Add e2e test suite for Airflow event-driven DAGs with Apache Kafka by jason810496 · Pull Request #64833 · apache/airflow

jason810496 · 2026-04-07T08:51:40Z

Why

Airflow 3 introduces event-driven DAGs with Assets, AssetWatchers, and MessageQueueTrigger to enable reactive, message-driven pipelines. This PR adds a comprehensive e2e
test suite to verify that Kafka-based event routing works correctly, including proper handling of malformed messages and dead-letter queues.

What

Dag pattern: Implemented a producer-consumer pattern where producer sends 9 messages (8 valid orders + 1 malformed) to a Kafka topic, and consumer triggers 9 times via AssetWatcher
Test verification: Consumer Dag uses retries (3), short-circuit on malformed messages, and routes failures to a dead-letter queue topic
CI integration: Extended selective-checks to detect event-driven test file changes and trigger the suite; added workflow inputs/outputs for test mode selection
Kafka integration: The kafka.yaml and update_run.sh are copied from breeze --integration kafka

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below) Claude Code following the guidelines

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds an event-driven (Kafka-backed) E2E test suite for Airflow 3 Assets/AssetWatchers and wires it into Breeze selective checks + CI workflows so the suite runs when relevant files change.

Changes:

Introduces Kafka producer/consumer example DAGs and a new E2E test validating triggers, retries/DLQ behavior, and Kafka offsets.
Adds Kafka docker-compose + init script and updates the E2E harness to spin up Kafka and create required topics.
Extends Breeze selective checks and GitHub workflows to support an event_driven E2E test mode.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
dev/breeze/src/airflow_breeze/utils/selective_checks.py	Adds a new selective-checks file group + toggle for event-driven E2E tests.
dev/breeze/src/airflow_breeze/commands/testing_commands.py	Extends `--e2e-test-mode` choices with `event_driven`.
dev/breeze/doc/images/output_testing_airflow-e2e-tests.txt	Updates generated CLI help output artifact.
dev/breeze/doc/images/output_testing_airflow-e2e-tests.svg	Updates generated CLI help output artifact (includes `event_driven`).
airflow-e2e-tests/tests/airflow_e2e_tests/event_driven_tests/test_event_driven.py	Adds Kafka event-driven E2E test validating DAG triggers + Kafka offsets.
airflow-e2e-tests/tests/airflow_e2e_tests/event_driven_tests/init.py	Adds package init for the new test module.
airflow-e2e-tests/tests/airflow_e2e_tests/e2e_test_utils/clients.py	Adds `list_dag_runs` helper used by the new test.
airflow-e2e-tests/tests/airflow_e2e_tests/dags/example_event_driven.py	Adds example producer/consumer event-driven DAGs (Assets + Kafka trigger + DLQ).
airflow-e2e-tests/tests/airflow_e2e_tests/constants.py	Adds Kafka docker directory path constant.
airflow-e2e-tests/tests/airflow_e2e_tests/conftest.py	Adds `event_driven` mode setup, Kafka file copying, and topic creation.
airflow-e2e-tests/docker/kafka/update_run.sh	Adds broker init script for Kafka container startup.
airflow-e2e-tests/docker/kafka.yml	Adds Kafka broker docker-compose definition for E2E runs.
.github/workflows/ci-amd-arm.yml	Plumbs through the new selective-check output for event-driven E2E tests.
.github/workflows/airflow-e2e-tests.yml	Updates workflow input docs to include `event_driven` mode.
.github/workflows/additional-prod-image-tests.yml	Adds a new job to run event-driven E2E tests when selected.

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 6 comments.

Copilot · 2026-04-28T13:38:32Z

+        # 5. All 9 DAG runs should succeed
+        for run in consumer_runs:
+            assert run["state"] == "success", (
+                f"Expected all consumer runs to succeed, but run {run['dag_run_id']} "
+                f"has state '{run['state']}'"
+            )
+


This test asserts every consumer DAG run is success, but later it expects one run to have a process_message task in failed state. In Airflow, a DagRun is marked failed if any task instance is in a failed state once the run finishes (see DagRun.update_state). Either adjust the expectations to allow one failed DagRun, or change the consumer DAG logic so the malformed message path does not leave a task instance in failed while still producing to the DLQ.

Suggested change

# 5. All 9 DAG runs should succeed

for run in consumer_runs:

assert run["state"] == "success", (

f"Expected all consumer runs to succeed, but run {run['dag_run_id']} "

f"has state '{run['state']}'"

)

# 5. Expect 8 successful consumer runs and 1 failed run for the malformed message

successful_runs = [run for run in consumer_runs if run["state"] == "success"]

failed_runs = [run for run in consumer_runs if run["state"] == "failed"]

assert len(successful_runs) == 8, (

f"Expected 8 successful consumer runs, got {len(successful_runs)}: "

f"{[(run['dag_run_id'], run['state']) for run in consumer_runs]}"

)

assert len(failed_runs) == 1, (

f"Expected 1 failed consumer run for the DLQ path, got {len(failed_runs)}: "

f"{[(run['dag_run_id'], run['state']) for run in consumer_runs]}"

)

Copilot · 2026-04-28T13:38:33Z

+            r"^providers/apache-kafka/.*",
+            r"^providers/common-messaging/.*",


The provider path regexes here don’t match the repository layout. Providers live under providers/apache/kafka/ and providers/common/messaging/, so these patterns will never match and CI selective-checks won’t enable the event-driven e2e suite when the Kafka/common-messaging providers change. Update the regexes to the correct directory paths.

Suggested change

r"^providers/apache-kafka/.*",

r"^providers/common-messaging/.*",

r"^providers/apache/kafka/.*",

r"^providers/common/messaging/.*",

Copilot · 2026-04-28T13:38:33Z



+def _copy_kafka_files(tmp_dir):
+    """Copy Kafka compose file, init script, and provider source into the temp directory."""


This docstring says it copies “provider source” into the temp directory, but the function only copies kafka.yml and update_run.sh. Please either update the docstring to match what’s actually being copied, or add the missing copy step if provider source is required for the integration.

Suggested change

"""Copy Kafka compose file, init script, and provider source into the temp directory."""

"""Copy the Kafka compose file and update script into the temp directory."""

Copilot · 2026-04-28T13:38:33Z

+"""Airflow Kafka connection
+AIRFLOW_CONN_KAFKA_DEFAULT='{
+  "conn_type": "general",
+  "extra": {
+    "bootstrap.servers": "broker:29092",
+    "group.id": "kafka_default_group",
+    "security.protocol": "PLAINTEXT",
+    "enable.auto.commit": false,
+    "auto.offset.reset": "latest"
+  }
+}'
+"""
+"""Kafka Command to verify messages are being produced to the topic:
+
+# Create Topic
+/bin/kafka-topics --bootstrap-server broker:29092 --create --topic fizz_buzz
+/bin/kafka-topics --bootstrap-server broker:29092 --create --topic dlq
+
+
+# Get offsets for the topic to verify messages are being produced
+/bin/kafka-get-offsets --bootstrap-server broker:29092 --topic fizz_buzz
+/bin/kafka-get-offsets --bootstrap-server broker:29092 --topic dlq
+
+# List consumer groups to verify our consumer group is being registered
+/bin/kafka-consumer-groups --bootstrap-server broker:29092 --list


These triple-quoted strings are standalone expression statements (not module docstrings), so they execute as no-ops at import time and are easy to miss/forget. Convert them to real comments, a proper module docstring, or constants referenced from documentation/tests.

Suggested change

"""Airflow Kafka connection

AIRFLOW_CONN_KAFKA_DEFAULT='{

"conn_type": "general",

"extra": {

"bootstrap.servers": "broker:29092",

"group.id": "kafka_default_group",

"security.protocol": "PLAINTEXT",

"enable.auto.commit": false,

"auto.offset.reset": "latest"

}

}'

"""

"""Kafka Command to verify messages are being produced to the topic:

# Create Topic

/bin/kafka-topics --bootstrap-server broker:29092 --create --topic fizz_buzz

/bin/kafka-topics --bootstrap-server broker:29092 --create --topic dlq

# Get offsets for the topic to verify messages are being produced

/bin/kafka-get-offsets --bootstrap-server broker:29092 --topic fizz_buzz

/bin/kafka-get-offsets --bootstrap-server broker:29092 --topic dlq

# List consumer groups to verify our consumer group is being registered

/bin/kafka-consumer-groups --bootstrap-server broker:29092 --list

# Airflow Kafka connection

# AIRFLOW_CONN_KAFKA_DEFAULT='{

# "conn_type": "general",

# "extra": {

# "bootstrap.servers": "broker:29092",

# "group.id": "kafka_default_group",

# "security.protocol": "PLAINTEXT",

# "enable.auto.commit": false,

# "auto.offset.reset": "latest"

# }

# }'

#

# Kafka commands to verify messages are being produced to the topic:

#

# Create Topic

# /bin/kafka-topics --bootstrap-server broker:29092 --create --topic fizz_buzz

# /bin/kafka-topics --bootstrap-server broker:29092 --create --topic dlq

#

# Get offsets for the topic to verify messages are being produced

# /bin/kafka-get-offsets --bootstrap-server broker:29092 --topic fizz_buzz

# /bin/kafka-get-offsets --bootstrap-server broker:29092 --topic dlq

#

# List consumer groups to verify our consumer group is being registered

# /bin/kafka-consumer-groups --bootstrap-server broker:29092 --list

#

Copilot · 2026-04-28T13:38:34Z

+"""Airflow Kafka connection
+AIRFLOW_CONN_KAFKA_DEFAULT='{
+  "conn_type": "general",
+  "extra": {
+    "bootstrap.servers": "broker:29092",
+    "group.id": "kafka_default_group",
+    "security.protocol": "PLAINTEXT",
+    "enable.auto.commit": false,
+    "auto.offset.reset": "latest"
+  }
+}'
+"""
+"""Kafka Command to verify messages are being produced to the topic:
+
+# Create Topic
+/bin/kafka-topics --bootstrap-server broker:29092 --create --topic fizz_buzz
+/bin/kafka-topics --bootstrap-server broker:29092 --create --topic dlq
+
+
+# Get offsets for the topic to verify messages are being produced
+/bin/kafka-get-offsets --bootstrap-server broker:29092 --topic fizz_buzz
+/bin/kafka-get-offsets --bootstrap-server broker:29092 --topic dlq
+
+# List consumer groups to verify our consumer group is being registered
+/bin/kafka-consumer-groups --bootstrap-server broker:29092 --list
+
+# Get current offsets for the consumer group to verify messages are being consumed
+/bin/kafka-consumer-groups --bootstrap-server broker:29092 --describe --group group_1
+"""


Same issue here: this is a standalone triple-quoted string (a no-op expression), not a docstring. Prefer comments or documentation in a README/rst, or fold this into the module docstring if it’s meant to be user-facing guidance.

Suggested change

"""Airflow Kafka connection

AIRFLOW_CONN_KAFKA_DEFAULT='{

"conn_type": "general",

"extra": {

"bootstrap.servers": "broker:29092",

"group.id": "kafka_default_group",

"security.protocol": "PLAINTEXT",

"enable.auto.commit": false,

"auto.offset.reset": "latest"

}

}'

"""

"""Kafka Command to verify messages are being produced to the topic:

# Create Topic

/bin/kafka-topics --bootstrap-server broker:29092 --create --topic fizz_buzz

/bin/kafka-topics --bootstrap-server broker:29092 --create --topic dlq

# Get offsets for the topic to verify messages are being produced

/bin/kafka-get-offsets --bootstrap-server broker:29092 --topic fizz_buzz

/bin/kafka-get-offsets --bootstrap-server broker:29092 --topic dlq

# List consumer groups to verify our consumer group is being registered

/bin/kafka-consumer-groups --bootstrap-server broker:29092 --list

# Get current offsets for the consumer group to verify messages are being consumed

/bin/kafka-consumer-groups --bootstrap-server broker:29092 --describe --group group_1

"""

# Airflow Kafka connection:

# AIRFLOW_CONN_KAFKA_DEFAULT='{

# "conn_type": "general",

# "extra": {

# "bootstrap.servers": "broker:29092",

# "group.id": "kafka_default_group",

# "security.protocol": "PLAINTEXT",

# "enable.auto.commit": false,

# "auto.offset.reset": "latest"

# }

# }'

#

# Kafka commands to verify messages are being produced to the topic:

#

# Create Topic

# /bin/kafka-topics --bootstrap-server broker:29092 --create --topic fizz_buzz

# /bin/kafka-topics --bootstrap-server broker:29092 --create --topic dlq

#

# Get offsets for the topic to verify messages are being produced

# /bin/kafka-get-offsets --bootstrap-server broker:29092 --topic fizz_buzz

# /bin/kafka-get-offsets --bootstrap-server broker:29092 --topic dlq

#

# List consumer groups to verify our consumer group is being registered

# /bin/kafka-consumer-groups --bootstrap-server broker:29092 --list

#

# Get current offsets for the consumer group to verify messages are being consumed

# /bin/kafka-consumer-groups --bootstrap-server broker:29092 --describe --group group_1

Copilot · 2026-04-28T13:38:34Z

+            print(f"Asset event: {event}")
+            process_one_message(cast("str", event.extra["payload"]))
+        return True


typing.cast expects a type as its first argument, not a string. Using cast("str", ...) defeats static checking and is inconsistent with normal typing usage. Use cast(str, ...) or (preferably) validate/cast the payload more explicitly if it can be non-string.

- Use timedelta(seconds=1) for retry_delay instead of bare int - Return exactly expected_count runs from _wait_for_consumer_dag_runs - Replace fixed 30s sleep with polling for Kafka consumer group registration - Pin --partitions 1 --replication-factor 1 for deterministic topic creation - Add kafka.yml pattern to selective checks file group - Remove container_name and unused JMX port from kafka.yml - Regenerate breeze CLI doc images

jason810496

The CI failure require additional PR to fix the AwaitTrigger first:

  airflow-triggerer-1  | 2026-05-02T15:25:00.831436Z [error    ] Trigger ID 1 exited with error 'AwaitMessageTrigger' object has no attribute '_task_instance' [airflow.jobs.triggerer_job_runner] error_detail=[{'exc_type': 'AttributeError', 'exc_value': "'AwaitMessageTrigger' object has no attribute '_task_instance'", 'exc_notes': [], 'syntax_error': None, 'is_cause': False, 'frames': [{'filename': '/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/triggerer_job_runner.py', 'lineno': 1238, 'name': 'cleanup_finished_triggers'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/greenback/_impl.py', 'lineno': 119, 'name': 'greenback_shim'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/greenback/_impl.py', 'lineno': 208, 'name': '_greenback_shim'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/greenback/_impl.py', 'lineno': 84, 'name': 'trampoline'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/outcome/_impl.py', 'lineno': 185, 'name': 'send'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/triggerer_job_runner.py', 'lineno': 1387, 'name': 'run_trigger'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/airflow/triggers/base.py', 'lineno': 104, 'name': 'task_instance'}], 'is_group': False, 'exceptions': []}] loc=triggerer_job_runner.py:929
  airflow-triggerer-1  | 2026-05-02T15:25:00.832067Z [error    ] Trigger exited without sending an event. Dependent tasks will be failed. [airflow.jobs.triggerer_job_runner] loc=triggerer_job_runner.py:929 name='ID 1'
  airflow-triggerer-1  | 2026-05-02T15:25:02.858280Z [error    ] Trigger ID 1 exited with error 'AwaitMessageTrigger' object has no attribute '_task_instance' [airflow.jobs.triggerer_job_runner] error_detail=[{'exc_type': 'AttributeError', 'exc_value': "'AwaitMessageTrigger' object has no attribute '_task_instance'", 'exc_notes': [], 'syntax_error': None, 'is_cause': False, 'frames': [{'filename': '/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/triggerer_job_runner.py', 'lineno': 1238, 'name': 'cleanup_finished_triggers'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/greenback/_impl.py', 'lineno': 119, 'name': 'greenback_shim'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/greenback/_impl.py', 'lineno': 208, 'name': '_greenback_shim'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/greenback/_impl.py', 'lineno': 84, 'name': 'trampoline'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/outcome/_impl.py', 'lineno': 185, 'name': 'send'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/triggerer_job_runner.py', 'lineno': 1387, 'name': 'run_trigger'}, {'filename': '/home/airflow/.local/lib/python3.10/site-packages/airflow/triggers/base.py', 'lineno': 104, 'name': 'task_instance'}], 'is_group': False, 'exceptions': []}] loc=triggerer_job_runner.py:929

parkhojeong

i reviewed about term usage :)

parkhojeong · 2026-05-03T09:19:37Z

+        """Poll until *dag_id* is registered (parsed & serialized) and reachable via the API.
+
+        On remote CI runners the DAG processor may need extra time to parse & serialize
+        new DAG files, so calling ``un_pause_dag`` immediately can return a 404. Retry


Suggested change

new DAG files, so calling ``un_pause_dag`` immediately can return a 404. Retry

new Dag files, so calling ``un_pause_dag`` immediately can return a 404. Retry

parkhojeong · 2026-05-03T09:19:44Z

+
+        On remote CI runners the DAG processor may need extra time to parse & serialize
+        new DAG files, so calling ``un_pause_dag`` immediately can return a 404. Retry
+        until the DAG exists or *timeout* seconds elapse.


Suggested change

until the DAG exists or *timeout* seconds elapse.

until the Dag exists or *timeout* seconds elapse.

parkhojeong · 2026-05-03T09:19:53Z

+                    raise
+                last_error = exc
+            time.sleep(check_interval)
+        raise TimeoutError(f"DAG {dag_id} was not registered within {timeout}s. Last error: {last_error}")


Suggested change

raise TimeoutError(f"DAG {dag_id} was not registered within {timeout}s. Last error: {last_error}")

raise TimeoutError(f"Dag {dag_id} was not registered within {timeout}s. Last error: {last_error}")

parkhojeong · 2026-05-03T09:20:03Z

        )

+    def list_dag_runs(self, dag_id: str, limit: int = 100):
+        """List DAG runs for a given DAG."""


Suggested change

"""List DAG runs for a given DAG."""

"""List Dag runs for a given DAG."""

parkhojeong · 2026-05-03T09:20:12Z

+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""E2E tests for the event-driven DAG pattern using Apache Kafka.


Suggested change

"""E2E tests for the event-driven DAG pattern using Apache Kafka.

"""E2E tests for the event-driven Dag pattern using Apache Kafka.

parkhojeong · 2026-05-03T09:20:51Z

+        """Return a mapping of task_id -> state for a consumer DAG run."""
+        response = self.airflow_client.get_task_instances(CONSUMER_DAG_ID, run_id)


Suggested change

"""Return a mapping of task_id -> state for a consumer DAG run."""

response = self.airflow_client.get_task_instances(CONSUMER_DAG_ID, run_id)

"""Return a mapping of task_id -> state for a consumer Dag run."""

response = self.airflow_client.get_task_instances(CONSUMER_DAG_ID, run_id)

parkhojeong · 2026-05-03T09:21:05Z

+        1. Unpause the consumer DAG so the triggerer starts the AssetWatcher.
+        2. Wait for the Kafka MessageQueueTrigger to begin polling.
+        3. Trigger the producer DAG and wait for it to succeed.
+        4. Wait for 9 consumer DAG runs to reach a terminal state.
+        5. All 9 DAG runs succeed. Verify task-level behavior:


Suggested change

1. Unpause the consumer DAG so the triggerer starts the AssetWatcher.

2. Wait for the Kafka MessageQueueTrigger to begin polling.

3. Trigger the producer DAG and wait for it to succeed.

4. Wait for 9 consumer DAG runs to reach a terminal state.

5. All 9 DAG runs succeed. Verify task-level behavior:

1. Unpause the consumer Dag so the triggerer starts the AssetWatcher.

2. Wait for the Kafka MessageQueueTrigger to begin polling.

3. Trigger the producer Dag and wait for it to succeed.

4. Wait for 9 consumer Dag runs to reach a terminal state.

5. All 9 Dag runs succeed. Verify task-level behavior:

parkhojeong · 2026-05-03T09:21:13Z

+
+        # 3. Trigger producer and wait for it to complete
+        producer_state = self.airflow_client.trigger_dag_and_wait(PRODUCER_DAG_ID)
+        assert producer_state == "success", f"Producer DAG did not succeed. Final state: {producer_state}"


Suggested change

assert producer_state == "success", f"Producer DAG did not succeed. Final state: {producer_state}"

assert producer_state == "success", f"Producer Dag did not succeed. Final state: {producer_state}"

parkhojeong · 2026-05-03T09:21:19Z

+        producer_state = self.airflow_client.trigger_dag_and_wait(PRODUCER_DAG_ID)
+        assert producer_state == "success", f"Producer DAG did not succeed. Final state: {producer_state}"
+
+        # 4. Wait for all 9 consumer DAG runs


Suggested change

# 4. Wait for all 9 consumer DAG runs

# 4. Wait for all 9 consumer Dag runs

parkhojeong · 2026-05-03T09:21:33Z

+
+
+def _create_kafka_topics(compose_instance):
+    """Create Kafka topics required by the event-driven DAG."""


Suggested change

"""Create Kafka topics required by the event-driven DAG."""

"""Create Kafka topics required by the event-driven Dag."""

boring-cyborg Bot added area:dev-tools backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch labels Apr 7, 2026

kaxil requested a review from Copilot April 10, 2026 19:55

Copilot AI reviewed Apr 10, 2026

View reviewed changes

jason810496 force-pushed the ci/airflow-e2e-tests/event-driven-dag branch from 351c307 to f72d2aa Compare April 28, 2026 13:26

jason810496 requested a review from Copilot April 28, 2026 13:29

Copilot started reviewing on behalf of jason810496 April 28, 2026 13:29 View session

Copilot AI reviewed Apr 28, 2026

View reviewed changes

jason810496 force-pushed the ci/airflow-e2e-tests/event-driven-dag branch 2 times, most recently from 5eb37d7 to 9ef8404 Compare May 1, 2026 15:36

jason810496 added 7 commits May 2, 2026 22:34

Add e2e test suite for Airflow event-driven DAGs with Apache Kafka

72319b7

Add 'event_driven' option to E2E test mode in testing_commands.py

b8aabaa

Fix AwaitTrigger and airflow e2e test

4425edf

Add retry for un_pause_dag method

9901780

Add additional requirements for Kafka integration in event-driven setup

9e77b37

Add pytest output

8e10497

jason810496 force-pushed the ci/airflow-e2e-tests/event-driven-dag branch from 9ef8404 to 8e10497 Compare May 2, 2026 14:35

jason810496 commented May 3, 2026

View reviewed changes

jason810496 mentioned this pull request May 3, 2026

Fix AwaitMessageTrigger missing _task_instance attribute #66302

Draft

1 task

parkhojeong reviewed May 3, 2026

View reviewed changes

-        # 5. All 9 DAG runs should succeed
-        for run in consumer_runs:
-            assert run["state"] == "success", (
-                f"Expected all consumer runs to succeed, but run {run['dag_run_id']} "
-                f"has state '{run['state']}'"
-            )
+        # 5. Expect 8 successful consumer runs and 1 failed run for the malformed message
+        successful_runs = [run for run in consumer_runs if run["state"] == "success"]
+        failed_runs = [run for run in consumer_runs if run["state"] == "failed"]
+        assert len(successful_runs) == 8, (
+            f"Expected 8 successful consumer runs, got {len(successful_runs)}: "
+            f"{[(run['dag_run_id'], run['state']) for run in consumer_runs]}"
+        )
+        assert len(failed_runs) == 1, (
+            f"Expected 1 failed consumer run for the DLQ path, got {len(failed_runs)}: "
+            f"{[(run['dag_run_id'], run['state']) for run in consumer_runs]}"
+        )

		r"^providers/apache-kafka/.*",
		r"^providers/common-messaging/.*",



		def _copy_kafka_files(tmp_dir):
		"""Copy Kafka compose file, init script, and provider source into the temp directory."""

	"""Copy Kafka compose file, init script, and provider source into the temp directory."""
	"""Copy the Kafka compose file and update script into the temp directory."""

	new DAG files, so calling ``un_pause_dag`` immediately can return a 404. Retry
	new Dag files, so calling ``un_pause_dag`` immediately can return a 404. Retry

	until the DAG exists or timeout seconds elapse.
	until the Dag exists or timeout seconds elapse.

	raise TimeoutError(f"DAG {dag_id} was not registered within {timeout}s. Last error: {last_error}")
	raise TimeoutError(f"Dag {dag_id} was not registered within {timeout}s. Last error: {last_error}")

	"""List DAG runs for a given DAG."""
	"""List Dag runs for a given DAG."""

	"""E2E tests for the event-driven DAG pattern using Apache Kafka.
	"""E2E tests for the event-driven Dag pattern using Apache Kafka.

		"""Return a mapping of task_id -> state for a consumer DAG run."""
		response = self.airflow_client.get_task_instances(CONSUMER_DAG_ID, run_id)

	assert producer_state == "success", f"Producer DAG did not succeed. Final state: {producer_state}"
	assert producer_state == "success", f"Producer Dag did not succeed. Final state: {producer_state}"

	# 4. Wait for all 9 consumer DAG runs
	# 4. Wait for all 9 consumer Dag runs



		def _create_kafka_topics(compose_instance):
		"""Create Kafka topics required by the event-driven DAG."""

Conversation

jason810496 commented Apr 7, 2026

Why

What

Was generative AI tooling used to co-author this PR?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

parkhojeong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants