Skip to content

Add e2e test suite for Airflow event-driven DAGs with Apache Kafka#64833

Draft
jason810496 wants to merge 2 commits intoapache:mainfrom
jason810496:ci/airflow-e2e-tests/event-driven-dag
Draft

Add e2e test suite for Airflow event-driven DAGs with Apache Kafka#64833
jason810496 wants to merge 2 commits intoapache:mainfrom
jason810496:ci/airflow-e2e-tests/event-driven-dag

Conversation

@jason810496
Copy link
Copy Markdown
Member

Why

Airflow 3 introduces event-driven DAGs with Assets, AssetWatchers, and MessageQueueTrigger to enable reactive, message-driven pipelines. This PR adds a comprehensive e2e
test suite to verify that Kafka-based event routing works correctly, including proper handling of malformed messages and dead-letter queues.

What

  • Dag pattern: Implemented a producer-consumer pattern where producer sends 9 messages (8 valid orders + 1 malformed) to a Kafka topic, and consumer triggers 9 times via AssetWatcher
  • Test verification: Consumer Dag uses retries (3), short-circuit on malformed messages, and routes failures to a dead-letter queue topic
  • CI integration: Extended selective-checks to detect event-driven test file changes and trigger the suite; added workflow inputs/outputs for test mode selection
  • Kafka integration: The kafka.yaml and update_run.sh are copied from breeze --integration kafka
Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below) Claude Code following the guidelines

@boring-cyborg boring-cyborg bot added area:dev-tools backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch labels Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant