Conversation
|
Since this PR needs approval the community, maybe put in draft to avoid any accidental merge? |
|
Looks good for me. Would you add yourself as CODEOWNER for this? Do you know somebody (even non committer) which would help as steward? |
563ed98 to
55b67e5
Compare
3c85c36 to
014dd51
Compare
47f961b to
aceb1a5
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a new IBM MQ provider to Apache Airflow, integrating IBM MQ with Airflow’s event-driven scheduling via the MessageQueueProvider abstraction and adding an IBMMQHook plus an MQ trigger.
Changes:
- Adds
apache-airflow-providers-ibm-mqprovider package (hook, trigger, message-queue provider) with unit/system tests. - Wires the new provider into the monorepo/workspace (extras, members, CI docker-compose test sources, mypy paths, labels, CODEOWNERS, issue template).
- Adds provider documentation pages and updates spelling wordlist and Breeze docs artifacts.
Reviewed changes
Copilot reviewed 62 out of 70 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py |
Implements IBMMQHook with sync/async consume/produce and reconnection/backoff. |
providers/ibm/mq/src/airflow/providers/ibm/mq/triggers/mq.py |
Adds AwaitMessageTrigger that polls IBM MQ via the hook and yields TriggerEvent. |
providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py |
Adds IBMMQMessageQueueProvider to integrate with common.messaging’s MessageQueueTrigger. |
providers/ibm/mq/tests/** |
Adds unit tests for hook/trigger/provider and a system-test example DAG. |
providers/ibm/mq/docs/**, providers/ibm/mq/README.rst |
Adds provider docs, connection docs, and generated README/changelog. |
pyproject.toml, uv.lock |
Registers provider in workspace + extras and locks optional ibmmq dependency. |
.github/*, scripts/ci/docker-compose/* |
CI wiring, labels, CODEOWNERS, and issue-template updates for the new provider. |
airflow-core/docs/extra-packages-ref.rst |
Adds the ibm-mq extra to docs. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 56 out of 62 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (3)
providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py:1
stop_eventis created once outside the retry loop and is unconditionally set infinally. After the first iteration, the event remains set permanently, causing subsequent retries to immediately short-circuitconsume()(itswhile not stop_event.is_set()loop won’t run). Create a newthreading.Event()per iteration (or clear it before each call), and only set it to signal the background thread when cancellation/early-exit is needed.
providers/ibm/mq/tests/system/ibm/mq/example_dag_message_queue_trigger.py:1- This example DAG defines
process_messagebut never calls it, so the DAG will end up with no tasks. Also, theMessageQueueTriggerargument namescheme=appears inconsistent with the provider docs snippet inqueues/mq.py(which showsqueue=)—one of these is likely incorrect and will break the example. Callprocess_message()inside the DAG context and align the trigger constructor argument name with the actualMessageQueueTriggerAPI used by the project.
providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py:1 - Logging the full raw
messageat WARNING level can leak sensitive payloads into logs and can be very large/binary (log noise / potential failures). Prefer logging only metadata (e.g., message length / a small prefix) and keep full payload logging at DEBUG if needed.
|
Good review copilot :D |
|
@dabla Could you squash all commits into 1 please? |
95f7f62 to
e6a3207
Compare
done and did some fixes related to Github Copilot remarks |
…t both cases for queue and scheme
…ult one was provided
…for permanent failures - Log quiet queues (no message) at DEBUG, connection broken at WARNING, and permanent errors at ERROR. - Only suppress MQMIError for MQRC_NO_MSG_AVAILABLE; all other MQMIError are now raised immediately. - Prevents infinite retries on permanent errors (e.g., unauthorized, unknown queue). - Ensures operators receive clear signals for actionable failures.
…or if a result is found" This reverts commit 0522933
With the introduction of event-driven scheduling and the MessageQueueProvider abstraction in Airflow, it has become significantly easier to trigger DAGs from external message brokers as described in Astronomer's guide on event-driven scheduling.
Many enterprises still rely heavily on IBM MQ as their primary enterprise messaging backbone. However, at the moment there is no official Airflow provider supporting IBM MQ.
This implementation consists of:
This allows IBM MQ to function similarly to Kafka, SQS, etc., within the Airflow event-driven scheduling framework.
The implementation is built on top of the open-source IBM MQ Python wrapper:
IBM has recently released and documented their modern Python binding here:
https://community.ibm.com/community/user/blogs/dylan-goode/2025/10/16/new-python-binding-for-ibm-mq
The hook supports:
The MessageQueueProvider implementation integrates with Airflow's event-driven scheduling so that DAGs can be triggered based on IBM MQ messages.
Why this might make sense:
I am willing to act as initial maintainer and code owner, of course this is purely a proposition.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.