Skip to content

Add IBM MQ provider#62790

Open
dabla wants to merge 17 commits intoapache:mainfrom
dabla:feature/ibm-mq-provider
Open

Add IBM MQ provider#62790
dabla wants to merge 17 commits intoapache:mainfrom
dabla:feature/ibm-mq-provider

Conversation

@dabla
Copy link
Copy Markdown
Contributor

@dabla dabla commented Mar 3, 2026

With the introduction of event-driven scheduling and the MessageQueueProvider abstraction in Airflow, it has become significantly easier to trigger DAGs from external message brokers as described in Astronomer's guide on event-driven scheduling.

Many enterprises still rely heavily on IBM MQ as their primary enterprise messaging backbone. However, at the moment there is no official Airflow provider supporting IBM MQ.

This implementation consists of:

  • An IBMMQHook
  • A MessageQueueProvider implementation for IBM MQ
  • The ability to trigger DAGs from IBM MQ events
  • Standard producer/consumer patterns from within tasks

This allows IBM MQ to function similarly to Kafka, SQS, etc., within the Airflow event-driven scheduling framework.

The implementation is built on top of the open-source IBM MQ Python wrapper:

  • IBM MQ Python ibmmq library

IBM has recently released and documented their modern Python binding here:

https://community.ibm.com/community/user/blogs/dylan-goode/2025/10/16/new-python-binding-for-ibm-mq

The hook supports:

  • Secure connections (TLS) (to be done)
  • Queue get/put operations
  • Configurable polling behaviour

The MessageQueueProvider implementation integrates with Airflow's event-driven scheduling so that DAGs can be triggered based on IBM MQ messages.

Why this might make sense:

  • IBM MQ is still widely used in regulated industries (banking, insurance, government).
  • Many enterprises using Airflow also run IBM MQ.
  • This would allow IBM MQ to be a first-class citizen in Airflow's event-driven ecosystem.
  • The dependency is officially maintained by IBM and open source.

I am willing to act as initial maintainer and code owner, of course this is purely a proposition.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@vincbeck
Copy link
Copy Markdown
Contributor

vincbeck commented Mar 3, 2026

Since this PR needs approval the community, maybe put in draft to avoid any accidental merge?

@dabla dabla marked this pull request as draft March 3, 2026 15:25
@jscheffl
Copy link
Copy Markdown
Contributor

jscheffl commented Mar 4, 2026

Looks good for me. Would you add yourself as CODEOWNER for this? Do you know somebody (even non committer) which would help as steward?

Comment thread providers/ibm/mq/provider.yaml
@dabla dabla force-pushed the feature/ibm-mq-provider branch from 563ed98 to 55b67e5 Compare March 10, 2026 19:25
@dabla dabla force-pushed the feature/ibm-mq-provider branch 2 times, most recently from 3c85c36 to 014dd51 Compare March 18, 2026 20:21
Copy link
Copy Markdown
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 5 new issues from the 30 commits pushed since last review.

Comment thread providers/ibm/mq/tests/unit/ibm/mq/hooks/test_mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/triggers/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/docs/connections/mq.rst Outdated
Comment thread providers/ibm/mq/provider.yaml
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/get_provider_info.py
Comment thread providers/ibm/mq/pyproject.toml
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/__init__.py
@dabla dabla force-pushed the feature/ibm-mq-provider branch 2 times, most recently from 47f961b to aceb1a5 Compare March 23, 2026 17:02
@kaxil kaxil requested a review from Copilot March 24, 2026 23:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new IBM MQ provider to Apache Airflow, integrating IBM MQ with Airflow’s event-driven scheduling via the MessageQueueProvider abstraction and adding an IBMMQHook plus an MQ trigger.

Changes:

  • Adds apache-airflow-providers-ibm-mq provider package (hook, trigger, message-queue provider) with unit/system tests.
  • Wires the new provider into the monorepo/workspace (extras, members, CI docker-compose test sources, mypy paths, labels, CODEOWNERS, issue template).
  • Adds provider documentation pages and updates spelling wordlist and Breeze docs artifacts.

Reviewed changes

Copilot reviewed 62 out of 70 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Implements IBMMQHook with sync/async consume/produce and reconnection/backoff.
providers/ibm/mq/src/airflow/providers/ibm/mq/triggers/mq.py Adds AwaitMessageTrigger that polls IBM MQ via the hook and yields TriggerEvent.
providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py Adds IBMMQMessageQueueProvider to integrate with common.messaging’s MessageQueueTrigger.
providers/ibm/mq/tests/** Adds unit tests for hook/trigger/provider and a system-test example DAG.
providers/ibm/mq/docs/**, providers/ibm/mq/README.rst Adds provider docs, connection docs, and generated README/changelog.
pyproject.toml, uv.lock Registers provider in workspace + extras and locks optional ibmmq dependency.
.github/*, scripts/ci/docker-compose/* CI wiring, labels, CODEOWNERS, and issue-template updates for the new provider.
airflow-core/docs/extra-packages-ref.rst Adds the ibm-mq extra to docs.

Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py
Comment thread providers/ibm/mq/docs/changelog.rst Outdated
Comment thread providers/ibm/mq/docs/changelog.rst Outdated
Comment thread providers/ibm/mq/tests/unit/ibm/mq/__init__.py Outdated
Comment thread providers/ibm/mq/tests/unit/ibm/mq/hooks/test_mq.py
Comment thread airflow-core/docs/extra-packages-ref.rst
Comment thread providers/ibm/mq/README.rst Outdated
@dabla dabla marked this pull request as ready for review March 25, 2026 18:58
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 56 out of 62 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (3)

providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py:1

  • stop_event is created once outside the retry loop and is unconditionally set in finally. After the first iteration, the event remains set permanently, causing subsequent retries to immediately short-circuit consume() (its while not stop_event.is_set() loop won’t run). Create a new threading.Event() per iteration (or clear it before each call), and only set it to signal the background thread when cancellation/early-exit is needed.
    providers/ibm/mq/tests/system/ibm/mq/example_dag_message_queue_trigger.py:1
  • This example DAG defines process_message but never calls it, so the DAG will end up with no tasks. Also, the MessageQueueTrigger argument name scheme= appears inconsistent with the provider docs snippet in queues/mq.py (which shows queue=)—one of these is likely incorrect and will break the example. Call process_message() inside the DAG context and align the trigger constructor argument name with the actual MessageQueueTrigger API used by the project.
    providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py:1
  • Logging the full raw message at WARNING level can leak sensitive payloads into logs and can be very large/binary (log noise / potential failures). Prefer logging only metadata (e.g., message length / a small prefix) and keep full payload logging at DEBUG if needed.

Comment thread providers/ibm/mq/provider.yaml Outdated
Comment thread airflow-core/docs/extra-packages-ref.rst
Comment thread providers/ibm/mq/docs/connections/mq.rst Outdated
Comment thread providers/ibm/mq/README.rst Outdated
@kaxil
Copy link
Copy Markdown
Member

kaxil commented Apr 10, 2026

Good review copilot :D

@kaxil
Copy link
Copy Markdown
Member

kaxil commented Apr 17, 2026

@dabla Could you squash all commits into 1 please?

@dabla dabla force-pushed the feature/ibm-mq-provider branch from 95f7f62 to e6a3207 Compare April 17, 2026 12:53
@dabla
Copy link
Copy Markdown
Contributor Author

dabla commented Apr 17, 2026

@dabla Could you squash all commits into 1 please?

done and did some fixes related to Github Copilot remarks

Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/tests/system/ibm/mq/example_dag_message_queue_trigger.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py Outdated
Comment thread providers/ibm/mq/tests/unit/ibm/mq/hooks/test_mq.py Outdated
dabla added 14 commits April 22, 2026 15:34
…for permanent failures

- Log quiet queues (no message) at DEBUG, connection broken at WARNING, and permanent errors at ERROR.
- Only suppress MQMIError for MQRC_NO_MSG_AVAILABLE; all other MQMIError are now raised immediately.
- Prevents infinite retries on permanent errors (e.g., unauthorized, unknown queue).
- Ensures operators receive clear signals for actionable failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants