Skip to content

Implement inter-topic dependency management #45

@stefan-cardnell-rh

Description

@stefan-cardnell-rh

Add an optional functionality for inter-topic dependency management, to resolve the following issues:

  1. If an Event 1 for entity A fails in a topic and is sent to the non-blocking retry or dead letter queue, it may be desired that Event 2 for the same entity A should also not be processed until Event 1 succeeds (Event 2 should be put in a holding queue). Essentially, the order of events for the same entity needs to be respected.
  2. It could be that messages in some topics should not be processed before messages in other topics. These dependent messages should be put in a holding queue until their dependency in the other topic has been processed.

"Entity" can be determined by the message key normally, but ideally it should be overideable.

Sources for implementation:

  1. https://www.confluent.io/blog/error-handling-patterns-in-kafka/#pattern-4
  2. https://medium.com/altitudehq/kafka-retries-and-maintain-order-of-retry-events-313482044351 - see "Maintain order of retry events"
  3. https://www.youtube.com/watch?v=FO2ptQNQKhM&ab_channel=DanielTammadge

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions