In the class RedisQues the method checkQueues() is called periodically to notify not active / not empty queues to be processed. This process gets heavier the more queues exist.
When having multiple instances of RedisQues deployed, it will be called even more.
See logs from one of our hosts. We have deployed 4 instances of RedisQues
2025-02-12 21:11:56,267 prod crush INFO RedisQues - periodic queue check is triggered now
2025-02-12 21:11:54,658 prod crush INFO RedisQues - periodic queue check is triggered now
2025-02-12 21:11:51,136 prod crush INFO RedisQues - periodic queue check is triggered now
2025-02-12 21:11:51,136 prod crush INFO RedisQues - periodic queue check is triggered now
2025-02-12 21:11:51,136 prod crush INFO RedisQues - periodic queue check is triggered now
2025-02-12 21:11:51,135 prod crush INFO RedisQues - periodic queue check is triggered now
2025-02-12 21:11:51,135 prod crush INFO RedisQues - periodic queue check is triggered now
2025-02-12 21:11:51,132 prod crush INFO RedisQues - periodic queue check is triggered now
You can see that the check is made multiple times almost at the same time.
@hiddenalpha you refactored this method some time ago to be more robust using UpperBoundParallel and PeriodicSkipScheduler. Maybe you could have another look to further improve this.
In the class
RedisQuesthe methodcheckQueues()is called periodically to notify not active / not empty queues to be processed. This process gets heavier the more queues exist.When having multiple instances of
RedisQuesdeployed, it will be called even more.See logs from one of our hosts. We have deployed 4 instances of
RedisQuesYou can see that the check is made multiple times almost at the same time.
@hiddenalpha you refactored this method some time ago to be more robust using
UpperBoundParallelandPeriodicSkipScheduler. Maybe you could have another look to further improve this.