After upgrading from KCL 2.x to 3.0.2, we observed a increase in `ConditionalCheckFailedException` our multi-stream consumer setup.

[yukyongsoo]

Environment

• KCL version: 3.0.2
• AWS SDK version: 2.29.24
• Lease Table: DynamoDB (on-demand)
• Number of KCL worker instances: 4
• Number of streams: 2 (using MultiStreamTracker and Non-fanOut)
• Shard count per stream: 4 and 1

ERROR MESSAGE

THREAD: LeaseRenewer-000N

The conditional request failed (Service: DynamoDb, Status Code: 400, Request ID: NDBIG6THU1RKAGN8RIKG5VL8F7VV4KQNSO5AEMVJF66Q9ASUAAJG)

STACK TRACE ( But It't not helpful )

software.amazon.awssdk.services.dynamodb.model.conditionalcheckfailedexception: The conditional request failed (Service: DynamoDb, Status Code: 400, Request ID: 7L2OU96QQC38PQ229LAHQQ9BT7VV4KQNSO5AEMVJF66Q9ASUAAJG)
software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException: The conditional request failed (Service: DynamoDb, Status Code: 400, Request ID: 7L2OU96QQC38PQ229LAHQQ9BT7VV4KQNSO5AEMVJF66Q9ASUAAJG)
at software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException$BuilderImpl.build(ConditionalCheckFailedException.java:236)
at software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException$BuilderImpl.build(ConditionalCheckFailedException.java:169)
at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.unmarshall(AwsJsonProtocolErrorUnmarshaller.java:95)
at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.handle(AwsJsonProtocolErrorUnmarshaller.java:71)
at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.handle(AwsJsonProtocolErrorUnmarshaller.java:42)
at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:95)
at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52)
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$DataCountingPublisher$1.onComplete(ResponseHandler.java:519)
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:254)
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:77)
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:375)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:402)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:833)

comparison data ( one day occurred )

2.X -> success: 151 fail: 0
3.X -> success: 255 fail: 321

Current Configuration ( The rest is default )

maxLeasesToStealAtOneTime: 3
idleTimeBetweenReadsInMillis: 10000

Change Configuration For Tested

maxLeasesToStealAtOneTime: 1
failoverTimeMillis: 20000

but still same result...

can we reduce this error? or we ignore this ?