Hi,
we have 4 shards for one Kinesis stream. We have 4 microservices sharing those 4 shards. We have the following config:
enhancedFanOutEnabled: false
pollingIntervalMillis: 3000
failureMaxRetries: 3
failureBackoffTimeMillis: 3000
failoverTimeMillis: 15000
2025-07-15 12:30:28 ERROR software.amazon.kinesis.retrieval.polling.PrefetchRecordsPublisher:590 event-stream:shardId-000000000003 : Exception thrown while fetching records from Kinesis software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for Shard - nnnnnnnnnnnn/event-stream/shardId-000000000003 (Service: Kinesis, Status Code: 400, Request ID: d6285a7b-8624-e11f-b6c7-e5ffc5432b9b, Extended Request ID: PydAvRQwL1u4yZhxK/kY2fLsIeR4yk6gIRR5aXD5RSzPlj43nmrWn1AoR9wxQct7I0uvoYMlYYBBTLmQhwxNitZCBMMVP+cV) (SDK Attempt Count: 4) at software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException$BuilderImpl.build(ProvisionedThroughputExceededException.java:156) at software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException$BuilderImpl.build(ProvisionedThroughputExceededException.java:104) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:127) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:110) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:253) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Rate exceeded for Shard - nnnnnnnnnnnn/event-stream/shardId-000000000003 (Service: Kinesis, Status Code: 400, Request ID: c0802fb4-066c-d38e-a06f-9035450b190a, Extended Request ID: y340KyG6KLh5rx5W2alm1blxNUYnYea43QwWYu432LxyfLRt8OCCbYIYD54joYLBvcGv1sK7CMiY+H+ingPEXclMIm8BWemz) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Rate exceeded for Shard - nnnnnnnnnnnn/event-stream/shardId-000000000003 (Service: Kinesis, Status Code: 400, Request ID: cef3c2dd-0e5b-4ff2-ae1c-7d5c4d3c8576, Extended Request ID: +hoPk0IOUCEG5+U5RyU1iOZBk5aWb6KFVm4VZCt4S0uL78eFKTs+vZPegzeu4gB3b6qbiddgif7ncIdOX9ao/Jz3acUL15Rs) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Rate exceeded for Shard - nnnnnnnnnnnn/event-stream/shardId-000000000003 (Service: Kinesis, Status Code: 400, Request ID: d201a160-b773-1d98-b2ee-1ee2f414d71c, Extended Request ID: NrS3ZuISkENliAwGGvUI9wS94tAgA/iWYphmZ2Rsirb1wpifKIJr0dmqvrTZQQK+Xj0T/dzhujyZaR+OnHOuBe4S9xvcweTm)
The configuration seems pretty standard, so how do we tweak to avoid this from happening?
Hi,
we have 4 shards for one Kinesis stream. We have 4 microservices sharing those 4 shards. We have the following config:
enhancedFanOutEnabled: false
pollingIntervalMillis: 3000
failureMaxRetries: 3
failureBackoffTimeMillis: 3000
failoverTimeMillis: 15000
2025-07-15 12:30:28 ERROR software.amazon.kinesis.retrieval.polling.PrefetchRecordsPublisher:590 event-stream:shardId-000000000003 : Exception thrown while fetching records from Kinesis software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for Shard - nnnnnnnnnnnn/event-stream/shardId-000000000003 (Service: Kinesis, Status Code: 400, Request ID: d6285a7b-8624-e11f-b6c7-e5ffc5432b9b, Extended Request ID: PydAvRQwL1u4yZhxK/kY2fLsIeR4yk6gIRR5aXD5RSzPlj43nmrWn1AoR9wxQct7I0uvoYMlYYBBTLmQhwxNitZCBMMVP+cV) (SDK Attempt Count: 4) at software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException$BuilderImpl.build(ProvisionedThroughputExceededException.java:156) at software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException$BuilderImpl.build(ProvisionedThroughputExceededException.java:104) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:127) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:110) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:253) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Rate exceeded for Shard - nnnnnnnnnnnn/event-stream/shardId-000000000003 (Service: Kinesis, Status Code: 400, Request ID: c0802fb4-066c-d38e-a06f-9035450b190a, Extended Request ID: y340KyG6KLh5rx5W2alm1blxNUYnYea43QwWYu432LxyfLRt8OCCbYIYD54joYLBvcGv1sK7CMiY+H+ingPEXclMIm8BWemz) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Rate exceeded for Shard - nnnnnnnnnnnn/event-stream/shardId-000000000003 (Service: Kinesis, Status Code: 400, Request ID: cef3c2dd-0e5b-4ff2-ae1c-7d5c4d3c8576, Extended Request ID: +hoPk0IOUCEG5+U5RyU1iOZBk5aWb6KFVm4VZCt4S0uL78eFKTs+vZPegzeu4gB3b6qbiddgif7ncIdOX9ao/Jz3acUL15Rs) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Rate exceeded for Shard - nnnnnnnnnnnn/event-stream/shardId-000000000003 (Service: Kinesis, Status Code: 400, Request ID: d201a160-b773-1d98-b2ee-1ee2f414d71c, Extended Request ID: NrS3ZuISkENliAwGGvUI9wS94tAgA/iWYphmZ2Rsirb1wpifKIJr0dmqvrTZQQK+Xj0T/dzhujyZaR+OnHOuBe4S9xvcweTm)The configuration seems pretty standard, so how do we tweak to avoid this from happening?