-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding support for Handling backpressure on KPLMesageHandler for KPL #249
Labels
Milestone
Comments
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 8, 2024
…plMessageHandler. Added Unit Tests.
Opened up a PR for the fix: |
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 14, 2024
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 14, 2024
…max capacity. Implemented exponential retry pattern.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 21, 2024
…vadocs. Added links and references to the properties. Removed commented code from checkstyle.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 21, 2024
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 22, 2024
…or KPL Message Handler.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 22, 2024
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 22, 2024
…style failures.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 22, 2024
…o maxInFlightRecordsInitialBackoffDuration. Corrected capacity loop entry condition to use greater than zero.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 24, 2024
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 29, 2024
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 29, 2024
…ed KPLBackpressureException.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 29, 2024
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 29, 2024
…s with RetryAdvice.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 29, 2024
…he KplMessageHandler class level.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 29, 2024
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 31, 2024
…ndler" This reverts commit b7ae296.
siddharthjain210
added a commit
to siddharthjain210/spring-integration-aws
that referenced
this issue
Dec 31, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
In order to prevent the KPL Native process from getting overwhelmed in high load situations, the user of the KPL Producers library must ensure to protect against the backpressure added on the kinesis producer native process. In some cases, if backpressure is not handled, the KPL native process can hung up, swallow exceptions.
References: https://github.com/awslabs/amazon-kinesis-producer?tab=readme-ov-file#back-pressure
The same capability is also required by the spring cloud stream aws kinesis binder project:
spring-cloud/spring-cloud-stream-binder-aws-kinesis#230
Current Behavior
The current implementation of Kinesis KPL message handler in the library doesn't support handling backpressure.
Context
References: https://github.com/awslabs/amazon-kinesis-producer?tab=readme-ov-file#back-pressure
The same capability is also required by the spring cloud stream aws kinesis binder project:
spring-cloud/spring-cloud-stream-binder-aws-kinesis#230
How has this issue affected you?
In one of the applications working on KCL-KPL mode, the native process has started eating up exceptions.
What are you trying to accomplish?
Ensure that
What other alternatives have you considered?
Even if the above recommendations are there, the KPL can still end in adverse situations in very high load. Hence adding this capability would certainly help out.
Are you aware of any workarounds?
Not that I can think of. If we take spring cloud stream kinesis binder as a project, it becomes impossible for a user to configure all the above. Hence the capability is needed in the library so that it can be used.
The text was updated successfully, but these errors were encountered: