Event Driven Apps : OOPS I lost the Event..
With Event Driven Apps, as the name suggests the most important piece is Events. If not designed properly, App risk loosing the events and never getting those back; and if that happens it may create some serious issues.
In this Blog — I will again consider scenarios where Queues and Durable Topics are used as event delivery mechanism.
Not sure of which Mechanism to choose / Queues or Topics??
Chose Topics but how to deal with Duplicate or missed Events??
Lost Event??
When I said event lost — I meant event was delivered to the Application; but somehow Application was unable to process the event completely (because of any reasons).
Once that happens — event though event could not be processed, Message Server does not serves the App the same event back; it instead shares next event. This scenario if happens is what I refer as “Lost Event”.
Why would it Happen?
There can be many reasons but the most common is that the Application processing the Event Errored out without completing the Event Processing.
If that happens — when application asks for events from Message Server, application will not receive same message event, but rather new event.
Is Missed Event required?
Not all applications need to worry about missed message/event. If the application is time-sensitive; such as live sports, live stock market stream etc., application need not worry about unprocessed event.
Most the applications (especially in the micro-services world) though would like all the messages/events to be delivered. Or else it may lead to application in confused/corrupted state.
What can be done?
The problem can be solved in ensuring that the Message server gives you the event back, whenever Application fails to complete its processing for the event.
Acknowledgement Mode
Whenever an application receives a message from the Message Server, it confirms (or acknowledge) back to the Message Server that Application has received the message.
Automatic:
By default — in most of the Message Servers — Acknowledgement happens automatically whenever message is delivered to the application; and once that happens Message Server moves on to the next message.
If Application Supports/ This feature is great for parallel processing of the messages.
Custom (or Client):
If Application happens to choose Client Acknowledgement — Message Server will not move on to next message until Application confirms Message Delivery Explicitly.
In this scenario, if application fails to process the message (say Message 3) completely, it will be redelivered.
This processing is not ideal for parallel processing.
Ideal Solution
As I always say
Every Coin has Two faces / हर सिक्के के दो पहलु होते हैं.
Both faces has to be looked into.
Whenever architecting the solution, in most cases both Ack Mode and parallel processing has to be balanced.
So ideally keep ack mode as Client (or Custom). Application should receive the message and before processing the message it should store the message in some persistent storage, acknowledge the message and then processing.
- Receive the Message
- Store the Message in some persistence location (Database, Local Storage etc.)
- Acknowledge the Message. Message Server will publish next message if it has any; and your application can perform parallel processing.
- Process the Message. In case of failure, reprocess the message from the local repository.
These are just general guidelines that Architect must consider while Architecting or Designing the solution. There are other factors such as Message TTL, Max no of Parallel Processes, Corrupted Message etc. those also have to be kept in mind.
Feel free to drop your comments or any confusion that you may have.
