
May 25, 2024 | 5 minutes read
Scenario
- Order service, managed by the Dev team.
- Analytics service, managed by the Analytics team.
- Audit service, managed by the Auditing team.
When a user places an order, the Order service publishes an ORDER_CREATED event on the order topic. Both the Analytics and Audit services consume this event and then carry out the further processing.

(i): Published ORDER_CREATED event consumed by the Analytics and Audit service
Now, suppose the Dev team needs to change the data type of the date present in orderDetails to include the time along with the date when the order was placed. Here’s the process they’ll follow:
- The Dev team updates the data type of the date field from LocalDate to ZonedDateTime.
- They communicate this exact change to the Analytics and Auditing teams.
- The Analytics and Auditing teams update their instances of the Order Event based on the specified changes.
- Once all three services have integrated the change, they coordinate and release simultaneously.

(ii): Change the data type of date to ZonedDateTime
Common Challenges and Pitfalls
Firstly, let’s consider the declaration of the event. As shown in the image (iii), each service maintains its own copy of the Order event. The order service serializes the Order event, defined by class com.abc.order.OrderEvent into the stream of bytes and publish it to the Order topic. The analytics and audit service consumes this stream of bytes and deserializes it to the class com.abc.analytics.OrderEvent and com.abc.audit.OrderEvent respectively.

(iii): Each service declaring its own copy of the Order event
What happens if the new event schema (let’s call it V2) isn’t compatible with the original schema (V1), and vice versa?
Consider this, updating the data type of order date from LocalDate (V1) to ZonedDateTime (V2) is not backward compatible. This means the stream of bytes representing the V2 event cannot be deserialized back into the model representing the V1 event, and vice versa. Any V1 events still lingering in the Kafka topic, waiting to be consumed, become problematic. The updated Analytics and Audit services, expecting V2 events, won’t be able to consume these V1 events.


(v): Audit service not updated to consume the V2 events
Conclusion
As we delve into Kafka’s role in helping microservices communicate, we encounter unexpected hurdles. What seemed like a simple task of updating the event schema unveiled hidden complexities. We learned about compatibility issues with schema evolution, challenges with the synchronizing of releases, and the need for a single source of truth in declaring event models. To tackle these common challenges and steer clear of pitfalls, we need to find answers to these questions:
- How can we restrict teams from making breaking changes to the event structure?
- How can we ensure that poison pill messages are not published by the producer services to the Kafka topic?
- How can we decouple schema modification from the release of consumer services so that changes can be independently released without waiting on consumer services?
- How do we maintain a single source of truth for our schemas and adhere to the DRY principle?