Dual Write Problem

• https://www.youtube.com/watch?v=FpLXCBr7ucA&t=259s
• occurs when we need to update two separate systems
• For example, update database and write events to Kafka
• Because these systems are not linked there is no way to update them in transactional fashion
• Patterns to mitigate the issue:
  • Change Data Capture (CDC)
    • See Database_Change_Data_Capture
  • Transactional Outbox
  • Event Sourcing
    • See Event_Sourcing
  • Listen to Yourself

sequenceDiagram
    participant Req
    participant Microservice
    participant Database
    participant Kafka
    Req->>Microservice: 
    Microservice-->>Database: Save to DB
    Microservice-->>Kafka: Send Event

Transactional Outbox Pattern

• Whenever we update database, we can also update outbox table in the same transaction
• Have a separate process that asynchronously monitors the table
  • whenever it sees a new event, it can deliver it to Kafka
  • Once event is successfully delivered, it might be removed from outbox table
  • generally we have another thread to monitor the table in the same microservice
• Delivery guarantee: at least once
  • We may retry to keep the Kafka in sync with outbox table, and retry sending events
  • Downstream system should be prepared to handle any duplicates