Consuming CDC Events: Patterns for Building Read Models

From Infrastructure to Implementation

In the previous article, we covered the infrastructure side of MongoDB CDC with Debezium - the oplog retention problem, why Atlas shared tiers fail, and how to configure a production-ready connector. Now let's focus on what happens after events land in Kafka: how to consume them reliably and build read-optimized projections.

The goal is simple: react to changes in your authoritative data store and maintain derived views for specific query patterns. Duplicate delivery makes this harder than the description suggests. Consumers also need a recovery path for failed processing.

Event Handling Pattern

Once events flow through Kafka, your consumers need to handle them reliably. The core pattern is straightforward:

Algorithm 1 ProcessCDCEvent
Require: event: Debezium CDC event from Kafka
Ensure: Updated search index state
1: op ← event.operation
2: key ← event.documentKey
3: if op ∈ {CREATE, UPDATE} then
4: data ← event.payload.after
5: Upsert(index, key, data)
6: else if op = DELETE then
7: Delete(index, key)
8: end if
9: Commit(event.offset)
10: return SUCCESS

Key Patterns

Route each event using Debezium's op field. A value of c denotes creation; u denotes an update. Delete events carry d.

Commit the offset after successful processing. Send failed events to a dead letter queue so they can be investigated. Kafka's at-least-once delivery also requires handlers to tolerate duplicates: use idempotent upserts.

Building Read Models

The search service consumes CDC events to maintain a read-optimized index. Here's the pattern using Redis with RediSearch:

Algorithm 2 BuildSearchIndex
Require: event: CDC event with user data
1: userId ← event.key
2: userData ← event.payload.after
3: HSET("user:" + userId, {userId, username}) // RediSearch auto-indexes
4: if event.op = DELETE then
5: DEL("user:" + userId)
6: end if

The key insight: RediSearch watches hash keys with a matching prefix and automatically maintains the full-text index. Your CDC handler just needs to write/delete hashes.

Recovery Mechanism: Batch Rebuild

Reliable oplog retention does not protect the Redis index itself. A fallback is still needed if Redis loses data or the index becomes corrupted. Implement a batch rebuild endpoint:

Algorithm 3 RebuildSearchIndex
Require: batchSize: number of records per batch
Require: offset: starting position
Ensure: Rebuilt search index
1: users ← FetchFromSource(batchSize, offset) // Bypasses CDC
2: EnsureIndexExists()
3: for user in users do
4: Upsert(index, user.id, user)
5: end for
6: return {processed: |users|}

This runs parallel to CDC processing, allowing index recovery without stopping the real-time pipeline.

Data Ownership and Cross-Service CQRS

It's critical that the single source of truth and authoritative owner of the data remains the user service. In this architecture, we follow the database-per-microservice pattern. You could argue that search isn't a separate domain from user service - however, search has shared concerns across many services (posts, products, users) and requires information from multiple systems to build a rich search experience.

This pattern can be viewed as a form of cross-service CQRS: separating read and write models where the write model lives in the authoritative service (user service with MongoDB) and the read model lives in a specialized query service (search service with Redis). The CDC pipeline is the synchronization mechanism between these models.

Recovery as a First-Class Concern

Replication failures can leave a read model out of sync with its source. Some failures leave stale records behind; others introduce incorrect data. Your system must be recoverable from the beginning. Design for idempotent recovery paths - the ability to rebuild any downstream read model from scratch without corrupting state or losing data.

The batch rebuild API shown above is just one example. I plan to cover fault tolerance in a later post. The approach treats downstream services as disposable and gives them a path from an arbitrary state back to a consistent one. That may involve recreating the service or recovering its existing state.

Failure Modes and Recovery

Conclusion

Building CDC consumers is fundamentally about handling uncertainty gracefully. Consumers must account for out-of-order delivery as well as duplicates. A downstream store can fail while either is being processed.

The patterns covered here - idempotent upserts, operation-based routing, dead letter queues, and batch rebuild APIs - form a foundation for reliable event processing. The key insight is treating your read models as disposable: if you can rebuild them from scratch at any time, you're resilient to almost any failure mode.

Combined with the infrastructure setup from Part 1, you now have a complete picture of production CDC with MongoDB, Kafka, and Debezium.