TL;DR
Kafka’s exactly-once delivery sounds like magic, but it’s more like a really good magic trick. Within Kafka’s ecosystem, it’s real and powerful. But the moment your pipeline touches an external system (S3, Elasticsearch, a database), that guarantee starts to crack. This article breaks down what exactly-once really means, where it works, where it falls apart, and how to build systems that don’t implode when reality hits.
🔥 Introduction
Here’s a story every distributed systems engineer knows: You build a beautiful Kafka pipeline. You enable all the exactly-once flags. You feel invincible. Then someone asks, „So we’ll never see duplicates in the database, right?“ And your stomach drops.
Because the truth is, Kafka’s exactly-once delivery is kind of like those „unlimited data“ phone plans. Yeah, it’s unlimited… until you hit the fine print. Kafka can absolutely guarantee that each message gets processed exactly once, but only if you stay inside Kafka’s world. The second you write to S3, index into Elasticsearch, or update a Postgres row, all bets are off.
This isn’t Kafka’s fault. It’s just physics. Or more specifically, it’s distributed systems doing what distributed systems do: being annoying.
Let’s pull back the curtain on what exactly-once actually means, how Kafka makes it work, and why your production incidents still involve duplicate records showing up at 3 a.m.
The Three Flavors of Message Delivery (And Why Two of Them Suck)
Kafka gives you three options for message delivery, and understanding them is like understanding your coffee order: get it wrong and you’re going to have a bad time.
At-most-once is the „fire and forget“ approach. Your producer yeets messages at Kafka and doesn’t wait around to see if they landed. If something crashes, that message is gone forever. This is great for throughput and terrible for basically everything else. Maybe you use this for UI metrics that nobody cares about. Maybe you don’t use it at all. It’s the decaf of delivery guarantees.
At-least-once is the default, and honestly, it’s what most of us live with. Messages will definitely arrive, but they might arrive twice. Or three times. Or seventeen times if you’re having a really bad week. The producer retries on failure. The consumer processes messages before marking them complete. If anything crashes in between, you process the same message again. It’s reliable in the „your data won’t disappear“ sense, but duplicates are your problem now.
Exactly-once is the dream. Every message processed exactly one time. No losses, no duplicates. Kafka introduced idempotent producers and transactional APIs to make this happen. It works. But it works with conditions, caveats, and a whole lot of engineering underneath.
Here’s the part that trips everyone up: exactly-once isn’t a switch you flip. It’s a property you architect. And it only holds if every piece of your system speaks the same transactional language.
How Messages Actually Move Through Kafka (And Where Things Go Wrong)
Let’s walk through the journey of a Kafka message, because understanding the failure modes is the only way to avoid them.
Producers send messages to brokers. If the producer doesn’t wait for an acknowledgment (acks=0), it’s fast but reckless. The message might never make it. If the producer waits for confirmation (acks=all), it’s safer, but now you have a new problem: what if the broker writes the message, crashes before sending the ack, and the producer retries? Boom. Duplicate.
Kafka’s solution? Idempotent producers. Since version 0.11, producers can attach a unique ID and sequence numbers to messages. If the broker sees the same sequence number twice, it drops the duplicate. No double-write, no duplicate in the log. This is huge. It means that producer retries don’t cause duplicates anymore, at least not at the broker level.
Brokers store messages and replicate them for durability. Once a message is committed, it’s not going anywhere. Brokers don’t spontaneously duplicate things. That’s good. But they also don’t care whether you’ve processed a message once or seventeen times. That’s on the consumer.
Consumers read messages by offset. This is where things get spicy. If you commit the offset before processing (maybe auto-commit is ticking along in the background), and then your app crashes mid-processing, Kafka thinks you handled those messages. You didn’t. They’re gone. At-most-once strikes again.
If you process first and then commit, a crash means Kafka will redeliver those messages when you restart. You’ve avoided data loss, but now you might process duplicates. At-least-once is the default for a reason: it’s safer than losing data, even if it’s messier.
So how do you get exactly-once consumption? You need to commit the offset atomically with the processing outcome. Kafka’s transactional API lets you do this. You can read a message, do some work, produce output to another Kafka topic, and commit the input offset, all in a single transaction. If anything fails, none of it commits. If it succeeds, all of it commits. It’s beautiful. It works. And it only works if your entire pipeline lives in Kafka.
Apache Flink: Making Exactly-Once Actually Work (In Kafka’s World)
Apache Flink took Kafka’s transactional guarantees and ran with them. Flink gives you exactly-once processing in streaming applications, and it does it by being incredibly paranoid about state.
Checkpointing is Flink’s secret weapon. Periodically, Flink snapshots the entire state of your application: all the variables, all the positions in Kafka, everything. If something crashes, Flink rewinds to the last checkpoint. It’s like save-stating a video game. You don’t lose progress because you can always reload.
When Flink reads from Kafka, it stores the Kafka offsets in each checkpoint. On failure, it resets to those offsets and re-reads the messages. This means messages might be processed multiple times during a failure, but Flink also resets its internal state, so from the outside, it looks like nothing happened. Exactly-once semantics achieved.
Writing to Kafka from Flink uses transactions. Flink opens a Kafka transaction for each checkpoint interval, writes all the output messages, and commits the transaction only when the checkpoint succeeds. If the job crashes before the checkpoint, the transaction aborts. None of the output becomes visible. If it succeeds, everything commits atomically. Downstream consumers (using read_committed isolation) only see each message once, even if Flink retried the whole batch.
This is genuinely impressive engineering. Flink extends exactly-once beyond just Kafka-to-Kafka pipelines by coordinating checkpoints with Kafka transactions. But here’s the kicker: this only works for Kafka sinks. What about databases? What about S3?
Flink has a two-phase commit sink for external systems. If your target system supports transactions (or something transaction-like), Flink can integrate it with checkpoints. On each checkpoint, Flink pre-commits the writes, stores a handle in the checkpoint, and then commits for real when the checkpoint succeeds. If it fails, Flink rolls back.
For file systems, Flink writes to a temp file and only moves it to the final location on checkpoint commit. If the checkpoint fails, the temp file gets discarded. Exactly-once to HDFS or S3, achieved.
But this requires cooperation. The external system has to support some form of transactional rollback. If it doesn’t, Flink can’t help you. You’re back to at-least-once.
The Achilles‘ Heel: External Systems That Don’t Speak Kafka
This is where the illusion breaks.
Kafka can guarantee exactly-once delivery inside its own ecosystem. Flink can extend that to systems that support transactions. But most of the real world doesn’t speak Kafka’s transactional language. And that’s where duplicates creep back in.
You can’t atomically commit to Kafka and an external system. Not unless that system is designed for it. You can write to S3 and then commit your Kafka offset, but if you crash between those two steps, you’re screwed. Either S3 has the data and Kafka thinks it doesn’t (so it’ll replay and create a duplicate), or Kafka thinks you processed it but S3 never got it (data loss).
There’s no magic here. The absence of a distributed transaction means you have to pick: risk duplicates or risk data loss. Most people pick duplicates.
Retries make this worse. Let’s say you write to an external API and get a timeout. Did the write succeed? You don’t know. If you retry, you might duplicate the write. If you don’t retry, you might lose the data. The external system doesn’t give you idempotency guarantees, so you’re gambling either way.
Some systems make this easier. If you can use an idempotent write (like an upsert by primary key), duplicates become harmless. You process the same message twice, but both writes produce the same final state. Elasticsearch can do this if you set document IDs correctly. Databases can do this with upsert or merge logic. S3 can do this if you use deterministic file names (the second write just overwrites the first).
But not every use case fits this pattern. If you’re appending to a log or creating a new record each time, idempotency doesn’t save you. You’ll see duplicates.
Timeouts, partial failures, and split-brain scenarios make it even worse. You write half a batch to S3, then crash. Do you replay the whole batch and risk duplicating the first half? Do you skip it and risk losing the second half? Kafka’s transactional API can abort a batch cleanly, but S3 doesn’t have that. You’re on your own.
The bottom line: exactly-once only extends as far as your transactional boundaries. If your pipeline crosses into systems that don’t support coordinated commits, you’re effectively back to at-least-once, and you need to design for it.
A Real Example: Kafka to S3 and Elasticsearch (Where Duplicates Happen)
Let’s make this concrete. You have a Kafka topic full of events. You want to:
- Write each event to S3 for long-term storage.
- Index each event in Elasticsearch for search.
You’ve enabled Kafka’s exactly-once settings. You’re using Flink or Kafka Connect or a custom consumer. Doesn’t matter. Here’s what can go wrong.
Scenario: You process a message, write to S3, then try to index it in Elasticsearch. The Elasticsearch call times out.
You don’t know if the indexing succeeded or failed. The timeout could mean the request never made it, or it could mean Elasticsearch processed it but the response got lost. You can’t be sure.
Because you’re cautious, you don’t commit the Kafka offset. Maybe your app crashes intentionally to restart from a clean state. When you come back up, Kafka redelivers the message (because the offset wasn’t committed).
Now you write to S3 again. Duplicate data in your data lake. You also retry the Elasticsearch indexing. If the first attempt truly failed, this is fine. But if it succeeded despite the timeout, you now have two documents in Elasticsearch for the same event. Oops.
This isn’t a bug in Kafka. This isn’t even a bug in your code. This is just what happens when you can’t atomically commit across multiple systems.
How do you fix it?
Option 1: Make the writes idempotent. Use a deterministic S3 key based on the event ID. The second write overwrites the first. Use the event ID as the Elasticsearch document ID. The second index operation updates the existing document instead of creating a duplicate. Now replays are harmless.
This works great if your use case supports it. The Kafka Connect S3 sink does exactly this: it uses deterministic file names so replays overwrite instead of duplicate.
Option 2: Two-phase commit with an outbox pattern. Instead of writing directly to S3 and Elasticsearch, write to a local database within a transaction that also stores the Kafka offset. Then a separate process reads from that database and pushes to S3 and Elasticsearch. This gives you exactly-once from Kafka to the database, and at-least-once from the database to external systems. It’s complicated, but it works.
Option 3: Accept at-least-once and clean up later. Process duplicates downstream. Use unique constraints in your database. Run deduplication jobs. This is often the most pragmatic approach. It’s not pretty, but it’s realistic.
The key insight: Kafka’s exactly-once guarantees don’t magically extend to your entire infrastructure. Once you cross the boundary into external systems, you’re back to dealing with distributed systems problems the old-fashioned way.
🎯 Conclusion
Kafka’s exactly-once delivery is real. Within Kafka’s ecosystem, it’s a genuine achievement. Idempotent producers prevent duplicate writes on retry. Transactional consumers tie input offsets to output messages, so everything commits atomically or not at all. Frameworks like Flink build on this to give you exactly-once stream processing with checkpoints and coordinated commits.
But the moment your pipeline touches an external system, the guarantees start to fray. S3 doesn’t speak Kafka’s transactional protocol. Elasticsearch doesn’t coordinate commits with Kafka offsets. APIs time out, and you can’t be sure if the write succeeded. In those cases, exactly-once becomes more of an aspiration than a reality.
This doesn’t mean you’re doomed. It means you need to design for it. Make your writes idempotent where possible. Use unique IDs. Accept that at-least-once is the practical guarantee for external integrations, and build deduplication or cleanup strategies accordingly.
Kafka didn’t solve the Two Generals Problem. It didn’t break the laws of distributed systems. It just gave us really good tools to handle the chaos within a well-defined boundary. And honestly, that’s more than enough.
The illusion isn’t that Kafka claims to do the impossible. The illusion is that marketing materials make it sound like you can flip a switch and never worry about duplicates again. You can’t. But with careful design, you can get pretty damn close.
Now go build something reliable. And when duplicates inevitably show up in production, at least you’ll know why.