Ordering Is Not a Transport Problem

Your partition count welds ordering granularity to parallelism. A sequence number in the outbox and a conditional UPDATE in the consumer unbolt them.

Two events leave your service one millisecond apart. address.updated with the new street, then address.updated with a typo fix on the same street. Your consumer applies the typo fix, then applies the old street on top of it, and now a customer’s package is going to the wrong building. Nothing crashed, nothing retried, and no alert fired. The system did exactly what it was told, in the wrong order, and you will find out about it from support.

So you reach for a Kafka partition key or a FIFO queue, and the ordering problem goes away. What is less obvious is what you handed over to get it. A partition is read by one consumer in a group, and that is not a config limit you can tune away. It is the thing that makes the ordering true. Ordering and parallelism are the same dial, and the granularity of both is now welded to a partition count you picked before you understood the workload.

There is a third option almost nobody reaches for first, and it starts from a different premise. Ordering is not a transport property. It is a property of the data, and your producer already knows the order at the moment it writes the row. If you run a database on both ends, you are one column and one WHERE clause away from owning it yourself.

what a partition key really charges you

Kafka gives you ordering inside a partition, and the mechanism is the whole cost: one reader position moving through an append-only log, which means one consumer per ordered stream. SQS FIFO makes the same promise with less ceremony and the same ceiling, scoped to a message group.

So your only way up is more partitions, which means more independent ordered streams, which means your ordering granularity is welded to your partition count. Choose it wrong at the start and you get to find out how much fun repartitioning a live topic is. Lock-in rides along too, since your correctness now depends on one broker’s specific semantics.

One piece of terminology worth fixing here, because I see it conflated constantly. The thing that gives you transport ordering is the partition key, not the idempotency key. An idempotency key says „if you see this twice, it is the same operation, do it once.“ A partition key says „route everything with this value to the same log.“ They are frequently the same string in practice, usually some resource ID, which is exactly why people mix them up. But they answer different questions, and only one of them has anything to do with order.

For plenty of systems, that trade is fine. Take it and move on. But it is worth knowing you made a trade, because most teams do not notice they made one.

put the order in the payload

Every message carries two extra fields. A resource identifier saying what this message is about, and a sequence number saying where it falls in that resource’s history.

{
  "resource_id": "customer-8412",
  "sequence": 47,
  "event": "address.updated",
  "payload": { "street": "Hauptstrasse 12" }
}

That sequence number is a third thing again, distinct from both keys above: it does not route and it does not deduplicate, it ranks.

The consumer keeps a table of the highest sequence it has processed per resource. A message arrives, the consumer looks up customer-8412, sees 46, and 47 is next, so it processes it and writes 47. Another message arrives with sequence 45. That is older than what you already applied. Drop it.

That is the whole mechanism. The queue can deliver in any order it likes, redeliver whatever it wants, run twelve consumers in parallel, and the outcome is the same. Which means the queue just got demoted to something that moves messages. It does not guarantee order, it does not need exactly-once or FIFO semantics, and it does not need to be the same product next year. Kafka, SQS, RabbitMQ, NATS, a webhook fan-out, some of them at once. The correctness argument now lives in your producer’s transaction and your consumer’s UPDATE, both of which are your code in your database, both of which you can test without a broker running. That is not a small thing when a team wants to swap transports and the answer is „sure“ instead of a six-week correctness review.

before you build it, check that it applies

Three questions, cheap to run against your own system. Ask them now, because two of them disqualify the pattern outright and you should find that out here rather than four sections from now.

First, are your events state-carrying? This is the one that decides everything, and it turns on a case that looks like a bug and is not. Message 48 arrives before 47. It is newer than 46, so the update succeeds and you process it. Then 47 shows up, gets rejected as stale, and is dropped forever. If every message contains the full current state of the resource, that is correct and it is the point: applying 48 and dropping 47 leaves you at 48, which is where you wanted to be, and you got there without waiting. But if 47 says „add 50 euros“ and 48 says „add 30 euros,“ you just lost 50 euros silently. Delta events need a real gap buffer, which means parking 48, waiting for 47, applying in order, and deciding what to do when 47 never comes. That buffer needs a timeout policy, and it quietly reintroduces the waiting you adopted this pattern to avoid.

Second, does your consumer own a database? The check-and-update has to be atomic, and that requires state on the consumer side. If your consumer is a stateless function that forwards to a third-party API, this pattern is not for you.

Third, is your per-resource volume high enough to care? The entire payoff is escaping the one-consumer-per-ordered-stream ceiling. If you are not near it, you are buying a table and a column to solve a problem you do not have.

Three yeses and the rest of this is worth building. Anything else, use a partition key. I would push most teams toward state-carrying events regardless, and not only for this: they make consumers idempotent by construction, and they turn „did we miss one“ from a correctness question into a freshness question.

the producer half is one more column

Monotonic per resource sounds like a distributed-counter problem. It is not, if you already run the outbox pattern, where the event goes into a table in the same transaction as the state change and a separate relay publishes it later.

So the event already lives in a transactional row before anything is published. The sequence number is one more column, allocated in that same transaction:

INSERT INTO outbox (resource_id, sequence, event_type, payload)
SELECT 'customer-8412',
       COALESCE(MAX(sequence), 0) + 1,
       'address.updated',
       '{"street": "Hauptstrasse 12"}'::jsonb
FROM outbox WHERE resource_id = 'customer-8412';

Under a unique constraint on (resource_id, sequence), two concurrent transactions writing events for the same customer cannot both get 47. One commits, the other fails on the constraint and retries into 48. The database is doing the serialization, which is the thing databases have been good at for forty years.

One trap worth naming. Reach for a global database sequence and you will get a number that is monotonic but has gaps, because sequences hand out values outside the transaction and a rollback does not return them. That is fine here. The consumer compares magnitudes, it never asserts that 47 follows 46. But it means you cannot use a gap to detect a lost message, and if you want that, you need per-resource numbering with the counter inside the transaction.

Notice what the relay is now free to do. It can publish out of order. It can publish the same row twice after a crash between publish and mark-as-sent. It can publish rows for one customer across four different partitions of four different topics. None of that changes the outcome, because the order is in the row.

the WHERE clause is the concurrency control

The consumer half is where people get it wrong.

The naive implementation is a read, then a compare, then a write, and it is broken under concurrency in the exact way this pattern was supposed to prevent. Two consumers pick up sequence 47 and 48 for the same customer at the same moment. Both read last_sequence = 46. Both decide they are next. Both process. Both write. Whichever transaction commits second wins, and if that is 47, you just applied the older event last and the typo fix is gone. You built the machinery and reintroduced the bug.

The check and the update have to be one atomic operation.

UPDATE consumer_state
SET last_sequence = 47
WHERE resource_id = 'customer-8412' AND last_sequence < 47;

Zero rows updated means the message is stale, so acknowledge it and discard. One row updated means you own it, and you do the work in that same transaction. No locks you manage, no distributed coordinator, no leader election. The first message for a resource has no row to update, so in practice this is an INSERT ... ON CONFLICT (resource_id) DO UPDATE carrying the same last_sequence < predicate on the conflict clause.

That „same transaction“ carries an assumption worth making explicit, and it brings us back to the key we set aside earlier. It only holds if the work is a database write. Call an external API and the call is outside the transaction, so a rollback after a successful call leaves the sequence unadvanced and the side effect already delivered, and your retry does it twice. If the consumer’s real work is an HTTP request, you need that endpoint to be idempotent, and the sequence number is a reasonable idempotency key to hand it. Three keys, and the one that ranks has just become the one that deduplicates.

the bill, and the case against paying it

One row per resource, growing with your resource count rather than your event count, which is the good news. The bad news is that dead resources never clean themselves up, so you need a retention story before the row count becomes someone’s on-call problem.

The stronger objection is not operational. If your partition count is sized correctly for your workload and you have no intention of changing transports, Kafka’s guarantee costs you nothing and you get to skip all of this: no sequence column, no unique constraint, no consumer state table, no gap-buffer decision, and no class of bugs where someone writes the naive read-compare-write and passes code review because it looks obviously correct. Fewer moving parts you own is a real engineering argument, and „the broker already solved this“ is often the right answer. The pattern here earns its keep when the ceiling actually binds or the transport is genuinely in question. Otherwise it is machinery in search of a problem.

Which is the thing worth taking away even if you never write the column. Ordering is not something you buy from a broker. It is a fact about your data, and your producer already knows it at the moment it writes the row. Every extra hop between that moment and your consumer is a chance for the transport to scramble a truth it never had any business owning. If the answer to „where does ordering live in your system“ is „in Kafka,“ you picked a concurrency limit at the same time, and you probably did not know you were picking one.

Write it down in the message.