TL;DR
Distributed systems are messy. Events happen everywhere, all at once. Some of them matter to each other. Some don’t. If you get the ordering wrong, you end up deleting users that were never created or charging accounts that don’t have money yet. This article breaks down the theory behind concurrency and causality, shows you how consensus algorithms like Raft keep things in check using leader terms and epochs, and explains how Kafka maintains order using partitions and keys. By the end, you’ll understand how these systems stay consistent even when everything is happening in parallel.
Introduction
Picture this: two things happen at almost the same time. One process creates a new user account. Another one deletes a user. In a single-threaded program, you’d know exactly which happened first because everything runs in sequence. Easy.
But in a distributed system? All bets are off. These events could be happening on different servers, in different data centers, maybe even on different continents. Did the deletion happen before the creation? Were they truly simultaneous? Get this wrong and you’ll delete a user that was never created. Or worse, you’ll create a user that magically reappears after deletion like some kind of database zombie.
Understanding event ordering isn’t just academic. It’s the difference between a system that works and one that silently corrupts your data.
So what does it mean for events to be parallel versus sequential? How do causal relationships define order when there’s no global clock? And why does any of this matter for real systems?
We’re going to start with the theory (using analogies that don’t suck), then move to practice. We’ll look at how consensus algorithms like Raft enforce order, how linearizability gives you a single consistent timeline, and how Apache Kafka maintains ordering at scale using partitions and keys. By the end, you’ll see that parallelism and causality are two sides of the same coin when building reliable distributed systems.
When Things Truly Happen Together
Let’s start simple. What does it mean for two events to be parallel?
In everyday life, parallel events are things happening at the same time without depending on each other. Two people editing different documents. Two chefs working on separate dishes. Neither one is waiting for the other.
Sequential events have a clear order. You write a sentence word by word. You don’t write the period before the first letter.
In distributed systems, events are concurrent (truly parallel) if there’s no causal link between them. Leslie Lamport put it this way: „two events are concurrent if neither can causally affect the other.“ If event A didn’t influence event B and B didn’t influence A, they’re parallel. The system has no inherent way to say which came first. It’s like two separate threads of history running side by side.
Take our user example. Creating user Alice on Server 1 and deleting user Bob on Server 2 at the same time are parallel events. Alice’s creation doesn’t depend on Bob’s deletion. There’s no natural order. They can happen independently.
But if we’re talking about the same user? That’s different. Creating Alice’s account and deleting Alice’s account are not independent. Deleting Alice should only happen after she’s created. If these events happen concurrently, you’ve got a logical paradox. Whether events can be parallel depends on context and what data they affect.
On a single machine, we enforce sequentiality to avoid this confusion. One operation after another. In distributed systems, many components operate at once. Without a defined order, different parts of the system might see events in different sequences.
This is where we need to talk about time and causality. Not wall-clock time (that doesn’t work in distributed systems). Logical time. The happens-before relationships that give events an order even when everything’s running in parallel.
The Happens-Before Relationship
Not all events are independent. Many have cause-and-effect relationships.
Causality is when one event influences or triggers another. We say event A causes event B (or A happens-before B) if B can only happen after A has occurred. Maybe B is a reaction to A.
Classic example: sending and receiving a message. The send event must happen before the receive event. The receive is caused by the send. There’s a clear order: send then receive.
Same thing with our user scenario. Creating a user account must happen before deleting that same user. The deletion depends on the creation. You can’t delete what isn’t created. This dependence is a causal relationship.
Lamport formalized this with the happens-before relation. In simple terms, if two events are related by a chain of cause and effect (one happened in the same process after the other, or one sent a message that the other received), then one happens before the other. If not, the events are concurrent.
So if Server 1 creates Alice and sends a notification to Server 2, and Server 2 logs that creation, the notification creates a causal link. Everyone should agree that the creation happened before the log event. But if Server 2 was simultaneously deleting Bob (unrelated to Alice), those two event streams don’t intersect. They’re concurrent.
Here’s an analogy. Two chefs in a kitchen. Chef A is making a salad. Chef B is baking a cake. If Chef B needs chopped nuts from Chef A, then Chef A chopping nuts must happen before Chef B mixes them into the batter. That’s a causal relationship.
But if Chef A is chopping carrots for the salad while Chef B is preheating the oven? Those events are independent. Either could happen first or they could overlap. It doesn’t matter. They’re parallel events with no direct connection.
In distributed systems, we lack a global clock to timestamp events perfectly. So we rely on logical time. Each node tags events with a logical timestamp (Lamport clocks or vector clocks) to track ordering. A happens-before relationship gets captured by these clocks. A message carries a timestamp so the receiver knows roughly what happened when.
But the fundamental rule remains: if no sequence of cause leads from Event A to Event B, they’re concurrent. The system must treat either order as potentially valid. If there is a cause-effect link, the system cannot reorder them. The cause must come before the effect.
When Causality Goes Wrong
Why do we care about identifying causality? Because if a distributed system treats causally related events as parallel (or vice versa), weird things happen.
Imagine this scenario. Event 1: „Add $100 to Alice’s bank account.“ Event 2: „Send payment of $50 from Alice’s account.“ If these events occur on different servers, we must know the order.
If the payment gets processed before the deposit due to lack of ordering, Alice’s account could overdraw even though she had enough money after the deposit. The deposit causally enables the payment. Reordering them breaks correctness.
On the flip side, truly parallel events might not need any particular ordering. If Alice deposits $100 and Bob deposits $200 to his own account at the same time, those can happen in any order relative to each other. It won’t matter to the correctness of each account’s balance. In a distributed setting, it’s fine for different parts of the system to see those in different orders, as long as each account’s updates are ordered.
So causal relationships impose a necessary order (cause then effect). Independent events have no inherent order. Distributed systems must figure out which is which.
This is where algorithms and consistency models come in. They coordinate and agree on an ordering of events that respects causality where needed. Let’s see how this plays out in consensus algorithms, the backbone of many distributed databases and systems.
Making a Cluster Agree on Order
How do you make a cluster of machines agree on the order of events when many events can happen in parallel?
Enter distributed consensus algorithms. Consensus algorithms (Paxos, Raft, Zab) are designed to make multiple servers act as one. They replicate a log of events in the same order everywhere. Even if events are initiated concurrently on different nodes, the cluster agrees on a single sequence (a total order) in which to apply these events.
Let’s focus on Raft. It’s more understandable than Paxos.
Raft imagines the system as a replicated state machine. Every server maintains a log of events that change the system state. The goal is for all servers to have identical logs in the same order, so they all transition through the same states.
How? Raft uses a leader to impose order. One node is elected leader. All changes go through it. The leader decides where each new event goes in the log (essentially assigning it a position in the sequence). It broadcasts the event to followers, who append it in the same position in their logs. As long as the leader is in charge, everyone gets events in the same order as the leader intended.
But what if the leader fails? What if messages are delayed and arrive late, out of order?
Raft handles this with terms (epochs of time) and strict rejection of old information. Each Raft term starts with an election. Servers vote to choose a leader for that term. The term number increases every time there’s a new election. Think of it as a logical clock that ticks up each time leadership changes.
This term acts like a version of the timeline. Only the leader of the current term should be sending commands. If a server hears from an old leader with a past term, it knows to ignore those messages. Servers reject any messages from obsolete terms.
Raft uses the term number to attach a causal context to messages. A message from term 4 cannot affect term 5 because the system has moved on. If Server A is in term 5 and receives a log entry request tagged with term 4, it concludes „this is a stale message from an old leader“ and discards it.
This ensures that an outdated event (one decided by a former leader but never fully processed before a leadership change) won’t confuse the cluster or violate the new order established under the new leader.
Think of term changes like changing the guard. Imagine a kingdom where each term is the reign of a new king. King Alice (term 4) was ordering events, but then King Bob (term 5) took over. If one of Alice’s couriers shows up late with a command after Bob is king, the court says, „Sorry, that order is from the previous reign. It’s no longer valid.“
The new king’s orders supersede the old. Any late arrivals from the old reign are ignored. This is exactly how Raft treats old leader messages using term numbers as reign identifiers. It’s a causal cut-off. The cause (leadership change) defines a before/after such that anything before is obsolete after.
To maintain consistency, Raft also ensures that events don’t get lost or reordered. The leader only marks an event as committed (finalized in the sequence) when a majority of servers have it in their log. This majority agreement acts like a safety net. It’s the cluster’s way of saying „we all got this event in position X, we consider it firmly ordered.“
If another event Y was being decided at the same time, either it got a different position in the log or it waits until the first one is decided. By doing this for every event, Raft achieves a total order broadcast of events to all nodes. All nodes apply the events in the exact same sequential order, which means the state remains identical.
The key takeaway: distributed consensus brings order to chaos. It takes potentially parallel happenings and forces a single sequence, respecting causality. No effect happens before its cause, thanks to mechanisms like term checks and majority agreement. Consensus algorithms serialize the distributed world, turning it into a series of one-after-the-other events, at least from the cluster’s perspective.
This property is crucial for building strongly consistent systems. It gives rise to the concept of linearizability, which we’ll explore next.
One Timeline, Even When Everything’s Parallel
If consensus gives us an ordered log, what does the user or application experience?
Ideally, it experiences operations as if they occurred one by one, in a consistent order cluster-wide. Even if under the hood many were happening in parallel. This illusion is called linearizability.
It’s one of the strongest consistency models. It basically means the system behaves like there’s a single copy of the data and a single timeline of operations. Every operation seems to take effect at an instant in time. If one operation completed before another started in real-world time, it will show up before the second in the sequence.
Another way to put it: linearizability ensures that all nodes (and all clients) see operations occur in the same one-at-a-time order, respecting the real-time order in which requests were made. Even if operations were done on different machines concurrently, the system won’t expose a view that contradicts a single sequential history.
This is exactly what a consensus-ordered log provides. The log is that single history.
For example, if client A updates a record and then client B reads it, and the update was acknowledged before the read request, linearizability guarantees B will see the update. As if the update happened instantaneously at some point before the read.
If the system weren’t linearizable, B’s read might return stale data, seeing the world as if the update hadn’t happened yet. That’s a consistency anomaly.
Let’s break down why linearizability is tied to causality and ordering. Suppose we have two events. Event A: „Alice’s account is credited $100.“ Event B: „Alice’s account is debited $50.“ If Alice’s credit finished at 10:00:05 and her debit started at 10:00:10 (after she got confirmation of A), then any linearizable system must show A before B in the global order. So her account never goes negative in the interim. This respects real-time ordering of requests.
In a non-linearizable system, it might be possible for some replica to apply B before seeing A, which would temporarily violate correctness. Linearizability prevents that by enforcing the real-time causal order. Since A causally preceded B (from the client’s perspective), all servers apply A then B in that order.
How do systems achieve linearizability? Typically by using total order broadcast or consensus under the covers.
Total Order Broadcast is a communication primitive that ensures all nodes receive all messages in the exact same order. Consensus and total order broadcast are closely related. In fact, you can prove that solving one lets you solve the other.
Intuitively, if you broadcast every operation (as a message) to all replicas with a guarantee of a consistent order, then each replica can apply the operations in that order. Voila. The state evolves the same everywhere, as if following a single timeline of events.
To achieve total order broadcast, two key properties are required:
Reliable delivery: No message is lost. If one node gets the message, eventually all correct nodes get it. Everyone sees the event.
Ordered delivery: All nodes deliver messages in the same relative order. If Event X is delivered before Event Y on one node, every node will deliver X before Y.
Consensus algorithms like Raft ensure both. Reliability via retries and quorum (so messages eventually get to everyone even with failures). Ordering via the leader’s log sequencing and commitment rule. That’s why we say consensus gives you an atomic broadcast or total order broadcast mechanism. The result is a linearizable log of events.
From a narrative perspective, linearizability means the distributed system can tell a single story of what happened, one event at a time, that all observers agree on. That story respects the actual chronology of cause and effect.
It’s like having multiple authors (servers) writing a book but through some magic, the book’s chapters always end up in a sensible order no matter who writes them. Everyone has the same edition of the book. This magic is expensive (it can impact performance because we often need to wait for coordination), but it greatly simplifies understanding the state of the system.
Here’s a simple timeline to illustrate linearizability. Imagine two clients and a key-value store. Client 1 writes X=5, then Client 2 (after seeing Client 1’s success) reads X. In a linearizable store, the read must return 5, because the write will appear before the read in the global order (since real-time order was that way). If the read came back with the old value (say 0), that would mean the operations didn’t appear in the right order to Client 2. Linearizability forbids that scenario.
Essentially, linearizability means no surprises. Once an operation is done, all later operations will see its effects.
Not every system needs full linearizability across all operations. Sometimes, we can loosen things for better performance. But many critical systems (distributed databases, configuration services like ZooKeeper or Chubby) rely on linearizability for simplicity and correctness.
Having explored the theoretical ideals of total ordering and linearizability, let’s look at a practical system that deals with ordering in a slightly different way. Apache Kafka shows that understanding causality and ordering isn’t only for fancy consensus algorithms. It’s also baked into how we handle streams of events at scale.
Kafka’s Pragmatic Take on Ordering
Apache Kafka is a distributed event streaming platform. Essentially a messaging system plus log storage. It’s designed to handle very high throughput by partitioning data across brokers (servers).
Kafka’s approach to ordering is pragmatic. It provides strong ordering guarantees within a partition, but events in different partitions are not globally ordered. This design acknowledges that not all events need to be totally ordered relative to each other. Only those that are related (possibly causally) should be ordered.
Here’s how it works. A topic in Kafka is split into partitions, each being an independent, ordered log. When producers publish messages, they attach a key to each message (or rely on round-robin if no key). The key determines which partition a message goes to.
Critically, Kafka’s rule is that all messages with the same key will always go to the same partition. Why? Because that guarantees those messages can be read in the exact order they were written.
Within a single partition, Kafka assigns each message a sequential offset (0, 1, 2, …) as they arrive. A partition is managed by one broker leader at a time, which appends messages in order. It’s very much like a single log file. Totally ordered internally.
For example, if we use the user ID as the key for user-related events, then all events for user Alice (ID 42, say) will end up in Partition X (some partition determined by hashing 42). That means if Alice’s account is created, then updated, then deleted, those three events will be in one partition in the exact order they occurred.
Any consumer reading that partition will see „Create Alice“ then „Update Alice“ then „Delete Alice“ in that sequence. Never jumbled. Kafka guarantees that ordering per partition. It’s a core feature of its design.
However, events with different keys (Alice’s events versus Bob’s events) might reside in different partitions. Kafka does not guarantee an order between events in different partitions. Bob’s events might be in Partition Y and could intermix in time in any way relative to Alice’s when you look at the whole system.
But that’s usually fine. Alice and Bob are independent. No direct causal link typically. If some application did need a relationship between Alice’s events and Bob’s, it would have to enforce it via using the same key or doing cross-partition coordination at a higher level.
Here’s an analogy. Think of Kafka’s topic like a set of parallel lanes on a highway. Each lane (partition) keeps cars (messages) in order, but cars in different lanes can pass each other independently. If you care about the order of two specific cars, you’d better put them in the same lane. That’s what keys do. They funnel related events into the same lane.
As long as two events are in the same partition, Kafka ensures the first produced will be the first consumed, the second produced will be second, and so on.
From a causality standpoint, Kafka’s model assumes that the key defines a context of causality or dependency. All events with the same key are potentially causally related or at least need ordering (so Kafka totally orders them). Events with different keys are assumed to be unrelated (or okay to treat as concurrent), so Kafka doesn’t attempt to order those globally.
This is a compromise that gives high throughput and scalability. Partitions can be processed in parallel across different brokers and consumers. But you still get ordering where it typically matters (within each entity or context).
Let’s make this concrete. Suppose we have a „user-events“ topic with 3 partitions. The key is the user ID. User Alice (ID 42) maps to Partition 1. Now imagine the sequence of actions: Alice is created, then Alice is deleted, then Alice is recreated.
All these events have key=42, so Kafka sends them to Partition 1. Partition 1 logs them as [„Create Alice“, „Delete Alice“, „Create Alice“] in that exact order. A consumer reading Partition 1 will see those in order and can react accordingly. Perhaps the deletion and re-creation are two separate actions in a short time, but they’re processed in sequence.
There’s no way that „Create Alice“ (the first one) would ever be seen after „Delete Alice.“ Kafka’s partitioning makes such reordering impossible.
User Bob (ID 77) might be on Partition 2. His events will be ordered on that partition. But consumers reading both Partition 1 and 2 have to deal with interleaving. Maybe Partition 2’s events come in between Alice’s events when merging streams. Since Alice and Bob are independent, this doesn’t violate causality. It just means there’s no single timeline of all events, only per partition timelines. And that’s acceptable for most uses of Kafka.
One more thing. Kafka also ensures that within a partition, a single consumer reads the messages in order. It won’t have two consumers race and possibly handle messages out of order. Kafka’s consumer group mechanism assigns entire partitions to individual consumers, so each partition’s events are processed by one consumer at a time, preserving the order they were stored. This way, even on the consuming side, the causal order per key/partition is respected.
Kafka’s design shows a practical application of the principles of parallelism and causality. Identify what needs ordering and what can be parallel. By partitioning, it acknowledges that total order (one big sequential log of everything) is not always necessary. And indeed doesn’t scale for huge throughput. But it provides ordering in the dimensions that matter (per key).
This is a common pattern in distributed systems. Weaken the guarantees just enough to gain efficiency, but still keep causality where it counts.
Orchestrating Chaos
In a distributed system, we’re conductors of an orchestra where each musician has their own clock and might start or stop independently. Parallel events are happening all over. Yet to produce a harmonious result, we need the right cues to ensure things occur in a sensible order.
Parallelism gives us performance and scale. Many things happening at once. But causality gives us the storytelling, the logic of „this before that“ which is crucial for correctness.
We started by examining what it means for events to be truly parallel (independent with no causal ties) versus sequential (strictly ordered). Without a unifying clock, parallel events are those that have no way of influencing each other. Lamport’s insight: such events are concurrent because neither knows about the other.
Then we introduced causality as the glue that links events. The arrows of time saying „A led to B, so everyone must see A then B.“
Following that trail, we arrived at distributed consensus algorithms like Raft, which exist to impose a consistent order on a distributed system. Raft uses leadership and terms (epochs) to ensure there’s at most one sequence of decisions at a time. It smartly rejects old information that could violate the current order. This gives the system a way to agree on a single history of events, achieving a total order broadcast so all nodes apply the same operations in the same order.
With a consistent log in place, we discussed linearizability, the property that makes a distributed system as easy to understand as a non-distributed one by providing the illusion of a single timeline. Linearizability is essentially the end-goal of enforcing proper ordering. It respects real-time causality and gives clients a simple, reliable view of the world. No reading old state or seeing things out of order. It’s a strong guarantee that consensus-based systems often provide. Even if under the hood messages flew around in parallel, the outcome is as if they happened in a neat sequence.
Finally, we looked at Apache Kafka as a real-world system that handles ordering in a nuanced way. Kafka ensures ordering on a per-key basis by routing keys to partitions (each a totally ordered log). It respects causality for related events (those sharing a key), while allowing unrelated events to be processed in parallel without a global order.
This design highlights an important engineering principle. You don’t always need a single total order on everything, as long as you group things such that causally related events are ordered within their group. Kafka’s partitions are those groups, and the use of keys gives developers control over what needs ordering.
Distributed systems live at the intersection of parallelism and order. Embracing parallelism means dealing with concurrency, understanding which events can go their separate ways and which must be kept in lockstep. Causality is our compass in this regard.
Techniques like logical clocks, consensus protocols, and careful data partitioning are the tools we use to navigate the sea of events, ensuring that the end result is consistent and correct. Whether you’re a junior dev debugging a race condition or a tech lead designing a cross-datacenter replication service, the same fundamentals apply. Know your events, understand their relationships, and make sure the system respects those relationships.
As you design or work with distributed systems, remember the story of the user creation and deletion. If you get the ordering wrong, you might delete a user that was never created or show an update that never happened. But with the right ordering guarantees in place, the story has a clear timeline. Every creation, update, deletion happens in a sensible sequence.
By orchestrating parallelism with an eye on causality, we ensure our distributed world doesn’t descend into chaos. Instead, it works in concert, like a well-timed symphony where every note (or event) falls in the right order. That’s the beauty of understanding parallelism and causality in distributed systems. It allows us to build systems that are both fast and correct, taking advantage of doing many things at once, without ever losing the plot of the story those things tell.