Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Asynchronous Processing and Messaging

System Design HLD-09.05

Ordering, partition keys and parallelism

Why you get per-key ordering instead of global ordering, how to choose a partition key, hot keys, head-of-line blocking and sequence numbers.

Advanced 10 min Last reviewed

The big idea

A shopping service emits cart updated, checked out and paid for each order. If a consumer sees paid before checked out, it may reject the payment, or ship an order that was never confirmed. Order matters. But a busy shop emits thousands of these events a second, far more than one consumer can handle in sequence. You need parallelism, and parallelism is the enemy of order.

The way out is to notice that you almost never need global order. Nobody cares whether Alice's checkout is processed before or after Bob's. You need order per entity: all of Alice's events in sequence, all of Bob's in sequence, and different customers in parallel. Logs deliver exactly that if you choose the right partition keyThe value used to pick a message's partition; messages with the same key always go to the same partition and keep their relative order..

Think of a supermarket with many checkout lanes. Each customer's items go through one lane in the order they are placed on the belt, while twenty customers are served at once in twenty lanes. Nobody needs the whole store to scan items in one global sequence.

How it works

Step 1: Global order costs all parallelism

A log is ordered within a partition and unordered across partitions. So global order needs one partition, which can be read by only one consumer in a group ( Kafka internals: the distributed log ). Your throughput is then the speed of a single consumer, and a single slow message stalls everything. Global order is occasionally worth it, such as a low-volume stream of configuration changes, but it does not scale.

Step 2: Per-key order through the partition key

Send every event about one entity with that entity's id as the key. The key is hashed to choose a partition, so all of Alice's events land in the same partition, in the order they were produced, and one consumer reads them in that order. Different keys spread across partitions and are processed in parallel.

Events keyed by customer id land in fixed partitions Events for alice, bob and carol are produced with the customer id as the key. All of alice's events glow along the same path into partition 3, in order: cart, checkout, paid. Bob's events go to partition 0. Carol's single event also hashes to partition 3. Partitions 1 and 2 receive events for other customers. Each partition is read by one consumer, so per-customer order is preserved while customers are processed in parallel. key = alice key = bob alice cart, checkout, paid bob signup, cart, checkout carol signup partition 0 bob, bob, bob partition 1 other keys partition 2 other keys partition 3 alice, alice, carol, alice consumer 1 consumer 4 Figure 1. The key decides the partition, so all of alice's events take the same glowing path into partition 3 and are consumed in order. Bob's go to partition 0. Different customers proceed in parallel; one customer's events never race each other.
keys.pyruns with python3 (standard library)
import zlib
from collections import Counter

P = 4
def partition(key):
    return zlib.crc32(key.encode()) % P      # stable hash: same key -> same partition

events = [("alice", "cart+book"), ("bob", "signup"), ("alice", "checkout"),
          ("carol", "signup"), ("bob", "cart+pen"), ("alice", "paid"), ("bob", "checkout")]

parts = {p: [] for p in range(P)}
for key, what in events:
    parts[partition(key)].append(f"{key}:{what}")
for p, log in parts.items():
    print(f"partition {p}: {log}")

# Every alice event is in one partition, in order. Across partitions there is no order.
alice = [e for e in parts[partition("alice")] if e.startswith("alice")]
print("alice in order:", alice == ["alice:cart+book", "alice:checkout", "alice:paid"])

# A hot key: one celebrity account produces most of the traffic.
traffic = ["celebrity"] * 900 + [f"user{i}" for i in range(100)]
load = Counter(partition(k) for k in traffic)
print("load per partition:", dict(sorted(load.items())))

Output

partition 0: ['bob:signup', 'bob:cart+pen', 'bob:checkout']
partition 1: []
partition 2: []
partition 3: ['alice:cart+book', 'alice:checkout', 'carol:signup', 'alice:paid']
alice in order: True
load per partition: {0: 24, 1: 26, 2: 24, 3: 926}

The first half shows per-key order: alice's three events sit in one partition, in order. The empty partitions are a reminder that hashing a handful of keys is lumpy; with millions of keys it evens out.

Step 3: Choosing a key

Pick the entity whose events must not be reordered, usually the one whose state the consumer updates:

  • Bank transfers: the account id, so debits and credits to one account apply in order.
  • Order lifecycle: the order id. Two orders by the same customer can run in parallel.
  • Chat messages: the conversation id, so messages appear in the order sent.

Two pressures pull on the choice. The key must be coarse enough that everything needing mutual order shares it: if a consumer checks a customer's credit limit across all their orders, the customer id may be the right key, not the order id. It must be fine enough to spread load: a key with few distinct values, such as country, gives a few giant partitions.

Step 4: Hot keys

The second half of the code is the danger: one celebrity account producing 90 percent of the traffic sends 90 percent of the load to one partition and one consumer, while the others idle. Adding partitions does not help, because one key always maps to one partition. The fixes all trade away something:

  • Split the key (key salting, Hot partitions and the celebrity problem ): celebrity#0 to celebrity#7 spreads the load but loses ordering across the sub-keys.
  • Handle the hot entity separately: a dedicated topic or consumer for known heavy hitters.
  • Make the work smaller: aggregate or sample hot-key events before the heavy processing.

Step 5: Head-of-line blocking

Per-partition order has a cost. If a message is slow or keeps failing, everything behind it in that partition waits, including other customers who happen to share the partition. This is head-of-line blockingWhen one stuck or slow item at the front of an ordered queue delays every item behind it, even unrelated ones.. A single poison message can halt a partition for hours.

Consumers need a policy for this:

  • Timeouts on each message, so a hung call does not block forever.
  • Bounded retries, then park the message in a retry or dead letter topic and move on ( Retries, dead letter queues and poison messages ). If later events for the same key must not overtake it, park those too, or mark the key as blocked until the parked message is handled.
  • Key-level parallelism inside a partition: process different keys from the same partition concurrently while keeping each key sequential, so one slow key only blocks itself.

Going deeper

Order can break before the broker

Ordering in a partition is the order the broker received messages. A producer that retries a failed send while later messages succeed can reorder them. Kafka's idempotent producer uses per-partition sequence numbers so the broker keeps the original order even with retries in flight. Two different producers writing the same key have no order between them at all, unless they coordinate.

Sequence numbers and versions

When order cannot be fully guaranteed end to end, put it in the data. The producer stamps each event with a per-entity sequence number or version (order 42, version 7). The consumer applies an event only if it is the next version, buffers or rejects events from the future, and ignores stale ones. This also makes the consumer safe against duplicates ( Delivery semantics ).

Changing the partition count

hash(key) % partitions changes when the partition count changes, so for a while a key's old events sit in one partition and its new events in another, and they can be consumed out of order. Plan partition counts ahead, or drain and switch over deliberately.

Trade-offs

OptionWins whenLoses when
Single partition (global order) Low-volume streams where every event must be in one sequence Throughput is one consumer; one bad message stops everything
Key by entity id (per-key order) The usual choice: order where it matters, parallel everywhere else Hot keys concentrate load; cross-entity ordering is gone
No key (round-robin) Even load when events are independent, such as metrics or clicks Any ordering between related events
Salted keys for hot entities Spreading a single heavy key over many partitions Ordering for that entity; readers must merge sub-keys

Common mistakes

  • Asking for global ordering when per-entity ordering is all the business needs.
  • Choosing a low-cardinality key (country, status) and creating a few overloaded partitions.
  • Choosing a key too fine (event id), so related events lose their mutual order.
  • Retrying a failing message forever in place and blocking every key in its partition.
  • Increasing the partition count on a live keyed topic without planning for the reshuffle.

In the wild

Kafka, Amazon Kinesis (partition key to shard), Azure Event Hubs and Google Cloud Pub/Sub (ordering keys) all offer the same contract: order is kept for messages that share a key, not across keys. Amazon SQS FIFO queues express it with message group ids, processing each group in order while different groups proceed in parallel.

Check yourself

  1. A bank processes account transactions from a log. Which key keeps each account's debits and credits in order?

    Show answer

    B. Account id All events for one account share the key, so they go to one partition and are consumed in order.

  2. You key by country. Most traffic comes from one country. What happens?

    Show answer

    B. One partition and its consumer take most of the load while others idle Low-cardinality keys with skewed traffic create hot partitions that more partitions cannot fix.

  3. One message for customer 17 keeps failing and the consumer retries it forever. Who is affected?

    Show answer

    B. Every key in that partition waits behind it Head-of-line blocking: the partition is processed in order, so everything behind the stuck message waits.

  4. You salt a hot key into 8 sub-keys. What do you give up?

    Show answer

    B. Ordering across the 8 sub-keys for that entity Different sub-keys land in different partitions, which have no order between them.

Recap

  • Logs order messages within a partition only; global order means a single partition and no parallelism.
  • Key events by the entity whose order matters to get per-entity order with parallelism across entities.
  • Good keys are coarse enough to group related events and fine enough to spread load.
  • Hot keys overload one partition; splitting them trades away that entity's ordering.
  • Head-of-line blocking needs timeouts, bounded retries and parking; sequence numbers protect order end to end.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up