Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Asynchronous Processing and Messaging

System Design HLD-09.03

Kafka internals: the distributed log

How Kafka works inside: topics, partitions and offsets, replication with in-sync replicas, consumer groups and rebalancing, retention and compaction.

Intermediate 18 min Last reviewed

The big idea

In Message queues vs pub/sub vs event streams you met the event log: an append-only list of messages that readers walk through at their own pace. One log on one machine is easy. Kafka's achievement is making that log huge, fast and fault tolerant: millions of messages a second, kept for days, surviving the loss of servers, read by dozens of independent applications.

It does that with three ideas. Split each log into partitions so many machines share the load. Replicate each partition so a dead server loses nothing. Let consumer groups divide partitions among their members so reading scales too. Almost every behaviour you will see in production, good or surprising, follows from those three.

Think of a bank with many tellers' ledgers instead of one giant ledger. Each customer's transactions always go to the same ledger, so their history is in order. Each ledger has photocopies in other branches in case of fire. Auditors split the ledgers among themselves, one auditor per ledger, each with a bookmark showing how far they have checked.

How it works

Step 1: Topics, partitions and offsets

A topicA named stream of messages in Kafka, such as 'orders', split into one or more partitions. is a named stream, such as orders. It is split into partitionsOne ordered, append-only log that holds a subset of a topic's messages; the unit of parallelism, replication and ordering in Kafka., and each partition is a separate append-only log stored on disk. Each message in a partition gets the next sequential offset: 0, 1, 2 and so on.

A Kafka topic with three partitions and one consumer group A producer sends messages with keys. The key decides the partition: user-7 always goes to partition 0. The topic orders has three partitions drawn as append-only logs with offsets. Partition 0 has entries up to offset 7, partition 1 up to 5, partition 2 up to 6. A consumer group with two consumers reads them: consumer A owns partitions 0 and 1 and has committed offsets 6 and 5, consumer B owns partition 2 at offset 3. hash(key) % 3 = 0 Producer key = user-7 partition 0 0 1 2 3 4 5 | 6 7 partition 1 0 1 2 3 4 | 5 partition 2 0 1 2 | 3 4 5 6 Consumer A p0 @6, p1 @5 Consumer B p2 @3 Figure 1. The key picks the partition, so all of user-7's events land in partition 0, in order. Each partition is its own log. In the group, each partition is read by exactly one consumer, which commits how far it has read (the bar marks that committed offset).

A producer chooses the partition for each message. If the message has a key (a user id, an order id), the default partitioner hashes the key, so the same key always lands in the same partition. Messages without a key are spread across partitions for balance.

Ordering is the key consequence: Kafka guarantees order within a partition, not across a topic. All of user-7's events are in partition 0, in the order they were written. An event in partition 0 and one in partition 2 have no defined order relative to each other ( Ordering, partition keys and parallelism ).

On disk, each partition is a series of segment files. New messages are appended to the active segment; old segments are deleted or compacted as a unit. Writes are sequential, reads of recent data are usually served from the operating system's page cache, and consumers fetch in batches. That is how a modest cluster moves a very large volume of data.

Step 2: Brokers and replication

A Kafka server is a brokerA Kafka server that stores partitions and serves producers and consumers., and a cluster has several. Each partition is replicated to several brokers, set by the topic's replication factor (3 is common). For each partition, one replica is the leader, which handles all writes and, by default, reads, and the others are followers that continuously fetch from the leader, just like the leader-follower replication in Replication: leader-follower . Leadership is spread so every broker leads some partitions.

The leader tracks which followers are caught up: the in-sync replicas (ISR)In-sync replicas: the set of replicas (including the leader) that are fully caught up with the leader of a partition within a configured time.. A follower that falls too far behind is dropped from the ISR, and rejoins once it catches up. The producer chooses how much durability it wants with acks:

  • acks=0: do not wait. Fastest, may lose messages.
  • acks=1: the leader has written it. Lost if the leader dies before followers copy it.
  • acks=all: every replica in the ISR has it. Combined with min.insync.replicas=2 and replication factor 3, an acknowledged message survives the loss of any one broker, and writes are refused if fewer than 2 replicas are in sync.

If a leader's broker dies, a new leader is chosen from the ISR, so it has every acknowledged message. Cluster metadata, such as which broker leads which partition, is managed by a controller. Modern Kafka runs those controllers as a Raft-based quorum called KRaft ( Consensus: Raft ) instead of the external ZooKeeper ensemble it used for years.

Step 3: Consumer groups and rebalancing

A consumer groupA set of consumers sharing one group id; Kafka assigns each partition of a topic to exactly one consumer in the group so they share the work. is a set of consumers that share the work of reading a topic. Kafka assigns each partition to exactly one member of the group. Members periodically commit the offset they have processed up to, stored in an internal Kafka topic, so if a consumer restarts, it resumes from its last commit.

That rule, one consumer per partition per group, gives three properties at once:

  • Ordering per partition: one reader per partition means messages are processed in order.
  • Scaling up to the partition count: 6 partitions can keep 6 consumers busy. A 7th sits idle.
  • Independent groups: a different group, such as fraud alongside billing, gets its own assignment and offsets and reads the same data without affecting the first.

When a consumer joins, leaves or stops sending heartbeats, the group rebalancesRedistributing a topic's partitions among the current members of a consumer group after a member joins, leaves or fails.: partitions are reassigned among the members still present. Older protocols paused the whole group during a rebalance; newer cooperative protocols move only the partitions that have to change hands, so frequent rebalances hurt much less.

Consumer lagHow far a consumer group is behind: the newest offset in a partition minus the group's committed offset., the newest offset minus the committed offset, is the number to watch. Growing lag means consumers cannot keep up.

Step 4: Retention and compaction

Kafka does not delete messages when they are read. Each topic has a retention policy: delete segments older than a time (7 days by default) or beyond a size. Any consumer can rewind to any offset still retained.

Log compaction is the other policy: Kafka keeps at least the latest message for each key and removes older ones in the background. A compacted topic is a durable table of "latest value per key", perfect for a changelog of user profiles or configuration. A message with a key and a null value, a tombstone, deletes that key during compaction.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Start with 4 partitions and 2 consumers and press Play. Lag climbs: 2 consumers handle 2 messages per tick against 3 produced.
  2. While it plays, raise consumers to 4. The group rebalances, each consumer gets one partition, and lag drains. Now go to 6: two consumers sit idle.
  3. Set partitions to 2 and consumers to 4. No number of consumers helps, because only 2 can be active. That is why the partition count is chosen up front with future throughput in mind.

Going deeper

Choosing the number of partitions

Partitions cap consumer parallelism, so teams over-provision somewhat: if you expect to need 12 consumers next year, do not start with 3 partitions. But each partition costs open files, memory and replication work, and more partitions mean more leaders to move when a broker fails, so very large counts per broker become a problem. Adding partitions later is possible but changes which partition a key maps to, breaking per-key ordering at the moment of change.

Delivery and offsets

When you commit the offset decides your delivery semantics. Commit before processing and a crash loses messages (at most once); commit after processing and a crash replays some (at least once). Kafka also offers idempotent producers and transactions for exactly-once processing inside Kafka, covered in Delivery semantics .

Trade-offs

OptionWins whenLoses when
Many partitions High throughput and many parallel consumers More overhead per broker; ordering only per partition; repartitioning breaks key placement
acks=all with min.insync.replicas=2 Acknowledged messages survive a broker failure Higher write latency; writes refused when too few replicas are in sync
acks=1 Lower latency where losing a few messages on failover is acceptable Durability when the leader dies before replicating
Compacted topic Latest value per key kept forever: changelogs, state restore Full history; not every message is kept

Common mistakes

  • Expecting global ordering across a topic; order holds only within a partition.
  • Starting with too few partitions, then hitting a parallelism ceiling no number of consumers can fix.
  • Using acks=1 or a replication factor of 1 for data you cannot lose.
  • Doing slow work between polls so the consumer misses its deadline, triggering endless rebalances.
  • Not monitoring consumer lag, the clearest signal that a pipeline is falling behind.

In the wild

Check yourself

  1. A topic has 6 partitions. A consumer group has 8 consumers. How many are actively reading?

    Show answer

    B. 6 Each partition goes to exactly one consumer in a group, so at most 6 are busy and 2 sit idle.

  2. You need all events for one order to be processed in sequence. What do you do?

    Show answer

    B. Use the order id as the message key so they share a partition Same key means same partition, and order is guaranteed within a partition.

  3. With replication factor 3, acks=all and min.insync.replicas=2, two of the three brokers for a partition fail. What happens to writes?

    Show answer

    B. They are refused until enough replicas are in sync Kafka refuses writes it cannot replicate to at least 2 in-sync replicas, choosing consistency over availability.

  4. What does log compaction keep?

    Show answer

    B. At least the latest message for each key Older values for a key are removed, so the topic becomes a durable latest-value-per-key table.

  5. Consumer lag on a partition keeps growing. What does it mean?

    Show answer

    B. Consumers are processing slower than producers are writing to it Lag is newest offset minus committed offset; growth means the consumers cannot keep up.

Recap

  • A topic is split into partitions; each partition is an append-only log with sequential offsets.
  • The message key picks the partition, and ordering is guaranteed only within a partition.
  • Partitions are replicated with a leader and followers; acks=all plus the ISR protects acknowledged writes.
  • A consumer group gives each partition to one member, so partitions cap parallelism; groups read independently.
  • Retention keeps messages after reading for replay; compaction keeps the latest value per key.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up