System Design HLD-09.10
Stream processing
Computing over unbounded data: batch vs stream, tumbling, sliding and session windows, event time vs processing time, watermarks and stateful operators.
The big idea
A fraud team wants to flag a card used more than 5 times in one minute. A batch job that runs every night finds it 20 hours too late. What they need is a computation that runs continuously over an endless stream of card events and produces results within seconds. That is stream processingContinuously computing results over unbounded data as each event arrives, instead of periodically over a fixed dataset..
The hard part is not speed; it is that the data never ends. "Count events per minute" is easy on a file: read it, group, done. On a stream you must decide which minute an event belongs to, when a minute is complete enough to report, and what to do with an event from 12:00:59 that shows up at 12:03 because a phone was in a tunnel. Windows, event time and watermarks are the answers.
Think of counting votes from polling stations. You report per-hour totals, but some ballot boxes arrive late by truck. You count each ballot in the hour it was cast, not the hour it arrived, and you decide how long to wait for stragglers before announcing each hour's result.
How it works
Step 1: Batch vs stream
A batch job processes a bounded dataset: yesterday's logs, last month's orders. It sees all the data, so results are complete, but they arrive late. A stream job processes unbounded data as it arrives. Results are fast but must be produced before all the data could possibly be there. Many systems run both, and modern engines such as Apache Flink treat a batch as a stream that happens to end.
Stateless steps, such as filtering or transforming each event, are easy to stream. The interesting ones keep state across events: counts, sums, joins, "has this card been seen in the last minute". State is grouped into windows.
Step 2: Windows
A windowA finite slice of an unbounded stream, defined by time or activity, over which an aggregation such as a count or sum is computed. cuts the endless stream into finite pieces you can aggregate:
- Tumbling windows are fixed-size and non-overlapping: 12:00 to 12:01, 12:01 to 12:02. Each event belongs to exactly one. Good for per-minute dashboards and billing.
- Sliding (hopping) windows are fixed-size but overlap: 1 minute long, starting every 10 seconds. Each event belongs to several. Good for moving averages and "more than 5 in any one-minute span".
- Session windows have no fixed size: a session lasts while events keep coming and closes after a gap of inactivity, say 30 minutes. Good for user sessions and bursts of activity.
Step 3: Event time vs processing time
Every event has two times. Event time is when it happened, stamped by the device or service that produced it. Processing time is when the stream processor sees it. They differ because of network delays, retries, consumer lag, and phones that queue events while offline ( Time and clocks ).
If you window by processing time, results are simple and immediate but wrong whenever arrival is delayed: a burst of buffered events lands in the wrong minute, and replaying yesterday's data produces different answers than the first run. Windowing by event timeThe time at which an event actually occurred, as recorded in the event itself, as opposed to when it was processed. gives correct, reproducible results, at the cost of the next question: when is a window done?
Step 4: Watermarks
A watermarkA stream processor's running estimate that no more events older than time T are expected; when the watermark passes a window's end, the window is considered complete and its result is emitted. is the engine's declaration "I believe I have now seen all events up to time T". When the watermark passes the end of a window, the window is closed, its result is emitted, and its state can be freed.
A common strategy: watermark = largest event time seen so far minus an allowed delay. A small delay gives fast results but drops more stragglers. A large delay catches more late events but holds every result back by that much. There is no setting that is both instant and complete; you choose per use case.
Events that arrive after their window closed are late. Engines let you drop them, send them to a side output for separate handling, or keep windows open a little longer ("allowed lateness") and emit corrected results.
Try it
Try this:
- With tumbling windows of 10 s and a 2 s watermark delay, the late event is dropped: its window (10 to 20 s) closed when the watermark reached 24 s. Raise the delay until it is counted. What is the cost?
- Switch to sliding windows. The same event belongs to two windows. With a small delay, find the setting where it misses one window but is still counted in the other.
- Switch to session windows with a gap of 6 s, then raise the delay. When the late event is accepted, watch it bridge two sessions into one.
Going deeper
Stateful operators and fault tolerance
Window contents, running counts and join buffers are state, and a crash must not lose them. Engines keep state locally for speed (in memory or an embedded store such as RocksDB) and make it durable: Flink takes periodic consistent checkpoints of all operator state together with the input offsets, and Kafka Streams backs each state store with a changelog topic. After a failure, the job restores the last checkpoint and replays input from the matching offsets. Combined with transactional sinks, this is how stream processors achieve exactly-once results ( Delivery semantics ).
Joins on streams
Joining two streams, such as ad impressions with clicks, needs a time bound: "match a click to an impression from the previous 30 minutes". The engine buffers each side for that long as state. Joining a stream with a table, such as enriching orders with the current customer record, keeps the table as state updated from its own changelog, often fed by CDC ( Transactional outbox and change data capture ).
Engines
Apache Flink processes event by event with event time, watermarks and checkpoints as first-class features. Kafka Streams is a library that runs inside your own services and reads and writes Kafka. Spark Structured Streaming runs streams as a series of small batches. Managed services such as Google Cloud Dataflow run Apache Beam pipelines, whose model introduced much of the vocabulary used here.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Batch processing | Complete, simple results where hours of delay are fine (reports, training data) | Freshness |
| Stream, processing-time windows | Simple monitoring where approximate timing is acceptable | Correctness under delays; replays give different answers |
| Stream, event-time windows + watermarks | Correct, reproducible results within seconds or minutes | Complexity, state size and a latency vs completeness choice |
| Long allowed lateness | Late-heavy sources such as mobile devices | More state held longer, and results that change after first emitted |
Common mistakes
- Windowing by processing time and getting wrong numbers whenever consumers lag or replay.
- Setting the watermark delay with no measurement of how late events actually arrive.
- Silently dropping late events instead of counting them or routing them to a side output.
- Unbounded state: session windows or joins with no timeout, growing until the job runs out of memory.
- Forgetting that downstream consumers may receive updated results for a window that was already reported.
In the wild
Stream processing runs fraud detection, real-time dashboards, recommendations and alerting at most large consumer companies. Apache Flink, Kafka Streams and Spark Structured Streaming are the common open source engines, and the managed Google Cloud Dataflow service runs pipelines written with Apache Beam, whose model of windows, watermarks and triggers was described in the Dataflow paper from Google.
Check yourself
-
You need 'more than 5 transactions in any 60-second span'. Which window fits best?
Show answer
B. Sliding 60 s with a small slide Tumbling windows miss bursts that straddle a boundary; overlapping sliding windows catch any 60-second span (to within the slide).
-
Events are counted per minute by processing time. Consumers fall 10 minutes behind and catch up. What happens to the counts?
Show answer
B. The backlog lands in the catch-up minutes, distorting the counts Processing-time windows measure when events were seen, not when they happened.
-
The watermark is 'max event time seen minus 5 s'. You raise it to minus 60 s. What changes?
Show answer
A. Fewer late events are dropped, but every window's result is emitted about 55 s later A watermark trades completeness for latency; waiting longer catches stragglers at the cost of delay.
-
How do stream processors keep window state safe across crashes?
Show answer
B. Periodic checkpoints of state with input offsets (or changelog topics), restored and replayed after failure Restoring state and input position together lets the job resume as if the crash never happened.
Recap
- Stream processing computes continuously over unbounded data; batch waits for complete data.
- Tumbling, sliding and session windows cut the stream into finite pieces to aggregate.
- Event time gives correct, reproducible results; processing time is simpler but wrong under delay.
- Watermarks decide when a window is complete, trading latency against dropped late events.
- State is checkpointed with input offsets so jobs recover exactly where they left off.