System Design HLD-19.21
Design an ad click aggregator
Design an ad click aggregator: high-volume event ingestion, windowed stream aggregation, dedupe, exactly-once sinks and reconciliation with batch.
The problem
Every time someone clicks an ad, an advertiser owes money. Advertisers want to see clicks per ad per minute on a dashboard, almost live, and the business needs exact totals per hour to bill them. A large ad network sees billions of clicks a day.
It is a counting problem, but counting at this volume, in real time, with money attached, has three traps. Events are duplicated (clients retry, pipelines redeliver), so naive counts are too high. Events are late (a phone was offline, a region's pipeline lagged), so a minute's count is never quite final. And the fast path can be wrong, so there must be a slower, exact path that checks it. The standard answer is a streaming pipeline for freshness plus a batch pipeline for truth, reconciled.
Requirements
Functional
- Record every ad click and redirect the user to the advertiser's page.
- Show clicks per ad per minute, with a delay of about a minute.
- Produce exact hourly and daily totals per ad and campaign for billing.
- Filter invalid clicks (duplicates, bots) before they are counted.
Non-functional
- Scale: 1 billion clicks a day; 2 million active ads.
- Freshness: dashboard counts within about a minute.
- Correctness: billing counts exact, each valid click counted once.
- Durability: no raw click is ever lost, so any number can be recomputed.
Out of scope
- Impressions (same design, ten times more events), ad serving and auctions, and fraud models (a filter hook).
Estimates
- Clicks: 1 billion a day is about 11,600 per second on average; plan for 5x peaks, about 58,000 per second.
- Raw log: at about 200 bytes per event, 200 GB a day, about 73 TB a year, kept in cheap object storage for recomputation and audits.
- Aggregates: one row per ad per minute with clicks. Most ads get no click in most minutes, so if 2 million ads average 30 clicked minutes a day, that is 60 million rows a day, a few GB.
- Hot keys: a viral ad might get a large share of a region's clicks, so partitioning by ad id alone can overload one partition.
API
GET /click?ad=8812&imp=ab12...&sig=... -> 302 Location: https://advertiser.example/landing
GET /v1/ads/{ad_id}/clicks?from=2026-10-04T10:00Z&to=...&granularity=minute
-> [{ minute, clicks }]
GET /v1/campaigns/{id}/billing?date=2026-10-04 # exact, from the batch path
The click URL carries a signed impression id, which becomes the base of the click id used for deduplication and lets the tracker reject forged clicks.
Data model
- Click event (log, partitioned by ad id, Kafka internals: the distributed log ):
click_id, ad_id, campaign_id, event_time, user_hash, ip_hash, country. - Raw archive (object storage): the same events in hourly files, the source of truth for batch jobs.
- Minute aggregates (an OLAP or wide-column store):
ad_id, minute -> clicks, source (stream or batch), version. - Billing aggregates (relational):
campaign_id, hour -> clicks, cost, written only by the batch path after reconciliation.
High-level design
- Ingest path The click tracker validates the signed click URL, appends the event to a durable log partitioned by ad id, and redirects the user. Logging is fast and asynchronous so the redirect is never slowed down.
- Stream path A stream processor drops duplicate click ids, filters invalid clicks and counts per ad in one-minute event-time windows. When the watermark passes a window plus its allowed lateness, the count is upserted into the aggregate store for dashboards.
- Batch path Raw events are archived hourly. A batch job recomputes exact counts with full deduplication and late events included, reconciliation overwrites any stream counts that differ, and billing uses only these totals.
Deep dives
Deep dive 1: Windows, watermarks and late events
A click belongs to the minute it happened (event time), not the minute it arrived. The stream processor groups clicks into one-minute tumbling windows by event time ( Stream processing ). But when is minute 10:04 complete? Events can always arrive later. A watermark is the processor's estimate that "all events before time T have arrived", usually the newest event time seen minus some slack. A window is emitted when the watermark passes its end plus an allowed lateness; events arriving after that are too late for the stream.
Choose the lateness and you choose the trade-off: wait longer and counts are more complete but the dashboard is staler; wait less and it is fresher but more clicks miss their window. Late events need not be thrown away: they can update the window again (if the sink accepts updates) or be left to the batch path.
Deep dive 2: Deduplication and exactly-once counting
Duplicates come from client retries, tracker retries to the log, and the stream processor itself replaying from its last checkpoint after a crash ( Delivery semantics ). Three defences:
- Click id dedupe: the stream processor keeps the click ids it has seen recently (in its state store, with a TTL of a few hours, or a Bloom filter to save memory, Probabilistic data structures ) and drops repeats.
- Checkpointed state: the processor periodically snapshots its state (window counts, seen ids) together with its log offsets. After a crash it restarts from the snapshot and replays from those offsets, so every event affects the state exactly once.
- Idempotent sink: writes are upserts keyed by
(ad_id, minute)that set the count rather than add to it, so writing the same window twice is harmless.
Together these give "effectively once" counts in the stream. Here is the whole picture on 3,000 clicks over five minutes, with 3 percent client retries and 2 percent of phones that were offline for a while:
# Count clicks per ad per minute two ways: a streaming job (dedupe, 1-minute
# windows, closes a window 20 s after the watermark passes it) and a batch
# job over the full raw log the next hour. Then reconcile.
import random
from collections import Counter
rnd = random.Random(21)
events = [] # (arrival s, click id, ad, event time s)
for i in range(3000):
t = rnd.uniform(0, 300) # five minutes of clicks
ad = rnd.choice("ABC")
delay = rnd.expovariate(1 / 2) # usual network delay
if rnd.random() < 0.02:
delay += rnd.uniform(30, 120) # phone was offline: very late
events.append((t + delay, f"c{i}", ad, t))
if rnd.random() < 0.03: # client retry: same click
events.append((t + delay + rnd.uniform(0.1, 5), f"c{i}", ad, t))
events.sort()
LATENESS = 20
stream, seen, late, dupes = Counter(), set(), 0, 0
watermark = 0.0
for arrival, cid, ad, t in events:
watermark = max(watermark, t - 5) # newest event time - slack
if cid in seen:
dupes += 1
continue
seen.add(cid)
minute = int(t // 60)
if (minute + 1) * 60 + LATENESS < watermark:
late += 1 # window already closed
continue
stream[(ad, minute)] += 1
batch = Counter()
for cid, ad, t in {(c, a, t) for _, c, a, t in events}: # exact dedupe
batch[(ad, int(t // 60))] += 1
print(f"raw events {len(events)}, duplicates dropped {dupes}, "
f"too late for the stream {late}")
print(f"stream total {sum(stream.values())}, "
f"batch total {sum(batch.values())}")
diffs = {k: batch[k] - stream[k] for k in batch if batch[k] != stream[k]}
print(f"windows that differ: {len(diffs)} of {len(batch)}")
for (ad, minute), d in sorted(diffs.items())[:4]:
print(f" ad {ad} minute {minute}: stream {stream[(ad, minute)]}, "
f"batch {batch[(ad, minute)]} (+{d})")Output
raw events 3089, duplicates dropped 89, too late for the stream 36
stream total 2964, batch total 3000
windows that differ: 12 of 15
ad A minute 0: stream 196, batch 197 (+1)
ad A minute 1: stream 204, batch 208 (+4)
ad A minute 2: stream 196, batch 198 (+2)
ad A minute 3: stream 187, batch 190 (+3) Deduplication removed all 89 retries. But 36 clicks arrived after their window had closed, so the stream counted 2,964 and the batch job, which simply sees every event, counted 3,000. Twelve of the fifteen ad-minute windows were a few clicks short. On a dashboard that is fine; on an invoice it is not.
Deep dive 3: Reconciliation with batch
The batch path recomputes the same aggregates from the raw archive, an hour or a day later, with complete data and exact deduplication. Reconciliation ( Reconciliation and correctness jobs ) compares the two per (ad, minute):
- Small differences, as in the sample, are expected from late events; the batch value overwrites the stream value (the row's
sourcebecomesbatch), so dashboards converge to the exact numbers. - Large or systematic differences mean a bug or an outage in one path, and raise an alert before anyone is billed.
- Billing reads only batch-reconciled totals.
Running two pipelines doing the same computation is the lambda architecture. Its alternative, the kappa architecture, uses one streaming codebase and reprocesses the log from the start when a fix is needed. Lambda costs two implementations that must agree; kappa costs long log retention and replay capacity. With money involved, many teams accept the cost of an independent exact check.
Deep dive 4: Hot ads and scaling ingestion
Partitioning the log by ad id keeps each ad's clicks together, but one viral ad can overwhelm its partition ( Hot partitions and the celebrity problem ). The fix is two-stage aggregation: spread a hot ad's events across several partitions with a random suffix (ad_8812#3), count partial windows per suffix, and add the partials in a second, much smaller stage. The tracker itself is stateless and scales horizontally behind a load balancer; it only appends to the log and redirects.
Try it
Try this:
- Choose tumbling windows, the shape the aggregator uses. Lower the watermark delay until the late event misses its window. Which count is now wrong?
- Raise the delay until it is counted. What did that cost the dashboard?
- In the sample, 36 clicks were too late. Which path counts them, and when does the dashboard show the corrected value?
Bottlenecks and failure modes
| Failure | Impact | Mitigation |
|---|---|---|
| Duplicate events | Advertisers overbilled | Click id dedupe, checkpointed state, idempotent upserts |
| Late events | Stream counts too low | Allowed lateness; batch recompute overwrites |
| Stream processor crash | Gap or double counts | Restore from checkpoint and replay from saved offsets |
| Viral ad | Hot partition | Salted keys and two-stage aggregation |
| Bug in one pipeline | Wrong numbers | Reconciliation alerts on large differences |
| Raw events lost | Cannot recompute or audit | Durable replicated log; archive before expiry |
Trade-offs we made
| Option | Wins when | Loses when |
|---|---|---|
| Stream plus batch with reconciliation (chosen) | Fresh dashboards and exact billing; independent check | Two pipelines that must agree |
| Stream only (kappa) | One codebase; reprocess by replaying the log | Exactness depends on one pipeline; replays are heavy |
| Short allowed lateness (chosen) | Dashboard within about a minute | More events left to the batch correction |
| Upserts that set counts (chosen) | Replays and retries are harmless | The sink must support keyed updates |
Extensions
- Impressions and click-through rate, joining two streams by impression id.
- Invalid traffic detection with rules and models, feeding refunds.
- Rollups to hourly and daily for long ranges.
- Ad-hoc queries over the raw archive with a SQL engine.
In the wild
Where you will see this: advertising platforms, but also payment volume dashboards, usage-based cloud billing, video view counts and API metering. Wherever counts drive money, the same shape appears: a durable event log, a streaming path for freshness, an exact batch path for truth, and reconciliation between them.
Check yourself
-
Why are clicks counted by event time rather than arrival time?
Show answer
B. A click belongs to the minute it happened, even if it arrives late Billing and dashboards are about when clicks happened; delays must not move them between minutes.
-
What does a watermark tell the stream processor?
Show answer
B. An estimate that all events before a time have arrived, so windows before it can be closed Windows close when the watermark passes their end plus the allowed lateness.
-
The stream processor crashes and replays from its last checkpoint. Why are counts not doubled?
Show answer
B. State and offsets are restored together, and the sink upsert sets rather than adds counts Checkpointed state plus idempotent writes make replay safe.
-
Why does billing use the batch totals?
Show answer
B. They include late events and exact deduplication and are reconciled against the stream The exact path is the system of record; the stream is for freshness.
Recap
- Log every click durably first; the raw log makes every number recomputable.
- Count per ad in event-time windows closed by a watermark with allowed lateness.
- Dedupe by click id, checkpoint state with offsets, and upsert counts so replays are harmless.
- A batch job recomputes exact counts; reconciliation overwrites stream counts and feeds billing.
- Split hot ads with salted keys and two-stage aggregation.
Next up
- Crack the system design round learning path