Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Design Case Studies

System Design HLD-19.02

Design a pastebin

Design a pastebin: metadata in a database, content in object storage behind a CDN, unguessable ids, size limits, access control and expiry cleanup.

Beginner 15 min Last reviewed

The problem

A pastebin lets someone paste a block of text, such as a log excerpt, a config file or a code snippet, and get back a link to share. Anyone with the link can read it; some pastes are public, some unlisted, some private to the owner, and many expire after an hour or a day.

It looks like the URL shortener with bigger values, and the comparison is useful: the read path and id generation are similar. What changes is the size of the payload. A short link is 500 bytes; a paste can be a megabyte. That single difference moves the content out of the database and into object storage, and brings size limits, abuse prevention and cleanup into the design.

Requirements

Functional

  • Create a paste with text, optional title, syntax language, visibility (public, unlisted, private) and expiry (10 minutes to never).
  • Read a paste by its link; private pastes only by their owner.
  • Delete your own paste.
  • Pastes expire automatically.

Non-functional

  • Scale: 1 million new pastes a day, about 10 reads per paste on average, popular pastes far more.
  • Size: average 10 KB, hard limit 1 MB per paste.
  • Latency: reads p99 under 200 ms worldwide.
  • Durability: a paste must not be lost before it expires.
  • Unlisted pastes must be practically impossible to find without the link.

Out of scope

  • Editing pastes (treat a change as a new paste), comments, search across public pastes.

Estimates

estimates.pyruns with python3 (standard library)
# Back-of-the-envelope numbers for a pastebin.
NEW_PER_DAY = 1e6      # new pastes per day
READS_PER_PASTE = 10   # average views per paste
AVG_KB = 10            # average paste size (limit is 1 MB)
META_BYTES = 300       # id, owner, expiry, visibility, size, key
DAY = 86_400

w = NEW_PER_DAY / DAY
r = w * READS_PER_PASTE
print(f"writes: {w:.0f}/s avg   reads: {r:.0f}/s avg, {3 * r:.0f}/s at 3x peak")

content_tb = NEW_PER_DAY * 365 * AVG_KB / 1e9
meta_gb = NEW_PER_DAY * 365 * META_BYTES / 1e9
print(f"per year: content {content_tb:.2f} TB, metadata {meta_gb:.0f} GB")

# Read bandwidth if every view downloaded the full paste
mbps = r * AVG_KB * 8 / 1000
print(f"read bandwidth: {mbps:.0f} Mbit/s avg")

# How guessable are random base62 ids after 10 years of pastes?
pastes_10y = NEW_PER_DAY * 365 * 10
for k in (8, 12):
    space = 62 ** k
    print(f"{k}-char ids: a random guess hits a paste 1 in {space / pastes_10y:,.0f}")

Output

writes: 12/s avg   reads: 116/s avg, 347/s at 3x peak
per year: content 3.65 TB, metadata 110 GB
read bandwidth: 9 Mbit/s avg
8-char ids: a random guess hits a paste 1 in 59,819
12-char ids: a random guess hits a paste 1 in 883,908,702,027

The load is small: about 12 writes and 116 reads per second, a few hundred at peak. Two numbers shape the design anyway:

  • Content is 33 times bigger than metadata: 3.65 TB a year of text against about 110 GB of metadata. Big blobs and small searchable records want different stores.
  • Id length is a security decision: with 3.65 billion pastes after ten years, an 8-character random id is hit by 1 random guess in about 60,000, so a scraper making a million guesses would find unlisted pastes. Twelve characters push that to about 1 in 900 billion.

API

POST /v1/pastes
  body: { "content": "...", "title": "nginx.conf", "language": "nginx",
          "visibility": "unlisted", "expires_in": "1d" }
  201 Created { "id": "q8ZtR2mKx0aB", "url": "https://paste.example/q8ZtR2mKx0aB" }
  413 Payload Too Large over 1 MB

GET /v1/pastes/{id}           metadata and content (403 if private and not the owner)
GET /raw/{id}                 plain text, cacheable
DELETE /v1/pastes/{id}        owner only

Data model

Metadata in a database ( Choosing the right database ), keyed by id:

FieldNotes
id12 random base62 characters, primary key
owner_idnullable for anonymous pastes
visibilitypublic, unlisted, private
language, title, size_bytes
created_at, expires_atexpires_at indexed for cleanup
content_keyobject storage key, for example p/q8/q8ZtR2mKx0aB

Content in object storage, one object per paste, compressed. Text compresses well, often several times smaller. Object storage is cheap per GB, durable by design and scales without sharding work, which is exactly what large, write-once, read-by-key blobs need. Keeping blobs out of the database also keeps the database small enough to hold its working set in memory.

High-level design

Pastebin architecture with write, read and cleanup paths Write path: a client posts to the paste service, which checks size and rate limits, uploads the content to object storage and then inserts the metadata row. Read path: the client fetches through a CDN; on a miss the paste service reads metadata, checks visibility and expiry, and streams content from object storage. Cleanup path: a scheduled job finds expired rows, deletes the objects, then the rows, and purges the CDN. POST /pastes 1. put blob 2. insert row GET /raw/id on miss check row stream find expired delete purge Client CDN public + unlisted Paste service limits, auth Object store content blobs Metadata DB key = id Cleanup job every few minutes Figure 1. Metadata and content take different routes. Writes store the blob first and the row second; reads come mostly from the CDN; a background job removes what has expired.
  1. Write path The paste service rejects content over 1 MB, applies a per-client rate limit, generates a random 12-character id, uploads the compressed blob, and only then inserts the metadata row. A crash between the two leaves an orphan blob, never a row pointing at nothing.
  2. Read path Public and unlisted pastes are served through the CDN with a cache lifetime no longer than the time to expiry. On a miss the service loads the row, checks visibility and expiry, and streams the blob. Private pastes skip the CDN.
  3. Cleanup path A scheduled job queries rows whose expires_at has passed, deletes their blobs, then their rows, in small batches. Reads also check expiry, so an expired paste is never served even if cleanup is behind.

Deep dives

Deep dive 1: Ids and access control

Unlike a URL shortener, a pastebin should never use counters: sequential ids let anyone enumerate every unlisted paste. Use random ids from a cryptographically secure generator. With 12 base62 characters the space is so large that collisions are vanishingly rare, but the insert should still be conditional ("insert if not exists") and retry on the rare clash.

Visibility is enforced on the server for every read that is not served by the CDN:

  • Public: listed, cacheable at the CDN.
  • Unlisted: not listed anywhere, readable by anyone with the link, cacheable. The long random id is the only protection, so add rate limits on 404s per client to slow down guessing.
  • Private: requires the owner's session ( Authentication vs authorization ), never cached at a shared CDN. For downloads, the service can hand out a short-lived pre-signed URLA URL to an object in storage that carries a signature and an expiry time, so anyone holding it can read that one object until it expires. to the object rather than streaming it.

Deep dive 2: Write ordering and size limits

The write touches two stores without a transaction between them. Order matters. Write the blob first, then the row. If the service crashes in between, you are left with an orphan blob that nothing references, which a weekly sweep can delete. The other order would leave a row pointing to missing content, and readers would see errors.

Size limits protect everything downstream. Enforce 1 MB at the gateway (reject on Content-Length before reading the body) and again in the service. Combine them with per-IP and per-account rate limits ( Rate limiting algorithms ), because a pastebin is an attractive free file host for spam and malware. Scanning public pastes for credentials and abuse asynchronously, after creation, is a common extension.

Deep dive 3: Expiry and cleanup

Expiry has two halves. Correctness is handled on read: if expires_at is in the past, return 404 or 410 even if the data still exists. Space is handled by a background job ( Design a distributed job scheduler covers schedulers) that walks an index on expires_at, deletes blobs and rows in batches of a few thousand, and spreads the work so it never spikes database load.

Two shortcuts help. Many object stores support lifecycle rules that delete objects automatically after a number of days, so pastes with fixed lifetimes can be stored under prefixes such as ttl-1d/ and expire without any job. And CDN entries should carry a cache lifetime no longer than the remaining time to expiry, so the CDN stops serving a paste on time without needing a purge.

Try it

Where does each piece of pastebin data or work belong?

Show answers
  • The 400 KB text of a log excerpt: Object store. Large, write-once blobs read by key belong in cheap, durable object storage.
  • Visibility and expiry time of a paste: Metadata DB. Small fields checked on every read and queried for cleanup belong in the database.
  • Deleting pastes whose expiry has passed: Background job. Cleanup runs off the request path in batches; reads check expiry for correctness.
  • Index of a user's own pastes, newest first: Metadata DB. Listing and sorting by owner and time is a database query over small rows.
  • Removing blobs that no row references: Background job. Orphans come from crashes between the two writes; a periodic sweep removes them.
  • The rendered, compressed paste content: Object store. Content goes to object storage, usually compressed, and is served through the CDN.

Try this:

  1. Sort all six items, then Check.
  2. Suppose you stored content in the database instead. Which estimate would make that painful first?
  3. Change the write order in your head to row first, blob second. What does a reader see after a crash between the two writes?

Bottlenecks and failure modes

FailureImpactMitigation
Crash between blob and row writeOrphan blobBlob first, row second; periodic orphan sweep
Viral public pasteOrigin overloadedCDN caching of public and unlisted pastes
Abuse as free file hostingStorage cost, legal risk1 MB limit, rate limits, async content scanning, takedown path
Id guessingUnlisted pastes exposed12-character random ids, rate limits on 404s
Cleanup job falls behindStorage grows, expired pastes still storedReads check expiry; lifecycle rules; batch size and lag alerts
Object store region outageReads failCross-region replication of the bucket for durability and failover

Trade-offs we made

OptionWins whenLoses when
Content in object storage (chosen) Cheap, durable, scales without sharding; keeps the DB small Two stores to keep consistent; one extra hop on reads
Content in the database One store, one transaction Terabytes of blobs bloat backups, replication and the cache
Random 12-character ids (chosen) Unguessable unlisted links, no coordination Longer URLs; a conditional insert for the rare clash
Expiry checked on read plus lazy cleanup (chosen) Always correct, cleanup can lag safely Storage holds expired data for a while

Extensions

  • Syntax highlighting rendered on the client, so the stored content stays plain text.
  • Deduplication by content hash for identical pastes, with reference counting so deletes stay correct.
  • Burn after reading: delete on first view, which needs an atomic "read and delete" on the metadata row and no CDN caching.
  • Secret scanning of public pastes to warn users who paste credentials.

In the wild

Where you will see this: public paste sites, the "gist" features of code hosting platforms, and the share-a-snippet buttons in chat tools and online code playgrounds. The same split, small metadata in a database and large immutable content in object storage behind a CDN, is the standard shape for image hosting, file attachments and build artifacts too.

Check yourself

  1. Why does paste content go to object storage rather than the metadata database?

    Show answer

    B. Content is large, write-once and read by key, and is many times the size of the metadata Separating blobs from small records keeps the database lean while object storage handles bulk, durable storage cheaply.

  2. Why are counter-based ids a bad idea for a pastebin?

    Show answer

    C. Sequential ids let anyone enumerate unlisted pastes An unlisted paste is protected only by its id being hard to guess, so ids must be random.

  3. The service writes the blob, then crashes before inserting the row. What is the result?

    Show answer

    B. An orphan blob that a sweep can delete later Blob-first ordering means a crash leaves only unreferenced data, never a row pointing at nothing.

  4. Cleanup is an hour behind. Can users still read expired pastes?

    Show answer

    B. No, because every read checks expires_at Correctness comes from the check on read; the job only reclaims space.

Recap

  • Small metadata goes in a database; large, immutable content goes in object storage behind a CDN.
  • Use long random ids: for unlisted pastes, the id is the only protection.
  • Write the blob first and the row second, so a crash leaves only orphans to sweep.
  • Enforce size and rate limits early; a pastebin attracts abuse.
  • Check expiry on every read; let a background job or lifecycle rules reclaim space.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up