System Design HLD-06.08
Object storage, block storage and file systems
Buckets and objects, durability through replication and erasure coding, presigned URLs, and when to use object, block or file storage.
The big idea
Photos, videos, PDFs, backups, logs and machine learning datasets are large, rarely changed and read as whole files. Putting them in a database bloats it, slows backups and wastes expensive storage; keeping them on an app server's disk ties them to one machine. Object storageStorage for files as immutable objects in buckets, addressed by key and accessed over HTTP, with very high durability and practically unlimited capacity., such as Amazon S3, Google Cloud Storage or Azure Blob Storage, is built exactly for them: a practically unlimited, very durable, cheap place for files, accessed over HTTP by a key such as user-uploads/avatars/42.jpg.
The standard pattern is simple and appears in almost every system design: metadata in the database, bytes in object storage. The database row says who owns the photo, when it was uploaded and where it lives; the object store holds the bytes; a CDN serves them to users. Clients often upload straight to the object store using a short-lived signed URL, so large files never pass through your application servers at all.
Think of a museum. The catalog (database) records each painting's title, artist and room. The paintings themselves are in the galleries and storerooms (object storage). Nobody staples paintings into the catalog.
How it works
Step 1: Three kinds of storage
- Object storage: flat namespace of buckets and keys; whole objects are written and read over HTTP; objects are immutable (you replace rather than edit them). Huge scale, high durability, low cost per gigabyte, higher latency per request. Images, video, backups, data lakes, static sites.
- Block storage: a raw virtual disk attached to one machine (EBS, persistent disks), formatted with a file system. Low latency, random reads and writes. The disk under databases and virtual machines.
- File storage: a shared network file system (NFS, EFS) that many machines mount as folders. Familiar file semantics; used for shared content and legacy applications.
Step 2: Durability
Object stores advertise extraordinary durability. They achieve it by storing each object redundantly across devices and data centers, with either full replicas (say three copies) or erasure codingSplitting data into fragments plus extra parity fragments, so the original can be rebuilt from any sufficient subset; durable like replication with far less extra storage.: the object is split into, for example, 10 data fragments plus 4 parity fragments spread over 14 disks, and any 10 of them can rebuild it. That tolerates 4 lost disks at 1.4 times the storage instead of 3 times. Background processes constantly check and repair fragments.
Step 3: Direct uploads with presigned URLs
A presigned URLA URL that grants temporary permission for one specific operation on one object, signed by your backend so the storage service can verify it. encodes one permission (this method, this key, until this time) and a signature made with a secret only your backend and the storage service know:
import hashlib, hmac
from urllib.parse import urlencode, urlparse, parse_qs
SECRET = b"storage-secret-known-only-to-your-backend-and-the-store"
def presign(method, bucket, key, expires_at):
"""Your backend signs exactly one operation on one object until a deadline."""
msg = f"{method}\n/{bucket}/{key}\n{expires_at}".encode()
sig = hmac.new(SECRET, msg, hashlib.sha256).hexdigest()[:32]
return f"https://store.example.com/{bucket}/{key}?" + urlencode({"expires": expires_at, "sig": sig})
def store_accepts(method, url, now):
"""The object store checks the signature itself; your app server is not involved."""
u = urlparse(url)
q = {k: v[0] for k, v in parse_qs(u.query).items()}
bucket, key = u.path.strip("/").split("/", 1)
if now > int(q["expires"]):
return "403 expired"
expected = presign(method, bucket, key, int(q["expires"]))
return "200 OK" if hmac.compare_digest(expected, url) else "403 bad signature"
url = presign("PUT", "user-uploads", "avatars/42.jpg", expires_at=1_700_000_300)
print(url)
print("upload within 5 minutes: ", store_accepts("PUT", url, now=1_700_000_100))
print("upload after it expired: ", store_accepts("PUT", url, now=1_700_000_301))
print("used to DELETE instead: ", store_accepts("DELETE", url, now=1_700_000_100))
print("key changed to admin.jpg: ", store_accepts("PUT", url.replace("42.jpg", "admin.jpg"), now=1_700_000_100))Output
https://store.example.com/user-uploads/avatars/42.jpg?expires=1700000300&sig=52916b83c50da5c3d460bd1a8855faa2
upload within 5 minutes: 200 OK
upload after it expired: 403 expired
used to DELETE instead: 403 bad signature
key changed to admin.jpg: 403 bad signature The store verifies the signature itself. A URL used after it expires, for a different method or for a different key is rejected. Your application servers stay out of the data path for the heavy transfer, and nothing gives the client general access to the bucket. Downloads of private files work the same way with presigned GET URLs.
Going deeper
Lifecycle and cost tiers
Object stores offer storage classes: frequently accessed, infrequently accessed, and archive tiers that are much cheaper per gigabyte but slower or costlier to read. Lifecycle rules move objects between tiers by age (logs to archive after 30 days, delete after a year) automatically.
Not a file system
Object stores have no real folders (the slashes are part of the key), no in-place edits, no locking and no rename. Listing millions of keys is slow. Code that treats a bucket like a local disk, writing small appends or renaming files, performs badly.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Object storage | Large immutable files: media, backups, logs, datasets, static assets | Small random edits, low-latency transactional access, file-system semantics |
| Block storage | Databases and VM disks needing low-latency random I/O | Sharing across many machines and very large, cheap capacity |
| File storage | Shared folders mounted by many servers, legacy apps | Cost and scale compared with object storage |
Common mistakes
- Storing large files as blobs in the database, bloating backups and replication.
- Uploading files through the app server, tying up its bandwidth and threads.
- Making buckets public by accident; use private buckets with presigned URLs or a CDN with signed access.
- Forgetting lifecycle rules, so storage costs grow forever.
Check yourself
-
Where should a user's profile photo bytes live?
Show answer
B. In object storage, with metadata in the database Bytes go to object storage; the database keeps the owner, key and size.
-
What does a presigned URL grant?
Show answer
B. Temporary permission for one operation on one object Method, key and expiry are all covered by the signature.
-
Why upload directly to the object store instead of through the app server?
Show answer
B. Large transfers then never pass through and tie up your servers The app only signs permission and records metadata; the store handles the bytes.
-
What does erasure coding offer compared with three full copies?
Show answer
B. Similar durability with much less extra storage For example, 10 data plus 4 parity fragments survive 4 losses at 1.4x storage.
Recap
- Object storage holds large immutable files in buckets, over HTTP, cheaply and durably.
- Block storage is a virtual disk for databases and VMs; file storage is a shared network file system.
- Durability comes from replication or erasure coding across devices and sites.
- Keep metadata in the database and bytes in the object store.
- Clients upload and download directly with short-lived presigned URLs.