System Design HLD-14.03
Service discovery
How services find each other when instances come and go: registries, client-side vs server-side discovery, and DNS-based discovery in Kubernetes.
The big idea
The orders service needs to call payments. Ten years ago you might have put payments.internal:8080 in a config file, pointing at two long-lived servers. Today payments runs as 3 to 40 containers whose addresses change with every deploy, every autoscaling event and every crashed instance. A hardcoded address is wrong within minutes.
Service discoveryThe mechanism by which a service finds the current network addresses of healthy instances of another service. solves this with a live directory. Instances register themselves when they start and are removed when they stop or fail health checks. Callers ask the directory, or a load balancer that uses it, for a healthy instance at the moment they need one.
Think of calling a company's main number instead of an employee's desk phone. People move desks, join and leave, but the switchboard always knows who is currently available and connects you.
How it works
Step 1: The registry
A service registryA database of the currently running instances of each service, kept up to date by registration and health checks or heartbeats. maps service names to healthy instance addresses. Instances register on startup (or a platform registers them), renew with heartbeats, and are removed when heartbeats stop or health checks fail. That is the same lease idea as in coordination services ( Coordination services (ZooKeeper, etcd) ); some registries are built on them.
import itertools
class Registry:
"""Instances register with a TTL and must heartbeat; expired ones disappear."""
def __init__(self, ttl=10):
self.ttl, self.entries = ttl, {} # (service, addr) -> last heartbeat
def register(self, service, addr, now):
self.entries[(service, addr)] = now
heartbeat = register
def lookup(self, service, now):
return sorted(a for (s, a), t in self.entries.items() if s == service and now - t < self.ttl)
reg = Registry()
for addr in ("10.0.1.7:8080", "10.0.2.4:8080", "10.0.3.9:8080"):
reg.register("payments", addr, now=0)
# Client-side discovery: look up the instances, then load balance across them yourself.
picks = itertools.cycle(reg.lookup("payments", now=1))
print("t=1 calls go to:", [next(picks) for _ in range(4)])
# 10.0.2.4 crashes and stops heartbeating; the others keep sending heartbeats.
for addr in ("10.0.1.7:8080", "10.0.3.9:8080"):
reg.heartbeat("payments", addr, now=8)
print("t=12 instances :", reg.lookup("payments", now=12))
reg.register("payments", "10.0.4.2:8080", now=13) # autoscaler added a new one
print("t=13 instances :", reg.lookup("payments", now=13))Output
t=1 calls go to: ['10.0.1.7:8080', '10.0.2.4:8080', '10.0.3.9:8080', '10.0.1.7:8080']
t=12 instances : ['10.0.1.7:8080', '10.0.3.9:8080']
t=13 instances : ['10.0.1.7:8080', '10.0.3.9:8080', '10.0.4.2:8080'] The crashed instance stops heartbeating and drops out of lookups after its TTL, and the new instance appears as soon as it registers. Callers never needed a config change.
Step 2: Client-side versus server-side discovery
- Client-side discovery: the caller queries the registry, gets the list of instances and picks one itself, often with a smart load-balancing algorithm and local health tracking ( Load balancing algorithms ). No extra hop, but every client needs a discovery library, in every language you use.
- Server-side discovery: the caller sends requests to a stable name or address belonging to a load balancer or proxy, which looks up healthy instances and forwards the request. Clients stay simple and language-agnostic, at the cost of an extra hop and another component to run. A service mesh is a form of this with the proxy running next to each client ( Service mesh ).
Step 3: DNS-based discovery in Kubernetes
Kubernetes builds discovery in. A Service object gives a set of pods a stable virtual IP and a DNS name such as payments.shop.svc.cluster.local. The platform tracks which pods are ready (via readiness probes) and routes traffic for that IP to them, so callers simply use http://payments. A "headless" Service instead returns the individual pod IPs from DNS, for clients that want to do their own balancing.
DNS is convenient and universal, but caching can make it slow to reflect changes, which is one reason Kubernetes routes through a stable virtual IP rather than handing out changing pod addresses in DNS answers by default.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Static config / fixed hostnames | A few long-lived servers that rarely change | Dynamic environments, autoscaling and frequent deploys |
| Client-side discovery | Low latency and smart client load balancing | A discovery library in every client language |
| Server-side discovery (load balancer) | Simple, language-agnostic clients | An extra hop and component to operate |
| Platform DNS (Kubernetes Services) | Zero code: call a name | Less control over balancing; DNS caching if used directly |
Common mistakes
- Hardcoding instance IPs in configuration.
- Registering instances before they are ready to serve, or never deregistering them on shutdown.
- Health checks that do not reflect whether the instance can actually serve.
- Clients that cache DNS answers forever and keep calling dead instances.
- Making the registry itself a single point of failure.
In the wild
Kubernetes Services and cluster DNS provide discovery for most containerized systems today. HashiCorp Consul offers a registry with health checks and DNS and HTTP interfaces, and Netflix's open source Eureka popularized client-side discovery for JVM services. Cloud load balancers combined with auto scaling groups provide server-side discovery without a separate registry.
Check yourself
-
Why can't callers simply hardcode instance addresses in a containerized system?
Show answer
B. Instances come and go with deploys, scaling and failures, so addresses change constantly Discovery provides a live view of which instances exist right now.
-
In client-side discovery, who chooses which instance receives a request?
Show answer
B. The calling client, after looking up the instance list The client gets the list and load balances itself.
-
An instance crashes without deregistering. How does it leave the registry?
Show answer
B. Its heartbeats or health checks stop, so its entry expires TTLs and health checks make the registry self-correcting.
-
In Kubernetes, how does the orders pod usually call payments?
Show answer
B. Through the payments Service's stable DNS name, routed to ready pods Services give a stable name and virtual IP in front of changing pods.
Recap
- Service discovery keeps a live directory of healthy instances so callers never hardcode addresses.
- Instances register and heartbeat; failed ones expire from the registry.
- Client-side discovery: the caller looks up and balances itself; server-side: a load balancer does it.
- Kubernetes provides discovery through Services with stable DNS names and virtual IPs.
- Health checks, deregistration and registry availability decide how well discovery works.