System Design HLD-14.04
Service mesh
Moving networking out of application code into sidecar proxies: mTLS, retries, timeouts, traffic shifting and telemetry, plus control plane vs data plane.
The big idea
With 200 microservices written in Go, Java, Python and Node, every service needs the same network behavior: encrypt traffic between services, verify who is calling, set timeouts and retries with backoff, open circuit breakers, shift 5 percent of traffic to a canary, and emit latency and error metrics for every call. Implementing all of that in a library for every language, and keeping hundreds of services on current versions, is a large, never-ending project.
A service meshAn infrastructure layer that handles service-to-service communication through proxies deployed alongside every service, configured centrally, providing security, reliability and telemetry without application changes. moves that logic out of the application. A small proxy runs next to every service instance, as a sidecarA helper container deployed in the same pod as an application container, sharing its network, such as a proxy that handles all of the app's traffic., and all of the service's inbound and outbound traffic flows through it. The proxies implement mTLS, retries, timeouts, traffic splitting and telemetry uniformly, whatever language the service is written in. A central control plane tells every proxy what to do.
Think of an office building where every employee's mail goes through a mailroom on their floor. The mailroom checks identities, signs and seals envelopes, retries failed deliveries and keeps logs, so employees just write letters. Building management sets the mailroom rules for every floor at once.
How it works
Step 1: Data plane and control plane
The data plane is the set of proxies that carry the actual traffic. Envoy is the most common data plane proxy ( Reverse proxies (Nginx, Envoy, HAProxy) ). The control plane holds the desired configuration, such as "retry payments calls twice with a 200 ms timeout" or "send 5 percent of catalog traffic to v2", and pushes it to the proxies, along with the certificates each proxy uses to prove its identity. Operators change behavior for the whole fleet by changing configuration, not code.
Step 2: What the mesh provides
- Security: mutual TLSMutual TLS: both sides of a connection present certificates, so each verifies the other's identity and the traffic is encrypted. between every pair of services, with certificates issued and rotated automatically, plus policies such as "only orders may call payments" ( TLS and HTTPS ).
- Reliability: timeouts, retries with budgets, circuit breaking and outlier detection (ejecting hosts that keep failing), applied consistently ( Timeouts, retries, backoff and jitter , Circuit breakers and bulkheads ).
- Traffic management: weighted routing for canaries and blue-green, routing by header (send employees to v2), mirroring traffic to a new version ( Deployment strategies ).
- Telemetry: request rate, errors and latency for every service-to-service call, and trace spans, without instrumenting code ( Logs, metrics and traces ). Applications must still forward trace headers for traces to connect.
Step 3: The costs
- Latency: every call passes through two extra proxies, adding typically around a millisecond or less per hop, which compounds in deep call chains.
- Resources: a proxy in every pod uses CPU and memory; across thousands of pods it adds up.
- Operational complexity: the mesh is now critical infrastructure. A bad mesh configuration or control plane bug can break every service at once, and debugging has a new layer.
- Not a replacement for good design: a mesh can retry a failed call, but it cannot make a non-idempotent operation safe to retry.
To reduce sidecar overhead, newer designs run shared per-node proxies instead of one per pod (for example Istio's ambient mode), or move some functions into the kernel.
Going deeper
When a mesh is worth it
A mesh pays off with many services, several languages, strict security requirements (encrypt everything, verify every caller) and the platform team to run it. With a handful of services in one language, a good shared library or a simple gateway may be far cheaper. Many organizations adopt it incrementally, starting with mTLS and telemetry before moving retries and routing into the mesh.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Libraries in each service | Few services, one language, minimal infrastructure | Re-implementing per language and upgrading every service for changes |
| Sidecar service mesh | Many services and languages needing uniform mTLS, policy, traffic control and telemetry | Per-hop latency, resource overhead and operational complexity |
| Node-level or sidecarless mesh | Mesh features with lower per-pod overhead | Newer, with different isolation trade-offs |
| API gateway only | Controlling traffic entering the system from outside | No control over service-to-service traffic |
Common mistakes
- Adopting a mesh for five services because it is popular.
- Enabling retries at the mesh layer and in application code, multiplying load during failures.
- Assuming the mesh makes tracing automatic when applications do not forward trace headers.
- Pushing mesh configuration changes to every service at once without staged rollout.
- Not monitoring the control plane as the critical dependency it is.
In the wild
Istio and Linkerd are the most widely used open source service meshes: Istio uses Envoy proxies with a central control plane, and Linkerd uses its own lightweight proxy. Cloud providers offer managed meshes, and many large companies built internal meshes on Envoy, which was created at Lyft. Kubernetes is the most common environment, where sidecars are injected into pods automatically.
Check yourself
-
What is the data plane in a service mesh?
Show answer
B. The proxies that carry service-to-service traffic Proxies alongside each service form the data plane; the control plane configures them.
-
Which capability does a mesh give a Python service without code changes?
Show answer
B. Mutual TLS, retries, timeouts and per-call metrics The sidecar handles network concerns; business logic and idempotency stay in the app.
-
Retries are configured in the mesh and also in the application code. What is the risk?
Show answer
B. Retries multiply during failures, amplifying load on the struggling service Retry at one layer only; multiple layers multiply attempts.
-
Why might a team with 6 services in one language skip a mesh?
Show answer
B. The operational and resource cost may outweigh benefits a shared library or gateway already provides A mesh pays off at larger scale and with many languages.
Recap
- A service mesh moves networking concerns into proxies next to every service.
- The data plane (proxies) carries traffic; the control plane configures proxies and issues certificates.
- It provides mTLS and access policy, timeouts, retries, circuit breaking, traffic shifting and telemetry, uniformly across languages.
- Costs: extra latency per hop, resources per pod and a critical new layer to operate.
- Worth it with many services and languages and a platform team; overkill for small systems.