Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Architecture Styles

System Design HLD-14.04

Service mesh

Moving networking out of application code into sidecar proxies: mTLS, retries, timeouts, traffic shifting and telemetry, plus control plane vs data plane.

Advanced 10 min Last reviewed

The big idea

With 200 microservices written in Go, Java, Python and Node, every service needs the same network behavior: encrypt traffic between services, verify who is calling, set timeouts and retries with backoff, open circuit breakers, shift 5 percent of traffic to a canary, and emit latency and error metrics for every call. Implementing all of that in a library for every language, and keeping hundreds of services on current versions, is a large, never-ending project.

A service meshAn infrastructure layer that handles service-to-service communication through proxies deployed alongside every service, configured centrally, providing security, reliability and telemetry without application changes. moves that logic out of the application. A small proxy runs next to every service instance, as a sidecarA helper container deployed in the same pod as an application container, sharing its network, such as a proxy that handles all of the app's traffic., and all of the service's inbound and outbound traffic flows through it. The proxies implement mTLS, retries, timeouts, traffic splitting and telemetry uniformly, whatever language the service is written in. A central control plane tells every proxy what to do.

Think of an office building where every employee's mail goes through a mailroom on their floor. The mailroom checks identities, signs and seals envelopes, retries failed deliveries and keeps logs, so employees just write letters. Building management sets the mailroom rules for every floor at once.

How it works

Step 1: Data plane and control plane

Pods with sidecar proxies forming a mesh, configured by a control plane Three pods: orders, payments and catalog. Each pod has the application container and a sidecar proxy. Calls from orders to payments and catalog travel proxy to proxy over mutual TLS, glowing. A control plane above pushes configuration, certificates and routing rules to every proxy and receives telemetry from them. orders pod payments pod catalog pod mTLS Control plane config, certificates, routing rules app proxy proxy app proxy app Figure 1. Each pod pairs the application with a sidecar proxy. Service-to-service traffic flows proxy to proxy over mutual TLS (the data plane). The control plane pushes routing rules and certificates to every proxy and collects telemetry, without touching application code.

The data plane is the set of proxies that carry the actual traffic. Envoy is the most common data plane proxy ( Reverse proxies (Nginx, Envoy, HAProxy) ). The control plane holds the desired configuration, such as "retry payments calls twice with a 200 ms timeout" or "send 5 percent of catalog traffic to v2", and pushes it to the proxies, along with the certificates each proxy uses to prove its identity. Operators change behavior for the whole fleet by changing configuration, not code.

Step 2: What the mesh provides

  • Security: mutual TLSMutual TLS: both sides of a connection present certificates, so each verifies the other's identity and the traffic is encrypted. between every pair of services, with certificates issued and rotated automatically, plus policies such as "only orders may call payments" ( TLS and HTTPS ).
  • Reliability: timeouts, retries with budgets, circuit breaking and outlier detection (ejecting hosts that keep failing), applied consistently ( Timeouts, retries, backoff and jitter , Circuit breakers and bulkheads ).
  • Traffic management: weighted routing for canaries and blue-green, routing by header (send employees to v2), mirroring traffic to a new version ( Deployment strategies ).
  • Telemetry: request rate, errors and latency for every service-to-service call, and trace spans, without instrumenting code ( Logs, metrics and traces ). Applications must still forward trace headers for traces to connect.

Step 3: The costs

  • Latency: every call passes through two extra proxies, adding typically around a millisecond or less per hop, which compounds in deep call chains.
  • Resources: a proxy in every pod uses CPU and memory; across thousands of pods it adds up.
  • Operational complexity: the mesh is now critical infrastructure. A bad mesh configuration or control plane bug can break every service at once, and debugging has a new layer.
  • Not a replacement for good design: a mesh can retry a failed call, but it cannot make a non-idempotent operation safe to retry.

To reduce sidecar overhead, newer designs run shared per-node proxies instead of one per pod (for example Istio's ambient mode), or move some functions into the kernel.

Going deeper

When a mesh is worth it

A mesh pays off with many services, several languages, strict security requirements (encrypt everything, verify every caller) and the platform team to run it. With a handful of services in one language, a good shared library or a simple gateway may be far cheaper. Many organizations adopt it incrementally, starting with mTLS and telemetry before moving retries and routing into the mesh.

Trade-offs

OptionWins whenLoses when
Libraries in each service Few services, one language, minimal infrastructure Re-implementing per language and upgrading every service for changes
Sidecar service mesh Many services and languages needing uniform mTLS, policy, traffic control and telemetry Per-hop latency, resource overhead and operational complexity
Node-level or sidecarless mesh Mesh features with lower per-pod overhead Newer, with different isolation trade-offs
API gateway only Controlling traffic entering the system from outside No control over service-to-service traffic

Common mistakes

  • Adopting a mesh for five services because it is popular.
  • Enabling retries at the mesh layer and in application code, multiplying load during failures.
  • Assuming the mesh makes tracing automatic when applications do not forward trace headers.
  • Pushing mesh configuration changes to every service at once without staged rollout.
  • Not monitoring the control plane as the critical dependency it is.

In the wild

Istio and Linkerd are the most widely used open source service meshes: Istio uses Envoy proxies with a central control plane, and Linkerd uses its own lightweight proxy. Cloud providers offer managed meshes, and many large companies built internal meshes on Envoy, which was created at Lyft. Kubernetes is the most common environment, where sidecars are injected into pods automatically.

Check yourself

  1. What is the data plane in a service mesh?

    Show answer

    B. The proxies that carry service-to-service traffic Proxies alongside each service form the data plane; the control plane configures them.

  2. Which capability does a mesh give a Python service without code changes?

    Show answer

    B. Mutual TLS, retries, timeouts and per-call metrics The sidecar handles network concerns; business logic and idempotency stay in the app.

  3. Retries are configured in the mesh and also in the application code. What is the risk?

    Show answer

    B. Retries multiply during failures, amplifying load on the struggling service Retry at one layer only; multiple layers multiply attempts.

  4. Why might a team with 6 services in one language skip a mesh?

    Show answer

    B. The operational and resource cost may outweigh benefits a shared library or gateway already provides A mesh pays off at larger scale and with many languages.

Recap

  • A service mesh moves networking concerns into proxies next to every service.
  • The data plane (proxies) carries traffic; the control plane configures proxies and issues certificates.
  • It provides mTLS and access policy, timeouts, retries, circuit breaking, traffic shifting and telemetry, uniformly across languages.
  • Costs: extra latency per hop, resources per pod and a critical new layer to operate.
  • Worth it with many services and languages and a platform team; overkill for small systems.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy