System Design HLD-10.02
Sagas: choreography and orchestration
Sagas split a business transaction into local transactions with compensations, coordinated by events or by an orchestrator, plus semantic locks.
The big idea
A travel site books a trip: reserve a flight, book a hotel, rent a car, charge the card. Four services, four databases, and possibly external partners. Two-phase commit is out ( Two-phase commit ): the airline will not hold a prepared transaction for you. Yet if the car rental fails, the customer must not be left with a flight and a hotel they did not want.
A sagaA business transaction split into a sequence of local transactions, each with a compensating action that semantically undoes it if a later step fails. gives up on making the whole thing atomic. Instead, each step is an ordinary local transaction that commits immediately, and each has a compensating actionAn action that semantically undoes the effect of an earlier committed step, such as a refund for a charge or a cancellation for a booking. that undoes it. If step three fails, run the compensations for steps two and one, in reverse. The end result is all or nothing, eventually, even though every step committed along the way.
Think of planning a dinner party. You book the restaurant, order flowers and hire a musician. If the musician cancels and the party is off, you do not rewind time; you call the restaurant to cancel and return the flowers. Each undo is its own new action, and some might cost a fee.
How it works
Step 1: Local transactions and compensations
Designing compensations is where the real work is:
- Compensations are semantic, not technical rollbacks. You cannot un-send an email; you send a correction. You may not get the full refund; a cancellation may carry a fee. The business decides what "undo" means.
- They must be retryable and idempotent. A compensation that fails must be retried until it succeeds, so running it twice must be safe ( Idempotency and safe retries ).
- Order steps by how hard they are to undo. Put steps that cannot be compensated, such as shipping a parcel or sending money to an external bank, last. The step after which the saga can no longer be undone is the pivot; steps after it should be ones that can be retried until they succeed.
Step 2: Choreography
In choreographyCoordinating a saga by having each service react to other services' events, with no central controller., there is no coordinator. Each service listens for events and reacts: the flight service publishes FlightReserved, the hotel service hears it and books a room, publishes HotelBooked, and so on. On failure, the car service publishes CarRentalFailed, and the hotel and flight services each react by compensating.
It is loosely coupled and needs no extra component. But the flow exists only implicitly, spread across services' event handlers. With more than a few steps it becomes hard to see the whole process, easy to create cycles, and hard to answer "where is booking 8812 stuck?".
Step 3: Orchestration
In orchestrationCoordinating a saga with a central component that tells each service which step to run next and which compensations to run on failure., a saga orchestrator owns the flow. It sends a command to the flight service, waits for the reply, then commands the hotel, and so on. On failure, it issues the compensating commands in reverse. The orchestrator persists the saga's state after every step, so if it crashes it resumes where it left off.
The whole process lives in one place: easy to read, test, monitor and change. The cost is a central component that knows about every participant. For most non-trivial business flows, teams prefer orchestration, often built on a workflow engine ( Workflow engines ).
Try it
Try this:
- With the car failing and orchestration selected, step through. Note that two compensations run, in reverse order, and payment never happens.
- Make the payment fail instead. Now three compensations run. Is "release seat" really free for the business? This is why expensive-to-undo steps go last.
- Switch to choreography and replay. The same actions happen, but each line is a service reacting to an event rather than obeying a command. Which log would you rather debug at 3 a.m.?
Going deeper
The missing I in ACID
Sagas are atomic in the end, but not isolated. While the saga runs, other requests can see its intermediate state: a hotel room that is booked but will be cancelled a second later, or a balance already debited for a payment that will be refunded. Common countermeasures:
- Semantic locks: mark records with a pending state, such as
booking.status = PENDING, so other operations know they are in flux and can wait, reject or handle them specially. The final step or the compensation clears the flag. - Commutative updates: design operations so their order does not matter, such as increments and decrements of a balance rather than overwrites.
- Re-reading values: verify that data has not changed before a later step acts on it.
Failures inside failures
What if a compensation fails? It must be retried, with backoff, until it succeeds, and if it truly cannot (the airline's API rejects the cancellation), the saga ends in a state that needs a human. Every saga system needs an escape hatch: alerts, a dashboard of stuck sagas, and tools to finish or repair them by hand, backed by reconciliation ( Reconciliation and correctness jobs ).
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Choreography (events) | Short flows of 2 to 4 steps between loosely coupled services | Visibility of the end-to-end flow; harder debugging as steps grow |
| Orchestration (coordinator) | Longer business processes that need monitoring, timeouts and clear error handling | A central component coupled to every participant |
| Two-phase commit | A few databases that support it, needing strict isolation | Availability; most services and external APIs cannot take part |
Common mistakes
- Writing compensations that are not idempotent, so a retried refund pays out twice.
- Assuming a compensation is a perfect rollback, ignoring fees, emails already sent and other side effects.
- Putting the hardest-to-undo step first instead of last.
- Ignoring the lack of isolation, letting other requests act on data a saga is about to compensate.
- Having no way to find and repair sagas that are stuck halfway.
In the wild
Airbnb
Avoiding double payments in a distributed payments system (opens in a new tab)Airbnb's 2019 post explains that its move to a service-oriented architecture made it harder to keep payment data consistent across distributed calls, network failures and timeouts, risking double charges and double payouts. The team built Orpheus, a general-purpose idempotency library that splits each request into pre-RPC, RPC and post-RPC phases, recording the request and then the response in the database, and classifying failures as retryable (network and 5XX errors) or non-retryable (validation errors). The authors report five nines of consistency for payments while annual payment volume doubled. It is the kind of idempotent, retry-safe step that every saga participant and compensation needs.
Check yourself
-
A saga's third step fails. What happens to steps one and two?
Show answer
B. Their compensating actions run, in reverse order Each completed step committed locally, so undoing it requires its own compensating transaction.
-
Why should a step like 'ship parcel' come last in a saga?
Show answer
B. It is hard or impossible to compensate, so it should only run once everything that could fail has succeeded Steps after the pivot should not need compensation; irreversible actions belong there.
-
While a saga runs, another request reads a hotel booking that will be cancelled in a second. Which countermeasure helps?
Show answer
B. A semantic lock: mark the booking PENDING until the saga completes or compensates Sagas lack isolation; a pending marker tells other operations the data is not final.
-
A 9-step order flow is implemented with choreography and nobody can tell where orders get stuck. What would help most?
Show answer
B. Moving to orchestration, where one component owns and records the flow An orchestrator makes the process explicit, persistent and observable.
Recap
- A saga is a sequence of local transactions, each with a compensating action.
- On failure, compensations for completed steps run in reverse; they must be idempotent and retryable.
- Choreography coordinates through events; orchestration uses a central coordinator that is easier to follow.
- Sagas are not isolated: use semantic locks and commutative updates, and put irreversible steps last.
- Plan for stuck sagas with alerts, dashboards and manual repair.