System Design HLD-10.03
Workflow engines
Durable execution with engines like Temporal and Cadence: workflows vs activities, event history and replay, durable timers, retries and timeouts.
The big idea
Write the trip-booking saga by hand and you end up building a small database-backed state machine: a table of saga states, a scheduler to resume them, retry counters, timeout jobs, a way to recover after a crash. It works, but most of the code is plumbing, and the business logic, "reserve, book, rent, charge, and undo on failure", is buried inside it.
A workflow engineA system that runs long-lived business processes durably: it records each step's outcome so a process survives crashes and restarts and resumes exactly where it left off. lets you write that logic as ordinary code, a function with loops, ifs and try/except, and makes it durable. Every step's result is recorded. If the machine running it crashes halfway, another machine picks it up and continues from exactly the same point, as if the crash never happened. That idea is called durable executionExecution of ordinary-looking code whose progress is persisted, so it survives process crashes, restarts and long waits..
Think of following a long recipe with a notebook beside you, ticking off each completed step. If you are called away and someone else takes over, they read your ticks, skip what is done, and carry on from the first unticked line. Nobody chops the onions twice.
How it works
Step 1: Workflows and activities
Workflow engines split code into two kinds:
- An activityIn a workflow engine, a single unit of real work with side effects, such as calling an API or writing to a database; it can fail, time out and be retried. does real work with side effects: charge a card, call the airline API, send an email. Activities can fail, so the engine retries them according to a policy with backoff and timeouts.
- A workflow is the orchestration logic that calls activities in order and decides what to do with their results. It must be deterministic: given the same inputs and activity results, it makes the same decisions. No direct clock reads, random numbers or network calls inside workflow code; those go in activities or through the engine's APIs.
Step 2: History and replay
The engine keeps an append-only event historyThe durable log a workflow engine keeps for each workflow run, recording every activity scheduled and completed, timer and signal, used to rebuild the workflow's state by replay. for each workflow run: started, activity scheduled, activity completed with its result, timer fired, signal received. Workers are stateless. To run or resume a workflow, a worker loads its history and replays the workflow code from the top. Whenever the code asks for an activity that the history says already completed, the engine hands back the recorded result instantly instead of running it again. When the code reaches a step with no history, that is new work: run it and append the result.
calls = [] # real side effects, to count them
def activity(name): # an activity: real work with side effects
def run(arg):
calls.append(name)
return f"{name}({arg}) ok"
run.__name__ = name
return run
charge, reserve, email = activity("charge"), activity("reserve"), activity("email")
def order_workflow(order): # workflow code: plain, deterministic
r1 = yield (charge, order)
r2 = yield (reserve, order)
r3 = yield (email, order)
return [r1, r2, r3]
def run_worker(history, order, crash_after=None):
"""Runs the workflow; completed steps are answered from the history instead of re-run."""
wf, result, step = order_workflow(order), None, 0
try:
while True:
fn, arg = wf.send(result)
if step < len(history): # replay: we already did this step
result = history[step]
else: # new step: run it, then record it
result = fn(arg)
history.append(result)
if crash_after == step:
return "worker crashed"
step += 1
except StopIteration as done:
return done.value
history = [] # stored durably by the engine
print("first worker :", run_worker(history, "A17", crash_after=1))
print("history :", history)
print("second worker:", run_worker(history, "A17"))
print("side effects :", calls, "(each ran once)")Output
first worker : worker crashed
history : ['charge(A17) ok', 'reserve(A17) ok']
second worker: ['charge(A17) ok', 'reserve(A17) ok', 'email(A17) ok']
side effects : ['charge', 'reserve', 'email'] (each ran once) The first worker charges and reserves, then crashes. The second worker runs the same workflow function from the start, but the first two steps come straight from the history, so the card is not charged again. Only the email runs for real. This is the same idea as event sourcing ( Event-driven architecture and event sourcing ), applied to the progress of a program, and it is why workflow code must be deterministic: replay must take the same path it took the first time.
One subtlety: an activity can finish its side effect and crash before its completion is recorded. The engine then retries it, so activities are effectively at-least-once and should be idempotent ( Idempotency and safe retries ).
Step 3: Timers, retries and timeouts
Because state lives in the history, a workflow can wait for a long time at almost no cost. sleep(30 days) before sending a renewal reminder is a durable timer stored by the engine; no process sits blocked for a month. The same goes for waiting on an external signal, such as a human approving a refund.
Retries and timeouts are declared, not hand-coded: "retry the charge up to 5 times with exponential backoff, fail if it does not finish in 2 minutes". When retries are exhausted, the activity raises an error in the workflow code, and an ordinary try/except runs the saga's compensations ( Sagas: choreography and orchestration ).
Step 4: Temporal, Cadence and the alternatives
Cadence was built at Uber and open sourced in 2017. Temporal is a fork started by Cadence's original creators, and both follow the model above: workflows and activities written in a general-purpose language, an event history per run, and replay. Other systems express workflows differently: AWS Step Functions and Azure Durable Functions are managed services, the first defined as a JSON state machine, the second as code with replay. Apache Airflow schedules batch data pipelines as task graphs and is not built for per-request business transactions.
Going deeper
Versioning running workflows
A workflow may run for weeks, and you will deploy new code in the meantime. If the new code takes a different path when replaying an old history, replay breaks. Engines provide versioning APIs so a workflow can branch on "which version of this code was I started with", and old versions are removed once no runs use them.
What a workflow engine costs
You run (or pay for) another critical piece of infrastructure, and each step adds a round trip to the engine and a write to its history, so it suits business-level steps, not tight loops. Very long histories slow down replay, which engines address by "continuing as new": ending one run and starting a fresh one with the current state.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Hand-written state machine | One or two simple flows; no new infrastructure | Lots of fragile plumbing for retries, timers and recovery |
| Workflow engine (Temporal, Cadence) | Long-running, multi-step business processes, sagas, human-in-the-loop waits | Another critical system; determinism rules and versioning discipline |
| Managed state machine (Step Functions) | Teams on one cloud wanting no servers to run | Vendor lock-in; logic expressed as configuration rather than code |
| Choreographed events | Short, loosely coupled reactions between services | No single place to see or manage the end-to-end process |
Common mistakes
- Calling the clock, random numbers or the network directly in workflow code, breaking replay.
- Writing non-idempotent activities, so a retry after a crash charges twice.
- Changing workflow code without versioning while old runs are still in flight.
- Putting high-frequency, low-value work in a workflow and drowning the engine in history events.
- Using a batch scheduler such as Airflow for per-request business transactions.
In the wild
Uber
Conducting better business with Uber's open source orchestration tool, Cadence (opens in a new tab)Uber's 2019 post introduces Cadence, the orchestration engine it built so its many microservices could work together, and made open source in 2017. It describes how, in an asynchronous microservice architecture, simple code turns into a mass of callbacks, and how Cadence instead lets engineers implement activities that combine into workflows run by workers, with the engine providing persistence, queues and timers. The post lists uses at Uber from sign-up flows and order fulfillment to machine learning pipelines, including coordinating the standard steps of an Uber Eats order.
Check yourself
-
A worker crashes after two of five activities completed. What does the next worker do?
Show answer
B. Replays the workflow against the history, skipping completed activities, then runs the third Recorded results are returned during replay, so completed steps are not re-executed.
-
Why must workflow code be deterministic?
Show answer
B. Replay must take exactly the same path as the original run to line up with the recorded history If replay made different decisions, the recorded results would no longer match the code's steps.
-
Where should a call to a payment API go?
Show answer
B. In an activity, with a retry policy and an idempotency key Side effects belong in activities, which the engine retries; idempotency keeps retries safe.
-
A workflow must wait 30 days before sending a reminder. What does a workflow engine do?
Show answer
B. Records a durable timer; no process is blocked, and the workflow resumes when it fires Timers are part of the durable history, so long waits cost almost nothing.
Recap
- Workflow engines give durable execution: ordinary code whose progress survives crashes.
- Activities do side-effecting work with retries and timeouts; workflows orchestrate and must be deterministic.
- An event history records every step; workers replay it to resume exactly where a run left off.
- Durable timers, signals and declarative retries replace hand-written state machines.
- Costs: running the engine, determinism rules, versioning, and idempotent activities.