RL in the real world

The big idea

DQN played many millions of Atari frames, and policy-gradient agents fall over thousands of times in simulation before they walk. In a game that costs nothing: reset and try again. A real robot that falls breaks; a cooling plant that tries a random setting can overheat servers; a recommender that explores badly loses users. Three facts shape real-world reinforcement learning:

• Exploration is expensive or unsafe, so you cannot learn by trial and error on the live system.
• Simulators are wrong in small ways the agent will find and exploit.
• Rewards are hard to specify, and the agent optimizes exactly what you wrote.

Think of training a pilot. Nobody learns by crashing real planes. Pilots train in simulators built to be uncomfortable and varied (gusts, failures, bad weather), study logs of past flights, and fly real aircraft with hard limits and an instructor who can take over. Real-world RL uses the same three tools: varied simulation, learning from logged data, and constraints with a human in control.

How it works

Step 1: Simulation and the reality gap

Training in a simulator is cheap, fast and safe. The catch is the reality gapThe difference between a simulator and the real system (dynamics, sensing, delays, appearance) that makes a policy trained in simulation behave differently on real hardware.: friction, delays, sensor noise, lighting and wear are never modelled exactly, and a policy that is optimal for the simulator can sit right on the edge of failure for reality.

Domain randomizationTraining in simulation with randomly varied physics, sensing or appearance in every episode, so the policy must work across all of them and treats the real world as one more variation. attacks the gap head-on. Instead of making the simulator perfect, make it varied: randomize masses, friction, delays, textures and camera positions in every episode, so the policy has to work across all of them. Tobin and colleagues (2017) trained an object detector only on simulated images with deliberately non-realistic random textures and transferred it to a real robot, accurate to 1.5 cm; in their words, with enough variability, the real world may appear to the model as just another variation. The price is a more conservative policy that is not the best possible for any single setting.

Step 2: Offline RL from logged data

Many organizations already have years of logs: operators' choices in a plant, treatments given to patients, items shown to users. Offline RLReinforcement learning from a fixed dataset of previously collected transitions, with no further interaction with the environment during training. learns a policy from that fixed dataset without touching the live system.

The hard part is distributional shiftThe mismatch between the actions in the logged data and the actions a new policy would take; value estimates for rarely logged actions are unreliable and often too optimistic.. A Q-learning target takes a max over actions, and the max happily picks an action the logs barely contain, whose value estimate is a guess. Errors in the optimistic direction get selected and then propagated by bootstrapping. The CQL paper names this directly: standard off-policy methods can fail offline because of overestimated values for actions outside the data. Offline methods therefore stay close to the data: they penalize values of poorly supported actions (CQL learns a Q-function whose policy value is a lower bound on the true value), or restrict the policy to actions the logs cover. Before deploying, evaluate the new policy on the logs with off-policy estimators, which needs the logged action probabilities ( Multi-armed bandits ).

Step 3: Safety constraints and human control

Even a well-trained policy should not act unchecked. Common layers, several of which appear in the case study below:

• Hard constraints on actions and predicted states (temperature limits, torque limits), checked outside the learned model.
• Uncertainty filters: discard actions the model is not confident about.
• Redundant verification by an independent, simpler control system before an action is applied.
• Narrow operating limits at first, widened as evidence accumulates.
• A human override that hands control back to conventional rules at any time.

Going deeper

Code: the reality gap and the offline trap

import numpy as np

# Part 1, sim-to-real. A controller pushes a cart to position 1 with force = gain x (1 - measured x).
# The simulator's sensor is instant; the real sensor reports the position a few steps late.
def cost(gain, delay, steps=60):
    x = v = total = 0.0; seen = [0.0] * (delay + 1)
    for _ in range(steps):
        seen.append(x)
        v = 0.5 * v + 0.1 * gain * (1 - seen[-1 - delay]); x += v      # acts on a stale reading
        total += min(abs(1 - x), 10)                                    # distance from target, capped
    return total
gains = np.linspace(0.5, 20, 40)
best_sim = gains[np.argmin([cost(g, 0) for g in gains])]               # tuned in the perfect simulator
best_rand = gains[np.argmin([np.mean([cost(g, d) for d in range(4)]) for g in gains])]   # delay randomized 0-3
for delay in [0, 1, 2]:
    print(f"real sensor delay {delay}: tuned-in-sim gain {best_sim:.1f} cost {cost(best_sim, delay):6.1f}   "
          f"randomized gain {best_rand:.1f} cost {cost(best_rand, delay):6.1f}")

# Part 2, offline RL. Logged data from an old policy that tried 8 actions very unevenly. Pick a new
# action from the logs alone. True mean rewards: action 3 is best; actions 6 and 7 are rarely logged.
rng = np.random.default_rng(1)
true = np.array([0.50, 0.55, 0.52, 0.60, 0.40, 0.45, 0.35, 0.30])
counts = np.array([400, 300, 300, 200, 100, 50, 3, 2])                      # how often each was logged
naive_pick, safe_pick = [], []
for trial in range(1000):
    means = np.array([rng.normal(t, 0.3, n).mean() for t, n in zip(true, counts)])
    naive_pick.append(np.argmax(means))                                     # trust every average
    safe_pick.append(np.argmax(means - 2 * 0.3 / np.sqrt(counts)))          # pessimistic lower bound
for name, picks in [("naive: highest logged average", naive_pick), ("conservative: lower bound", safe_pick)]:
    picks = np.array(picks)
    print(f"{name:31s} true reward {true[picks].mean():.3f}, picked a rarely logged action {np.mean(picks >= 6):.0%}")

real sensor delay 0: tuned-in-sim gain 10.0 cost    1.3   randomized gain 1.0 cost    4.0
real sensor delay 1: tuned-in-sim gain 10.0 cost  516.5   randomized gain 1.0 cost    3.5
real sensor delay 2: tuned-in-sim gain 10.0 cost  545.0   randomized gain 1.0 cost    4.3
naive: highest logged average   true reward 0.555, picked a rarely logged action 16%
conservative: lower bound       true reward 0.597, picked a rarely logged action 0%

Sim-to-real. A controller pushes a cart to a target with force proportional to the measured error. The simulator's position sensor is instant; the real one reports positions a step or two late. Tuning the gain in the perfect simulator picks 10.0, which scores 1.3 there. On the real cart with a one-step delay the same gain oscillates out of control: cost 516.5 (each step's error is capped at 10). Tuning instead over randomized delays of 0 to 3 steps picks a gentler 1.0. It is worse in the perfect simulator (4.0 against 1.3) but stays at 3.5 and 4.3 with real delays of 1 and 2 steps. That is domain randomization in miniature: give up a little in the best case to avoid disaster in the cases you did not model.

Offline action choice. Logs from an old policy cover eight actions very unevenly: hundreds of samples for the first few, 3 and 2 for the last two. Over 1,000 repetitions, picking the action with the highest logged average chooses one of the two barely-logged actions 16 percent of the time, purely because a few lucky samples look great, and earns a true reward of 0.555. Picking by a pessimistic lower bound (the average minus two standard errors) never chooses them and earns 0.597, close to the best action's 0.60. Conservative offline RL methods apply the same pessimism to every value estimate.

Rewards and side effects

Real objectives are rarely one number. A cooling controller should save energy and also keep temperatures, pressures and equipment wear in bounds. Put everything into one weighted reward and the agent trades them in ways you did not intend; better to optimize the main objective subject to explicit constraints, as in the case study, and to watch for behaviour that scores well but looks wrong.

Trade-offs

Common mistakes

• Tuning a policy to one simulator and shipping it: the reality gap shows up as instability, not as a small drop.
• Running standard off-policy Q-learning on a fixed log and trusting its highest-valued actions.
• Putting safety in the reward ("-1000 for overheating") instead of in hard constraints the agent cannot trade away.
• Deploying without a fallback controller and a clear human override.
• Evaluating a new policy on logs without the logged action probabilities, so the estimate is biased.

In the wild

Check yourself

Recap