- Learn
- Machine Learning
- Reinforcement Learning
ML-M13
Reinforcement Learning
Learning by trial and error: agents, rewards and Markov decision processes, bandits, Q-learning, deep Q-networks, policy gradients and PPO, and what changes outside of games.
Supervised learning is told the right answer for every example. Reinforcement learning is only told how well things went, often much later, and must work out which of its own actions deserve the credit. That makes it the tool for problems that are sequences of decisions: games, robots, control systems, recommendations over time, and the fine-tuning of language models.
You will start with the agent-environment loop and Markov decision processes, then the simplest case, the multi-armed bandit, where the whole difficulty is exploration against exploitation. Q-learning learns the value of every action in every state from experience, and deep Q-networks replace its table with a neural network. Policy gradient and actor-critic methods, including PPO, adjust the policy directly. The module ends with what makes real-world reinforcement learning hard: simulators that differ from reality, learning from logged data, and safety constraints.
Lessons
- ML-13.01 RL fundamentals and MDPs In reinforcement learning, an agent observes a state, takes an action and receives a reward and a new state, trying to maximize total reward over time. Intermediate 14 min
- ML-13.02 Multi-armed bandits A multi-armed bandit is the simplest RL problem: repeatedly choose among options with unknown payoffs. Intermediate 12 min
- ML-13.03 Q-learning Q-learning learns the value of taking each action in each state. Intermediate 14 min
- ML-13.04 Deep Q-Networks Deep Q-networks replace the Q-table with a neural network so RL can handle large state spaces such as raw game pixels. Advanced 12 min
- ML-13.05 Policy gradients and actor-critic, PPO Policy gradient methods directly adjust a policy's parameters to make rewarding actions more likely. Advanced 15 min
- ML-13.06 RL in the real world Real-world RL is harder than games: exploration can be costly or unsafe, simulators differ from reality and rewards are hard to specify without loopholes. Advanced 8 min