Reinforcement Learning

Reinforcement Learning

Reinforcement learning (RL) is a fundamentally different approach to machine learning. Instead of learning from a fixed dataset of examples, an agent learns by interacting with an environment and receiving feedback on its actions.

The feedback comes as rewards or penalties. The agent's goal is to figure out what behaviour maximises its cumulative reward over time.

The Agent-Environment Loop

RL involves three key elements:

• Agent: the AI that takes actions (e.g., a game-playing AI, a robot)
• Environment: the world the agent operates in (e.g., a game board, a physical space)
• Reward signal: feedback on whether an action was good or bad

The loop:

1. Agent observes the current state of the environment
2. Agent takes an action
3. Environment transitions to a new state
4. Agent receives a reward (positive or negative)
5. Agent updates its strategy based on what it learned
6. Repeat

Over many iterations, the agent learns which actions in which situations lead to the highest long-term rewards.

Famous Examples

Game playing

• AlphaGo (DeepMind, 2016): Trained via RL to play the game of Go. Defeated the world champion — a feat many thought was a decade away.
• AlphaZero: Learned chess, Go, and Shogi from scratch, reaching superhuman level in hours with no human game data — just the rules and a reward signal.
• OpenAI Five: Beat professional Dota 2 teams after training for the equivalent of 45,000 years of gameplay.

Real-world applications

• Robotics: robots learn to walk, grasp objects, and navigate by trial and error
• Data centre cooling: DeepMind's RL system reduced Google's data centre cooling energy by 40%
• Drug discovery: RL explores chemical spaces to find promising molecule candidates
• RLHF: Reinforcement Learning from Human Feedback — how ChatGPT was tuned to be helpful and safe (more on this later)
• Reasoning models: today's frontier models also use RL after pre-training, but with rewards based on whether an answer is actually correct (e.g. on maths or code), not just human preference. This is what teaches a model to work through a problem step by step before answering

Why RL Is Hard

RL sounds elegant but has practical challenges:

• Sparse rewards: In many real tasks, feedback is rare. A robot might take 10,000 actions before getting any signal that tells it whether it's doing well.
• Exploration vs exploitation: Should the agent try new things (explore) or stick with what already works (exploit)? Getting this balance wrong leads to bad outcomes.
• Sample inefficiency: RL often requires millions of trials to learn what a human picks up in minutes. Games are ideal training grounds because you can simulate them millions of times cheaply.
• Real-world risk: You can't let a robot crash 10,000 times to learn to walk. Simulation is essential, but the real world always surprises.

Despite these challenges, RL is responsible for some of the most impressive AI results in history.