COURSE 14Advanced10 IN-DEPTH LESSONS

Reinforcement Learning

Learning through interaction and reward: Markov Decision Processes (MDPs), Q-learning, policy gradients, PPO, and Reinforcement Learning from Human Feedback (RLHF).

Start Lesson 1: Reinforcement learning overview →← All 29 Courses

Course Syllabus & Units

Unit 1

Overview

1 lessons
1.1L2

Reinforcement learning overview

**Reinforcement learning (RL)** trains an agent through trial and error: the agent takes actions, gets rewards over time, and improves a policy. Stanford CS234 describes RL as a paradigm for learning to make good decisions in tasks spann...

Read lesson →
Unit 2

Methods

3 lessons
2.1L2

Actor-critic overview

**Actor-critic** methods learn a policy (the **actor**) and a value helper (the **critic**) that reduces the noise of pure “multiply by total return” policy gradients. Berkeley CS285 pairs “Policy Gradients” with “Actor Critic” in the sa...

Read lesson →
2.2L2

Policy gradients overview

**Policy gradient** methods learn a policy directly—nudging action probabilities in directions that raise expected reward—rather than only filling a Q-table. CS234 and CS285 both devote substantial lecture time to policy search / policy ...

Read lesson →
2.3L2

Q-learning intuition

**Q-learning** learns a score for “take this action in this state”—how good that choice is, assuming you’ll act well afterward. CS234’s modules list Q-learning after tabular MDP planning and policy evaluation; Berkeley CS285 covers value...

Read lesson →
Unit 3

Risks

1 lessons
3.1L2

Reward hacking cases

**Reward hacking** (also called specification gaming) is when an agent maximizes the *measured* reward in ways that violate the designer’s intent—loopholes, unsafe shortcuts, or metric games.

Read lesson →
Unit 4

Alignment

1 lessons
4.1L2

RLHF at a high level

**RLHF** (Reinforcement Learning from Human Feedback) fine-tunes a model using human preference data as a reward signal. InstructGPT (Ouyang et al.) describes three stages: (1) supervised fine-tuning on demonstrations, (2) train a reward...

Read lesson →
Unit 5

Mdps

1 lessons
5.1L2

MDP, value, and policy foundations

A **Markov decision process (MDP)** is the standard math story for sequential decisions: you are in a state, you pick an action, the world moves, you get a reward. CS234 opens with tabular MDP planning, then policy evaluation, then learn...

Read lesson →
Unit 6

Deep Rl

1 lessons
6.1L2

Deep RL, policy gradients, and PPO

**Deep RL** uses neural networks as policies and/or value functions. **Proximal Policy Optimization (PPO)** is a popular on-policy algorithm that tries to take useful policy steps without updating so far that performance collapses. OpenA...

Read lesson →
Unit 7

Games

1 lessons
7.1L2

Game RL: AlphaGo and MuZero

**AlphaGo** combined deep neural networks with search to master Go, defeating Fan Hui 5–0 (2015) and Lee Sedol 4–1 (2016). **MuZero** (DeepMind, Nature paper; blog Dec 2020) learned to master Go, chess, shogi, **and** Atari *without bein...

Read lesson →
Unit 8

Contrast

1 lessons
8.1L2

Game RL vs LLM RLHF (contrast)

A deliberate contrast between **classical/game reinforcement learning** (MDPs, self-play, MCTS, AlphaGo/MuZero) and **RL from human feedback for language models** (InstructGPT, summarization-from-feedback, Anthropic HH-RLHF). MASTER’s re...

Read lesson →