Lecture 3 · Planning by Dynamic Programming

Slides: David Silver - RL Course (UCL 2015) · Previous: Lecture 2 · Markov Decision Processes · Next: Lecture 4 · Model-Free Prediction
Companion: Silver RL Lecture 3 — Question List (your own stuck points, in Korean) · Contraction Mapping & Bellman Convergence

Read this instead of watching. Each slide is embedded below with only what the slide itself does not say — the reasoning, the caveats, and the questions worth being able to answer.

---

The one idea

L2 wrote down the Bellman equations and left them as equations. L3 does one thing to them:

Turn the equality into an assignment, and iterate.

Everything else follows from which Bellman equation you iterate. Slide p.26 is the entire lecture in one table:

Contents

• 1 · What dynamic programming is
• 2 · Iterative policy evaluation
• 3 · Policy iteration
• 4 · Why greedy improvement works
• 5 · How much evaluation is enough
• 6 · Value iteration
• 7 · The summary, and complexity
• 8 · Asynchronous dynamic programming
• 9 · The bridge to model-free RL
• Study checklist
• Gaps to close on your own
• Language notes

---

1 · What dynamic programming is

p.3–6 · Requirements and Scope ⏱ 2:18

Neither word means what it looks like. Dynamic = the problem has a sequential or temporal structure. Programming = mathematical programming, as in linear programming — optimizing a "program", which for mathematicians means what we call a policy. Nothing to do with writing code.

The shortest-path example makes both DP properties concrete at once: the shortest path from here to the wall decomposes through a midpoint (optimal substructure), and the sub-path from that midpoint gets reused by many different starting points (overlapping subproblems).

The two planning problems, and note that the second contains the first:

• Prediction: given MDP + policy → $v_\pi$. Equivalently, flatten the MDP under $\pi$ into an MRP and evaluate that.
• Control: given MDP → $v_*$, and hence $\pi_*$.

DP is far older and broader than RL — scheduling, sequence alignment, shortest paths, the Viterbi algorithm. What is special here is only that the Bellman equation supplies the recursion.

---

2 · Iterative policy evaluation

p.7–8 · The Update ⏱ 12:56

Read the backup diagram as every state taking a turn at being the root. One sweep = each state gets one one-step lookahead, using the previous iteration's values at the leaves.

Synchronous means the whole new value function is computed from the whole old one before anything is swapped in. That is a choice, not a requirement — p.29 drops it.

The starting point $v_1$ is arbitrary; zeros everywhere is the natural choice and means "I expect no reward anywhere."

p.9–11 · Small Gridworld ⏱ 19:45

$4\times4$ grid, two terminal corners, $-1$ per step, uniform random policy, $\gamma=1$. Since every step costs exactly $-1$, the value of a state is minus the expected number of steps to termination — the question "how good is this state?" and "how long until I get out?" are the same question.

Work one cell by hand and the mechanism becomes concrete. At $k=2$, a cell adjacent to the terminal corner: three directions give $-1+(-1)=-2$, and the one leading to the terminal gives $-1+0=-1$. Average over the four with probability $\tfrac14$ each:

which the slide prints as $-1.7$ — it is truncated to fit, not rounded to something else. Several cells that look identical differ past the displayed decimal.

---

3 · Policy iteration

p.12–13 · Evaluate, Improve, Repeat ⏱ 29:41

Up-arrows evaluate, down-arrows act greedily, and the two converge on $v_*$ and $\pi_*$ from any starting point.

Two facts the slide does not state:

• After the first improvement your policy is deterministic and stays deterministic, because $\arg\max$ returns a single action. That is consistent with the L2 theorem that a deterministic optimal policy always exists, so restricting the search to deterministic policies costs nothing.
• The convergence rate is independent of the initialization, though the initialization obviously affects how much work you do in practice — start at $\pi_*$ and you finish immediately.

This diagram is the one to memorize. It recurs in L5 with sampled evaluation, in L6 with function approximation, and in L7 with a gradient step replacing the greedy jump.

p.14–15 · Jack's Car Rental ⏱ 36:34

Two rental locations, up to 20 cars each, move up to 5 overnight, $10 per car rented, Poisson demand and returns. Location 2 receives more requests than returns, so it drains — the policy has to keep feeding it.

Reading the figure: the five squares are policies drawn as contour maps — axes are cars at each location, and the contour label is how many cars to move. The surface at the end is the value function, in dollars of expected lifetime revenue.

Unlike the gridworld, one improvement step is not enough: it takes four or five iterations to stabilize.

---

4 · Why greedy improvement works

p.16 · The Policy Improvement Theorem ⏱ 46:30

The proof is two moves, and separating them is what makes it readable.

Move 1 — one step is not worse. By construction $\pi'$ picks the maximizing action, so

The inequality is just a max is at least as large as any particular element. What it says: follow $\pi'$ for one step, then $\pi$ forever, and you do at least as well as $\pi$ throughout.

Move 2 — telescope it. Apply Move 1 again inside the expectation, at the next state, then again, unrolling:

Each step converts one more "then follow $\pi$" into "then follow $\pi'$". One step at a time is what turns a one-step improvement into a whole-policy improvement.

p.17 · When Improvement Stops ⏱ 52:36

Set the inequality to equality: $v_\pi(s)=\max_a q_\pi(s,a)$ — which is the Bellman optimality equation. So $v_\pi=v_*$ and $\pi$ is optimal.

That is the whole convergence argument for policy iteration: the only way improvement can stop is by satisfying the optimality equation.

---

5 · How much evaluation is enough

p.18–19 · Modified and Generalized Policy Iteration ⏱ 58:54

This slide answers the question the gridworld raised: must policy evaluation converge before you improve? No. Options, from most to least work: run to convergence; stop at $\epsilon$-convergence; stop after a fixed $k$; stop after $k=1$.

And $k=1$ turns out to be value iteration — evaluate once, improve, and the two collapse into a single max.

---

6 · Value iteration

p.20 · Principle of Optimality ⏱ 1:01:54

An optimal policy splits into an optimal first action and an optimal policy from wherever you land. The theorem says $\pi$ achieves $v_*(s)$ if and only if it achieves $v_*(s')$ for every reachable $s'$.

The "if and only if" is what licenses working backwards: solve the successors, then a one-step lookahead settles the predecessor.

Unreachable states are not an exception — they carry zero probability and simply do not appear in the equation.

p.21–22 · Value Iteration Intuition ⏱ 1:06:28

Shortest path with $-1$ per step. Values propagate outward from the goal: after $k$ sweeps, every state within $k$ steps has its correct value.

p.23–24 · Value Iteration ⏱ 1:12:38

Same machinery as policy evaluation with one symbol changed: $\sum_a\pi(a|s)$ becomes $\max_a$. Expectation equation → optimality equation.

Why $k=1$ modified policy iteration equals value iteration: evaluating the greedy ($\arg\max$) policy for exactly one step is taking the max. The two steps fuse, and no explicit policy is ever built.

---

7 · The summary, and complexity

p.26 · Synchronous DP Algorithms ⏱ 1:24:03

The slide the lecture asks you to take away. Three algorithms, each an iterated Bellman equation, differing only in which one.

Complexity per iteration, for $m$ actions and $n$ states:

---

8 · Asynchronous dynamic programming

p.27–31 · Three Ways to Choose States ⏱ 1:29:27

Drop the requirement that an iteration touches every state. Back up states individually, in any order.

The convergence condition is weak and worth remembering: it still converges as long as every state continues to be selected eventually. The order is free; only the coverage matters.

In-place keeps one copy of $v$ instead of two, so a backup immediately sees values updated earlier in the same sweep.

Prioritized sweeping orders by Bellman error — how much this state's value would change if updated. Large error = large downstream consequences, so do it first.

The practical picture: with rewards concentrated somewhere far away, most of the state space sits at $0$ and updating it recomputes $0=0+\gamma\cdot 0$ forever. Prioritization spends effort only where something is actually changing. The cost is needing reverse dynamics — which states lead into the one you just updated — and a priority queue.

Real-time DP picks states by running the agent and backing up whatever it visits. If your robot is on this bit of carpet, the far corner of the room is not worth computing.

---

9 · The bridge to model-free RL

p.32–33 · Full-Width and Sample Backups ⏱ 1:36:18

Every backup so far considered every action and every successor state. Two costs follow, and the second is the one that ends the lecture:

1. Bellman's curse of dimensionality. $n=|\mathcal{S}|$ grows exponentially in the number of state variables, so DP tops out around millions of states — fine for medium problems, hopeless for Go's $10^{170}$, where even a single backup is unaffordable.
2. You must know $\mathcal{P}$ and $\mathcal{R}$. The expectation is not computable otherwise.

---

Study checklist

Answer these without the notes before moving on to L4. Each item links back to where it is worked out; open the toggle only after you have tried.

1. What single operation turns L2's Bellman equations into L3's algorithms, and what does each choice give you?
→ p.7–8 · The Update ⏱ 12:56

Answer

Turn the equality into an assignment and iterate.
Bellman expectation equation → iterative policy evaluation (prediction). Expectation + greedy improvement → policy iteration. Bellman optimality equation → value iteration. That is slide p.26 and it is the whole lecture.

2. Which two properties must a problem have for DP, and what supplies each in an MDP?
→ p.3–6 · Requirements and Scope ⏱ 2:18

Answer

Optimal substructure — supplied by the Bellman equation, which decomposes a state's value into one step plus the successor's value.
Overlapping subproblems — supplied by the value function, which caches "reward obtainable from here" and is reused everywhere that state appears.
The reframe worth keeping: the value function is a cache.

3. In the small gridworld, what does the right-hand column show, and what does it imply?
→ p.9–11 · Small Gridworld ⏱ 19:45

Answer

The greedy policy with respect to $v_k$. By $k=3$ it is already optimal, while the value function is far from converged ($v_\infty\approx-14,-20,-22$).
Two implications: any value function can produce a better policy — you do not need $v_\pi$ — and running evaluation to convergence is wasted work. That is modified policy iteration, and at $k=1$, value iteration.
Also: $-1.7$ is $\tfrac34(-2)+\tfrac14(-1)=-1.75$ truncated for display.

4. Prove that greedy policy improvement does not make things worse, and say why greedy is not myopic.
→ p.16 · The Policy Improvement Theorem ⏱ 46:30

Answer

One step: $q_\pi(s,\pi'(s))=\max_a q_\pi(s,a)\ge q_\pi(s,\pi(s))=v_\pi(s)$ — a max is at least any particular element. Follow $\pi'$ once then $\pi$ forever and you do no worse.
Telescope: apply that inequality again at the next state, and again, converting one more "then $\pi$" into "then $\pi'$" each time, until $v_\pi(s)\le v_{\pi'}(s)$.
Not myopic because $\arg\max_a q_\pi(s,a)$ maximizes one step of action plus the value function, which already summarizes all future reward under $\pi$. It will trade immediate reward for a better successor.

5. Why can policy iteration not get stuck at a local optimum?
→ p.17 · When Improvement Stops ⏱ 52:36

Answer

Policy improvement defines a partial order: $\pi'\ge\pi$ iff $v_{\pi'}(s)\ge v_\pi(s)$ in every state. Each greedy step produces a policy $\ge$ its predecessor in that sense, so you can only climb.
The failure you would fear — better in some states, worse in others — is excluded by the "every state" quantifier; there is no sideways move to be trapped in.
And when climbing stops, $v_\pi(s)=\max_a q_\pi(s,a)$, which is the Bellman optimality equation, so $\pi$ is optimal.

6. Does policy evaluation have to converge before you improve?
→ p.18–19 · Modified and Generalized Policy Iteration ⏱ 58:54

Answer

No. Options: run to convergence, stop at $\epsilon$-convergence, stop after fixed $k$, or stop after $k=1$ — all converge to $\pi_*$.
$k=1$ is exactly value iteration: evaluating the $\arg\max$ policy for one step is taking the max, so the two steps fuse and no explicit policy is built.
The general frame is GPI — any evaluation, any improvement, alternating — and the rest of the course swaps components into it without changing the diagram.

7. Value iteration's numbers go down while the improvement theorem says values go up. Resolve it.
→ p.23–24 · Value Iteration ⏱ 1:12:38

Answer

They describe different objects. The improvement theorem is about the value functions of actual policies under policy iteration, which do increase.
Value iteration's intermediate $v_k$ is generally not $v_\pi$ for any policy $\pi$ — it is an intermediate construct on the way to $v_*$, so no monotonicity claim was ever made about it.
Policy iteration produces a real policy's value at every step; value iteration works directly in value space and never builds a policy until the end.

8. State the principle of optimality and say why "work backwards from the goal" is only intuition.
→ p.20 · Principle of Optimality ⏱ 1:01:54

Answer

$\pi$ achieves $v_*(s)$ if and only if it achieves $v_*(s')$ for every reachable $s'$. The "only if" is what lets you solve successors first and settle the predecessor with a one-step lookahead.
The algorithm does not know where the goal is — or whether one exists. It sweeps every state every iteration and works on continuing, looping, stochastic MDPs with no terminal state at all. The backwards picture is a legible special case, not a requirement.

9. Give the complexity of DP with $v$ and with $q$, and say why the course later prefers the expensive one.
→ p.26 · Synchronous DP Algorithms ⏱ 1:24:03

Answer

$O(mn^2)$ per iteration with $v$; $O(m^2n^2)$ with $q$ — because $q$'s roots and leaves are both state-action pairs.
$q$ removes the model. Acting from $v$ needs $\arg\max_a(\mathcal{R}^a_s+\gamma\sum_{s'}\mathcal{P}^a_{ss'}v(s'))$, and that lookahead is $\mathcal{P},\mathcal{R}$. With $q$ it is $\arg\max_a q(s,a)$ and needs nothing.
In L3 you have the model, so $v$ is cheaper and sufficient. From L5 you do not, and the constant factor is trivially worth paying.

10. Name the three asynchronous DP schemes and the condition that keeps convergence.
→ p.27–31 · Three Ways to Choose States ⏱ 1:29:27

Answer

In-place (one copy of $v$, so backups see values updated earlier in the same sweep), prioritized sweeping (order by Bellman error; needs reverse dynamics and a priority queue), real-time DP (back up the states the agent actually visits).
Condition: convergence holds as long as every state continues to be selected eventually — the order is entirely free.
Ordering is worth a factor of $n$: on a corridor swept goal-backwards, in-place value iteration finishes in one sweep.

11. What two problems do full-width backups have, and what single change fixes both?
→ p.32–33 · Full-Width and Sample Backups ⏱ 1:36:18

Answer

(1) Bellman's curse of dimensionality — $|\mathcal{S}|$ grows exponentially in the number of state variables, and for large problems even one backup is unaffordable. (2) You must know $\mathcal{P}$ and $\mathcal{R}$ to compute the expectation.
The fix is sampling. Back up a sampled transition $\langle S,A,R,S'\rangle$ instead of expanding the full expectation: the cost becomes constant in $|\mathcal{S}|$, and the model disappears because the environment performs the expectation for you.
Sampling was introduced for efficiency and delivered model-freedom as a side effect. That is the step from planning to RL.

Gaps to close on your own

• Slide p.34, approximate dynamic programming, is passed over. It is the first appearance of a function approximator $\hat v(s,\mathbf{w})$ in the course: sample a subset of states, compute Bellman targets for them, fit the approximator, repeat — fitted value iteration. This is the direct ancestor of L6, and of DQN's fixed Q-targets in particular.
• The value iteration demo (cs.ubc.ca/~poole/demos/mdp/vi.html) is recommended and not shown. It is the same gridworld at a size where you can watch values propagate.
• Modified policy iteration's convergence for general $k$ is asserted and not proved; only the $k=1$ case is identified with value iteration.
• Prioritized sweeping needs reverse dynamics — the predecessors of a state. In a learned or sampled setting that is not free, and the lecture does not say how to get it.
• The relationship between policy iteration and expectation-maximization is mentioned as real but subtle, explored in recent papers, and left there.

Language notes

Phrasings worth stealing for your own technical writing and talks: