Lecture 2 · Markov Decision Processes

Slides: David Silver - RL Course (UCL 2015) · Previous: Introduction to RL (L1) · Next: Lecture 3 · Planning by Dynamic Programming
Companion: Silver RL Lecture 2 — Question List (your own stuck points, in Korean) · Markov Decision Process (MDP)

Read this instead of watching. Each slide is embedded below with only what the slide itself does not say — the reasoning, the caveats, and the questions worth being able to answer.

---

The one idea

L2 builds the MDP by adding one ingredient at a time, and each addition creates exactly one new question:

Each layer inherits everything below it, so once you can evaluate an MRP you can evaluate any MDP with a fixed policy — because fixing a policy collapses an MDP back into an MRP (p.27).

Contents

• 1 · The Markov property
• 2 · Markov reward processes
• 3 · The Bellman equation for MRPs
• 4 · Adding actions
• 5 · Value functions and the Bellman expectation equation
• 6 · Optimality
• 7 · The Bellman optimality equation
• Study checklist
• Gaps to close on your own
• Language notes

---

1 · The Markov property

p.3–6 · Markov Property and Markov Processes ⏱ 3:05

"The future is independent of the past given the present." The operational phrasing is better: the state is a sufficient statistic of the future, so once you know $S_t$ you may throw the history away.

The slide claims almost all RL problems can be formalized as MDPs, and the three examples are each doing work:

• Optimal control is a continuous MDP — the differential-dynamics framing is the same object with continuous $\mathcal{S}$ and $\mathcal{A}$.
• Partially observable problems are not exceptions; they convert into MDPs (p.53 shows how).
• Bandits are MDPs with one state. Which is why L9 can spend an hour on bandits and call it reinforcement learning.

$\mathcal{P}$ has one row per state, each summing to 1. That single matrix is the entire dynamics.

p.7–9 · Student Markov Chain ⏱ 7:44

Three classes, Facebook as a distraction with a $0.9$ self-loop, a pub, and Sleep as a terminal state.

A terminal state needs no special machinery — it is just an absorbing state with a self-loop of probability 1 collecting zero reward forever. Worth knowing early, because it means "episodic" and "continuing" are not two different formalisms.

The samples are variable-length sequences drawn from the chain. There is no reward yet and nothing to decide; this is only what happens.

---

2 · Markov reward processes

p.10–13 · Reward, Return, and Discounting ⏱ 13:04

Two things to keep straight from the start:

• $\mathcal{R}_s=\mathbb{E}[R_{t+1}\mid S_t=s]$ is immediate reward — one step only.
• $G_t$ is a random variable, one sample of a whole trajectory. There is no expectation in it yet. That arrives with the value function two slides later, and the distinction is the source of most early confusion.

p.14–18 · The Value Function ⏱ 23:04

The returns are random; the value function is not. $v(s)$ is an expectation over trajectories — a fixed number attached to a state. Holding that apart from $G_t$ is what makes the next hour parse.

The four sample returns from $C_1$ at $\gamma=\tfrac12$ give $-2.25,-3.125,-3.41,-3.20$ — all different, all legitimate samples of the same quantity. Averaging enough of them estimates $v(C_1)$, which is exactly Monte-Carlo evaluation in L4, arriving three lectures early without being named.

---

3 · The Bellman equation for MRPs

p.19–21 · The Decomposition ⏱ 29:52

The derivation is four lines and each has a name worth knowing: unroll $G_t$; factor out $\gamma$; recognize the remainder as $G_{t+1}$; then apply the law of iterated expectations to replace $\mathbb{E}[G_{t+1}]$ with $\mathbb{E}[v(S_{t+1})]$.

That last step is the only non-mechanical one — it is what turns a statement about trajectories into a statement about states.

p.22–23 · Matrix Form and Direct Solution ⏱ 38:41

Note also what linearity depended on: averaging over a fixed policy. Once you introduce a $\max$ (p.48), linearity is gone and so is the closed form.

---

4 · Adding actions

p.24–25 · The MDP ⏱ 43:19

$\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$. Both $\mathcal{P}$ and $\mathcal{R}$ now carry a superscript $a$ — think of one transition matrix per action.

The reward depending on the action is not a technicality. A trading agent that buys £1,000 of goods pays £1,000 for the action, regardless of where it lands. Costs attach to what you do, not only to where you end up.

p.26–27 · Policies ⏱ 46:23

Policies are stationary — they depend on the state, not on $t$. That is not a restriction but a consequence: the Markov property says $s$ already contains everything relevant, so a time index would add nothing.

---

5 · Value functions and the Bellman expectation equation

p.28–29 · Two Value Functions ⏱ 51:01

The subscript $\pi$ is new and necessary: with actions there is no longer one expectation. Different policies induce different distributions over trajectories, so "how good is this state" is ill-posed until you say under what behavior.

$q_\pi$ is the one that will matter. It is the quantity you compare when choosing — "should I go left or right" is answered by comparing $q$ values, not $v$ values.

p.30–35 · The Backup Diagrams ⏱ 54:51

The verification at $C_3$ with $\pi=$ uniform:

Note the equation refers to $v_\pi(C_3)=7.4$ on both sides — the state can reach itself through the pub. Self-consistency, not a computation you could run forward.

Same linear structure, same closed form, now via the induced MRP: $v_\pi=(I-\gamma\mathcal{P}^\pi)^{-1}\mathcal{R}^\pi$.

---

6 · Optimality

p.37–39 · Optimal Value Functions ⏱ 1:08:33

The maximum is over all policies — so $v_*$ tells you the best performance achievable in the MDP, without telling you what achieves it.

p.40–42 · The Optimal Policy Theorem ⏱ 1:16:07

Define $\pi\ge\pi'$ iff $v_\pi(s)\ge v_{\pi'}(s)$ for all $s$. Note it is a partial order: two policies each better somewhere are simply incomparable.

And the practical payoff, stated repeatedly: once you have $q_*$, you are done. Read off the argmax and act. No lookahead, no model, no further computation.

---

7 · The Bellman optimality equation

p.43–47 · Optimality Backups ⏱ 1:22:57

Study gives $-2$ then a state worth $8$; Facebook gives $-1$ then a state worth $6$. The immediate reward is worse and the total is better — a one-line demonstration that greedy on $q_*$ is not myopic, which is the misreading L3 has to correct explicitly.

p.48 · Solving It ⏱ 1:29:52

---

Study checklist

Answer these without the notes before moving on to L3. Each item links back to where it is worked out; open the toggle only after you have tried.

1. Give the three layers of the formalism and the question each new ingredient creates.
→ p.24–25 · The MDP ⏱ 43:19

Answer

Markov process $\langle\mathcal{S},\mathcal{P}\rangle$ — what happens?
Markov reward process $\langle\mathcal{S},\mathcal{P},\mathcal{R},\gamma\rangle$ — adds reward — how good is it?
Markov decision process $\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$ — adds actions — what should I do?
Each inherits the layer below, and fixing a policy collapses an MDP back to an MRP, which is why prediction and control separate cleanly.

2. State the Markov property operationally, and say what to do when it fails.
→ p.3–6 · Markov Property and Markov Processes ⏱ 3:05

Answer

$\mathbb{P}[S_{t+1}|S_t]=\mathbb{P}[S_{t+1}|S_1,\dots,S_t]$ — the state is a sufficient statistic of the future, so the history can be discarded.
When it fails, either allow non-stationarity and track the drifting solution, or — better — absorb the missing information into the state. If transition probabilities depend on how long you have been somewhere, that duration is part of the state. "Non-Markov" usually means your state is incomplete.

3. Distinguish $G_t$ from $v(s)$, and give the honest ranking of reasons to discount.
→ p.10–13 · Reward, Return, and Discounting ⏱ 13:04

Answer

$G_t$ is a random variable — one sampled trajectory's discounted reward. $v(s)=\mathbb{E}[G_t|S_t=s]$ is an expectation, a fixed number per state. Averaging sampled $G_t$ estimates $v$ — that is Monte-Carlo evaluation, arriving early.
Discounting: (1) mathematically convenient — stated as the real reason; (2) bounds returns in cyclic processes; (3) represents an imperfect model — trusting a ten-year plan means trusting your model that long; (4) interest rates for financial reward; (5) humans and animals genuinely prefer immediate reward. If all sequences terminate, $\gamma=1$ is fine.

4. Derive the Bellman equation for an MRP and say what kind of statement it is.
→ p.19–21 · The Decomposition ⏱ 29:52

Answer

Unroll $G_t$; factor out $\gamma$; recognize the remainder as $G_{t+1}$; apply the law of iterated expectations to get $\mathbb{E}[v(S_{t+1})]$. Result: $v(s)=\mathbb{E}[R_{t+1}+\gamma v(S_{t+1})|S_t=s]=\mathcal{R}_s+\gamma\sum_{s'}\mathcal{P}_{ss'}v(s')$.
It is a consistency condition, not a formula. It does not compute $v$; it characterizes it. Satisfy it everywhere and you have the value function. Every algorithm in L3–L8 forces it to hold.

5. Why can an MRP be solved in closed form, and why does that not settle the subject?
→ p.22–23 · Matrix Form and Direct Solution ⏱ 38:41

Answer

Because the Bellman expectation equation is linear: $\mathbf{v}=\mathcal{R}+\gamma\mathcal{P}\mathbf{v}\Rightarrow\mathbf{v}=(I-\gamma\mathcal{P})^{-1}\mathcal{R}$.
It costs $O(n^3)$, which is hopeless at scale — and the linearity depended on averaging over a fixed policy. Introduce a $\max$ and both the linearity and the closed form are gone.
The three named replacements are the rest of the course: dynamic programming, Monte-Carlo, TD learning.

6. What changed between the Student MRP and the Student MDP diagrams?
→ p.24–25 · The MDP ⏱ 43:19

Answer

The state space. Pub was a state; it is now an action. The diagram gained black action-nodes between the white state-nodes, and that is where randomness lives: you choose Pub, then the environment's dice pick $C_1$, $C_2$, or $C_3$.
Every other action has a black dot too, with transition probability 1 — just not drawn. All actions share the same two-stage structure: your choice, then the environment's roll.

7. What are the two half-steps of a backup, and who controls each?
→ p.30–35 · The Backup Diagrams ⏱ 54:51

Answer

$v\to q$ (white to black): average over your policy, $v_\pi(s)=\sum_a\pi(a|s)q_\pi(s,a)$. You control it.
$q\to v$ (black to white): average over the environment, $q_\pi(s,a)=\mathcal{R}^a_s+\gamma\sum_{s'}\mathcal{P}^a_{ss'}v_\pi(s')$. You do not.
Stack them and you get the full backup. There is no third kind of step in the course — later lectures only change how the averages are computed, or replace one with a max.

8. State the optimal policy theorem and the two conditions worth attaching to it.
→ p.40–42 · The Optimal Policy Theorem ⏱ 1:16:07

Answer

For any MDP there exists $\pi_*\ge\pi$ for all $\pi$ (with $\ge$ meaning $v_\pi(s)\ge v_{\pi'}(s)$ in every state), all optimal policies achieve $v_*$ and $q_*$, and there is always a deterministic optimal policy.
It rules out needing to switch policies depending on where you are — a single policy dominates everywhere at once.
Conditions: the order is partial, so incomparable policies exist; and determinism holds for a Markov, fully observed MDP. Under aliasing or partial observability the theorem does not apply and a stochastic policy can strictly win (L7).

9. Write both Bellman equations for $v$ and say precisely what differs.
→ p.43–47 · Optimality Backups ⏱ 1:22:57

Answer

Expectation: $v_\pi(s)=\sum_a\pi(a|s)\big(\mathcal{R}^a_s+\gamma\sum_{s'}\mathcal{P}^a_{ss'}v_\pi(s')\big)$.
Optimality: $v_*(s)=\max_a\big(\mathcal{R}^a_s+\gamma\sum_{s'}\mathcal{P}^a_{ss'}v_*(s')\big)$.
Exactly one half-step changed. The $q\to v$ step is identical — still an average over $\mathcal{P}$, because you never control the dice. The $v\to q$ step went from $\sum_a\pi(a|s)$ to $\max_a$, because now you are choosing.
Average over what you do not control; maximize over what you do.

10. Why is solving an MDP defined as finding $q_*$ rather than $v_*$?
→ p.37–39 · Optimal Value Functions ⏱ 1:08:33

Answer

$v_*$ tells you how much reward is available but not how to get it — choosing would still need a one-step lookahead through $\mathcal{P}$ and $\mathcal{R}$, i.e. a model.
$q_*$ finishes the job: $\pi_*(s)=\arg\max_a q_*(s,a)$, no model needed. Once you have $q_*$, read off the argmax and act.
This is exactly the argument L5 reconstructs as its first substitution ($V\to Q$) when it removes the model.

11. Why is there no closed-form solution to the Bellman optimality equation?
→ p.48 · Solving It ⏱ 1:29:52

Answer

The $\max$ makes it non-linear. $(I-\gamma\mathcal{P})^{-1}$ worked only because the expectation equation was linear in $v$.
The replacements named on the slide are the course: value iteration and policy iteration (L3), Q-learning and Sarsa (L5).
"Iterative" here means maintaining an estimate of the value function and improving it step by step — not a numerical routine for a fixed computation. That is the kind of iteration that survives losing $\mathcal{P}$.

12. The slide says $q_*(\text{Pub})=8.4$. Check it.
→ p.37–39 · Optimal Value Functions ⏱ 1:08:33

Answer

$q_*(\text{Pub})=1+\big(0.2(6)+0.4(8)+0.4(10)\big)=1+8.4=\mathbf{9.4}$.
$8.4$ is the transition term with the $+1$ immediate reward dropped. The decision is unchanged — studying gives $10>9.4$ — but the margin is far narrower than the slide suggests, and $9.4$ appears in later printings.

Gaps to close on your own

• Technical conditions for undiscounted MDPs. The lecture defers to Bertsekas (Dynamic Programming and Optimal Control, vol. 2) for when undiscounted MDPs are well defined. If you ever want $\gamma=1$ without guaranteed termination, this is the gap.
• The optimal policy theorem is stated, not proved. Your own Silver RL Lecture 2 — Question List Q3 already asked whether the proof is worth knowing; the proof itself is in Puterman or Sutton & Barto.
• Why the Bellman equation has a unique solution is assumed throughout and answered in L3 by the contraction mapping theorem → Contraction Mapping & Bellman Convergence.
• Sherman-Morrison for incremental matrix inversion is mentioned as reducing $O(n^3)$ to $O(n^2)$ in special cases. It returns in L6 for LSTD.

Language notes

Phrasings worth stealing for your own technical writing and talks: