Silver RL Lecture 5 — Question List

Status: pre-annotation. This captures the Q&A from a 2026-08-19 chat session, before any Zotero highlights were marked on the L5 slides. Once the corresponding pages are annotated, re-run zotero-harvest to convert this into the page-anchored format used by Silver RL Lecture 4 — Question List.
Source: Lecture 5 · Model-Free Control · David Silver - RL Course (UCL 2015) · Previous: Silver RL Lecture 4 — Question List

Q1. What exactly is bootstrapping?
Q2. On-policy vs. off-policy — what is actually different?
Q3. What is GLIE, and what does $\epsilon_k=1/k$ actually mean?
Status notes

Q1. What exactly is bootstrapping?

Asked: "bootstrap, on-policy, off policy 이런 용어 좀 쉽고 명확히 이해시켜줘" — 2026-08-19 chat, right after reading Lecture 5 · Model-Free Control.

Definition: bootstrapping means the update target is built using the algorithm's own current estimate, instead of waiting for the real outcome.

V(S_t) \leftarrow V(S_t) + \alpha\big(\text{target}_t - V(S_t)\big)

The difference between methods is entirely in what target is:

\begin{aligned} \text{MC (no bootstrap)} &: G_t = \sum_{k=0}^{T-t-1}\gamma^k R_{t+k+1} \\ \text{TD(0) (bootstrap)} &: R_{t+1} + \gamma V(S_{t+1}) \end{aligned}

$G_t$ is the actual return the episode produced — no $V$ estimate appears inside it, so nothing can be wrong except sampling noise. TD(0)'s target replaces everything after the first reward with $V(S_{t+1})$ , a guess that can itself be wrong. That is bootstrapping: fixing one guess using another guess, rather than waiting for ground truth.

Analogy

Checking a bank balance by taking yesterday's balance and adding today's transactions, instead of re-auditing the account from scratch. Fast, but if yesterday's number was already off, the error carries forward.

Why it matters for L5 specifically: everything in model-free control bootstraps off $Q$ instead of $V$ . Sarsa's target $R+\gamma Q(S',A')$ and Q-learning's target $R+\gamma\max_{a'}Q(S',a')$ are both one-step bootstraps — same mechanism as TD(0), just on the state-action value. → L5 notes §5, §8

This restates L4 Q1 in the control setting — the concept doesn't change from prediction to control, only which function ( $V$ vs $Q$ ) gets bootstrapped. Full target-family comparison (MC / TD(0) / $n$ -step / $\lambda$ -return) is in Model-Free Prediction.

Q2. On-policy vs. off-policy — what is actually different?

Asked: same session, immediately following Q1.

The one-line distinction: is the policy you're acting with the same as the policy you're learning about?

	Behaviour policy (acting)	Target policy (learning about)
On-policy	$\pi$	same $\pi$
Off-policy	$\mu$	different $\pi$

Sarsa is on-policy by construction, not by convention. The update

Q(S,A) \leftarrow Q(S,A) + \alpha\big(R+\gamma Q(S',A') - Q(S,A)\big)

is a sample of the Bellman expectation equation for $q_\pi$ . That equation's right-hand side is an expectation over your own policy from $S'$ onward — so the only way $Q(S',A')$ is a valid sample of it is if $A'$ is actually drawn from the same $\pi$ you're evaluating. Break that and the target no longer estimates the thing it claims to. → L5 notes, p.22 question

Q-learning is off-policy: it splits the action actually executed from the action used inside the bootstrap.

Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha\Big(R_{t+1}+\gamma \max_{a'}Q(S_{t+1},a') - Q(S_t,A_t)\Big)

$A_{t+1}$ (what you actually do next) is drawn from an exploratory $\epsilon$ -greedy $\mu$ ; the $\max_{a'}$ inside the target implicitly samples the greedy $\pi$ instead. You explore with one policy while learning the value of a different, better one.

Why bother with off-policy at all (from L5 notes §7):

Learn from watching someone else act, and conclude you could have done better than they did — not mere imitation.
Reuse old data — GPI walks through $\pi_1,\pi_2,\dots$ and discarding each policy's data is wasteful (this is why DQN-style methods are off-policy).
Explore however you like, while still converging to the deterministic optimal policy.
Ask several counterfactual questions off one single stream of experience.

The non-obvious causal link: off-policy Monte-Carlo needs an importance-sampling correction multiplied across the whole episode,

G_t^{\pi/\mu}=\frac{\pi(A_t|S_t)}{\mu(A_t|S_t)}\cdot\frac{\pi(A_{t+1}|S_{t+1})}{\mu(A_{t+1}|S_{t+1})}\cdots\frac{\pi(A_T|S_T)}{\mu(A_T|S_T)}\;G_t,

and this ratio decays toward zero over a long episode — not just "high variance," but unusable. This is what forces off-policy learning to bootstrap: a one-step TD target only needs one importance ratio, because everything past the bootstrap point is already summarized inside $V$ or $Q$ . Q-learning goes one step further and needs no importance sampling at all, because sampling the Bellman optimality equation (which already contains the $\max$ ) never required an expectation over $\mu$ to begin with — it's sampling a different equation, not correcting a mismatched one.

	Bellman equation sampled	Converges to
Sarsa	expectation equation for $q_\pi$	$q_\pi$
Q-learning	optimality equation for $q_*$	$q_*$

So Sarsa-vs-Q-learning is not a style choice — it's a choice of which Bellman equation you're sampling, and on/off-policy status falls out of that choice mechanically.

Q3. What is GLIE, and what does $\epsilon_k=1/k$ actually mean?

Asked: "GLIE에서 epsilon=1/k가 된다는게 뭔소린지 설명좀. GLIE가 뭔지 잘 이해 못했어 특히 두번째조건" — 2026-08-25 chat session.

GLIE = Greedy in the Limit with Infinite Exploration. It's the requirement a behaviour schedule (how exploratory episode $k$ 's policy is) must satisfy for MC control / Sarsa to converge to the true optimum $q_*$ , not merely to some self-consistent value.

\text{1) } \lim_{k\to\infty}N_k(s,a)=\infty \qquad\qquad \text{2) } \lim_{k\to\infty}\pi_k(a\mid s)=\mathbf{1}\big(a=\arg\max_{a'}Q_k(s,a')\big)

Why condition 2 exists — the part that doesn't just fall out of intuition. The target we actually want to solve is the Bellman optimality equation, which has a $\max$ inside it:

q_*(s,a)=\mathbb{E}\big[R+\gamma\max_{a'}q_*(s',a')\big]

If the behaviour policy stays soft forever — e.g. a fixed $\epsilon=0.1$ -greedy policy that never anneals — then the one-step target $R+\gamma Q(S',A')$ , with $A'$ drawn from that same soft policy, is not sampling the equation above. It is sampling the Bellman expectation equation for that specific soft $\pi$ :

q_\pi(s,a)=\mathbb{E}\Big[R+\gamma\sum_{a'}\pi(a'\mid s')q_\pi(s',a')\Big]

$q_\pi$ (weighted average over actions) and $q_*$ (max over actions) are different numbers — a policy that keeps a permanent chance of acting randomly is worth strictly less than the true optimum. Without condition 2, the algorithm converges perfectly — just to the wrong target. The behaviour has to actually become greedy for the sampled equation to turn into the optimality equation instead of a fixed soft policy's expectation equation.

Why $\epsilon_k=1/k$ satisfies both. Condition 2 is immediate: $\epsilon_k\to0 \Rightarrow$ greedy-action probability $\to1$ . Condition 1 is the subtler one — it isn't about whether $\epsilon_k$ shrinks, it's about whether the sum of exploration probability across all episodes still diverges:

\sum_{k=1}^{\infty}\epsilon_k=\sum_{k=1}^{\infty}\frac1k=\infty \qquad(\text{harmonic series})

Even though each episode's exploration chance shrinks, the cumulative exploration budget over infinite episodes is still infinite, which is what guarantees every $(s,a)$ pair keeps getting hit forever rather than being abandoned.

Schedule	$\sum_k \epsilon_k$	Cond. 1 (infinite exploration)	Cond. 2 (eventually greedy)	Converges to
constant $\epsilon=0.1$	$\infty$	✓	✗ (never reaches 0)	$q_\pi$ of the soft policy, not $q_*$
$\epsilon_k=1/k^2$	$\pi^2/6<\infty$	✗	✓	some $(s,a)$ can stop being explored early and stay wrong
$\epsilon_k=1/k$	$\infty$	✓	✓	GLIE satisfied

$1/k$ sits exactly on the boundary: it decays to zero (condition 2), but not so fast that the sum converges (condition 1). This is the same shape as the Robbins-Monro step-size condition ( $\sum\alpha=\infty,\ \sum\alpha^2<\infty$ ) seen in Sarsa's convergence conditions and in L4 Q2 — "travel far enough everywhere, but eventually settle." The lecture notes' own caveat — "It is a valid schedule, not a good one" — means other schedules satisfying the same two properties (e.g. $1/\sqrt{k}$ ) also work in theory; $1/k$ is just the simplest textbook example, not a claim that it's practically efficient.

Status notes

Not yet Zotero-annotated. Slide p.5 (On- and Off-Policy Learning) and pp.31–37 (importance sampling, Q-learning) are the natural highlight targets once the lecture is watched/re-read with this in hand.
When annotated, this file should be re-harvested into the same page-anchored, zotero://open-pdf/... format as Silver RL Lecture 4 — Question List, with an annotation-coverage table.
Grounding sources: Lecture 5 · Model-Free Control (§1, §4, §5, §7, §8) and Model-Free Prediction for the MC/TD backdrop behind Q1.

Contents

Q1. What exactly is bootstrapping?

Q2. On-policy vs. off-policy — what is actually different?

Q3. What is GLIE, and what does ϵk=1/k\epsilon_k=1/kϵk​=1/k actually mean?

Status notes

Q3. What is GLIE, and what does $\epsilon_k=1/k$ actually mean?