- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the benefit of including expert data in policy gradient updates without applying per-decision importance sampling correction. The authors argue that off-policy samples can be beneficial, as they can provide "action" and "exploration" guidance. Furthermore, they argue that even actions that are very improbable under the target policy can receive significant gradient updates.
The authors test their hypothesis using PPO on a gridworld and three language tasks of increasing difficulty. The results seem to support the authors' hypothesis.
Critical strengths
Counterintuitive findings The authors claim, essentially, that using an unbiased policy gradient estimator that uses off-policy data without proper importance sampling correction can be beneficial. Their theoretical argument applies to policy gradient without clipping, which makes the statement most interesting, since the clipping mechanism in PPO is exactly designed to prevent off-policy updates (which, even though PPO is considered on-policy, are still present, since the optimization policy diverges from the one used to collect samples as gradient updates are performed). If the authors' reasoning were correct, then this would be an important result. Even more interestingly, the authors define a mechanism to determine whether data is useful, summarized in equation 2.
Small Scale Experiments I like that the authors tested their hypothesis on the Grid-Maze, where they show gradient directions, providing good intuition for the claims, and an important, statistically significant result obtainable with the small problem size (same goes for countdown, word reversal).
Critical weaknesses
Poor concept formalization and sloppy notation: Many mathematical quantities are not defined in the main text (e.g., $\Delta_\rho). The same goes for results or findings, which are often presented at an intuitive (ambiguous) level (e.g., the authors often refer to "improvement" without formalizing what they mean). It is not clear whether the theoretical results admit early episode termination (before H steps). If so, some formulas seem incorrect.
Logical passages missing: In many parts of the paper, I cannot follow the argument's logic. Examples are sections 3.2 and 3.3. I have detailed more precise questions in the appropriate textbox. This is my main concern.
Clear improvement analysis. At the end of the day, beyond all the reasoning, much of which I am not able to follow, the critical question that is important to answer is the following (informally): (for small enough steps)? Under what conditions does this hold? The policy gradient theorem already identifies the optimal gradient direction for policy improvement and prescribes using on-policy samples.
Use of clipped objective in the experiments. The authors motivate their approach assuming an unclipped objective (essentially, OffPAC's objective), but test their experiments using PPO. As I have previously remarked, PPO's clipping surrogate objective is actually designed to make off-policy updates less harmful. In a way, this weakens the empirical support for the theory.
GSM8K. Is run only with one seed, which does not tell much about the algorithm's expected performance.
Questions for authors
Clarification
-
Background, line 098. The term is unclear to me. 1) On the left side of the equation, there is no time dependence, but such dependence appears on the right-hand side. 2) The right-hand side is "normalized" by the horizon . But what happens to episodes that terminate before steps?
-
Notice that, in the finite-horizon undiscounted setting, the value and Q-functions and the state visitation are time-dependent. I assume that we can still write Q(s, a) as a shorthand for Q(s, a, t), since in the transformer architecture there is a time embedding.
-
I think the statement 142--151 is vague and unclear. Terms like are not properly defined (one needs to consult the appendix), and it is unclear what kind of improvement line 140 refers to. I.e., an improvement from a policy gradient update?
-
I don't understand how Eq. 4 supports the claim in Section 3.2. It feels completely disconnected to me.
-
Can you concisely explain how you derive eq 5?
-
I don't understand the reasoning in 197--200; in particular, I don't understand where comes from. Why $ \pi_\text{old}(a|s)$ is at the numerator? Is it because we are looking at the first step of the optimization? Why should one assume the teacher contribution is negligible compared to the online contribution? The idea of using importance sampling is to make them "equivalent" in expectation.
-
What is the author's take on my comment in the "Critical Weaknesses"? Clear improvement analysis. At the end of the day, beyond all the reasoning, much of which I am not able to follow, the critical question that is important to answer is the following (informally): (for small enough steps)? Under what conditions does this hold? The policy gradient theorem already identifies the optimal gradient direction for policy improvement and prescribes using on-policy samples.
4 comments
- Clarification of lines 197–200
We evaluate the gradient at the start of the update, where . For the chosen action, teacher sampling probability times importance weight is
Online sampling gives the same product, . This equates the action weighting, not the gradients: importance sampling equates expectations for the same integrand, whereas source targets and visitation distributions differ. We do not assume the teacher gradient is negligible. Both directional gradients share , so both can be small absolutely without the teacher being small relative to the online source.
- Unclipped analysis and clipped experiments
The exact identities concern the unclipped surrogate. For positive advantage, PPO clipping suppresses the gradient only above ratio ; a tiny teacher ratio therefore remains active. Action and exploration guidance can persist through active contributions. However, ratio- and sign-dependent clipping changes effective margins, so the unclipped conditions and relative-gradient comparison need not carry over unchanged. We now state this qualification in Sections 3.1 and 3.3.
- Multiple training seeds
We have repeated Countdown, Word Reversal, and GSM8K with three independent training seeds. The main language learning curves now report mean development accuracy with ±1 standard deviation bands (Section 4, Figure 6). Held-out test results also report means and standard deviations across three training seeds (Table 1); Appendix B describes the evaluation protocol.
Held-out test accuracy (%) as mean ± standard deviation across three training seeds (Section 4, Table 1). OPD denotes on-policy distillation.
Greedy decoding
| Method | Countdown | Word Reversal | GSM8K |
|---|---|---|---|
| SFT | 33.7 ± 0.8 | 63.74 ± 5.29 | 66.54 ± 2.69 |
| OPD | 37.0 ± 0.7 | 74.54 ± 4.05 | 67.45 ± 0.68 |
| On-Policy PPO | 21.7 ± 3.1 | 95.80 ± 1.22 | 68.49 ± 1.15 |
| Off-Policy PPO | 43.6 ± 4.9 | 96.52 ± 1.24 | 71.17 ± 0.91 |
Sampled decoding (temperature 0.8)
| Method | Countdown | Word Reversal | GSM8K |
|---|---|---|---|
| SFT | 21.4 ± 0.7 | 61.36 ± 4.55 | 62.52 ± 1.56 |
| OPD | 26.6 ± 2.9 | 71.68 ± 4.80 | 63.58 ± 0.29 |
| On-Policy PPO | 20.4 ± 2.1 | 94.24 ± 1.19 | 50.14 ± 1.55 |
| Off-Policy PPO | 41.3 ± 3.5 | 94.95 ± 1.45 | 61.71 ± 0.62 |
We would appreciate clarification of any remaining concerns and would be happy to address them.
Thanks for the Reply
Dear authors, I've read your reply carefully. I will update my review accordingly and consider it during the reviewers-AC discussion.
Thank you for reading our response carefully and considering the revisions. We appreciate your engagement and would be happy to clarify last-minute questions if you have any.
Thank you for the detailed review. We address your questions below.
The corrected state frequency is
dρ(s)=H1∑t=0H−1Prρ(st=s).
The state contains the full autoregressive prefix, whose length determines the time step, so Q and V need no separate time argument. Episodes terminating early are padded through H with time-indexed absorbing states and zero subsequent rewards. This preserves episode return and normalizes dρ. Absorbing-state values and advantages are zero. We have corrected the occupancy definition and added the padding convention in Background and the appendix.
For a chosen action a⋆, let
wb=1−πold(a⋆∣s)πold(b∣s),b=a⋆.
We define
Δρ(s)=Aρ(s,a⋆)−∑b=a⋆wbAρ(s,b).
This compares the chosen action with the student's weighted alternatives, using source ρ's values. A state-only baseline cancels because the weights sum to one. The definition now appears at its first use in Section 3.1.
The margin–visitation condition guarantees an increase in the frozen mixed surrogate:
L(πδ)−L(πold)=2δ[dπold(s)Δπold(s)+dμ(s)Δμ(s)]>0,
where πδ increases a⋆'s probability at s and reduces alternatives proportionally. This does not guarantee increased return. The separate student–teacher equality result concerns return: with unbiased advantage targets, the mixed gradient is H−1∇J, so a sufficiently small step increases return if ∇J is nonzero. Section 3.1 now distinguishes these claims.
We do not establish
J(θ+Δθoff−policy)≥J(θ+Δθon−policy)
in general. The exact return gradient gives the steepest first-order increase among parameter-space directions of equal Euclidean norm. Our motivation concerns finite batches: online rollouts may miss useful prefixes or provide weak action-preference signals. Teacher data can supply coverage and favorable value margins at those prefixes. Our analysis characterizes these surrogate signals; whether their benefit outweighs the off-policy bias is evaluated empirically in Sections 4 and 5.
We acknowledge that the performance-difference identity did not provide strong enough support for our claim. We have replaced that discussion in Section 3.2 with a direct explanation of prefix-distribution bias: token-level correction retains the teacher's prefix distribution, supplying learning signals at prefixes that finite student rollouts may miss. This is the source of exploration guidance, provided the teacher's action preferences are useful.
With frozen targets and alternative weights, importance weighting gives the conditional surrogate
ℓρ(x;s)gρ(s):=∂xℓρ(x;s)∣x0=a∑πx(a∣s)Aρ(s,a)=σ(x)Aρ(s,a⋆)+[1−σ(x)]b=a⋆∑wbAρ(s,b),=ε(1−ε)Δρ(s).
Here σ(x)=1/(1+e−x) and σ(x0)=ε. The shared factor is the logistic derivative. After visitation weighting, relative source strength depends on both margins and visitation. Section 3.3 now includes the conditional surrogate and its derivative.