- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+2)
- Soundness
- 2(+1)
Read review
Paper summary
This paper analyzes gradient variance in on-policy distillation. Under bounded feedback and score-moment assumptions, it establishes sharp worst-case horizon scaling of for token-local and for full-return and reward-to-go (RTG) unnormalized rollout gradients. It provides matching OPD constructions and analyzes variance reduction using baselines that depend on the generated prefix.
Critical strengths
The theoretical analysis appears sound and tightens the previous variance bound. These results should interest the theoretical reinforcement learning community.
Critical weaknesses
-
As the paper acknowledges, the analysis builds on known estimand relationships [1], score orthogonality [2], and the optimal baseline coefficient [3]. The contribution statement should distinguish these ingredients from the new OPD constructions and restricted-control results, and explain what new theoretical insight those results provide.
-
Full-return and RTG share the same cubic worst-case rate despite different optimization outcomes. Token-local estimates a different gradient, so its lower variance alone cannot establish superiority for the sequence objective. The rate comparison therefore offers limited guidance for estimator selection.
-
I found the paper's presentation can be largely improved in terms of the display of the empirical results and the organization of theoretical results. Section 5.1 compresses distinct control settings without sufficiently explaining their relationship. Figure 3 labels fitted slopes as exact exponents, while Table 1 omits uncertainty intervals, making the empirical scaling claims harder to assess.
[1] Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.
[2] Rui Yuan, Robert M. Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient.
[3] Ronan Keane and H. Oliver Gao. Variance reduction for score functions using optimal baselines.
Questions for authors
-
What new insights do the matching OPD constructions and restricted-control results provide beyond the earlier variance bounds and baseline analysis?
-
What guidance do the current results offer for choosing among the three updates for practioner? Please distinguish theoretical guarantees from findings specific to the tested settings.
-
Could you report confidence intervals for the fitted exponents in Figure 3 and Table 1? Do they support the reported difference in variance growth between token-local and the sequence estimators?
4 comments
Follow-up for Reviewer 1L5n
Dear Reviewer 1L5n,
Thank you again for your careful review. Your question about what guidance the variance bounds offer practitioners helped us clarify how the theory informs sampling budgets and what the training results suggest about choosing among the three update rules.
We posted our response on November 13 and wanted to check in before the discussion closes. We would appreciate hearing whether these explanations address your concerns. If anything is still unclear, we would be happy to clarify it while the discussion is open.
Thank you for your time.
Best regards, The Authors
I appreciate the authors’ detailed response and the clearer presentation. On W1, I now understand the main contribution as sharpening earlier variance bounds and showing that a single OPD instance can attain all three rates, even at a parameter value where the three mean updates agree. The paper also shows that an optimal time-only baseline can leave cubic worst-case variance. My main concern remains W2. Full-return and RTG share the same worst-case rate, so the rate alone gives little guidance on which to use. Across the tested prompts, token-local consistently has lower fitted variance slopes than both sequence estimators. However, a bound that is tight in the worst case can still be loose for a particular prompt distribution in a realistic OPD training setting. The analysis does not establish how selecting or reweighting training prompts would affect gradient variance or estimator choice. Given this remaining concern, I have decided to keep my overall score.
Practical Guidance and Estimator Selection for a Target Prompt Distribution
Thank you for highlighting the question of estimator choice for a specific prompt distribution.
Our concrete recommendation is to use paired pilot rollouts from the target prompt distribution to select the sequence estimator with the smaller gradient second moment, then verify that choice using independent rollouts. Full-return and RTG estimate the same sequence gradient, so this second-moment comparison directly compares their variances. In the new multi-prompt experiments, all six pilot trials selected RTG; independent evaluation found 62.4%–64.6% lower full-parameter gradient variance than full-return.
The theory further guides sampling budgets and control-variate design. For sequence gradients normalized by the number of generated tokens, worst-case variance grows linearly with generation length, providing a budget-scaling principle for maintaining a fixed estimation accuracy. A baseline depending only on token position can still leave the original variance-growth order, motivating prefix-dependent controls fitted with a score-energy-weighted objective.
For prompt selection and reweighting, the starting point is to fix the target prompt distribution, then allocate samples according to prompt-conditional gradient statistics. Under a fixed rollout budget and an unchanged target, the ideal sampling probability is proportional to the original prompt probability multiplied by the square root of its conditional gradient second moment, with importance weighting preserving the original target. Larger differences in these moments call for a corresponding adjustment in sampling allocation; when the moments are similar, the allocation stays close to the original distribution.
OPD attainability, control information, and estimator choice
Q1/W1: What new insights do the matching OPD constructions and restricted controls provide? The new insight from the matching constructions is that raw full/RTG variance can grow by an extra factor of T relative to token-local even when all three mean updates agree at one parameter point; the restricted-control results reveal that optimizing a time-only baseline need not remove cubic worst-case growth.
Theorems 4–5 realize this separation in one full-support, normalized OPD model with shared parameters and fixed probability floors, score/feedback bounds and positive gap as T grows. Thus the raw sequence estimators' cubic upper bound is tight within OPD.
The restricted-control result identifies a separate obstacle: even an optimally chosen time-only baseline can leave cubic worst-case variance. Over this distribution class, improving its coefficients alone cannot remove that order; the control needs richer information or the estimator must change. This is an information restriction, distinct from inaccurate baseline fitting. Time-only centering can still reduce particular instances to quadratic; deterministic local KL is sufficient for RTG, with Lemma 25's correction needed when it depends on the prefix.
Corollary 7 then quantifies what a learned control must achieve: its variance equals the full-prefix oracle residual plus ΣE[D_t(bhat_t−b*_t)²], with independent fitting and conditional score energy D_t. Thus a quadratic oracle residual is preserved when total weighted fitting error is O(T²). In the nondegenerate scalar/binary case, the full-prefix oracle can even attain zero residual. These results connect the information supplied to a control, the accuracy of its fit and the variance left after fitting.
Fu identifies the omitted future-feedback terms and earlier upper bounds; Yuan derives score-based moment bounds; Keane–Gao derives the optimal conditional baseline. Our additions explain attainability and the consequences of restricting or learning those coefficients (Introduction, K.1; upper-bound comparison in Table R1).
Q2/W2: What guidance do the current results offer for choosing among the three updates? Our practical guidance is to try token-local first in the tested GPT-2 reverse-KL training setting, and RTG first among sequence estimators when retaining the full sequence-gradient terms is required.
These choices use the same evaluation metric but impose different requirements on the update. Full/RTG include how early token choices change the distribution of later feedback; token-local differentiates with that prefix distribution held fixed. This is the theoretical distinction to check before selecting the rule.
Selection supported by training. In the GPT-2 three-way comparison, token/RTG/full final-to-initial KL ratios are .027/.355/.700 at T=512 with clipping and a common learning rate (Table 20). Within the two sequence estimators, matched GPT-2 and Qwen results favor RTG. Qwen3-8B→4B gives lower final KL for RTG in 13 of 18 correlated seed/horizon/prompt endpoints. At T=64 its mean training KL stays near initialization; at T=256 it decreases while full-return increases (K.5). These are experimental reasons to prioritize a rule in the tested settings.
Budget supported by theory. Once the rule is chosen, the bounds specify the worst-case rollout budget for fixed absolute MSE of its own mean. For token-averaged gradients, variance is Θ(1) for token-local and Θ(T) for full/RTG. Maintaining that estimation precision therefore requires a constant-order versus linear-in-T worst-case budget. Full and RTG share this order; their empirical comparison, rather than the exponent, motivates trying RTG first.
Q3/W3: Can you report confidence intervals, and do they support the variance-growth difference? Yes: all four full-parameter sequence-minus-token slope-difference intervals lie above zero, supporting faster variance growth over the measured horizons. We report them below; revised Table 1 also gives individual fitted-slope intervals. These approximate 95% simultaneous intervals are constructed separately for each model:
The paired bootstrap and interval construction are detailed in Appendix K.2. Both differences are positive at all twelve original GPT-2 checkpoints (Tables 9/J1); all 36 layers appear in Appendix K.3.
W3: Theory organization and experimental presentation. Section 5.1 (p. 7) now explains how available information determines residual variance: time-only controls establish the restricted case, full-prefix controls provide the ideal reference, and fitted controls quantify approximation error. The centering results specify when leading terms cancel. Figure 3 (p. 9) labels estimates “fitted slopes”; Table 1 supplies slope intervals for all six Qwen/Llama settings.