acceptodds
Under review as a conference paper at ICLR 2027

Sharp Horizon Scaling of Gradient Variance in On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student on self-generated sequences using positionwise teacher feedback. Temporal credit rules pair feedback with scores differently, but their tight horizon dependence remains unclear. Using a score–feedback matrix and conditional score centering, we show that past links affect variance without changing the mean, while future links can carry sequence credit. Under bounded feedback and scores, arbitrary prefix dependence, and shared parameters, we establish tight worst-case total variance of for token-local and for full-return and reward-to-go (RTG) in unnormalized rollout updates. One shared-parameter OPD construction attains all three rates at a fixed nonzero teacher–student gap, tightening the previous quartic full-return bound. Our control analysis characterizes removable variance and approximation costs; even optimal time-only controls retain cubic worst-case sequence variance. Full-parameter Qwen and Llama measurements show nearly linear token-local variance and faster sequence variance growth, revealing different length sensitivities. In controlled GPT-2 training with matched objectives and rollout budgets, RTG improves optimization, reducing normalized final reverse KL from 0.73 to 0.35 at . This advantage persists without gradient clipping.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, temporal credit assignment, gradient variance, reward-to-go, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1L5nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+2)
Soundness
2(+1)
Read review
Paper summary

This paper analyzes gradient variance in on-policy distillation. Under bounded feedback and score-moment assumptions, it establishes sharp worst-case horizon scaling of Θ(T2)\Theta(T^2) for token-local and Θ(T3)\Theta(T^3) for full-return and reward-to-go (RTG) unnormalized rollout gradients. It provides matching OPD constructions and analyzes variance reduction using baselines that depend on the generated prefix.

Critical strengths

The theoretical analysis appears sound and tightens the previous variance bound. These results should interest the theoretical reinforcement learning community.

Critical weaknesses
  1. As the paper acknowledges, the analysis builds on known estimand relationships [1], score orthogonality [2], and the optimal baseline coefficient [3]. The contribution statement should distinguish these ingredients from the new OPD constructions and restricted-control results, and explain what new theoretical insight those results provide.

  2. Full-return and RTG share the same cubic worst-case rate despite different optimization outcomes. Token-local estimates a different gradient, so its lower variance alone cannot establish superiority for the sequence objective. The rate comparison therefore offers limited guidance for estimator selection.

  3. I found the paper's presentation can be largely improved in terms of the display of the empirical results and the organization of theoretical results. Section 5.1 compresses distinct control settings without sufficiently explaining their relationship. Figure 3 labels fitted slopes as exact exponents, while Table 1 omits uncertainty intervals, making the empirical scaling claims harder to assess.

[1] Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.

[2] Rui Yuan, Robert M. Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient.

[3] Ronan Keane and H. Oliver Gao. Variance reduction for score functions using optimal baselines.

Questions for authors
  1. What new insights do the matching OPD constructions and restricted-control results provide beyond the earlier variance bounds and baseline analysis?

  2. What guidance do the current results offer for choosing among the three updates for practioner? Please distinguish theoretical guarantees from findings specific to the tested settings.

  3. Could you report confidence intervals for the fitted exponents in Figure 3 and Table 1? Do they support the reported difference in variance growth between token-local and the sequence estimators?

4 comments
AuthorsOpenReview ↗

OPD attainability, control information, and estimator choice

Q1/W1: What new insights do the matching OPD constructions and restricted controls provide? The new insight from the matching constructions is that raw full/RTG variance can grow by an extra factor of T relative to token-local even when all three mean updates agree at one parameter point; the restricted-control results reveal that optimizing a time-only baseline need not remove cubic worst-case growth.

Theorems 4–5 realize this separation in one full-support, normalized OPD model with shared parameters and fixed probability floors, score/feedback bounds and positive gap as T grows. Thus the raw sequence estimators' cubic upper bound is tight within OPD.

The restricted-control result identifies a separate obstacle: even an optimally chosen time-only baseline can leave cubic worst-case variance. Over this distribution class, improving its coefficients alone cannot remove that order; the control needs richer information or the estimator must change. This is an information restriction, distinct from inaccurate baseline fitting. Time-only centering can still reduce particular instances to quadratic; deterministic local KL is sufficient for RTG, with Lemma 25's correction needed when it depends on the prefix.

Corollary 7 then quantifies what a learned control must achieve: its variance equals the full-prefix oracle residual plus ΣE[D_t(bhat_t−b*_t)²], with independent fitting and conditional score energy D_t. Thus a quadratic oracle residual is preserved when total weighted fitting error is O(T²). In the nondegenerate scalar/binary case, the full-prefix oracle can even attain zero residual. These results connect the information supplied to a control, the accuracy of its fit and the variance left after fitting.

Fu identifies the omitted future-feedback terms and earlier upper bounds; Yuan derives score-based moment bounds; Keane–Gao derives the optimal conditional baseline. Our additions explain attainability and the consequences of restricting or learning those coefficients (Introduction, K.1; upper-bound comparison in Table R1).

Q2/W2: What guidance do the current results offer for choosing among the three updates? Our practical guidance is to try token-local first in the tested GPT-2 reverse-KL training setting, and RTG first among sequence estimators when retaining the full sequence-gradient terms is required.

These choices use the same evaluation metric but impose different requirements on the update. Full/RTG include how early token choices change the distribution of later feedback; token-local differentiates with that prefix distribution held fixed. This is the theoretical distinction to check before selecting the rule.

Selection supported by training. In the GPT-2 three-way comparison, token/RTG/full final-to-initial KL ratios are .027/.355/.700 at T=512 with clipping and a common learning rate (Table 20). Within the two sequence estimators, matched GPT-2 and Qwen results favor RTG. Qwen3-8B→4B gives lower final KL for RTG in 13 of 18 correlated seed/horizon/prompt endpoints. At T=64 its mean training KL stays near initialization; at T=256 it decreases while full-return increases (K.5). These are experimental reasons to prioritize a rule in the tested settings.

Budget supported by theory. Once the rule is chosen, the bounds specify the worst-case rollout budget for fixed absolute MSE of its own mean. For token-averaged gradients, variance is Θ(1) for token-local and Θ(T) for full/RTG. Maintaining that estimation precision therefore requires a constant-order versus linear-in-T worst-case budget. Full and RTG share this order; their empirical comparison, rather than the exponent, motivates trying RTG first.

Q3/W3: Can you report confidence intervals, and do they support the variance-growth difference? Yes: all four full-parameter sequence-minus-token slope-difference intervals lie above zero, supporting faster variance growth over the measured horizons. We report them below; revised Table 1 also gives individual fitted-slope intervals. These approximate 95% simultaneous intervals are constructed separately for each model:

ModelFull−tokenRTG−token
Qwen[1.1180,1.7554][1.0686,1.7214]
Llama[.7068,2.0127][.7563,2.1317]

The paired bootstrap and interval construction are detailed in Appendix K.2. Both differences are positive at all twelve original GPT-2 checkpoints (Tables 9/J1); all 36 layers appear in Appendix K.3.

W3: Theory organization and experimental presentation. Section 5.1 (p. 7) now explains how available information determines residual variance: time-only controls establish the restricted case, full-prefix controls provide the ideal reference, and fitted controls quantify approximation error. The centering results specify when leading terms cancel. Figure 3 (p. 9) labels estimates “fitted slopes”; Table 1 supplies slope intervals for all six Qwen/Llama settings.

AuthorsOpenReview ↗

Follow-up for Reviewer 1L5n

Dear Reviewer 1L5n,

Thank you again for your careful review. Your question about what guidance the variance bounds offer practitioners helped us clarify how the theory informs sampling budgets and what the training results suggest about choosing among the three update rules.

We posted our response on November 13 and wanted to check in before the discussion closes. We would appreciate hearing whether these explanations address your concerns. If anything is still unclear, we would be happy to clarify it while the discussion is open.

Thank you for your time.

Best regards, The Authors

I appreciate the authors’ detailed response and the clearer presentation. On W1, I now understand the main contribution as sharpening earlier variance bounds and showing that a single OPD instance can attain all three rates, even at a parameter value where the three mean updates agree. The paper also shows that an optimal time-only baseline can leave cubic worst-case variance. My main concern remains W2. Full-return and RTG share the same worst-case rate, so the rate alone gives little guidance on which to use. Across the tested prompts, token-local consistently has lower fitted variance slopes than both sequence estimators. However, a bound that is tight in the worst case can still be loose for a particular prompt distribution in a realistic OPD training setting. The analysis does not establish how selecting or reweighting training prompts would affect gradient variance or estimator choice. Given this remaining concern, I have decided to keep my overall score.

AuthorsOpenReview ↗

Practical Guidance and Estimator Selection for a Target Prompt Distribution

Thank you for highlighting the question of estimator choice for a specific prompt distribution.

Our concrete recommendation is to use paired pilot rollouts from the target prompt distribution to select the sequence estimator with the smaller gradient second moment, then verify that choice using independent rollouts. Full-return and RTG estimate the same sequence gradient, so this second-moment comparison directly compares their variances. In the new multi-prompt experiments, all six pilot trials selected RTG; independent evaluation found 62.4%–64.6% lower full-parameter gradient variance than full-return.

The theory further guides sampling budgets and control-variate design. For sequence gradients normalized by the number of generated tokens, worst-case variance grows linearly with generation length, providing a budget-scaling principle for maintaining a fixed estimation accuracy. A baseline depending only on token position can still leave the original variance-growth order, motivating prefix-dependent controls fitted with a score-energy-weighted objective.

For prompt selection and reweighting, the starting point is to fix the target prompt distribution, then allocate samples according to prompt-conditional gradient statistics. Under a fixed rollout budget and an unchanged target, the ideal sampling probability is proportional to the original prompt probability multiplied by the square root of its conditional gradient second moment, with importance weighting preserving the original target. Larger differences in these moments call for a corresponding adjustment in sampling allocation; when the moments are similar, the allocation stays close to the original distribution.

Reviewer_fqufOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how gradient variance scales with generation length in on-policy distillation (OPD) under three temporal credit assignment rules: token-local, full-return, and reward-to-go (RTG). The authors introduce a unified score–feedback matrix formulation that represents these three rules through different masks, allowing a systematic characterization of their expected updates and gradient variances. The theoretical analysis establishes tight worst-case variance rates of Θ(T²) for token-local updates and Θ(T³) for full-return and RTG estimators, improving the previously established O(T⁴) upper bound for full-return. The authors also provide matching teacher–student constructions with shared parameters and investigate prefix-based control variates, including their variance-reduction capabilities and the costs associated with estimating control coefficients. Experiments with Qwen, Llama, and GPT-2 illustrate different variance growth patterns across the three estimators and examine the effectiveness of prefix-based variance reduction. Controlled GPT-2 training experiments further demonstrate improved optimization performance with RTG compared to full-return updates under matched objectives and rollout budgets.

Critical strengths
  1. The paper provides detailed and rigorous proofs establishing tight worst-case gradient variance bounds for the three temporal credit assignment rules. The proposed score–feedback matrix formulation offers a unified framework that makes the relationships among token-local, full-return, and RTG explicit and enables a systematic comparison of their gradient variances. This is a meaningful theoretical contribution that supports a better understanding of temporal credit assignment in on-policy distillation.

  2. The paper provides experimental evaluations with Qwen, Llama, and GPT-2, illustrating different variance growth patterns and examining the relationship between theoretical predictions and empirical observations. The variance decomposition also provides useful insights into the sources of gradient variance, while the proposed scalar prefix controls demonstrate measurable variance reduction when sufficient samples are available. These results strengthen the theoretical analysis and provide useful guidance for potential practical applications.

Critical weaknesses
  1. Although the paper establishes tight worst-case gradient variance bounds, the practical implications of these findings and their value for future research are not sufficiently clear. A more explicit discussion of how these theoretical results can guide practical on-policy distillation and algorithm design would strengthen the contribution.

  2. The paper evaluates the effectiveness of the proposed prefix controls primarily through variance reduction measurements at fixed model parameters. However, it does not demonstrate whether these learned controls improve end-to-end optimization or downstream task performance. Additional experiments incorporating learned prefix controls into the training process would provide stronger evidence of their practical effectiveness.

  3. Although the paper includes gradient variance measurements with Qwen and Llama models, the full-parameter optimization experiments are conducted using GPT-2. Evaluating the optimization performance with a larger teacher and a smaller student from a modern model family, such as Qwen models of different sizes, would help establish the generalizability of the experimental findings.

  4. In Table 10, only 4.9%–5.6% of internal probe coordinates show nominally significant differences between the full-return and token-local mean gradients, which is close to the 5% null rate. Therefore, these tests do not provide clear empirical evidence of a detectable estimand gap in the evaluated settings, despite the theoretical characterization of their different learning targets.

Questions for authors
  1. The paper evaluates gradient variance using Qwen and Llama models, while the end-to-end optimization experiments are conducted with GPT-2. Could the authors explain the choice of GPT-2 for these experiments and discuss whether similar optimization improvements would be expected with modern teacher–student model pairs? Would it be possible to conduct additional experiments using a larger Qwen teacher and a smaller Qwen student?

  2. Section 5 introduces prefix-based variance reduction, and the experiments demonstrate that learned controls can reduce gradient variance when sufficient training samples are available. However, their impact on end-to-end optimization remains unclear. Could the authors evaluate whether incorporating these learned prefix controls into the training process improves optimization performance or downstream task results?

4 comments
AuthorsOpenReview ↗

Learned controls improve training; Qwen extends the sequence comparison

Q2/W2: Does incorporating learned prefix controls improve end-to-end optimization? Yes: incorporating learned prefix controls into GPT-2 full-parameter training lowers final reverse KL on the training and both held-out prompts, including at the same learning-rollout budget. With gradient clipping, the three-seed mean final KL/token is:

PromptRaw RTG, 100 updatesControlled RTG, 100Raw RTG, 180
Training.094616.042299.081127
Held-out 1.076462.028700.070547
Held-out 2.101079.029053.073443

The improvement holds for every seed and prompt against both baselines. The control arm uses 200 update rollouts plus 160 independent fitting rollouts; the raw baseline uses all 360 rollouts for 180 updates. The two baselines therefore compare equal update counts and equal learning-rollout budgets. Fitting data are independent of updates and evaluation.

Appendix K.4 (pp. 45–46) reports every seed, cross-seed standard deviations, backward-call counts and total elapsed times, alongside the update-axis average KL.

Q1/W3: Why GPT-2, and can you test a larger Qwen teacher with a smaller student? We chose GPT-2 for affordable, repeated full-parameter comparisons of update rules and learned controls; the added Qwen3-8B→4B runs compare full-return with RTG in a modern teacher–student pair. RTG finishes with lower KL in 13 of 18 correlated seed/horizon/prompt comparisons.

At T=64, mean training KL stays near its initial value under RTG (.290268 to .290668), while full-return reaches .338008. At T=256, RTG reduces it from .236733 to .229853, while full-return reaches .252852. The relative advantage therefore comprises stability at the shorter horizon and an absolute reduction at the longer one. Appendix K.5 (p. 46) reports three-seed mean initial and final KL for each horizon on the training and two held-out prompts.

In GPT-2, the sharpened-copy teacher and fixed training prompt isolate the update rule under a common training setup. This also supports the repeated learned-control experiment in K.4. The Qwen comparison in K.5 extends the full-return/RTG training evidence to a larger teacher and smaller modern student.

W1: How do these results guide practical OPD and algorithm design? Our guidance for algorithm design is to give a control access to prefix-dependent variation, fit it using a score-weighted criterion, and check its variance reduction on fresh rollouts before using it in training.

The time-only lower bound shows why coefficient fitting alone is insufficient to remove cubic worst-case growth over the stated class: even the optimal time-only baseline encounters that barrier. Corollary 7 then identifies the fitting target for richer controls. An error at a prefix with large score energy contributes more variance than the same error elsewhere. Independent fitting data and held-out gradient measurements let a practitioner assess whether the fitted control improves on the raw estimator. The GPT-2 experiment above implements this direction with the combined five-feature control: at equal learning-rollout budget, mean final training KL is .042299 versus raw RTG's .081127.

The rates also guide sampling once an estimator is selected. To estimate its token-averaged mean to fixed absolute MSE, the worst-case rollout budget is constant-order for token-local and linear in T for full/RTG under our feedback and score-moment assumptions. This gives a budget for the required estimation precision; the matched training comparisons provide the separate evidence for choosing the update rule.

W4: How should Table 10 be interpreted? Table 10 does not clearly distinguish the full and token-local mean gradients at the evaluated probes and rollout budgets. Its 4.9%–5.6% full-minus-token detection rates are close to the null calibration: full-minus-RTG, whose population mean is zero, yields 4.7%–5.4% under the same nominal 5% test.

The theoretical identity specifies that the full-minus-token mean difference equals the expected contribution of early tokens to later feedback; its magnitude depends on the model, prompt and parameter point. Table 10 tests whether that contribution is detectable in the selected coordinates. The variance-growth comparison is evaluated separately in Figure 3 and Table 1, where all four paired sequence-minus-token slope intervals are positive. The revision makes these two experimental questions explicit.

AuthorsOpenReview ↗

Follow-up for Reviewer fquf

Dear Reviewer fquf,

Thank you again for your careful review. Your suggestion to compare a larger teacher with a smaller student guided our additional Qwen3-8B to 4B experiments. We also reported training results with learned prefix controls in response to your question about their practical benefit.

We posted our response on November 13 and wanted to check in before the discussion closes. We would appreciate hearing whether these additional experiments address your concerns. If anything is still unclear, we would be happy to clarify it while the discussion is open.

Thank you for your time.

Best regards, The Authors

Thank you for the authors' response and additional experiments. I appreciate their efforts to address my questions and concerns.

The additional explanations and results provide useful evidence supporting the paper's contributions and address several of my original concerns, particularly W2.

Although some limitations remain regarding computational overhead and the generalizability of the practical improvements, I find the new evidence encouraging and maintain my positive assessment of the paper.

AuthorsOpenReview ↗

Thank You for Your Positive Feedback

Thank you for recognizing the additional explanations and experiments, and for your positive assessment of the paper. We also appreciate your specific suggestions throughout the discussion, which helped us explain the connection between the theory and algorithm design more clearly.

Reviewer_u3DBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes complete-rollout gradient variance for three OPD credit rules under arbitrary prefix dependence and shared parameters. A score-feedback matrix unifies their expected updates and variance bounds. It proves tight quadratic token-local and cubic full-return/RTG worst-case rates, with valid OPD lower-bound constructions, and develops optimal scalar prefix controls, approximation-cost identities, and limits under restricted information. Empirical studies measure gradient covariance in Qwen/Llama and compare GPT-2 optimization under matched sequence objectives.

Critical strengths
  1. The main theoretical advance is a tight characterization within valid OPD processes. Conditional score orthogonality removes a slack horizon factor from the full-return bound; the more distinctive contribution is matching lower examples that respect normalized teacher/student distributions, full support, fixed gaps, and shared parameters. The underlying score identity is classical, but these compatibility and tightness statements are substantive.
  2. The estimand distinction is handled correctly. The paper separates zero-mean past links from potentially informative future links. It does not compare token-local and sequence variance while silently assuming their gradients are identical.
  3. The control analysis adds useful substance. The optimal-prefix projection identity gives an exact weighted error cost for approximate controls. The time-only minimax barrier and full-prefix scalar-binary representation distinguish an attainable oracle from what restricted information can accomplish. The common witness becoming quadratic after simple centering is also explicitly acknowledged, rather than hidden.
  4. Several empirical checks directly address plausible objections. Full-parameter covariance traces extend beyond a single output direction. Appendix C.3 checks sensitivity to the largest rollout pair; C.7 uses an independently estimated signal direction; C.11 removes gradient clipping; and C.12 includes token-local training. These additions make the evidence considerably stronger than the abstract alone suggests.
Critical weaknesses

W1. The empirical scaling evidence needs a clearer uncertainty summary. Exponents are fitted over a small range of horizons and relatively few independent rollout pairs. Table 7 shows that a single pair contributes up to 58.6% of the token-local trace at one horizon and 22-26% for the sequence estimators. Deleting the largest contribution preserves the ordering, which is reassuring, but does not provide confidence intervals for the reported all-parameter slopes. Re-pairing the same trajectories primarily checks a numerical estimator choice, not independent sampling uncertainty. Please provide paired, rollout-level uncertainty for the slope differences, accounting for any shared trajectory prefixes across horizons. This can use existing samples.

W2. The optimization lesson should foreground the strongest result and its limits. RTG beating full-return is a useful comparison because they share the sequence target. However, Table 20 shows token-local substantially outperforming both on the same reverse-KL evaluation, across both horizons and clipping conditions. The paper does report this in Section 6, so it is not a missing experiment. It should also be reflected in the abstract/conclusion's practical interpretation: unbiased sequence credit costs variance and does not win in this controlled training setup. The sharpened-copy GPT-2 teacher and one training prompt support a mechanism study, not a general recommendation for modern reasoning or agent distillation.

W3. Fixed-gap asymptotics need a more prominent finite-horizon interpretation. The raw worst-case rates hold with a teacher-student gap fixed independently of T. For the fixed-teacher witness, the cubic coefficient contains a^2 with a=log cosh(delta), hence is of order delta^4 near teacher matching; the quadratic component is of order delta^2. The leading-term crossover for full-return is approximately 2 delta^2/a^2, around 803 at delta=0.1 and 3,203 at delta=0.05. These are illustrative witness calculations, not predictions for actual LLMs. Appendix E.5 already states the gap dependence, so this is not an omitted assumption; its consequence for practical interpretation deserves more emphasis. Separately, measured quantiles and fitted slopes do not establish the horizon-uniform bounded or fourth-moment assumptions. The empirical curves should be presented as descriptive measurements rather than validation of a minimax exponent.

W4. The paper could distinguish inherited theory from its new results more efficiently. The score-centering, RTG mean identity, and optimal score-function baseline have substantial prior foundations. The manuscript cites them, but the breadth of the theorem list can obscure that the distinctive results are the sharp OPD-compatible lower bounds, restricted-information residual rates, and their synthesis. A compact comparison with the precise prior bounds and assumptions would make the contribution easier to assess. No additional broad experimental campaign is needed for a theory-focused contribution.

Questions for authors
  1. What are uncertainty intervals for the all-parameter exponent differences, with resampling at the independent rollout or trajectory-cluster level?

  2. Given Table 20, what practical circumstances would justify paying for sequence credit rather than token-local training? Please distinguish a different target from demonstrated downstream benefit.

  3. Can the main text explain when the cubic term dominates at a finite teacher-student gap, and separate that issue from observed score-energy/feedback moments? How should the result be interpreted near teacher matching?

  4. Can you place the previous full-return bound and your new bound side by side with their exact assumptions and normalization conventions?

2 comments
AuthorsOpenReview ↗

Paired slope intervals, practical choice, and finite-horizon theory

Q1/W1: What are the all-parameter slope-difference intervals under rollout-level resampling? For full−token and RTG−token, respectively, the approximate 95% simultaneous slope-difference intervals are [1.1180,1.7554] and [1.0686,1.7214] for Qwen, and [.7068,2.0127] and [.7563,2.1317] for Llama. Resampling uses independent, non-overlapping rollout pairs; all four intervals support positive differences over the measured horizons.

Each pair contributes ½||g_i−g_j||² over all parameters, including normalization and tied embeddings. Within each horizon/seed cell, we resample these pair-level observations with the same weights across the three estimators. Bonferroni propagation of marginal intervals accommodates cross-horizon dependence, including possible shared prefixes.

These are approximate 95% simultaneous intervals within each model cohort, conditional on fixed seed strata and valid bootstrap marginal coverage. The broad Llama token interval [.2633,1.9700] reflects heavy-tailed contributions. Appendix K.2 gives the method and K.3 all 36 layer results.

Q2/W2: What practical circumstances justify the extra cost of sequence credit? Sequence credit is worth considering when early choices substantially affect later feedback: an opening may match the teacher locally yet lead to higher-KL continuations. Full/RTG assign that later cost to the early choice (Eq. 7). At fixed parameters and the same number of independent rollouts, this improves sequence-gradient mean-squared error when the squared omitted mean-gradient term exceeds the extra variance of the sample mean.

Appendix K.1 (p. 44) states this MSE criterion. Table 20 favors token-local: token/RTG/full final-to-initial KL ratios are .027/.355/.700 at T=512 with clipping and a common learning rate. Its final-KL and curve-area advantage also holds across C.12's paired seeds, horizons and clipping settings. RTG beating full-return informs the choice within the two sequence estimators. The revised abstract and conclusion foreground token-local's advantage; the experiments specify the single-prompt, sharpened-copy teacher setup. These experiments measure reverse-KL optimization; downstream benefit in the motivating scenario remains to be tested.

Q3/W3: When does cubic growth dominate at finite length, especially near teacher matching? When the cubic term dominates. In Proposition 14's fixed-teacher construction, it dominates when a²T²/[2δ²(T−1)]≫1, with a=log cosh δ and T>1. The large-T criterion is T≫2δ²/a². Section 4 (p. 6) states this finite-length condition.

Difference from measured score energy and feedback moments. The crossover compares the relative sizes of two variance terms; the common score scale α² cancels. Measured score energy describes gradient scale, while feedback moments and quantiles describe sampled log-ratio magnitudes and tails. Horizon-uniform bounds or moment conditions constrain the whole process family; these measurements describe sampled models and lengths. Figure 3 therefore labels fitted slopes, with their empirical role explained in K.2.

Interpretation near teacher matching. As δ approaches zero, the cubic coefficient shrinks as δ⁴ and the quadratic coefficient as δ², moving the crossover to longer horizons: T_cross≈2δ²/a²∼8/δ². In this construction, δ=.1 gives about 803 and δ=.05 about 3203. Halving the gap roughly quadruples the crossover length. Quadratic growth over finite lengths is thus compatible with cubic asymptotics at fixed positive gap.

Q4/W4: How do the previous and new full-return bounds compare under explicit assumptions? Bounds. For the same full-return product G=(Σr_t)(Σs_t), Fu's quartic upper bound becomes cubic here. Table R1 (p. 5) compares:

ResultVar_tr(G) ≤Var_tr(G/T) ≤
Fu, Appendix D.2B_r²B_s²T⁴B_r²B_s²T²
Our Proposition 2R²σ²T³R²σ²T

Exact assumptions. Fu assumes pathwise |r_t|≤B_r and ||s_t||≤B_s. Proposition 2 assumes F_t-measurable feedback with |r_t|≤R, E[s_t|F_{t−1}]=0, E[||s_t||²|F_{t−1}]≤σ², and square integrability. These hold almost surely, with constants uniform in T. For regular on-policy scores under Fu's bounds, R=B_r and σ=B_s suffice. Conditional score orthogonality removes one horizon factor.

Normalization convention. G is the unnormalized rollout gradient; G/T averages it over tokens. Var_tr is the covariance trace, so Var_tr(G/T)=Var_tr(G)/T². Thus raw T⁴ and T³ bounds become T² and T after token averaging.

Inherited and new results. Fu identifies omitted future-feedback terms; Yuan derives score-based moment bounds; Keane–Gao gives the optimal conditional baseline. Our OPD constructions establish sharpness with normalized distributions and shared parameters; fixed probability floors and positive gap are restrictions of these lower constructions. Our control results quantify restricted-information residual growth and the exact weighted approximation cost (Introduction, K.1).

AuthorsOpenReview ↗

Follow-up for Reviewer u3DB

Dear Reviewer u3DB,

Thank you again for your careful review. Your questions about uncertainty in the fitted slopes and the behavior near teacher matching helped us focus our response. We provided the full-parameter slope-difference intervals from rollout-level resampling and explained when the cubic variance term dominates in the fixed-teacher construction.

We posted our response on November 13 and wanted to check in before the discussion closes. We would appreciate hearing whether these results and explanations address your concerns. If anything is still unclear, we would be happy to clarify it while the discussion is open.

Thank you for your time.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.