acceptodds
Withdrawn from ICLR 2027

Randomized Horizons with Centered History for On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Teacher-logit on-policy distillation learns from a student's own continuations, making response length a joint budget for generation, teacher supervision, and student differentiation. A fixed short horizon saves work but removes late supervision from the training target. We study an alternative that retains the mean of a specified full-horizon semi-gradient while evaluating response tails intermittently. Centered-history OPD combines a continuation decision shared across the batch with a learned prediction of the omitted tail gradient. A weighted loss produces the student update in one reverse traversal, and a contrast between continuation outcomes updates the prediction without computing a separate tail gradient. We characterize the additional covariance, the role of causal centering, and the distinction between expected retained work and runtime. On MATH, the observed mean correctness is 61.662% versus 60.333% for plain OPD with a 0.6B student, and 75.410% versus 74.623% with a 1.7B student. These comparisons use all three available centered-history runs and both available plain-OPD runs at each scale. The results motivate randomized supervision allocation as a practical design direction for on-policy distillation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM, On-Policy Distillation, Randomized horizons

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_hoMaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes centered-history on-policy distillation (OPD), a randomized horizon estimator for OPD after the student model generates prefixes. The core idea is to preserve the mean of the semi-gradient for a specified full horizon while intermittently evaluating response tails. All rollouts in the same batch first generate to a common prefix boundary, after which they share a random decision to either continue reasoning or stop the responses. For the decision to stop a response, a parameter-space vector b is used to predict the omitted subsequent response. The paper also characterizes the additional covariance, the role of causal centering, and the distinction between expected retained work and runtime. Experiments on MATH with Qwen3-0.6B and Qwen3-1.7B show that, compared with plain OPD accuracies of 60.333% and 74.623%, centered-history OPD achieves accuracies of 61.662% and 75.410%, respectively.

Critical strengths

1.The method has some degree of novelty. The estimator is clearly constructed and technically meaningful. 2.The paper clearly characterizes, in theory, the relationship among computational cost, tail-prediction error, continuation probability, and variance. 3.The experiments provide preliminary evidence that centered-history OPD can improve accuracy.

Critical weaknesses

1.The main claim of the method should concern resource savings rather than accuracy improvement, yet the final experimental design measures only accuracy, making the experimental setup highly unreasonable. 2.The experimental setup is overly cursory, using only one benchmark and one teacher; the experimental evidence is insufficient to support the effectiveness of the method. 3.Necessary ablation experiments are missing, making it impossible to determine the contribution of each component. 4.The experimental setup is not sufficiently comprehensive and lacks detailed information such as random seeds, response-level results, and question-level confidence intervals. 5.The theoretical guarantee remains limited to the raw semi-gradient at a single step before clipping.

Questions for authors

1.Please provide more detailed experimental settings, including random seeds and other parameters, as well as more detailed results on resource consumption, GPU-hours, and related costs. 2.Please compare the method with more baselines under comparable experimental conditions. 3.Please add ablation and analysis experiments to examine the various claims about the method and its theoretical proof in greater detail

Reviewer_TJSzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In on-policy distillation, longer responses lead to higher computational costs for generation, teacher scoring, and the student’s backpropagation. This paper proposes computing the latter portion (the "tail") of a response only with a certain probability pp. When this segment is included in the computation, it is weighted by 1/p1/p; when it is excluded, a learned "history vector" bb is used to approximate the missing gradient. This approach ensures that the expected gradient remains identical to that of full-length OPD, while significantly reducing the overall computational burden.

Critical strengths

Clear and easy to follow. The paper motivates the problem well: longer on-policy rollouts cost more, and truncating them drops late supervision. Figures 1–2 and Table 1 lay out the method and the order of each update step concisely, so it is easy to understand and to reimplement.

Critical weaknesses
  1. Scalability to large-scale training. The method keeps an extra FP32 vector b the size of the trainable parameters, which costs as much memory as another Adam moment. The experiments only train rank-16 LoRA, where this cost is negligible, but full-parameter training of large students would pay it in full. The paper does not discuss how b would work with ZeRO/FSDP sharding or gradient accumulation, since the update needs the globally reduced, unclipped gradient. In addition, the shared coin forces every rank to pause at the prefix boundary, which conflicts with asynchronous or continuous-batching rollout systems.

  2. No explanation for the gains over plain OPD. By the paper's own Proposition 1, the estimator has the same mean as full-rollout OPD and strictly larger variance, so at best it should match OPD. Yet it reports gains of +1.3 and +0.8 points with no analysis of where they come from. The natural ablations (b = 0, z = 0) are missing. So are diagnostics such as the fraction of batches that reach the cut, ‖S − b‖ versus ‖S‖, gradient variance, and training curves, so it is unclear whether the gains come from the proposed mechanism.

  3. Limited experiments. The only baseline is full-rollout OPD; fixed truncation, randomized truncation without history, and prior horizon methods (TOPD, ADWIN, FastOPD) are all missing. Although the method is motivated by efficiency, the paper reports no wall-clock or token measurements, and the 37.5% saving is purely analytical. Evaluation covers one benchmark, one teacher, small students, and only 128 training updates. There is no sensitivity analysis for p, ℓ, or η, and with 2–3 runs per method and no confidence intervals, the reported gains are not statistically convincing.

Questions for authors

Please see the weakness section.

Reviewer_o2mkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Randomized Horizons with Centered History in order to reduce the computational cost of on-policy distillation (OPD). It utilizes a shared Bernoulli continuation decision to randomly truncate response tail. It uses a learned history-based control variate to compensate for omitted gradient contributions. The key result is that the estimator preserves the mean of the full-horizon semi-gradient before clipping and optimizer updates. They provide experimental results for Qwen3 students of 0.6B and 1.7B parameters which shows modest improvements over standard OPD on MATH.

Critical strengths

Key strengths:

The paper addresses an important practical problem - reducing the computational cost of on-policy distillation while preserving the expected.

The history-based compensation mechanism is appealing because it estimates omitted tail contributions without requiring an additional reverse traversal.

The shared continuation decision offers an interesting opportunity to reduce batch-level critical-path computation.

Critical weaknesses

Here are some of the weakness in the paper. The major is is the lack of detailed experiments for supporting the claims. The paper is addressing an important problem with a reasonably good solution. But they need to conduct more experimental studies with more rigor to be able to substantiate the claims.

  1. Limited statistical evidence. The reported improvements are modest (+1.33 percentage points for the 0.6B model and +0.79 points for the 1.7B model), based on only two baseline and three proposed-method runs. There are no confidence intervals, significance tests, or paired-seed comparisons.

  2. Efficiency claims are not experimentally validated. The paper estimates a 37.5% reduction in retained token positions under its illustrative setting, but provides no end-to-end training time, throughput, GPU-hour, or memory measurements. Thus, the central computational-efficiency benefit remains unsubstantiated.

  3. Missing ablations. The contributions of randomized continuation, history-based compensation, and centered updates are not isolated. Comparisons against simpler alternatives, such as truncation without compensation or compensation without centering, are needed.

  4. Limited experimental scope. Evaluation is restricted to one benchmark, one teacher, two student sizes, and a fixed set of horizon and continuation parameters. Comparisons with existing adaptive- or shortened-horizon OPD approaches are also missing.

Questions for authors

Here are some of the questions I would like the authors to reflect on and address if possible:

What are the measured end-to-end training-time and GPU-hour savings compared with standard OPD?

Do the reported accuracy improvements remain statistically significant across more random seeds, with confidence intervals computed at the question level?

What is the individual contribution of randomized continuation, history compensation, and centering?

Can the authors provide diagnostics of the history predictor's accuracy and the resulting gradient variance reduction?

How does the method compare with existing horizon-reduction approaches at matched computational budgets?

Reviewer_sgj1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the tradeoff between computation and supervision coverage in on-policy distillation (OPD). The authors propose Centered-History OPD, which preserves the gradient in expectation over the random truncation. A continuation decision is shared across the batch, and a learned predictor compensates for omitted tail gradients. Proposed implementation choices allow to achieve the theoretical benefits without prohibitive cost, making it practically deployable in real training pipelines. The authors demonstrate the feasibility in an empirical study.

Critical strengths
  1. The authors prove that the centered-history gradient estimator preserves the conditional mean of the full-horizon semi-gradient before clipping, with added covariance proportional to (1-p)/p times the residual S-b. This provides formal guarantees that the randomized approach doesn't bias learning, only adds variance that can be controlled through better prediction. In my understanding this is an important results making it safe to deploy the method even if empirical gains are modest.

  2. The method computes prefix and retained-tail gradients through a single backward pass by sharing the student's key-value computation graph. History learning uses the contrast between continuation outcomes without additional teacher queries or traversals. Thus, the method is computationally feasible at scale making theoretical benefits realizable in practical implementations.

Critical weaknesses

Limited empirical evidence. See questions to the authors.

Questions for authors
  1. Would it be reasonable / possible to provide actual runtime measurements (wall-clock time, GPU hours) comparing centered-history vs. plain OPD? The theoretical 37.5% position reduction might benefit from empirical validation.
  2. Which component (shared continuation, centered history, or learned compensation) contributes most to the performance gains? Would it be possible to run relevant ablations ?
  3. Would it be possible to see how the performance generalizes on other benchmarks beyond MATH or on different teacher-student architectures?
  4. Would it be possible to improve the readability of the paper by introducing / explaining non standard terminology, e.g. "causal centering".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.