- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to reuse trajectories already generated by the student to reduce the cost of repeatedly generating complete responses during on-policy distillation. The authors propose REVO, which separates distribution shift during trajectory reuse into two components: the difference between stored prefixes and those visited by the current policy, and the difference between previously sampled tokens and the current student distribution at a fixed prefix.
For the latter, REVO resamples next-token candidates from the current student distribution at each update; for the former, it stabilizes training using geometrically normalized and capped prefix weights. The method additionally uses the variance of the student–teacher log-probability ratio, termed RKL variance, to assign two levels of training weights to token positions, prioritizing positions with stronger remaining supervision signals.
The experiments show that REVO, with 50 rollout iterations and multiple parameter updates per iteration, matches or exceeds OPD baselines trained for 200 rollout iterations. The authors also provide cross-domain evaluations, training-time comparisons, and component ablations, showing that the gains do not simply come from increasing the number of parameter updates.
Critical strengths
-
The paper clearly distinguishes two sources of distribution shift during trajectory reuse and provides practical ways to address them.
Resampling the current token at a fixed prefix provides supervision under the current student distribution without requiring the entire trajectory to be regenerated. For prefix mismatch, which cannot be eliminated in the same way, the authors use stabilized weights and explicitly acknowledge their approximate nature. This design takes advantage of the distillation setting and makes both the computational costs and the sources of approximation relatively easy to understand.
-
The efficiency gains are supported by substantial experimental evidence and cannot be explained simply by additional updates.
Across multiple student–teacher scales, REVO achieves better average performance with 50 rollout iterations than OPD with 200 iterations. The paper also includes an OPD(U10) baseline that performs the same ten updates per rollout iteration but performs substantially worse than REVO, indicating that the improvements cannot be attributed solely to more parameter updates.
Furthermore, the appendix reports that, with a Qwen3-1.7B student and Qwen3-4B teacher, 50 REVO iterations take approximately 5.5 hours, whereas 200 standard OPD iterations take approximately 10 hours. This shows that the benefits extend beyond rollout counts to actual training time. Component ablations and comparisons with alternative prioritization signals further support the method’s design.
Critical weaknesses
-
The objective underlying gradient alignment and the relationship between stochastic weighting and the theoretical weights need to be stated more precisely.
Section 3 first defines the reverse-KL gradient at a fixed prefix and then averages it over prefixes visited by the current policy. This is not necessarily equivalent to directly differentiating the overall objective defined as the average KL over the current policy’s prefix distribution: the latter also involves derivatives arising from the dependence of that distribution on the parameters. One-step resampling exactly handles the current-token distribution at a fixed prefix, but it does not automatically resolve this distinction between objectives. Treating prefixes as fixed data is a reasonable training convention, but the paper should clarify whether REVO and the OPD baseline follow exactly the same convention.
In addition, according to Algorithm 1, the same resampled tokens are used to estimate both RKL variance and the gradient, followed by threshold-based weighting and within-batch normalization. Proposition 4.1, however, analyzes a local descent bound under deterministic weights. Stopping gradients through the weights only prevents differentiation through them; it does not remove the statistical dependence between the estimated weights and gradients. A gap therefore remains between the practical update and the deterministic weighting analyzed theoretically. It is reasonable to present the two-level rule as a theoretically motivated practical design, but the paper should also explain whether this dependence introduces an additional selection effect.
-
Direct evidence connecting RKL variance to effective updates realizable by the actual model remains limited.
The authors explicitly distinguish RKL variance from gradient-estimation noise and report a strong correlation between RKL variance and logit-space gradient signal-to-noise ratio in Figure 4. The appendix also discusses the Fisher–Jacobian projection, showing that the parameter-space relationship additionally depends on the model’s local structure. My concern is therefore not that the paper confuses these two variances or ignores the effects of parameterization.
Rather, a large difference between teacher and student output distributions does not necessarily mean that the current student can effectively reduce that difference through parameter updates. Some correction directions may not be locally realizable by the current parameters, and updates from different positions may conflict when combined. The high- versus low-variance training comparison in Figure 3 and the alternative-weighting ablations support the practical usefulness of the metric, but they do not directly establish that high-RKL-variance positions also have higher signal-to-noise ratios or larger one-step gains in the actual parameter space. This limits the strength of the current explanation of the weighting mechanism.
Questions for authors
-
Which objective does the paper’s “on-policy gradient” correspond to?
Does it refer to the conditional KL gradient at a fixed prefix, averaged over the current policy’s prefix distribution, or to the derivative of the full objective that includes the prefix visitation distribution?
Please clarify whether the prefix distribution is treated as fixed during an update and whether REVO and the OPD baseline use the same convention. If the method uses a semi-gradient update that does not differentiate through the prefix distribution, I suggest stating this explicitly and distinguishing the exactness of current-token resampling from the approximation in prefix correction.
-
How does using the same tokens to estimate both variance and gradients affect the actual update?
Algorithm 1 appears to use the same samples for both estimates; please confirm whether this is also the case in the implementation. If so, positions assigned higher weights may also be those that happen to exhibit larger fluctuations in the current sample. Thresholding and within-batch normalization may further affect this dependence.
How do the authors interpret this effect: is it an intended emphasis on stronger learning signals observed in the current sample, or a side effect of finite-sample estimation? Could you compare shared-sample and independent-sample estimation of weights and gradients while controlling sampling cost, or use more accurate variance estimates in a small-scale diagnostic to assess the effect of this dependence?
-
Does high RKL variance still correspond to more effective updates in the actual parameter space?
On a small set of representative prefixes, could you measure the gradient signal-to-noise ratio for the actual trainable parameters, or compare the KL reduction from a single update using the same small step size, and examine its relationship with RKL variance?
I would particularly like to understand which prefixes exhibit large teacher–student distribution differences but only small corrections that the student can realize. This would help determine whether high variance mainly identifies worthwhile learning opportunities or sometimes amplifies teacher differences that the student cannot currently absorb, thereby clarifying the metric’s scope of applicability.
3 comments
Response to Reviewer xUua (2/2)
W2 & Q3. Does RKL variance remain informative in parameter space?
We thank the reviewer for this insightful question. To directly address this concern, we extend the Figure 4 analysis to the full parameter space of Qwen3-1.7B and examine actual one-step KL reduction, including cases with substantial teacher–student support mismatch.
Experimental setup. We evaluate 256 prefixes from 64 training prompts, sampled at four relative response positions using Qwen3-1.7B-Base as the student and Qwen3-4B as the teacher.
1. Parameter-space gradient reliability. For each prefix , we compute the exact parameter gradient through vector-Jacobian products. We estimate the gradient-estimator variance using vocabulary-level moments, with exact contributions from high-probability tokens and unbiased importance sampling for the remaining tail.
We measure the SNR of the gradient estimator as
The Spearman correlation between RKL variance and parameter-space gradient SNR is , demonstrating that the strong association between RKL variance and gradient reliability persists in the actual parameter space.
2. Actual one-step KL reduction. We independently update each prefix using its exact parameter gradient with the same learning rate , then measure the resulting conditional KL reduction.
| RKL variance quartile | Median relative KL reduction |
|---|---|
| Q1 (lowest) | |
| Q2 | |
| Q3 | |
| Q4 (highest) |
The Spearman correlation between and absolute KL reduction is , showing that higher-variance prefixes are strongly associated with larger achievable KL reductions under individual parameter updates.
3. Prefixes with substantial teacher–student support mismatch. We further examine local optimization progress at prefixes with substantial support mismatch. We define support mismatch as the teacher probability mass outside the student's 99% nucleus. Among the 256 prefixes, 10 exhibit substantial mismatch (; median 0.22), accounting for 16% of the total conditional KL. These prefixes typically have confident student predictions (median ), while the teacher assigns substantial probability to tokens rarely sampled by the student.
To isolate optimization progress from gradient magnitude, we perform fixed-norm updates along each prefix's exact KL gradient, with over all trainable parameters. These updates achieve a median relative KL reduction of 9% for high-mismatch prefixes, compared with 47% for the remaining prefixes.
Furthermore, the median is 0.36 for high-mismatch prefixes, compared with 2.28 for lower-mismatch prefixes. This suggests that RKL variance places relatively less emphasis on large but difficult-to-correct discrepancies than KL magnitude alone. Consistent with this observation, RKL-variance weighting outperforms sampled-KL-based weighting in our training ablation (19.6 vs. 17.3 Avg@16).
4. Shared-parameter updates. We acknowledge that a complete theoretical analysis of shared-parameter updates, where gradients from different prefixes interact, remains future work. Nevertheless, using sample-level signals to guide prioritization in shared-parameter training is a common practice [1–3]. Following this practice, REVO uses per-prefix RKL variance to weight updates in batched training, and we empirically verify that this remains effective when gradients from different prefixes are combined. To this end, we apply the REVO estimator to all 256 prefixes, average their gradients, and perform a single gradient step without optimizer state.
The summed conditional KL decreases by 2.1% and 14.5% at step sizes and , respectively, showing a net KL reduction for the aggregated update. In full training, variance-guided weighting further improves Avg@16 from 18.2 to 19.6 (Table 3).
Overall, RKL variance is strongly associated with parameter-space gradient reliability and achievable KL reduction, while relatively de-emphasizing the rare high-mismatch prefixes (10 of 256) where local optimization progress is limited.
We have incorporated these analyses into Appendix C.8 of the revised manuscript.
References
[1] Katharopoulos, A., & Fleuret, F. (2018). Not All Samples Are Created Equal: Deep Learning with Importance Sampling. ICML 2018, 2525–2534.
[2] Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized Experience Replay. ICLR 2016.
[3] Mindermann, S., Brauner, J. M., Razzak, M. T., et al. (2022). Prioritized Training on Points that Are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022, 15630–15649.
Thank you again for your detailed and constructive review. As we are about halfway through the discussion period, we would like to kindly follow up on our response.
We would like to first briefly summarize the main clarifications and additional analyses:
-
Gradient convention (Q1). REVO and all OPD baselines use conditional KL gradients at fixed student-generated prefixes, as now clarified in Section 3.
-
Shared-sample estimation (W1 & Q2). Weighting gains persist when variance and gradients are estimated from independent samples (19.2 for 8+8 and 19.8 for 16+16, versus 18.2 for uniform weighting and 19.6 for REVO). Thus, the gains do not require sample–weight dependence (Appendix C.6).
-
Parameter-space relevance (W2 & Q3). RKL variance strongly correlates with parameter-space gradient SNR () and absolute one-step KL reduction (). For prefixes with substantial teacher–student support mismatch, we observe smaller median relative KL reductions (9% vs. 47% under fixed-norm updates) and lower median (0.36 vs. 2.28). We also verify that aggregated shared-parameter updates reduce the summed conditional KL (Appendix C.8).
We would greatly appreciate it if you could let us know whether our response addresses your concerns, and we would be happy to clarify any remaining questions.
Response to Reviewer xUua (1/2)
Q1. Which objective does the on-policy gradient refer to?
We thank the reviewer for the question. We clarify that our formulation follows the standard sampled-token OPD convention. As defined in Section 3, gton denotes the expected conditional RKL gradient over student-generated prefixes, rather than the gradient of the full objective that differentiates through the prefix visitation distribution.
Both REVO and the OPD baselines follow the same convention, treating sampled prefixes as fixed during each learner update. Under this formulation, one-step resampling eliminates current-token distribution mismatch at each fixed prefix, while geometric prefix correction provides a stabilized approximation to the prefix importance ratio.
We have clarified this formulation in Section 3 of the revised manuscript.
W1 & Q2. How does shared-sample estimation affect updates?
We thank the reviewer for raising this question. We confirm that our implementation follows Algorithm 1, using the same K=16 samples for both variance and gradient estimation, followed by threshold-based weighting and batch normalization.
We agree that this introduces statistical dependence and may lead to a finite-sample selection effect, where positions with unusually large estimated variance receive higher weights. Our intention is to prioritize positions based on their underlying RKL variance, while sharing samples allows both estimators to leverage all K=16 candidates under a fixed sampling budget.
To examine this effect, we compare REVO with two independent-sample variants: Independent (8+8), using 8 samples for each estimator, and Independent (16+16), using 16 samples for each. All other configurations remain unchanged.
The weighting gains persist under independent estimation, demonstrating that they do not require sample–weight dependence. Under the same total sampling budget, independent estimation does not outperform REVO. With 16 independent samples per estimator, performance remains comparable.
These results support our choice of shared-sample estimation as an effective approach, allowing both estimators to leverage all available samples without increasing the sampling budget.
We have incorporated this analysis into Appendix C.6 of the revised manuscript.