acceptodds
Under review as a conference paper at ICLR 2027

REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse

Abstract

On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its reliance on frequently refreshed student rollouts often incurs substantial generation cost. We introduce REVO, an off-policy distillation framework that improves rollout efficiency by reusing each student rollout for multi-step learner updates. REVO addresses prefix-level and current-token policy mismatch through stabilized prefix weighting and one-step resampling from the current student, which enables repeated updates without regenerating full trajectories. To prioritize informative token positions within reused rollouts, REVO uses the variance of the student-teacher log-probability ratio to quantify the remaining token-level learning signal and guide repeated optimization. Across multiple student–teacher scales, REVO with only 50 rollout iterations matches or exceeds OPD baselines trained for 200 iterations on both in-domain and cross-domain reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Off-policy distillation, Knowledge distillation, Large language models, Rollout efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xUuaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to reuse trajectories already generated by the student to reduce the cost of repeatedly generating complete responses during on-policy distillation. The authors propose REVO, which separates distribution shift during trajectory reuse into two components: the difference between stored prefixes and those visited by the current policy, and the difference between previously sampled tokens and the current student distribution at a fixed prefix.

For the latter, REVO resamples next-token candidates from the current student distribution at each update; for the former, it stabilizes training using geometrically normalized and capped prefix weights. The method additionally uses the variance of the student–teacher log-probability ratio, termed RKL variance, to assign two levels of training weights to token positions, prioritizing positions with stronger remaining supervision signals.

The experiments show that REVO, with 50 rollout iterations and multiple parameter updates per iteration, matches or exceeds OPD baselines trained for 200 rollout iterations. The authors also provide cross-domain evaluations, training-time comparisons, and component ablations, showing that the gains do not simply come from increasing the number of parameter updates.

Critical strengths
  1. The paper clearly distinguishes two sources of distribution shift during trajectory reuse and provides practical ways to address them.

    Resampling the current token at a fixed prefix provides supervision under the current student distribution without requiring the entire trajectory to be regenerated. For prefix mismatch, which cannot be eliminated in the same way, the authors use stabilized weights and explicitly acknowledge their approximate nature. This design takes advantage of the distillation setting and makes both the computational costs and the sources of approximation relatively easy to understand.

  2. The efficiency gains are supported by substantial experimental evidence and cannot be explained simply by additional updates.

    Across multiple student–teacher scales, REVO achieves better average performance with 50 rollout iterations than OPD with 200 iterations. The paper also includes an OPD(U10) baseline that performs the same ten updates per rollout iteration but performs substantially worse than REVO, indicating that the improvements cannot be attributed solely to more parameter updates.

    Furthermore, the appendix reports that, with a Qwen3-1.7B student and Qwen3-4B teacher, 50 REVO iterations take approximately 5.5 hours, whereas 200 standard OPD iterations take approximately 10 hours. This shows that the benefits extend beyond rollout counts to actual training time. Component ablations and comparisons with alternative prioritization signals further support the method’s design.

Critical weaknesses
  1. The objective underlying gradient alignment and the relationship between stochastic weighting and the theoretical weights need to be stated more precisely.

    Section 3 first defines the reverse-KL gradient at a fixed prefix and then averages it over prefixes visited by the current policy. This is not necessarily equivalent to directly differentiating the overall objective defined as the average KL over the current policy’s prefix distribution: the latter also involves derivatives arising from the dependence of that distribution on the parameters. One-step resampling exactly handles the current-token distribution at a fixed prefix, but it does not automatically resolve this distinction between objectives. Treating prefixes as fixed data is a reasonable training convention, but the paper should clarify whether REVO and the OPD baseline follow exactly the same convention.

    In addition, according to Algorithm 1, the same resampled tokens are used to estimate both RKL variance and the gradient, followed by threshold-based weighting and within-batch normalization. Proposition 4.1, however, analyzes a local descent bound under deterministic weights. Stopping gradients through the weights only prevents differentiation through them; it does not remove the statistical dependence between the estimated weights and gradients. A gap therefore remains between the practical update and the deterministic weighting analyzed theoretically. It is reasonable to present the two-level rule as a theoretically motivated practical design, but the paper should also explain whether this dependence introduces an additional selection effect.

  2. Direct evidence connecting RKL variance to effective updates realizable by the actual model remains limited.

    The authors explicitly distinguish RKL variance from gradient-estimation noise and report a strong correlation between RKL variance and logit-space gradient signal-to-noise ratio in Figure 4. The appendix also discusses the Fisher–Jacobian projection, showing that the parameter-space relationship additionally depends on the model’s local structure. My concern is therefore not that the paper confuses these two variances or ignores the effects of parameterization.

    Rather, a large difference between teacher and student output distributions does not necessarily mean that the current student can effectively reduce that difference through parameter updates. Some correction directions may not be locally realizable by the current parameters, and updates from different positions may conflict when combined. The high- versus low-variance training comparison in Figure 3 and the alternative-weighting ablations support the practical usefulness of the metric, but they do not directly establish that high-RKL-variance positions also have higher signal-to-noise ratios or larger one-step gains in the actual parameter space. This limits the strength of the current explanation of the weighting mechanism.

Questions for authors
  1. Which objective does the paper’s “on-policy gradient” correspond to?

    Does it refer to the conditional KL gradient at a fixed prefix, averaged over the current policy’s prefix distribution, or to the derivative of the full objective that includes the prefix visitation distribution?

    Please clarify whether the prefix distribution is treated as fixed during an update and whether REVO and the OPD baseline use the same convention. If the method uses a semi-gradient update that does not differentiate through the prefix distribution, I suggest stating this explicitly and distinguishing the exactness of current-token resampling from the approximation in prefix correction.

  2. How does using the same tokens to estimate both variance and gradients affect the actual update?

    Algorithm 1 appears to use the same samples for both estimates; please confirm whether this is also the case in the implementation. If so, positions assigned higher weights may also be those that happen to exhibit larger fluctuations in the current sample. Thresholding and within-batch normalization may further affect this dependence.

    How do the authors interpret this effect: is it an intended emphasis on stronger learning signals observed in the current sample, or a side effect of finite-sample estimation? Could you compare shared-sample and independent-sample estimation of weights and gradients while controlling sampling cost, or use more accurate variance estimates in a small-scale diagnostic to assess the effect of this dependence?

  3. Does high RKL variance still correspond to more effective updates in the actual parameter space?

    On a small set of representative prefixes, could you measure the gradient signal-to-noise ratio for the actual trainable parameters, or compare the KL reduction from a single update using the same small step size, and examine its relationship with RKL variance?

    I would particularly like to understand which prefixes exhibit large teacher–student distribution differences but only small corrections that the student can realize. This would help determine whether high variance mainly identifies worthwhile learning opportunities or sometimes amplifies teacher differences that the student cannot currently absorb, thereby clarifying the metric’s scope of applicability.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xUua (1/2)

Q1. Which objective does the on-policy gradient refer to?

We thank the reviewer for the question. We clarify that our formulation follows the standard sampled-token OPD convention. As defined in Section 3, gtong_t^{\mathrm{on}} denotes the expected conditional RKL gradient over student-generated prefixes, rather than the gradient of the full objective that differentiates through the prefix visitation distribution.

Both REVO and the OPD baselines follow the same convention, treating sampled prefixes as fixed during each learner update. Under this formulation, one-step resampling eliminates current-token distribution mismatch at each fixed prefix, while geometric prefix correction provides a stabilized approximation to the prefix importance ratio.

We have clarified this formulation in Section 3 of the revised manuscript.


W1 & Q2. How does shared-sample estimation affect updates?

We thank the reviewer for raising this question. We confirm that our implementation follows Algorithm 1, using the same K=16K=16 samples for both variance and gradient estimation, followed by threshold-based weighting and batch normalization.

We agree that this introduces statistical dependence and may lead to a finite-sample selection effect, where positions with unusually large estimated variance receive higher weights. Our intention is to prioritize positions based on their underlying RKL variance, while sharing samples allows both estimators to leverage all K=16K=16 candidates under a fixed sampling budget.

To examine this effect, we compare REVO with two independent-sample variants: Independent (8+8), using 8 samples for each estimator, and Independent (16+16), using 16 samples for each. All other configurations remain unchanged.

MethodResampling KKAvg@16Gain vs. Uniform (pp)
Uniform weighting1618.2—
Independent (8+8)1619.2+1.0
REVO (shared samples)1619.6+1.4
Independent (16+16)3219.8+1.6

The weighting gains persist under independent estimation, demonstrating that they do not require sample–weight dependence. Under the same total sampling budget, independent estimation does not outperform REVO. With 16 independent samples per estimator, performance remains comparable.

These results support our choice of shared-sample estimation as an effective approach, allowing both estimators to leverage all available samples without increasing the sampling budget.

We have incorporated this analysis into Appendix C.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer xUua (2/2)

W2 & Q3. Does RKL variance remain informative in parameter space?

We thank the reviewer for this insightful question. To directly address this concern, we extend the Figure 4 analysis to the full parameter space of Qwen3-1.7B and examine actual one-step KL reduction, including cases with substantial teacher–student support mismatch.

Experimental setup. We evaluate 256 prefixes from 64 training prompts, sampled at four relative response positions using Qwen3-1.7B-Base as the student and Qwen3-4B as the teacher.

1. Parameter-space gradient reliability. For each prefix ctc_t, we compute the exact parameter gradient gt=∇θDKL(Pθ(⋅∣ct)∥PT(⋅∣ct))g_t=\nabla_\theta D_{\mathrm{KL}} (P_\theta(\cdot\mid c_t)\|P_T(\cdot\mid c_t)) through vector-Jacobian products. We estimate the gradient-estimator variance using vocabulary-level moments, with exact contributions from high-probability tokens and unbiased importance sampling for the remaining tail.

We measure the SNR of the K=16K=16 gradient estimator as

Γtθ=∥gt∥22E∥g^t−gt∥22.\Gamma_t^\theta=\frac{\|g_t\|_2^2}{\mathbb{E}\|\hat g_t-g_t\|_2^2}.

The Spearman correlation between RKL variance and parameter-space gradient SNR is ρ=0.930\rho=0.930, demonstrating that the strong association between RKL variance and gradient reliability persists in the actual parameter space.

2. Actual one-step KL reduction. We independently update each prefix using its exact parameter gradient with the same learning rate η=10−5\eta=10^{-5}, then measure the resulting conditional KL reduction.

RKL variance quartileMedian relative KL reduction
Q1 (lowest)<0.01%<0.01\%
Q20.09%0.09\%
Q310.0%10.0\%
Q4 (highest)44.7%44.7\%

The Spearman correlation between EtE_t and absolute KL reduction is ρ=0.97\rho=0.97, showing that higher-variance prefixes are strongly associated with larger achievable KL reductions under individual parameter updates.

3. Prefixes with substantial teacher–student support mismatch. We further examine local optimization progress at prefixes with substantial support mismatch. We define support mismatch mtm_t as the teacher probability mass outside the student's 99% nucleus. Among the 256 prefixes, 10 exhibit substantial mismatch (mt≥0.1m_t\geq0.1; median 0.22), accounting for 16% of the total conditional KL. These prefixes typically have confident student predictions (median pmax⁡=0.98p_{\max}=0.98), while the teacher assigns substantial probability to tokens rarely sampled by the student.

To isolate optimization progress from gradient magnitude, we perform fixed-norm updates along each prefix's exact KL gradient, with ∥Δθ∥2=10−3\|\Delta\theta\|_2=10^{-3} over all trainable parameters. These updates achieve a median relative KL reduction of 9% for high-mismatch prefixes, compared with 47% for the remaining prefixes.

Furthermore, the median Et/KLtE_t/\mathrm{KL}_t is 0.36 for high-mismatch prefixes, compared with 2.28 for lower-mismatch prefixes. This suggests that RKL variance places relatively less emphasis on large but difficult-to-correct discrepancies than KL magnitude alone. Consistent with this observation, RKL-variance weighting outperforms sampled-KL-based weighting in our training ablation (19.6 vs. 17.3 Avg@16).

4. Shared-parameter updates. We acknowledge that a complete theoretical analysis of shared-parameter updates, where gradients from different prefixes interact, remains future work. Nevertheless, using sample-level signals to guide prioritization in shared-parameter training is a common practice [1–3]. Following this practice, REVO uses per-prefix RKL variance to weight updates in batched training, and we empirically verify that this remains effective when gradients from different prefixes are combined. To this end, we apply the REVO estimator to all 256 prefixes, average their gradients, and perform a single gradient step without optimizer state.

The summed conditional KL decreases by 2.1% and 14.5% at step sizes η=10−5\eta=10^{-5} and η=10−4\eta=10^{-4}, respectively, showing a net KL reduction for the aggregated update. In full training, variance-guided weighting further improves Avg@16 from 18.2 to 19.6 (Table 3).

Overall, RKL variance is strongly associated with parameter-space gradient reliability and achievable KL reduction, while relatively de-emphasizing the rare high-mismatch prefixes (10 of 256) where local optimization progress is limited.

We have incorporated these analyses into Appendix C.8 of the revised manuscript.


References

[1] Katharopoulos, A., & Fleuret, F. (2018). Not All Samples Are Created Equal: Deep Learning with Importance Sampling. ICML 2018, 2525–2534.

[2] Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized Experience Replay. ICLR 2016.

[3] Mindermann, S., Brauner, J. M., Razzak, M. T., et al. (2022). Prioritized Training on Points that Are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022, 15630–15649.

Thank you again for your detailed and constructive review. As we are about halfway through the discussion period, we would like to kindly follow up on our response.

We would like to first briefly summarize the main clarifications and additional analyses:

  • Gradient convention (Q1). REVO and all OPD baselines use conditional KL gradients at fixed student-generated prefixes, as now clarified in Section 3.

  • Shared-sample estimation (W1 & Q2). Weighting gains persist when variance and gradients are estimated from independent samples (19.2 for 8+8 and 19.8 for 16+16, versus 18.2 for uniform weighting and 19.6 for REVO). Thus, the gains do not require sample–weight dependence (Appendix C.6).

  • Parameter-space relevance (W2 & Q3). RKL variance strongly correlates with parameter-space gradient SNR (ρ=0.93\rho=0.93) and absolute one-step KL reduction (ρ=0.97\rho=0.97). For prefixes with substantial teacher–student support mismatch, we observe smaller median relative KL reductions (9% vs. 47% under fixed-norm updates) and lower median Et/KLtE_t/\mathrm{KL}_t (0.36 vs. 2.28). We also verify that aggregated shared-parameter updates reduce the summed conditional KL (Appendix C.8).

We would greatly appreciate it if you could let us know whether our response addresses your concerns, and we would be happy to clarify any remaining questions.

Reviewer_DsPvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

REVO proposes a method for reducing the cost of student rollouts typically used in on-policy distillation by reusing student rollouts for a batch (i.e. multiple times) which makes the method off-policy. The claim is that an off-policy-esque method can approach or beat on-policy distillation gains using the proposed setup. In order to learn better from the existing rollouts, a variance metric is used to select which tokens to use from any particular rollout. As the student gets multiple updates before the rollouts are generated again, the rollouts get stale (or further away from the policy), making this method a gradual switch from on-policy to off-policy within each rollout batch. The aim is to preserve the benefit of dense token-level supervision from the on-policy setup, but save on the generation cost of the student rollouts. To incorporate the current student's state into the learning process, the tokens are resampled for a fixed prefix and used for grading instead so that the learning uses the current student, not a stale one. The innovation is in how often the rollouts are generated (only once per many student updates), how the tokens are sampled from the current student state (so that learning stays fresh) and the fact that only those tokens that are going to be graded next by the teacher are sampled. The divergence between the student and the teacher is used as a way to decide which tokens to reuse so as to maximize the learning gain from a small generation cost. The results show that similar or better accuracy can be reached with 4x fewer student rollouts.

Critical strengths

The cost reduction is real - full generations are replaced by targeted token sampling for only the portion that is about to be graded

The method achieves faster learning given the number of rollouts

The results are decent in terms of improvements (quality) but the main point of the paper is cost saving (so comparison of quality with OPD at 4x more rollouts is good)

Critical weaknesses

The paper is difficult to follow - communicating the method is interleaved with distinguishing the work from other works which breaks the flow, supporting justifications and comparisons that are not necessary for the core argument are also interleaved.

It is not clear why the student rollout cost is a concern to an extent that justifies the complex inner loop/outer loop setup

"no generation" is not technically true, tokens are sampled from the current student for a known prefix

Questions for authors

Is there an opportunity to save some of the teacher outputs too?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DsPv

W1. Comparisons and justifications in the method description

We thank the reviewer for the feedback. We agree that a clear and coherent presentation of the methodology is important, and would like to clarify the organization and motivation of Section 4.

Section 4 follows the two main components of REVO. Section 4.1 first distinguishes prefix mismatch from current-token mismatch, then introduces one-step resampling and stabilized prefix correction to address them. Section 4.2 introduces RKL variance to quantify the remaining distillation signal, examines its relationship with gradient reliability, and develops the variance-guided weighting rule.

The brief comparisons with existing methods are intended to clarify the motivation for and distinctions between specific design choices, while broader discussions of related work are provided in Section 2. Similarly, the supporting theoretical and empirical analyses provide the rationale for these design choices and help explain their roles in effective rollout reuse.

For a concise, step-by-step overview of the complete training procedure, we also point to Algorithm 1 (Appendix B.4), which is referenced in Section 4.2.


W2 & W3. Why does rollout cost matter, and what does "no generation" mean?

We thank the reviewer for raising this point. Autoregressive rollout generation is a major computational bottleneck in on-policy LLM post-training, accounting for 66–72% of total training time in RLVR [1]. This bottleneck also arises in OPD, where student rollout generation consumes the majority of training time [2].

The key computational distinction lies in sequential autoregressive decoding versus parallel forward evaluation [3]. Generating a trajectory of length TT requires TT sequential decoding steps, whereas evaluating the next-token distributions at all positions of an existing trajectory can be performed in parallel through a single forward pass. Regarding "no generation", REVO avoids additional autoregressive rollout generation (Section 4.1), while next-token candidates are sampled from the current student at stored prefixes, as the reviewer notes.

The inner/outer-loop structure refreshes OPD rollouts every NN gradient updates: the outer loop collects a rollout batch, and the inner loop performs NN updates on the stored prefixes without additional autoregressive generation.

In our profiling (Appendix C.5, Table 10), rollout generation accounts for about 70% of standard OPD runtime. In REVO, generating one rollout batch takes 128.7 seconds, whereas the nine additional student log-probability evaluations and resampling signal constructions take only 52.1 and 14.2 seconds. In the 1.7B-to-4B setting, REVO thus reduces training time from 9.9 to 5.5 hours (Table 9) while improving the six-benchmark mean from 27.3 to 29.6 (Table 1), using 50 instead of 200 rollout iterations. In the revision, we further report end-to-end GPU-hours across all three scales (Appendix C.5, Table 9), where REVO achieves 1.751.75–2.09×2.09\times speedups over standard OPD with higher accuracy.


Q1. Is there an opportunity to save some of the teacher outputs too?

We thank the reviewer for this suggestion. We first clarify that OPD does not involve teacher-generated rollouts. Instead, rollouts are generated by the student, while the teacher only evaluates next-token distributions at student-generated prefixes through parallel forward passes.

If the reviewer refers to caching teacher outputs at these prefixes, this is indeed possible since the teacher and stored prefixes remain fixed across learner updates. However, caching the teacher distributions for an entire rollout batch incurs substantial memory overhead. For Qwen3 (vocabulary size 151,936), caching BF16 logits for 32 trajectories at the maximum sequence length of 8,192 tokens requires approximately 79.7 GB, or 19.9 GB per GPU even with ideal sharding across four GPUs. Our measured peak reserved GPU memory is already 72.1 GB per 80-GB A100, making GPU-resident caching impractical without additional memory management.

Moreover, the nine additional teacher scoring passes account for only 8.5% of REVO's per-step runtime (Appendix C.5, Table 10). We agree that teacher-output caching could further reduce this overhead through engineering optimizations, but this is orthogonal to our primary focus on reducing the number of autoregressive student rollouts.


References

[1] Gao, W., Zhao, Y., An, D., et al. (2026). RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching. NSDI 2026.

[2] Zhou, Z., Li, J., Tang, H., Wu, Y. N., & Terzopoulos, D. (2026). Less Is More: Early Stopping Rollout for On-Policy Distillation. arXiv:2605.27028.

[3] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., ... & Stoica, I. (2023, October). Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles (pp. 611-626).

Thank you again for your review. As we are about halfway through the discussion period, we would like to kindly follow up on our response.

We would like to first briefly summarize the main clarifications and additional analyses:

  • Presentation (W1). Section 4 follows REVO's two components: Section 4.1 stabilizes off-policy updates, and Section 4.2 develops variance-guided weighting. Algorithm 1 (Appendix B.4) provides a step-by-step overview of the full procedure.
  • Rollout cost and "no generation" (W2 & W3). Rollout generation accounts for about 70% of standard OPD runtime. REVO avoids additional autoregressive rollouts and only samples next-token candidates at stored prefixes. In the 1.7B-to-4B setting, this reduces training time from 9.9 to 5.5 hours while improving the six-benchmark mean from 27.3 to 29.6, and REVO achieves 1.75–2.09× GPU-hour speedups across all three scales.
  • Teacher outputs (Q1). OPD involves no teacher rollouts. Caching teacher distributions is possible but incurs substantial GPU memory overhead, while the additional teacher scoring passes account for only 8.5% of REVO's per-step runtime.

We would be happy to address any remaining or new questions. If our response has resolved your concerns, we would be grateful if you could consider updating your assessment.

Thank you for the response and the additional profiling data.

The runtime breakdown and GPU-hour speedups clearly answer my question on the rollout computational bottleneck and motivate the generation (outer-loop) followed by scoring and sampling (inner-loop) design. Your memory and runtime analysis for caching teacher scoring distributions (79.7 GB for 8.5% runtime saving) settles my question on teacher outputs. On presentation, as no changes are proposed to the main text, my experience as a reader stays the same. My overall assessment remains unchanged, as the new information supports the computational motivation, but neither improves the clarity of the main text nor changes my view of the contribution.

Dear Reviewer DsPv,

Thank you for your follow-up. We appreciate your confirmation that we have addressed all your technical concern including the bottleneck of student rollout and teacher caching.

Regarding your assessment of the contribution, we appreciate if you could clarify which specific limitation remains beyond the presentation concern. Our understanding is that the contribution-related concern has been adequately addressed in our revision as we mentioned earlier. We believe our contribution is still sufficient centering around enabling effective rollout reuse in OPD with off-policy stabilization and variance-guided token prioritization and receives a sufficient (4x) rollout efficiency experimentally.

We would appreciate that clarification on the comments on contribution and thank you again for your feedback.

Best regards,

Authors.

Reviewer_BXaROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

REVO improves the rollout efficiency of reverse-KL language-model distillation by repeatedly updating on stored student trajectories. It resamples current-policy tokens at fixed prefixes, applies a capped geometric prefix correction, and prioritizes token positions using the estimated variance of student–teacher log-probability ratios. Experiments cover three Qwen3 student–teacher scales, cross-domain evaluations, additional model families, and component and runtime analyses.

Critical strengths
  1. The method makes a useful distinction between prefix mismatch and current-token mismatch. Resampling addresses the latter without regenerating trajectories, while the paper explicitly identifies the geometric prefix weight as a biased stabilization surrogate rather than an exact importance correction.

  2. The empirical gains have meaningful controls. At 1.7B, Table 1 reports a six-benchmark mean of 29.6 after 50 rollout iterations, versus 27.3 for OPD after 200 and 25.3 for OPD(U10) after 50. Table 3 separately tests token correction, prefix correction, and variance weighting; Table 8 compares prioritization signals at approximately matched high-weight fractions.

  3. The paper goes beyond counting rollout iterations. It reports approximately 5.5 versus 10 hours for REVO and OPD in one setting, includes larger-batch OPD controls, and evaluates cross-domain transfer and Llama/Gemma models. These results support practical value beyond a single math benchmark.

Critical weaknesses
  1. The comparison with closely related stale-data distillation estimators is incomplete. AsyncOPD already studies current-student reverse-KL signals and multi-sample Monte Carlo under staleness. Table 3 compares against clipped importance sampling, but not an unclipped, current-signal multi-sample estimator under a matched teacher-access budget. This leaves the incremental value of resampling over a stronger corrected-reuse baseline unclear.

  2. The theory motivates the weighting rule but does not establish its behavior in the implemented optimizer. Proposition 4.1 assumes deterministic scaling of a local logit-gradient estimator; REVO estimates priorities from the same samples as the gradient and normalizes them across the batch. Proposition 4.2 additionally requires local realizability. The appendix acknowledges the logit/parameter-space distinction, but sample–weight dependence and shared-parameter interactions remain untested.

  3. The efficiency evidence is encouraging but limited in scope. Runtime is profiled for one student–teacher pair against standard OPD, rather than through accuracy-versus-GPU-hour curves for competitive corrected-reuse methods. Fifty U10 iterations involve 500 learner updates, so fourfold fewer rollouts should not be interpreted as fourfold less total computation; the reported wall-clock reduction is about 1.8-fold.

Questions for authors
  1. How does REVO compare with an unclipped current-signal importance-sampling baseline using multiple samples, with matched rollout, learner-update, and teacher-access budgets?

  2. Does the variance-weighting gain persist if priorities are estimated from samples independent of the gradient samples? This would test the role of sample–weight dependence.

  3. Across reuse updates and prefix lengths, how much does the geometric correction depart from the exact prefix ratio, and when does this affect performance?

  4. Can you report accuracy versus GPU-hours for REVO and a competitive corrected-reuse baseline, including teacher scoring, and clarify whether the measured runtime advantage extends to the other model scales?

5 comments
AuthorsOpenReview ↗

Response to Reviewer BXaR (1/3)

We thank the reviewer for the careful review. We address the questions and concerns below.

Unless otherwise specified, all additional experiments use Qwen3-1.7B-Base as the student and Qwen3-4B as the teacher, following the experimental configurations in Appendix C.1 (Table 5). Rollout-reuse ablations use 50 rollout iterations with 10 learner updates per iteration (U10). All reported Avg@16 results are averaged over AIME24, AIME25, and AMC23.

W1 & Q1. How does REVO compare with AsyncOPD?

We first clarify a key methodological distinction: AsyncOPD's multi-sample estimator uses behavior-policy samples with importance weighting to estimate current-policy gradients. In contrast, REVO resamples tokens directly from the current student PθP_\theta at every learner update, eliminating the need for current-token IS while separately addressing prefix mismatch through stabilized weighting.

Importantly, AsyncOPD primarily studies policy staleness caused by asynchronous rollout consumption, whereas REVO targets repeated optimization on the same student-generated trajectories. Although both involve off-policy distribution shift, repeated reuse introduces additional sample–parameter dependence when the same finite set of actions is repeatedly optimized, potentially leading to over-optimization. The unbiasedness of IS for a fixed policy does not automatically eliminate this dependence.

Related studies in LLM reinforcement learning have examined substantial rollout staleness [1] and multi-epoch rollout reuse [2], with the latter highlighting the risk of overfitting from excessive reuse. To our knowledge, systematic multi-update rollout reuse remains relatively underexplored in reverse-KL OPD.

Beyond addressing policy mismatch, REVO further introduces variance-guided prioritization for repeated rollout reuse. As the student evolves, the remaining distillation signal varies across token positions. REVO estimates RKL variance at each learner update to prioritize positions with stronger and more reliable learning signals.

To directly address the reviewer’s concern, we implement an AsyncOPD-style multi-sample estimator using K=16K=16 behavior-policy samples, current-policy RKL signals, and unclipped IS, under matched rollout and learner-update budgets and matched teacher forward passes.

MethodAvg@16
OPD15.1
OPD(U10)15.0
OPD(U10) + PPO-style clipped IS16.7
OPD(U10) + Multi-sample unclipped IS (K=16K=16)17.1
OPD(U10) + One-step resampling (K=16K=16)17.8
Full REVO19.6

The multi-sample unclipped IS baseline achieves 17.1, compared with 17.8 for one-step resampling. With all components combined, REVO achieves 19.6, supporting the effectiveness of current-policy resampling and variance-guided prioritization under repeated rollout reuse.

We have incorporated this comparison into Section 5.3 (Table 3), with implementation details provided in Appendix C.1.

References

[1] Tian, M., Xie, Y., & Wei, C. (2026). How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning. arXiv:2605.17570.

[2] Li, J., Zhou, P., Meng, R., Vadera, M. P., Li, L., & Li, Y. (2026). Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs. Findings of EACL 2026, 6227–6243.

AuthorsOpenReview ↗

Response to Reviewer BXaR (2/3)

W2 & Q2. How do the theoretical assumptions and sample–weight dependence affect practical weighting?

1. Theoretical assumptions and parameter-space relevance.

Proposition 4.1 provides an oracle motivation for variance-guided weighting, rather than a guarantee for the implemented optimizer. To examine whether its logit-space motivation extends to the actual parameter space, we evaluate 256 representative prefixes using Qwen3-1.7B-Base and all its 1.7B trainable parameters. We find that RKL variance is strongly correlated with parameter-space gradient SNR (Spearman ρ=0.930\rho=0.930), consistent with the logit-space observation in Figure 4.

Regarding Proposition 4.2, local realizability is required only for the equality, while the general bound gt⊤Ft†gt≤Etg_t^\top F_t^\dagger g_t \le E_t holds without this condition. We have revised Proposition 4.2 to state this general bound, with local realizability as the equality condition (proof in Appendix A.1). Importantly, REVO's practical variance-guided weighting does not depend on local realizability.

2. Sample–weight dependence.

We agree that using the same samples for variance and gradient estimation introduces statistical dependence. Our motivation for shared-sample estimation is that under a fixed sampling budget, independent estimation reduces the number of samples available to each estimator. Sharing samples allows both estimators to leverage all candidates.

To examine this effect, we evaluate two independent-sample variants: Independent (8+8), using 8 samples for each estimator, and Independent (16+16), using 16 samples for each.

MethodResampling KKAvg@16
Uniform weighting1618.2
Independent (8+8)1619.2
REVO (shared samples)1619.6
Independent (16+16)3219.8

The weighting gains persist under independent estimation, demonstrating that they do not require sample–weight dependence. Under the same total sampling budget, independent estimation does not outperform REVO. With 16 independent samples per estimator, performance remains comparable.

These results support our choice of shared-sample estimation as an effective approach, allowing both estimators to leverage all available samples without increasing the sampling budget.

3. Shared-parameter updates.

We acknowledge that a complete theoretical analysis of shared-parameter updates, where gradients from different prefixes interact, remains future work. Nevertheless, using sample-level signals to guide prioritization in shared-parameter training is a common practice [3–5]. Following this practice, REVO uses per-prefix RKL variance to weight updates in batched training, and we empirically verify that this remains effective when gradients from different prefixes are combined.

To examine shared-parameter interactions, we apply the REVO estimator to 256 prefixes, aggregate their gradients, and perform a single gradient step.

The summed conditional KL decreases by 2.1% and 14.5% at step sizes η=10−5\eta=10^{-5} and η=10−4\eta=10^{-4}, respectively, showing a net KL reduction for the aggregated update.

The effectiveness of variance-guided weighting in full training is further supported by Table 3, where it improves Avg@16 from 18.2 to 19.6.

These additional analyses are included in Appendix C.6 and C.8 of the revised manuscript.

References

[3] Katharopoulos, A., & Fleuret, F. (2018). Not All Samples Are Created Equal: Deep Learning with Importance Sampling. ICML 2018, 2525–2534.

[4] Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized Experience Replay. ICLR 2016.

[5] Mindermann, S., Brauner, J. M., Razzak, M. T., et al. (2022). Prioritized Training on Points that Are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022, 15630–15649.

AuthorsOpenReview ↗

Response to Reviewer BXaR (3/3)

W3 & Q4. How does REVO compare in GPU-hours across scales?

We emphasize that REVO primarily targets rollout efficiency, aiming to extract more learning from each expensive autoregressive student rollout. While runtime also depends on system implementations and engineering optimizations, we provide end-to-end GPU-hour comparisons to demonstrate the practical efficiency gains of REVO.

All experiments use 4×4\times A100 GPUs. REVO uses 50 rollout iterations with 10 learner updates per iteration, while standard OPD uses 200 iterations. We additionally include the multi-sample unclipped IS baseline from W1 & Q1. Both reuse methods use the same experimental setup.

Student / TeacherMethodGPU-hoursAvg@16
0.6B / 1.7BOPD37.68.8
REVO18.010.9
1.7B / 4BOPD39.616.8
Multi-sample unclipped IS (U10)21.217.1
REVO22.019.6
4B / 8BOPD53.230.5
REVO30.431.2

Across the three scales, REVO achieves 2.09×2.09\times, 1.80×1.80\times, and 1.75×1.75\times speedups over standard OPD, respectively, while achieving higher accuracy.

REVO also achieves higher accuracy than multi-sample unclipped IS with comparable GPU-hours. This demonstrates that current-policy resampling and variance-guided prioritization provide better optimization performance at comparable computational cost.

We have incorporated these GPU-hour comparisons and the accuracy–runtime analysis into Appendix C.5 (Table 9 and Figure 9).

Q3. How does geometric correction compare with exact IS?

Geometric prefix correction is an intentional stabilization rather than an exact importance correction. As motivated by prior work [6], cumulative importance ratios over long sequences can exhibit extreme or vanishing weights, while geometric normalization provides a more stable alternative. Unlike GSPO, which uses a sequence-level ratio, REVO applies geometric normalization only to the preceding prefix, since current-token mismatch is eliminated through one-step resampling.

1. Exact vs. geometric prefix ratios. We track the evolution of prefix weights over successive learner updates on the same stored rollout batch. At each update, we compute both exact cumulative IS and geometric prefix weights on the same stored prefixes. The table reports mean / median weights on a representative rollout batch with substantial policy drift.

Inner updatePrefix lengthExact IS (Mean / Median)Geometric (Mean / Median)
41–150.95 / 0.850.974 / 0.984
42k–4k190 / 1.01.000 / 1.000
91–150.76 / 0.0060.668 / 0.596
92k–4k0.27 / 2.5×10−62.5\times10^{-6}0.995 / 0.996

Exact cumulative IS becomes highly skewed under reuse. For 2k–4k prefixes, the mean weight reaches 190 with a median of 1.0 at update 4, and by update 9 the median drops to 2.5×10−62.5\times10^{-6} while the mean remains 0.27. Extreme weights thus concentrate updates on a few positions, while vanishing weights suppress learning signals from most stored prefixes in later updates. Geometric normalization reduces these extreme scales through length normalization and capping, keeping contributions stable across reuse updates.

2. Effect on performance. To isolate the effect of prefix correction, we follow Table 3 and compare three variants using one-step resampling without RKL-variance weighting. All other configurations remain unchanged.

Prefix correctionAvg@16
Exact cumulative prefix IS16.5
No prefix correction17.8
Geometric prefix correction18.2

Under our default U10 setting, where stored prefixes are reused across multiple updates, exact cumulative prefix IS underperforms even the uncorrected baseline, consistent with the weight distributions above, while geometric prefix correction achieves 18.2.

We have incorporated the weight-distribution analysis and performance comparison into Appendix C.9 of the revised manuscript.

References

[6] Zheng, C., Liu, S., Li, M., et al. (2025). Group Sequence Policy Optimization. arXiv:2507.18071.

Comments on the paper

2 comments

Dear Area Chair and Reviewers,

We thank all reviewers for their constructive feedback. We are encouraged that the reviewers find the distinction between prefix mismatch and current-token mismatch useful (BXaR, xUua) and the efficiency gains well supported by controlled experiments and wall-clock measurements (BXaR, xUua), and that DsPv notes REVO reduces generation cost while achieving faster learning per rollout.

We have uploaded a revised manuscript, with all revisions highlighted in blue. The main changes are summarized below.

New experiments and analyses.

  • Multi-sample unclipped IS baseline (BXaR): an AsyncOPD-style estimator with current-policy RKL signals under matched budgets (Section 5.3, Table 3; Appendix C.1).

  • End-to-end efficiency analysis (BXaR, DsPv): GPU-hour comparisons across model scales and with multi-sample unclipped IS, together with an accuracy–runtime plot, complementing the existing per-step runtime breakdown (Appendix C.5, Table 9, Figure 9).

  • Shared- vs. independent-sample estimation (BXaR, xUua): independent-sample variants for RKL variance and gradient estimation (Appendix C.6, Table 11).

  • Parameter-space analysis (BXaR, xUua): parameter-space gradient SNR, one-step KL reduction, prefixes with teacher–student support mismatch, and shared-parameter updates (Appendix C.8, Figure 10, Table 13).

  • Exact vs. geometric prefix correction (BXaR): weight distributions under repeated reuse and the corresponding training comparison (Appendix C.9, Figure 11, Table 14).

Clarifications.

  • Proposition 4.2 (BXaR): now states the general bound gt⊤Ft†gt≤Etg_t^\top F_t^\dagger g_t \le E_t, with local realizability as the equality condition (Section 4.2, Appendix A.1).

  • Gradient convention (xUua): Section 3 clarifies that REVO and all OPD baselines use conditional reverse-KL gradients at fixed student-generated prefixes.

  • Pointers to new analyses (BXaR, xUua): Section 4.2 adds pointers to the shared-sample and parameter-space analyses.

We address each reviewer's questions in our response to each review, and we welcome any further questions or suggestions.

AuthorsOpenReview ↗

Final Justification from Authors

We thank the reviewers and the Area Chair for their time and constructive feedback. As the author–reviewer discussion approaches its conclusion, we briefly summarize how the main concerns have been addressed in our revised manuscript.

Reviewer BXaR

  1. AsyncOPD comparison: An AsyncOPD-style multi-sample unclipped IS baseline achieves 17.1 Avg@16, compared with 19.6 for REVO (Section 5.3, Table 3).
  2. Theoretical assumptions: Proposition 4.2 now states the general bound gt⊤Ft†gt≤Etg_t^\top F_t^\dagger g_t \le E_t, with local realizability required only for equality (Appendix A.1). RKL variance strongly correlates with parameter-space gradient SNR (ρ=0.93\rho=0.93; Appendix C.8).
  3. Sample–weight dependence: Weighting gains persist when variance and gradients are estimated from independent samples (Appendix C.6, Table 11).
  4. GPU-hours: REVO achieves 1.75–2.09× end-to-end speedups over standard OPD across three scales with higher accuracy, while outperforming multi-sample unclipped IS at comparable GPU-hours (Appendix C.5, Table 9).
  5. Exact vs. geometric correction: Exact cumulative IS exhibits highly skewed weights under reuse and underperforms even no correction (16.5 vs. 17.8), while geometric correction achieves 18.2 (Appendix C.9).

Reviewer xUua

  1. Gradient convention: Section 3 clarifies that REVO and standard sampled-token OPD follow the same fixed-prefix conditional reverse-KL gradient convention.
  2. Shared-sample estimation: Independent-sample experiments demonstrate that the weighting gains do not require sample–weight dependence (Appendix C.6).
  3. Parameter-space relevance: RKL variance strongly correlates with parameter-space gradient SNR (ρ=0.93\rho=0.93) and one-step KL reduction (ρ=0.97\rho=0.97). We further analyze high-mismatch prefixes and shared-parameter updates (Appendix C.8).

Reviewer DsPv

  1. The reviewer confirmed that our runtime analysis addressed the question of the rollout computational bottleneck and resolved the question regarding teacher outputs.

Overall, REVO enables effective rollout reuse in OPD through one-step resampling, stabilized prefix correction, and variance-guided token prioritization. Across three model scales, it matches or outperforms OPD trained with 4× more rollout iterations while achieving 1.75–2.09× end-to-end speedups.

We thank the reviewers and the Area Chair again for their consideration. We would welcome any remaining questions or comments before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.