acceptodds
Under review as a conference paper at ICLR 2027

Effective-Supervision-Aware On-Policy Distillation

Abstract

On-policy distillation (OPD) provides token-level teacher supervision on student-generated trajectories. Yet these trajectories differ in answer correctness and in their agreement with teacher predictions, raising a practical question: how should training updates be allocated among them? We study effective-supervision-aware allocation in OPD: using outcome and teacher-based signals to prioritize trajectories for training without changing the distillation loss. As a concrete implementation, we introduce the Reliable Trajectory Buffer (RTB). RTB uses answer correctness and teacher-assigned likelihood to select current trajectories for replacement, teacher–student predictive compatibility to admit previously encountered mistakes to a replay buffer, and rescoring after replacement to refresh distillation signals. Across six mathematical reasoning benchmarks, with Qwen3-8B as teacher, RTB improves the aggregate score from 25.66 to 28.01 for Qwen3-1.7B-Base and from 35.68 to 38.03 for Qwen3-4B-Base. It also reaches higher scores earlier when measured by newly sampled student trajectories. Ablations support the roles of selective replacement and compatibility-based admission.

Primary area:
generative models
Keywords:
Large Language Models, On-Policy Distillation, Trajectory Selection, Selective Replay

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZczXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper attempts to answer the question of how does one really perform on-policy distillation when there could be scenarios where the teacher could potentially give higher mass (in terms of probability) to inherently incorrect responses (by the student). The paper tries to filter out candidate responses where this mismatch occurs and replaces them with other candidate responses.

Critical strengths

Here are some strengths of the paper:

  1. The paper attempts to solve an interesting relevant problem: how does one really perform on-policy distillation when there could be scenarios where the teacher could potentially give higher mass (in terms of probability) to inherently incorrect responses (by the student).

  2. Some figures in the paper (figure 3) provides a solid intuition of what the paper is attempting to do.

Critical weaknesses

Here are some weaknesses:

  1. The paper is littered with multiple writing issues, that make reading this paper extremely difficult. For example:

    • In equation 1, the term li,tKDl_{i, t}^{KD} is never defined. (I know what the authors want to say but it's not defined until equation 3, which again does not have the same explicit notation).
    • In equation 2, top−ktrtop-k_{tr} is not defined. (Again I know what that authors want to say, but this is not easy to digest in one reading)
    • It is not clear why predictive compatibility is defined (just after equation 3).
    • There are issues with references: in line numbers 146 and 148, there are question marks.
    • In table 2 and section 5.3 correct replacement and error replacement are referenced but they dont seem to be defined anywhere, and it's hard to understand what the authors want to convey.
    • The Δ\Delta in table 1, is not absolute percentage change but its relative. This makes it appear as if the gains are much higher than they actually are. I suggest that the authors go back to absolute percent gains or explicitly call out that these numbers are relative gains.
  2. The base model's performance (Qwen-3-1.7B-Base) reported by the authors in table 1, does not seem to match the performance in the Qwen 3 tech report (Table 19): ([1], link: https://arxiv.org/pdf/2505.09388). The results for AIME24 and AIME25 (in the report) are higher than the method proposed by the authors (and the base model performance reported by the authors is much lower). It's not clear why and is a cause for concern. This could be because the authors evaluated it on non-thinking mode or the evaluation lengths are different. However, this should be explicitly stated in the paper, and then should be reconciled with the setting in the tech report.

  3. Some critical baselines are missed:

    • RGOPD [2] is a paper that seems to exactly do what the authors are suggesting. There is no comparison to this work and hence the methods merits are unclear.
    • From what I understand the authors do seem to have access to the ground truth (line 255). Then clear baselines would be to compare against SFT and/or GRPO.
  4. The paper does not report any standard deviations. This is extremely concerning for the results on the datasets that the paper reports as some of them are about 30 problems (for example AIME). Hence it's not clear if the gains reported are actually statistically significant.

References

[1] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025)

[2] Akhondzadeh, Mohammad Sadegh, et al. "Reward-Gated On-Policy Distillation." arXiv preprint arXiv:2607.04037 (2026).

Questions for authors

Here are some questions:

  1. Please refer to weakness 2. With regards to this, can the authors explain why there is a discrepancy between performance of the base model reported by the authors and the performance stated in the tech report? Moreover, can it be made sure that instead of creating a method for a specific setting, is it possible to perform distillation in a setting that matches the initial performance that is stated in the tech report?

  2. Please refer to weakness 3. With regards to this, can the authors compare against the baselines suggested?

  3. Please refer to weakness 4. With regards to this, can the authors provide standard deviations?

  4. In table 5, why is the training setting (in terms of temperature) different from evaluation?

Note: In its current form, due to the various issues in terms of writing as well as other weaknesses raised I am recommending rejection. However, if the authors could provide evidence addressing the weaknesses raised, Im happy to revisit my decision.

4 comments

We thank the reviewer for the constructive comments. We have revised the manuscript to improve the clarity and consistency of the mathematical notation, methodological descriptions, and experimental presentation. We address the relevant comments below.

W1 : Mathematical Notation and Predictive Compatibility

Thank you for pointing out the notation issues and the unclear motivation for predictive compatibility. We have revised the Preliminaries section to explicitly define the mathematical symbols before their first use and maintain consistent notation throughout the formulation.

Specifically, we now define the student and teacher distributions, trajectory prefixes, and valid response lengths before introducing the distillation objective. We also explicitly define the truncated student and teacher distributions on the shared support before presenting the token-level reverse-KL loss. Furthermore, we clarify the distinction between $ k_{\mathrm{tr}} $ and $ k_{\mathrm{diag}} $, explaining their respective roles in the distillation objective and predictive compatibility measurement.

We have also clarified the motivation for predictive compatibility. Since the truncated reverse-KL objective is computed over the student's retained top-$ k $ support, higher overlap indicates that more of the teacher's high-probability token alternatives are represented within this support, making the teacher's local preferences more accessible to the distillation objective. Thus, predictive compatibility serves as a trajectory selection signal indicating the availability of locally exploitable teacher supervision, rather than a direct measure of trajectory quality or expected learning gains.

W2: Discrepancy with the Qwen3 Technical Report

We thank the reviewer for pointing out this discrepancy. The key difference is that our experiments use Qwen3-1.7B-Base and Qwen3-4B-Base as students, whereas Table 19 of the Qwen3 Technical Report reports results for the officially post-trained Qwen3-1.7B in the thinking mode. These checkpoints have different training histories and are therefore not directly comparable.

Our choice of Base checkpoints follows common practice in recent OPD studies. For example, Fast and Effective On-Policy Distillation from Reasoning Prefixes [1], Prune-OPD [2], and Entropy-Aware On-Policy Distillation of Language Models [3] also adopt pretrained Base models as students. This setting allows us to evaluate our method from a consistent pretrained initialization. Moreover, the official Qwen3 post-training pipeline already incorporates both off-policy and on-policy distillation for lightweight models. In particular, Qwen3-32B and Qwen3-235B-A22B are explicitly used as teachers in the on-policy distillation stage (Section 4.5 of the Qwen3 Technical Report). Starting from post-trained checkpoints would introduce prior distillation effects, making it harder to isolate the contribution of our method.

We will clarify the distinction between Base and post-trained checkpoints, as well as the evaluation settings, in the revised manuscript.

References

[1] Dongxu Zhang et al. Fast and Effective On-Policy Distillation from Reasoning Prefixes. Findings of ACL, 2026.

[2] Zhicheng Yang et al. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning. arXiv:2605.07804, 2026.

[3] Woogyeol Jin et al. Entropy-Aware On-Policy Distillation of Language Models. arXiv:2603.07079, 2026.

W3: Clarification of Trajectory Replacement Operations

Thank you for identifying the ambiguity in the ablation terminology. We have revised Section 5.3 and Table 2 to consistently use the terms Correct-Rollout Replacement and Unreliable-Error Replacement, aligning the ablation descriptions with the corresponding mechanisms in the Method section.

Specifically, Correct-Rollout Replacement selects the fraction of correct rollouts with the highest teacher log-likelihood for replacement. Unreliable-Error Replacement targets incorrect rollouts whose teacher log-likelihood exceeds the mean teacher log-likelihood of correct responses to the same prompt, selecting the highest-likelihood candidates among them. We also clarify the ablation settings by separately disabling these two replacement operations to examine their individual contributions.

W4: Clarification of Relative Performance Changes

We thank the reviewer for catching this ambiguity. We have revised the corresponding label in Table 1 from $ \Delta $ vs. OPD to Relative change vs. OPD (%). This explicitly indicates that the reported values represent relative percentage changes rather than absolute percentage-point differences. The underlying experimental results remain unchanged.

W5: Clarification of Baseline Selection

We appreciate the reviewer’s suggestions regarding additional baselines. Our primary objective is to improve the training efficiency of OPD, rather than to establish a comparison between different post-training paradigms. SFT and GRPO represent alternative post-training approaches with different optimization objectives and training pipelines. Our work instead investigates how to allocate training updates more effectively within the OPD framework. Therefore, we use vanilla OPD under matched experimental settings as the primary baseline, allowing us to isolate the improvements brought by RTB. We do not claim that RTB outperforms SFT or GRPO.

Regarding RG-OPD [4], although it also incorporates verifier feedback and teacher information into OPD, its intervention mechanism differs from ours. RG-OPD uses a reward–teacher likelihood gate to determine whether newly sampled trajectories should contribute to the distillation loss, applying reverse-KL supervision only when the verifier reward and teacher–student likelihood gap are directionally consistent. In contrast, RTB focuses on trajectory allocation through selective replacement of current rollouts and compatibility-based admission and replay of historical mistakes. Thus, RG-OPD primarily addresses when teacher supervision should be applied, whereas RTB addresses which trajectories should receive optimization updates and how useful historical mistakes can be retained for future training. These mechanisms address different aspects of OPD and are complementary rather than directly interchangeable.

We will clarify the relationship between RG-OPD and RTB in the Related Work section.

W6: Citation Formatting

Thank you for identifying the citation issues. We have corrected the missing references and citation formatting in the revised manuscript and recompiled the PDF to ensure that all references are properly rendered.

Q4: Different Training and Evaluation Temperatures

We thank the reviewer for raising this question. The different temperature settings serve distinct purposes. During training, we set the student rollout and teacher temperatures to 1.0 to preserve sufficient diversity in the generated trajectories. This is particularly important for RTB, as diverse student-generated trajectories provide a broader pool of candidate errors for identifying and retaining teachable mistakes. During evaluation, we use a lower temperature of 0.7 with top-p = 0.95 to obtain more focused generations while retaining sampling diversity. All methods are evaluated using the same decoding configuration to ensure fair comparisons.

This setting is also consistent with recent OPD studies. For example, Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe [1], Tail-Aware Top-k On-Policy Distillation [2], and Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation [3] all adopt a training temperature of 1.0 and an evaluation temperature of 0.7 with top-p = 0.95. We will clarify the rationale for using different temperatures in the revised manuscript.

References

[1] Yaxuan Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.

[2] Huipeng Huang and Hongxin Wei. Tail-Aware Top-k On-Policy Distillation. arXiv:2608.14728, 2026.

[3] Bingxi Hou et al. Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation. arXiv:2610.21601, 2026.

[4] Mohammad Sadegh Akhondzadeh et al. Reward-Gated On-Policy Distillation. arXiv:2607.04037, 2026.

I thank you for the comments. With regards to the updated version of the paper, I think it's hard to judge it without reading it. With regards to W2, I apologize, I meant to write table 20. I think table 20 of the same Qwen report shows much higher gains for Math 500 (im not sure why). This discrepancy needs to be checked. With regards to W5, I think the baseline listed is highly relevant, the mechanism is different but the intent in my opinion is similar and hence cannot be disregarded. Q3, was not addressed and for Q4, I think this needs to be tested to see what temperature combination really works. Considering all of this, I dont think my concerns have been sufficiently addressed and hence I maintain my score.

We thank the reviewer for the follow-up comments. We respond to your newly raised concerns concisely below.

  1. Qwen3 technical report Table 20 discrepancy We have rechecked Table 19 and Table 20. Both report results on Qwen3 Instruct checkpoints (thinking / non-thinking), while our experiments use Qwen3 Base, the raw pretrained checkpoint without any post-training. This fundamental difference in model checkpoints accounts for the performance gap. We will clearly highlight this distinction in the revised manuscript.

  2. Temperature mismatch between training and evaluation We have conducted the requested temperature ablation, with results as follows:

TemperatureAIME24AIME25AMC23MATH-500
1.00.08540.05410.37500.6620
0.70.08960.05210.37810.6680
0.50.08960.05000.37960.6560
0.30.08330.05210.36560.6660

The performance remains stable across this temperature range, and no single temperature consistently outperforms others on all benchmarks. Our original setting (training T=1.0 for trajectory diversity, evaluation T=0.7 for consistent comparison) follows standard OPD conventions.

Reviewer_cgEBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Summary

Authors introduce: Reliable Trajectory Buffer. Uniform OPD uses all trajectories equally, even for correct ones, and uninformative errors. Paper proposes using outcome and teacher-based signals to prioritize trajectories for training without changing the distillation loss.

  • RTB is a trajectory selection/allocation setup
  • For newly generated trajectories, it uses answer correctness and teacher-assigned likelihood to select which trajectories to replace rather than optimize as sampled.
  • For previously encountered mistakes, it uses teacher-student predictive compatibility to decide which candidates to admit to a replay buffer.
  • After replacement, RTB re-scores the inputs to refresh the training signals.

Problem being solved

student-generated trajectories do not always offer equally useful opportunities for distillation A correct trajectory may: (a) already align closely with the teacher, or (b) may still contain big predictive inconsistency Incorrect trajectories can similarly also have student & teacher agreeing to different degrees in the prefix

Process

Fig 3 in the paper gives a good overview. The process relies on having a verifiable final-answer state. They use math problems in this case.

Seleting trajectories:

  • For correct answers: based on teacher’s likelihood traj.
  • For incorrect: keep the ones where trajectory is at least close to teacher’s high-likelihood response to avoid noisy gradients. High token overlap indicates that even with wrong result, student & teacher distributions are locally aligned

Setup:

  1. Training data: DAPO-Math-17K; ~17,000 math problems with verifiable answers; reverse KL distillation loss used.
  2. Model initialization: Fixed Qwen3-8B teacher; Qwen3-1.7B-Base / Qwen3-4B-Base student
  3. Rollout collection: Sample 4 responses per prompt at T=1.0; batch = 64; total = 256 trajectories
  4. Outcome verification: Evaluate each response with the mathematical-answer verifier
  5. Replacemensts: Get teacher log-likelihood; identify teacher-supported correct responses + teacher-confidence-mismatch errors
  6. Defines “Predictive compatibility” as the number of common tokens b/w teacher and student’s high prob (top 16) tokens for the fixed prefix. Rank incorrect response prefixes by average top-16 overlap; retain up to 16 entries per training step.
  7. Replace selected trajectories - uniform sampling with replacement
  8. Recompute teacher & student distributions over the new training batch whenever replay succeeds
  9. Optimization: Apply same truncated reverse-KL distl. loss
  10. Evals: 6 math benchmarks; 2 students; replacement-target ablations, random-admission comparison, and pseudo-label experiment

Required Theory

  1. On-policy Post training: training on student model’s own generated trajectories with RL (as opposed to off-policy, which uses a teacher model’s logits, or final token sequence to do distillation on student - SFT)
  2. On-policy Distillation: grading student model’s trajectories with a teacher and training on that with distillation instead of RL to get over the sparse reward issue with RL.

Novelty

Some algorithmic novelty in combining known techniques and usign them in a novel way:

  1. Correctness / teacher-likelihood based filtering of trajectories.
  2. Compatibility-ranked admission of historical errors.
  3. Current-policy rescoring while preserving the original OPD loss.
Critical strengths
  1. Good problem thesis: standard trajectory supervision overlooks heterogeneity in corrective learning opportunities.
  2. Existing OPD loss preserved; no additional learned reward model.
  3. Easy to implement: FIFO replay, explicit admission budget, current-policy rescoring.
  4. Good improvements: across 2 student sizes; 6 mathematical reasoning benchmarks.
  5. Good ablation study: replacment target selection; compatibility-based versus random admission.
  6. Explicit acknowledgment of off-policy replay and absence of total-compute efficiency evidence.
Critical weaknesses
  1. Only checked in mathematical reasoning benchmarks
  2. Only tested with single teacher family; 2 same-family students
  3. Needs more statistical metrics using multiple runs of the setup (ie, std deviation, confidence intervals) to check the consistency of the better-than-OPD results
  4. While the authors mention SCOPE and other papers, there are no metrics comparing against these techniques. Only OPD is compared
  5. Re. “compatibility” scoring: I’m not fully convinced high overlap of high-prob tokens is a good signal. High overlap can just indicate already trained space of trajectories, with just an incorrect answer. Conversely, low overlap can still indicate the trajectories are actually very useful. They should try to get a causal signal linking admitted trajectories to task improvement
Questions for authors
  1. Some papers in the “RELATED WORK” section have not been correctly cited/formatted (‘?’ sign in the pdf)
  2. In related work/elsewhere, please add a short section on how/why your work is novel and is not already covered by existing related works.
  3. How many buffer capacities ( C ) were tried? Some ablations with that will be useful
  4. Any details on the rescoring overhead in terms of compute will be useful
  5. Do trajectories from older student checkpoints mess with the training stability in later checkpoints? Do you have some way of accounting for ordering of related errors in training steps?
  6. Any other verifiable tasks tested (non-math)?
3 comments

We thank the reviewer for the insightful and constructive feedback. In response, we have carefully revised the manuscript to address the concerns regarding methodological clarity, experimental evaluation, and presentation. Our detailed responses to each comment are provided below.

W1 & Q2: Novelty and Relationship to Existing Methods

We appreciate the suggestion to clarify the novelty of RTB. OPD differs from RL-based on-policy post-training: although both use student-generated trajectories, OPD optimizes token-level teacher supervision rather than outcome-based rewards.

The central contribution of our work is identifying and addressing the heterogeneity of effective supervision in OPD. Specifically, answer correctness alone does not determine whether a trajectory provides useful opportunities for teacher-guided learning. RTB operationalizes this insight through three complementary mechanisms: (1) selective replacement of current trajectories based on correctness and teacher likelihood; (2) compatibility-based admission and replay of historical mistakes; and (3) rescoring under the current models while preserving the original distillation loss.

These mechanisms distinguish RTB from related approaches that address different aspects of distillation. SCOPE routes correct and incorrect rollouts to different optimization objectives; RA-OPD filters trajectories based on reward–distillation conflicts; FiRe-OPD combines trajectory filtering with token reweighting; Prune-OPD uses predictive compatibility to detect distribution drift; and DistiLLM explores adaptive reuse of historical student outputs. In contrast, RTB jointly reallocates current optimization updates and preserves compatible historical mistakes for future replay, without modifying the underlying distillation objective.

While these methods share the broader goal of improving distillation efficiency, their different objectives and intervention mechanisms do not constitute directly interchangeable baselines. Our experiments therefore focus on comparisons with vanilla OPD and controlled ablations of RTB's components. We do not claim empirical superiority over these methods. We will further clarify these distinctions and the specific novelty of RTB in the Related Work section.

W2: Predictive Compatibility as a Selection Signal

We agree that high overlap does not necessarily imply greater learning potential, nor does low overlap mean that a trajectory is uninformative. Importantly, we use predictive compatibility as a selection signal, not as a direct estimate of expected learning gain.

Specifically, answer correctness identifies unresolved task-level errors, while top-$ 16 $ overlap measures the agreement between teacher and student high-probability token candidates on the corresponding prefixes. Combining these signals helps identify potentially teachable mistakes, without assuming that overlap alone determines trajectory value.

Our controlled ablation provides empirical support for this criterion. Replacing overlap-based admission with random admission, while keeping the remaining procedure unchanged, reduces the aggregate score from 28.01 to 26.93, with overlap-based admission outperforming random selection on five of the six benchmarks. This supports the usefulness of compatibility-based selection for downstream performance in our setting, although it does not establish an item-level causal relationship between overlap and individual learning gains. We will clarify this distinction and avoid interpreting compatibility as a direct causal measure.

W1 & Q6: Evaluation Scope and Generalizability

We use Qwen3-8B as the teacher and Qwen3-1.7B-Base and Qwen3-4B-Base as students. Using models from the same family provides a shared tokenizer and vocabulary for token-level distribution matching, allowing us to isolate the effect of trajectory allocation without introducing cross-tokenizer alignment factors. We acknowledge that generalization to other teacher families remains to be evaluated.

Regarding task domains, we choose mathematical reasoning because it provides reliable outcome verification, which is required by RTB's correctness-based trajectory selection. This is a controlled testbed for studying supervision allocation rather than a mathematical-task-specific assumption of the method.

We also clarify an important detail concerning all-wrong prompt groups. When all sampled responses for a prompt are incorrect, the correct-response mean used by the confidence-mismatch replacement rule is unavailable, so this particular replacement criterion cannot be applied. However, these incorrect trajectories remain eligible for compatibility-based buffer admission. Thus, all-wrong groups are not excluded from RTB's replay mechanism.

More broadly, RTB relies on verifiable outcomes and meaningful teacher–student predictive compatibility, rather than mathematical reasoning itself. Extending it to coding tasks with execution-based verification is a natural direction, whereas instruction following and open-ended dialogue would require appropriate task-specific evaluation criteria. These extensions remain unexplored, and we will clarify the current evaluation scope in the revised manuscript.

Statistical Reliability

We acknowledge that the reported results are based on single runs and do not include standard deviations or confidence intervals. Nevertheless, the improvements are observed across both student scales and are broadly distributed across benchmarks: RTB improves 14 of the 18 benchmark-specific metrics, matches two, and decreases on two. The controlled ablations further support the contributions of the individual components.

We will explicitly clarify the single-run setting and avoid making statistical-significance claims. Multi-seed evaluation would provide additional evidence regarding variance and reproducibility.

Q3–Q5: Buffer Configuration, Rescoring Overhead, and Replay Stability

Buffer configuration (Q3). We use a fixed FIFO buffer capacity of $ C=256 $ trajectories and an admission budget of up to $ M=16 $ incorrect trajectories per training step. We use this configuration throughout our experiments rather than conducting a capacity sweep. The capacity controls the amount of historical data retained, while the admission budget controls how many new mistakes are introduced into the buffer at each step. Neither changes the underlying selection criteria or distillation objective. We will make this fixed configuration explicit.

Rescoring overhead (Q4). Whenever at least one replacement succeeds, RTB recomputes student and teacher scores over the entire resulting batch to ensure consistent distillation signals. This introduces an additional batched forward-scoring pass but requires no additional trajectory generation. Importantly, autoregressive rollout generation and batched scoring over existing trajectories have different computational profiles; an additional scoring pass should therefore not be directly interpreted as doubling the overall training cost. Our sample-efficiency comparison measures progress relative to newly generated student trajectories, not matched FLOPs or wall-clock time. We will further clarify this distinction and avoid claiming lower total computational cost.

Replay stability (Q5). RTB uses three mechanisms to manage historical trajectories: (1) a bounded FIFO buffer that evicts the oldest entries when capacity is exceeded; (2) delayed admission, so newly selected trajectories become available for replay only from the next training step; and (3) current-model rescoring. In particular, the buffer stores prompts and response prefixes rather than policy-dependent scores, which are recomputed under the current student and fixed teacher after successful replacement. Thus, replayed trajectories may originate from earlier student policies, but their distillation signals are refreshed using the current models. We do not use explicit age-dependent weighting, nor do we claim that replay makes historical trajectories strictly on-policy.

Q1: Citation Formatting

Thank you for pointing out these issues. We have added the missing bibliography entries for SCOPE and RA-OPD, corrected the corresponding citation issues, and recompiled the PDF to ensure that all references are properly rendered in the revised manuscript.

Thanks for detailed responses

Main remaining concern is experimental validation. RTB improves over vanilla OPD - but, there are should be some direct comparisons with closely related trajectory-selection methods. Also, the compute-matched comparisons use online training pass rates & not separate benchmarks

Maintaining rating

Reviewer_SWMYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training trajectory allocation in on-policy distillation (OPD) for LLM mathematical reasoning. Noting that uniformly optimizing all student-generated rollouts can waste updates on saturated correct rollouts or unanchored errors with low corrective value, the authors propose the Reliable Trajectory Buffer (RTB). RTB modulates which trajectories are trained on without altering the underlying token-level reverse-KL distillation loss.

Critical strengths

Rather than proposing another variation of divergence objectives (e.g., skew KL, f-divergences), RTB directly addresses the data-efficiency bottleneck of OPD by optimizing which trajectories receive updates. The insight of coupling outcome verification with distribution compatibility—specifically filtering out unanchored "confidence mismatch" mistakes while preserving "teachable mistakes" via replay—is conceptually clean and complementary to existing distillation losses.

Solid Empirical Performance and Informative Diagnostics:RTB consistently outperforms vanilla OPD across two student model scales (Qwen3-1.7B and 4B) on six demanding competition-level math benchmarks. The ablation studies convincingly justify both error replacement and correct-sample replacement, demonstrate the superiority of overlap-based admission over random replay, and correlate downstream performance gains with higher token overlap, Kendall's τ\tau, and reduced teacher entropy.

Critical weaknesses

Unaccounted Wall-Clock and FLOP Overhead from Full-Batch Rescoring:Whenever a replacement occurs (which happens at almost every step with α=0.5\alpha=0.5), RTB discards existing policy scores and reruns full-batch student and teacher forward passes. This means every optimization step incurs approximately double the forward inference/scoring FLOPs compared to vanilla OPD. Presenting faster convergence exclusively in terms of "training steps" (Figure 1 and Figure 5) is somewhat misleading, as vanilla OPD trained for 1.5$\times$–2$\times$ more steps might consume equivalent total wall-clock FLOPs.

A fair wall-clock / FLOP-matched comparison is missing. Evaluation Restricted to a Single Domain (Math) with Rigid Group Verification:All experiments are conducted exclusively on mathematical reasoning with verifiable answers (DAPO-Math-17K). The replacement rule relies on grouping sampled responses per prompt (G=4G=4) and computing a prompt-specific baseline bT(x)b_T(x) using correct rollouts Cs(x)\mathcal{C}_s(x). On tasks where all sampled student rollouts are incorrect (∣Cs(x)∣=0\vert{}\mathcal{C}_s(x)\vert{}=0), the error replacement rule is bypassed entirely. The applicability and robustness of this framework to non-math domains (e.g., coding, instruction following, or open-ended dialogue) remain unverified.

Questions for authors

NA

2 comments

We thank the reviewer for the thoughtful and constructive assessment, particularly for recognizing the motivation and empirical effectiveness of RTB. We appreciate the concerns regarding computational overhead and experimental generalizability, which we address below.

W1: Computational Overhead and Efficiency Comparison

We thank the reviewer for highlighting the importance of accounting for the additional computational overhead introduced by full-batch rescoring. To address this concern, we conducted a detailed profiling of two 200-step online distillation runs using Qwen3-1.7B as the student and Qwen3-8B as the teacher. Both methods use identical student and teacher models, training data, and batch configurations.

Computational Cost Estimation. We estimate FLOPs using the standard parameter-based approximation, with $ 2NT $ for a forward pass and $ 4NT $ for a backward pass, where $ N $ denotes the number of non-embedding parameters and $ T $ denotes the number of processed tokens. Specifically, the student and teacher have 1.409B and 6.946B non-embedding parameters, respectively. The total estimated computation is $ (10N_s+2N_t)T ,accountingforstudent−sidecomputationandteacherforwardpasses.Thestudentrolloutandlog−probabilityforwardcomputation(, accounting for student-side computation and teacher forward passes. The student rollout and log-probability forward computation ( 4N_sT $) is included in the student-side total rather than counted separately.

ComponentFLOPs per tokenAverage FLOPs per step (Vanilla / RTB)
Student forward (generation + log-prob)$ 4N_s $ = 5.64 GFLOPs8.71 / 3.14 TFLOPs
Student-side total (including training and backward)$ 10N_s $ = 14.09 GFLOPs21.78 / 7.85 TFLOPs
Teacher forward$ 2N_t $ = 13.89 GFLOPs21.47 / 7.74 TFLOPs
Total27.98 GFLOPs43.24 / 15.59 TFLOPs

Wall-Clock Time and Total Computational Cost. Over 200 training steps, Vanilla OPD takes 11.43 hours, whereas RTB takes only 6.82 hours, reducing total training time by 40.3%. The corresponding average step times are 205.8 and 122.8 seconds, respectively. RTB also processes substantially fewer tokens, resulting in a 2.77$ \times $ reduction in estimated total FLOPs.

MetricVanilla OPDRTBVanilla / RTB
Total training time11.43 h6.82 h1.68$ \times $
Average step time205.8 s122.8 s1.68$ \times $
Total processed tokens309.0 M111.4 M2.77$ \times $
Student forward FLOPs (generation + log-prob)1.742 PFLOPs0.628 PFLOPs2.77$ \times $
Total student-side FLOPs4.355 PFLOPs1.571 PFLOPs2.77$ \times $
Teacher forward FLOPs4.293 PFLOPs1.548 PFLOPs2.77$ \times $
Total estimated FLOPs8.648 PFLOPs3.119 PFLOPs2.77$ \times $

The student forward FLOPs above are a subset of the total student-side FLOPs and are not counted twice.

Breakdown of Per-Step Runtime. To further understand the overhead, we decomposed the average step time into individual training stages:

StageVanilla OPDRTB
Rollout generation105.0 s (51.0%)54.3 s (44.3%)
Reward computation51.6 s (25.1%)22.5 s (18.3%)
Advantage estimation / trajectory buffer8.2 s (4.0%)30.7 s (25.0%)
Actor update40.8 s (19.8%)15.2 s (12.4%)
Total205.8 s122.8 s

Within reward computation, log-probability calculation decreases from 10.9 to 3.9 seconds, and rule-based reward computation decreases from 39.0 to 13.6 seconds. Distillation and top-k computation increases from 1.0 to 4.8 seconds, reflecting the additional scoring operations in RTB. Although trajectory-buffer management introduces additional overhead (8.2 to 30.7 seconds), the reductions in rollout generation, reward computation, and actor updates more than compensate for it. Overall, RTB reduces average step time by 40.3%.

Compute-Matched Comparison. Importantly, the faster convergence is not merely an artifact of comparing training steps. We further aligned the two training curves by cumulative estimated FLOPs and compared their online training pass rates, measured by the EMA of true_reward/mean with a decay factor of 0.05.

Cumulative FLOPsVanilla OPDRTBDifference
0.5 PFLOPs0.0080.0080.000
1.0 PFLOPs0.0080.0080.000
1.5 PFLOPs0.0070.019+0.012
2.0 PFLOPs0.0050.061+0.056
2.5 PFLOPs0.0040.074+0.070
3.0 PFLOPs0.0040.075+0.071
3.12 PFLOPs0.0030.075+0.072

RTB achieves substantially higher training pass rates under the same computational budget, particularly beyond 1.5 PFLOPs. For example, at 2.5 PFLOPs, RTB reaches 0.074, compared with 0.004 for Vanilla OPD.

Computational Cost to Reach the Same Performance. We also compare the estimated computation and wall-clock time required to reach identical online training pass-rate thresholds.

Target pass rateVanilla OPD (FLOPs / Time)RTB (FLOPs / Time)FLOPs ratioTime ratio
0.025.53 PFLOPs / 7.41 h1.52 PFLOPs / 3.25 h3.6$ \times $2.3$ \times $
0.056.54 PFLOPs / 8.72 h1.89 PFLOPs / 4.09 h3.5$ \times $2.1$ \times $
0.077.15 PFLOPs / 9.50 h2.20 PFLOPs / 4.77 h3.3$ \times $2.0$ \times $

These results show that RTB requires approximately 3.3--3.6$ \times $ less estimated computation and 2.0--2.3$ \times $ less wall-clock time to reach the same intermediate training pass-rate thresholds.

Overall, these experiments demonstrate that RTB's efficiency improvements extend beyond sample efficiency measured by training steps. Although trajectory selection and rescoring introduce additional computational overhead, the resulting reduction in processed tokens and the associated savings in other training stages lead to lower overall computational costs and faster attainment of comparable training performance. We will incorporate these results and clarify the distinction between sample efficiency and compute efficiency in the revised manuscript.

W2: Generalizability and All-Wrong Prompt Groups

We agree that the current evaluation is limited to mathematical reasoning with verifiable answers. Our experimental setting follows a common practice in recent OPD studies, using DAPO-Math-17K for training and established mathematical reasoning benchmarks, including AIME24/25, AMC23, MATH-500, Minerva Math, and OlympiadBench, for evaluation. This setting provides a controlled environment for studying trajectory-level supervision quality, as final-answer correctness can be reliably verified.

More importantly, the primary contribution of this work is to identify an overlooked issue in OPD: not all student-generated trajectories provide equally useful opportunities for teacher supervision. In particular, some incorrect trajectories exhibit weak predictive compatibility with the teacher and therefore receive limited corrective signals despite dense token-level supervision. RTB attempts to identify and handle such trajectories through outcome verification and predictive compatibility.

Regarding all-wrong prompt groups, our replacement rule uses the mean teacher likelihood of correct responses within the same prompt group as a reference. Consequently, when all sampled responses for a prompt are incorrect, this reference is unavailable, and the corresponding errors are not selected for replacement. However, these trajectories are not discarded: incorrect responses from both mixed and all-wrong groups remain eligible for compatibility-based buffer admission and can still contribute teachable mistakes for future replay.

Regarding broader applicability, RTB currently relies on verifiable outcomes and meaningful teacher--student predictive compatibility. Extending it to coding tasks with execution-based verification is a promising direction, while applications to instruction following and open-ended dialogue would require appropriate task-specific evaluation criteria, which remain unexplored. We will clarify these applicability conditions and acknowledge the current limitations in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.