acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-Refined Distillation

Abstract

On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student’s own rollouts. In this work, we identify a common structural cause underlying OPD, which we call prefix failure. Under prefix failure, dense per-token supervision induces a bimodal teacher mixture and fragmented gradients that token-level loss truncation or reweighting fail to address. We thus propose Trajectory-Refined Distillation (TRD), a trajectory-level correction method that revises the student’s rollout under the teacher guidance while within on-policy support. By correcting problematic prefixes before distillation, TRD mitigates prefix failure at its source. Moreover, TRD improves exploration by exposing the student to alternative valid derivations under teacher guidance, even when the original rollouts are already correct. TRD can also be applied to on-policy self-distillation (OPSD), a parameter-sharing variant that uses the student model conditioned on privileged information as the teacher. TRD improves average math accuracy while remaining competitive on code. Under OPSD, it solves AMOBench questions left unsolved by the Qwen3-8B base model, nearly doubling the strongest dense-KL baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, On-policy Distillation, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w161OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows and addresses a limitation of OPSD, where KD loss is computed along student-generated trajectories that may contain incorrect reasoning steps, arguing that incorrect prefixes can lead to fragmented or misleading supervision signals. To address this problem, the authors propose Trajectory-Refined Distillation (TRD). Instead of directly distilling on the student's sampled trajectory, the method asks the teacher to revise it and then performs logit KD along the refined trajectory.

Critical strengths
  1. Simple proposed approach.
  2. Good experimental results show the effectiveness of TRD.
Critical weaknesses
  1. The idea of refining student-generated trajectories using teacher feedback before training has been explored in prior work, including Critique-Guided Distillation (ICML 2026) and Multi-Granularity Semantic Revision (ACL 2026). These closely related approaches are not discussed or experimentally compared. Although TRD uses a different KL-based training objective, its specific novelty relative to these methods is insufficiently established.

  2. Although the authors provide some justification, TRD generates refined trajectories using the teacher rather than the student. Therefore, these trajectories are not on-policy.

Questions for authors
  1. See weaknesses.
  2. Why is refinement necessary for all trajectories rather than selectively correcting incorrect ones? Additionally, how do the authors determine whether a single refinement step is sufficient?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors to Reviewer w161

We sincerely thank Reviewer w161 for the insightful comments. The two related works and the question on selective refinement helped us sharpen the positioning of TRD.

W1. "These closely related approaches are not discussed or experimentally compared." We clarify that Kapusuzoglu et al. (2026) refine at the trajectory level, but in SFT rather than OPD. Liu et al. (2026) can be viewed as OPD, but their correction is at the token level: one flagged token is replaced and the student regenerates the rest, so the earlier prefix is kept. As argued in the submission (lines 65 to 66), recovering from prefix failure "requires a trajectory-level improvement". We will cite and discuss both in Sec. 2. Table R5 summarizes them. To our knowledge, TRD is the first trajectory-level refinement in the OPD setup.

Table R5. Comparison with related works.

MethodSetupRefinement levelTraining signal
Kapusuzoglu et al. (2026)SFTTrajectoryCross-entropy
Liu et al. (2026)OPDTokenKL
TRD (ours)OPDTrajectoryKL

TRD is also well motivated rather than a heuristic: it follows from our analysis of prefix failure, which is inherent in the post-hoc supervision paradigm of OPD (Sec. 4). Prefix failure is also a recognized problem in the community (Liao et al. 2026; Xu et al. 2026).

W2. "these trajectories are not on-policy." We agree that yry_r is not an on-policy sample, and TRD does not require it to be. (1) We require the support constraint of Eq. (4) (lines 271 to 276), in the sense of the RL literature on trust regions and support constraints (Schulman et al. 2015; Kumar et al. 2019; Kostrikov et al. 2022): the constructed trajectories should stay in the region where the student has support (Fig. 1, left). Formally, the student's density along the trajectory is bounded below (see Table R4, response to KJ3t). Eq. (4) is a construction target, not strictly enforced: yry_r may fall outside the support, as our Limitations note. We will state this in Sec. 5 and correct our imprecise "on-policy" wording throughout. (2) Strict on-policy sampling is also not required: distilling on fixed offline student rollouts can match or outperform fully on-policy distillation (Wu et al. 2026; Zhao et al. 2026).

Q1. "Why is refinement necessary for all trajectories?" (1) A correct final answer does not imply a correct reasoning path, and locating failed prefixes is intractable (line 270), so correct rollouts cannot be assumed free of prefix failure. (2) Refining a correct yoy_o can expose new reasoning paths that the student rarely samples while maintaining its support (Sec. 5), which improves exploration and Pass@kk (Sec. 6.3; Table R7). (3) We added a failed-only variant (Table R6): it improves over OPD (mean 36.7 vs 35.8) but is below TRD (37.9), so refining both failed and correct rollouts matters. The same holds for Pass@128 (Table R7).

Table R6. Failed-only refinement on Qwen3-4B-Instruct-2507 (separate teacher, %).

Avg@16AIME24AIME25HMMT25BeyondAIMEAMOBenchMean
Base59.142.928.328.28.133.3
+OPD60.046.730.432.09.935.8
+TRD (failed-only)62.347.430.733.110.136.7
+TRD65.447.933.232.610.337.9
Pass@16
Base73.366.743.354.023.152.1
+OPD80.073.350.061.033.359.5
+TRD (failed-only)83.376.746.761.035.960.7
+TRD83.376.750.062.035.961.6

Table R7. Pass@kk (%) on AMOBench, same setting (Fig. 3).

kk181664128
+OPD9.928.233.343.646.7
+TRD (failed-only)10.128.235.943.648.7
+TRD10.328.235.948.753.8

Q2. "whether a single refinement step is sufficient?" One step raises the verifier pass rate from 65.8% to 81.4% on Qwen3-8B (Fig. 6). More steps may raise it, but each step moves the trajectory farther from yoy_o and may fail to maintain the student's support: one step raises the per-token NLL (Table R4). We thus use a single step as the trade-off in Eq. (4): mitigating prefix failure while staying within the student's support.

References

Kapusuzoglu B, et al. (2026) Critique-guided distillation. ICML.

Kostrikov I, et al. (2022) Offline RL with implicit Q-learning. ICLR.

Kumar A, et al. (2019) Stabilizing off-policy Q-learning. NeurIPS.

Liao B, et al. (2026) Multi-turn OPD with prefix replay. arXiv.

Liu X, et al. (2026) Multi-granularity semantic revision. ACL.

Schulman J, et al. (2015) Trust region policy optimization. ICML.

Wu Y, et al. (2026) Lightning OPD. arXiv.

Xu H, et al. (2026) Pass the baton. arXiv.

Zhao S, et al. (2026) When do we need on-policy distillation? arXiv.

Thank you for the response. The failed-only refinement ablation addresses my question about the need to refine both correct and incorrect trajectories. However, my main concern regarding novelty remains unresolved. The cited works are closely related to TRD, as they share the same central idea of refining student-generated trajectories using teacher feedback before training. Although the authors clarify the differences in refinement mechanisms and training objectives, it remains unclear whether these distinctions constitute a sufficiently significant contribution. Moreover, without direct experimental comparisons with these closely related methods, it is difficult to assess what additional benefit TRD offers over existing approaches.

I appreciate the authors' acknowledgment that the refined trajectories are not strictly on-policy, although the clarifications have not yet been incorporated into a revised manuscript. Regarding the number of refinement steps, while the authors show that a single step improves trajectory correctness, they do not provide experiments comparing different numbers of refinement steps. Therefore, it remains unclear whether one step is sufficient or whether additional refinement could yield further improvements.

Overall, while the additional experiment addresses one of my concerns, the main issue regarding novelty remains insufficiently resolved. I therefore maintain my original score of 2.

AuthorsOpenReview ↗

Follow Up Responses to Reviewer w161

Thank you for the follow-up. We address the remaining points.

1. Novelty. We agree that refining student trajectories with teacher feedback is shared with Kapusuzoglu et al. (2026) and Liu et al. (2026). Our contribution is not refinement by itself. It is (i) the analysis of prefix failure, which explains why dense KL on a failed student prefix does not teach the correction path (Sec. 4), and (ii) the resulting design: the refined complete trajectory serves as the context of full-vocabulary KL in OPD, including privileged self-distillation. These distinctions determine the outcome: under the same setup, both cited methods fall below TRD (Table R9).

2. Direct comparisons. Table R9 compares TRD with both families, and with TOPD raised by Reviewer KJ3t. TRD achieves the best mean; SCRG is the closest, and TOPD is on par with vanilla OPD. The ordering follows from what each method changes. Token-level correction (SKD, SCRG) and segment-level guidance (TOPD) keep the earlier student prefix as the supervision context, so the KL is still evaluated on a failed prefix. CGD rewrites the whole trajectory but trains with cross-entropy, which discards the teacher distribution and falls below OPD. TRD is the only method that both replaces the context with a refined complete trajectory and keeps full-vocabulary KL.

Table R9. Avg@16 (%) on Qwen3-1.7B with a separate teacher, under the same student, teacher, and data. TOPD has no official code and is our reimplementation from the paper.

MethodTeacher correctionSetupLossAIME24AIME25HMMT25BeyondAIMEAMOBenchMean
+OPDnoneOPDKL44.435.123.321.22.425.3
+SKDtokenOPDKL40.231.519.615.51.421.6
+SCRG (Liu et al. 2026)tokenOPDKL45.135.724.220.93.025.8
+CGD (Kapusuzoglu et al. 2026)trajectorySFTCE40.931.419.217.71.222.1
+TOPD (Jiang and Ferraro 2026)segment (50 tokens)OPDKL + OT46.234.721.621.22.425.2
+TRDtrajectoryOPDKL49.437.524.420.13.026.9

3. Number of refinement steps. Table R10 compares zero to three refinement steps and reports the trajectory statistics together with the final performance. One step gives the best final performance. A second step adds little correctness and does not improve performance. A third step degrades the trajectories, which become much shorter and less correct, and performance drops below OPSD. One step is therefore sufficient: additional refinement brings no further gain and eventually over-compresses the trajectories. Each additional step also requires another teacher sampling pass over the training set, so the computation cost grows with the number of steps (Tab. 12). The reference y⋆y^\star, although fully correct, lies mostly outside the student's support and is also below one step.

Table R10. Refinement steps on Qwen3-4B-Instruct-2507 with a privileged self-teacher. ϵ\epsilon is set so that 95% of yoy_o lie inside the ϵ\epsilon-support (see Table R4).

StepsVerifier pass (%)NLL/tokenMean lengthIn ϵ\epsilon-support (%)Mean Avg@16 (%)
0 (yoy_o, OPSD)81.30.1862,7779536.3
1 (yry_r, TRD)88.30.5659966737.7
290.20.5931,0227137.4
387.60.5416787733.9
y⋆y^\star (reference)1000.9725602037.0

4. Revised manuscript. We will revise the manuscript accordingly: the "support constraint" wording in Sec. 5, the discussion of Kapusuzoglu et al. (2026) and Liu et al. (2026) in Sec. 2, and Tables R5 to R10.

We hope these results address the remaining concerns.

I thank the authors for the additional experiments and clarifications. The new results strengthen the empirical evaluation of the paper. However, after reconsidering the proposed method compared to the related work, novelty remains my main concern Teacher-guided refinement of student-generated trajectories is already explored in prior work, including Critique-Guided Distillation (ICML 2026) and Multi-Granularity Semantic Revision (ACL 2026). Moreover, TOPD, as highlighted by another reviewer, already investigates trajectory-level guidance in on-policy distillation. While TRD differs by applying full-vocabulary KL along complete teacher-refined trajectories, it remains unclear whether this combination constitutes a substantial methodological contribution. The improvement over the strongest related baseline (SCRG) is also relatively marginal (+1.1). That is why I maintain my original score of 2.

Reviewer_KJ3tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies erroneous prefixes in on-policy distillation and argues that they create a bimodal teacher mixture and fragmented gradients. It proposes Trajectory-Refined Distillation, which samples a student trajectory, asks a teacher to revise it while conditioning on the original rollout, and then applies full-vocabulary forward KL along the refined trajectory. The method also extends to self-distillation using the student conditioned on privileged information as the teacher. Experiments span mathematics and code, several student scales, and separate-teacher and self-teacher settings; TRD generally improves average mathematics accuracy and remains competitive on code.

Critical strengths
  1. The method is straightforward and yields reasonably consistent mathematics gains across student scales, separate teachers, and self-teacher settings, suggesting practical value in teacher-guided trajectory revision.
  2. The evaluation covers multiple distillation and reinforcement-learning baselines and includes trajectory-source, Best-of-N, timing, quality, and coverage analyses.
Critical weaknesses
  1. The novelty scope should be stated more cautiously. TOPD [1] and related trajectory-correction and reflective-distillation work already address limitations of token-level supervision at the trajectory level, so “first trajectory-level optimization design” appears overly broad. If a priority claim is retained, it should be limited to using teacher-revised student trajectories as full-vocabulary OPD contexts and extending this design to privileged self-distillation.
  2. The central formalization and mechanistic evidence remain insufficient.
    • Support constraint: If support means positive probability under a standard softmax model, Equation (4) holds for almost every finite sequence and does not show that yry_r is close to the student distribution. If it means the actual top-k 20 sampling policy, the paper does not verify that teacher-rewritten tokens remain in the student's candidate set.
    • On-policy status: Although yoy_o is sampled from the student, the trajectory used for KL training is yr∼πT(⋅∣x,yo)y_r\sim\pi_T(\cdot\mid x,y_o). Conditioning on yoy_o does not ensure that yry_r lies in the student's typical set or has high student probability, while the median-length reduction from 7.7K to 0.88K suggests that refinement may amount to a substantial rewrite.
    • Prefix-failure validation: The reported KL, perplexity-gap, and correction-token diagnostics neither locate the earliest error nor directly demonstrate a bimodal teacher distribution between continuing the failed path and initiating a correction. The causal link between prefix correction and downstream gains therefore remains unestablished.
    • Result attribution: The main results lack independent seeds, uncertainty, and significance tests. Controls using the same teacher without the original rollout, correcting only failed trajectories, matching trajectory length, and hard-label training on yry_r are also missing.

[1] Jiang, Yuxuan and Ferraro, Francis. “Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance.” arXiv:2606.00305, 2026.

Questions for authors
  1. Given TOPD [1] and existing trajectory-correction paradigms, could the authors narrow the “first trajectory-level optimization design” claim and clarify what is new beyond near-future guidance and corrected-trajectory training, apart from using full-vocabulary OPD contexts?
  2. Does support mean positive probability, a typical set, a probability threshold, or the actual top-k sampling support? Please report student sequence likelihood, token-rank coverage, edit distance from yoy_o, or another distributional-proximity measure for yry_r to substantiate the student-anchored and on-policy claims.
  3. Could the authors annotate the earliest failure positions in actual rollouts, measure the teacher's semantic probability mass on continuing versus correcting near those positions, and verify that yry_r repairs the identified failures?
  4. Were the main results obtained with independent seeds or checkpoints? Please also compare against the same teacher without yoy_o, failed-only refinement, a length-matched control, and hard-label training on yry_r to separate correction, compression, and soft-KL effects.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors to Reviewer KJ3t (1/2)

We thank Reviewer KJ3t for the precise and constructive review.

W1 and Q1. "'first trajectory-level optimization design' appears overly broad." Thank you for pointing us to TOPD, a valuable and closely related paper. TOPD is available only as an arXiv preprint. Under the ICLR 2027 Reviewer Guidelines, authors "may be excused for not knowing about papers" available only on arXiv. We will cite and discuss it in Sec. 2.

W2a and Q2. "Does support mean positive probability, a typical set, a probability threshold...?" Eq. (4) is a trajectory-level support constraint (lines 271 to 276), in the sense of the RL literature on trust regions and support constraints (Schulman et al. 2015; Kumar et al. 2019; Kostrikov et al. 2022): the constructed trajectories should stay in the region where the student has support (Fig. 1, left). Formally, it is a probability threshold: the student's density along the trajectory is bounded below (Kumar et al. 2019). As you request, Table R4 reports the student's per-token NLL: yry_r lies between yoy_o and y⋆y^\star (0.565 vs 0.186 and 0.972 on Qwen3-4B-Instruct-2507; 0.345 vs 0.282 and 0.551 on Qwen3-8B), i.e., it is closer to the student's distribution than the reference while having a higher verifier pass rate than yoy_o. It is not a token-level top-kk condition or a typical-set requirement. Eq. (4) describes the construction target and is not optimized directly or strictly enforced (lines 276 to 277). Empirically, yty_t and y⋆y^\star, which are not anchored to a student rollout, trail yry_r (Tab. 5; best or tied in 16 of 20 settings). We will state this in Sec. 5 and correct our imprecise wording throughout.

Table R4. Per-token NLL of the training trajectories under the student, on one DeepScaleR training batch (128 samples; privileged self-teacher).

StudentTrajectoryNLL/tokenMean lengthVerifier pass (%)
Qwen3-4B-Instruct-2507yoy_o0.1862,77781.3
yry_r0.56599688.3
y⋆y^\star0.972560100
Qwen3-8Byoy_o0.2823,13396.1
yry_r0.3451,08098.4
y⋆y^\star0.551560100

W2b. "refinement may amount to a substantial rewrite" A support constraint does not require yry_r to resemble yoy_o token by token. The 7.7K to 0.88K reduction is measured under OPSD on Qwen3-8B, where the refiner is the student's own backbone conditioned on yoy_o and y⋆y^\star (Eq. (5)), not an external model. We agree that conditioning on yoy_o does not guarantee the constraint and yry_r may fall outside the support; the Limitations paragraph acknowledges this (lines 483 to 485).

W2c and Q3. "neither locate the earliest error nor directly demonstrate a bimodal teacher distribution" We agree; the submission states the diagnostics "do not directly locate failed prefixes" (line 738) and are "a supporting diagnostic" rather than direct evidence (line 765). They support the mechanism: on yoy_o the teacher places 6 to 8‰ of its next-token mass on correction-related tokens, against below 2‰ on yry_r (Fig. 5, right). Locating the failure position is intractable ("would require searching O(∣V∣k)O(|V|^k) continuations of length kk", line 270), hence the surrogate of Eq. (4). Prefix failure follows from the OPD training paradigm itself, as shown analytically in Sec. 4.2. It is also a recognized problem in the community (Liao et al. 2026; Xu et al. 2026).

Causal link: (1) Best-of-NN improves with NN, i.e., with more correct on-policy rollouts (Fig. 4); (2) correctness alone is not sufficient: y⋆y^\star does not outperform yry_r (Tab. 5); (3) so the gain requires correct trajectories within the student's support (Eq. (4)). Refining only failed rollouts also improves over OPD (W2d).

AuthorsOpenReview ↗

Rebuttal by Authors to Reviewer KJ3t (2/2)

W2d and Q4. "lack independent seeds, uncertainty, and significance tests"

(a) Seeds. The main results are single training runs. We follow the standard protocol for evaluating reasoning LLMs, which controls variance on the sampling side rather than through training seeds: each score is computed from 16 independent samples per question, as in the Qwen3 technical report (Yang et al. 2025).

(b) Controls. The submission partially separates these factors. (i) Without yoy_o: Tab. 5 fixes the teacher and the sampling budget and varies only the trajectory (see W2a). (ii) Failed trajectories only: a failed-only variant improves over OPD (mean 36.7 vs 35.8) but is below TRD (37.9), so refining both failed and correct rollouts matters (Tables R6 and R7, response to w161). Best-of-NN selection remains below TRD at 7.8x its wall-clock (Fig. 4). (iii) Length: y⋆y^\star is the shortest trajectory (Fig. 6, left) yet does not outperform yry_r overall, and at evaluation the students trained with OPD and TRD produce closely matching correct-rollout lengths (18.9K vs 18.5K chars; Fig. 6, right). (iv) Hard labels on yry_r: SFT on yry_r is 2.8 points below TRD (Table R3, response to BNcF).

Summary of contributions. (1) We identify prefix failure in OPD (Sec. 4). (2) Token-level remedies leave the failed prefix unchanged; we propose trajectory refinement (Sec. 5). (3) TRD improves math accuracy across four configurations and five benchmarks and remains competitive on code (Tabs. 1 to 4).

Please let us know whether these responses address your concerns.

References

Kostrikov I, et al. (2022) Offline RL with implicit Q-learning. ICLR.

Kumar A, et al. (2019) Stabilizing off-policy Q-learning via bootstrapping error reduction. NeurIPS.

Liao B, et al. (2026) Multi-turn OPD with prefix replay. arXiv.

Schulman J, et al. (2015) Trust region policy optimization. ICML.

Xu H, et al. (2026) Pass the baton. arXiv.

Yang A, et al. (2025) Qwen3 technical report. arXiv.

Thank you for the response and additional controls. The clarification of the on-policy terminology and the new NLL, failed-only, and SFT analyses are helpful. However, the support constraint remains unenforced, the proposed mechanism is only indirectly validated, and training-seed uncertainty and comparisons with closely related methods remain missing. I therefore retain my original score of 2.

AuthorsOpenReview ↗

Follow Up Responses to Reviewer KJ3t

Thank you for the follow-up and for acknowledging the new analyses. We address the remaining points.

1 and 2. "the support constraint remains unenforced, the proposed mechanism is only indirectly validated" We answer these two points together.

Role of Eq. (4). Eq. (4) is a surrogate target for mitigating prefix failure, not a constraint that the algorithm is meant to enforce. Since locating the failure position is intractable (line 270), we "relax the target to a trajectory-level surrogate" (line 271), which "is not optimized directly in the OPD update" and "defines an upstream trajectory-construction task" (lines 276 to 280). We then discuss how to optimize it through two extremes (lines 281 to 289): yoy_o (q=πθq=\pi_\theta) satisfies the support constraint by construction but does not mitigate prefix failure, and y⋆y^\star (q=π⋆q=\pi^\star) attains Verify=1\mathrm{Verify}=1 but generally violates it. TRD constructs yry_r between the two (Eq. (5)).

Where yry_r lies. Table R4 confirms the intended ordering: the student's per-token NLL of yry_r lies between yoy_o and y⋆y^\star on both models. Setting the threshold ϵ\epsilon so that 95% of yoy_o lie inside the ϵ\epsilon-support, 67% of yry_r and 20% of y⋆y^\star lie inside on Qwen3-4B-Instruct-2507 (72% and 14% on Qwen3-8B).

What is validated directly. Prefix failure itself is a structural property of OPD: dense KL is evaluated only on the prefixes the student visits, so on a failed prefix it supervises the correction-onset token but not the correction path (Sec. 4.2). The token-level diagnostics are presented only as supporting evidence (line 765). The remedy is validated by interventions that change one factor at a time: (a) refining only the failed rollouts already improves over OPD (Table R6); (b) with the trajectories fixed to yry_r, replacing KL by cross-entropy removes the gain (Table R3); (c) correctness alone is not sufficient: y⋆y^\star is 100% correct but has the highest student NLL and the lowest ϵ\epsilon-support coverage, and it does not outperform yry_r (Tab. 5). The gain therefore follows trajectories that are both more correct and within the student's support, as Eq. (4) describes.

3. "training-seed uncertainty" Table R8 reports three training seeds for OPD and TRD. The variation across seeds is small relative to the gap between the two methods, the seed means closely match the single runs of the submission, and TRD remains above OPD on the mean.

Table R8. Avg@16 (%) over three training seeds (mean ± std) on Qwen3-1.7B with a separate teacher.

MethodAIME24AIME25HMMT25BeyondAIMEAMOBenchMean
+OPD43.6 ± 0.835.5 ± 0.322.6 ± 0.721.5 ± 1.02.4 ± 0.125.1
+TRD48.9 ± 0.837.9 ± 0.623.7 ± 0.820.7 ± 0.63.0 ± 0.026.8

4. "comparisons with closely related methods" Table R9 adds direct comparisons with TOPD (TOPD has no official code and is our reimplementation from the paper) and with the two refinement-based methods raised by Reviewer w161. TRD achieves the best mean; SCRG is the closest, and TOPD is on par with vanilla OPD. The ordering follows from what each method changes. Token-level correction (SKD, SCRG) and segment-level guidance (TOPD) keep the earlier student prefix as the supervision context, so the KL is still evaluated on a failed prefix. CGD rewrites the whole trajectory but trains with cross-entropy, which discards the teacher distribution and falls below OPD. TRD is the only method that both replaces the context with a refined complete trajectory and keeps full-vocabulary KL.

Table R9. Avg@16 (%) on Qwen3-1.7B with a separate teacher, under the same student, teacher, and data.

MethodTeacher correctionSetupLossAIME24AIME25HMMT25BeyondAIMEAMOBenchMean
+OPDnoneOPDKL44.435.123.321.22.425.3
+SKDtokenOPDKL40.231.519.615.51.421.6
+SCRG (Liu et al. 2026)tokenOPDKL45.135.724.220.93.025.8
+CGD (Kapusuzoglu et al. 2026)trajectorySFTCE40.931.419.217.71.222.1
+TOPD (Jiang and Ferraro 2026)segment (50 tokens)OPDKL + OT46.234.721.621.22.425.2
+TRDtrajectoryOPDKL49.437.524.420.13.026.9

In summary, each remaining point now has new evidence: support coverage and interventional validation (points 1 and 2), training seeds (Table R8), and direct comparisons (Table R9). We hope these results address your concerns.

References

Jiang and Ferraro (2026) Bridging reasoning trajectories in on-policy distillation via near-future guidance. arXiv.

Reviewer_BNcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Trajectory-Refined Distillation (TRD), which generates student trajectories, refines them with a teacher, and applies token-level KL distillation along the refined prefixes to mitigate supervision mismatch caused by erroneous reasoning prefixes. Experiments across Qwen3 models, two teacher regimes, and math and code benchmarks show consistent improvements in average math accuracy and competitive code performance. However, the claimed on-policy property and the attribution of gains remain insufficiently established.

Critical strengths
  1. The method is simple, integrates naturally with OPD/OPSD pipelines, and addresses supervision mismatch on erroneous trajectories.
  2. TRD achieves the highest math macro-average across all four model/teacher configurations, suggesting applicability across settings.
  3. The paper analyzes trajectory sources, Best-of-N, length, and compute; total pipeline time is close to vanilla OPSD in the 8B setting.
Critical weaknesses
  1. The on-policy claim and proposed mechanism are insufficiently established. The nonzero-probability constraint in Eq. (4) is generally too weak for an untruncated softmax model to meaningfully distinguish reference solutions from refined trajectories. If support instead refers to top-k/top-p sampling, teacher refinement does not guarantee feasibility. Moreover, differences between supervision contexts on a fixed erroneous trajectory do not establish that iterative OPD cannot learn corrections. The reported KL, perplexity, and correction-token statistics do not directly verify the proposed bimodal distribution or gradient fragmentation.
  2. Confounded comparisons and a baseline discrepancy limit attribution. TRD changes the supervision trajectories, teacher context, and trajectory length simultaneously, making it difficult to isolate prefix repair. In Table 5, distillation on reference trajectories and TRD differ by only approximately 0.04 percentage points in 8B math macro-average accuracy. Additionally, Appendix C.2 describes clipping the aggregate KL at each position, whereas the cited OPSD method clips individual vocabulary-level contributions. These operations are not equivalent and may affect baseline strength; clarification and a faithful implementation are needed (Zhao et al., Section 3.2).
  3. Compute matching and statistical reporting are incomplete. In the separate-teacher settings, TRD takes approximately 1.69–1.75 times the total time of vanilla OPD, motivating equal-total-GPU-time comparisons against the closest distillation baselines. Main results lack repeated training runs and paired confidence intervals. AMOBench contains only 39 evaluated questions, so the coverage difference of 9/23 versus approximately 5/23 requires stability analysis. Under the single-evaluation definition in Appendix C.6, Table 4's 33.8% Pass@16 cannot result from conventionally rounding an integer solved count over 39 questions; the aggregation protocol or reported value needs clarification.
Questions for authors
  1. How is the on-policy constraint defined and verified? Are trajectories refreshed during training, and does the self-teacher use current parameters?
  2. Can you control teacher context and trajectory length, compare KL with cross-entropy on the same refined trajectories, and verify the clipping baseline?
  3. Do the gains persist under equal total GPU time and multiple random seeds? Please provide AMOBench solved counts and explain the 33.8% result.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors to Reviewer BNcF (1/2)

We thank Reviewer BNcF for the careful and specific review. We respond in two comments.

W1. "The on-policy claim and proposed mechanism are insufficiently established."

  1. "The nonzero-probability constraint in Eq. (4) is generally too weak" Eq. (4) is a support constraint (lines 271 to 276). It is used in the sense of the RL literature on trust regions and support constraints (Schulman et al. 2015; Kumar et al. 2019; Kostrikov et al. 2022): the constructed trajectories should stay in the region where the student has support, as illustrated in Fig. 1 (left), rather than be on-policy or high-likelihood samples. Formally, the student's density along the trajectory is bounded below by a threshold, as in Kumar et al. (2019); the measured per-token NLL is in Table R4 (response to Reviewer KJ3t). Eq. (4) describes the target of trajectory construction and is not optimized directly or strictly enforced (lines 276 to 277); we acknowledge that yry_r may fall outside the support (Limitations, lines 483 to 485). We will state this in Sec. 5 and correct our imprecise "on-policy" wording throughout.
  2. "do not establish that iterative OPD cannot learn corrections" This is our motivation for this paper. The fixed erroneous trajectory is yoy_o with post-hoc supervision: on it, dense KL supervises the correction-onset token but not the correction path (Sec. 4.2). We do not claim that iterative OPD cannot learn corrections; the submission states that this signal "can trigger self-correction" (lines 256 to 258). TRD supplies the correction path directly through yry_r instead of leaving the student to discover it.
  3. "do not directly verify the proposed bimodal distribution or gradient fragmentation" The diagnostics support the bimodal teacher mixture of Sec. 4.1: on yoy_o the teacher places 6 to 8‰ of its next-token mass on correction-related tokens, against below 2‰ on yry_r (Fig. 5, right). We present this as supporting rather than direct evidence (line 765). Locating the failure position is intractable ("would require searching O(∣V∣k)O(|V|^k) continuations of length kk", line 270), hence the surrogate of Eq. (4). Prefix failure follows from the OPD post-hoc training paradigm itself, as shown analytically in Sec. 4.2. It is also a recognized problem in the community (Liao et al. 2026; Xu et al. 2026).

W2. "Confounded comparisons and a baseline discrepancy limit attribution."

  1. "difficult to isolate prefix repair" The submission partially separates these factors in three ways, and we add a fourth. (i) Tab. 5 fixes the teacher and the sampling budget and varies only the supervision trajectory. yry_r lies between the two extremes: yoy_o stays in the student's support but keeps failed prefixes, and y⋆y^\star is correct but may fall outside that support; yry_r is best or tied in 16 of 20 settings. (ii) Best-of-NN repairs failures by selection alone, with the teacher context unchanged (Tab. 6, Fig. 4). It improves with NN, which is consistent with removing failed prefixes helping, but at N=8N=8 it costs 7.8 times TRD's wall-clock and remains below TRD. (iii) On length, y⋆y^\star is the shortest trajectory (Fig. 6) yet not the best, so shorter supervision alone does not explain the gain. (iv) Refining only the failed rollouts already improves over OPD (Table R6 in our response to Reviewer w161), which isolates prefix repair. Together, correctness helps ((ii), (iv)) but is not sufficient ((i)), so the gain requires correct trajectories within the student's support (Eq. (4)).
  2. "differ by only approximately 0.04 percentage points" We agree with this observation. However, this is one of the four macro-averages in Tab. 5 (Qwen3-8B Avg@16, 50.02 vs 50.06). The other three favor TRD: Qwen3-8B Pass@16 (74.0 vs 76.6), Qwen3-4B Avg@16 (37.0 vs 37.7), and Qwen3-4B Pass@16 (59.1 vs 64.1).
  3. "clarification and a faithful implementation are needed" We agree. App. C.2 caps the per-position KL, whereas Zhao et al. (Sec. 3.2) cap each vocabulary-level term. We added their pointwise variant as a baseline on both OPSD settings (Table R1) and will include it in the paper. It performs close to vanilla OPSD and remains below TRD on average on both models.

Table R1. Avg@16 (%) under the privileged self-teacher, with the pointwise clipping of Zhao et al. added.

ModelMethodAIME24AIME25HMMT25BeyondAIMEAMOBenchMean
Qwen3-4B-Instruct-2507Base59.142.928.328.28.133.3
+OPSD60.348.830.832.39.536.3
+OPSD (w/ pointwise clip)60.148.431.232.89.836.5
+TRD63.149.432.132.711.237.7
Qwen3-8BBase72.359.730.535.68.941.4
+OPSD74.865.440.339.615.747.2
+OPSD (w/ pointwise clip)74.765.940.139.916.347.4
+TRD76.569.244.542.817.350.1
AuthorsOpenReview ↗

Rebuttal by Authors to Reviewer BNcF (2/2)

W3 and Q3. "Compute matching and statistical reporting are incomplete."

(a) Compute. "motivating equal-total-GPU-time comparisons" Thank you for this suggestion. We added a vanilla OPD baseline matched to TRD's total GPU time (Table R2). With the extra budget, vanilla OPD does not improve and is slightly below its original version (mean 24.8 vs 25.3), which suggests that for this reasoning model vanilla OPD is already saturated at about 40K training samples. It remains 2.1 points below TRD (26.9).

Table R2. Avg@16 (%) on Qwen3-1.7B with a separate teacher, with vanilla OPD matched to TRD's total GPU time.

ModelMethodAIME24AIME25HMMT25BeyondAIMEAMOBenchMean
Qwen3-1.7BBase44.833.821.220.11.924.4
+OPD44.435.123.321.22.425.3
+OPD (budget-matched)43.135.322.920.92.024.8
+TRD49.437.524.420.13.026.9

(b) Seeds and intervals. "lack repeated training runs and paired confidence intervals" We follow the standard protocol for evaluating reasoning LLMs, which controls variance on the sampling side rather than through training seeds: each score is computed from 16 independent samples per question (Avg@16 or Pass@16), as in the Qwen3 technical report (Yang et al. 2025). The conclusions are also consistent across four student and teacher configurations and five benchmarks.

(c) AMOBench. "the aggregation protocol or reported value needs clarification" Thank you for catching this. The 33.8% entry is our transcription error; the correct value is 13/39 = 33.3%, and we will fix it in the paper. On stability, the solved counts for Qwen3-8B at K=16K=16 are Base 16, vanilla OPSD 20, and TRD 24 of 39 questions, and the coverage gap also holds for Qwen3-4B with a separate teacher as the sampling budget grows to k=128k=128 (Fig. 3).

Q1. "How is the on-policy constraint defined and verified?" The constraint is the support constraint of Eq. (4) (see W1, point 1), and Tab. 5 examines it empirically. Trajectories are refreshed every epoch: yoy_o is sampled from the current student and then refined (Algorithm 1, line 2). The self-teacher is fixed during training: the same base model produces the privileged teacher logits, while LoRA adapters update only the student branch (App. C.2, lines 1050 to 1051). All baselines share this protocol.

Q2. "Can you control teacher context and trajectory length, ...?" Teacher context and trajectory length are outcomes of the refinement and cannot be set independently; all methods share the same maximum response length (16,384 tokens, Tab. 8). At evaluation, for Qwen3-4B-Instruct-2507 with a separate teacher on AMOBench, the students trained with vanilla OPD and TRD produce closely matching correct-rollout length distributions (18.9K vs 18.5K chars on average; Fig. 6, right), so the gain does not come from shorter outputs. The clipping baseline is verified in Table R1 (W2, point 3). For KL versus cross-entropy on the same refined trajectories, we added SFT on yry_r (Table R3): it is on par with SFT on y⋆y^\star (mean 24.1 vs 24.5) and 2.8 points below TRD (26.9), so the gain comes from KL supervision along yry_r, not from the refined trajectories alone.

Table R3. Avg@16 (%) on Qwen3-1.7B (separate teacher): SFT (cross-entropy) versus TRD (KL).

MethodTraj.AIME24AIME25HMMT25BeyondAIMEAMOBenchMean
+SFTy⋆y^\star44.533.322.719.82.124.5
+SFTyry_r44.732.821.119.32.624.1
+TRDyry_r49.437.524.420.13.026.9

Please let us know whether these responses address your concerns, in particular the clarification of the support constraint under W1.

Summary of contributions. (1) We identify prefix failure in OPD (Sec. 4). (2) Token-level remedies leave the failed prefix unchanged; we propose trajectory refinement (Sec. 5). (3) TRD improves math accuracy across four configurations and five benchmarks and remains competitive on code (Tabs. 1 to 4).

References

Kostrikov I, et al. (2022) Offline reinforcement learning with implicit Q-learning. ICLR.

Kumar A, et al. (2019) Stabilizing off-policy Q-learning via bootstrapping error reduction. NeurIPS.

Liao B, et al. (2026) Multi-turn on-policy distillation with prefix replay. arXiv.

Schulman J, et al. (2015) Trust region policy optimization. ICML.

Xu H, et al. (2026) Pass the baton: Trajectory-relayed on-policy distillation. arXiv.

Yang A, et al. (2025) Qwen3 technical report. arXiv.

AuthorsOpenReview ↗

Follow Up Responses to Reviewer BNcF

We would like to share additional results obtained during the discussion period, since they bear on your W1, W2, W3, and Q3.

Training seeds (W3, Q3). We ran three training seeds for OPD and TRD (Table R8, in our follow-up to Reviewer KJ3t). The variation across seeds is small relative to the gap between the two methods, and the seed means closely match the single runs of the submission.

Support constraint (W1, Q1). We measured how often the trajectories lie inside the student's ϵ\epsilon-support, with ϵ\epsilon set so that 95% of yoy_o lie inside: most yry_r lie inside, while most y⋆y^\star lie outside (follow-up to Reviewer KJ3t, points 1 and 2).

Attribution (W2). Table R9 (in our follow-ups to Reviewers KJ3t and w161) compares TRD with closely related methods under the same student, teacher, and data. TRD achieves the best mean. Token-level and segment-level methods keep the failed student prefix as the supervision context, and CGD replaces KL by cross-entropy.

Refinement steps. Table R10 (in our follow-up to Reviewer w161) compares one to three refinement steps: one step is best, and further steps bring no gain and eventually degrade performance.

We hope these additional results are helpful for your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.