- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows and addresses a limitation of OPSD, where KD loss is computed along student-generated trajectories that may contain incorrect reasoning steps, arguing that incorrect prefixes can lead to fragmented or misleading supervision signals. To address this problem, the authors propose Trajectory-Refined Distillation (TRD). Instead of directly distilling on the student's sampled trajectory, the method asks the teacher to revise it and then performs logit KD along the refined trajectory.
Critical strengths
- Simple proposed approach.
- Good experimental results show the effectiveness of TRD.
Critical weaknesses
-
The idea of refining student-generated trajectories using teacher feedback before training has been explored in prior work, including Critique-Guided Distillation (ICML 2026) and Multi-Granularity Semantic Revision (ACL 2026). These closely related approaches are not discussed or experimentally compared. Although TRD uses a different KL-based training objective, its specific novelty relative to these methods is insufficiently established.
-
Although the authors provide some justification, TRD generates refined trajectories using the teacher rather than the student. Therefore, these trajectories are not on-policy.
Questions for authors
- See weaknesses.
- Why is refinement necessary for all trajectories rather than selectively correcting incorrect ones? Additionally, how do the authors determine whether a single refinement step is sufficient?
4 comments
Thank you for the response. The failed-only refinement ablation addresses my question about the need to refine both correct and incorrect trajectories. However, my main concern regarding novelty remains unresolved. The cited works are closely related to TRD, as they share the same central idea of refining student-generated trajectories using teacher feedback before training. Although the authors clarify the differences in refinement mechanisms and training objectives, it remains unclear whether these distinctions constitute a sufficiently significant contribution. Moreover, without direct experimental comparisons with these closely related methods, it is difficult to assess what additional benefit TRD offers over existing approaches.
I appreciate the authors' acknowledgment that the refined trajectories are not strictly on-policy, although the clarifications have not yet been incorporated into a revised manuscript. Regarding the number of refinement steps, while the authors show that a single step improves trajectory correctness, they do not provide experiments comparing different numbers of refinement steps. Therefore, it remains unclear whether one step is sufficient or whether additional refinement could yield further improvements.
Overall, while the additional experiment addresses one of my concerns, the main issue regarding novelty remains insufficiently resolved. I therefore maintain my original score of 2.
Follow Up Responses to Reviewer w161
Thank you for the follow-up. We address the remaining points.
1. Novelty. We agree that refining student trajectories with teacher feedback is shared with Kapusuzoglu et al. (2026) and Liu et al. (2026). Our contribution is not refinement by itself. It is (i) the analysis of prefix failure, which explains why dense KL on a failed student prefix does not teach the correction path (Sec. 4), and (ii) the resulting design: the refined complete trajectory serves as the context of full-vocabulary KL in OPD, including privileged self-distillation. These distinctions determine the outcome: under the same setup, both cited methods fall below TRD (Table R9).
2. Direct comparisons. Table R9 compares TRD with both families, and with TOPD raised by Reviewer KJ3t. TRD achieves the best mean; SCRG is the closest, and TOPD is on par with vanilla OPD. The ordering follows from what each method changes. Token-level correction (SKD, SCRG) and segment-level guidance (TOPD) keep the earlier student prefix as the supervision context, so the KL is still evaluated on a failed prefix. CGD rewrites the whole trajectory but trains with cross-entropy, which discards the teacher distribution and falls below OPD. TRD is the only method that both replaces the context with a refined complete trajectory and keeps full-vocabulary KL.
Table R9. Avg@16 (%) on Qwen3-1.7B with a separate teacher, under the same student, teacher, and data. TOPD has no official code and is our reimplementation from the paper.
| Method | Teacher correction | Setup | Loss | AIME24 | AIME25 | HMMT25 | BeyondAIME | AMOBench | Mean |
|---|---|---|---|---|---|---|---|---|---|
| +OPD | none | OPD | KL | 44.4 | 35.1 | 23.3 | 21.2 | 2.4 | 25.3 |
| +SKD | token | OPD | KL | 40.2 | 31.5 | 19.6 | 15.5 | 1.4 | 21.6 |
| +SCRG (Liu et al. 2026) | token | OPD | KL | 45.1 | 35.7 | 24.2 | 20.9 | 3.0 | 25.8 |
| +CGD (Kapusuzoglu et al. 2026) | trajectory | SFT | CE | 40.9 | 31.4 | 19.2 | 17.7 | 1.2 | 22.1 |
| +TOPD (Jiang and Ferraro 2026) | segment (50 tokens) | OPD | KL + OT | 46.2 | 34.7 | 21.6 | 21.2 | 2.4 | 25.2 |
| +TRD | trajectory | OPD | KL | 49.4 | 37.5 | 24.4 | 20.1 | 3.0 | 26.9 |
3. Number of refinement steps. Table R10 compares zero to three refinement steps and reports the trajectory statistics together with the final performance. One step gives the best final performance. A second step adds little correctness and does not improve performance. A third step degrades the trajectories, which become much shorter and less correct, and performance drops below OPSD. One step is therefore sufficient: additional refinement brings no further gain and eventually over-compresses the trajectories. Each additional step also requires another teacher sampling pass over the training set, so the computation cost grows with the number of steps (Tab. 12). The reference , although fully correct, lies mostly outside the student's support and is also below one step.
Table R10. Refinement steps on Qwen3-4B-Instruct-2507 with a privileged self-teacher. is set so that 95% of lie inside the -support (see Table R4).
| Steps | Verifier pass (%) | NLL/token | Mean length | In -support (%) | Mean Avg@16 (%) |
|---|---|---|---|---|---|
| 0 (, OPSD) | 81.3 | 0.186 | 2,777 | 95 | 36.3 |
| 1 (, TRD) | 88.3 | 0.565 | 996 | 67 | 37.7 |
| 2 | 90.2 | 0.593 | 1,022 | 71 | 37.4 |
| 3 | 87.6 | 0.541 | 678 | 77 | 33.9 |
| (reference) | 100 | 0.972 | 560 | 20 | 37.0 |
4. Revised manuscript. We will revise the manuscript accordingly: the "support constraint" wording in Sec. 5, the discussion of Kapusuzoglu et al. (2026) and Liu et al. (2026) in Sec. 2, and Tables R5 to R10.
We hope these results address the remaining concerns.
I thank the authors for the additional experiments and clarifications. The new results strengthen the empirical evaluation of the paper. However, after reconsidering the proposed method compared to the related work, novelty remains my main concern Teacher-guided refinement of student-generated trajectories is already explored in prior work, including Critique-Guided Distillation (ICML 2026) and Multi-Granularity Semantic Revision (ACL 2026). Moreover, TOPD, as highlighted by another reviewer, already investigates trajectory-level guidance in on-policy distillation. While TRD differs by applying full-vocabulary KL along complete teacher-refined trajectories, it remains unclear whether this combination constitutes a substantial methodological contribution. The improvement over the strongest related baseline (SCRG) is also relatively marginal (+1.1). That is why I maintain my original score of 2.
Rebuttal by Authors to Reviewer w161
We sincerely thank Reviewer w161 for the insightful comments. The two related works and the question on selective refinement helped us sharpen the positioning of TRD.
W1. "These closely related approaches are not discussed or experimentally compared." We clarify that Kapusuzoglu et al. (2026) refine at the trajectory level, but in SFT rather than OPD. Liu et al. (2026) can be viewed as OPD, but their correction is at the token level: one flagged token is replaced and the student regenerates the rest, so the earlier prefix is kept. As argued in the submission (lines 65 to 66), recovering from prefix failure "requires a trajectory-level improvement". We will cite and discuss both in Sec. 2. Table R5 summarizes them. To our knowledge, TRD is the first trajectory-level refinement in the OPD setup.
Table R5. Comparison with related works.
TRD is also well motivated rather than a heuristic: it follows from our analysis of prefix failure, which is inherent in the post-hoc supervision paradigm of OPD (Sec. 4). Prefix failure is also a recognized problem in the community (Liao et al. 2026; Xu et al. 2026).
W2. "these trajectories are not on-policy." We agree that yr is not an on-policy sample, and TRD does not require it to be. (1) We require the support constraint of Eq. (4) (lines 271 to 276), in the sense of the RL literature on trust regions and support constraints (Schulman et al. 2015; Kumar et al. 2019; Kostrikov et al. 2022): the constructed trajectories should stay in the region where the student has support (Fig. 1, left). Formally, the student's density along the trajectory is bounded below (see Table R4, response to KJ3t). Eq. (4) is a construction target, not strictly enforced: yr may fall outside the support, as our Limitations note. We will state this in Sec. 5 and correct our imprecise "on-policy" wording throughout. (2) Strict on-policy sampling is also not required: distilling on fixed offline student rollouts can match or outperform fully on-policy distillation (Wu et al. 2026; Zhao et al. 2026).
Q1. "Why is refinement necessary for all trajectories?" (1) A correct final answer does not imply a correct reasoning path, and locating failed prefixes is intractable (line 270), so correct rollouts cannot be assumed free of prefix failure. (2) Refining a correct yo can expose new reasoning paths that the student rarely samples while maintaining its support (Sec. 5), which improves exploration and Pass@k (Sec. 6.3; Table R7). (3) We added a failed-only variant (Table R6): it improves over OPD (mean 36.7 vs 35.8) but is below TRD (37.9), so refining both failed and correct rollouts matters. The same holds for Pass@128 (Table R7).
Table R6. Failed-only refinement on Qwen3-4B-Instruct-2507 (separate teacher, %).
Table R7. Pass@k (%) on AMOBench, same setting (Fig. 3).
Q2. "whether a single refinement step is sufficient?" One step raises the verifier pass rate from 65.8% to 81.4% on Qwen3-8B (Fig. 6). More steps may raise it, but each step moves the trajectory farther from yo and may fail to maintain the student's support: one step raises the per-token NLL (Table R4). We thus use a single step as the trade-off in Eq. (4): mitigating prefix failure while staying within the student's support.
References
Kapusuzoglu B, et al. (2026) Critique-guided distillation. ICML.
Kostrikov I, et al. (2022) Offline RL with implicit Q-learning. ICLR.
Kumar A, et al. (2019) Stabilizing off-policy Q-learning. NeurIPS.
Liao B, et al. (2026) Multi-turn OPD with prefix replay. arXiv.
Liu X, et al. (2026) Multi-granularity semantic revision. ACL.
Schulman J, et al. (2015) Trust region policy optimization. ICML.
Wu Y, et al. (2026) Lightning OPD. arXiv.
Xu H, et al. (2026) Pass the baton. arXiv.
Zhao S, et al. (2026) When do we need on-policy distillation? arXiv.