- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FiRe-OPD combines trajectory filtering and token reweighting for on-policy distillation. It discards student rollouts with low length-normalized teacher likelihood, then scales token advantages using teacher confidence and student uncertainty. The paper evaluates strong-to-weak, same-size single-teacher, and math/code multi-teacher distillation with a Qwen3-4B student, and reports component and hard/soft ablations.
Critical strengths
- FiRe-OPD provides a simple way to select student trajectories and redistribute supervision within them without an external verifier. The component ablation supports combining the two levels: the full method reaches 60.83, compared with 59.30 for trajectory filtering alone.
- The experiments cover three distillation relationships and report improved aggregate performance in each. The multi-teacher code result is particularly encouraging, with an average of 64.16 versus ExOPD's 62.53.
Critical weaknesses
- The paper does not sufficiently establish the specific limitation of existing selective OPD methods that its combination resolves. TIP (Xu et al., 2026) already considers student uncertainty and teacher–student disagreement, while SCOPE (Zheng et al., 2026) uses teacher perplexity to adjust trajectory supervision. FiRe's verifier-free, two-level design differs, but the introduction presents separate treatment of the two granularities as a key limitation without directly diagnosing its consequences. The method then treats low teacher likelihood as evidence of unreliable supervision, although this score alone does not establish either teacher–student divergence or correction reliability. Appendix B supports the default filtering rule empirically, but does not establish the proposed explanation in terms of noisy gradients. These gaps weaken the claimed insight beyond a combination of existing selection and weighting ideas.
- The experiments support the tested recipe more clearly than the claimed optimization principle or a robust advantage over strong alternatives. Table 6 compares all four hard/soft combinations, but the manuscript does not specify the alternative trajectory-weighting function, token-selection threshold, normalization, or comparable tuning effort. Their ranking therefore cannot be attributed confidently to granularity alone. Against the strongest reported aggregate baseline in each setting, the gains are 0.67 for strong-to-weak math, 0.53 for single-teacher math, 0.85 for multi-teacher math, and 1.63 for code. No independent training variation or validation-based parameter-selection procedure is reported. Avg@8 averages sampled answers and does not establish robustness across training runs.
References
- Hou, W., Peng, S., Wang, W., et al. (2026). Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe. arXiv preprint arXiv:2605.03677.
- Xu, Y., Sang, H., Zhou, Z., et al. (2026). TIP: Token Importance in On-Policy Distillation. arXiv preprint arXiv:2604.14084.
- Yang, W., Liu, W., Xie, R., et al. (2026). Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv preprint arXiv:2602.12125.
- Zheng, B., Ma, X., Liang, Y., et al. (2026). SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting. arXiv preprint arXiv:2604.10688.
Questions for authors
- Could the authors provide evidence relating teacher likelihood to correction reliability or the advantage distribution on retained and discarded trajectories, to substantiate the interpretation of filtering?
- Could the authors provide the exact alternative objectives and selection rates in Table 6, including normalization and how tuning and optimization budgets were made comparable?
- Could the authors clarify parameter and checkpoint selection, provide uncertainty for the main baseline comparisons, and explain why Uni-OPD (Hou et al., 2026) is listed in §4.1 but absent from the result tables?
3 comments
Response to q2
The exact definitions are summarized below. Here, denotes teacher confidence and denotes student confusion.
| Level | Mode | Mathematical form | Implementation / threshold |
|---|---|---|---|
| Trajectory | Hard | traj_weight = 1{norm_teacher_logprob >= τ_traj} | Bottom N% discarded; the rest receive weight 1. τ_traj is the batch-level traj_skip_percentile (default N=20%). |
| Trajectory | Soft | traj_weight = sigmoid(k * (norm_teacher_logprob - τ_traj)) | k=10 (default traj_soft_k); uses the same τ_traj as the hard gate, matching the cutoff location. |
| Token | Soft | token_weight = (1 + α * tc) * (1 + β * sc) | α=β=1.0; tc is teacher confidence and sc is student confusion. |
| Token | Hard | token_mask = 1{tc > τ_tc and sc > τ_sc} | τ_tc and τ_sc are batch-level 70th percentiles, chosen so that the retained fraction is comparable to the soft gate's effective mass. |
The four groups A (Hard+Soft), B (Soft+Soft), C (Hard+Hard), and D (Soft+Hard) share the same optimizer, batch size, rollout budget, and evaluation protocol. The details of the soft/hard selection ablations have been added to the appendix.
Response to q4
Parameter and checkpoint selection. Details are in §4.1.
Uni-OPD. We acknowledge Uni-OPD as valuable concurrent work that presents a systematic dual-perspective framework for improving student exploration and teacher supervision, supported by extensive evaluations across tasks and modalities. Uni-OPD combines data balancing with outcome-guided margin calibration, whereas FiRe-OPD focuses on verifier-free supervision allocation through teacher-likelihood-based trajectory filtering and soft token-level reweighting. The two methods therefore address related challenges from different methodological perspectives. Due to computational constraints, we did not evaluate Uni-OPD experimentally. Its inclusion in the list of evaluated baselines was a writing error; we will correct this and retain a discussion of its contributions and differences from our method in the related work.
Response to W2
We report three independent test runs:
| Benchmark | Seed A | Seed B | Seed C | Mean ± Std (n=3) |
|---|---|---|---|---|
| AIME24 | 59.17 | 62.08 | 60.83 | 60.69 ± 1.46 |
| AIME25 | 53.33 | 51.62 | 52.92 | 52.62 ± 0.89 |
| MATH500 | 93.65 | 93.77 | 93.73 | 93.72 ± 0.06 |
| AMC2023 | 92.50 | 93.52 | 93.13 | 93.05 ± 0.51 |
| OlympiadBench | 70.38 | 70.38 | 70.47 | 70.41 ± 0.05 |
| MinervaMATH | 43.01 | 43.80 | 43.47 | 43.43 ± 0.40 |
| HMMT Feb | 32.50 | 30.42 | 32.08 | 31.67 ± 1.10 |
| HMMT Nov | 39.33 | 41.25 | 40.00 | 40.19 ± 0.97 |
| Average | 60.48 | 60.86 | 60.83 | 60.72 ± 0.21 |
Three-seed mean 60.72 ± 0.21, all three runs above OPD (58.70) and ExOPD (60.16), margins +2.02 and +0.56 (95% CI [60.21, 61.23]).
Thanks for the detailed rebuttal and new experiments. Clarifying the roles of the trajectory filter and token weights—along with the diagnostics showing higher correctness (84.6% vs. 33.3%) and lower gradient variance—gives much stronger empirical motivation for the filter, and the note on Uni-OPD resolves my baseline question.
However, two main concerns keep me at a 2 (Weak Reject):
-
Generalizability of Table 6: Switching from soft trajectory weighting to hard filtering helps under soft token weights (58.68 → 60.83) but slightly hurts under hard token selection (58.55 → 58.23). While this supports your setup here, it leaves open when this design transfers elsewhere. Reporting the exact fraction of retained tokens and loss normalization details would help isolate the effect of supervision volume.
-
Significance of the strong-to-weak gains: While FiRe-OPD's three runs (60.72 ± 0.21) beat ExOPD (60.16), "independent test runs" is ambiguous as to whether seeds varied during training or only evaluation. Clarifying this and reporting baseline variance under the same protocol is needed to gauge that 0.56-point margin.
We thank the reviewer for the detailed assessment and the constructive suggestions on methodological positioning, experimental controls, and the transparency of model selection and baseline reporting.
Response to W1
The contribution is not “using two confidence signals,” but allocating supervision at two different units: trajectory gating decides whether a complete student path participates in updates; token weighting decides the relative emphasis within a retained path, without binary removal of individual tokens. The two operations serve different purposes and are not interchangeable.
The difference from existing concurrent methods is concrete. The concurrent work TIP performs hard token selection based on student uncertainty and teacher–student divergence; other concurrent work SCOPE routes trajectories by correctness and applies teacher-perplexity-weighted KL on incorrect trajectories. FiRe-OPD differs in two respects: (i) at the trajectory level, it filters only by teacher likelihood, without an external verifier; (ii) at the token level, it uses soft multiplicative reweighting that preserves all token gradients, whereas TIP irreversibly discards tokens. The 2×2 factorial supports this choice: hard token selection (C, 58.23) is substantially worse than soft token reweighting (A, 60.83), especially on AIME25 (+6.67) and HMMT Nov (+6.25).
Response to q1
Teacher likelihood measures the teacher’s relative support for a student-generated path; it is not an answer-correctness classifier. At a fixed prefix, expected teacher NLL decomposes into student entropy plus student-to-teacher KL, so low likelihood reflects distributional mismatch rather than a quality label.
We directly measured what the filter discards on a batch of student-generated trajectories. We applied the length-normalized 20th-percentile rule to trajectories from the initial Qwen3-4B, scored on identical tokens by Qwen3-30B-A3B-Instruct-2507 (temperature/top-p 1.0/1.0, response cap 16,384).
The correct-discard rate is 8.3%; 91.7% of correct trajectories are retained. The retained group has higher accuracy (84.6% vs. 33.3%), and discarded trajectories carry more negative and more dispersed token advantages. This demonstrates that the filter mainly removes incorrect trajectories whose supervision is more negative and less consistent, while also ensuring that the retained set keeps a high accuracy.
We directly measured gradients on the same trajectory pool used for the selection diagnostic. We drew four microbatches from the full pool and four from the likelihood-retained pool, using whole trajectories with matched token counts (paired difference below 5%).
The mean gradient norm drops from 5.679 to 5.249, and the gradient variance trace.
E∥g∥2−∥Eg∥2
drops from 5.570 to 2.152—a 61.4% reduction on the retained pool. After filtering, microbatch gradients fluctuate substantially less. This matches the design motivation: the trajectories removed by the filter are exactly those whose updates are larger and less consistent, so dropping them stabilizes the local update.
We include the above findings in Appendix. We have also revised the relevant discussion in the main text to clarify the rationale for trajectory filtering. Specifically, we frame teacher likelihood as a signal of teacher–student alignment along a student-generated path, rather than a direct measure of answer correctness or supervision quality. We also connect this motivation to the gradient analysis in Appendix, which shows lower local gradient variation on the retained pool. This revision more clearly explains the motivation and empirical support for allocating supervision differently across trajectories instead of supervising all paths uniformly.