acceptodds
Under review as a conference paper at ICLR 2027

StageOPD: Stage-aware Loss Reweighting for On-policy Distillation

Abstract

Current on-policy distillation (OPD) methods mainly improve how teacher supervision is collected on student-generated trajectories, but typically aggregate the resulting token-level losses with uniform or fixed token weights. In long reasoning trajectories, such token-level aggregation favors long routine reasoning spans, while short but informative regions, such as correction boundaries, teacher-guided spans, and the student’s immediate continuation after guidance, receive much less optimization weight simply because they contain fewer tokens. We introduce STAGEOPD, a simple stage-aware loss-reweighting method for on- policy distillation. STAGEOPD decomposes each trajectory into routine student reasoning, correction boundaries, teacher-guided spans, and post-guidance recovery, assigns an explicit loss budget to each stage, and normalizes the resulting weights within each sequence so that the total optimization weight of a stage is determined by its role rather than its token count, without requiring additional rollout or teacher computation. Experiments with a Qwen3-1.7B-Non-Thinking student and a Qwen3-4B-Instruct-2507 teacher on DAPO-Math-17K show that STAGEOPD achieves 45.58 ± 0.08 average accuracy across eight mathematical reasoning benchmarks, outperforming the strongest baseline by 0.11 percentage points. Ablation studies further show that removing the student-reasoning anchor, the teacher-guidance budget, or sequence-level weight normalization reduces average accuracy by 1.98, 1.68, and 1.08 points, respectively; a Recovery-window study further achieves its best result at 64 tokens. These results demonstrate the effectiveness of stage-aware loss allocation for on-policy distillation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
on policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d2LtOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the work proposed a new loss design for OPD, demonstrated that stage-aware reweighting outperforms fixed or uniform weighting. The experiments on distillation for training of student models, authors also conducted systematic ablation studies. The improvement was demonstrated on math benchmarks, with 0.11 points stronger than the best baseline.

Critical strengths

The stage-aware design of the training strategy is reasonable and the authors conducted careful research, carried out detailed experiments, clearly demonstrating the performance improvement.

Critical weaknesses

the improvement over baseline is very limited. The authors are encouraged to demonstrate other advantages of this specially designed loss, or explain why the improvement is so small, whether this is due to some choices in this work, such as models? If this level of improvement is hitting the ceiling, then does it worthwhile to further pursue this approach?

Questions for authors
  1. How to define the boundaries of the trajectories and why four stages.
  2. Whether the chosen models are suitable for this study, are the improvements in performance depending on the choice of the models?
  3. What are the computing costs? Additional evaluation of the weighting parameters at token level will add to the total cost of the computing? By how much?
Reviewer_koMSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a role-based weighting method for on-policy distillation, assigning learning weights to predefined roles within supervision traces.

Critical strengths

The paper tackles an important issue: short but critical segments of supervision traces may receive insufficient weight during training. However, this issue is not specific to on-policy distillation, and the specific connection to that setting is insufficiently established.

Critical weaknesses
  • Implicit and strong assumption:

    • The method assumes that supervision traces can be segmented into predefined roles, which may not the case. The segmentation of each role itself may also require auxiliary supervision.
  • Limited novelty:

    • The method still needs to allocate a pre-defined budget csc_s, analogous to pre-defined re-weighting schemes.
    • Although there is on-policy distillation in the title, the paper is very loosely connected to that concept.
  • Performance: 0.11 percent over baseline is a marginal improvement. The budget choices are insufficiently justified. How generalizable this method can be applied to other data/model settings remains unresolved.

  • Writing:

    • This paper is written with scrambled logic chains and is difficult to follow.
    • Figures are barely readable with very tiny fonts.

Overall, I do not consider the submission to meet the acceptance bar for ICLR.

Questions for authors

How much of the reported improvement comes from normalizing by stage length rather than the hand-chosen budgets?

2 comments

Thanks for authors' response which partially addressed my questions. However, I still have remaining concerns: 1 about the novelty: why or whether it is significantly different from predefined reweighing methods? 2 the quality of writing and figure presentation, 3 and the marginal performance improvement over prior art.

Overall, I intend to maintain my rating.

Thank you for engaging with our response. We address your three remaining points directly.

1. How StageOPD differs from predefined reweighting. The two methods fix different quantities.

  • A predefined token weight αs\alpha_s fixes the per-token importance. A stage's total share of the loss is then nsαs/∑unuαun_s\alpha_s / \sum_u n_u\alpha_u, so it still depends on how many tokens the stage has. In our traces the Trigger stage receives only 0.09% of the loss despite a 4× weight (Table 2).
  • A stage budget csc_s fixes the stage's total share, cs/∑ucuc_s / \sum_u c_u, in every response, whatever its length (Eq. 16), while keeping the mean token weight at 1 (Eq. 17).

In other words, fixed token weights act like stage budgets that grow and shrink with stage length; StageOPD removes that dependence.

This matters specifically in on-policy distillation, because stage lengths change as the student learns. Teacher tokens fall from 10.98% to 1.75% of all tokens during training (Table 10). Under a fixed αT\alpha_T, the Teacher Body's share of the loss shrinks with them. Under StageOPD it stays at 12.8–14.7% across steps 66–80 (Table 9). A single fixed α\alpha cannot hold that share constant, because the right value would have to change for every response.

The gradient view (App. B.2) gives a second way to see the difference: each stage contributes its average token gradient, and the budgets mix these stage-level averages. This is closer to per-task loss balancing in multi-task learning than to per-token reweighting.

The experiments are consistent with this:

  • Tuning the fixed weights over 27 configurations reaches 45.19, still below StageOPD's 45.58 (Table E2).
  • Setting the budgets equal to the same average shares that the fixed weights produce already gives +0.17 (Table E7b). So delivering a stage's share per response, rather than only on average, helps on its own.

2. Writing and figures. In the revision we:

  • redrew Figures 1–4 with fonts of at least 9pt and removed the duplicated overlay text;
  • rebuilt Figure 1 to show directly how the two weighting rules allocate the loss, using measured shares (4.7% vs. 18.9% for the three short stages);
  • stated the link to on-policy distillation in the introduction and moved the gradient derivation to the appendix.

If a particular section or figure is still hard to follow, we would be grateful if you could point us to it, and we will revise it.

3. Size of the improvement over prior art. We agree that the 0.11-point margin over FastOPD is not significant; the revision now says so explicitly (Sec. 4.2). However, FastOPD and StageOPD change different parts of the pipeline:

  • FastOPD changes how trajectories are built;
  • StageOPD changes how the losses on those trajectories are combined.

The fair test for an aggregation method is therefore to hold the trajectories fixed. Under that test:

  • StageOPD improves over fixed token weights by +0.59 points, 95% CI [0.31, 0.87] (Table E1);
  • it improves over uniform per-token weighting by +0.86 (Table E2);
  • it adds +0.50 inside the official Relay-OPD pipeline (Table E2);
  • the gain is similar across model pairs and trajectory controllers (+0.53 to +0.62, Table E6).

Because the two methods are complementary, combining them gives 45.93, which is +0.49 over FastOPD with a 95% CI of [0.22, 0.77] (Table E1). StageOPD therefore improves on prior art when added to it, rather than competing with it, at +0.3% step time.

We hope this clarifies the distinction. If it addresses your concerns, we would sincerely appreciate it if you would reconsider your rating, and we are happy to continue the discussion.

Reviewer_X1H3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StageOPD, a stage-aware loss reweighting method for on-policy distillation. It divides reasoning trajectories into four stages (Normal, Trigger, Teacher Body, and Recovery) and assigns different loss budgets to each stage. The method requires no additional teacher computation. Experiments on eight mathematical reasoning benchmarks show an average accuracy of 45.58, slightly outperforming FastOPD (45.47). Ablation studies are also provided to evaluate the main components.

Critical strengths
  1. StageOPD introduces a lightweight modification to the existing OPD loss without additional teacher inference or rollouts. This makes it easy to integrate into existing training pipelines.
  2. The paper identifies a reasonable limitation of token-level loss aggregation in long reasoning trajectories. The stage-level weight allocation analysis and component ablations help explain the behavior of the proposed method.
Critical weaknesses
  1. StageOPD only improves over FastOPD by 0.11 points (45.58 vs. 45.47), which is small relative to run-to-run variation, and does not consistently outperform it across benchmarks. The fixed-token comparison uses weights of (1, 4, 2, 1.5), but their selection is not justified. Similarly, the proposed stage budgets lack sufficient explanation, and the implementation of SARD-FULL is unclear. Without properly tuned baselines and weight sensitivity analysis, it is difficult to determine whether the gains come from the proposed method or specific hyperparameter choices.

  2. StageOPD builds on Relay-OPD [1], mainly modifying its loss aggregation. Moreover, the method relies on predefined reflection tokens and is evaluated only on one Qwen3 teacher-student pair and mathematical reasoning tasks. Its effectiveness across different models and trajectory construction methods remains unclear.

[1] Xu H, Xu X, Hong H, et al. Pass the baton: Trajectory-relayed on-policy distillation[J]. arXiv preprint arXiv:2607.26057, 2026.

Questions for authors
  1. The improvement over FastOPD is only 0.11 points. Could the authors provide additional evidence that this improvement is statistically significant?
  2. How were the fixed-token weights (1, 4, 2, 1.5) and the proposed stage budgets selected? Could the authors clarify the implementation of SARD-FULL and provide sensitivity analysis or comparisons with better-tuned fixed weights?
  3. Since StageOPD builds on Relay-OPD, could the authors provide a direct comparison under the same experimental settings to better demonstrate the contribution of the proposed loss reweighting?
  4. How sensitive is the method to the predefined reflection-token set? Could the authors evaluate whether StageOPD generalizes to different model families or trajectory construction methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.