- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Step-wise On-policy Distillation (SOD) to improve tool-integrated reasoning (TIR) in small language models. The authors identify a limitation of conventional on-policy distillation (OPD): erroneous tool calls can cause cascading state divergence between the student and teacher, potentially making subsequent teacher supervision unreliable. To address this issue, SOD introduces a step-wise reweighting mechanism that adjusts distillation strength based on student–teacher divergence. Specifically, it reduces supervision in highly divergent steps while restoring stronger guidance when the student realigns with the teacher. The proposed method combines this adaptive distillation objective with GRPO to leverage both dense token-level supervision and outcome-level rewards. Experiments with Qwen3-0.6B and Qwen3-1.7B students on mathematical reasoning, scientific question answering, and coding benchmarks demonstrate consistent improvements over several RL and distillation baselines. Additional experiments in agentic environments and ablation studies further investigate the effectiveness and generalizability of the proposed approach.
Critical strengths
-
The paper addresses the student–teacher distribution shift caused by error accumulation during multi-turn tool use. This is an important challenge for on-policy distillation of small language models, and designing a distillation strategy tailored to tool-interaction steps is a reasonable approach.
-
The experimental evaluation is comprehensive, with clear performance improvements. The authors evaluate their method at two model scales across four major benchmarks and compare it against related approaches, including OPD, TIP, and SCOPE. The results show relatively consistent improvements. Additional experiments in other agent environments are provided in the appendix, further strengthening the empirical evidence.
-
The method is simple and inexpensive to implement. SOD computes step-wise weights using existing student–teacher log probabilities, without requiring additional model forward passes. The ablation studies and intervention experiments on low-weight steps provide further support, suggesting that the performance gains are not merely attributable to increased training complexity.
Critical weaknesses
-
The reliability of the GRPO baseline is questionable. Figure 3 shows severe entropy collapse for GRPO, and Table 1 indicates that both model sizes perform worse than their SFT counterparts on AIME2025. These observations suggest that the GRPO training setup may suffer from substantial stability issues. Prior work, such as SimpleTIR, has proposed stabilization techniques for multi-turn tool use, yet the authors do not compare against these methods. Consequently, some of the reported improvements may stem from an insufficiently optimized baseline rather than the effectiveness of SOD itself.
-
The core assumption is insufficiently validated. SOD uses student–teacher divergence as a proxy for the reliability of teacher supervision. However, a larger distributional discrepancy does not necessarily imply that the teacher's supervision is unreliable; it may instead reflect valuable corrective signals from the teacher. Moreover, the derivation in Appendix G.5 does not establish that the proposed divergence score is monotonically consistent with the true KL divergence. This leaves the theoretical justification for the weighting mechanism insufficiently substantiated.
-
The algorithmic novelty is relatively limited. SOD essentially applies step-wise reweighting to the existing OPD loss. A closer examination of Eq. (7) reveals that its multiplicative form can be simplified to a weight determined solely by the divergence at the first step and the current step, rather than genuinely modeling the full history of divergence changes. The paper also lacks a direct comparison with simpler step-wise divergence-based weighting schemes. Consequently, the unique advantages of the proposed design over standard adaptive weighting approaches remain unclear.
Questions for authors
-
GRPO baseline stability. In Figure 3, GRPO exhibits a sharp entropy collapse, while its AIME 2025 accuracy is even lower than the SFT baseline for both model sizes. This raises concerns about whether GRPO was sufficiently optimized. Have the authors investigated the cause of this collapse, particularly the gradient behavior and tool-calling failures?
-
Comparison with stabilized GRPO. Previous work, such as SimpleTIR, has identified similar instability in multi-turn tool-integrated RL and proposed effective stabilization techniques. Have the authors considered these techniques for the GRPO baseline? It would be important to understand whether SOD maintains its advantage over a properly stabilized GRPO implementation.
-
Checkpoint selection. Figure 3 shows substantial performance degradation in later training stages, particularly for the 1.7B OPD baseline. Were the results in Table 1 obtained from the final checkpoint or the best validation checkpoint? Was the same selection criterion applied consistently across all methods?