acceptodds
Withdrawn from ICLR 2027

SOD: Step-wise On-Policy Distillation for Small Language Model Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-integrated reasoning (TIR) is difficult to scale to small language models due to instability in long-horizon tool interactions and limited model capacity. While reinforcement learning methods like group relative policy optimization provide only sparse outcome-level rewards. Recently, on-policy distillation (OPD) has gained popularity by supplying dense token-level supervision from a teacher on student-generated trajectories. However, our experiments indicate that applying OPD to TIR leads to a critical failure mode: erroneous tool calls tend to cascade across subsequent reasoning steps, progressively amplifying student-teacher divergence and rendering the teacher's token-level supervision increasingly unreliable. To address this, we propose SOD, a step-wise on-policy distillation framework for small language model agents, which adaptively reweights distillation strength at each step based on step-level divergence. Therefore, SOD can attenuate potentially misleading teacher signals in high-divergence regions while preserving dense guidance in well-aligned states. Experiments on challenging math, science, and code benchmarks show that SOD achieves up to 20.86% improvement over the second-best baseline. Notably, our 0.6B student achieves 26.13% on AIME 2025, demonstrating effective transfer of agentic reasoning to lightweight models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Agent Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9uSNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Step-wise On-policy Distillation (SOD) to improve tool-integrated reasoning (TIR) in small language models. The authors identify a limitation of conventional on-policy distillation (OPD): erroneous tool calls can cause cascading state divergence between the student and teacher, potentially making subsequent teacher supervision unreliable. To address this issue, SOD introduces a step-wise reweighting mechanism that adjusts distillation strength based on student–teacher divergence. Specifically, it reduces supervision in highly divergent steps while restoring stronger guidance when the student realigns with the teacher. The proposed method combines this adaptive distillation objective with GRPO to leverage both dense token-level supervision and outcome-level rewards. Experiments with Qwen3-0.6B and Qwen3-1.7B students on mathematical reasoning, scientific question answering, and coding benchmarks demonstrate consistent improvements over several RL and distillation baselines. Additional experiments in agentic environments and ablation studies further investigate the effectiveness and generalizability of the proposed approach.

Critical strengths
  1. The paper addresses the student–teacher distribution shift caused by error accumulation during multi-turn tool use. This is an important challenge for on-policy distillation of small language models, and designing a distillation strategy tailored to tool-interaction steps is a reasonable approach.

  2. The experimental evaluation is comprehensive, with clear performance improvements. The authors evaluate their method at two model scales across four major benchmarks and compare it against related approaches, including OPD, TIP, and SCOPE. The results show relatively consistent improvements. Additional experiments in other agent environments are provided in the appendix, further strengthening the empirical evidence.

  3. The method is simple and inexpensive to implement. SOD computes step-wise weights using existing student–teacher log probabilities, without requiring additional model forward passes. The ablation studies and intervention experiments on low-weight steps provide further support, suggesting that the performance gains are not merely attributable to increased training complexity.

Critical weaknesses
  1. The reliability of the GRPO baseline is questionable. Figure 3 shows severe entropy collapse for GRPO, and Table 1 indicates that both model sizes perform worse than their SFT counterparts on AIME2025. These observations suggest that the GRPO training setup may suffer from substantial stability issues. Prior work, such as SimpleTIR, has proposed stabilization techniques for multi-turn tool use, yet the authors do not compare against these methods. Consequently, some of the reported improvements may stem from an insufficiently optimized baseline rather than the effectiveness of SOD itself.

  2. The core assumption is insufficiently validated. SOD uses student–teacher divergence as a proxy for the reliability of teacher supervision. However, a larger distributional discrepancy does not necessarily imply that the teacher's supervision is unreliable; it may instead reflect valuable corrective signals from the teacher. Moreover, the derivation in Appendix G.5 does not establish that the proposed divergence score is monotonically consistent with the true KL divergence. This leaves the theoretical justification for the weighting mechanism insufficiently substantiated.

  3. The algorithmic novelty is relatively limited. SOD essentially applies step-wise reweighting to the existing OPD loss. A closer examination of Eq. (7) reveals that its multiplicative form can be simplified to a weight determined solely by the divergence at the first step and the current step, rather than genuinely modeling the full history of divergence changes. The paper also lacks a direct comparison with simpler step-wise divergence-based weighting schemes. Consequently, the unique advantages of the proposed design over standard adaptive weighting approaches remain unclear.

Questions for authors
  1. GRPO baseline stability. In Figure 3, GRPO exhibits a sharp entropy collapse, while its AIME 2025 accuracy is even lower than the SFT baseline for both model sizes. This raises concerns about whether GRPO was sufficiently optimized. Have the authors investigated the cause of this collapse, particularly the gradient behavior and tool-calling failures?

  2. Comparison with stabilized GRPO. Previous work, such as SimpleTIR, has identified similar instability in multi-turn tool-integrated RL and proposed effective stabilization techniques. Have the authors considered these techniques for the GRPO baseline? It would be important to understand whether SOD maintains its advantage over a properly stabilized GRPO implementation.

  3. Checkpoint selection. Figure 3 shows substantial performance degradation in later training stages, particularly for the 1.7B OPD baseline. Were the results in Table 1 obtained from the final checkpoint or the best validation checkpoint? Was the same selection criterion applied consistently across all methods?

Reviewer_7LrjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that tool-induced errors can make teacher supervision unreliable on student-generated histories. The paper proposes SOD, which adaptively weights the OPD loss at each interaction step using student–teacher disagreement, and report improvements when combined with GRPO.

Critical strengths

1.The paper studies an interesting problem that tool-induced errors can make teacher supervision unreliable on student-generated histories, which can be important in on-device LLM agents. 2. The writing and presentation are obviously excellent. 3.The method is simple and easy-to-understand.

Critical weaknesses
  1. The central motivation is insufficiently established. Figure 1 shows that increasing disagreement and teacher entropy after errors, but do not directly evaluate whether the teacher’s corrective guidance becomes incorrect or harmful. Uncertainty and disagreement alone do not establish unreliable supervision.
  2. Large disagreement can also indicate valuable teacher correction of student mistakes. Appendix F.5 shows that increasing selected weights hurts performance, but lacks controls matching overall distillation strength or gradient scale. It therefore does not isolate misleading guidance from optimization effects, which is a major concern to me in this paper.
  3. The theoretical justification contains substantive errors. Appendix G.4, Eq. (27), incorrectly substitutes a lower bound on the unweighted second moment into an upper-bound expression. Appendix G.2’s claim that SNR vanishes as overlap approaches zero also does not follow with a fixed support threshold.
  4. Contribution of this paper needs sharper positioning against "TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents", which already describes error compounding, escalating KL, and unreliable supervision.
  5. Table 1’s OPD baseline and Table 2’s uniform-weight ablation appear equivalent under the stated objective and training setup, yet report 1.7B averages of 36.27 and 34.70. Please clarify whether these are independent runs or differ in configuration. 6.The claimed code (in Anonymous GitHub) is inaccessible. At review time, the supplied anonymous link reports that its configured commit no longer exists, preventing inspection of the implementation and verification of the reported settings.
Questions for authors
  1. Table 1’s OPD baseline and Table 2’s uniform-weight ablation appear equivalent under the stated objective and training setup, yet report 1.7B averages of 36.27 and 34.70. Please clarify whether these are independent runs or differ in configuration.
  2. What exact gradient surrogate implements the sampled-token OPD objective, and which quantities(including the adaptive weights) are detached during optimization?
Reviewer_2CQGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOD, a step-wise reweighting method for on-policy distillation of small tool-using language models. The motivation is that tool errors can increase student–teacher divergence and reduce the reliability of subsequent teacher supervision. SOD uses the average absolute log-probability gap within each response to adjust the distillation weight, and combines this loss with GRPO. The authors report gains on math, science, and coding benchmarks with 0.6B and 1.7B students, along with experiments using additional teachers and agent environments.

Critical strengths
  1. The method is simple to implement and uses log-probabilities already computed during distillation. This makes it a practical change to existing training pipelines.

  2. SOD performs best on all four main benchmarks at both student sizes. The inclusion of TIP and SCOPE is useful, since they also adapt the distillation signal.

  3. The experimental analysis is extensive. I appreciate the teacher-quality comparisons, anchor perturbations, and results without GRPO, which help assess where the gains come from.

Critical weaknesses
  1. I have serious concerns about the theoretical analysis. In Appendix G.4, Eq. (27) substitutes a lower bound on the unweighted second moment into the upper bound in Eq. (26). This does not give the claimed upper bound. In Appendix G.2, the conclusion after Eq. (22) also does not follow: with the support threshold fixed, letting the overlap tend to zero does not make the displayed denominator diverge. These steps are used to justify the method's variance argument and need correction.

  2. The monotonicity claim in Appendix G.5 is false in general. Fix the teacher at q=(0.1, 0.9). Students p=(0.2, 0.8) and p'=(0.01, 0.99) have KL divergences of 0.0444 and 0.0713, but expected absolute log-ratios of 0.2329 and 0.1174. The ordering reverses even without sampling noise; Jensen's inequality does not resolve this. Also, Eq. (7) simplifies exactly to min((d_1+epsilon)/(d_k+epsilon), 1+delta), with all intermediate terms cancelling. The analysis should use this simpler form and explain what relationship to KL is actually needed. A direct comparison with full KL would be informative.

  3. Does high divergence indicate that the teacher is wrong, or that its supervision is difficult for the student to learn from? Appendix F.5 shows that increasing weights on these steps hurts performance, but this also changes gradient magnitude and variance. Measuring the teacher's recovery accuracy on the same corrupted contexts, and comparing against uniform or randomized weights with matched total distillation strength, would help distinguish these explanations.

  4. According to Appendix D.3, the OPD baseline uses GRPO plus unweighted OPD. This appears equivalent to the uniform-weight SOD ablation, yet the reported 1.7B averages are 36.27 and 34.70. What accounts for the difference under the shared settings? Separately, Appendix F.2 reports the best validation success during training. Results on an independent test set after checkpoint selection are needed for these environments.

Questions for authors

see Critical Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.