- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies a practical issue in on-policy distillation: whether the student is learning actual capabilities or merely imitating the teacher’s style. It identifies the style mismatch problem in cross-teacher OPD, analyzes its underlying behavior, and proposes a corresponding mitigation method. The problem is clearly defined and well motivated, and the proposed approach achieves consistent improvements across multiple benchmarks and model settings.
Critical strengths
The paper addresses a realistic and practically relevant problem in OPD: when the SFT reference and the OPD teacher come from different models, the student may learn stylistic differences rather than actual capabilities. The problem is concrete and clearly defined, and the paper provides a useful insight into how cross-teacher mismatch can affect OPD supervision. The proposed solution is also reasonable and well aligned with the identified issue.
Critical weaknesses
1.Limited task settings. The experiments focus mainly on single-turn reasoning and code-generation tasks. It remains unclear whether the proposed method can generalize to more realistic long-horizon agentic settings, where the student interacts with an environment and the teacher must supervise evolving states and tool-use decisions.
2.Engineering-heavy solution. Although the underlying insight is elegant and practically relevant, the proposed solution is relatively engineering-oriented and less conceptually elegant. The paper would benefit from a clearer theoretical justification and comparisons with simpler alternatives for mitigating cross-teacher style mismatch.
Questions for authors
Could the authors evaluate whether the proposed method also works in agentic settings, especially on tasks with clear outcome rewards such as SWE? If effective, this would make the method more practically valuable, since such settings involve not only long-horizon reasoning but also tool use, environment interaction, and potential effects from the agent harness. It would be particularly interesting to see whether the method can distinguish stylistic mismatch from genuine policy differences in these more complex settings.
3 comments
Thanks for response
I maintain my score. please add the discussion to camera-ready.
Thanks very much for your suggestions that helped us strengthen our work. We are grateful for your time and support.
Rebuttal for Reviewer rFd8
W1 / Q1. Agentic settings and tool use.
We added a multi-turn tool-use experiment. The teacher is Qwen3-30B-A3B-Thinking-2507 and the student is Klear-AgentForge-8B, an agentic tool-use model. Training uses student rollouts on ToolMind tool-use tasks, including multi-turn ones, for 150 updates with the paper's settings. We evaluate on BFCL v3 (4,441 items in 17 categories), whose multi-turn tasks run stateful, executable environments over many turns and grade the final state.
2.0 matches or improves over the base model in every group, by 1.4 points overall and 3.8 points on multi-turn tasks, while Lightning OPD falls below the base model in every group. Here the main style mismatch is tool-call formatting. Lightning OPD imitates the teacher's format and breaks the student's native one, whereas 2.0 removes this recurring disagreement and keeps the tool-use signal.
Full SWE-style training does not fit in the rebuttal period. We have added this experiment to Appendix F.11 (Table 19) and list SWE settings as future work in the Limitations (Appendix G).
W2. Engineering-heavy solution, theoretical justification and simpler alternatives.
The method rests on one statistical idea. Regress the teacher–reference disagreement d=ℓT−ℓR on features z that cannot encode whether a step is correct (token identity, coarse position and surprisal), then subtract the prediction. The lookup tables are group means, the prior masses are standard shrinkage for rare groups, and cross-fitting is the standard double-ML device. The whole estimator runs in under 30 seconds on one CPU core.
Theory. We have added a formal analysis of the training signal with proofs in Appendix D (Propositions D.1 to D.3).
Simpler alternatives. A moment-matched global affine control (same mean and scale, no conditioning on z) is 1.2 points (Qwen3-4B) and 1.1 points (Klear-8B) below 2.0 on AIME24 (Table 6). On Qwen3-4B, 2.0 also beats IW-OPD by 3.5 points and TA-OPD by 6.1 points on the math average, and both on each of AIME24, AIME25 and HMMT25 (Table 1).
We have added this regression view to the method section (Section 3.4, "Regression view").