- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether an agent policy can be learned from a stronger teacher without imitating its chain-of-thought, by training on the teacher's actions with an empty reasoning block. The student is held near its initial model with a KL penalty on offline replay data, and may also be given short summaries of the teacher's reasoning. Students are evaluated with thinking enabled, where they reason in their own way while acting like the teacher.
Critical strengths
1.The method transfers agent capabilities using only actions and optional reasoning summaries, making it applicable to closed teachers. The improvements on AppWorld and SWE-bench are substantial. 2.LoRA fine-tuning on 270 fixed teacher trajectories, no RL, no environment interaction during training, and a replay buffer built once offline rather than sampled on-policy. This is a low-cost recipe that others can run, and the offline replay corpus also makes the setup easy to reproduce.
Critical weaknesses
1.The recipe combines action imitation, summary supervision and standard replay-based KL regularization. Cross-format transfer is an interesting empirical finding, but the mechanism remains unclear 2.CoT-proxies significantly outperform full-CoT in only one of four open-teacher settings. Capability retention is tested mainly on AIME25 and replay-domain continuations, which does not establish broad preservation of the student’s original abilities.
Questions for authors
1.Why is action-only clearly worse than full-CoT for Qwen3-4B on AppWorld (17.9 vs 29.2) but roughly tied for Qwen3-32B on SWE-bench (14.3 vs 21.1 — well, closer)? What determines when the teacher's CoT is actually needed? 2.With thinking disabled, CoT-proxy drops from 21.7% to 5.1% on SWE-bench. Since the gain seems to live in the student's own reasoning, would you call this policy distillation, or is it mostly teaching the student to reason better on the task?
3 comments
Thank you again for your review. We would be grateful to hear if anything in our response is still unclear, or if there are other points you would like us to address.
As the discussion period ends soon, we would appreciate it if you could take our response and the new results into consideration in your final score.
Thank you for your review. We address each of your points below.
The final recipe combines three complementary components, and we ablate each of them, showing that: 1) action-only training (action imitation) does transfer long-horizon behaviour of the teacher to some degree; 2) reasoning supervision via proxy (in form of summaries) provides additional signal and matches performance of full CoT supervision; 3) replay-based KL regularization helps to retain model's existing capabilities, which in case of action-only and CoT-proxy training is important, as we do not want the model to lose its initial reasoning capability.
We do not have a complete mechanistic explanation of cross-format transfer, but our ablations narrow it down: the gain from CoT-proxy largely disappears when thinking is disabled at inference, when the summaries are placed inside the reasoning block, and when the summaries are shuffled across turns. The transfer itself was also shown independently in AI safety work by Chua et al. (2025).
To strengthen the empirical evidence for cross-format transfer, we trained Qwen3-4B on SWE-bench trajectories distilled from the closed teacher (GPT Luna):
These results are the best we've got on SWE, surpassing even 32B student trained on open teacher's trajectories. The pattern of CoT-proxy outperforming action-only holds here as well.
We agree, and we do not aim to show that CoT-proxy is always superior to full-CoT. Our claim is that it is at least as good in the long-horizon distillation setting (it has the highest mean in all settings and is never significantly worse) which makes it an alternative to standard full-CoT distillation that doesn't need the teacher's verbatim reasoning. This is what enables distillation from closed teachers, where full-CoT training is not possible.
We have additionally evaluated the checkpoints on IFEval and a stratified 1,000-question MMLU-Pro subset. Every cell is the mean over the same three seeds as Table 3:
The pattern here is the same -- anchoring helps with retaining the original capabilities outside of target domain.
We would like to point out that in both of these cases full-CoT is significantly better than action-only (+11.3 and +6.7 points, p = 0.002 in both, as shown in Appendix D.4).
We do not see a single factor that predicts the size of the gap across the settings, and we do not claim one. What holds is that CoT-proxy matches or exceeds full-CoT, which means that the teacher's verbatim CoT is not needed -- a short summary of its reasoning is enough.
We would call it policy distillation that is executed through the student's own reasoning. This ablation shows that the best performance on these tasks requires reasoning at inference, and this is what motivates cross-format transfer -- we supervise only actions and summaries, and the student applies what it learned through its own reasoning, which is never supervised directly.
If you have any further questions, please let us know and we will try to address them before the end of the rebuttal.