- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Large Language Models (LLMs) have become increasingly the dominant paradigm for generally capable intelligent agents. While frontier LLMs have led to significant improvements in AI agentic capabilities, for various use cases, it is important to have smaller LLMs that can act as agents and be deployed locally without significant computational burdens. Training small models to have better agentic capabilities is therefore an important current research question.
However, small language models often have limited capacity, which makes training particularly sensitive to the difficulty of available experience. The paper argues that more data or more complex tasks do not necessarily lead to better learning, and often it is required to generate interaction experience that is actually learnable and useful for a model with limited capacity. The paper proposes a training pipeline targeting this problem: in particular, they propose using both the small model’s performance and behavior related feedback to generate custom new environments that are suitable for their learning, use a larger teacher model to generate useful experience in these custom environments, and then use supervised finetuning to improve the smaller models. The proposed training framework is then evaluated across a variety of benchmarks and compared against similar compute matched training frameworks, and shows favorable results.
Critical strengths
-
The paper is well-written and easy to follow.
-
The presented framework is well-principled and intuitive. I am confident that the framework works well once one has taken care of the implementation details carefully.
-
The problem of training small language models to be strong agents is timely and important.
Critical weaknesses
(Missing important details about the experiments presented in the paper)
The paper mentions:
A stronger teacher agent then generates successful demonstrations on the resulting tasks, yielding a separate training corpus for each student.
I could not find any details about the teacher agent. Which model is it? What is the teacher’s own performance? Are there any other teacher models used in the compared baselines in Table 1? For example, SETA in Table 1 uses Kimi-K2.5-Thinking as the teacher model. Could the performance improvement be solely based on the teacher’s strength? A controlled study of the teacher will significantly strengthen the paper. I will be happy to raise my score if the authors could provide this ablation, especially if they can use the same models they are training (e.g., Qwen3.5-27B) as the teacher model, since it will show that the method can be scaled to the largest models as well, significantly strengthening the paper’s claim.
(Thin Improvement Margins)
The improvement margins are thin, and it is unclear to me if they are statistically significant. Moreover, the main motivation of the paper is to train small language models to be strong agents, but the performance gains are the smallest on the 4B model (the smallest model tested in the paper), contradicting the motivation for the proposed framework.
(Missing experiments)
-
There is no ablation comparing behavior-guided editing against outcome-only gating. Without this ablation, it is unclear to me the relative significance of different parts of the framework.
-
Separate training data is built for 4B, 9B and 27B students. Could the authors train one model (eg., 4B) on another model’s training data (e.g., 9B)? Without it, it is unclear to me that student specific training data generation is helpful.
-
The paper is missing few key important baselines. For example, how does a well-tuned outcome reward only RL run on the modified/augmented dataset of environments perform? Since a teacher model is used to generate SFT data, why not run OPD or some other well-tuned distillation baselines and compare against the paper’s method?
(Extremely poorly written related works)
The paper’s relevant works are poorly written. It misses key citations of even the well-established methods that it uses for training its models. For example, the paper uses Direct Preference Optimization (DPO) on top of SFT to further improve the LLM agents’ performance. I could not find any citations to DPO [1] or its multi-turn extension [2] anywhere in the paper, even though these citations will be vital. Similarly, the idea of length normalized DPO can be traced back to SimPO [6] and Tulu 3 [7], but none of these are cited as well.
There is also a significant body of research studying curriculum learning for LLM agents that discuss the “frontier of capability” based curriculum. A few notable examples are PAPRIKA [3] and SEC [4], again not cited or discussed. Related works in environment generation such as SPADE [5], though concurrent, should at least be cited and discussed.
References
[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, https://arxiv.org/abs/2305.18290
[2] From to : Your Language Model is Secretly a Q-Function, https://arxiv.org/abs/2404.12358
[3] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543
[4] Self-Evolving Curriculum for LLM Reasoning, https://arxiv.org/abs/2505.14970
[5] SPADE: Self-Play in Adaptive Synthetic Executable Environments, https://arxiv.org/abs/2608.19197
[6] SimPO: Simple Preference Optimization with a Reference-Free Reward, https://arxiv.org/abs/2405.14734
[7] Tulu 3: Pushing Frontiers in Open Language Model Post-Training, https://arxiv.org/abs/2411.15124
Questions for authors
Please address the weaknesses mentioned above.