- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors propose an end-to-end training recipe for small agentic models. The authors apply a series of pre-training principles to guide long-horizon reasoning and downstream agentic behavior instead of focusing only on post-training. Following experiments show that the final model X-LLM achieves competitive performance on SWE-Bench Verified.
Critical strengths
- This paper provides useful empirical observations about the relationship between pre-training checkpoints and post-training performance. Further experiments indicate that the combination of pre-training and post-training principles improved the reasoning capabilities across several downstream tasks.
- This paper is well-organized, and the overall motivation is easy to understand.
Critical weaknesses
-
This work treat this combination of end-to-end training recipes as a unified process. Yet several principles that described in this work have been founded or investigated in many other studies [1,2,3,4,5,6,7].
-
The final model combines numerous ingredients: influence-guided data mixing, synthetic bridge data, two mid-training phases, reasoning-effort conditioning, on-policy distillation and etc. But there is no convincing stage-by-stage performance comparison establishing how much agentic capability is acquired during the whole process.
-
The title explicitly emphasizes on-device language models, yet the paper provides no practical deployment measurements. E.g., latency, inference throughput, memory footprint, energy consumption, or quantization experiments...
-
Other weaknesses are provided in the questions below
[1] Zhao, Changsheng, et al. "Mobilellm-r1: Exploring the limits of sub-billion language model reasoners with open training recipes." International Conference on Learning Representations. Vol. 2026. 2026.
[2]Chen, Mayee, et al. "Skill-it! a data-driven skills framework for understanding and training language models." Advances in Neural Information Processing Systems 36 (2023): 36000-36040.
[3]Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).
[4]Lauffer, Niklas, et al. "Imitation learning for multi-turn lm agents via on-policy expert corrections." arXiv preprint arXiv:2512.14895 (2025).
[5]Li, Gengsheng, et al. "On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents." arXiv preprint arXiv:2606.15912 (2026).
[6]Brown, Bradley, et al. "Large language monkeys: Scaling inference compute with repeated sampling." arXiv preprint arXiv:2407.21787 (2024).
[7]Liu, Qian, et al. "Regmix: Data mixture as regression for language model pre-training." International Conference on Learning Representations. Vol. 2025. 2025.
Questions for authors
My main concern is that the paper seems to propose an engineering study with an end-to-end training recipe instead of a research contribution introducing fundamentally new training principles.
Another question is that X-LLM uses a powerful teacher (Muse-Glimmer-30B, Meta AI, 2026) and a substantial multi-stage training pipeline. So the final performance does not clearly validate the individual principles proposed in the paper.
4 comments
[ X-LLM uses a powerful teacher (Muse-Glimmer-30B, Meta AI, 2026) and a substantial multi-stage training pipeline. So the final performance does not clearly validate the individual principles proposed in the paper. ]
The use of a teacher definitely helps the model training overall and improve upon its performance, but we would also state that simply using a teacher does not guarantee decent performance. Especially for multi-stage training, a weaker data and training recipe could easily lead to underfitting due to the teacher–student capacity gap (Zhang et al., 2023), optimization discrepancies between teacher and student predictive distributions (Stanton et al., 2021), or train–inference distribution mismatch and catastrophic forgetting across stages (Agarwal et al., 2024).
Zhang et al. (ACL 2023): Lifting the Curse of Capacity Gap in Distilling Language Models (arXiv:2305.12129) which documents how an oversized teacher–student capacity gap degrades student performance without structural sparse capacity.
Stanton et al. (NeurIPS 2021): Does Knowledge Distillation Really Work? (arXiv:2106.05945) which demonstrates that optimization difficulties and dataset choices heavily govern whether teacher alignment actually translates into held-out generalization.
Agarwal et al. (ICLR 2024): On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (arXiv:2306.13649) which shows that off-policy distillation on static datasets introduces exposure bias and distribution mismatch in autoregressive models.
Dear Authors,
Thank you for the detailed answer, especially on the ablation of deployment measurements for X-LLM. Though I am still concerned about the key contribution of this work for these ene-to-end training principles. Thus, I will raise my score to 2 (weak reject).
Best
Dear Reviewer,
Thank you for your thoughtful feedback and for reconsidering your assessment. We appreciate your recognition of our additional experiments and deployment measurements. We would also like to further clarify the scientific contribution of our end-to-end training study.
We agree that the individual training techniques are not necessarily novel in isolation. However, we believe that systematically investigating how these techniques interact across training stages, identifying effective design choices, and demonstrating their practical impact constitute meaningful research contributions.
This perspective is also reflected in prior work such as OLMoE (ICLR 2025 Oral), OLMo 2 (COLM 2025), and DataComp-LM (NeurIPS 2024), which demonstrate the research value of systematically studying training recipes, data mixtures, and empirical design decisions, even when building upon established techniques.
Importantly, our work goes beyond presenting a strong final checkpoint or simply combining existing methods. We investigate how pre-training and mid-training choices affect downstream agentic capabilities, how reasoning and agentic supervision interact during post-training, and how trajectory-level teacher correction affects multi-turn tool-use performance. Our controlled experiments provide empirical evidence for these interactions.
We believe these findings are particularly valuable for compact agentic models, where limited model capacity and training budgets make balancing reasoning and agentic capabilities challenging. The resulting 2.5B model achieves strong mathematical reasoning and agentic performance, including results competitive with substantially larger open models.
Together with our reproducible training recipe and planned checkpoint release, we believe these empirical insights offer both scientific and practical value to the community.
We hope this clarification helps address your remaining concern about the contribution of our work.
Thank you for your feedback. We address each of your comments and questions as follows:
[ This work treat this combination of end-to-end training recipes as a unified process. Yet several principles that described in this work have been founded or investigated in many other studies [1,2,3,4,5,6,7]. ]
We thank the reviewer for raising these connections to prior literature. We agree that data mixing [2, 7], repeated sampling [6], large-scale distillation [3], and on-policy multi-turn training [4, 5] have strong foundations, and we expanded our Related Work to discuss [2, 4, 6, 7] explicitly alongside [1, 3, 5]. Rather than claiming to invent these individual components in isolation, our contribution lies in co-designing and diagnosing how pre-training, mid-training, and post-training interact when building a compact (2.5B) agentic model from scratch under a tight token budget (1.2T pre-training tokens). Specifically, our findings differ from and complement [1–7] in four ways: Forecasting post-training readiness rather than base loss ([1, 2, 7]): Unlike RegMix [7] or Skill-It! [2], which optimize data mixtures against base-model validation loss, and MobileLLM-R1 [1], which focuses strictly on single-turn math/code reasoning, we introduce synthetic cross-domain bridge data to mitigate domain interference and use a ridge regression readiness predictor, combining capability-weighted zero-shot scores with pre-training sampling lift (16, repurposing inference-time scaling [6] as a training-time diagnostic), to detect when further pre-training stops yielding post-training agentic gains. Reproducibility at small scale ([3]): Unlike Qwen3 [3], which relies on a 36T-token proprietary corpus without releasing its pre-training mixture or full agentic pipeline, we provide an end-to-end recipe that achieves a 10-point lead on SWE-Bench Verified among open models up to 7B parameters. We will release our model checkpoints along with all the data-mixing recipes. We have clarified this positioning alongside the added citations in the revised paper.
[ The title explicitly emphasizes on-device language models, yet the paper provides no practical deployment measurements. E.g., latency, inference throughput, memory footprint, energy consumption, or quantization experiments… ]
At X-LLM's size, a 32K-token agent session fits in the memory of 8 GB devices.The following table presents measurements for four models evaluated using llama.cpp on an Apple M4 Max laptop. (4-bit weights, 8-bit KV cache, one request):
Memory allocations are device-independent, so the iPhone column follows directly from these measurements: models up to approximately 4B parameters fit within the memory budget, consistent with the size class of Apple's ~3B on-device model (arXiv:2507.13575). Similarly, an 8 GB laptop, such as an M1 MacBook Air, can accommodate all four models. X-LLM requires 3.0 GiB of memory, comprising 1.5 GiB for model weights, 1.1 GiB for the KV cache, and 0.4 GiB for the compute buffer. All four models use llama.cpp's standard Q4_K_M GGUF quantization.
[ My main concern is that the paper seems to propose an engineering study with an end-to-end training recipe instead of a research contribution introducing fundamentally new training principles. ]
We respectfully disagree that a research contribution must introduce a fundamentally new training algorithm. Our goal is to establish empirically grounded principles for training small agentic models end-to-end, an area where open and reproducible recipes remain limited.
Beyond integrating existing techniques, our study provides several insights: (1) pretraining sampling lift and capability-specific signals can predict downstream post-training potential; (2) stronger long-horizon reasoning does not necessarily translate into better multi-turn agentic performance; and (3) trajectory-level correction provides more effective supervision for multi-turn tool use than local correction.
We believe these findings, together with a reproducible end-to-end training recipe and released checkpoints, as well as superior SWE-bench performance compared to all evaluated open-source models, including those with up to 7B parameters, constitute a valuable research contribution by advancing both our understanding of small agentic model training and its practical accessibility.