acceptodds
Under review as a conference paper at ICLR 2027

From Reasoning to Agents: Building the Next Generation of On-Device Language Models

Abstract

Recent advances in agentic models have enabled a growing range of applications, driving increasing demand for training models specialized for downstream agentic tasks. Existing work, however, either develops proprietary construction recipes for large-scale models or optimizes post-training in isolation, leaving the end-to-end training of small agentic models poorly understood. We show that downstream agentic performance is largely determined by the capabilities and behavioral support established during pre-training; post-training sharpens what the base model can already express, but does not reliably recover what it cannot. We therefore ask which pre-training signals predict post-training gains. We find that sampling lift, the improvement obtainable by sampling multiple rollouts over greedy decoding, serves as a good indicator, suggesting that pre-training should preserve diverse behavioral support rather than prematurely concentrating on narrow solution modes. Since pre-training metrics differ substantially in predictive power, we weight them by their measured correlation with post-training outcomes, providing a principled basis for checkpoint selection and data-mixture allocation, and allowing pre-training to be concentrated on the capabilities most predictive of downstream performance. Building on these principles, we study how capabilities equipped in general pre-training are converted into reliable agentic behavior. We find that long-horizon reasoning does not transfer directly to multi-turn tool use, while separating the decoding pattern and injecting trajectory-level supervision enables more effective learning from the student's evolving state. Together, these results form an end-to-end recipe for small agentic models. Trained from scratch with only 1.2T tokens, our 2.5B model matches or surpasses the 7.3B-parameter OLMo-3 model trained with 5x more tokens, achieving the best overall results on AIME24, GSM8K, HumanEval, and SWE-Bench.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Xu5oOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors propose an end-to-end training recipe for small agentic models. The authors apply a series of pre-training principles to guide long-horizon reasoning and downstream agentic behavior instead of focusing only on post-training. Following experiments show that the final model X-LLM achieves competitive performance on SWE-Bench Verified.

Critical strengths
  1. This paper provides useful empirical observations about the relationship between pre-training checkpoints and post-training performance. Further experiments indicate that the combination of pre-training and post-training principles improved the reasoning capabilities across several downstream tasks.
  2. This paper is well-organized, and the overall motivation is easy to understand.
Critical weaknesses
  1. This work treat this combination of end-to-end training recipes as a unified process. Yet several principles that described in this work have been founded or investigated in many other studies [1,2,3,4,5,6,7].

  2. The final model combines numerous ingredients: influence-guided data mixing, synthetic bridge data, two mid-training phases, reasoning-effort conditioning, on-policy distillation and etc. But there is no convincing stage-by-stage performance comparison establishing how much agentic capability is acquired during the whole process.

  3. The title explicitly emphasizes on-device language models, yet the paper provides no practical deployment measurements. E.g., latency, inference throughput, memory footprint, energy consumption, or quantization experiments...

  4. Other weaknesses are provided in the questions below

[1] Zhao, Changsheng, et al. "Mobilellm-r1: Exploring the limits of sub-billion language model reasoners with open training recipes." International Conference on Learning Representations. Vol. 2026. 2026.

[2]Chen, Mayee, et al. "Skill-it! a data-driven skills framework for understanding and training language models." Advances in Neural Information Processing Systems 36 (2023): 36000-36040.

[3]Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).

[4]Lauffer, Niklas, et al. "Imitation learning for multi-turn lm agents via on-policy expert corrections." arXiv preprint arXiv:2512.14895 (2025).

[5]Li, Gengsheng, et al. "On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents." arXiv preprint arXiv:2606.15912 (2026).

[6]Brown, Bradley, et al. "Large language monkeys: Scaling inference compute with repeated sampling." arXiv preprint arXiv:2407.21787 (2024).

[7]Liu, Qian, et al. "Regmix: Data mixture as regression for language model pre-training." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors

My main concern is that the paper seems to propose an engineering study with an end-to-end training recipe instead of a research contribution introducing fundamentally new training principles.

Another question is that X-LLM uses a powerful teacher (Muse-Glimmer-30B, Meta AI, 2026) and a substantial multi-stage training pipeline. So the final performance does not clearly validate the individual principles proposed in the paper.

4 comments

Thank you for your feedback. We address each of your comments and questions as follows:

[ This work treat this combination of end-to-end training recipes as a unified process. Yet several principles that described in this work have been founded or investigated in many other studies [1,2,3,4,5,6,7]. ]

We thank the reviewer for raising these connections to prior literature. We agree that data mixing [2, 7], repeated sampling [6], large-scale distillation [3], and on-policy multi-turn training [4, 5] have strong foundations, and we expanded our Related Work to discuss [2, 4, 6, 7] explicitly alongside [1, 3, 5]. Rather than claiming to invent these individual components in isolation, our contribution lies in co-designing and diagnosing how pre-training, mid-training, and post-training interact when building a compact (2.5B) agentic model from scratch under a tight token budget (1.2T pre-training tokens). Specifically, our findings differ from and complement [1–7] in four ways: Forecasting post-training readiness rather than base loss ([1, 2, 7]): Unlike RegMix [7] or Skill-It! [2], which optimize data mixtures against base-model validation loss, and MobileLLM-R1 [1], which focuses strictly on single-turn math/code reasoning, we introduce synthetic cross-domain bridge data to mitigate domain interference and use a ridge regression readiness predictor, combining capability-weighted zero-shot scores with pre-training sampling lift (16, repurposing inference-time scaling [6] as a training-time diagnostic), to detect when further pre-training stops yielding post-training agentic gains. Reproducibility at small scale ([3]): Unlike Qwen3 [3], which relies on a 36T-token proprietary corpus without releasing its pre-training mixture or full agentic pipeline, we provide an end-to-end recipe that achieves a 10-point lead on SWE-Bench Verified among open models up to 7B parameters. We will release our model checkpoints along with all the data-mixing recipes. We have clarified this positioning alongside the added citations in the revised paper.

[ The title explicitly emphasizes on-device language models, yet the paper provides no practical deployment measurements. E.g., latency, inference throughput, memory footprint, energy consumption, or quantization experiments… ]

At X-LLM's size, a 32K-token agent session fits in the memory of 8 GB devices.The following table presents measurements for four models evaluated using llama.cpp on an Apple M4 Max laptop. (4-bit weights, 8-bit KV cache, one request):

ModelMemory, 32K contextFits an 8 GB iPhone (about 4 GiB per app)Time to first token, 4K / 32K promptDecode tokens/s, 4K / 32K
X-LLM (2.5B)3.0 GiBYes1.7 s / 32 s137 / 84
Qwen3.5-2B2.0 GiBYes1.1 s / 11 s190 / 159
Qwen3.5-4B3.8 GiBYes2.7 s / 31 s101 / 81
Qwen3.5-9B6.1 GiBNo5.1 s / 52 s66 / 56

Memory allocations are device-independent, so the iPhone column follows directly from these measurements: models up to approximately 4B parameters fit within the memory budget, consistent with the size class of Apple's ~3B on-device model (arXiv:2507.13575). Similarly, an 8 GB laptop, such as an M1 MacBook Air, can accommodate all four models. X-LLM requires 3.0 GiB of memory, comprising 1.5 GiB for model weights, 1.1 GiB for the KV cache, and 0.4 GiB for the compute buffer. All four models use llama.cpp's standard Q4_K_M GGUF quantization.

[ My main concern is that the paper seems to propose an engineering study with an end-to-end training recipe instead of a research contribution introducing fundamentally new training principles. ]

We respectfully disagree that a research contribution must introduce a fundamentally new training algorithm. Our goal is to establish empirically grounded principles for training small agentic models end-to-end, an area where open and reproducible recipes remain limited.

Beyond integrating existing techniques, our study provides several insights: (1) pretraining sampling lift and capability-specific signals can predict downstream post-training potential; (2) stronger long-horizon reasoning does not necessarily translate into better multi-turn agentic performance; and (3) trajectory-level correction provides more effective supervision for multi-turn tool use than local correction.

We believe these findings, together with a reproducible end-to-end training recipe and released checkpoints, as well as superior SWE-bench performance compared to all evaluated open-source models, including those with up to 7B parameters, constitute a valuable research contribution by advancing both our understanding of small agentic model training and its practical accessibility.

[ X-LLM uses a powerful teacher (Muse-Glimmer-30B, Meta AI, 2026) and a substantial multi-stage training pipeline. So the final performance does not clearly validate the individual principles proposed in the paper. ]

The use of a teacher definitely helps the model training overall and improve upon its performance, but we would also state that simply using a teacher does not guarantee decent performance. Especially for multi-stage training, a weaker data and training recipe could easily lead to underfitting due to the teacher–student capacity gap (Zhang et al., 2023), optimization discrepancies between teacher and student predictive distributions (Stanton et al., 2021), or train–inference distribution mismatch and catastrophic forgetting across stages (Agarwal et al., 2024).

Zhang et al. (ACL 2023): Lifting the Curse of Capacity Gap in Distilling Language Models (arXiv:2305.12129) which documents how an oversized teacher–student capacity gap degrades student performance without structural sparse capacity.

Stanton et al. (NeurIPS 2021): Does Knowledge Distillation Really Work? (arXiv:2106.05945) which demonstrates that optimization difficulties and dataset choices heavily govern whether teacher alignment actually translates into held-out generalization.

Agarwal et al. (ICLR 2024): On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (arXiv:2306.13649) which shows that off-policy distillation on static datasets introduces exposure bias and distribution mismatch in autoregressive models.

Dear Authors,

Thank you for the detailed answer, especially on the ablation of deployment measurements for X-LLM. Though I am still concerned about the key contribution of this work for these ene-to-end training principles. Thus, I will raise my score to 2 (weak reject).

Best

Dear Reviewer,

Thank you for your thoughtful feedback and for reconsidering your assessment. We appreciate your recognition of our additional experiments and deployment measurements. We would also like to further clarify the scientific contribution of our end-to-end training study.

We agree that the individual training techniques are not necessarily novel in isolation. However, we believe that systematically investigating how these techniques interact across training stages, identifying effective design choices, and demonstrating their practical impact constitute meaningful research contributions.

This perspective is also reflected in prior work such as OLMoE (ICLR 2025 Oral), OLMo 2 (COLM 2025), and DataComp-LM (NeurIPS 2024), which demonstrate the research value of systematically studying training recipes, data mixtures, and empirical design decisions, even when building upon established techniques.

Importantly, our work goes beyond presenting a strong final checkpoint or simply combining existing methods. We investigate how pre-training and mid-training choices affect downstream agentic capabilities, how reasoning and agentic supervision interact during post-training, and how trajectory-level teacher correction affects multi-turn tool-use performance. Our controlled experiments provide empirical evidence for these interactions.

We believe these findings are particularly valuable for compact agentic models, where limited model capacity and training budgets make balancing reasoning and agentic capabilities challenging. The resulting 2.5B model achieves strong mathematical reasoning and agentic performance, including results competitive with substantially larger open models.

Together with our reproducible training recipe and planned checkpoint release, we believe these empirical insights offer both scientific and practical value to the community.

We hope this clarification helps address your remaining concern about the contribution of our work.

Reviewer_4maTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an end-to-end recipe for training a 2.5B on-device LLM, combining teacher-assisted pretraining, capability-oriented data mixtures, effort-conditioned SFT, and trajectory distillation.

Critical strengths
  1. Strong reported mathematical reasoning and SWE-bench Verified results; the latter reaches 27.0%, versus 15.6% for Qwen3.5-2B.
  2. Informative comparison of different teacher-intervention schemes, including a negative filtering result and explicit training-token statistics
Critical weaknesses
  1. The practical benefit of training from scratch remains unverified. Table 5 compares finished models without applying the same post-training recipe to an existing checkpoint of similar size. This leaves unclear whether the proposed pretraining pipeline offers a better performance–cost trade-off than adapting an available model.
  2. The central pretraining claims lack controlled evidence. No matched experiment demonstrates that equivalent supervision introduced during post-training cannot recover the proposed advantage. The claimed 3–5% bridge-data improvement also lacks detailed ablation results.
  3. The small-model-specific rationale is insufficiently established as the used data mixing, effort conditioning, and trajectory correction are broadly applicable techniques. Experiments on one 2.5B configuration do not establish how these choices address capacity-related limitations or how their benefits vary with model scale.
  4. The OPD comparisons confound intervention granularity with other changes. Naive OPD uses reverse KL, whereas teacher-suffix training uses cross-entropy; training rows and supervised-token budgets also differ.
  5. Several baseline scores raise evaluation concerns. Gemma and Nemotron obtain only 1.8% and 5.5% on HumanEval despite achieving 44.0% and 43.8% on LiveCodeBench. These discrepancies warrant validation of prompting and answer extraction before HumanEval is used as headline evidence. They suggest a possible evaluation issue, rather than confirming one.
  6. Training/evaluation separation is insufficiently documented and no benchmark decontamination procedure is reported. This particularly limits the evidential value of longstanding public benchmarks such as HumanEval; contamination remains an unresolved concern.
Questions for authors
  1. Which GPU models were used, and what were the wall-clock time and GPU hours for each stage? Please include teacher inference and data generation, and clarify whether the reported 128 GPUs include teacher resources.
  2. Are Fig4 and Fig5 based on separate runs? They extend to 1M pretraining steps, whereas Table 7 specifies 600k; please identify their training configurations.
  3. Can you provide representative baseline failures and quantify answer-extraction or tool-format errors separately from task-solving failures?
3 comments

Thank you for your feedback. We address each of your comments and questions as follows:

[ Why training from scratch? ]

Training from scratch gives us direct control over the pretraining data distribution and recipe, allowing us to study how pretraining choices interact with later reasoning and agentic post-training. This is particularly useful for agentic tasks, for which public end-to-end training studies remain limited.

We agree that comparisons of final model performance alone do not establish a superior performance–cost trade-off over adapting existing pretrained models. Our primary objective is not to demonstrate that training from scratch is more cost-efficient than adaptation, but rather to investigate how pretraining, mid-training, and post-training can be jointly designed to develop compact models with strong reasoning and agentic capabilities. A controlled comparison against adapting similarly sized pretrained models under equivalent supervision and evaluation protocols would be valuable future work.

[ No matched experiment demonstrates that equivalent supervision introduced during post-training cannot recover the proposed advantage. ]

In our pipeline, mid-training belongs to the pre-training phase (Section 3) and precedes post-training (Section 4). We applied the identical two-stage SFT (general instruction, then agentic coding) to three checkpoints with different pre-training settings:

Checkpoint of the same pre-training runMid-trainingPost-trainingSolved, 95-task subset of SWE-Bench Verified
389K stepsCapability-concentratedTwo-stage SFT24/95 (25.3%)
600K stepsNoneTwo-stage SFT12/95 (12.6%)
600K steps (final X-LLM)Capability-concentratedFull X-LLM post-training (Section 4)31/95 (32.6%)

The same post-training procedure did not close the gap resulting from insufficient pre-training.

[ The claimed 3–5% bridge-data improvement lacks detailed ablation results. ]

We agree that the bridge-data claim requires a matched ablation. We found them to be helpful across benchmarks in our 50K training step ablation, as seen below:

Benchmarkw/ Bridgew/o BridgeΔ_abs (pp)Δ_rel
MMLU38.4736.88+1.59+4.31%
MMLU-Redux40.3038.97+1.33+3.41%
MMLU-Pro16.5315.92+0.61+3.83%
ARC-Challenge45.1543.90+1.25+2.85%
ARC-Easy70.6667.69+2.97+4.39%
HellaSwag66.3063.97+2.33+3.64%
PIQA76.2274.28+1.94+2.61%
SIQA47.3946.02+1.37+2.98%
WinoGrande64.4862.96+1.52+2.41%
BoolQ63.1561.75+1.40+2.27%
GSM8K47.9945.94+2.05+4.46%
MATH16.4015.70+0.70+4.46%
BBH25.0724.26+0.81+3.34%
BBEH3.433.34+0.09+2.69%
GPQA13.4813.02+0.46+3.53%
SuperGPQA10.4510.06+0.39+3.88%
HumanEval+27.4426.73+0.71+2.66%
MBPP44.4442.96+1.48+3.45%
CRUXEval24.2523.12+1.13+4.89%
Mean39.0337.76+1.27+3.48%

Incorporating cross-domain bridge data yields consistent improvements across all 19 evaluated benchmarks at 50K training steps, with an average absolute gain of 1.27 percentage points and an average relative improvement of 3.48% compared to training without bridge data. We have included this comparison in the Appendix of the revised paper.

[ Insufficient justification of small-model-specific design choices and their scalability. ]

Our choice of 2.5B parameters is motivated by the memory and computational constraints of on-device inference. For example, recent smartphones such as the iPhone 16 and iPhone 17 have 8 GB of RAM, while Pro models offer up to 12 GB, with memory shared across the operating system, applications, and model execution. These constraints motivate compact models that balance capability with resource requirements. Consistent with this motivation, Apple's on-device foundation models have adopted approximately 3B dense parameters (Apple, 2024; 2025).

We agree that the training techniques are broadly applicable across a range of model sizes. However, our data design explicitly considers the limited capacity of small models to accommodate heterogeneous and noisy data. We therefore prioritize clean, task-relevant data, remove redundancy, and tailor components such as bridging data to the target model scale. Moreover, our data influence scores are computed at the target model size, making the resulting data mixture model-aware. While we do not explicitly quantify the degree of model specificity, model capacity is an important consideration throughout our data design. Furthermore, our contribution is to establish an effective end-to-end training recipe for reasoning and agentic capabilities within a deployment-relevant 2.5B parameter budget. We acknowledge that cross-scale experiments would further clarify capacity-dependent effects; however, reproducing the full training pipeline at multiple scales requires substantial additional computation and is beyond the scope of this study.

[Apple Intelligence Foundation Language Models (2024)](https://arxiv.org/abs/2407.21075) [Apple Intelligence Foundation Language Models: Tech Report 2025](https://arxiv.org/abs/2507.13575)

[ The OPD comparisons confound intervention granularity with other changes. Naive OPD uses reverse KL, whereas teacher-suffix training uses cross-entropy; training rows and supervised-token budgets also differ. ]

Regarding KL vs. CE: The difference in supervision objectives follows naturally from the respective methods: standard OPD applies token-level reverse KL to student-generated trajectories, whereas teacher-suffix correction uses cross-entropy to imitate teacher-generated recovery trajectories. Applying reverse KL to these suffixes would constitute a different training objective rather than a direct replacement. More importantly, One-Turn and Till-End OPD both use cross-entropy, allowing a more direct comparison of correction horizons. Till-End improves BFCL accuracy from 24.88% to 30.75%, despite using 25% fewer supervised tokens under the same candidate-episode budget.

Regarding training rows and token budgets: Since rollout generation is the dominant computational cost in OPD, we control for the same 12,180 candidate scenario-sample slots and rollout horizon across all variants (Table 4). The different row counts arise from trajectory construction rather than unequal candidate sampling budgets. Naive OPD produces one row per episode (12,180), while One-Turn expands episodes into per-decision probes (59,377). Till-End retains 10,967 rows after excluding episodes that terminated before teacher intervention. Verified Till-End further filters the trajectories to 4,082 rows. Despite these differences, Till-End achieves the highest BFCL accuracy (30.75%) while using 25% fewer supervised loss tokens than other methods.

[ Several baseline scores raise evaluation concerns. ]

Thank you; both cells were evaluation defects. HumanEval and MBPP use a completion-style harness, while LiveCodeBench serves each model with its chat template, and only the former was affected. Gemma-4-E2B's prompts lacked the <bos> token the model is trained with, and Nemotron-3-Nano-4B's native Hugging Face generation path for its hybrid Mamba–Transformer architecture produced incoherent code. Regenerating the same prompts on a corrected path, with unchanged post-processing and tests, gives HumanEval 67.1 and 72.0. Answer extraction was not the cause: an independent re-scorer reproduces every verdict.

We then re-evaluated these two models and OLMo-3-7B end to end. The updated table reports our own runs for every cell of these three models, replacing the developer-reported numbers some cells had used; no other baseline changed. We have also verified that the remaining baseline results were correctly evaluated, and confirm that all baseline results were obtained under a unified evaluation framework and consistent evaluation protocols as those used for our final model. With these corrections, Nemotron-3-Nano-4B (4B) has the best AIME24 and HumanEval scores, while X-LLM remains best among models under 3.5B parameters on both and best overall on SWE-Bench Verified.

[ Can you provide representative baseline failures and quantify answer-extraction or tool-format errors separately from task-solving failures? ]

We separated failures by type from the per-item outputs. Representative baseline cases: Answer extraction: Gemma-4-E2B does not write \boxed{} unless told to, so the extractor reads the last number in its prose. With an explicit boxed-answer instruction and its reasoning mode on, its MATH500 rises from 50.8 to 77.4. Tool format: on BFCL-Parity, OLMo-3-7B solves 256/369 trials when its native tool calls are written to the answer file, but 1/369 when it must write the call JSON itself (SmolLM3: 259 vs 0). Truncation: all 98 unparseable answers among Qwen3.5-2B's 1,319 GSM8K answers are repetition loops cut at the token limit.

[ No benchmark decontamination procedure is reported. ]

Our decontamination procedure uses n-gram matching, following the approach described in [1]. Practically, we removed 13-gram candidates overlap between benchmarks and pretraining sets, or source from pretraining datasets that were decontaminated.

[1] Singh et al. (2024), “Evaluation Data Contamination in LLMs: How Do We Measure It and (When) Does It Matter?”

[ GPU configurations, training costs, and teacher inference overhead. ]

Teacher inference ran inside each training job on the same GPUs as students, so the counts below include the teacher; no separate teacher hardware was used.

StageGPUsTraining timeGPU-hours (median step time × steps × GPUs)
Pre-training (online distillation)32×8 H100≈16 days≈98K
Mid-training, 2 phases (distillation)16×8 B200 / H100≈5 days≈15K
SFT, 2 stages16×8 A100, then H100 / B200≈4.5 days≈14K
Total≈127K

Pre-training time is computed from the median step time of our distillation setup with the Muse-Glimmer-30B teacher, measured on H100. The 16×8 = 128 GPUs in Table 7 refer to mid-training and post-training; pre-training used 256 GPUs, which we corrected in the revision. Generating synthetic bridge data is estimated to require approximately 287 GPU-hours, based on 300 equivalent runs of the one-GPU data generation job, whose recorded end-to-end runtime was 57 minutes 22 seconds.

[ Are Fig4 and Fig5 based on separate runs? They extend to 1M pretraining steps, whereas Table 7 specifies 600k; please identify their training configurations. ]

Thank you for noting the ambiguity.Figures 4 and 5 use checkpoints of a single pre-training run, the same run that produced X-LLM's base model. The run was scheduled for 1M updates. Figures 4–5 evaluate its checkpoints before and after one fixed post-training recipe (Section 3.2), while X-LLM continues from its 600K-step checkpoint (Table 7) into mid-training and post-training. The table lists both configurations.

Training configuration of the run behind Figures 4–5 and X-LLM

ConfigurationPre-training (Figures 4–5; X-LLM's base)Mid-training (X-LLM only)
Updates1M scheduled; X-LLM continues from the 600K-step checkpointTwo phases of 50K (≈105B tokens each)
Tokens per update128 sequences × 16,384 tokens ≈ 2.1M≈2.1M
GPUs32×8 H10016×8 B200 / H100
Teacher and lossMuse-Glimmer-30B, online, pure forward KLMuse-Glimmer-30B, pure forward KL
Peak LR and schedule1.2 × 10⁻³, 7,500 warmup steps, cosine decay to zero over 1M updates3.63 × 10⁻⁴, no warmup, linear decay to zero
Reviewer_RtX2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to develop compact language models capable of both long-horizon reasoning and multi-turn agentic behavior through a jointly designed pretraining and post-training pipeline. The authors introduce X-LLM, a 2.5B-parameter model trained from random initialization with 1.2T pretraining tokens, followed by mid-training, supervised fine-tuning, and on-policy distillation.

The paper presents three main findings. First, it proposes sampling lift (pass@k − pass@1) as a diagnostic of retained behavioral support during pretraining and develops a capability-weighted predictor of downstream post-training performance using zero-shot benchmark scores. Second, it uses task-conditioned reasoning-effort tokens to distinguish long-horizon single-turn reasoning from multi-turn tool use. Third, it compares on-policy distillation strategies and finds that supervising teacher-completed trajectories from student-visited states yields stronger agentic performance than token-level or single-turn correction.

Experiments span mathematical reasoning, coding, function calling, software engineering, and out-of-distribution agentic benchmarks. X-LLM achieves 27.0% on SWE-bench Verified, compared with 15.6% for Qwen3.5-2B, while also obtaining 62.8% on HumanEval and 93.4% on GSM8K. These results demonstrate promising capabilities for a compact model, although the strength of the evidence varies across the proposed methodological components.

Critical strengths

1. Strong empirical performance from an integrated training pipeline for compact agentic models.

The paper addresses an important practical challenge: developing small language models that support both reasoning and agentic behavior with relatively limited student training-token budgets. The final 2.5B model achieves 27.0% on SWE-bench Verified, compared with 15.6% for Qwen3.5-2B, while obtaining strong results on HumanEval (62.8%), GSM8K (93.4%), and AIME24 (51.5% under the reported 16-sample averaging protocol). The evaluation covers multiple capability families rather than relying exclusively on a single benchmark, providing meaningful evidence for the effectiveness of the integrated training pipeline. These results constitute a substantial empirical contribution, even though the individual sources of improvement are not fully isolated.

2. Informative empirical investigation of trajectory-level on-policy distillation.

The comparison of on-policy distillation strategies in Table 3 provides valuable evidence about how teacher intervention affects multi-turn agentic learning. On BFCL V4 multi-turn evaluation, accuracy increases from 14.25% for the full-SFT initialization to 19.00% with Naive OPD, 24.88% with One-Turn OPD, and 30.75% with Till-End OPD. The decrease to 16.50% with Verified Till-End OPD further highlights the potential trade-off between filtering for successful trajectories and retaining coverage of difficult recovery states. Although these experiments do not completely control for differences in training budgets and supervision objectives, they offer useful practical evidence that the design of trajectory-level supervision matters for agentic performance.

Critical weaknesses

1. Insufficient validation of the pretraining diagnostics and downstream predictor.

The interpretation of sampling lift (pass@k − pass@1) as retained behavioral diversity is not fully justified, since the metric can decrease as pass@1 improves due to saturation rather than genuine capability collapse. Moreover, the downstream predictor is fitted on a relatively small set of early checkpoints and evaluated only on later checkpoints from the same training trajectory. Without comparisons against simpler baselines or validation across independent training runs and data mixtures, it remains unclear whether the identified predictive relationships generalize beyond training-progress correlations. This weakens the evidence supporting the paper's central pretraining principles.

2. Insufficiently controlled ablations and incomplete efficiency evaluation.

Several experiments do not fully isolate the contributions of the proposed mechanisms. Table 1 lacks a joint-training baseline without HIGH/LOW effort conditioning, making it difficult to attribute improvements specifically to this design. Similarly, the OPD variants differ substantially in training-token budgets and supervision objectives, complicating attribution of the gains to trajectory-level correction alone. Furthermore, the efficiency claims do not account for the computational cost of teacher-model distillation, and the paper provides no direct on-device latency or memory measurements. These limitations weaken the causal interpretation and practical efficiency claims of the proposed training recipe.

Questions for authors

1. Reliability of pretraining signals. Could the authors clarify how sampling lift distinguishes genuine behavioral diversity loss from pass@k saturation? Additionally, could they compare the downstream predictor against simpler baselines or provide further evidence of its generalization?

2. Contribution of training mechanisms. Could the authors provide a joint-training ablation without HIGH/LOW effort conditioning to better isolate its contribution? A brief clarification of how capability-aware data mixing contributes to the final performance would also be helpful.

3. Distillation efficiency and practical deployment. Given the different training budgets across OPD variants, could the authors clarify whether Till-End OPD maintains its advantage under comparable computational budgets? Additionally, reporting approximate teacher-distillation costs or representative on-device inference measurements would help substantiate the efficiency claims.

4 comments

Thank you for your feedback. We address each of your comments and questions as follows:

[ Justify sampling lift (pass@k − pass@1) as retained behavioral diversity; comparisons against simpler baselines. ]

The goal of the sampling lift based predictive relationship serves to provide a preliminary signal for us to understand about the existing training run, such as answering the question whether it is experiencing diminishing returns or not. Thus, it is not being used to predict beyond the current training runs. On the other hand, we also used it to examine the relationship between zero-shot benchmarks and downstream post-trained performance, so more so as an anecdotal examination of how our datamix is performing or to find out about the domains that are lacking in our training data.

Table: GSM8K pass@k accuracy at different checkpoints and the sampling lift

Checkpointp@1 %p@2 %p@4 %p@8 %p@16 %Δ16 pp (derived)
100k40.254.265.576.087.547.3
200k49.663.573.382.090.641.0
300k59.473.983.390.696.937.5
400k57.870.680.488.496.939.1
500k64.378.487.192.996.932.6
600k64.776.785.692.496.932.2
700k67.078.885.790.193.826.8

For each datamixture change, we ran a new datamixing run to obtain their weights to ensure that improvements can be observed across scales. One such table is the following where we compare datamix weights against baseline weights:

Model sizeFLOPsUniform NLLOurs NLLAbsolute dropRelative drop
2.5B1.9e1914.012.02.014%

[ Controlled ablations for HIGH/LOW effort conditioning. ]

Starting from the checkpoint before X-LLM's agentic reasoning SFT stage, we trained that stage with and without the effort label (both without the 14K function-calling rows), keeping data, proportions and steps identical, and evaluated the untagged model without an effort instruction. The two match within noise (untagged vs tagged: AIME24 49.9 vs 49.7, averaged over 32 samples; MATH500 90.2 vs 91.4; HumanEval 62.8 vs 62.2). At inference, switching X-LLM between HIGH and LOW likewise changes neither accuracy nor output length; length follows the task (Table 2). The gains in Table 1 therefore come from training jointly on both data types; the effort labels serve as a taxonomy for organizing heterogeneous reasoning data rather than as an inference-time control, and we will state this explicitly in the paper.

[ How capability-aware data mixing contributes to the final performance. ]

Capability-aware mixing sets the pre-training and mid-training mixtures from each candidate dataset's influence on the target capabilities (Section 3.1); mid-training uses it to concentrate tokens on code, mathematics, instruction following and tool use (Section 3.3). To measure what this mid-training contributes to final performance, we applied the identical two-stage SFT (general instruction, then agentic coding) to two checkpoints of the same pre-training run:

Checkpoint of the same pre-training runMid-trainingPost-trainingSolved, 95-task subset of SWE-Bench Verified
389K stepsCapability-concentratedTwo-stage SFT24/95 (25.3%)
600K stepsNoneTwo-stage SFT12/95 (12.6%)
600K steps (final X-LLM)Capability-concentratedFull X-LLM post-training (Section 4)31/95 (32.6%)

Under identical post-training, the mid-trained checkpoint solves twice as many tasks.

[ Whether Till-End OPD maintains its advantage under comparable computational budgets. ]

We clarify that rollout generation is the dominant computational cost in on-policy distillation. Therefore, we control for the number of candidate episodes (12,180) and rollout horizon across all variants, as shown in Table 4, providing a fair comparison under the same rollout sampling budget. Till-End OPD achieves the highest accuracy (30.75%) while consuming fewer training tokens than both Naive and One-Turn OPD.

Importantly, Verified Till-End OPD also generates the same candidate episodes, but retains fewer training examples due to post-generation filtering. Generating additional episodes to match the number of retained examples would instead give Verified OPD a larger rollout budget, making the comparison unfair.

[ Missing direct on-device latency and memory measurements and teacher-distillation compute costs. ]

  • On-device latency or memory measurements

At X-LLM's size, a 32K-token agent session fits in the memory of 8 GB devices.The following table presents measurements for four models evaluated using llama.cpp on an Apple M4 Max laptop. (4-bit weights, 8-bit KV cache, one request):

ModelMemory, 32K contextFits an 8 GB iPhone (about 4 GiB per app)Time to first token, 4K / 32K promptDecode tokens/s, 4K / 32K
X-LLM (2.5B)3.0 GiBYes1.7 s / 32 s137 / 84
Qwen3.5-2B2.0 GiBYes1.1 s / 11 s190 / 159
Qwen3.5-4B3.8 GiBYes2.7 s / 31 s101 / 81
Qwen3.5-9B6.1 GiBNo5.1 s / 52 s66 / 56

Memory allocations are device-independent, so the iPhone column follows directly from these measurements: models up to approximately 4B parameters fit within the memory budget, consistent with the size class of Apple's ~3B on-device model (arXiv:2507.13575). Similarly, an 8 GB laptop, such as an M1 MacBook Air, can accommodate all four models. X-LLM requires 3.0 GiB of memory, comprising 1.5 GiB for model weights, 1.1 GiB for the KV cache, and 0.4 GiB for the compute buffer. All four models use llama.cpp's standard Q4_K_M GGUF quantization. We’ve also incorporated this table into the revised manuscript

  • Distillation cost:

We will add this accounting. We use the standard estimates: training a model with N parameters on D tokens costs about 6ND FLOPs for the forward and backward passes [1], and a forward pass alone costs about 2ND. The teacher, Muse-Glimmer-30B (26.5B parameters in its text forward pass), runs forward-only: per token it costs about 2 × 26.5B ≈ 53 GFLOPs, versus 6 × 2.49B ≈ 15 GFLOPs to train the student, so a distilled token costs about 4.5× a cross-entropy token. Even with every teacher pass counted, X-LLM's total training compute is about half or less of what each model below used for pre-training alone:

ParametersTokensTraining compute (FLOPs)
X-LLM student, all stages (6ND)2.49B≈1.6T2.4 × 10^22
Teacher forward passes in pre-training (2ND)26.5B1.26T6.7 × 10^22
Teacher forward passes in mid-training (2ND)26.5B0.21T1.1 × 10^22
X-LLM total1.0 × 10^23
SmolLM3-3B, pre-training only (6ND)3.08B11.2T2.1 × 10^23
OLMo-3-7B, pre-training only (6ND)7.30B5.9T2.6 × 10^23
Qwen3-1.7B, pre-training only (6ND)1.7B36T3.7 × 10^23
Qwen3-4B, pre-training only (6ND)4.0B36T8.6 × 10^23

Pre-training through SFT took about 127K GPU-hours, teacher inference included. The teacher's own training is not incurred because it is an existing model, and top-k teacher logits could be cached once and reused across runs or students.

[1] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

Thank you for your detailed rebuttal. It addresses my concerns, so I will raise my original score to 3.

Thank you for your response and positive assessment of our work. We appreciate the time and effort you devoted to reviewing our paper.

Reviewer_CYz7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies how pretraining data composition and ordering shape the agentic trainability of small LLMs. It introduces sampling lift as a diversity diagnostic, effort-conditioned SFT to balance reasoning and tool use, and trajectory-level on-policy distillation using teacher continuations from student states. With 1.2T pretraining tokens, the 2.5B X-LLM achieves strong results on SWE-bench, mathematical reasoning, and coding benchmarks, outperforming Qwen3.5-2B on selected metrics and showing competitive tool-use generalization.

Critical strengths

1. The topic bears strong practical research value. Most existing works on small‑agent models treat pre‑trained models as off‑the‑shelf products and only focus on optimizing post‑training stages.

2. The paper is generally well written, and the overall methodological pipeline is clearly presented.

3. The empirical results are competitive. The 2.5B X‑LLM achieves impressive performance on multiple benchmarks including SWE‑Bench, mathematical reasoning and code generation.

Critical weaknesses

1. The paper interprets the drop in sampling-lift as contraction of behavioral diversity, yet this inference lacks sufficient supporting evidence. When the success rate across multiple samples gradually saturates, a declining sampling-lift may merely reflect an increased probability of generating correct answers, rather than a reduction in the diversity of useful solutions.

2. The 3%–5% zero-shot gain from 3% cross-domain bridging data is stated without supporting ablations, baselines or calculation details.

3. The on-device claim seems only supported by using a 2.5B model smaller than 7B variants.

Questions for authors

X-LLM shows a substantial advantage on SWE-Bench Verified, but this advantage is not consistently observed on BFCL and BFCL-Parity. How do the authors explain these performance differences across tasks? Drawing on analyses of individual training stages or components, could the authors clarify which agentic capabilities benefit most from the proposed approach and the scope of its applicability?

3 comments

Thank you for your feedback. We address each of your comments and questions as follows:

[ Does declining sampling lift truly indicate reduced behavioral diversity, or simply higher success rates? ]

We agree that sampling lift is not a standalone diversity signal, but we found it to correlate well with how much gain the model can obtain after post-training. Thus,we report it jointly with pass@1 and pass@16, so saturation is visible rather than hidden. The GSM8K results below support this use: pass@1 improves 64.3 -> 67.0 from 500k to 700k while sampling headroom narrows 32.6->26.8 pp, exactly the remaining room post-training can exploit, and it also shows that at 700K the post-training gain decreases, just like sampling lift does.

Table: GSM8K pass@k accuracy at different checkpoints and the sampling lift

Checkpointp@1 %p@2 %p@4 %p@8 %p@16 %Δ16 pp (derived)
100k40.254.265.576.087.547.3
200k49.663.573.382.090.641.0
300k59.473.983.390.696.937.5
400k57.870.680.488.496.939.1
500k64.378.487.192.996.932.6
600k64.776.785.692.496.932.2
700k67.078.885.790.193.826.8

[ Ablations for 3%–5% zero-shot gain from cross-domain bridging data ]

We agree that the bridge-data claim requires a matched ablation. We found them to be helpful across benchmarks in our 50K training step ablation, as seen below:

Benchmarkw/ Bridgew/o BridgeΔ_abs (pp)Δ_rel
MMLU38.4736.88+1.59+4.31%
MMLU-Redux40.3038.97+1.33+3.41%
MMLU-Pro16.5315.92+0.61+3.83%
ARC-Challenge45.1543.90+1.25+2.85%
ARC-Easy70.6667.69+2.97+4.39%
HellaSwag66.3063.97+2.33+3.64%
PIQA76.2274.28+1.94+2.61%
SIQA47.3946.02+1.37+2.98%
WinoGrande64.4862.96+1.52+2.41%
BoolQ63.1561.75+1.40+2.27%
GSM8K47.9945.94+2.05+4.46%
MATH16.4015.70+0.70+4.46%
BBH25.0724.26+0.81+3.34%
BBEH3.433.34+0.09+2.69%
GPQA13.4813.02+0.46+3.53%
SuperGPQA10.4510.06+0.39+3.88%
HumanEval+27.4426.73+0.71+2.66%
MBPP44.4442.96+1.48+3.45%
CRUXEval24.2523.12+1.13+4.89%
Mean39.0337.76+1.27+3.48%

Incorporating cross-domain bridge data yields consistent improvements across all 19 evaluated benchmarks at 50K training steps, with an average absolute gain of 1.27 percentage points and an average relative improvement of 3.48% compared to training without bridge data. We have included this comparison in the Appendix of the revised paper.

[ Supporting evidence for on-device claim ]

At X-LLM's size, a 32K-token agent session fits in the memory of 8 GB devices.The following table presents measurements for four models evaluated using llama.cpp on an Apple M4 Max laptop. (4-bit weights, 8-bit KV cache, one request):

ModelMemory, 32K contextFits an 8 GB iPhone (about 4 GiB per app)Time to first token, 4K / 32K promptDecode tokens/s, 4K / 32K
X-LLM (2.5B)3.0 GiBYes1.7 s / 32 s137 / 84
Qwen3.5-2B2.0 GiBYes1.1 s / 11 s190 / 159
Qwen3.5-4B3.8 GiBYes2.7 s / 31 s101 / 81
Qwen3.5-9B6.1 GiBNo5.1 s / 52 s66 / 56

Memory allocations are device-independent, so the iPhone column follows directly from these measurements: models up to approximately 4B parameters fit within the memory budget, consistent with the size class of Apple's ~3B on-device model (arXiv:2507.13575). Similarly, an 8 GB laptop, such as an M1 MacBook Air, can accommodate all four models. X-LLM requires 3.0 GiB of memory, comprising 1.5 GiB for model weights, 1.1 GiB for the KV cache, and 0.4 GiB for the compute buffer. All four models use llama.cpp's standard Q4_K_M GGUF quantization. We’ve also incorporated this table into the revised manuscript

[ Why does X-LLM perform better on SWE-Bench but not consistently on BFCL? Clarify which agentic capabilities benefit most from the proposed approach and the scope of its applicability. ]

The benchmarks measure different capabilities. SWE-Bench Verified is long-horizon and execution-grounded: many turns of repository exploration, editing and test execution. BFCL scores the format and relevance of individual function calls and short API sessions, and BFCL-Parity requires writing an exact call JSON and stopping. Our recipe concentrates on long-horizon trajectories, and the agentic reasoning SFT stage contains only a small function-calling set (14K rows). Accordingly, X-LLM's BFCL score is limited mainly by multi-turn function calling (12.9) and memory (15.7), while it reaches 64.5 on the multi-step web-search category. The approach therefore benefits long-horizon, execution-feedback tasks most; short-horizon function-call formatting depends mainly on targeted function-calling data, which is complementary to the recipe.

Thanks for the rebuttal. I have no other concerns, good luck.

Thank you for your response and for taking the time to review our paper. We appreciate your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and Area Chair

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the importance of compact agentic models, the strong performance of X-LLM, and the value of investigating pre-training and post-training interactions.

In response, we conducted additional experiments, strengthened our analyses, and clarified our contributions. We summarize the key contributions and updates below.

Our main contributions. We would like to emphasize that our work goes beyond assembling existing training techniques. Our goal is to systematically investigate how pre-training, mid-training, and post-training can be jointly designed to develop strong reasoning and agentic capabilities in compact language models. In particular, our study provides three key insights:

  1. Pre-training matters for downstream agentic capabilities. We investigate how capability-aware data mixing, cross-domain bridging, and pre-training diagnostics relate to downstream performance. Additional controlled experiments demonstrate that pre-training and mid-training choices substantially affect agentic capabilities under identical post-training procedures.

  2. Reasoning and agentic capabilities require complementary training signals. We study how to jointly develop strong mathematical reasoning, coding, and multi-turn tool-use capabilities without sacrificing one for another, clarifying the roles of heterogeneous reasoning data and effort conditioning.

  3. Trajectory-level supervision matters for multi-turn agentic learning. Controlled comparisons show that teacher-generated trajectory suffixes provide more effective supervision than local, single-turn correction, highlighting the importance of student-visited states and subsequent recovery trajectories.

These findings lead to an end-to-end training recipe for X-LLM, a 2.5B-parameter model trained from scratch with only 1.2T pre-training tokens. X-LLM outperforms Qwen3.5-2B on all four mathematical reasoning and three coding benchmarks, achieving 27.0% on SWE-Bench Verified versus 15.6%. It delivers mathematical reasoning performance competitive with OLMo-3-7B (91.2% vs. 92.0% on MATH500 and 93.4% vs. 93.3% on GSM8K), while achieving the highest SWE-Bench Verified, GSM8K, and Terminal-Bench scores among all evaluated models up to 7B. These results, empirical findings, and reproducible recipe contribute to understanding and developing compact agentic models.

Common concerns across reviewers:

  1. Sampling lift and bridge-data ablation (CYz7, RtX2, 4maT): We clarified sampling lift's interpretation and limitations, added checkpoint-level analyses and comparisons, and provided a matched ablation quantifying the contribution of cross-domain bridge data.
  2. On-device inference and computational efficiency (CYz7, RtX2, 4maT, Xu5o): We added memory, latency, and throughput measurements using llama.cpp, showing that X-LLM requires approximately 3.0 GiB at 32K context under the evaluated quantization settings. We also accounted for training compute, including teacher inference.
  3. OPD methodology and comparison fairness (RtX2, 4maT): We clarified supervision objectives, rollout budgets, and training-token accounting. Existing results show that Till-End outperforms One-Turn correction (30.75% vs. 24.88%) under the same cross-entropy objective and matched candidate-episode budget, supporting the importance of correction horizon.

Reviewer-specific concerns:

  • CYz7 — SWE-Bench and BFCL performance differences: We added category-level BFCL analysis and clarified which agentic capabilities benefit most from our training recipe.

  • RtX2 — Effort conditioning and capability-aware training: We added a controlled effort-label ablation and downstream evidence for capability-concentrated mid-training.

  • 4maT — Evaluation reliability, configurations, and pre-training contributions: We corrected baseline discrepancies, clarified decontamination procedures and configurations, and added evidence linking earlier training stages to downstream agentic performance.

  • Xu5o — Contributions beyond an engineering recipe and stage-wise evidence: We clarified distinctions from prior work and cross-stage interactions. A new checkpoint comparison shows 24/95 versus 12/95 solved SWE-Bench Verified tasks under identical post-training procedures, demonstrating the impact of earlier training-stage choices.

We believe these additions substantially address the reviewers' concerns and strengthen our empirical evidence and contributions. We have also revised the paper to incorporate these major updates. We welcome further feedback from the reviewers and Area Chair.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.