- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a 256K-context midtraining recipe for preparing base language models for agentic coding, combining natural software-engineering data, a small proportion of agent trajectories, and pretraining replay. After identical SFT, the recipe improves Nemotron 3.5 Lightning on SWE-bench Verified, Multilingual, and Pro by 3.8, 7.0, and 3.5 percentage points, respectively, and improves the larger Nemotron 3 Super on DeepSWE by 4.0 points, with additional gains across agent harnesses.
Critical strengths
• Understanding how to construct the intermediate training stage between pretraining and agentic post-training is valuable. The descriptions of data construction, filtering, serialization, and sampling provide useful implementation guidance. • Applying identical short and full SFT to the starting and midtrained checkpoints helps establish that the added midtraining stage can improve the resulting agents. Evaluating after 100B SFT tokens is particularly useful for checking whether the improvements survive substantial downstream training.
Critical weaknesses
• The paper shows that the complete midtraining pipeline improves downstream performance, but it does not establish that the proposed natural-data-heavy recipe is responsible for these gains. In Table 1, most of the downstream improvement comes from adding agent trajectories, yet there is no trace-only or simpler trace-based control under a comparable training budget. Therefore, it remains unclear whether the proposed data composition provides meaningful benefits beyond additional agentic training. • A key motivation for limiting agent trajectories to 10–15% is that excessive SFT-like training may hurt subsequent RL trainability. However, the RL experiment contains only 20 updates and, as the authors acknowledge, does not demonstrate improved RL trainability. Without comparisons across trace proportions or against longer SFT under a meaningful RL budget, this central justification for the proposed mixture remains unverified.
Questions for authors
The topic is valuable, and the results show that midtraining can improve downstream agent performance. However, I do not think the current experiments sufficiently validate the specific recipe and its key motivations. These concerns are the main reason for my current score, and I would be willing to raise it if they can be addressed in the rebuttal.