- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how well a software engineering model can be trained under a fixed compute budget, given specific amounts of pre-training and agent data. It also explores related scaling laws for using small-scale experiments to predict performance under larger training budgets. Finally, the authors argue that the value of pre-training must be measured according to the task and the budget, offering a valuable counterexample to the assumption that general pre-training is always a necessary stepping stone to acquiring specialized capabilities.
Critical strengths
- The author's experimental design is very thoughtful. Instead of immediately scaling up to run full-scale experiments, they started with small-scale tests, predicted how to allocate the budget most cost-effectively, and obtained a great scaling law curve.
- This article provides a strong argument for the one-student hypothesis.
- The author constructed and utilized multiple test sets to investigate their experimental results and claims, and proved that their claims remained valid across various datasets and among different datasets.
Critical weaknesses
- I feel the article's title could be a bit more straightforward. The current title makes it feel like yet another new auto-research competition, seeing how powerful a software engineering agent can be trained within specific budget. But the paper is about the authors themselves conducting a comprehensive and in-depth study.
- Regarding the authors' claim that their hypothesis can also explain the phenomenon where a strong teacher isn't necessarily better for a weak student, I see two main issues: 1) Their "one student hypothesis" states that we should look at the imitation loss on held-out teacher demonstrations. However, in their experiments, they didn't actually check whether the two student models have the same imitation loss on a common, held-out set of teacher demonstrations. Moreover, it's hard to determine whether you should use the imitation loss from the weak teacher or the strong teacher. If you factor that in, their hypothesis might become untestable, because a student model trained by a weak teacher and one trained by a strong teacher should each have a relatively low imitation loss on their respective teachers. 2) The evaluation in this paper is done entirely on SFT models, and the results might change after RL. At the same time, there are many other papers offering different explanations for this phenomenon. If the authors want to argue this point, they really need to patch up the logical gaps in the first part and discuss related works.
Questions for authors
NA
3 comments
Thank you for the detailed response and additional experiments. The clarification on how imitation loss is measured is helpful, but my overall assessment is unchanged. I will keep my score.
We are glad the clarification was helpful. Thank you again for your positive assessment of our work.
Dear Reviewer ZGMq,
Thank you for your thoughtful review. Below, we address your main concerns.
We score each model on held-out demonstrations from the teacher it was trained on, and fit two separate scaling laws, one per teacher. Each predicts pass@1 after SFT on that teacher, by relating imitation loss to pass@1 (Equation 6, Figure 7 three leftmost panels). To plot the two teacher laws on a single axis (Figure 7, rightmost panel), we use an empirical regularity. Take any student and SFT budget, and fine-tune the student once on each teacher. The two resulting imitation losses satisfy LQwen 3.6≈0.91LQwen Coder−0.049 (Figure 15 left, Appendix M; see [1] for similar relations across training datasets). We can therefore place each Qwen 3.6 fine-tune at the Qwen Coder loss its student would reach with the same SFT budget. This only shifts and rescales the loss axis of the Qwen 3.6 law, and no model is ever scored on the other teacher's held-out set.
Because both teacher laws hold for every student (our "one-student hypothesis"), as does the aforementioned regularity, a single threshold decides between the two teachers. That is, Qwen Coder is the better teacher for a given student and SFT budget if the student, fine-tuned on Qwen Coder, would attain an imitation loss above approximately 0.148 bpb (Figure 15, top). Equivalently, Qwen 3.6 is the better teacher if the student, fine-tuned on Qwen 3.6, would attain an imitation loss below approximately 0.086 bpb (by applying the linear rule above, Figure 15, bottom).
We only measure a model's imitation loss on the teacher it was fine-tuned on, never on the other one. The rules above lead to testable predictions. For each student, the table gives the SFT budget beyond which Qwen 3.6 is predicted to be the better teacher (Figure 15 right, Appendix M), with the loss-to-pass@1 laws fitted on all students or refitted without that student, and compares it with the empirical observations (Figure 7, three leftmost panels).
Our claims concern SFT only, and we make no claim about what happens after RL. Whether students with the same imitation loss respond differently to subsequent RL is a very interesting research question that we leave for future work.
We already discuss other explanations for why a stronger teacher can yield a weaker student in our related work (Section 1.2 and Appendix J). Prior work attributes this phenomenon to a capacity gap between teacher and student. We instead consider models with similar sizes but different pre-training choices.
In response to other reviewers' comments, we have further tested the one-student hypothesis. Without refitting, the ability law fitted on our from-scratch scaling sweep (Section 2.1) predicts the post-SFT pass@1 of SmolLM3 base and mid-trained, Qwen 3 4B, Talkie 13B web and vintage, and Marin's Grug and Snowball 67B-A2B from their imitation loss alone, to within 2.4pp (Figure 13, Appendix K). Combined with a loss law fitted on three smaller Marin models, it also predicts the final pass@1 of Snowball (1.6e23 FLOPs, ~6,600x the pre-training compute of our largest sweep model) to within 1.6pp (Figure 14, Appendix L). This further validates our methodology for predicting student performance from imitation loss.
We hope our response addresses your concerns, and would be grateful if you would consider raising your score.
[1] Brandfonbrener et al. (2025). Loss-to-Loss Prediction: Scaling Laws for All Datasets. TMLR.