acceptodds
Under review as a conference paper at ICLR 2027

nanoswe: Speedrunning SWE-bench

Abstract

Speedrunning has become a popular engineering challenge in machine learning: achieve the best performance within a fixed compute budget. We use it as a scientific tool to study how models acquire specialized skills and how far those skills transfer. We speedrun SWE-bench by training coding agents from scratch on demonstrations from stronger models. With 192 B200-hours of training compute, we reach 25.5% pass@1 on SWE-bench Verified, outperforming GPT-4o. Scaling laws fitted to smaller runs forecast this result to within 0.3 percentage points. At matched pass@1, our agents solve largely the same SWE-bench problems as fine-tunes of a model pre-trained on trillions of tokens, and generalize about as well to new bugs and codebases. Beyond SWE-bench, however, our speedrun's capabilities are strikingly jagged: on MMLU, they do worse than GPT-2. What, then, does pre-training buy for SWE-bench? Controlled experiments support a one-student hypothesis: pre-training helps a student imitate its teacher, but once we know how well it does so, how it was pre-trained barely matters. The hypothesis holds even when comparing a model trained on the web against a "vintage" one trained only on text from before 1931. Up to the SWE-bench scores we reach, pre-training is interchangeable with specialization data. We release nanoswe as an open project for studying specialization through speedrunning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, coding agents, speedrun

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_ZGMqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how well a software engineering model can be trained under a fixed compute budget, given specific amounts of pre-training and agent data. It also explores related scaling laws for using small-scale experiments to predict performance under larger training budgets. Finally, the authors argue that the value of pre-training must be measured according to the task and the budget, offering a valuable counterexample to the assumption that general pre-training is always a necessary stepping stone to acquiring specialized capabilities.

Critical strengths
  1. The author's experimental design is very thoughtful. Instead of immediately scaling up to run full-scale experiments, they started with small-scale tests, predicted how to allocate the budget most cost-effectively, and obtained a great scaling law curve.
  2. This article provides a strong argument for the one-student hypothesis.
  3. The author constructed and utilized multiple test sets to investigate their experimental results and claims, and proved that their claims remained valid across various datasets and among different datasets.
Critical weaknesses
  1. I feel the article's title could be a bit more straightforward. The current title makes it feel like yet another new auto-research competition, seeing how powerful a software engineering agent can be trained within specific budget. But the paper is about the authors themselves conducting a comprehensive and in-depth study.
  2. Regarding the authors' claim that their hypothesis can also explain the phenomenon where a strong teacher isn't necessarily better for a weak student, I see two main issues: 1) Their "one student hypothesis" states that we should look at the imitation loss on held-out teacher demonstrations. However, in their experiments, they didn't actually check whether the two student models have the same imitation loss on a common, held-out set of teacher demonstrations. Moreover, it's hard to determine whether you should use the imitation loss from the weak teacher or the strong teacher. If you factor that in, their hypothesis might become untestable, because a student model trained by a weak teacher and one trained by a strong teacher should each have a relatively low imitation loss on their respective teachers. 2) The evaluation in this paper is done entirely on SFT models, and the results might change after RL. At the same time, there are many other papers offering different explanations for this phenomenon. If the authors want to argue this point, they really need to patch up the logical gaps in the first part and discuss related works.
Questions for authors

NA

3 comments

Dear Reviewer ZGMq,

Thank you for your thoughtful review. Below, we address your main concerns.

In their experiments, they didn't actually check whether the two student models have the same imitation loss on a common, held-out set of teacher demonstrations. Moreover, it's hard to determine whether you should use the imitation loss from the weak teacher or the strong teacher.

We score each model on held-out demonstrations from the teacher it was trained on, and fit two separate scaling laws, one per teacher. Each predicts pass@1 after SFT on that teacher, by relating imitation loss to pass@1 (Equation 6, Figure 7 three leftmost panels). To plot the two teacher laws on a single axis (Figure 7, rightmost panel), we use an empirical regularity. Take any student and SFT budget, and fine-tune the student once on each teacher. The two resulting imitation losses satisfy LQwen 3.6≈0.91 LQwen Coder−0.049L_{\text{Qwen 3.6}} \approx 0.91\,L_{\text{Qwen Coder}} - 0.049 (Figure 15 left, Appendix M; see [1] for similar relations across training datasets). We can therefore place each Qwen 3.6 fine-tune at the Qwen Coder loss its student would reach with the same SFT budget. This only shifts and rescales the loss axis of the Qwen 3.6 law, and no model is ever scored on the other teacher's held-out set.

Because both teacher laws hold for every student (our "one-student hypothesis"), as does the aforementioned regularity, a single threshold decides between the two teachers. That is, Qwen Coder is the better teacher for a given student and SFT budget if the student, fine-tuned on Qwen Coder, would attain an imitation loss above approximately 0.148 bpb (Figure 15, top). Equivalently, Qwen 3.6 is the better teacher if the student, fine-tuned on Qwen 3.6, would attain an imitation loss below approximately 0.086 bpb (by applying the linear rule above, Figure 15, bottom).

If you factor that in, their hypothesis might become untestable, because a student model trained by a weak teacher and one trained by a strong teacher should each have a relatively low imitation loss on their respective teachers.

We only measure a model's imitation loss on the teacher it was fine-tuned on, never on the other one. The rules above lead to testable predictions. For each student, the table gives the SFT budget beyond which Qwen 3.6 is predicted to be the better teacher (Figure 15 right, Appendix M), with the loss-to-pass@1 laws fitted on all students or refitted without that student, and compares it with the empirical observations (Figure 7, three leftmost panels).

StudentPredicted cut-off, all studentsPredicted cut-off, student held outMatches observations?
SmolLM3 base8B6.7BYes: Qwen Coder better at every budget we train
SmolLM3 mid-trained0.93B1.1BYes: Qwen 3.6 better from about 1B
Qwen 3 4B0.14B0.12BYes: Qwen 3.6 better starting from 0.18B

The evaluation in this paper is done entirely on SFT models, and the results might change after RL. At the same time, there are many other papers offering different explanations for this phenomenon.

Our claims concern SFT only, and we make no claim about what happens after RL. Whether students with the same imitation loss respond differently to subsequent RL is a very interesting research question that we leave for future work.

We already discuss other explanations for why a stronger teacher can yield a weaker student in our related work (Section 1.2 and Appendix J). Prior work attributes this phenomenon to a capacity gap between teacher and student. We instead consider models with similar sizes but different pre-training choices.

In response to other reviewers' comments, we have further tested the one-student hypothesis. Without refitting, the ability law fitted on our from-scratch scaling sweep (Section 2.1) predicts the post-SFT pass@1 of SmolLM3 base and mid-trained, Qwen 3 4B, Talkie 13B web and vintage, and Marin's Grug and Snowball 67B-A2B from their imitation loss alone, to within 2.4pp (Figure 13, Appendix K). Combined with a loss law fitted on three smaller Marin models, it also predicts the final pass@1 of Snowball (1.6e23 FLOPs, ~6,600x the pre-training compute of our largest sweep model) to within 1.6pp (Figure 14, Appendix L). This further validates our methodology for predicting student performance from imitation loss.

We hope our response addresses your concerns, and would be grateful if you would consider raising your score.

[1] Brandfonbrener et al. (2025). Loss-to-Loss Prediction: Scaling Laws for All Datasets. TMLR.

Thank you for the detailed response and additional experiments. The clarification on how imitation loss is measured is helpful, but my overall assessment is unchanged. I will keep my score.

We are glad the clarification was helpful. Thank you again for your positive assessment of our work.

Reviewer_rYTYOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines, from a number of angles, the question of "what is the best model that you can train for swe-bench given an allocation of one B200 node day". It examines scaling laws, variations of the base model, etc.

Critical strengths
  1. The problem setting is an interesting one that I don't think has been examined before, and it was studied well from a number of angles.
  2. Overall, the paper was interesting and engaging to read. It was refreshing in the pile of other papers that I had to review.
Critical weaknesses

None

Questions for authors
  1. How many different functional forms were tried to fit the SWE-bench accuracy scaling law? It seems to be some sort of quadratic function, but looking at the fit it doesn't seem extremely good so I'm a little bit incredulous that it was actually able to hit close to 25.5 without significant parameter tuning (which would weaken the result).

  2. If you could do a more qualitative analysis of the types of problems that the speed-run models were able to do well/poorly compared to smollm, that would be interesting. I expect that the speedrun models would be lacking world knowledge, but have good procedural knowledge of how to fix bugs, so certainly there's a ceiling in some bug types but maybe not others.

1 comment

Dear Reviewer rYTY,

Thank you for your positive assessment of our work.

Number of functional forms tried and quality of the fit

The curve in Figure 1 is not fitted to the sweep models: it is the predicted efficiency frontier, the best pass@1 reachable at each compute budget according to our two laws (Figure 2). Most sweep models are not compute-optimal, so they lie below it by construction.

We tried 6 different functional forms, of type L = E + A N^{−α} + S(N, D). We chose S(C, D)=C^{-gamma}D^{-beta} because it led to the lowest BIC metric (fit error relative to parameters) on the scaling sweep. For the ability law (Equation 3), we tried only the linear form and kept it, since it already fits well (R² = 0.98, with residuals at the level of measurement noise).

As further evidence that the law is not specific to our sweep, we test the same functional form, refitted on data that was not public when we chose it: Marin's Snowball pre-training ladder. The law predicts Snowball's test loss after 7.55B fine-tuning tokens to within 2.1%. Chaining this prediction into the ability law of Section 2.1 gives 41.3% pass@1, against 39.7% measured (Figure 14, Appendix L). The prediction misses by only 1.6pp, despite extrapolating ~136x in pre-training compute for the perplexity law and ~6,600x for the ability law. The functional forms of the paper thus hold well beyond our own scaling sweep.

What are the types of problems that the speedrun models were able to do well compared to smollm?

Thank you for the suggestion. We ran this analysis and added it to Appendix N. We annotated every SWE-bench problem with a 14-item rubric (LLM-scored in two independent passes, inter-pass reliability 0.78–0.94), including whether the fix requires knowledge not derivable from the repository and whether a routine reproduce–localize–edit loop suffices. Both point in the direction you expect (Pearson correlation r = −0.11 and r = +0.13 with the difficulty difference), but neither is statistically distinguishable from zero given the small sample size.

What does show a significant effect is guidance in the issue: the speedruns fall behind when the issue does not propose a fix or the fault is hard to localize (|r| ≈ 0.3 for each). Together, these items explain about 7% of the variance, small but well above chance (Figure 16, Appendix N). This result is consistent with the speedruns doing relatively better when the difficulty lies mostly in executing a plan that the issue already lays out.

However, we stress that these differences are small and rest on few problems. No rubric item correlates with the difference beyond |r| = 0.3, and only about 17 of the SWE-bench problems show a large difference in difficulty between SmolLM3 and the speedruns.

Reviewer_Dg2ZOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that surprisingly small pre-trained LLM models, and uninitialized LLM models can both be trained to perform well on SWE-bench Verified, beating GPT-4o and Claude 3 Opus. The paper shows that how well the model performs on SWE-Bench test is best determined by how well the model does imitating the teacher, what the imitation loss is on the teacher's solutions, not on whether it was pre-trained or not or what it was pretrained on. The paper shows that finetuning on the SWE-Bench domain on a sufficiently large enough dataset will result in an LLM that does well on SWE-Bench testset, even if it is a randomly initialized LLM with no broad pre-training exposure. The paper shows that outside the SWE-Bench domain the models trained from scratch do poorly on other domains like HumanEval, GSM8k, MMLU, where pre-trained models to much better outside of the domain. The paper creates a nice framework for "speed-running" SWE Bench and establishes some baseline benchmarks.

Critical strengths

The paper sets up a framework for other researchers to use for "speed-running" on SWE-Bench, with good initial baselines. The paper shows how for SWE-bench Verified a randomly initialized student LLM can be trained to perform well using relatively few domain specific training tokens, but the randomly initialized LLM won't do well outside the SWE-Bench domain. In comparison the paper shows a standard pre-trained LLM will learn quicker than randomly initialized, and do much better outside the SWE-Bench domain after being fine-tuned on SWE-Bench data. The paper shows for in-domain performance on SWE-Bench test set, the imitation loss of the student on the teacher's output predicts how well the student will do on the SWE-Bench testset. The paper shows scaling laws holding in an agentic task, and multiple smaller runs can find coefficients to reliably predict performance on larger models & longer runs.

Critical weaknesses

The result that the student's loss on the teacher's training trajectories for SWE-Bench corresponds to the student's performance on SWE-Bench isn't super surprising, it seems like for the in-domain test sets that would be expected.

Questions for authors

In figure 7 left, shouldn't the lines start at the same point at 0.0B SFT tokens? It doesn't appear they would.

3 comments

Dear Reviewer Dg2Z,

Thank you for your positive assessment of our work. Below, we answer your question on Figure 7 and address your main concern about the surprisingness of our findings.

In figure 7 left, shouldn't the lines start at the same point at 0.0B SFT tokens?

Yes, the fitted curves meet at 0 SFT tokens. However, the plot is log-scaled and thus can't show 0 tokens. We draw curves starting at 0.1B, where the choice of teacher already matters.

student's loss on the teacher's training trajectories for SWE-Bench corresponds to the student's performance on SWE-Bench isn't super surprising, it seems like for the in-domain test sets that would be expected.

We agree that within one model family, lower imitation loss should mean higher in-domain pass@1. Our findings are much stronger: the relationship between imitation loss and performance is the same for students whose pre-training differs widely. The ability law of Section 2.1, fitted only on our from-scratch sweep models, predicts without refitting the post-SFT pass@1 of SmolLM3 base and mid-trained, Qwen3-4B, Talkie web and vintage, and Marin's Grug and Snowball 67B-A2B (1.6e23 FLOPs), to within 2.4pp (RMSE 1.8pp, over pass@1 of 15–40%, see new Figure 13, Appendix K).

This is not implied by the test set being in-domain. Imitation loss is teacher-forced, while pass@1 is closed-loop over many turns, where errors compound [1]. How well a student recovers from its own errors is not measured by imitation loss and plausibly depends on pre-training, so students with equal imitation loss could well differ in pass@1. Moreover, in light of prior work, it is highly surprising that the relationship between imitation loss and performance is invariant across widely different pre-training choices. Loss-to-performance relations shift with pre-training data [2, 3] and can fail across model families after SFT [4], and equal pre-training loss can hide different downstream ability [5, 6, 7]. It is also widely held that SFT cannot compensate for a weak base [8, 9, 10, 11]. We show instead one important domain (simple agentic issue fixing) where pre-training and specialization are largely interchangeable, at least at the compute scales considered.

Second, and just as surprisingly, this relationship can be learned from small models with little pre-training and still holds for models with orders of magnitude more pre-training FLOPs. Combining the ability law from our 105 B200-hour sweep with a loss law fitted on Marin's three-model Snowflake ladder, we predict Snowball's post-SFT pass@1 at 41.3%, against 39.7% measured (Figure 14, Appendix L). The ability law was fitted on models with at most 7B pre-training tokens, yet extrapolates to one with 10T (~6,600x more pre-training FLOPs). Our methodology thus opens the door to co-designing pre- and post-training via end-to-end scaling ladders, which has so far been difficult because pre-training metrics are poor predictors of post-training performance [12, 13].

We hope our response addresses your initial concern, and would be grateful if you would consider raising your score.

[1] Ross et al. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS.

[2] Mayilvahanan et al. (2025). LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws. ICML.

[3] Isik et al. (2025). Scaling Laws for Downstream Task Performance in Machine Translation. ICLR.

[4] O'Neill et al. (2026). Post-Training Science for Supervised Fine-Tuning. arXiv:2609.01244.

[5] Liu et al. (2023). Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models. ICML.

[6] Tay et al. (2022). Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers. ICLR.

[7] Tay et al. (2023). Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? Findings of EMNLP.

[8] Akter et al. (2026). Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data. ICLR.

[9] Liu et al. (2026). Midtraining Bridges Pretraining and Posttraining Distributions. ICML.

[10] Gudibande et al. (2024). The False Promise of Imitating Proprietary Language Models. ICLR.

[11] Ye et al. (2025). LIMO: Less is More for Reasoning. COLM.

[12] Zeng et al. (2025). Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs? arXiv:2504.12491.

[13] Han et al. (2026). Weight Decay Improves Language Model Plasticity. ICML.

Thanks for the updates, I like the paper and hope it gets accepted.

Thank you for your response and for your positive assessment of our work.

Comments on the paper

1 comment

We thank all reviewers for their feedback and for their very positive assessment of our work. We have uploaded a revised paper with new appendix sections (K–N, in blue) that address the reviewers' concerns. We highlight two results.

First, the ability law of Section 2.1, fitted only on small from-scratch models, predicts without refitting the post-SFT pass@1 of models with widely different pre-training: SmolLM3 base and mid-trained, Qwen3-4B, Talkie 13B web and vintage (pre-1931 text only), and Marin's Grug and Snowball 67B-A2B. It is accurate to within 2.4pp (RMSE 1.8pp, over pass@1 of 15–40%, Appendix K). This is strong validation of the one-student hypothesis introduced in the paper.

Second, we apply the paper's methodology to Marin's new Snowball 67B-A2B model. From its three-model pre-training ladder, which was not public at the time of submission, we predict Snowball's final pass@1 at 41.3%, against 39.7% measured (Appendix L). That is a gap of only 1.6pp, despite extrapolating ~136x in pre-training FLOPs for the loss law and ~6,600x for the ability law. These results suggest that our methodology can forecast post-SFT performance (i.e., after the cold-start phase before RL) from small pre-training ladders, opening the door to closer co-design of pre- and post-training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.