acceptodds
Under review as a conference paper at ICLR 2027

SourceWorldBench: Can Coding Models Predict Test and Profiler Outcomes?

11 views

Abstract

Coding benchmarks increasingly focus on realistic repository-level software engineering tasks approached via agentic harnesses, where the models are allowed to explore repository contents and execute code. Supporting code execution for agentic harnesses requires sophisticated infrastructure, so code world modeling, i.e., simulating the program's execution, has recently gained traction. Performance optimization, in particular, is a common software engineering task that often requires high-load or long-running workloads and could substantially benefit from simulation. However, while simulating the logical code state during execution and end-to-end performance optimization are widely studied, it remains unknown whether models can reliably, say, simulate a profiler. Thus, we introduce SourceWorldBench, a repository-level benchmark comprising six tasks from various aspects of modeling software performance: test outcome prediction, execution time and traced memory estimation, and runtime and memory hotspot localization. We derive instances from an end-to-end performance optimization benchmark SWE-fficiency, augment them with agent-generated patches, and execute test suites and performance workloads to construct the benchmark labels. We evaluate six proprietary and open-weight models under four context collection baselines, ranging from no retrieval to privileged execution-informed context. For all six tasks, the best configurations outperform simple task-specific heuristics marking the possibility of progress, while many of the considered configurations do not, highlighting a substantial room for improvement. For example, for the task of test outcome prediction, Claude Opus 5 and GPT-5.6 Sol score 45.1 and 41.5 Macro F1 respectively, while a random coin flip scores 49.5. These findings establish SourceWorldBench as a challenging testbed for studying and improving optimization-adjacent code world modeling capabilities in isolation.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model Evaluation, Code Reasoning, Repository-level Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oJQ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SourceWorldBench, a repository-level benchmark that tests whether coding models can predict execution outcomes without running code. Its six tasks cover three capabilities: test outcome prediction (Single and Anchored), workload time and traced-memory estimation, and time and memory hotspot localization. Instances are built from SWE-fficiency tasks and agent-generated patches, and labels are obtained by executing tests and profiling workloads with cProfile and tracemalloc. Six models are evaluated under four context conditions: no retrieval, BM25, read-only agents, and an execution-trace oracle. The authors conclude that the tasks are challenging, since many configurations fail to beat simple heuristics.

Critical strengths
  • Meaningful problem decomposition. End-to-end optimization benchmarks report a single speedup that cannot be attributed to specific capabilities. Isolating behavior prediction, resource estimation, and bottleneck localization, with labels grounded in real execution, is a useful direction for diagnosing performance-related reasoning in code models.
  • Careful statistical reporting and full release. The paper uses paired bootstrap intervals with Holm correction and validates the Lite subset. It also releases the datasets, all model predictions, and the evaluation harness.
Critical weaknesses
  1. The low scores may be systematically underestimated, so the "challenging" conclusion is not solid.

    • On Single Test Status, the "below random" macro-F1 mostly reflects models predicting PASSED almost exclusively. Under balanced accuracy and MCC, the same models are above the random heuristic.
    • Only 2 of 23 configurations beat a constant predictor on Memory Prediction, and only 5 of 23 beat a call-count heuristic on Time Hotspots. These gaps likely reflect the label definitions and profiler overhead as much as model capability.
    • The read-only agents appear barely to explore, costing less per instance than no-retrieval on several tasks, and each configuration is run only once.
  2. The benchmark design is weakly aligned with its stated purpose of supporting optimization.

    • The labels are profiler readings rather than program performance: they include setup time and cProfile overhead, and tracemalloc sees only Python-level allocations. Measured speedups correlate with the original SWE-fficiency speedups at only ρ = 0.79 (Appendix H).
    • The tasks ask for absolute, machine-dependent seconds and bytes, while optimization mainly requires relative judgments, such as whether a patch is faster than the base.
    • There is no evidence that higher benchmark scores correspond to better optimization outcomes.
Questions for authors
  1. Under the debiased log-MAE, any constant predictor obtains the same score. Why do most configurations, including those given oracle context, perform worse than a constant on Memory Prediction?
  2. How do the hotspot labels and the call-count heuristic's advantage change if they are measured without cProfile, for example with a sampling profiler? What fraction of the measured time comes from workload setup?
1 comment

We appreciate your time giving such a detailed review that actually helped us investigate more to find and solve some technical difficulties.

On Single Test Status, the "below random" macro-F1 mostly reflects models predicting PASSED almost exclusively. Under balanced accuracy and MCC, the same models are above the random heuristic.

That is true, we need to fix the wording around that result. It is actually expected that LLMs cannot predict test outcomes without relevant context. And from what we see in the table, under No-Retrieval context, OpenAI models tend to produce PASSED, while Claude produces random answers.

Only 2 of 23 configurations beat a constant predictor on Memory Prediction, and only 5 of 23 beat a call-count heuristic on Time Hotspots.

  • We reran Memory Prediction, fixing the problems described in Appendix K. Now, 22 of 23 configurations beat a constant predictor
  • With Time Hotspots, it’s much harder to beat call_count heuristic, since we measure cumulative time. And the score is 0.456 which is quite high.

The read-only agents appear barely to explore, costing less per instance than no-retrieval on several tasks, and each configuration is run only once.

Thank you for the catch. We explored agents' behavior more, found a harness bug, and re-evaluated all agentic experiments for memory and memory hotspots, and all remaining agentic experiments using the codex harness.

The labels are profiler readings rather than program performance: they include setup time and cProfile overhead, and tracemalloc sees only Python-level allocations. Measured speedups correlate with the original SWE-fficiency speedups at only ρ = 0.79 (Appendix H).

  • We still believe that the labels are fair, since we added data collection protocol to the prompt. We also see that within the OpenAI family of models we see expected ranking patterns, eg, for No-Retrieval Luna < Terra < Sol; for Sol No-Retrieval < BM25 < Execution Trace Oracle.
  • That is why the benchmark methodology is valid to estimate the progress on the presented tasks.

The tasks ask for absolute, machine-dependent seconds and bytes, while optimization mainly requires relative judgments, such as whether a patch is faster than the base.

  • We solve the problem of being machine-dependent with the right choice of metrics.
  • We do not focus completely on optimization end-to-end tasks, but rather build a methodological base for pushing code understanding / reasoning about code behavior tasks further.
  • Relative judgement benchmark is indeed a great idea worth developing as an independent project. But real-world scenarios do not always need time-only optimizations, it is essential to keep memory consumption under control. For example, comparison does not differentiate between “memory usage went up from 1MB to 2MB” and “from 1MB to 1GB”. Such a problem is not solvable with comparisons only.
  • Additionally, we believe that predicting absolute values is less straightforward, but more promising, than relative judgements. For now, we heavily rely on existing workloads and a repository test suite. However, in the real world, that condition is not always true. Our benchmark design allows the community to explore different scenarios beyond pre-defined ones.

There is no evidence that higher benchmark scores correspond to better optimization outcomes.

Thank you for your comment! Based on that, we conducted additional experiment.

  • We run the following 5 models that SWE-fficiency reports results for: claude-opus-4-5, gemini-3-flash, deepseek-v3.1, glm-4.6, gpt-5-mini. This model selection covers different capability tiers and includes open-weights and proprietary LLMs.
  • The objective of this analysis is to check cohort speedup performance on SWE-fficiency, conditioned on whether any of the top 20 gold hotspots is in top1 position.
  • We compare top-1 prediction in top-20 gold hotspots vs top-1 prediction NOT in top-20 gold hotspots. The results for correct patches are:
    • For faster than pre-edit patches: 67.0% vs 61.3% (+5.8 pp)
    • For faster than expert patches: 31.0% vs 23.5% (+7.5 pp)
  • As we can see, identifying hotspots is associated with higher observed optimization success. That motivates the need to explicitly evaluate models on code understanding benchmarks, such as ours.

... Why do most configurations, including those given oracle context, perform worse than a constant on Memory Prediction?

Thank you for your question again, we fixed the table.

How do the hotspot labels and the call-count heuristic's advantage change if they are measured without cProfile ...

That is a great question! However, the results are not easily transferable, since we prompted all of the configurations with all the details about the profiling.

You can read about our new experiment in our reply to reviewer nEyF. This experiment shows that the way we collect data actually helps in the end-to-end optimization task.

Reviewer_xZBZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SourceWorldBench, which decomposes execution prediction relevant to repository-level optimization into six tasks: predicting a single test's status; predicting multiple test outcomes after a patch, given that all passed in the original state; estimating workload runtime and peak traced memory; and ranking time and memory hotspot functions.

Critical strengths

None

Critical weaknesses

I think the quality falls well short of NeurIPS standards. Most of the paper looks like it was likely generated by AI, so I'm inclined to reject it.

Questions for authors

None

1 comment

Thank you for your review. We totally understand a frustration from AI generated papers. However, we would appreciate specific feedback on which aspects of our research could be improved.

I think the quality falls well short of NeurIPS standards.

We would like to address your concern, but can you please clarify which part of our paper is not good enough? Is it writing, the general idea, benchmark design, validity, or anything else?

Most of the paper looks like it was likely generated by AI

  • As far as we know, that is not prohibited by ICLR rules.
  • We have a detailed AI Use Statement. And we are happy to provide any additional artifacts such as Research Proposal document, Overleaf Repository with history, Online Boards with brainstorming sessions, etc., to the Area Chairs for independent assessment of our statement.
Reviewer_nEyFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks the question if coding LLMs can be used to get quick proxies for predictors -- predict unit test pass/fail outcomes, performance and resource usage, hotspots, etc. Paper presents data showing the tradeoffs with such an approach, including variation across different models, but also insights around when such approaches could work (and not work).

Critical strengths

I was very excited about the thesis of the paper. Fast LLM proxies for the kind of questions the paper tries to predict would be a game changer for recursive self-improvement loops. So, very timely and interesting research area.

I also liked the paper's balanced presentation of the results identifying both when things could work and when it would not. Good contribution to our knowledge in this area.

Critical weaknesses

Ultimately, the paper's results are just too mixed and I am struggling to figure out how to use it. "Challenging but tractable" is a bit non-actionable. I did appreciate the nuanced discussions on when things could work (execution traces, avoiding too much context rotting for memory prediction, etc) but I would have loved to have the paper take the next step in demonstrating the benefits from this approach in some real use-cases. I suspect using even a slightly flawed but fast predictor in the early stages of recursive exploration followed by more detailed models could be very interesting, and thinking about leveraging their insights to come up with a family of predictors that can change speed accuracy tradeoffs might also be very useful.

Questions for authors

See above

3 comments

We would like to thank the reviewer for the engagement and an insightful review. We agree that in current form the paper’s results may seem too mixed. Let us be more precise with the following clarifications.

"Challenging but tractable" is a bit non-actionable.

  • “Challenging” means that in the best case scenario when we have execution traces, LLMs still have room for improvement. It is still worth investigating how to close this gap: for example, with better long context handling or LLM-specific execution data preprocessing.
  • “Tractable” means that there is still a gap between BM25 baseline and Execution Trace Oracle baseline. The difference between these baselines is that the latter requires actual code execution which eliminates a necessity to run LLM-based solutions. For a practitioner it implies a straightforward action: how to build an execution-free solution that beats Execution Trace Oracle baseline.
  • Overall, our baselines show that the presented benchmark design indicates improvements on the presented tasks.

I would have loved to have the paper take the next step in demonstrating the benefits from this approach in some real use-cases

That is completely true. And we conducted the following experiment to address this concern.

  • We took a subsample of 50 SWE-fficiency tasks
  • We ran them with official SWE-fficiency instructions, using OpenAI models from our Table 1.
  • We evaluated produced patches with the official harness.
  • Then, we forced the model to think more in 3 different regimes.
    • Control: prompting the agent to double-check, and think more before submission, no additional feedback.
    • Time: we provided real values of time before and after patch.
    • Time Hotspots: we provided a sorted list of time hotspots after patch.
  • The results are: | Model | Regime | Harmonic mean, run 1 → run 2 | Geometric mean, run 1 → run 2 | At or above expert | Geometric mean, run 2 over run 1 speedups, no outliers | | --- | --- | --- | --- | --- | --- | | GPT-5.6 Luna | Control | 0.586 → 0.561 | 1.20 → 1.18 | 23 → 25 | 0.98× (0.93–1.04) | | GPT-5.6 Luna | Time | 0.586 → 1.079 | 1.20 → 1.73 | 23 → 31 | 1.44× (1.24–1.73) | | GPT-5.6 Luna | Time Hotspots | 0.586 → 0.313 | 1.20 → 1.59 | 23 → 35 | 1.32× (1.06–1.63) | | GPT-5.6 Sol | Time | 0.318 → 1.264 | 1.42 → 2.10 | 29 → 37 | 1.48× (1.21–1.91) | | GPT-5.6 Sol | Time Hotspots | 0.318 → 0.333 | 1.42 → 2.14 | 29 → 39 | 1.51× (1.09–2.06) |
    • We report harmonic mean as the official metric of SWE-fficiency. But it is very sensitive to outliers, so we provide geometric mean, and absolute numbers of successful optimizations (out of 50).
  • The results show that additional profiler data improves agentic performance significantly.

I suspect using even a slightly flawed but fast predictor in the early stages of recursive exploration followed by more detailed models could be very interesting, and thinking about leveraging their insights to come up with a family of predictors that can change speed accuracy tradeoffs might also be very useful.

We agree and that is a great idea. However, building such a complex system contradicts with the paper’s idea of isolating capabilities. Do you think it is substantial to include such systems in a newer version of the paper?

Reviewer_nEyFOpenReview ↗

Acknowledging author response

Thanks for the additional comments. The new results were interesting. To your last question, I do think the direction is interesting. I will leave the decision on whether it belongs to this paper or a different paper to the authors.

Thank you for your response! We agree that this is an interesting direction, but believe that exploring it thoroughly is beyond the scope of the current paper and deserves a separate study. We hope the additional results address your concerns and that you will consider them in your updated assessment of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Changes in the main text

After reading feedback from reviewers, we reran our baseline experiments and updated Table 1 and the results discussion. We did the following.

  • Analyzed model answers in Memory tasks and changed the prompt as was previously discussed in Appendix K.
  • Analyzed agentic trajectories for read-only agentic baselines. Found a bug that Codex harness rejected all the tool calls, fixed it and reran the experiments.

We will work on the affected Appendix sections after the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.