- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SourceWorldBench, a repository-level benchmark that tests whether coding models can predict execution outcomes without running code. Its six tasks cover three capabilities: test outcome prediction (Single and Anchored), workload time and traced-memory estimation, and time and memory hotspot localization. Instances are built from SWE-fficiency tasks and agent-generated patches, and labels are obtained by executing tests and profiling workloads with cProfile and tracemalloc. Six models are evaluated under four context conditions: no retrieval, BM25, read-only agents, and an execution-trace oracle. The authors conclude that the tasks are challenging, since many configurations fail to beat simple heuristics.
Critical strengths
- Meaningful problem decomposition. End-to-end optimization benchmarks report a single speedup that cannot be attributed to specific capabilities. Isolating behavior prediction, resource estimation, and bottleneck localization, with labels grounded in real execution, is a useful direction for diagnosing performance-related reasoning in code models.
- Careful statistical reporting and full release. The paper uses paired bootstrap intervals with Holm correction and validates the Lite subset. It also releases the datasets, all model predictions, and the evaluation harness.
Critical weaknesses
-
The low scores may be systematically underestimated, so the "challenging" conclusion is not solid.
- On Single Test Status, the "below random" macro-F1 mostly reflects models predicting PASSED almost exclusively. Under balanced accuracy and MCC, the same models are above the random heuristic.
- Only 2 of 23 configurations beat a constant predictor on Memory Prediction, and only 5 of 23 beat a call-count heuristic on Time Hotspots. These gaps likely reflect the label definitions and profiler overhead as much as model capability.
- The read-only agents appear barely to explore, costing less per instance than no-retrieval on several tasks, and each configuration is run only once.
-
The benchmark design is weakly aligned with its stated purpose of supporting optimization.
- The labels are profiler readings rather than program performance: they include setup time and cProfile overhead, and tracemalloc sees only Python-level allocations. Measured speedups correlate with the original SWE-fficiency speedups at only ρ = 0.79 (Appendix H).
- The tasks ask for absolute, machine-dependent seconds and bytes, while optimization mainly requires relative judgments, such as whether a patch is faster than the base.
- There is no evidence that higher benchmark scores correspond to better optimization outcomes.
Questions for authors
- Under the debiased log-MAE, any constant predictor obtains the same score. Why do most configurations, including those given oracle context, perform worse than a constant on Memory Prediction?
- How do the hotspot labels and the call-count heuristic's advantage change if they are measured without cProfile, for example with a sampling profiler? What fraction of the measured time comes from workload setup?
We appreciate your time giving such a detailed review that actually helped us investigate more to find and solve some technical difficulties.
That is true, we need to fix the wording around that result. It is actually expected that LLMs cannot predict test outcomes without relevant context. And from what we see in the table, under No-Retrieval context, OpenAI models tend to produce PASSED, while Claude produces random answers.
call_countheuristic, since we measure cumulative time. And the score is 0.456 which is quite high.Thank you for the catch. We explored agents' behavior more, found a harness bug, and re-evaluated all agentic experiments for memory and memory hotspots, and all remaining agentic experiments using the codex harness.
Thank you for your comment! Based on that, we conducted additional experiment.
claude-opus-4-5,gemini-3-flash,deepseek-v3.1,glm-4.6,gpt-5-mini. This model selection covers different capability tiers and includes open-weights and proprietary LLMs.Thank you for your question again, we fixed the table.
That is a great question! However, the results are not easily transferable, since we prompted all of the configurations with all the details about the profiling.
You can read about our new experiment in our reply to reviewer nEyF. This experiment shows that the way we collect data actually helps in the end-to-end optimization task.