acceptodds
Under review as a conference paper at ICLR 2027

When Code Crawls: Benchmarking Proactive Optimization in Coding Agents

Abstract

On benchmarks such as SWE-bench, coding agents now resolve a substantial share of real software tasks end to end. What those benchmarks score is functional correctness: runtime is not part of the score, so a program that returns the right answer after an hour still passes. Benchmarks that measure runtime either score the efficiency of generated code or ask for a performance improvement outright. In production, speed is part of usefulness, and engineers fix a step that runs far longer than it should instead of letting it finish. We study whether coding agents self-initiate performance optimization when performance is not an explicit task objective. We introduce CrawlBench, which runs the same repair task under two framings: the slowness is either stated in the prompt or left for the agent to discover and repair on its own. Every pathological case is paired with a healthy twin, so efficiency is the only quantity that varies. The reported pool is 192 cases: 88 single-bug Python artifacts, 12 held-out decay tiers, 19 two-bug stacks, 20 regressions mined from merged performance-fix pull requests, and 53 single-bug asymptotic replicas (28 in C, 25 in JavaScript). Across ten production models, agents told the program is slow fix 93.6–97.7% of the single-bug pool; run on the identical task without the hint, the same agents fix 23.5–90.2%. The gap spans +7.2 to +72.0 points. On all ten models, each run for three seeds, the implicit rate has sd ≤4.5 points. Every model repairs 93.6–97.7% once told. Implicit success still requires the agent to discover the slowness, judge it worth fixing, and modify the code unprompted. That self-initiation is what fails. Failure-mode triage finds a miss is the largest failure for every model, at 52–88% of unfixed implicit episodes: the submission stays correct but slow. Fast-and-wrong submissions are 2.0% of those unfixed episodes. We release the benchmark, harness, and all cases.

Primary area:
datasets and benchmarks
Keywords:
agents, coding, efficiency, proactiveness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LGtaOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CrawlBench, a benchmark for evaluating proactive efficiency awareness: whether coding agents initiate performance optimization when it is not an explicit task objective. Each case pairs a pathologically slow program with a functionally equivalent healthy twin. The same task is evaluated under two framings. Under explicit framing, the agent is told that the program is slow and asked to fix it. Under implicit framing, the agent is asked to run the program and report its output, without any mention of performance. A submission counts as fixed only if it is both correct and fast relative to the healthy twin. The main quantity of interest is the explicit-implicit gap in fix rates. The authors evaluate ten models using mini-SWE-agent. The main benchmark contains 88 single-bug Python cases covering asymptotic complexity, execution-model constants, external I/O latency, and declarative-engine cost. Additional experiments examine slowdown severity, two-bug stacks, regressions mined from real performance-fix PRs, and C and JavaScript replicas, bringing the reported pool to 192 cases. On the main benchmark, explicit fix rates range from 93.6% to 97.7%, while implicit fix rates range from 23.5% to 90.2%. The largest failure category for every model is miss: submissions that remain correct but slow. These results suggest that strong performance under explicit optimization requests does not consistently translate into proactive optimization.

Critical strengths

The main strength of this paper is its focus on the implicit setting. Existing work such as SWE-fficiency already studies performance optimization on real workloads, including bottleneck localization and the changes needed to improve runtime while preserving correctness. CrawlBench adds a useful question: when users do not explicitly ask for faster code, will an agent still recognize a serious performance problem and decide to address it? I find this distinction meaningful. An agent may know how to optimize a program once asked, yet consider its work complete as soon as the program produces the correct output. The high explicit fix rates, together with the frequent correct-but-slow outcomes under implicit framing, provide evidence of this gap. They suggest that, for many of the evaluated models under this scaffold, the runtime efficiency of the final program is not yet a consistently considered part of completing a task. The timing-visibility control provides some additional support for this observation. For the two models tested, showing the runtime of every command improves implicit fix rates by only 2 and 8 percentage. Although limited in scope, this result suggests that making timing information available may not be sufficient to trigger optimization. The paper therefore offers a useful perspective for agent evaluation and design: agents should be assessed both on their ability to fix performance problems when asked and on whether they appropriately initiate that work without an explicit request.

Critical weaknesses

I appreciate the authors’ effort to study whether coding agents self-initiate performance optimization when speed is not an explicit task objective. However, I remain unconvinced that the evidence presented establishes the broader value of the optimization behavior being measured. My concerns are substantive, and I encourage the authors to address each of them directly and in detail.

First, the paper has not sufficiently explained to what extent faster execution on the benchmark reflects meaningful proactive optimization capability. The explicit framing asks the agent to improve performance, while the implicit framing examines whether it initiates optimization on its own. In both settings, however, success is ultimately determined by whether the final submission produces matching output and meets the runtime criterion. These results show whether an agent meets the benchmark's performance requirement under different prompts, but do not sufficiently establish whether it identifies the right problem, chooses an appropriate modification, or makes an improvement of practical value. Real performance optimization involves more than obtaining a shorter runtime: it requires understanding why a program is slow, where it should be changed, and whether the change is worthwhile. The paper covers asymptotic complexity, execution-model constants, external I/O latency, and declarative-engine cost, demonstrating coverage of different sources of slowness. Yet covering different sources of slowness is not the same as distinguishing between optimizations of different quality. Likewise, leaving code unchanged does not by itself establish a lack of proactive efficiency awareness, and modifying code and meeting the runtime criterion does not by itself establish sound engineering judgment. The failure-mode triage and healthy-twin control provide relevant evidence, but do not resolve this distinction. The explicit-implicit gap is therefore a useful behavioral observation, but the paper still needs to explain which optimization capabilities it reveals and why those capabilities are meaningful measures of progress in coding agents.

More fundamentally, I question an approach to optimizing and evaluating agents that gives runtime such pure and overwhelming priority. SWE-fficiency already treats runtime improvement as a core objective in repository-level performance optimization. In the broader setting of today's LLM coding agents, I consider the more important question to be which aspects of a complex engineering architecture shape agent capability and, in turn, the runtime of the optimized code. A more fundamental direction is to identify recurring optimization patterns and use them to inform agent orchestration frameworks. This paper does not pursue that deeper connection between optimization behavior and agent orchestration. Instead, its central evaluation criterion remains whether the final submission meets the runtime requirement while preserving correctness. In my view, this framing is no longer adequate for the current stage of LLM coding-agent development.

Second, the paper needs a direct comparison of the healthy twin, the agent-modified version, and the original pathological version for the same task, with the performance changes tied to the actual code changes. The authors already provide the healthy and pathological versions and report their runtimes. The verifier also measures the agent's final submission against the pristine reference. What is still missing is a systematic analysis that brings the final agent-modified code into this comparison and explains exactly what optimization it achieves. For the same task, I need to see how long each of the three versions takes to run, why the original version is slow, how the healthy version addresses that problem, what the agent changes, and why those changes produce the reported speedup. If the agent adopts a different approach from the healthy twin, the authors should also explain how the approaches differ and how those differences affect performance. Only by examining the runtimes of all three versions alongside their code differences can we judge what problem the agent actually solves, rather than merely whether it passes the tests.

The current summaries of fix rates do not answer my concern about “shortcut-taking” by LLM coding agents. With runtime as the dominant optimization target, the incentives created by this evaluation criterion could encourage improvements that exist primarily at the level of the test rather than in the underlying program. The same label of “success” can conceal very different modifications and performance gains. For example, suppose we expect an agent to identify a quadratic bubble-sort routine and replace it with an efficient sorting algorithm. A patch that instead replays stored input and output pairs for repeated benchmark inputs could appear faster without addressing the sorting bottleneck. If accepted by the evaluation, this would be a test-specific shortcut rather than the improvement we intended to measure. The held-out-seed check is intended to prevent simple reuse of observed outputs, but it does not by itself establish what the successful patches actually optimize. The paper still needs to show whether those submissions make substantive improvements and how the evaluation distinguishes them from benchmark-specific shortcuts. The analysis of unfixed implicit episodes does not answer this question about successful submissions. Providing the three versions' runtimes and code differences would make it possible to examine which underlying optimization issues the agent and the healthy implementation address in common, where their approaches diverge, and what those differences mean for the value of the optimization.

Questions for authors

How do the authors position runtime improvement in evaluating coding agents, and what analysis of agent components and code changes establishes the benchmark’s value beyond measuring speedup?

5 comments
AuthorsOpenReview ↗

Evaluation value, agent components and code comparison

Thank you for your questions.

What the score measures. CrawlBench measures whether agents initiate correctness-preserving optimization without an explicit performance request. Users may overlook performance while still valuing it. Large slowdowns (16–10730×) motivate studying this behavior. To count as a successful repair, a submission must meet the runtime criterion while preserving output. Success is binary within 5× of the healthy reference on synthetic cases, so progressively larger speedups receive no extra credit.

The target is the full behavior, not an internal noticing stage. Healthy-twin controls test selectivity: healthy hot-path edits occur in 1.1–9.1% of episodes, versus 23.5–90.2% repair rates on slow code.

Agent components. We test whether guidance triggers optimization. Estimates use 88 cases per model and paired implicit baselines; intervals are case-cluster bootstrap 95% intervals.

Added guidanceFix-rate gain (pp), 95% intervalCoverage
Edit permission+0.9 [−3.2, 5.2]Four models, three seeds
Expected runtime+16.9 [12.8, 20.8]Four models, three seeds
Measured runtime in observations+4.3 [0.6, 8.0]Four models, seed 1
Generic pre-submission self-check+1.1 [−6.8, 9.7]Two models, seed 1

The first three use DeepSeek V4 Flash, Qwen3.8 27B, Claude Opus 5 and GPT 5.6 Sol. The self-check uses Qwen and Opus with "Before submitting, check whether your work fully accomplishes the task."

Expected runtime yields a larger estimated gain than displaying elapsed time alone. This suggests a scaffold component worth testing, not a general orchestration solution. Linking repair patterns to orchestration remains future work.

Replay shortcuts. Verification regenerates input from a held-out seed different from the agent's observed input. Replaying observed input–output pairs therefore cannot reliably pass the correctness check.

What the agent repairs. For A01 sort:

VersionImplementationRuntime
PathologicalBubble sort, quadratic complexity74.0 s
Healthysorted(), linearithmic complexity0.09 s
Agent-modifiedBottom-up merge sort, linearithmic complexity0.13 s

The original bubble sort has quadratic cost. The healthy version uses sorted(), while the agent uses bottom-up merge sort. Both replacements address the bottleneck with O(n log n) worst-case complexity. The implementations differ, but the speedup comes from fixing the same underlying issue. This ties the runtime improvement to the code change.

Thank you again for your thoughtful review. When you have a chance, we’d welcome your thoughts on the three-way code and runtime comparison in our response. If it leaves any part of your concern about what successful patches actually optimize unanswered, we’d be happy to clarify.

Reviewer_LGtaOpenReview ↗

Sorry for the late reply

Thank you for the detailed reply. Sorry for the late reply. Forgive me I am a little bit busy...

To be short, I have been convinced with all tests results and I believe these stats is worthy to tell the role of the harness and the short-cuts problem is considered. This raise my confidence in your paper. Please revise the paper and mention it in the appendix. This helps the community to raise the confidence in your paper.

However, I still want to see a full success case, maybe just A01, to see the relevant code diff, and the key commands and observations before and after the edit. You may also consider, if possible, what performance information did the harness expose before the agent changed the code, and which correctness or timing checks did the agent itself run before submission? Release a full life-turn of a code optimization process helps everyone to know your benchmark's working logics and considering the harness's role helps the programmer and the agent developer gets the advice in current era of agents.

The reply is more convincing and answer many of my questions. I want to raise the score. Sorry for the late reply and bringing you very limited revision times. However, although the implicit-explicit gap is interesting, and your reply has raised my confidence in the result, I am still unsure how much this gap tells us, and how the "crawling" itself tells us about when a coding agent should take the initiative to optimize code in everyday engineering work. This is the main reason I am only raising my score to weak rejection.

Reviewer_kqdpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether coding agents will fix code that works but runs unusually slowly when no one has asked them to optimize it. The authors introduce CrawlBench, which pairs two versions of each task: one explicitly requests a performance improvement, and the other does not. Each submission is checked against a healthy reference on the same input for both correctness and runtime. The benchmark contains 192 cases. The main study uses 88 Python cases with one performance bug each; extensions cover different degrees of slowdown, two-bug cases, performance fixes from real pull requests, and other programming languages. The authors tested ten models with three random seeds each. When asked to optimize, the models fixed 93.6–97.7% of cases. Without that request, fix rates ranged from 23.5% to 90.2%. The most common failure in the latter setting was to leave the program working correctly but still running slowly. The authors take this as evidence that an agent’s ability to fix slow code does not always translate into taking the initiative to do so.

Critical strengths
  1. The paper asks whether a model will optimize a program without being explicitly asked, adding a useful perspective to benchmarks that focus on code correctness. Testing the same task with and without a performance hint offers a direct way to examine this behavior.
  2. The paper is easy to follow and addresses a worthwhile research question.
Critical weaknesses
  1. The explicit prompt does more than say that the program is slow: it also gives an expected runtime, says how many bugs there are, and clearly asks for a fix. The implicit prompt only asks the agent to run the program and report the output. So the difference in fix rates may come from the performance goal, the runtime hint, the bug count, or how much guidance the agent gets, rather than only from whether it decides to optimize on its own.
  2. The authors need to justify why proactive performance optimization should be considered a necessary implicit user requirement when the stated task only asks the agent to run the program and report its output. Without this justification, treating the absence of optimization as a failure risks penalizing agents for reasonably following the task scope.
Questions for authors
  1. Could you compare the results while changing one prompt detail at a time: saying the program is slow, giving the healthy runtime, stating the number of bugs, and showing the actual runtime? Does the gap between the explicit and implicit settings remain after controlling for these details one by one?
  2. An incorrect submission does not by itself show that an attempted performance fix broke the program. DeepSeek V4 Flash produced 96 incorrect submissions (Table A2), but only 2 of its 202 unfixed runs modified performance-related files (Table A8). Could the authors explain what caused these errors and provide evidence that they resulted from performance fixes, rather than unrelated edits or execution failures?
3 comments
AuthorsOpenReview ↗

Evaluation goal, task scope and edit-statistics correction

Thank you for your questions.

W1/Q1. Evaluation goal. CrawlBench asks how readily agents initiate optimization without an explicit performance request, compared with what they repair when explicitly guided. An agent can optimize reliably when asked yet rarely do so on its own. This behavioral contrast is our target; identifying which prompt sentence drives it is an additional diagnostic. We use "motivation" to mean the observed tendency to initiate optimization, not an internal mental state. Both settings share the system instruction "You may fix any issue you encounter."

W2. Why performance matters without a request. Users can overlook performance during vibe coding while still wanting performant code. In offline interviews, 93 of 96 programmers (96.9%) agreed with this view. Omitting performance from a prompt can reflect an overlooked concern rather than indifference. This motivates studying agents that take initiative without requiring users to identify each issue.

These interviews concern stated views, not behavior on our tasks. Editing is permitted but not obligatory: reporting abnormal slowness without editing is also reasonable. The interviews motivate studying self-initiated optimization; they do not establish that every implicit task requires a code change.

Additional prompt controls. All runs use mini-SWE-agent, 88 cases, a 1200 s budget and held-out verification. I denotes the implicit prompt; S states slowness, F requests repair, B gives an expected runtime, and N states the bug count. Unmarked rows use three seeds (264 episodes per model); † rows use seed 1 (88 episodes).

ConditionDS V4 FlashQwen3.8 27BOpus 5GPT 5.6 Sol
C0 Implicit †23.943.242.075.0
C0 Implicit, three seeds (Table A1)23.542.442.075.4
C1 I + permission to edit (no mention of speed)24.243.643.276.1
C2 I + S (slowness only)62.583.086.796.2
C3 I + B (runtime budget only)37.963.662.986.4
C4 I + S + F (no budget, no count)94.793.296.695.5
C5 Explicit − N †94.393.296.696.6
C6 Explicit − B †94.393.297.797.7
C7 I + measured runtime in every observation †26.151.146.677.3
X Explicit †95.594.397.797.7

C1–C4 are paired with the three-seed implicit baseline; C5–C7 with the seed-1 re-runs. Endpoint re-runs differ from the original seed-1 results by at most two cases per model.

Pooled gains are +0.9 pp for edit permission, +36.3 for S, +16.9 for B, +49.1 for S+F, and +4.3 for shown runtime. Case-cluster bootstrap 95% intervals (88 clusters, 10,000 resamples) are respectively [−3.2, 5.2], [32.6, 40.0], [12.8, 20.8], [45.9, 52.5] and [0.6, 8.0]. Removing N or B from explicit changes the pooled seed-1 rate by −1.1 and −0.6 pp. N is tested by removal because mentioning a "performance bug" already supplies a performance cue. These controls characterize responsiveness to guidance, not an internal noticing stage.

Q2. Discrepancy between Tables A2 and A8. Table A8 contains incorrect statistics due to a bug in our analysis code. This was our error. Table A2 remains authoritative, including DeepSeek V4 Flash's 96 incorrect submissions. We will remove Appendix A2 and Table A8.

Thank you for the detailed response. I agree with the authors' perspective on agent proactivity. However, I am still unclear about how to define good proactivity. How can we determine whether proactively taking action is genuinely what the user needs, rather than an unnecessary intervention? How is this aspect captured and evaluated in the task definition of CrawlBench? Could the authors elaborate a little further on this point?

Thank you for the follow-up. Good proactivity means addressing a material problem within the user’s authorized scope and leaving working code alone. CrawlBench tests this with the same run-and-report request on two functionally equivalent versions: one pathologically slow and one already fast. We count verified repairs on the slow version and track edits to already-fast code as a proxy for unnecessary intervention. Flagging a slowdown can also serve the user well, though our repair metric credits only verified fixes.

Reviewer_n4VMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CrawlBench to test whether AI coding agents proactively detect and fix performance bottlenecks without being asked. The authors find that while agents reliably optimize slow code when explicitly instructed (fixing 93.6–97.7% of main cases), they rarely fix performance issues on their own in implicit settings (fixing as low as 23.5%).

Critical strengths

S1. Unlike prior efficiency benchmarks (SWE-fficiency, GSO, SWE-Perf), this paper evaluates whether agents fix slowness without being asked. Comparing explicit versus implicit prompts directly isolates the exact impact of the instruction hint

S2, Fixes require strict output correctness, tight runtime bounds, and newly generated test inputs from hidden seeds.

S3. The evaluation reveals some interesting insights, e.g., that agents fail not because they attempt optimizations poorly, but because they ignore performance entirely.

Critical weaknesses

W1. The gap between the explicit and implicit settings has another possible cause. In the implicit setting the agent is only asked to run the tool and report the output. It is never asked to change the code, so leaving the code unchanged is a reasonable response. The explicit prompt adds much more than "the program is slow": it also says "find the cause and fix it", gives the number of bugs, and gives a runtime budget. So the gap may come from these extra instructions, and not from the agent failing to notice the slowness. The paper admits that it cannot separate three steps (noticing the slowness, judging that it matters, and deciding to act), and the evidence for that split is weak (two models, one run each, and nine inspected runs). Two extra settings might be considered needed: one that allows the agent to edit code without mentioning slowness, and one that gives only the runtime budget.

W2. A key safety check is missing for most models. The check (the "healthy-twin control") gives the agent the already-fast version of the program and tests whether the agent leaves good code alone. The paper runs it for only two models (GLM 5.3 Flash and Qwen3.8 27B, one run each). The models with the highest implicit scores (Gemini 3.7 Flash at 90.2% and GPT 5.6) are not tested. A model that edits everything it sees would also get a high implicit score. So we cannot tell "good at noticing slow code" from "edits code more often".

W3. The headline numbers depend on how slow the authors made the cases, not only on the models. In the paper's own test, the same bug is made slower by using a bigger input. The implicit fix rate then rises from 20% (about 15 seconds) to 80% (about 110 seconds). Many main cases run only 7-30 seconds, where agents mostly do not act. So the range of 23.5-90.2% partly reflects how the authors chose the slowdowns: with slower cases the gap would be smaller, and with faster cases it would be larger. The agent also chooses its own time limit for each command, and hitting that limit can reveal the slowness. The paper does not analyze this.

W4. The way "slow" is defined and measured is weak. Runtime is measured on one noisy machine. The paper uses a ratio E (the pristine runtime divided by the agent's program runtime) with three different cut-offs: 0.1 to call a case pathologically slow, 0.2 for "fast enough" in the checker, and 0.5 for the real-fix cases. The paper neither explains nor tests this choice. It gives no measure of how much E varies between runs, and it does not repeat the measurements on other hardware. The statement "no annotator assigns the label" only means that the authors set the label when they built each case. Nobody outside checked that these are slowdowns a developer would actually want fixed.

W5. The paper does not show that the reference program is a good one. Both correctness and speed are judged against the pristine program. The paper only says that it is "not a proven lower bound" on runtime. It does not say how the pristine versions were chosen, profiled, or compared with other implementations. The appendix itself shows two agent edits that run faster than the pristine code, so the reference is not the best possible. An agent that beats the reference gets no extra credit. For the real-fix cases, the reference is simply the fix that was merged. The paper also does not say whether outputs must match exactly or within a tolerance, although it notes that some fast fixes silently change numeric results.

W6. The contribution over existing efficiency benchmarks is narrower than claimed. SWE-fficiency (498 real tasks) and GSO (102 tasks) also do not tell the agent where the slow code is, which contradicts the related-work section. On those benchmarks agents reach only under 0.23 times the expert's speedup (SWE-fficiency) and succeed on under 5% of tasks (GSO). So CrawlBench's explicit rate of 94-98% reflects easy, textbook planted bugs and a prompt that reveals the number of bugs, and the claim that "repair is mostly engineering" does not carry over to real repositories. Those papers also report agents stopping early, doing lazy optimization, and failing to locate the slow code. These findings are close to the "miss" category here, but the paper does not discuss them, and PerfBench is not cited. The new element is the explicit/implicit comparison, and it is applied only to small synthetic programs.

Questions for authors

Please have a look at W1-W6, specifically, the following questions Can you run the healthy-twin control (agents on the already-fast version) for all ten models? What happens under an implicit setting that allows edits without mentioning slowness, and under one that gives only the runtime budget? Can you group the results by how slow the program is, and report the fix rate when the agent's own time limit was hit? Why are three different thresholds of E used (0.1, 0.2, 0.5), and how were they chosen? How sensitive are the results to these thresholds, and how much does E vary from run to run? How did you ensure that each reference program is good or the best available? How many agent submissions ran faster than the reference, and how are outputs compared (exactly or with a tolerance)? Can you run an implicit setting on a subset of SWE-fficiency or GSO tasks? Is there a human or expert-developer baseline on the implicit prompts?

5 comments
AuthorsOpenReview ↗

Self-initiated optimization and healthy-twin control

Thank you for your questions. New runs use mini-SWE-agent, 88 cases, a 1200 s budget and held-out verification.

W1. Noticing versus deciding not to act. Our question is whether agents initiate optimization without a performance request, compared with repair when explicitly guided. Attributing non-repair to an internal stage is not essential to this evaluation. Users can overlook performance while still valuing it: 93 of 96 programmers (96.9%) endorsed this view in offline interviews. If an agent neither addresses nor reports a significant slowdown, the concern remains unresolved whichever stage led to that outcome.

Both settings already permit edits through the shared system instruction. We ran your two requested controls on four models over three seeds: repeating edit permission gives +0.9 pp [−3.2, 5.2], and giving only an expected runtime gives +16.9 pp [12.8, 20.8], relative to implicit. Intervals are case-cluster bootstrap 95% intervals (88 cases, 10,000 resamples). These diagnose dependence on guidance rather than a specific internal stage.

Statistics correction. Table A8 contains incorrect edit statistics due to a bug in our analysis code. Table A2 remains authoritative. We will remove Appendix A2 and Table A8.

W2. Healthy-twin control on all ten models. The original control covered two models. The full ten-model results are below, using the same implicit task and budget on already-fast workspaces. Edit and damage columns are seed-1 counts out of 88; implicit fix rate uses three seeds on the slow twins. FPR is healthy hot-path edits / 88, and TPR − FPR is a descriptive index in percentage points with different seed coverage.

ModelImplicit fix rate on slow twins (%)Any source editHot-path editDamageTPR − FPR
Claude Opus 542.096135.2
DeepSeek V4 Flash23.564119.0
DeepSeek V4 Pro34.575128.8
Gemini 3.7 Flash90.285184.5
GLM 5.242.4107134.4
GLM 5.3 Flash75.4128266.3
GPT 5.6 Luna72.797164.7
GPT 5.6 Sol75.464170.9
Kimi K354.5118145.4
Qwen3.8 27B42.451141.3

Eight seed-1 rows are new; the original GLM 5.3 Flash and Qwen3.8 27B rows were rechecked against their logs. Gemini, Sol, GLM 5.3 Flash and Luna also have healthy-control seeds 2–3; hot-path-edit rate sample SDs are 1.1, 0.7, 2.3 and 2.0 pp.

Healthy hot-path edit rates are 1.1–9.1%, versus 23.5–90.2% repair rates on slow code; damage occurs in 1–2 of 88 episodes per model. This supports selective repair. Edit rate and implicit fix rate have Spearman ρ = 0.27; ranking by TPR − FPR versus implicit fix rate has Kendall τ-b = 0.93. All 55 healthy hot-path edits preserve correctness and speed; the 11 damaged workspaces involve edits outside those hot paths.

AuthorsOpenReview ↗

Severity, timing, references and scope

W3. Severity and timeouts. Absolute fix rates depend on case severity. Grouping by construction-time pathological runtime gives:

Pathological runtimeCasesImplicit (%)Explicit (%)Gap (pp)Kendall τ-b of model ranking vs. overall
7 ≤ T < 20 s2541.395.253.90.93
20 ≤ T < 60 s1655.095.640.60.93
≥ 60 s4762.896.033.10.99

Slowdown-ratio groups of 16–100×, 100–1000× and above 1000× give implicit rates of 50.0%, 57.1% and 63.5%, with ranking τ-b of 0.96, 0.90 and 0.87. Severity changes absolute rates, while a gap remains across these groups.

Agent-chosen timeouts are part of the behavior being evaluated. The scaffold requires bounded commands, but the agent chooses how long to wait and whether a timeout warrants diagnosis and repair. A timeout supplies execution feedback, not a user request to optimize.

Timeouts fired in 402/2,640 implicit episodes: 331/402 fixed (82.3%), versus 1,129/2,238 (50.4%) otherwise. Within the ≥60 s group, rates are 292/341 (85.6%) and 594/1,069 (55.6%). These are associations, not causal estimates.

W4. Threshold choices. E = T_healthy / T_candidate. The thresholds serve different purposes:

PurposeCriterionMeaning
Synthetic admissionE < 0.1More than 10× slower than healthy
Synthetic repair successE ≥ 0.2Within 5× of healthy
Real-PR repair successE ≥ 0.5Within 2× of the maintainer's fix

The synthetic success bar allows constant-factor differences between valid implementations. Real regressions can be only a few times slower, so 0.2 could accept unchanged code. These are design choices, not tuned cut-offs.

Synthetic fast barImplicit (%)Explicit (%)Gap (pp)
0.158.097.039.0
0.255.395.740.4
0.3352.794.742.0
0.549.493.343.9

No model pair reverses order; some ties form or break. On real-PR cases, bars 0.33/0.5/0.67 give implicit rates 46.7/41.0/36.0% and explicit rates 95.8/92.5/89.2%. Removing the five least-severe synthetic cases gives 55.6% implicit and a 40.0 pp gap. Restricting to ≥50× and ≥100× slowdowns gives gaps of 38.8 and 37.2 pp.

Timing reliability. We interleave 10 healthy/buggy repetitions per case and run each correct candidate five times. Median CV of E is 2.6%; 31/2,640 verdicts flip (1.2%), including 27 near the 0.2 boundary. Four cases have CV above 9%; the other 84 are at or below 4.1%.

On M3 versus M2, all 88 admission labels and 2,613/2,640 verdicts (99.0%) reproduce; ranking Spearman ρ is 0.98. Both machines are Apple Silicon, so hardware coverage is limited. Runtime ratios are not developer judgments; for real-PR cases, merged performance fixes show that maintainers considered the regressions worth addressing.

W5. Reference quality. Synthetic references are pinned library implementations before injection, or canonical implementations for hand-written cases. Real-PR references apply the maintainer's diff to the PR merge-base. We do not claim runtime lower bounds.

Of 1,460 fixed implicit submissions, 138 (9.5%) beat the reference; maximum E is 1.42. Using the fastest correct implementation observed per case changes any model by at most 2.3 pp. Pooled implicit changes 55.3→54.1% and explicit 95.7→94.9%, without reversing a strict model ordering.

Output comparison. We compare normalized stdout fields exactly: strip trailing whitespace, apply declared field order and normalize numeric formatting. Printing uses six significant digits. Relative tolerance 1e-5 makes 14 implicit submissions correct, all fast-wrong cases differing only in the last printed digit. Pooled implicit moves 55.3→55.8% and explicit 95.7→96.0%; no pair reverses.

W6. Contribution. SWE-fficiency and GSO also require bottleneck localization; we correct our earlier description. CrawlBench adds paired tasks with and without a performance objective. Our 93.6–97.7% explicit success concerns planted bugs, not general repository-scale optimization. We recognize prior findings on early stopping, lazy optimization and localization failures, and include PerfBench (81 .NET performance bug-fix tasks).

We have not run implicit variants of SWE-fficiency or GSO. Our 20 real performance-fix PR cases give 41.0% implicit versus 92.5% explicit, a 51.5 pp gap; transfer to those other benchmarks remains untested.

Human baseline. Not run on our tasks. The interviews reported above concern developer preferences, not task-performance measurements.

Thank you again for your detailed feedback. When you have a chance, we’d welcome your thoughts on the expanded healthy-twin control and the runtime analyses in our responses. If any of your questions remain open, we’d be happy to clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.