- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces CrawlBench, a benchmark for evaluating proactive efficiency awareness: whether coding agents initiate performance optimization when it is not an explicit task objective. Each case pairs a pathologically slow program with a functionally equivalent healthy twin. The same task is evaluated under two framings. Under explicit framing, the agent is told that the program is slow and asked to fix it. Under implicit framing, the agent is asked to run the program and report its output, without any mention of performance. A submission counts as fixed only if it is both correct and fast relative to the healthy twin. The main quantity of interest is the explicit-implicit gap in fix rates. The authors evaluate ten models using mini-SWE-agent. The main benchmark contains 88 single-bug Python cases covering asymptotic complexity, execution-model constants, external I/O latency, and declarative-engine cost. Additional experiments examine slowdown severity, two-bug stacks, regressions mined from real performance-fix PRs, and C and JavaScript replicas, bringing the reported pool to 192 cases. On the main benchmark, explicit fix rates range from 93.6% to 97.7%, while implicit fix rates range from 23.5% to 90.2%. The largest failure category for every model is miss: submissions that remain correct but slow. These results suggest that strong performance under explicit optimization requests does not consistently translate into proactive optimization.
Critical strengths
The main strength of this paper is its focus on the implicit setting. Existing work such as SWE-fficiency already studies performance optimization on real workloads, including bottleneck localization and the changes needed to improve runtime while preserving correctness. CrawlBench adds a useful question: when users do not explicitly ask for faster code, will an agent still recognize a serious performance problem and decide to address it? I find this distinction meaningful. An agent may know how to optimize a program once asked, yet consider its work complete as soon as the program produces the correct output. The high explicit fix rates, together with the frequent correct-but-slow outcomes under implicit framing, provide evidence of this gap. They suggest that, for many of the evaluated models under this scaffold, the runtime efficiency of the final program is not yet a consistently considered part of completing a task. The timing-visibility control provides some additional support for this observation. For the two models tested, showing the runtime of every command improves implicit fix rates by only 2 and 8 percentage. Although limited in scope, this result suggests that making timing information available may not be sufficient to trigger optimization. The paper therefore offers a useful perspective for agent evaluation and design: agents should be assessed both on their ability to fix performance problems when asked and on whether they appropriately initiate that work without an explicit request.
Critical weaknesses
I appreciate the authors’ effort to study whether coding agents self-initiate performance optimization when speed is not an explicit task objective. However, I remain unconvinced that the evidence presented establishes the broader value of the optimization behavior being measured. My concerns are substantive, and I encourage the authors to address each of them directly and in detail.
First, the paper has not sufficiently explained to what extent faster execution on the benchmark reflects meaningful proactive optimization capability. The explicit framing asks the agent to improve performance, while the implicit framing examines whether it initiates optimization on its own. In both settings, however, success is ultimately determined by whether the final submission produces matching output and meets the runtime criterion. These results show whether an agent meets the benchmark's performance requirement under different prompts, but do not sufficiently establish whether it identifies the right problem, chooses an appropriate modification, or makes an improvement of practical value. Real performance optimization involves more than obtaining a shorter runtime: it requires understanding why a program is slow, where it should be changed, and whether the change is worthwhile. The paper covers asymptotic complexity, execution-model constants, external I/O latency, and declarative-engine cost, demonstrating coverage of different sources of slowness. Yet covering different sources of slowness is not the same as distinguishing between optimizations of different quality. Likewise, leaving code unchanged does not by itself establish a lack of proactive efficiency awareness, and modifying code and meeting the runtime criterion does not by itself establish sound engineering judgment. The failure-mode triage and healthy-twin control provide relevant evidence, but do not resolve this distinction. The explicit-implicit gap is therefore a useful behavioral observation, but the paper still needs to explain which optimization capabilities it reveals and why those capabilities are meaningful measures of progress in coding agents.
More fundamentally, I question an approach to optimizing and evaluating agents that gives runtime such pure and overwhelming priority. SWE-fficiency already treats runtime improvement as a core objective in repository-level performance optimization. In the broader setting of today's LLM coding agents, I consider the more important question to be which aspects of a complex engineering architecture shape agent capability and, in turn, the runtime of the optimized code. A more fundamental direction is to identify recurring optimization patterns and use them to inform agent orchestration frameworks. This paper does not pursue that deeper connection between optimization behavior and agent orchestration. Instead, its central evaluation criterion remains whether the final submission meets the runtime requirement while preserving correctness. In my view, this framing is no longer adequate for the current stage of LLM coding-agent development.
Second, the paper needs a direct comparison of the healthy twin, the agent-modified version, and the original pathological version for the same task, with the performance changes tied to the actual code changes. The authors already provide the healthy and pathological versions and report their runtimes. The verifier also measures the agent's final submission against the pristine reference. What is still missing is a systematic analysis that brings the final agent-modified code into this comparison and explains exactly what optimization it achieves. For the same task, I need to see how long each of the three versions takes to run, why the original version is slow, how the healthy version addresses that problem, what the agent changes, and why those changes produce the reported speedup. If the agent adopts a different approach from the healthy twin, the authors should also explain how the approaches differ and how those differences affect performance. Only by examining the runtimes of all three versions alongside their code differences can we judge what problem the agent actually solves, rather than merely whether it passes the tests.
The current summaries of fix rates do not answer my concern about “shortcut-taking” by LLM coding agents. With runtime as the dominant optimization target, the incentives created by this evaluation criterion could encourage improvements that exist primarily at the level of the test rather than in the underlying program. The same label of “success” can conceal very different modifications and performance gains. For example, suppose we expect an agent to identify a quadratic bubble-sort routine and replace it with an efficient sorting algorithm. A patch that instead replays stored input and output pairs for repeated benchmark inputs could appear faster without addressing the sorting bottleneck. If accepted by the evaluation, this would be a test-specific shortcut rather than the improvement we intended to measure. The held-out-seed check is intended to prevent simple reuse of observed outputs, but it does not by itself establish what the successful patches actually optimize. The paper still needs to show whether those submissions make substantive improvements and how the evaluation distinguishes them from benchmark-specific shortcuts. The analysis of unfixed implicit episodes does not answer this question about successful submissions. Providing the three versions' runtimes and code differences would make it possible to examine which underlying optimization issues the agent and the healthy implementation address in common, where their approaches diverge, and what those differences mean for the value of the optimization.
Questions for authors
How do the authors position runtime improvement in evaluating coding agents, and what analysis of agent components and code changes establishes the benchmark’s value beyond measuring speedup?
5 comments
Thank you again for your thoughtful review. When you have a chance, we’d welcome your thoughts on the three-way code and runtime comparison in our response. If it leaves any part of your concern about what successful patches actually optimize unanswered, we’d be happy to clarify.
Sorry for the late reply
Thank you for the detailed reply. Sorry for the late reply. Forgive me I am a little bit busy...
To be short, I have been convinced with all tests results and I believe these stats is worthy to tell the role of the harness and the short-cuts problem is considered. This raise my confidence in your paper. Please revise the paper and mention it in the appendix. This helps the community to raise the confidence in your paper.
However, I still want to see a full success case, maybe just A01, to see the relevant code diff, and the key commands and observations before and after the edit. You may also consider, if possible, what performance information did the harness expose before the agent changed the code, and which correctness or timing checks did the agent itself run before submission? Release a full life-turn of a code optimization process helps everyone to know your benchmark's working logics and considering the harness's role helps the programmer and the agent developer gets the advice in current era of agents.
The reply is more convincing and answer many of my questions. I want to raise the score. Sorry for the late reply and bringing you very limited revision times. However, although the implicit-explicit gap is interesting, and your reply has raised my confidence in the result, I am still unsure how much this gap tells us, and how the "crawling" itself tells us about when a coding agent should take the initiative to optimize code in everyday engineering work. This is the main reason I am only raising my score to weak rejection.
Evaluation value, agent components and code comparison
Thank you for your questions.
What the score measures. CrawlBench measures whether agents initiate correctness-preserving optimization without an explicit performance request. Users may overlook performance while still valuing it. Large slowdowns (16–10730×) motivate studying this behavior. To count as a successful repair, a submission must meet the runtime criterion while preserving output. Success is binary within 5× of the healthy reference on synthetic cases, so progressively larger speedups receive no extra credit.
The target is the full behavior, not an internal noticing stage. Healthy-twin controls test selectivity: healthy hot-path edits occur in 1.1–9.1% of episodes, versus 23.5–90.2% repair rates on slow code.
Agent components. We test whether guidance triggers optimization. Estimates use 88 cases per model and paired implicit baselines; intervals are case-cluster bootstrap 95% intervals.
The first three use DeepSeek V4 Flash, Qwen3.8 27B, Claude Opus 5 and GPT 5.6 Sol. The self-check uses Qwen and Opus with "Before submitting, check whether your work fully accomplishes the task."
Expected runtime yields a larger estimated gain than displaying elapsed time alone. This suggests a scaffold component worth testing, not a general orchestration solution. Linking repair patterns to orchestration remains future work.
Replay shortcuts. Verification regenerates input from a held-out seed different from the agent's observed input. Replaying observed input–output pairs therefore cannot reliably pass the correctness check.
What the agent repairs. For A01 sort:
sorted(), linearithmic complexityThe original bubble sort has quadratic cost. The healthy version uses
sorted(), while the agent uses bottom-up merge sort. Both replacements address the bottleneck with O(n log n) worst-case complexity. The implementations differ, but the speedup comes from fixing the same underlying issue. This ties the runtime improvement to the code change.