- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that repository-level performance benchmarks (SWE-fficiency, GSO) use the same small set of workloads both as optimization feedback and as the final evaluation. As a result, coding agents can earn inflated speedups by specializing to incidental properties of the visible workload, for example caching on object identity. It makes two contributions:
- WorkloadPro. This is a benchmark-augmentation framework that iteratively generates held-out workloads of two kinds. Robustness workloads mutate the values, shapes, and types of inputs. Diverse workloads change the workload structure and must cover new lines of code. A candidate is kept only if the human patch improves it by at least 10% and its runtime is at least 10 ms. The resulting suite has 3,415 workloads for 85 SWE-fficiency Lite tasks and 1,612 workloads for 71 GSO tasks.
- APO. This agentic optimizer first generates five guardrail workloads from the original workload, a cProfile report, and the relevant source code. A planner then proposes up to three distinct optimization targets. Each target is explored in an isolated branch, and an integration agent then picks the strongest candidate and refines it in a longer depth pass.
Critical strengths
The workload-generation design is careful, and the method has real-world validation. The paper explicitly recognizes that workloads can overfit to the ground-truth patch just as patches overfit to workloads. It responds by generating first and verifying relevance with the human patch only afterwards, and by requiring new coverage for diverse workloads.
Critical weaknesses
- The paper is not positioned against closely related prior work, and the comparison baselines are therefore incomplete.
- FormulaCode[1] already provides repository-level optimization tasks with on average about 265 community-maintained workloads per task, and studies how agents balance multiple workloads. This directly challenges the premise that existing benchmarks “typically provide only one or a few” workloads.
- The SWE-fficiency[2] paper itself reported workload overfitting as a failure mode. It also found that manually annotated workloads outperform LLM-generated ones. Since WorkloadPro’s validity rests on LLM-generated workloads, the paper should address this finding directly.
Without these comparisons, it is hard to judge how much of APO’s gain comes from the guardrails and breadth-then-depth search, and how much comes from profiling plus a stronger scaffold, which prior work already offers.
- Several aspects of the evaluation could bias the reported numbers.
- WorkloadPro is defined relative to the human patch. Every retained workload must be improved at least 10% by the human patch. The “generated” metric therefore rewards patches that resemble the human patch rather than measuring generalization in general. A correct optimization that targets a different bottleneck could be penalized. The paper should acknowledge this and ideally quantify it.
- The paper does not state which model generates the WorkloadPro workloads. If it is GPT-5.5, the same model as all APO subagents, there is a risk of circularity in favor of APO.
[1] FormulaCode: Evaluating Agentic Optimization on Large Codebases, ICML 2026
[2] SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?, ICML 2026
Questions for authors
-
Since retained workloads must be improved by the human patch, how often does a correct patch that targets a different but legitimate bottleneck score poorly on WorkloadPro? Could you report the generated-workload score of agent patches that a manual check finds correct but structurally different from the human patch?
-
How do you reconcile your use of LLM-generated workloads with SWE-fficiency’s finding that manually annotated workloads are better? How does WorkloadPro relate to FormulaCode’s community-maintained multi-workload setting?
-
Which model(s) generated the WorkloadPro workloads? If it was GPT-5.5, can you report results with workloads generated by a different model family?
-
Section 5.2 (line 376) states that 9/85 patches from OpenHands with GPT-5.5/Opus 4.7 regress, but Table 1 lists 7/85 for GPT-5.5. Which is correct?
5 comments
Response to Reviewer iWor (2/5)
W1&Q2 (2): The paper is not positioned against closely related prior work, and the comparison baselines are therefore incomplete.
SWE-fficiency's finding that manually annotated workloads are better. SWE-fficiency generated one workload per task by directly prompting Gemini 2.5 Flash once, using the human patch and related files, and did not filter the results. On 47% of these workloads, the human patch had no significant speedup. That is the main reason their manual workloads did better. WorkloadPro differs from that setup in multiple ways:
- We filter generated workloads. We keep a generated workload only if the human patch speeds it up by at least 10% and its baseline runtime is at least 10 ms. On the 30-task subset, Claude Opus 4.7 generated 900 workloads, and these filters removed 295 of them (32.8%). A workload on which the human patch shows no speedup does not make it into our benchmark. On average, the human patch improves performance by 51.5% on the kept workloads and by 59.5% on the manual original workloads.
- WorkloadPro is complementary to manual workloads. (we keep the manual workloads) Every task still includes the original manual workload, and we still report the improvement on it (Impr. Original). The generated workloads are extra inputs. Instead of claiming our generated workloads are better than human-written workloads, we primarily argue that WorkloadPro workloads are complementary to human-written workloads as it provides a more comprehensive evaluation in terms of how general the optimization is and could identify suspicious harmful specialization patches.
- We generate workloads interactively and iteratively. Instead of prompting a language model to generate workloads, WorkloadPro launches a specialized coding agent in a sandbox docker environment to generate workloads. The agent can explore the codebase, read the PR descriptions and the human patch. It also profiles the execution to understand the performance issue. In addition, instead of querying the language model once, WorkloadPro iteratively generates better workloads on top of the workloads from the last iteration incrementally, aiming to cover different patterns of inputs and cover new lines, improving the test adequacy of the workloads.
- WorkloadPro is fundamentally more scalable. WorkloadPro does not depend on either expensive human annotation or historical workload corpus. It turns the discovery of workloads into an automated search problem. We admit that it will also generate some less effective workloads, but our filter mechanism will only keep the workloads that exercise the bottleneck the human patch tackles. In contrast, SWE-fficiency completely resorts to rely on a single workload, which we have shown to be the typical causes of encouraging overoptimizing.
Response to Reviewer iWor (3/5)
W1&Q2 (3): The paper is not positioned against closely related prior work, and the comparison baselines are therefore incomplete.
The SWE-fficiency paper itself reported workload overfitting as a failure mode. SWE-fficiency does have some preliminary efforts of detecting and mitigating workload overfitting, and we cite it as motivation. However, SWE-fficiency's "prevent model reward hacking" mechanism only targets two specific patterns: stack introspection to detect the timing context and caching across timing repetitions. For the stack introspection, they employ rule-based methods such as AST analysis to statically parse the patch, and check whether the patch contains API calls in the blacklist of stack introspection. The labeled patches are considered failing. For mitigating caching across timing repetitions, they employ process isolation via rewriting the workloads using AST. However, in practice, a lot of reward hacking behaviors are beyond the scope of the two patterns.
For example, we inspected the 17 patches flagged by WorkloadPro and found they come from different patterns, e.g., one patch rebuilds a cache in the forked timing process via register_after_fork, which bypasses the process isolation directly; other patches memoize repeated calls, exit early based on the benchmark’s data order, or tune constants to the benchmark’s input size, and none of them can be captured by SWE-fficiency's mechanisms. They are specializing based on the single visible workload and do not require stack introspection at all. However, they all rely on some aggressive assumptions based on the original workload. When the new workloads break such assumptions, the performance regressions indicate risks of workload overfitting. In other words, diversified workload evaluation is more effective and comprehensive in terms of detecting workload overfitting.
We will add the detailed discussion of FormulaCode and comparison of manual vs language model/agent-generated workloads, as well as SWE-fficiency's overfitting detection efforts, in the revised version of the paper.
Contribution of profiling in APO. We gave OpenHands and Claude Code the same profiling information as APO and a $3 budget. Results are averaged over 5 runs, k = 1, T = 5%:
| Method | Rect. Generated | Impr. Original | Opt@1 | Overfit |
|---|---|---|---|---|
| APO (GPT-5.5), single $3 run | 45.4 ± 1.1 | 63.7 ± 0.2 | 79.3 ± 0.0 | 1.8 ± 0.8 / 29 |
| OpenHands (GPT-5.5) + profile, single $3 run | 37.8 ± 0.9 | 56.3 ± 0.0 | 62.1 ± 0.0 | 2.8 ± 0.4 / 29 |
| Claude Code (Opus 4.7) + profile, single $3 run | 38.2 ± 0.3 | 52.3 ± 0.2 | 61.4 ± 1.5 | 2.6 ± 0.5 / 29 |
| OpenHands (Opus 4.7) + profile, single $3 run | 28.7 ± 0.6 | 38.2 ± 0.1 | 42.8 ± 3.1 | 1.6 ± 0.5 / 29 |
| OpenHands (GPT-5.5) + profile, best of 3 × $1 | 43.7 ± 1.0 | 68.5 ± 0.3 | 74.7 ± 2.0 | 2.3 ± 0.6 / 29 |
| OpenHands (GPT-5.5) + profile, single $1 attempt (avg) | 26.1 ± 0.2 | 42.7 ± 0.1 | 49.8 ± 1.3 | 2.7 ± 0.0 / 29 |
| Claude Code (Opus 4.7) + profile, best of 3 × $1 | 31.2 ± 1.3 | 43.9 ± 0.2 | 51.7 ± 0.0 | 2.0 ± 1.0 / 29 |
| Claude Code (Opus 4.7) + profile, single $1 attempt (avg) | 21.0 ± 0.8 | 25.4 ± 0.2 | 34.5 ± 0.0 | 2.1 ± 0.4 / 29 |
| OpenHands (Opus 4.7) + profile, best of 3 × $1 | 13.0 ± 2.2 | 7.4 ± 0.3 | 17.2 ± 0.0 | 2.0 ± 1.0 / 29 |
| OpenHands (Opus 4.7) + profile, single $1 attempt (avg) | 10.7 ± 0.9 | -3.4 ± 0.4 | 13.0 ± 0.7 | 2.3 ± 0.3 / 29 |
With the same model, APO outperforms the profiling-guided single agent by 7.6 pp on held-out workloads and by 17 pp on Opt@1. The table shows that adding profiling alone does not necessarily improve the performance of baseline agents, i.e., it does not fully contribute to the improvement that APO achieves. Without profiling, they reach 41.6 (OpenHands, GPT-5.5) and 43.5 (Claude Code) on held-out workloads, still below APO (45.4).
Response to Reviewer iWor (1/5)
We are grateful for your time, effort, and leaving insightful comments. We address your comments and questions in the following.
W1&Q2 (1): The paper is not positioned against closely related prior work, and the comparison baselines are therefore incomplete.
Thanks for bringing up the related works. We believe WorkloadPro is complementary to existing works and will explicitly discuss them in the revised version.
Comparison with FormulaCode. A large number of workloads per task does not mean that sufficient workloads test the specific performance issue of the task, specifically:
Therefore, WorkloadPro is still complementary to benchmarks like FormulaCode where targeted workloads are rare.