acceptodds
Under review as a conference paper at ICLR 2027

Goodhart’s Law in Performance Benchmarks: Measuring and Mitigating Software Overoptimization

Abstract

Coding agents have shown growing promise for optimizing the performance of real-world software, motivating recent benchmarks that assess this capability through executable workloads. However, these benchmarks typically provide only one or a few visible workloads per task due to the substantial expert effort required to construct high-quality workloads. These workloads then act as both optimization feedback and the final evaluation. Agents can thus obtain large reported speedups by exploiting properties of the observed inputs rather than resolving the underlying performance bottleneck, producing patches that fail to generalize to other valid workloads. To systematically measure this workload overoptimization, we introduce WorkloadPro, an evaluation framework that expands existing benchmarks with diverse, held-out workloads while ensuring that each workload exercises the intended optimization. Measuring against these workloads shows that existing top-rankers substantially overstate performance improvements, and their rankings can reverse. To mitigate this problem, we introduce APO (Agentic Performance Optimization), a profile-guided agentic optimization framework that generates diverse workloads at test time and profiles them to explore diverse optimization targets against these workloads. Our preliminary evaluation using WorkloadPro shows that APO outperforms the top-ranking agents on SWE-fficiency, using the same budget and model, and produces fewer overoptimized patches. These findings show that workload diversity is important both for reliably evaluating coding agents and for guiding them toward performance optimizations that generalize.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
coding agents, performance optimization, reward hacking, benchmarking agents, test generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iWorOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that repository-level performance benchmarks (SWE-fficiency, GSO) use the same small set of workloads both as optimization feedback and as the final evaluation. As a result, coding agents can earn inflated speedups by specializing to incidental properties of the visible workload, for example caching on object identity. It makes two contributions:

  1. WorkloadPro. This is a benchmark-augmentation framework that iteratively generates held-out workloads of two kinds. Robustness workloads mutate the values, shapes, and types of inputs. Diverse workloads change the workload structure and must cover new lines of code. A candidate is kept only if the human patch improves it by at least 10% and its runtime is at least 10 ms. The resulting suite has 3,415 workloads for 85 SWE-fficiency Lite tasks and 1,612 workloads for 71 GSO tasks.
  2. APO. This agentic optimizer first generates five guardrail workloads from the original workload, a cProfile report, and the relevant source code. A planner then proposes up to three distinct optimization targets. Each target is explored in an isolated branch, and an integration agent then picks the strongest candidate and refines it in a longer depth pass.
Critical strengths

The workload-generation design is careful, and the method has real-world validation. The paper explicitly recognizes that workloads can overfit to the ground-truth patch just as patches overfit to workloads. It responds by generating first and verifying relevance with the human patch only afterwards, and by requiring new coverage for diverse workloads.

Critical weaknesses
  1. The paper is not positioned against closely related prior work, and the comparison baselines are therefore incomplete.
  • FormulaCode[1] already provides repository-level optimization tasks with on average about 265 community-maintained workloads per task, and studies how agents balance multiple workloads. This directly challenges the premise that existing benchmarks “typically provide only one or a few” workloads.
  • The SWE-fficiency[2] paper itself reported workload overfitting as a failure mode. It also found that manually annotated workloads outperform LLM-generated ones. Since WorkloadPro’s validity rests on LLM-generated workloads, the paper should address this finding directly.

Without these comparisons, it is hard to judge how much of APO’s gain comes from the guardrails and breadth-then-depth search, and how much comes from profiling plus a stronger scaffold, which prior work already offers.

  1. Several aspects of the evaluation could bias the reported numbers.
  • WorkloadPro is defined relative to the human patch. Every retained workload must be improved at least 10% by the human patch. The “generated” metric therefore rewards patches that resemble the human patch rather than measuring generalization in general. A correct optimization that targets a different bottleneck could be penalized. The paper should acknowledge this and ideally quantify it.
  • The paper does not state which model generates the WorkloadPro workloads. If it is GPT-5.5, the same model as all APO subagents, there is a risk of circularity in favor of APO.

[1] FormulaCode: Evaluating Agentic Optimization on Large Codebases, ICML 2026

[2] SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?, ICML 2026

Questions for authors
  1. Since retained workloads must be improved by the human patch, how often does a correct patch that targets a different but legitimate bottleneck score poorly on WorkloadPro? Could you report the generated-workload score of agent patches that a manual check finds correct but structurally different from the human patch?

  2. How do you reconcile your use of LLM-generated workloads with SWE-fficiency’s finding that manually annotated workloads are better? How does WorkloadPro relate to FormulaCode’s community-maintained multi-workload setting?

  3. Which model(s) generated the WorkloadPro workloads? If it was GPT-5.5, can you report results with workloads generated by a different model family?

  4. Section 5.2 (line 376) states that 9/85 patches from OpenHands with GPT-5.5/Opus 4.7 regress, but Table 1 lists 7/85 for GPT-5.5. Which is correct?

5 comments
AuthorsOpenReview ↗

Response to Reviewer iWor (1/5)

We are grateful for your time, effort, and leaving insightful comments. We address your comments and questions in the following.

W1&Q2 (1): The paper is not positioned against closely related prior work, and the comparison baselines are therefore incomplete.

Thanks for bringing up the related works. We believe WorkloadPro is complementary to existing works and will explicitly discuss them in the revised version.

Comparison with FormulaCode. A large number of workloads per task does not mean that sufficient workloads test the specific performance issue of the task, specifically:

  1. The workloads are general for the whole repository instead of targeting specific performance issues. We performed a coverage analysis on the 88 tasks whose human patch changes include Python code and found that the median fraction of workloads that execute the changed function is only 2.8% per task. In 15 tasks, no workload executes the changed functions. However, in evaluation, the performance score is averaged by all workloads. Consequently, the evaluation of workloads covering the performance issues could be affected by the noise from the other 97.2% irrelevant workloads.
  2. The distribution of abundance of workloads and covering workloads is very imbalanced across projects. We counted the number of available workloads and covering workloads per task and calculated the median number. The pandas project alone contributes most of the workloads. It turns out that the median number (available/covering) of pandas is 1146/25 while it is only 36/1 for all other projects.
  3. The historical ASV workloads have been public on the internet for years, so it remains unclear if there are risks of memorization or specialization. In contrast, WorkloadPro's held-out workloads are less prone to specialization or reward hacking.
  4. FormulaCode can only include projects that already maintain a large ASV suite historically maintained by a group of developers. On new projects where ASV suits are not available, using FormulaCode is not possible, and rebuilding such large scale ASV suites from scratch require substantial human efforts. In contrast, WorkloadPro is a fully automated pipeline that only requires the human patch and one proof-of-concept workload, so it also applies to benchmarks such as SWE-fficiency and GSO or any projects where the human patch and a workload exist.

Therefore, WorkloadPro is still complementary to benchmarks like FormulaCode where targeted workloads are rare.

AuthorsOpenReview ↗

Response to Reviewer iWor (2/5)

W1&Q2 (2): The paper is not positioned against closely related prior work, and the comparison baselines are therefore incomplete.

SWE-fficiency's finding that manually annotated workloads are better. SWE-fficiency generated one workload per task by directly prompting Gemini 2.5 Flash once, using the human patch and related files, and did not filter the results. On 47% of these workloads, the human patch had no significant speedup. That is the main reason their manual workloads did better. WorkloadPro differs from that setup in multiple ways:

  1. We filter generated workloads. We keep a generated workload only if the human patch speeds it up by at least 10% and its baseline runtime is at least 10 ms. On the 30-task subset, Claude Opus 4.7 generated 900 workloads, and these filters removed 295 of them (32.8%). A workload on which the human patch shows no speedup does not make it into our benchmark. On average, the human patch improves performance by 51.5% on the kept workloads and by 59.5% on the manual original workloads.
  2. WorkloadPro is complementary to manual workloads. (we keep the manual workloads) Every task still includes the original manual workload, and we still report the improvement on it (Impr. Original). The generated workloads are extra inputs. Instead of claiming our generated workloads are better than human-written workloads, we primarily argue that WorkloadPro workloads are complementary to human-written workloads as it provides a more comprehensive evaluation in terms of how general the optimization is and could identify suspicious harmful specialization patches.
  3. We generate workloads interactively and iteratively. Instead of prompting a language model to generate workloads, WorkloadPro launches a specialized coding agent in a sandbox docker environment to generate workloads. The agent can explore the codebase, read the PR descriptions and the human patch. It also profiles the execution to understand the performance issue. In addition, instead of querying the language model once, WorkloadPro iteratively generates better workloads on top of the workloads from the last iteration incrementally, aiming to cover different patterns of inputs and cover new lines, improving the test adequacy of the workloads.
  4. WorkloadPro is fundamentally more scalable. WorkloadPro does not depend on either expensive human annotation or historical workload corpus. It turns the discovery of workloads into an automated search problem. We admit that it will also generate some less effective workloads, but our filter mechanism will only keep the workloads that exercise the bottleneck the human patch tackles. In contrast, SWE-fficiency completely resorts to rely on a single workload, which we have shown to be the typical causes of encouraging overoptimizing.
AuthorsOpenReview ↗

Response to Reviewer iWor (3/5)

W1&Q2 (3): The paper is not positioned against closely related prior work, and the comparison baselines are therefore incomplete.

The SWE-fficiency paper itself reported workload overfitting as a failure mode. SWE-fficiency does have some preliminary efforts of detecting and mitigating workload overfitting, and we cite it as motivation. However, SWE-fficiency's "prevent model reward hacking" mechanism only targets two specific patterns: stack introspection to detect the timing context and caching across timing repetitions. For the stack introspection, they employ rule-based methods such as AST analysis to statically parse the patch, and check whether the patch contains API calls in the blacklist of stack introspection. The labeled patches are considered failing. For mitigating caching across timing repetitions, they employ process isolation via rewriting the workloads using AST. However, in practice, a lot of reward hacking behaviors are beyond the scope of the two patterns.

For example, we inspected the 17 patches flagged by WorkloadPro and found they come from different patterns, e.g., one patch rebuilds a cache in the forked timing process via register_after_fork, which bypasses the process isolation directly; other patches memoize repeated calls, exit early based on the benchmark’s data order, or tune constants to the benchmark’s input size, and none of them can be captured by SWE-fficiency's mechanisms. They are specializing based on the single visible workload and do not require stack introspection at all. However, they all rely on some aggressive assumptions based on the original workload. When the new workloads break such assumptions, the performance regressions indicate risks of workload overfitting. In other words, diversified workload evaluation is more effective and comprehensive in terms of detecting workload overfitting.

We will add the detailed discussion of FormulaCode and comparison of manual vs language model/agent-generated workloads, as well as SWE-fficiency's overfitting detection efforts, in the revised version of the paper.

Contribution of profiling in APO. We gave OpenHands and Claude Code the same profiling information as APO and a $3 budget. Results are averaged over 5 runs, k = 1, T = 5%:

MethodRect. GeneratedImpr. OriginalOpt@1Overfit
APO (GPT-5.5), single $3 run45.4 ± 1.163.7 ± 0.279.3 ± 0.01.8 ± 0.8 / 29
OpenHands (GPT-5.5) + profile, single $3 run37.8 ± 0.956.3 ± 0.062.1 ± 0.02.8 ± 0.4 / 29
Claude Code (Opus 4.7) + profile, single $3 run38.2 ± 0.352.3 ± 0.261.4 ± 1.52.6 ± 0.5 / 29
OpenHands (Opus 4.7) + profile, single $3 run28.7 ± 0.638.2 ± 0.142.8 ± 3.11.6 ± 0.5 / 29
OpenHands (GPT-5.5) + profile, best of 3 × $143.7 ± 1.068.5 ± 0.374.7 ± 2.02.3 ± 0.6 / 29
OpenHands (GPT-5.5) + profile, single $1 attempt (avg)26.1 ± 0.242.7 ± 0.149.8 ± 1.32.7 ± 0.0 / 29
Claude Code (Opus 4.7) + profile, best of 3 × $131.2 ± 1.343.9 ± 0.251.7 ± 0.02.0 ± 1.0 / 29
Claude Code (Opus 4.7) + profile, single $1 attempt (avg)21.0 ± 0.825.4 ± 0.234.5 ± 0.02.1 ± 0.4 / 29
OpenHands (Opus 4.7) + profile, best of 3 × $113.0 ± 2.27.4 ± 0.317.2 ± 0.02.0 ± 1.0 / 29
OpenHands (Opus 4.7) + profile, single $1 attempt (avg)10.7 ± 0.9-3.4 ± 0.413.0 ± 0.72.3 ± 0.3 / 29

With the same model, APO outperforms the profiling-guided single agent by 7.6 pp on held-out workloads and by 17 pp on Opt@1. The table shows that adding profiling alone does not necessarily improve the performance of baseline agents, i.e., it does not fully contribute to the improvement that APO achieves. Without profiling, they reach 41.6 (OpenHands, GPT-5.5) and 43.5 (Claude Code) on held-out workloads, still below APO (45.4).

Reviewer_fL3ZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets repository-level performance-optimization benchmarks (SWE-fficiency, GSO, SWE-Perf), where each task contains one or a few hand-written workloads that serve as both the agent’s optimization feedback and the final score. The authors observe the agents exploiting incidental properties of the visible input. In a motivating example, Claude Code reached 85% improvement on the visible workload but only ~18.0% improvement on held-out variants. To alleviate this, the authors make two contributions:

  1. WorkloadPro generates held-out workloads, using improvement under the human patch as a relevance filter and additional coverage as a diversity criterion.
  2. APO generates separate optimization-time guardrail workloads and uses profiling to explore multiple optimization targets before refining a selected candidate.

The evaluation covers 85 SWE-fficiency Lite tasks and 71 GSO tasks. All evaluated systems, including the human patches, achieve lower mean percentage improvements on the generated workloads. APO performs best among the evaluated agents and produces fewer patches that trigger the authors’ regression criterion.

Critical strengths
  • The authors use APO to submit 13 patches to real scientific software, out of which seven were accepted by the developers.
    • I have some concerns about the disclosure but this is a strong result for APO.
Critical weaknesses

I highly recommend the authors read this commentary on best practices for making agentic benchmarks: https://arxiv.org/abs/2507.02825

On the motivation of synthetic workloads. The primary motivation for workload generation is that existing benchmarks provide few workloads because of the “substantial expert effort required to construct high-quality workloads.” However, the cost of constructing new workloads does not establish a shortage of existing expert-generated workloads. The apparent scarcity may instead reflect how the selected benchmarks were constructed. Two datasets that come to mind that are able to mine diverse performance improvement workloads:

  • SWE-Pro is able to evaluate runtime and memory across varying workloads under noise-aware conditions.
  • FormulaCode mines >100 community-maintained workloads from similar repositories that SWEfficiency, GSO and SWE-Perf use.

These are not exact counterparts to WorkloadPro, but they demonstrate alternatives for obtaining and grounding multi-workload evaluation that the paper does not discuss. At minimum, the authors should explain where these approaches are insufficient for their target tasks and whether existing developer-maintained workloads can independently validate the generated ones.

On the realism of synthetic workloads. Now, it can be argued that expert-generated workloads may not exist for every performance issue. However, this makes it necessary to ensure that the generated workloads are grounded in real-world requirements and specifications for each repository. For example, a developer may reasonably prioritize a function’s performance on a particular range of inputs known to be important to the community, while preserving correctness more broadly and accepting specified performance tradeoffs elsewhere.

  • For example, pandas explicitly distinguishes optimization for in-memory datasets from scaling to larger-than-memory data, while Dask provides blocked execution for arrays that exceed memory.
  • Performance optimizations in scientific repositories need to be cognizant of the input distribution of their performance benchmarks as these values must conforms to physical laws (e.g. this PR from a quantum simulation library https://github.com/Qiskit/qiskit/pull/14458).

Presently, this personalization seems to happen via a requirement that the human patch improve each retained workload by at least 10%. However, the human patch is evidence of one useful implementation, not a specification of every performance tradeoff that maintainers would accept. Showing that the human patch benefits a workload does not establish that the workload falls within the intended usage regime or that a regression on it would make another optimization unacceptable.

There are other places where I feel realism is unnecessarily broken for ease of benchmark evaluation:

  • Any patch that causes generated workload to regress > 5% is automatically rejected. However, regressions are a regular (and accepted) part of optimization if they yield pareto-optimal improvement elsewhere depending on the workloads' importance to the user (No Free Lunch Theorems for Optimization; https://www.cs.ubc.ca/~hutter/earg/papers07/00585893.pdf).
  • In benchmarking models, its imperative that the true optimization objective be available, even if evaluation inputs are hidden. If success additionally requires protecting a broader workload family against specified regressions, that family and requirement should be communicated consistently rather than introduced only through evaluation.
  • One simple reason for why the human patches produce lower improvements on certain/new inputs regimes is because the optimized operations may account for a smaller fraction of the total execution time. Hence, the authors' observation that overall performance improvement drops on their input regimes may not directly characterize harmful specialization by the agents.

Overall, I agree with the authors' hypothesis that a single visible workload is reward hackable. However, the proposed evaluation needs to be able to distinguish between harmful specialization from intentional specialization. One solution for this is to (1) define the intended workload family and acceptable tradeoffs using project or issue evidence, (2) show a principled approach for how the generator preserves those requirements, (3) independently audit a representative sample of flagged patches against them.

Performance optimization is noisy.

  • WorkloadPro rejects any patch whenever any generated workload slows down by at least 5%. This makes the score sensitive to measurement uncertainty near that threshold. Consider the hypothetical scenario where the same patch, on multiple runs regresses on one of the workloads by 0%, 3%, and 6%. A performance engineer would remeasure until a confidence interval on the variance was established. But, here, the harness would simply reject the patch.
    • Concretely, please report whether the regressions that trigger rejection remain above the threshold when the same patches and workloads are remeasured multiple times, and whether the resulting overfit counts and rectified rankings remain stable.
  • WorkloadPro's suites contain between 6 and 66 workloads per task, and it seems that a single regression can cause a patch to be rejected. This raises a multiple-comparison concern (https://en.wikipedia.org/wiki/Multiple_comparisons_problem). i.e. each measurement incurs some noise so the suites with more workloads are more likely to be rejected due to random chance.
    • Concretely, please report run-to-run uncertainty for the updated leaderboard, and the variance of the uncertainty estimate for different task suites.
Questions for authors

In the weaknesses section.

7 comments
AuthorsOpenReview ↗

Response to Reviewer fL3Z (1/7)

Thank you for your insightful questions and detailed suggestions. The below is our responses.

W1.1: The cost of constructing new workloads does not establish a shortage of existing expert-generated workloads.

Thanks for bringing up these relevant works. We apologize for not discussing SWE-Opt (previously named SWE-Pro) and FormulaCode. They are indeed relevant to compare with for multi-workload evaluation. WorkloadPro complements them from the following aspects:

Expert efforts in workload generation. SWE-Opt involves extensive human efforts, including identifying associated unit tests, reading PR descriptions, code changes, and developer discussions, and selecting and validating generated parameterized performance tests. FormulaCode builds on top of historical expert efforts over years in writing workloads, i.e., the existing Airspeed Velocity (ASV) benchmarks.

WorkloadPro complements them by offering a fully automated and scalable pipeline, which does not depend on historical human-written workloads and expensive human annotation efforts. It can theoretically expand to any new applications to scale the evaluation up, as long as the human patch and one workload that reveals the performance issue are provided.

Quality of workloads and evaluation. FormulaCode is a good use of existing human efforts by using all ASV workloads in a repository for evaluation. WorkloadPro complements FormulaCode in the following perspectives:

  1. WorkloadPro generates targeted workloads for each specific task while FormulaCode relies on a set of universal workloads for the whole repository. The workloads that cover the patched functions are scarce among the whole set of workloads. We performed a coverage analysis on the 88 tasks whose human patch changes include Python code and found that the median fraction of workloads that execute the changed function is only 2.8% per task. However, in evaluation, the performance score is averaged by all workloads. Consequently, the evaluation of workloads covering the performance issues could be affected by the noise from the other 97.2% irrelevant workloads.
  2. The distribution of abundance of FormulaCode workloads and covering workloads is imbalanced across projects, and WorkloadPro complements FormulaCode on the projects where existing workloads are not abundant. We counted the number of available FormulaCode workloads and covering workloads per task and calculated the median number. The pandas project alone contributes most of the workloads, i.e., the median number (total/covering) of pandas is 1146/25 while it is only 36/1 for all other projects. In 15 tasks, no workload executes the changed functions
  3. The historical ASV workloads have been public in the internet for years, so it remains unclear if there are risks of model memorization. In contrast, WorkloadPro's held-out workloads are less prone to model memorization, as well as reward hacking. SWE-Opt also hides the parameterized performance tests from the agents.

SWE-Opt conducts a systematic parameter sweeps over input scales, data characteristics and function properties. It shares similar motivation and designs of the robustness workload generation of WorkloadPro. But WorkloadPro has an additional diverse workload generator where it iteratively changes the workload structure to reach new program paths, grounded by coverage analysis and human patches. The diverse workloads with various workload structures can exercise the intended performance bottleneck in different ways, thus improve the test adequacy of the benchmark and provide a more comprehensive evaluation.

In conclusion, our work is complementary because 1) it is fully automated, has no reliance on ASV benchmarks or human inspection thus scalable, 2) compared with FormulaCode, all generated workloads are specific to the performance bottleneck (validated by human patches), while the evaluation of FormulaCode could be affected by the noise of hundreds of irrelevant workloads, 3) compared with SWE-Opt, it has additional diverse workloads with different structures that exercise the intended bottleneck in different ways. We will add the detailed discussion in the revised version.

AuthorsOpenReview ↗

Response to Reviewer fL3Z (2/7)

W1.2: Can existing developer-maintained workloads independently validate the generated ones?

Good point. We examined the 30-task subset and found there are 7 tasks where the human patch adds or modifies asv benchmarks. We collected 49 patches from them (7 agents per task). 6 of them failed correctness tests. We then respectively used the asv workloads from the 7 tasks and WorkloadPro to flag potentially overoptimization patches. We applied the same rule as WorkloadPro, i.e., a patch is flagged if it is at least 5% slower on the asv benchmark that the human patch speeds up by at least 10%. It turns out that the two methods flagged the same 2 patches (Claude Code with and without profiling on scipy-8558, 2.5× and 4× slower on GaussianKDE.time_evaluate_many_points) and pass the other 41. This experiment indicates that the generated workloads can be validated by developer-maintained workloads. It is potentially a good proxy that approximates the developer-maintained workloads while being cheap and scalable (requiring no human efforts).

AuthorsOpenReview ↗

Response to Reviewer fL3Z (3/7)

W2.1 & W2.2: The human patch is evidence of one useful implementation, not a specification of every performance tradeoff that maintainers would accept. Regressions are a regular (and accepted) part of optimization if they yield pareto-optimal improvement elsewhere depending on the workloads' importance to the user.

The real-world input distributions are known to be non-uniform and vary from task to task, e.g., pandas targets in-memory data, and Qiskit's maintainers called some benchmarked circuit sizes "currently unrealistic." A good patch does not need to improve, or even preserve, performance under a uniform distribution over all valid inputs. WorkloadPro does not require a patch to optimize uniformly. Instead, it aims to find suspicious patches depending on aggressive, unrealistic specialization to the specific workloads.

The human patch is our anchor, as a best-effort proxy. Distinguishing valid specialization from harmful specialization requires a reference for which workloads matter, and tasks rarely come with such specifications. The human patch is the most reliable reference available, i.e., it passed maintainers' review and approximates the project's real-world requirements. We agree with the reviewer that it is imperfect, but the human patch is the best proxy we can find. In fact, using the human patch as the anchor is standard practice in existing performance benchmarks. SWE-fficiency keeps only workloads on which the expert edit shows a significant speedup. GSO keeps developer commits that show significant improvements on their workloads.

Specialization is acceptable. We aim to find narrow or wrong specialization. Human patches specialize too, and we fully agree that it is legitimate to specialize. That specialization itself encodes the developer's understanding of the input distribution, just as the pandas and Qiskit examples show. Robustness workloads vary only input construction (size, ordering, value distribution, data type), while diverse workloads must target the same bottleneck. We keep a generated workload only if the human patch speeds it up, so the kept workloads likely lie inside the input region the expert's specialization targets. Consequently:

  • A patch that performs well on these workloads likely shares the expert's assumptions about the workloads.
  • A patch that significantly regresses on them likely assumes a narrower distribution than the expert, typically the single workload it saw. These are the patches we aim to identify.
  • A patch that fixes a different performance issue than the intended one may regress on the held-out workloads and be flagged. We acknowledge this limitation. However, our evaluation asks whether the agent can resolve the specific performance issue addressed by the human patch. A fix for a different issue is out of scope under this criterion, even if it could be useful. Actually, we inspected 17 patches produced by agents that regress on some workloads from 30 randomly sampled tasks, and none of them is a legitimate fix of a different performance issue. In other words, such patches could exist in theory but are rare in our benchmark.

Regressions as trade-offs (W2.2). We agree that a regression can be an acceptable price for gains elsewhere, depending on how much each workload matters to the user. WorkloadPro does not reject such patches. Instead, we provide both rectified scores where the flagged patches are penalized with 0 score (i.e., Rect. Generated) and scores where no patch is penalized (i.e., Impr. Generated). Impr. Generated counts every workload as measured, without the flag. The flag affects only the rectified score, and it marks one specific trade-off, i.e., a slowdown of at least 5% on a workload that the human patch improves by at least 10%. Such a workload exercises the task's intended optimization, and the human patch shows that at least 10% improvement is achievable. Users who accept such trade-offs can compare agents by Impr. Generated, which is not affected by the flag.

Interpreting the results. Characterizing a project's input distribution is hard in general. We do not claim that our generated workloads or our "overoptimization" labels are the gold standard. The overoptimization number and rectified scores are signals to help users interpret results from different angles based on the needs. Compared with the original benchmark where only a single score from one or a few visible workloads is provided, which could be easily inflated by reward hacking or over-specialization, WorkloadPro gives a more comprehensive evaluation with numbers from different perspectives.

In the revision, we will:

  • (i) tone down the claims and use "flagged" to replace "overfitting";
  • (ii) clarify the human-patch anchor and its limitations explicitly;
  • (iii) advise users to interpret flagged numbers and rectified scores with caution.
Reviewer_8Gw6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates a significant limitation in performance-based coding-agent benchmarks: agents can achieve large reported speedups by optimizing for a small set of visible workloads, while the resulting improvements may not generalize to other valid workloads.

The authors introduce WorkloadPro, a benchmark augmentation framework that generates diverse held-out performance workloads. It constructs robustness workloads by varying input values, shapes, and types, and diverse workloads by exploring different execution paths. The generated workloads are filtered using human reference patches to ensure relevance to the intended optimization.

The paper also proposes APO (Agentic Performance Optimization), which combines profiling-guided workload generation with breadth-then-depth optimization. APO generates separate guardrail workloads during optimization, explores multiple potential bottlenecks in parallel, and refines the most promising optimization.

Experiments augment SWE-fficiency Lite and GSO with 5,027 generated workloads across 156 tasks. The results demonstrate substantial differences between reported improvements on original workloads and performance on held-out workloads. APO outperforms the strongest baselines by 7.9 and 10.8 percentage points on the generated workloads of the two benchmarks, respectively. The authors further demonstrate practical impact by identifying 13 performance issues across six open-source cyber-physical systems, with seven confirmed by maintainers and three patches merged.

Critical strengths
  1. Important and well-motivated problem. The paper addresses a real limitation of existing software performance benchmarks, where optimizing against a small set of visible workloads can produce misleading performance improvements. The matplotlib case study clearly demonstrates how iterative optimization can increasingly specialize a patch to benchmark inputs while reducing its effectiveness on other valid workloads. This is a meaningful problem for coding-agent evaluation and practical software optimization.

  2. Strong experimental evidence and practical impact. The authors construct a substantial held-out evaluation suite across two existing benchmarks, demonstrating that workload diversity can materially change measured performance and even reverse agent rankings. The proposed APO framework also shows encouraging improvements over existing agents. I particularly appreciate the real-world validation through contributions to open-source robotics systems, with confirmed performance issues and merged patches. This provides useful evidence of practical relevance beyond benchmark results.

Critical weaknesses
  1. The definition of overoptimization is not sufficiently precise or aligned with the benchmark evaluation.

This is my main concern. The paper uses the terms overoptimization, overfitting, and reward hacking somewhat interchangeably, while measuring several distinct phenomena: reduced speedups on generated workloads, performance regressions, and a threshold-based overfitting metric.

These measurements do not necessarily establish Goodhart's Law or reward hacking. A patch that improves one workload substantially more than others may represent legitimate workload-specific optimization rather than undesirable behavior. Similarly, a smaller speedup on held-out workloads does not by itself indicate overoptimization.

WorkloadPro primarily measures whether performance improvements generalize across diverse generated workloads. I would encourage the authors to more precisely define overoptimization, distinguish specialization from harmful regressions, and align the central claims with what the benchmark actually measures.

  1. Representativeness and potential selection bias of generated workloads.

WorkloadPro selects generated workloads based on whether the human reference patch achieves at least a 10% improvement. Although this helps ensure relevance to the intended optimization, it may bias the evaluation toward the specific optimization strategy implemented by the reference patch.

Furthermore, the generated workloads are not necessarily representative of real deployment workload distributions. The use of a 5% regression threshold to classify overfitting, and assigning zero improvement to patches exceeding that threshold on any generated workload, also requires stronger justification.

I would like to see additional analysis of workload representativeness and sensitivity to workload composition, regression thresholds, and scoring choices.

  1. Limited isolation of APO's methodological contributions.

APO combines profiling, workload diversification, parallel exploration, and iterative refinement. While the overall results are encouraging, the experiments do not sufficiently isolate the contributions of these components.

In particular, removing optimization-time workload generation only reduces held-out improvement from 47.0% to 46.3% on SWE-fficiency Lite and from 45.1% to 43.8% on GSO. These relatively modest differences raise questions about how much of the improvement comes from workload diversification versus profiling and broader optimization search.

More controlled ablations, particularly against a single-agent or multi-attempt optimization baseline with matched profiling information and compute budgets, would strengthen the methodological claims.

Questions for authors
  1. Problem definition: What precisely constitutes overoptimization in this setting? How does WorkloadPro distinguish harmful benchmark specialization or reward hacking from legitimate workload-specific optimization?

  2. Interpretation of the generalization gap: Even the human reference patches show lower improvements on generated workloads than on original workloads. How much of the observed performance gap should be attributed to overoptimization versus differences in the underlying workload distribution?

  3. Workload representativeness: How do the authors validate that the generated workloads represent meaningful software usage scenarios? Does filtering workloads based on human reference-patch improvements introduce bias toward that particular optimization?

  4. Evaluation metrics: How sensitive are the results and agent rankings to the 5% regression threshold, the number and composition of generated workloads, and alternative scoring schemes?

  5. APO contribution: Can the authors further isolate the contributions of workload generation, profiling, and breadth-then-depth optimization? In particular, would similar improvements be achieved by simpler profiling-guided agents under matched compute budgets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8Gw6 (1/4)

We are grateful for your time, effort, and leaving encouraging comments. We address your comments and questions in the following.

W1&Q1:The definition of overoptimization is not sufficiently precise or aligned with the benchmark evaluation.

The paper uses the terms overoptimization, overfitting, and reward hacking somewhat interchangeably. What precisely constitutes overoptimization in this setting? We apologize for the confusion. Reward hacking happens when the evaluation proxy is not strong enough, and an AI or reinforcement learning agent finds an unintended shortcut or loophole in its reward system, achieving high scores without actually accomplishing the real task. We use overoptimization to refer to such phenomenon in performance optimization, where there are no sufficient workloads and the agents hack the evaluation via aggressive operations like caching, identity checks, memoization, fast-paths (which break the semantics and skip significant computation), or it could speculatively assume a narrow input distribution based on the limited visible workloads, leading to optimizations with low generalizability. Operationally, a patch P is flagged if (i) it passes the correctness tests and (ii) on at least k WorkloadPro workloads, P is at least T%×\times slower than the baseline, where k=1 and T=5 in our experiments.

Our work focuses on overoptimization, while "overfitting" and "reward hacking" are both relevant terms, but they are more general concepts that are widely used in broader areas.

How does WorkloadPro distinguish harmful benchmark specialization or reward hacking from legitimate workload-specific optimization? In WorkloadPro, we rely on the human patch anchor to distinguish harmful benchmark specialization or reward hacking from legitimate workload-specific optimization. A generated workload is kept only if the developer's reference patch speeds it up by at least 10% over the baseline (workloads where the reference fix gives no gain are discarded). Each kept workload is direct evidence that (a) it exercises the intended performance issue, and (b) a correct speedup on the benchmark is achievable without slowing that workload down.

A patch that accelerates the target workload far more than others is a legitimate specialization, and WorkloadPro does not penalize it. A patch is flagged only if it slows down held-out workloads by more than 5%. We keep a held-out workload only if the human patch speeds it up by at least 10%. Therefore, a regression on such a workload cannot be explained as "that workload simply had nothing to optimize," because there exists a human patch that also optimizes this workload by ≥\geq 10%. For example, in pandas-52120 (slow groupby().min() on ordered Categoricals), the human patch routes the operation through existing Cython kernels. The OpenHands (GPT-5.5) patch instead adds a Python loop that stops once every group has seen the smallest category. In the benchmark this category happens to appear in the first few rows, so the patch reaches 34× speedup, versus 2.2× for the human patch. On generated workloads where the smallest category appears late or never, the loop scans the whole array and becomes 5.9× and 3.1× slower than the unpatched code, while the human patch is 6.7× and 2.1× faster. WorkloadPro flags the patch as overoptimization, whereas the original benchmark fails to recognize this problematic patch and ranks it far above the human patch.

We will define clearly what overoptimization is, clarify relevant terms, and clarify the difference between harmful specialization and legitimate optimization in the revised version.

AuthorsOpenReview ↗

Response to Reviewer 8Gw6 (2/4)

W2&Q3&Q4: Representativeness and potential selection bias of generated workloads.

Bias toward the human patch and representativeness. We agree that our workload filter depends on the human patch and could be biased towards it. The filter is needed because generated workloads can exercise code unrelated to the task, and tasks rarely specify which workloads matter. We use the human patch because it passed maintainers' review and reflects the project's real-world requirements. The filter only asks that a workload exercise the intended performance bottleneck. Agents do not need to follow the expert's strategy, and they are scored on their own speedups, which can exceed the expert's. A patch that fixes a different performance issue than the intended one may regress on the held-out workloads and still be flagged. We acknowledge this limitation. However, our evaluation asks whether the agent can resolve the specific performance issue addressed by the human patch. A fix for a different issue is out of scope under this criterion, even if it could be useful. Actually, we inspected 17 patches produced by agents that regress on some workloads from 30 randomly sampled tasks, and none of them is a legitimate fix of a different performance issue. In other words, such patches could exist in theory but are rare in our benchmark.

Representativeness. We do not guarantee that the generated workloads strictly follow real-world distributions. However, we use several designs to make them as representative as possible. Workload generation starts from each benchmark's official workloads, and the prompt explicitly asks for plausible, in-distribution API usage and inputs rather than artificial corner cases, so the model's knowledge of library usage is used to produce representative workloads. The human patch serves as an additional filter to make sure that the workloads exercise the intended performance bottleneck. For example, dask-7172 shows an example where the workload generator was not told how this function is used, but it still produced realistic workloads: some workloads call da.percentile(...).compute() on chunked arrays with commonly reported quantiles, and others sort each input to be "monotonic like a real percentile output" (from the code comment).

Sensitivity to the threshold, the workloads, and the scoring scheme. We ran the full sensitivity study on 30 randomly sampled tasks (5 repetitions), and the results show that WorkloadPro's conclusions are robust and not sensitive to T and k.

Thresholds T and k. We swept T ∈\in {2, 5, 10, 20}% and k ∈\in {1, 2, 3}. In every setting:

  • APO outperforms all baselines.
  • The two OpenHands (Opus 4.7) configurations are the bottom two.
  • Claude Code (Opus 4.7) patches are flagged the most.

Number and composition of workloads. We recalculate the scores using random 75%, 50%, and 25% subsets of the generated workloads (200 draws each) and using a single workload type. The top two and bottom two are unchanged in every variant except one. The number of detected overfitting patches grows with the number of workloads. For example, OpenHands (GPT-5.5) has 2.2 flags with 25% of the workloads and 4.2 with all of them. Rankings are stable even with fewer workloads.

Alternative scoring rules. We scored all agents under three additional rules: (i) geometric-mean speedup over the original and all generated workloads, with no threshold; (ii) worst-case speedup, the smallest speedup across all workloads; (iii) a noise-aware flag, where a patch is flagged only if the median slowdown over 5 runs on some generated workload exceeds 5%.

Under all three rules, APO remains in the top three and top one under most rules. The agent whose rank varies most is OpenHands (GPT-5.5): it performs the best on average-speedup metrics, but the performance drops when regressions on held-out workloads are counted, and even dropping to last under worst-case speedup. Therefore, the regressions could be hidden by average speedups, which is also a motivation of WorkloadPro's design.

We will add the sensitivity analysis and the discussion of representativeness and potential bias in the revision version.

AuthorsOpenReview ↗

Response to Reviewer 8Gw6 (3/4)

W3&Q5: Limited isolation of APO's methodological contributions.

Simpler agents using profiling information under matched budgets. We gave OpenHands and Claude Code the same profiling information as APO and a $3 budget. Results are averaged over 5 runs, k = 1, T = 5%:

MethodRect. GeneratedImpr. OriginalOpt@1Overfit
APO (GPT-5.5), single $3 run45.4 ± 1.163.7 ± 0.279.3 ± 0.01.8 ± 0.8 / 29
OpenHands (GPT-5.5) + profile, single $3 run37.8 ± 0.956.3 ± 0.062.1 ± 0.02.8 ± 0.4 / 29
Claude Code (Opus 4.7) + profile, single $3 run38.2 ± 0.352.3 ± 0.261.4 ± 1.52.6 ± 0.5 / 29
OpenHands (Opus 4.7) + profile, single $3 run28.7 ± 0.638.2 ± 0.142.8 ± 3.11.6 ± 0.5 / 29
OpenHands (GPT-5.5) + profile, best of 3 × $143.7 ± 1.068.5 ± 0.374.7 ± 2.02.3 ± 0.6 / 29
OpenHands (GPT-5.5) + profile, single $1 attempt (avg)26.1 ± 0.242.7 ± 0.149.8 ± 1.32.7 ± 0.0 / 29
Claude Code (Opus 4.7) + profile, best of 3 × $131.2 ± 1.343.9 ± 0.251.7 ± 0.02.0 ± 1.0 / 29
Claude Code (Opus 4.7) + profile, single $1 attempt (avg)21.0 ± 0.825.4 ± 0.234.5 ± 0.02.1 ± 0.4 / 29
OpenHands (Opus 4.7) + profile, best of 3 × $113.0 ± 2.27.4 ± 0.317.2 ± 0.02.0 ± 1.0 / 29
OpenHands (Opus 4.7) + profile, single $1 attempt (avg)10.7 ± 0.9-3.4 ± 0.413.0 ± 0.72.3 ± 0.3 / 29

With the same model, APO outperforms the profiling-guided single agent by 7.6 pp on held-out workloads and by 17 pp on Opt@1. The table shows that adding profiling alone does not necessarily improve the performance of baseline agents, i.e., it does not fully contribute to the improvement that APO achieves. Without profiling, they reach 41.6 (OpenHands, GPT-5.5) and 43.5 (Claude Code) on held-out workloads, still below APO (45.4).

Multi-attempt baseline. To separate breadth-then-depth search from compute, we split the \3budgetintothreeindependent3 budget into three independent \\1 attempts per task, each with profiling, and keep the best attempt by original-workload speedup. The strongest multi-attempt baseline, OpenHands (GPT-5.5), reaches 43.7 on held-out workloads (vs. 37.8 for its single \3run)butstaysbelowAPO(45.4)andOpt@1(74.7vs.79.3).ItdoesexceedAPOontheoriginalworkload(68.5vs.63.7).Inotherwords,selectingthefastestattemptcouldfavorpatchesthataremorespecializedtotheoriginalworkloadsandnotgeneraltootherworkloads.ForbothOpus4.7agents,three3 run) but stays below APO (45.4) and Opt@1 (74.7 vs. 79.3). It does exceed APO on the original workload (68.5 vs. 63.7). In other words, selecting the fastest attempt could favor patches that are more specialized to the original workloads and not general to other workloads. For both Opus 4.7 agents, three \\1 attempts score below one \3run(31.2vs.38.2forClaudeCode;13.0vs.28.7forOpenHands),becauseasingle3 run (31.2 vs. 38.2 for Claude Code; 13.0 vs. 28.7 for OpenHands), because a single \\1 attempt often fails to produce a correct patch.

Workload generation. The main role of workload generation is to mitigate potentially overoptimization patches on the fly, i.e., without it, the number of flagged patches increases from 3 to 5 and from 2 to 6, and rectified improvement drops by 2.1 and 2.7 pp. Workload generation does not directly contribute to speedup. We will discuss the contributions of each module in details in the revision.

AuthorsOpenReview ↗

Response to Reviewer 8Gw6 (4/4)

Q2: How much of the observed performance gap should be attributed to overoptimization versus differences in the underlying workload distribution?

We do find that part of the gap comes from the generated workloads themselves, i.e., human patches also drop from 61.5% to 46.8% on SWE-fficiency Lite, likely because the optimized code accounts for a smaller fraction of total execution time on new workloads. Although the absolute drop does not by itself indicate overoptimization, comparing the drop of an agent with the drop of the human patch and the drops of other agents might tell us something interesting, e.g., the agent shows particularly good performance on the original visible workloads but the drop is significantly more than the human patch and other agents, then it is a suspicious signal of harmful specialization and could potentially alarm the user of the risk of inflation.

To separate the effect of workloads (smaller fraction) and patches (overoptimization), we compare each agent with the human patch on the same tasks. For example, on SWE-fficiency Lite, the human patch keeps 76% of its original-workload improvement on the generated workloads, and the agents keep 60–69%. In other words, 7-16 percentage points performance improvement drops come from overoptimization of the patches generated by the agents.

In the revision, we will report these differences explicitly and add a per-workload normalization by the human patch, together with the runtime fraction of the patched functions on original vs. generated workloads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.