acceptodds
Withdrawn from ICLR 2027

SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As autonomous coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite. Reward hacking naturally arises in this setup, as the agent optimizes over the target of passing more automated test, the test suite cease to be a good measure for the developer to review the real progress of the agents. We introduce SpecBench, a benchmark of 30 systems-level programming tasks, spanning reference implementations from a 1,500-line JSON parser to a 110,000-line operating-system kernel. Each task includes two test suites: (i) visible validation tests that exercise specified features in isolation, and (ii) held-out tests that compose those same features to simulate real-world usage. Crucially, these held-out suites introduce no requirements beyond the visible validation suites, a genuinely implementation will inherently pass the validation suites as well as the held-out suites. Thus we can use the gap in pass rates on these two suites to quantify reward hacking. Large scale experiments reveal a consistent pattern: every frontier agent saturates the visible suite. Yet reward hacking persists, with smaller models demonstrating larger gaps between test suites. The gap also scales sharply with task length: the gap grows by 28 percentage points for every tenfold increase in code size. Failures range from subtle feature isolation to deliberate exploits, including a 2,900-line hash-table "compiler" that memorizes test inputs. SpecBench offers a principled testbed for measuring whether coding agents build genuine working systems or merely game the test suites developers hand them.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reward Hacking, Coding Agents Abstract:

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tPoqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark comprising 30 systems-level development tasks. It measures reward hacking through the gap in pass rates between visible tests of individual features and hidden tests of multi-feature interactions. Experiments show that stronger models and smaller tasks generally exhibit smaller gaps, but these gaps do not disappear entirely.

Critical strengths
  1. Splitting tests into visible and hidden sets and measuring the gap in pass rates seems like a reasonable way to assess whether coding agents engage in reward hacking.
Critical weaknesses
  1. It is unclear whether the requirements evaluated by the held-out tests can be correctly inferred from the task descriptions. The measured reward hacking gap may instead reflect insufficiently clear or comprehensive task specifications.
Questions for authors
  1. In real-world development, coding agents may receive only natural language description, without predefined unit tests. It would be helpful to include an additional experiment where agents receive only the task description, to better understand how the findings generalize to this setting.
  2. Could the authors provide more details on how the 30 tasks were constructed or selected, and how they assessed whether the task descriptions clearly and comprehensively specify the requirements covered by the held-out tests?
  3. I came across a paper on arXiv with the same title but seemingly different content (https://arxiv.org/abs/2605.30314). Could the authors clarify how it relates to this submission?
Reviewer_Z4C5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpecBench, a benchmark for measuring reward hacking by coding agents that build software systems from scratch. It has 30 tasks, from a JSON parser to an OS kernel, with reference implementations of about 1.5K to 110K lines. Each task has a natural-language specification and starter code. It also has a visible validation suite that tests single features and a hidden held-out suite that combines these features. The reward hacking gap is the validation pass rate minus the held-out pass rate. The authors argue that a genuine implementation should pass both suites. The study runs Codex, Claude Code, and OpenCode (with six other models) under three search loops (AIDE, Linear, Autoresearch).

Critical strengths
  1. The question is timely and the setup is simple. As agents write more code than people can review, the test suite becomes the main check. So it matters how much a passing visible suite overstates real quality. The task suite is a useful resource. It covers long, from-scratch system builds over a wide range of sizes, and every task has a reference implementation that passes all tests. Few existing benchmarks cover this setting.

  2. The experiments are broad and the examples are concrete. Across agents, models, and search loops, visible pass rates are near-saturated while hidden pass rates vary widely.

Critical weaknesses
  1. The gap does not isolate reward hacking. Because validation scores are near-saturated, the gap is mostly the held-out failure rate. An honest but imperfect system also passes single-feature tests more often than tests that combine several features. So a positive gap is expected even without any gaming. The paper's own evidence points this way. The LLM judge finds very few deliberate exploits, and the main SQL example is described as unintentional. Adding visible composition tests as hard as the hidden ones does not close the gap. A compiler built with human guidance also shows a clear gap, mainly because the hidden tests check error handling that its own tests never covered. The paper does not say how the specifications, reference implementations, and tests were built or checked. So the key claim that the hidden tests add no new requirements is not shown.

  2. The LLM judge is neither named nor checked against human labels.

Questions for authors
  1. Does the gap measure hacking? Can you show that the gap comes from optimizing against the visible tests, not from task difficulty?

  2. Inconsistent numbers. Figure 4 gives Claude Opus 4.6 a mean gap of about 8pp. Figure 5 gives Claude Code, which runs Opus 4.6, 43–48pp. Both are described as means over tasks and seeds. Which is correct? Other mismatches:

    • Table 3's rows sum to 1912 runs and 33458, not 2046 and 38094.
    • Table 5 lists 31 tasks.
  3. Test construction. How were the specifications, reference implementations, and tests built: by hand, adapted from open-source projects, or with an LLM? How did you check that each held-out test is implied by the specification and the visible tests?

Reviewer_RXmjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces systems-programming tasks with visible tests for individual features and hidden tests for feature compositions. It uses the pass-rate gap as an operational measure of reward hacking. Experiments across coding models, agent frameworks and search strategies examine relationships with task size, model capability and search budget, supplemented by case studies of explicit shortcuts and compositional failures.

Critical strengths
  • Systems-building tasks with hidden compositional tests address a useful gap in coding-agent evaluation.
  • The compiler example provides concrete evidence of harmful proxy optimization: search replaces a functional candidate scoring 53%/43% with a lookup-table implementation scoring 97%/0% on visible/held-out tests (§3.5).
Critical weaknesses
  • The central metric conflates hacking with ordinary implementation errors and compositional difficulty. The two suites intentionally test different distributions; a positive score gap alone cannot establish reward hacking. The cited Skalse et al. definition does not justify this equivalence. Consequently, the model-capability and horizon trends have plausible explanations unrelated to hacking.

  • Major results are difficult to reconcile. Figure 4 places Claude Opus 4.6 near an 8 pp mean gap, while Figure 5 shows roughly 39–64 pp for Claude Code, without explaining the population difference. Table 3’s rows sum to 1,912 runs, not the reported 2,046.

Questions for authors
  • What control separates proxy-induced shortcuts from ordinary compositional failure, beyond the individual exploit examples?

  • Can a canonical run manifest reconcile Figures 4–5 and Table 3, including task subsets, budgets, model configurations and aggregation weights?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.