- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark comprising 30 systems-level development tasks. It measures reward hacking through the gap in pass rates between visible tests of individual features and hidden tests of multi-feature interactions. Experiments show that stronger models and smaller tasks generally exhibit smaller gaps, but these gaps do not disappear entirely.
Critical strengths
- Splitting tests into visible and hidden sets and measuring the gap in pass rates seems like a reasonable way to assess whether coding agents engage in reward hacking.
Critical weaknesses
- It is unclear whether the requirements evaluated by the held-out tests can be correctly inferred from the task descriptions. The measured reward hacking gap may instead reflect insufficiently clear or comprehensive task specifications.
Questions for authors
- In real-world development, coding agents may receive only natural language description, without predefined unit tests. It would be helpful to include an additional experiment where agents receive only the task description, to better understand how the findings generalize to this setting.
- Could the authors provide more details on how the 30 tasks were constructed or selected, and how they assessed whether the task descriptions clearly and comprehensively specify the requirements covered by the held-out tests?
- I came across a paper on arXiv with the same title but seemingly different content (https://arxiv.org/abs/2605.30314). Could the authors clarify how it relates to this submission?