- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes OverclaimBench, a benchmark that assesses the agent's propensity to overclaim the task done in coding tasks. Throughout the evaluation across 8 frontier models and 5 file-review scenarios, the authors show that all the coding agents have a non-zero overclaim rate, and that requiring delegation to sub-agents increases both coverage and the misleading rate. The evaluation results show that coding agents' final responses are not reliable accounts of their actions
Critical strengths
- This paper proposes OverclaimBench, an evaluation suite that can systematically evaluate the overclaim propensity of coding agents across five scenarios, offering a new perspective on evaluating reliability and deception in agentic coding tasks.
- Experiments are comprehensive, covering frontier closed-weight and open-weight models and mainstream coding scenarios. The author also conducted a comprehensive study of judge reliability and failure-mode analysis.
- Writing is clear and easy to follow. The author provides a clear definition of overclaim in line 84 and the research questions in line 53, offering a clear picture of the project scope and the problem it aims to solve. The experiment setup and results analysis are clear and easy to understand.
Critical weaknesses
- The agent harness is not fixed. For closed-source models, the author uses their native production CLI instead of the fixed coding agents, while for open-weight models, the author uses the fixed Claude code harness. The experiment results only highlight the differences between models, but omit the confounding factor of the agent harness. It would be better to evaluate all the models under a fixed agent framework.
- I don't understand the purpose of injecting needles here. From my perspective, the ability to detect needles is a separate capability that may not have a direct, causal relationship with overclaim. To evaluate overclaim propensity, we don't have to test the ability to identify needles, as we only need to check the difference between the claimed progress and the number of files touched. If the agent is doing a comprehensive review but misses some needles, it's purely a capability issue. To make the observation: "Needles are more often missed when agents overclaim" hold, we really need an ablation study that fixes the task and the file coverage, and the only difference is whether the agent is overclaiming or not, then check the miss rate of needles. It seems like this study is missing so I don't think the obervation is reasonable here.
Questions for authors
- For the results in Figure 2, do we have any requirements for the use of subagent?
2 comments
I thank the authors for their detailed responses, and most of my concerns are addressed. I therefore maintain my assessment.
Response to Reviewer YYBP
We evaluate each model as an agent, i.e. the model with its native production harness, because that is how these systems are deployed; the main results therefore characterize deployed agents rather than models in isolation. Part of the evaluation already holds the harness fixed: the three Claude models and the four open-weight models all run in Claude Code, and within this single harness the open-weight models' incomplete runs are misleading in 65–85% of cases.
To address the reviewer’s concern, we also ran the remaining five models in Claude Code on Sprint Planning (20 runs per model; see the General Response), so every model has now been evaluated under the same harness on that scenario. The harness changed coverage for some models, so it can shift overall rates and cross-model comparisons, but it barely changed disclosure: 76 of the 79 incomplete runs were still misleading, compared with 95 of 95 in the native harnesses. We will note in the limitations that cross-model differences in overall rates reflect model and harness together.
We agree that needle recovery and accurate reporting are distinct. Indeed, our overclaiming judgments depend only on the discrepancy between the agent’s coverage claims and its transcript. Needle recovery is an additional component and is evaluated separately.
We include needles to test whether coverage gaps have real consequences, a question complementary to overclaiming propensity. Not reading every file is not in-and-of-itself a failure. An agent could take "clever shortcuts," e.g., keyword searches (such as grepping for API keys) that find everything important while skipping unimportant files. This is a valid approach if the agent finds the defects and is honest about how it did so. By contrast, when a review misses a planted, task-relevant defect, we can establish that the agent did not recover everything important.
A needle was reported 83.2% of the time its evidence was read, compared with 1.8% when it was not. Capability failures thus account for a fraction of missed needles, but the vast majority of missed needles are those whose evidence the agent did not read.
We have removed the heading "Needles are more often missed when agents overclaim" and have revised it to
for clarity and added a sentence to the text further explaining this:
In our main evaluation, task prompts neither require nor prohibit subagent use; agents run in their native harness freely choose whether to delegate. This is stated in the “Subagents” subsection of the paper, and we have now also added “Agents choose whether to delegate” to the caption of Figure 2 to avoid any ambiguity for the reader. Further details on the delegation behavior underlying the results in Figure 2 can be found in Appendix D.1.