acceptodds
Under review as a conference paper at ICLR 2027

Benchmark Graders Miss What Agents Leave Behind

Abstract

Benchmark graders can award passing scores to outputs that violate task requirements. Repairing such errors can change which agent is recommended, but the value of that change depends on performance on new outputs. We study this distinction using repeated coding-agent executions, archived public code generations, and a documented banking-benchmark correction. Additional checks reveal sensitive keys retained in nested outputs and invoices reset after an intervening payment. We compare the original-score and corrected-score choices on the same corrected target in another execution. Across four new model environments, mean selection benefits include gains, no change, and losses. The negative mean persists in one environment when the disputed migration check is omitted. On public code generations, selectors frozen before held-out scoring reduce mean pairwise loss from 0.098 to 0.052 passed tasks, while the full-pool leader remains unchanged. The banking correction also reduces mean selection loss while retaining adverse transfers. These results separate checking task requirements from evaluating the consequence of a revised recommendation. We release deterministic checks, paired score records, and a calculator for reproducing these comparisons.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, benchmark auditing, evaluation reliability, model selection, agent harnesses

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MZMzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that benchmark graders sometimes pass outputs that break task rules. It asks whether fixing the grader leads to better agent choices. To test this, it picks a winner with the old scores and a winner with the fixed scores, then compares both on a different run. Results are positive on EvalPlus and τ²-bench banking, but mixed (gains, zeros, losses) for agent recovery policies.

Critical strengths
  1. Good question. Most audit papers stop at "the ranking changes." This paper asks if the new ranking is actually better on new outputs. The test is simple and reusable.
  2. Concrete bugs and open process. The grader failures are real and easy to check (e.g., a report scored 1.0 that names the wrong largest overrun). The authors release their checks and report when each was built.
Critical weaknesses
  1. Too little data. Each new model has only two runs, and effects are under one pass out of 16. Just re-running the same plan changes more verdicts (7–17 cells) than the grader fix. So the mixed results may be noise.
  2. Results depend on the authors' own choices. Most negative results come from the authors' reading of task 086. Under the simpler reading, most changes disappear. The checks were also improved after seeing outputs.

The paper is also very hard to read: dense numbers, many private terms, and 93 pages.

Questions for authors
  1. What does the benefit look like if you remove the same number of passes at random? If it is similar, the grader fix has no clear effect.
  2. Can you add more runs per model, or a bootstrap over tasks?
  3. Why is your reading of task 086 the right one? Can you show main results with the simpler reading?
Reviewer_ckBCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper explores the effect of correcting flawed benchmark graders on tasks like HumanEval. It finds fixing those graders can change which e.g. model or configuration looks better, but on new outputs the corrected recommendation is sometimes better, sometimes no different, and sometimes worse.

Critical strengths

Reasonable methodology and benchmark choices, across diverse fields like coding/banking etc.

Critical weaknesses

The practical effect averaged over everything is small: about 0.05 tasks out of 164.

The paper is somewhat hard to read, e.g. Figure 1 caption uses a lot of technical jargon that it doesn't explain.

The fact that scores change with improved graders does not seem particularly notable. It would be more interesting to find that particularly popular benchmarks are flawed in some novel way.

Questions for authors

It seems significant results depend on single tasks like task-086. Do you think your findings would generalise to newer agentic coding benchmarks?

Reviewer_hk8bOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes repairing benchmark acceptance errors from improving the recommendation induced by repaired scores. It evaluates original-score and corrected-score choices on the same corrected target in different executions or generation panels. Recovery-agent artifact checks expose nested sensitive-key retention and invoice-state resets under an explicit state-preservation interpretation. Recovery selection benefits have mixed signs; frozen pairwise selectors on held-out public-code generations improve mean loss, while a documented banking correction also yields a mean gain with adverse transfers. The contribution is an empirical decision-audit application rather than a new regret framework.

Critical strengths

The evaluation cleanly separates score repair from the consequence of following a changed recommendation. Both selectors are assessed against the same corrected target on different outputs (§5, p. 5; App. K.13, pp. 80–81, Table 43). Frozen public-code selectors are tested on five unread panels, retaining unchanged choices and adverse transfers: nine of 231 pairs change, producing 28 gains, five zeros, and twelve losses among 45 changed transfers (App. L.2, pp. 88–90, Tables 49–50). This is materially stronger than showing a ranking change on the outputs used to derive it.

Concrete artifact failures and sensitivity analyses support an appropriately bounded audit. Task 042's explicitly named nested keys are checked with benign-field controls (App. K.5, pp. 68–69); contradictory largest-overrun assertions and malformed CSV provide additional direct examples (App. F.6, pp. 43–46, Tables 23–25). The manuscript distinguishes task 086's paid-update interpretation from static idempotence, reports the disappearing unique-winner replacement, and retains a negative DeepSeek mean after restoring task-086 labels (App. K.11, pp. 77–78, Table 41; App. K.14, pp. 82–83, Table 44). These disclosures prevent a universal-benefit narrative.

Critical weaknesses

The measured decision benefit has a narrow estimand and limited demonstrated practical reach. The approximately 47% held-out reduction is about 0.046 passed tasks out of 164, averaged uniformly over all 231 fixed pairs; 222 pairs retain the same choice, and the 22-model leader has zero loss under either selector (App. L.2, pp. 88–89, Tables 49–50). Those pair-panel comparisons share tasks and models. Each new recovery environment supplies only one execution pair, and the banking study reuses four trials (§5.1–5.3, pp. 5–8; App. K.14, pp. 82–83; App. L.3, pp. 90–91, Table 51). The authors disclose these facts, but they still limit inference about typical future recommendation value. Minimum remedy: foreground the already-reported absolute, per-panel and changed-choice results beside the relative headline, and make the fixed-pair weighting and conditional target explicit there. Genuine frozen-selector holdout exists; additional frozen evaluation and dependence-aware uncertainty would be needed only for stronger generalization across new tasks or releases, not to establish the reported finite comparisons.

“Corrected” recovery scores are not a comprehensive or independently adjudicated correctness target. Version 5 covers 25 of 50 obligations, with output-informed repair requests; its task-086 sequence differs from the retrospective probe on two graded cells (App. K.9, pp. 72–73). Static idempotence withdraws no task-086 passes under the double-engine rule, whereas paid-update checking drives several prominent selection changes (App. K.11, pp. 77–78, Table 41). Exact-key task-042 detection has stronger explicit-source support, but benign preservation controls for those shapes do not establish full conversion correctness. These are disclosed boundaries, yet the outcome called decision utility remains utility under this particular partial oracle. Minimum remedy: place a concise covered-target definition next to the main loss results, explicitly distinguishing no detected violation from verified correctness, and provide independent source-based adjudication of the small set of decisive task-042/086/087 cases. Retain full and narrow results; alternatively, keep all correctness and utility conclusions explicitly conditional on the implemented checks without presenting the target as general task ground truth.

Questions for authors

Decision population: What concrete selection setting motivates uniform weighting over the 231 fixed model pairs, and how should the approximately 0.046-task held-out gain be interpreted when the full-pool choice is unchanged? Please distinguish this finite-pair result from a claim about future tasks or releases (App. L.2, pp. 88–89, Tables 49–50).

Decisive oracle validity: For the narrow DeepSeek adverse transfer, can you supply source-anchored adjudication of task 087's missing regression-test obligation and task 042's nested-key exclusions, including benign-field acceptance, and explain why these suffice for the stated conditional target? This would strengthen the negative example that survives the disputed migration scope (App. K.14, p. 83).

Migration consistency: How do the two disagreements between version 5 and the retrospective 042/086 checks affect interpretation of the historical selector losses, and what source-grounded migration semantics justify the different rerun sequences and engine acceptance rules? Please reconcile the p. 73 version comparison with the static/paid-update analyses on pp. 77–81 (Apps. K.9, K.11, K.13).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.