- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper shows that benchmark graders sometimes pass outputs that break task rules. It asks whether fixing the grader leads to better agent choices. To test this, it picks a winner with the old scores and a winner with the fixed scores, then compares both on a different run. Results are positive on EvalPlus and τ²-bench banking, but mixed (gains, zeros, losses) for agent recovery policies.
Critical strengths
- Good question. Most audit papers stop at "the ranking changes." This paper asks if the new ranking is actually better on new outputs. The test is simple and reusable.
- Concrete bugs and open process. The grader failures are real and easy to check (e.g., a report scored 1.0 that names the wrong largest overrun). The authors release their checks and report when each was built.
Critical weaknesses
- Too little data. Each new model has only two runs, and effects are under one pass out of 16. Just re-running the same plan changes more verdicts (7–17 cells) than the grader fix. So the mixed results may be noise.
- Results depend on the authors' own choices. Most negative results come from the authors' reading of task 086. Under the simpler reading, most changes disappear. The checks were also improved after seeing outputs.
The paper is also very hard to read: dense numbers, many private terms, and 93 pages.
Questions for authors
- What does the benefit look like if you remove the same number of passes at random? If it is similar, the grader fix has no clear effect.
- Can you add more runs per model, or a bootstrap over tasks?
- Why is your reading of task 086 the right one? Can you show main results with the simpler reading?