acceptodds
Under review as a conference paper at ICLR 2027

A Claim Is Not a Measurement: Compliance Claims Change What LLM Evaluators Verify and Accept

Abstract

LLM evaluators increasingly decide which work an agent keeps, and are often given verification tools to do so reliably. A tool helps only if it is used and its result read correctly. Can report claims change verification and acceptance? Holding the work, its contract and a reliable on-request check fixed, we edit one report field to claim compliance without a matching measurement; a held-out verifier scores whether the evaluator obtains the deciding measurement and what it accepts. Across eight models, where verification was the pre-specified endpoint, the claim reduced it: a copied tolerance cut the deciding audit by 22.6 points in mathematics, and a favourable proxy cut audits from 99.5% to 29.0% in finance. In physics, echoing the requested value instead of the measured one raised false acceptance of non-compliant work by 44.5 points (task-cluster bootstrap [27.0, 51.9]); a matched-decimal execution agreed in sign. Evidence in the report did not guarantee a correct verdict: in a diagnostic execution, adding the measurement's raw result shrank a claim-versus-deleted-field contrast from 37.5 to 4.75 points, unevenly across models; some still certified non-compliant work. Unsupported claims also arise unedited: in one research-agent workflow, an AI-assisted audit finds admitted validation claims contradicted by the runs' own records and reused in later prompts. On one task, neither stating the defect nor replacing reported metrics with held-out scores established a change in delivered quality. Access to a verification tool is not verification: evaluations should measure whether checks are used and how their results are judged.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.