- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks if an LLM used as a judge can be fooled by what the agent writes in its own report. The setting is a judge that decides if the work produced by another agent should be accepted and considered for the next steps. The authors fix the task, the delivered work and the verification tool, and change only one field of the report: in one case it contains a value that looks compliant but was never measured, in the others the field is empty or contains the real measurement. They find that when the report shows a compliant-looking value, the judge checks less with the tool and accepts more wrong work, even if a single tool call would have revealed the problem. Giving the raw measurement inside the report helps on average, but some models still accept wrong work. The authors also show, on logs of a real ML agent, that this kind of unsupported claim appears spontaneously and is reused in later steps.
Critical strengths
S1. Clean experimental design: The core intervention is well controlled: the task, the work and the verification tool are the same in all conditions, and only one field of the report changes, so it is clear that the effect comes from there. Tool use and the final verdict are recorded as two separate outcomes, which separates a judge that skips verification from one that verifies and still accepts non-compliant work. The authors also test the judge on correct work and on clearly wrong work, and in these cases the claim does not change the decision, so the effect is present only when the problem is not visible without a measurement. This makes the main result credible, even if its relevance is not clear.
Critical weaknesses
- W1. Unclear framing: The paper does not commit to what the evaluator is expected to do. Section 3 says that trusting a report is not unreasonable, and Section 5 only points to contract-gated certification as the implied norm, without testing it; yet the observed behavior is labeled with normative terms (unsafe certification, false acceptance, reward-hacking surface). The evaluator prompt (Appendix Z.2, physics) does not say who wrote the report or that its values may be unverified, so trusting a plausible value seems a reasonable policy under these instructions. Section 5 supports this: once the raw measurement is in the report, most models detect the violation. The main finding may therefore reflect the instructions given to the evaluator more than a failure of verification, and without a stated ideal behavior it is hard to say what the effect means and why it matters.
- W2. Presentation: The paper is extremely hard to parse: the text is very dense and compressed, and the main text is difficult to follow on its own. Key terminology (for example contract, held-out verifier, deciding audit) is used in the text before being defined. There is no full example of what a report looks like and what the claim means in practice. The paper needs a systematic rewriting to clearly convey its goals and findings. In its current state, even if the results may be of interest, they are difficult to fully understand. Together with W3, this is the main reason for my recommendation: the work needs a deeper revision and a better contextualization of its goal.
- W3. Unclear practical relevance: The controlled experiments show that the evaluator is sensitive to a claim in the report, but the paper does not show that this matters in practice. First, the real-world setting does not match the controlled one: the whole of Section 6 is run on AIDE, whose reviewer, as stated in Section 6.2, already sees code and output and has no tool to request, so there is no verification to skip. Second, no real negative effect is found there: the audit (Section 6.1) only shows that unsupported claims exist and propagate, and the two continuation experiments (Section 6.3), run on one task with one model, find no detectable effect on the final results, as the authors themselves state. Third, the remedy that the authors themselves propose, always verifying before certifying, is not tested (Section 5). As a result, the paper documents a sensitivity, but it is not clear whether it has any impact in practice or how hard it is to fix.
Minor:
- Clarity in Table 1 can be improved. The contrasts switch order across columns (echo - measured vs. absent - copied), while the rate row always lists the claim first, so it is not always clear how the values in “Eight models” are computed.
- Appendix figures and tables are cited inconsistently: sometimes as “Appendix Figure 13”, sometimes as a bare “Figure 11”, although all of them are in the appendix. Either always use the “Appendix” prefix or number appendix figures/tables by section (e.g. Figure Q.1), so the reader knows where they are located.
Questions for authors
- Q1: Section 3 states that trusting a report is not unreasonable, yet the certifications are labeled unsafe. Under the instructions in the evaluator prompt (Appendix Z.2), what is the evaluator expected to do in each condition, and why? If the intended norm is contract-gated certification (Section 5), why was it not given to the evaluator or tested?
- Q2: The evaluator prompt (Appendix Z.2) says that at most two measurements may be requested. Do the authors think that removing this limit, or framing the measurements as expected rather than optional, would change how willing the evaluator is to use the available tool?
- Q3: Section 4.2 presents the effect as “more specific than a model believing its prompt”, but not checking a value that the report already provides seems to be exactly that. Can you explain better why it is not the same thing?
5 comments
Response to the second comment (1/2): the request limit, tested; purpose
Thank you for a specific, testable alternative. You kept your rating because the purpose and the practical relevance were unclear; we answer the purpose here, after the experiment you asked for (Sections 4.4 and 5, Table 3; Appendix G.6), and practical relevance in part 2, which adds a re-analysis of delivered quality in the unedited workflow (Section 6.2). The experiment was designed after your comment: eight models, 4,320 verified decisions.
Q2: two requests may explain much of the main pattern. Partly confirmed. In one run, the claim raised false acceptance over the measured value by 37.0 points [16.7, 47.0] with two requests and by 14.0 [7.0, 17.9] with five, which, as we verified before any call, suffice to check every requirement. The limit's effect is 23.0 [7.5, 30.2]: the limit enlarged the effect substantially but did not explain all of it. At five requests the effect came from four of eight models and stayed positive with unreadable replies counted least favourably (11.0 [3.9, 15.1], post hoc).
Q2: the instruction's cost may reflect the limit. Supported. With two requests, passing lenses under the instruction were accepted 63.6% and 54.5%; the lower acceptance came mainly from abstention. With five, they were accepted 98.9% and 96.6%, within the pre-set 10-point margin relative to the original prompt at the same five requests (98.9% and 94.3%; differences 0.0 [-3.4, 3.4] and +2.3 [-3.4, 9.1]). We removed the general statement that the instruction costs passing work.
W1: purpose. To your question, whether this is a failure of LLM evaluators or of pipelines whose prompts say nothing about where report values come from: the second. We do not present reliance on an unattributed report field as surprising. The question is what that reliance does to an evaluation pipeline that already has a reliable check, and which settings of the pipeline reduce the resulting errors. The controlled result concerns the correctness of the acceptance decision itself. This is relevant when such decisions govern selection or reward, while consequences for final outputs require separate evidence. The new experiment separates two settings one can change: a larger request limit shrank the claim's effect but still left errors (14.0 points); a measurement requirement with enough requests left no observed false acceptance while passing lenses stayed accepted, as did running every check in advance. This is evidence about pipeline design, not a claim that evaluators reason badly. The instruction changes the warning and the requirement together, so it does not separate "the model" from "the prompt".
Response to the second comment (2/2): practical relevance, delivered quality, readability
W3: practical relevance. You made three points; we answer them in order.
"[AIDE's] reviewer ... already sees code and output and has no tool to request, so there is no verification to skip." Agreed. Section 6 does not reproduce the controlled mechanism. It asks whether such claims arise without our editing, and whether they affect what the workflow keeps and delivers. Giving AIDE's reviewer a tool would study a modified workflow. The re-analysis below uses the original logged reviewer decisions; no check tool was added to those historical runs.
"No real negative effect is found there." We added a re-analysis of delivered quality, with no new model call (Section 6.2; Appendix H.4, Table H.7). In the 24 unedited runs of our paired experiment, we kept every step and every admission of the reviewer and changed only the final choice: the admitted nodes were ranked by a score on training rows held out from fitting, instead of the reported metric. AIDE's own rule, applied to the logs, reproduced all 24 delivered files, and the analysis rules were committed before new test scoring for this re-analysis. Test AUC of the delivered file was higher by 0.017 on average: [0.002, 0.035] by a bootstrap over runs, [-0.002, 0.035] by a t interval. Of the 17 runs whose file changed, 10 improved and 7 worsened; three runs account for about 92% of the net gain. This is supportive evidence that, at the final choice, ranking by the reported metric delivered worse files than a held-out score would have. It is not decisive: the analysis is post hoc (the delivered files' scores were known), covers one task and one model, and changes only the last step, not the search; the t interval includes zero; and it does not separate unsupported claims from other reasons a reported metric ranks nodes badly, such as scoring noise. Whether the claims themselves cost quality therefore remains open, and Sections 6.2 and 7 say so. In full new runs, keeping and ranking nodes by the held-out score gave +0.020 [-0.007, 0.048] (Section 6.2), compatible with no change and with a meaningful gain.
"The remedy that the authors themselves propose, always verifying before certifying, is not tested." It is now tested, in the request-limit experiment of part 1. With five requests, the instruction to accept only on a measurement, given to the evaluator, left no observed false acceptance in 200 decisions on failing lenses per report version (25 failing designs × 8 models; 6 and 4 replies unreadable) and kept passing lenses accepted (98.9% and 96.6%); running every check before the evaluator reads the report did the same (3 unreadable replies per version; Table 3). Observing no errors does not establish a zero error probability, and these remedies were not tested on what real agents deliver (Section 8). Section 8's last item, giving such an instruction enough requests, is therefore now a tested result rather than advice.
W2: readability. This revision changes the presentation, not the results. The figures of the main text are redrawn: Figures 1 and 3 now show the report lines themselves (one field with the claimed or the measured value; one value under two names) beside the rates, and Figure 2 draws each model's contrast in every pre-specified comparison on one axis, in the direction the claim predicts, where the main text had a table of signed values (now Appendix Table G.3). Section titles are short labels, long sentences in Sections 4.4, 5 and 6.2 are split, Sections 4.4 and 6.2 are divided into one paragraph per topic, each figure and table stands beside the paragraph that reads it, and the table of every pair of versions moved to Appendix I (Table I.2). The main text defines its terms in Section 3 and adds none for the new experiment; the appendix uses the main text's terms in most of its prose, and its reading guide lists the record names that remain. We would welcome pointers to the passages that remain hardest to read.
We would be grateful to know whether these results address your concerns, and whether they change your assessment.
Response to Authors
Dear Authors,
Thank you for the detailed response and the extensive revision
W1 / Q1. The response states that relying on the report is admissible under the given instructions. This clarifies the framing but not my concern: the paper then only shows that an evaluator not told that report values may be unmeasured uses them as evidence, which is expected. The instruction experiment supports this, since the effect largely disappears once evaluators are told. The purpose of the paper therefore remains unclear: is it a failure of LLM evaluators, or of pipelines whose prompts say nothing about where report values come from?
Q2. The framing part of my question was tested, but the request limit itself was not, and this leaves me with strong doubts about the setting. By the response's own count, verifying every requirement needs four or five requests, but only two are allowed, so the evaluator must trust the report for some requirements. Trusting what the report provides and measuring what it lacks is then reasonable, and would explain much of the main pattern. Read literally, the rule in the instruction experiment also cannot be followed with two requests, so its cost on compliant designs may largely reflect the limit.
Q3. Addressed: the response agrees that the results are compatible with reliance on the report, and the sentence in Section 4.2 was removed.
W2. The main text was almost entirely rewritten but still requires substantial effort to read, as many sentences remain compressed. The appendix, which contains the details needed to check the results, still uses different terms than the main text and relies on a translation table (Reading guide, p. 16). That said, the main text now defines its terms before use and uses them consistently, which I appreciate, and I will raise my Presentation score to two.
W3. The response agrees with all three points and the revision now states them, but none is fully resolved. The native workflow still has no check to skip, no downstream effect is established, and the remedy is tested only as a prompt instruction under the two-request limit discussed in Q2. The practical advice in Section 8 is generic, and its last item, giving such an instruction enough requests, is exactly what was not tested. The practical relevance of the finding therefore remains undemonstrated.
At this stage I am inclined to keep my rating, mainly because the purpose and practical relevance of the main finding remain unclear.