acceptodds
Under review as a conference paper at ICLR 2027

A Claim Is Not a Measurement: Compliance Claims Change What LLM Evaluators Verify and Accept

Abstract

LLM evaluators increasingly decide which work an agent keeps, and are often given verification tools to do so reliably. A tool helps only if it is used and its result read correctly. Can report claims change verification and acceptance? Holding the work, its contract and a reliable on-request check fixed, we edit one report field to claim compliance without a matching measurement; a held-out verifier scores whether the evaluator obtains the deciding measurement and what it accepts. Across eight models, where verification was the pre-specified endpoint, the claim reduced it: a copied tolerance cut the deciding audit by 22.6 points in mathematics, and a favourable proxy cut audits from 99.5% to 29.0% in finance. In physics, echoing the requested value instead of the measured one raised false acceptance of non-compliant work by 44.5 points (task-cluster bootstrap [27.0, 51.9]); a matched-decimal execution agreed in sign. Evidence in the report did not guarantee a correct verdict: in a diagnostic execution, adding the measurement's raw result shrank a claim-versus-deleted-field contrast from 37.5 to 4.75 points, unevenly across models; some still certified non-compliant work. Unsupported claims also arise unedited: in one research-agent workflow, an AI-assisted audit finds admitted validation claims contradicted by the runs' own records and reused in later prompts. On one task, neither stating the defect nor replacing reported metrics with held-out scores established a change in delivered quality. Access to a verification tool is not verification: evaluations should measure whether checks are used and how their results are judged.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, AI agents, evidence acquisition, reward hacking, agentic science

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KPs3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks if an LLM used as a judge can be fooled by what the agent writes in its own report. The setting is a judge that decides if the work produced by another agent should be accepted and considered for the next steps. The authors fix the task, the delivered work and the verification tool, and change only one field of the report: in one case it contains a value that looks compliant but was never measured, in the others the field is empty or contains the real measurement. They find that when the report shows a compliant-looking value, the judge checks less with the tool and accepts more wrong work, even if a single tool call would have revealed the problem. Giving the raw measurement inside the report helps on average, but some models still accept wrong work. The authors also show, on logs of a real ML agent, that this kind of unsupported claim appears spontaneously and is reused in later steps.

Critical strengths

S1. Clean experimental design: The core intervention is well controlled: the task, the work and the verification tool are the same in all conditions, and only one field of the report changes, so it is clear that the effect comes from there. Tool use and the final verdict are recorded as two separate outcomes, which separates a judge that skips verification from one that verifies and still accepts non-compliant work. The authors also test the judge on correct work and on clearly wrong work, and in these cases the claim does not change the decision, so the effect is present only when the problem is not visible without a measurement. This makes the main result credible, even if its relevance is not clear.

Critical weaknesses
  • W1. Unclear framing: The paper does not commit to what the evaluator is expected to do. Section 3 says that trusting a report is not unreasonable, and Section 5 only points to contract-gated certification as the implied norm, without testing it; yet the observed behavior is labeled with normative terms (unsafe certification, false acceptance, reward-hacking surface). The evaluator prompt (Appendix Z.2, physics) does not say who wrote the report or that its values may be unverified, so trusting a plausible value seems a reasonable policy under these instructions. Section 5 supports this: once the raw measurement is in the report, most models detect the violation. The main finding may therefore reflect the instructions given to the evaluator more than a failure of verification, and without a stated ideal behavior it is hard to say what the effect means and why it matters.
  • W2. Presentation: The paper is extremely hard to parse: the text is very dense and compressed, and the main text is difficult to follow on its own. Key terminology (for example contract, held-out verifier, deciding audit) is used in the text before being defined. There is no full example of what a report looks like and what the claim means in practice. The paper needs a systematic rewriting to clearly convey its goals and findings. In its current state, even if the results may be of interest, they are difficult to fully understand. Together with W3, this is the main reason for my recommendation: the work needs a deeper revision and a better contextualization of its goal.
  • W3. Unclear practical relevance: The controlled experiments show that the evaluator is sensitive to a claim in the report, but the paper does not show that this matters in practice. First, the real-world setting does not match the controlled one: the whole of Section 6 is run on AIDE, whose reviewer, as stated in Section 6.2, already sees code and output and has no tool to request, so there is no verification to skip. Second, no real negative effect is found there: the audit (Section 6.1) only shows that unsupported claims exist and propagate, and the two continuation experiments (Section 6.3), run on one task with one model, find no detectable effect on the final results, as the authors themselves state. Third, the remedy that the authors themselves propose, always verifying before certifying, is not tested (Section 5). As a result, the paper documents a sensitivity, but it is not clear whether it has any impact in practice or how hard it is to fix.

Minor:

  • Clarity in Table 1 can be improved. The contrasts switch order across columns (echo - measured vs. absent - copied), while the rate row always lists the claim first, so it is not always clear how the values in “Eight models” are computed.
  • Appendix figures and tables are cited inconsistently: sometimes as “Appendix Figure 13”, sometimes as a bare “Figure 11”, although all of them are in the appendix. Either always use the “Appendix” prefix or number appendix figures/tables by section (e.g. Figure Q.1), so the reader knows where they are located.
Questions for authors
  • Q1: Section 3 states that trusting a report is not unreasonable, yet the certifications are labeled unsafe. Under the instructions in the evaluator prompt (Appendix Z.2), what is the evaluator expected to do in each condition, and why? If the intended norm is contract-gated certification (Section 5), why was it not given to the evaluator or tested?
  • Q2: The evaluator prompt (Appendix Z.2) says that at most two measurements may be requested. Do the authors think that removing this limit, or framing the measurements as expected rather than optional, would change how willing the evaluator is to use the available tool?
  • Q3: Section 4.2 presents the effect as “more specific than a model believing its prompt”, but not checking a value that the report already provides seems to be exactly that. Can you explain better why it is not the same thing?
5 comments
Reviewer_KPs3OpenReview ↗

Response to Authors

Dear Authors,

Thank you for the detailed response and the extensive revision

W1 / Q1. The response states that relying on the report is admissible under the given instructions. This clarifies the framing but not my concern: the paper then only shows that an evaluator not told that report values may be unmeasured uses them as evidence, which is expected. The instruction experiment supports this, since the effect largely disappears once evaluators are told. The purpose of the paper therefore remains unclear: is it a failure of LLM evaluators, or of pipelines whose prompts say nothing about where report values come from?

Q2. The framing part of my question was tested, but the request limit itself was not, and this leaves me with strong doubts about the setting. By the response's own count, verifying every requirement needs four or five requests, but only two are allowed, so the evaluator must trust the report for some requirements. Trusting what the report provides and measuring what it lacks is then reasonable, and would explain much of the main pattern. Read literally, the rule in the instruction experiment also cannot be followed with two requests, so its cost on compliant designs may largely reflect the limit.

Q3. Addressed: the response agrees that the results are compatible with reliance on the report, and the sentence in Section 4.2 was removed.

W2. The main text was almost entirely rewritten but still requires substantial effort to read, as many sentences remain compressed. The appendix, which contains the details needed to check the results, still uses different terms than the main text and relies on a translation table (Reading guide, p. 16). That said, the main text now defines its terms before use and uses them consistently, which I appreciate, and I will raise my Presentation score to two.

W3. The response agrees with all three points and the revision now states them, but none is fully resolved. The native workflow still has no check to skip, no downstream effect is established, and the remedy is tested only as a prompt instruction under the two-request limit discussed in Q2. The practical advice in Section 8 is generic, and its last item, giving such an instruction enough requests, is exactly what was not tested. The practical relevance of the finding therefore remains undemonstrated.

At this stage I am inclined to keep my rating, mainly because the purpose and practical relevance of the main finding remain unclear.

AuthorsOpenReview ↗

Response to the second comment (1/2): the request limit, tested; purpose

Thank you for a specific, testable alternative. You kept your rating because the purpose and the practical relevance were unclear; we answer the purpose here, after the experiment you asked for (Sections 4.4 and 5, Table 3; Appendix G.6), and practical relevance in part 2, which adds a re-analysis of delivered quality in the unedited workflow (Section 6.2). The experiment was designed after your comment: eight models, 4,320 verified decisions.

Q2: two requests may explain much of the main pattern. Partly confirmed. In one run, the claim raised false acceptance over the measured value by 37.0 points [16.7, 47.0] with two requests and by 14.0 [7.0, 17.9] with five, which, as we verified before any call, suffice to check every requirement. The limit's effect is 23.0 [7.5, 30.2]: the limit enlarged the effect substantially but did not explain all of it. At five requests the effect came from four of eight models and stayed positive with unreadable replies counted least favourably (11.0 [3.9, 15.1], post hoc).

Q2: the instruction's cost may reflect the limit. Supported. With two requests, passing lenses under the instruction were accepted 63.6% and 54.5%; the lower acceptance came mainly from abstention. With five, they were accepted 98.9% and 96.6%, within the pre-set 10-point margin relative to the original prompt at the same five requests (98.9% and 94.3%; differences 0.0 [-3.4, 3.4] and +2.3 [-3.4, 9.1]). We removed the general statement that the instruction costs passing work.

W1: purpose. To your question, whether this is a failure of LLM evaluators or of pipelines whose prompts say nothing about where report values come from: the second. We do not present reliance on an unattributed report field as surprising. The question is what that reliance does to an evaluation pipeline that already has a reliable check, and which settings of the pipeline reduce the resulting errors. The controlled result concerns the correctness of the acceptance decision itself. This is relevant when such decisions govern selection or reward, while consequences for final outputs require separate evidence. The new experiment separates two settings one can change: a larger request limit shrank the claim's effect but still left errors (14.0 points); a measurement requirement with enough requests left no observed false acceptance while passing lenses stayed accepted, as did running every check in advance. This is evidence about pipeline design, not a claim that evaluators reason badly. The instruction changes the warning and the requirement together, so it does not separate "the model" from "the prompt".

AuthorsOpenReview ↗

Response to the second comment (2/2): practical relevance, delivered quality, readability

W3: practical relevance. You made three points; we answer them in order.

"[AIDE's] reviewer ... already sees code and output and has no tool to request, so there is no verification to skip." Agreed. Section 6 does not reproduce the controlled mechanism. It asks whether such claims arise without our editing, and whether they affect what the workflow keeps and delivers. Giving AIDE's reviewer a tool would study a modified workflow. The re-analysis below uses the original logged reviewer decisions; no check tool was added to those historical runs.

"No real negative effect is found there." We added a re-analysis of delivered quality, with no new model call (Section 6.2; Appendix H.4, Table H.7). In the 24 unedited runs of our paired experiment, we kept every step and every admission of the reviewer and changed only the final choice: the admitted nodes were ranked by a score on training rows held out from fitting, instead of the reported metric. AIDE's own rule, applied to the logs, reproduced all 24 delivered files, and the analysis rules were committed before new test scoring for this re-analysis. Test AUC of the delivered file was higher by 0.017 on average: [0.002, 0.035] by a bootstrap over runs, [-0.002, 0.035] by a t interval. Of the 17 runs whose file changed, 10 improved and 7 worsened; three runs account for about 92% of the net gain. This is supportive evidence that, at the final choice, ranking by the reported metric delivered worse files than a held-out score would have. It is not decisive: the analysis is post hoc (the delivered files' scores were known), covers one task and one model, and changes only the last step, not the search; the t interval includes zero; and it does not separate unsupported claims from other reasons a reported metric ranks nodes badly, such as scoring noise. Whether the claims themselves cost quality therefore remains open, and Sections 6.2 and 7 say so. In full new runs, keeping and ranking nodes by the held-out score gave +0.020 [-0.007, 0.048] (Section 6.2), compatible with no change and with a meaningful gain.

"The remedy that the authors themselves propose, always verifying before certifying, is not tested." It is now tested, in the request-limit experiment of part 1. With five requests, the instruction to accept only on a measurement, given to the evaluator, left no observed false acceptance in 200 decisions on failing lenses per report version (25 failing designs × 8 models; 6 and 4 replies unreadable) and kept passing lenses accepted (98.9% and 96.6%); running every check before the evaluator reads the report did the same (3 unreadable replies per version; Table 3). Observing no errors does not establish a zero error probability, and these remedies were not tested on what real agents deliver (Section 8). Section 8's last item, giving such an instruction enough requests, is therefore now a tested result rather than advice.

W2: readability. This revision changes the presentation, not the results. The figures of the main text are redrawn: Figures 1 and 3 now show the report lines themselves (one field with the claimed or the measured value; one value under two names) beside the rates, and Figure 2 draws each model's contrast in every pre-specified comparison on one axis, in the direction the claim predicts, where the main text had a table of signed values (now Appendix Table G.3). Section titles are short labels, long sentences in Sections 4.4, 5 and 6.2 are split, Sections 4.4 and 6.2 are divided into one paragraph per topic, each figure and table stands beside the paragraph that reads it, and the table of every pair of versions moved to Appendix I (Table I.2). The main text defines its terms in Section 3 and adds none for the new experiment; the appendix uses the main text's terms in most of its prose, and its reading guide lists the record names that remain. We would welcome pointers to the passages that remain hardest to read.

We would be grateful to know whether these results address your concerns, and whether they change your assessment.

Reviewer_pBn4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper purprotedly claims to explore if the claims made by a report affect the verification and acceptance of the report.

I believe AI was heavily used to write most, if not all, of this manuscript. There are several indicators that make this work very difficult to parse:

  • Run-on sentences littered throughout the paper
  • An abstract that quickly devolves into a recap of results without establishing the motivation and methodology
  • The absence of a clear structure to the paper
  • Arbitrary incorproration of metrics. For instance, the list of contributions in the introduction add mentions of some top-level numbers when the methodology has yet to be discussed.
Critical strengths

None

Critical weaknesses

This paper needs a complete rewrite to be understandable.

Questions for authors

N/a

1 comment
AuthorsOpenReview ↗

Thank you sincerely

Dear Reviewer pBn4,

As the discussion closes, we want to thank you sincerely. Your review was sharp, and it helped us a great deal. It was painful to read at first, because the questions, the design and the experiments came from long and careful work. But the fault was ours: we had many results and, within nine pages, we did not explain them clearly. We are sorry for that, and we thank you for saying so plainly. It is the reason the paper is clearer now. We understand your position on the revision, and we respect it.

You wrote that the other reviewers were better able to understand the manuscript. For your information, Reviewer KPs3, who followed the revisions, has since written that "the content of the paper (the experiments, results) is the same", while noting, as you did, that the text changed a great deal, and raised many questions along the way. With their guidance we improved the paper step by step and addressed the questions they raised. They have now raised their rating from 2 to 3.

Thank you again for your time.

Reviewer_z4vHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an important reliability problem in LLM-based agent systems, namely, whether unsupported claims of compliance can influence an LLM evaluator's decision to verify and accept an artifact, even when reliable verification tools are available. The paper is motivated by many agentic workflows rely on LLMs to evaluate generated code, scientific results, financial analyses, or other artifacts. The authors investigate whether statements contained in an agent's report can influence the evaluator's verification behavior and final acceptance decision. The experiments cover three domains, physics, mathematics and finance. The authors evaluate 8 LLMs and report substantial sensitivity to unsupported claims.

Critical strengths
  1. The paper presents timely research questions with practical use cases, although in controlled settings.
  2. The focus of the paper goes beyond conventional LLM-as-a-judge biases by examining whether reports influence the evaluator's evidence-gathering behavior.
  3. The paper shows interesting findings, for example , an evaluator may accept a non-compliant artifact because it does not request the necessary evidence, or because it misinterprets evidence already provided.
  4. The experiments cover physics, mathematics, and finance, which involve substantially different artifacts, contractual requirements, and verification procedures.
  5. Finally, what I enjoyed learning was that the paper show that access to evidence does not ensure correct evaluation, which I believe has potential to be extensively studied.
Critical weaknesses
  1. The controlled evaluation relies on relatively constrained and artificial verification environments which makes it unclear if the results will generalize to a real-world environment.
  2. Although the controlled experimental design is a strength, the evaluation settings are highly structured. Evaluators receive explicit contracts, and a fixed menu of tools.
  3. The paper does not study whether the observed negative behavior leads to meaningful downstream harm, or how significant the consequences of lack of verification is among various cases. I am wondering if one could cluster the different verified and unverified cases based on whether they led to dangerous behavior versus unwanted behavior which did not end up being a danger?
  4. The evidence from naturally occurring agent workflows is limited.
  5. The experiments demonstrate that changing a compliance-related report field can alter whether an evaluator requests verification. However, they do not determine why this happens. To be fair this is not reasonably within the scope, but I believe the paper does need a discussion on this point.
  6. Similarly, it would be nice if the paper would present some discussion about practical ways for addressing the observed issues.
  7. Finally, the presentation is often times difficult to follow and could be improved. In some cases the claims are difficult to understand.
Questions for authors

Please see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer z4vH

Thank you for the careful review and for the suggestions on what the paper should discuss. We number your weaknesses W1-W7 in the order you listed them.

W1-W2. Constrained, highly structured settings. We agree: written requirements, a fixed list of checks, a small request limit and one prompt per setting. That structure is what lets us hold the work and the check fixed and change only the report, and the Limitations (Section 7) now name each element. The three settings differ in artifact, requirement and check, and the effect appears in each where its outcome was pre-specified. Whether it holds with open-ended tools and implicit requirements is untested.

W3. Downstream harm; dangerous versus unwanted cases. We measured delivered quality in new runs on one task and could not establish a change (Section 6.2); we did not measure application-level harm or its severity, which Section 7 states. Two facts bear on severity without measuring harm:

  • In physics, the accepted lenses are not marginal failures: on the 25 failing designs the delivered f-number exceeds the required one by 11.9–42.9%.
  • In the unedited agent workflow, the 20 claims that the runs' own records contradict (under the recorded verdicts) are of two kinds: 9 present a training score as held-out validation, and 11 report a value the run's own measurement contradicts. All 20 entered later prompts.

Classifying cases as dangerous or merely unwanted would need a consequence measure per case, which we do not have.

W4. Limited evidence from natural workflows. Agreed. Section 6.1 now says that AIDE's reviewer has no tool to request, so this evidence shows that claims contradicted by run records arise and travel, not that checking is skipped.

W5. Why it happens. Section 8 now has a paragraph of possible explanations. Our evidence is behavioural and does not identify a mechanism. The replies fit evaluators taking a field presented as the required quantity as evidence for it: in the new Table 1 the evaluator computes the pupil from the repeated f-number and accepts, and in finance the field's name alone, on the same value, reduced checking. That an instruction changed the result (Section 5) shows a dependence on instructions, not why the check was skipped.

W6. Practical remedies. Section 5 tests the two natural remedies, and Section 8 draws the practical points:

  • Supplying the measured value lowered false acceptance in every setting, but in physics 37 of 400 decisions on failing lenses still accepted a lens whose reported measurement showed the failure, 36 of them from two models.
  • A warning or a provenance label left most false acceptance in place (Appendix Figure E.1).
  • An instruction to accept only on a measurement, with two requests allowed, left no detectable difference in false acceptance between the claim and the measured value (10.00% and 8.75%), but passing work was accepted only 60–69% of the time, most of the remaining verdicts being abstentions; the gain cannot be separated from that shift.

From these: evaluate tool-using evaluators on whether the decisive check is used and whether verdicts match the work, not only on whether tools are offered; make the fields a requirement depends on hold measurements of the delivered work, and test whether that evidence is used (how a pipeline should enforce this is untested); and give any measure-before-accept instruction enough requests, reporting its abstentions with its errors.

W7. Presentation. Agreed; see the general response for the rewrite.

AuthorsOpenReview ↗

Update on W1-W2, W6 and W3: a new experiment and a re-analysis

Since our reply, the revision has gained one experiment and one re-analysis that bear on your W1-W2, W6 and W3.

W1-W2. Constrained settings. One of the constraints you noted, the small request limit, is now varied. With five requests, enough to verify every requirement, the claim still raised false acceptance, by 14.0 points [7.0, 17.9] against 37.0 with two (Sections 4.4 and 5, Table 3; Appendix G.6). The limit explains part of the effect, not all of it. The other constraints (written requirements, a fixed list of checks) remain, as Section 7 states.

W6. "Practical ways for addressing the observed issues." Our reply reported that an instruction to accept only on a measurement cost passing work when evaluators had two requests. With five requests, no false acceptance was observed in 200 decisions on failing lenses per report version (25 failing designs × 8 models; 6 and 4 replies unreadable), and passing lenses were accepted 98.9% and 96.6%, within the pre-set 10-point margin relative to the original prompt at five requests. Running every check before the evaluator reads the report gave the same (3 unreadable replies per version). Observing no errors does not establish a zero error probability; Section 8 states these results and their scope.

W3. "Whether the observed negative behavior leads to meaningful downstream harm." We added a re-analysis of existing runs of the agent workflow, with no new model call (Section 6.2; Appendix H.4). For one task and one model, choosing the delivered file by a held-out score instead of the reported metric, among the same admitted nodes, gave test AUC higher by 0.017 on average ([0.002, 0.035] by a bootstrap over runs; [-0.002, 0.035] by a t interval). It is post hoc and does not separate unsupported claims from other ranking errors, so harm caused by the claims is still not established, and we still have no consequence measure per case with which to sort cases into dangerous and merely unwanted.

W7. Presentation. The figures are redrawn to show the report lines beside the rates, and long sentences in Sections 4.4, 5 and 6.2 are split.

We would be grateful to know whether our replies address your concerns.

AuthorsOpenReview ↗

Thank you sincerely

Dear Reviewer z4vH,

As the discussion closes, we want to thank you sincerely. Your review showed us that the idea of the paper had come through even though the writing was hard to follow, and that encouraged us a great deal. Your comments also shaped the revision: the discussion of why this happens and of practical remedies in Section 8 is there because of your suggestions. With your help we made the paper better.

If any of your concerns remains, we would be glad to hear it. Thank you again for your time and your encouragement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Update (revised PDF): a new experiment on the request limit, and a re-analysis of delivered quality

We have uploaded a revised PDF. Reviewer KPs3 asked whether the two-request limit in the physics setting explains the main result and the instruction's cost on passing work. We tested this in a new pre-specified experiment, designed after the reviews and reported as such (Sections 4.4 and 5, Table 3; Appendix G.6). In one run it compared two requests with five, which suffice to verify every requirement; crossed both with the instruction to accept only on a measurement; and added a condition in which every check is run before the evaluator reads the report. Eight models, 4,320 decisions, all verified.

  • The limit explains part of the effect, not all of it. The claim raised false acceptance by 37.0 points with two requests and by 14.0 points [7.0, 17.9] with five (difference 23.0 [7.5, 30.2]); at five the effect came from four of eight models.
  • With five requests, the instruction's cost on passing lenses was within the pre-set 10-point margin relative to the original prompt at the same five requests (acceptance 98.9% and 96.6% under the instruction, 98.9% and 94.3% under the original prompt; 63.6% and 54.5% under the instruction with two). With five requests and the instruction, no false acceptance was observed in 200 decisions on failing lenses per report version (25 failing designs × 8 models; 6 and 4 replies unreadable); with every check run in advance, none either (3 unreadable replies per version). Observing no errors does not establish that the underlying error probability is zero.
  • The abstract, introduction and Sections 3, 4.4, 5, 7 and 8 now state these results. The appendix uses the main text's terms in most of its prose; its reading guide lists the record names that remain.
  • Presentation, with no result changed: the figures of the main text are redrawn (Figures 1 and 3 now show the report lines themselves beside the rates; Figure 2 draws each model's contrast in the pre-specified comparisons where the main text had a table of signed values, now Appendix Table G.3), section titles are short labels, Sections 4.4 and 6.2 are split into one paragraph per topic, each figure and table stands beside the paragraph that reads it, and the table of every pair of versions moved to Appendix I (Table I.2). Figure and table numbers in our earlier comments refer to the 9 Nov revision: its Figures 2, 3 and 4 are now Figure 3, Figure 4 and Appendix Figure E.1, and its Table 4 is now Appendix Table G.12.

Reviewers KPs3 and z4vH also asked whether these claims matter for what an agent delivers. We added a re-analysis of existing runs, with no new model call (Section 6.2; Appendix H.4, Table H.7). In the 24 unedited runs of our paired downstream experiment, we kept every step and every admission by the workflow's feedback model, and changed only the final choice: the admitted nodes were ranked by a score on training rows held out from fitting, instead of the reported metric. Test AUC of the delivered file was higher by 0.017 on average ([0.002, 0.035] by a bootstrap over runs; [-0.002, 0.035] by a t interval), with most of the gain from three runs. This is supportive evidence at the final choice, not an established cost of the claims: it is post hoc, covers one task and one model, leaves the search unchanged, and does not separate unsupported claims from other reasons a reported metric ranks nodes badly.

The paper's question and other results are unchanged. Behaviour observed in September and November differed (one model, claude-opus-5, accepted most failing lenses in September and none in November; backends were not verified), so only contrasts within one run are reported (Section 7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.