acceptodds
Withdrawn from ICLR 2027

The Honest Agent Paradox: How LLM Judges Score a Human-Inspired Web Agent

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM judges decide most published results for web agents and are usually validated by test-retest self-consistency. We show that self-consistency cannot detect a large, systematic error, the Honest Agent Paradox: when the same verified result is reported in different words, LLM judges change their verdict, and the direction of the change depends on the judge and its prompt. We split it into Strictness (α), the false-negative rate on correct results reported with disclosed execution friction, and Gullibility (β), the false-positive rate on incorrect results reported as complete, which can offset in aggregate success rate while individual verdicts stay wrong. In a controlled probe of constructed reports whose correctness is known by construction, disclosing friction on 65 correct reports lowers the pass rate of gpt-4o, claude-3-5-sonnet and gemini-1.5-pro by 38.5 to 46.2 percentage points (p < 1e-5), and hedged wording alone costs 13.9 to 23.1 points; on 21 incorrect reports, confident assertion turns up to 38.1% of failures into passes. The production WebJudge moves the other way (disclosure 65/65, assertion 56/65, exact McNemar p = 0.00391), so rankings produced by different judges cannot be compared. We describe the agent harness behind our memory study, whose vision grounding, bounding-box snapping and trusted click cascade operate the page, and whose recovery machinery, which reports partial results with caveats, is where such disclosures originate. In a 240-run memory study on Online-Mind2Web, double-blind human adjudication shrinks an apparent 25.0 point gap between two memory architectures (11/24 vs. 17/24) to 8.3 points (18/24 vs. 20/24), and 10 of the 12 discordant tasks trace to how the reports were styled. We release the framing probe at https://anonymous.4open.science/r/honest-agent-paradox.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Agent Evaluation, Autonomous Web Agents, Evaluation Validity, Visual Grounding, Web Navigation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2CFkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether an LLM judge changes its verdict when a web agent reports the same result in a different way. It defines Strictness as false negatives associated with disclosing execution friction, and Gullibility as false positives associated with confident claims of completion. The authors test 86 constructed report units: 65 with correct target values and 21 with incorrect ones. Three retrospective judges pass far fewer correct reports under DISCLOSE than ASSERT, whereas the production WebJudge shows the opposite pattern (65/65 versus 56/65). On incorrect units, confident completion claims also receive undeserved passes. The paper then revisits a 240-run Online-Mind2Web memory experiment. The observed Static Ledger versus Full History gap is 25.0 points under WebJudge (11/24 versus 17/24) and 8.3 points under blind human adjudication (18/24 versus 20/24). In the authors' audit, 10 of 12 discordant tasks involve how results were reported or evidenced rather than a difference in the information retrieved.

Critical strengths

The controlled probe makes the concern about report framing concrete. I found the reversal across judges particularly informative: disclosure sharply reduces passes on correct reports for the retrospective panel, but increases them on WebJudge. That makes it difficult to treat framing sensitivity as a single penalty that could be corrected across judges. The memory experiment shows why this matters for a benchmark comparison. Human adjudication changes the observed Ledger versus Full History gap from six tasks to two, and the task audit helps locate the disagreements. I also found the discussion of test-retest agreement useful: a judge can give consistent verdicts while consistently misclassifying particular kinds of reports.

Critical weaknesses

The retrospective disclosure result still has an unresolved length confound. DISCLOSE reports are longer than ASSERT reports; the planned length-matched control did not meet its target, and the repaired control resolves this issue only for WebJudge. I am convinced the retrospective judges respond differently to these constructed conditions, but less certain how much of the reported gap is due specifically to disclosure. The case study shows that automated evaluation can change an observed comparison, but it does not establish the true performance gap between the memory policies. There are 24 tasks per arm, neither pairwise difference is statistically established, and the human results do not establish equivalence within five points. The 10 audited disagreements also include evidence and screenshot-related grading margins; they are not all instances of wording bias

Questions for authors
  1. Can you score the repaired length-matched ASSERT/DISCLOSE control on all 65 correct units with each retrospective judge? This is the result most likely to change my assessment of the headline disclosure effect.
  2. How many statements about retries or execution friction in the constructed DISCLOSE reports can be checked against the underlying trajectories? A breakdown by trace-supported versus added disclosure would help me interpret the probe.
  3. In the memory study, which discordant cases do you regard as report-framing effects specifically, and which depend on screenshots or other evidence available to the judge? This distinction affects how directly the case study supports the probe.
Reviewer_V5HsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that LLM judges of web agents react to how an agent frames its final report, not only to whether the result is correct, and defines two error rates for this, Strictness (α, failing correct reports that disclose execution friction) and Gullibility (β, passing incorrect reports that assert completion). On 86 constructed reports from 43 tasks, each written in four fixed framings and judged on text alone, a retrospective panel of gpt-4o, claude-3-5-sonnet and gemini-1.5-pro passes correct reports 38.5 to 46.2 points less often when they disclose friction than when they assert completion, while WebJudge with gpt-5.4-mini moves in the opposite direction. In a 24-task memory study run with the authors' own web-agent harness on Online-Mind2Web, human re-scoring shrinks a 25.0 point WebJudge gap between two memory policies to 8.3 points.

Critical strengths
  • The experimental setup and analysis are documented in detail.
  • The paper addresses an important and timely problem. Since LLM judges often determine the results of web agent benchmarks, a systematic framing bias in these judges would directly distort how agents are compared.
Critical weaknesses
  1. Hallucinated references
  • Several reference entries list authors who did not write the cited papers. For Agent Workflow Memory (arXiv:2409.07429), seven of the eight listed authors are not authors of the paper, whose actual authors are Z. Z. Wang, J. Mao, D. Fried and G. Neubig. For UI-TARS (arXiv:2501.12326), none of the ten names listed as authors 11 to 20 appears in the paper's author list. The WebArena entry lists "Yiming Yang" in place of Uri Alon. These errors suggest that parts of the bibliography were generated without verification, and all references should be checked against the original papers.
  1. Presentation
  • Figure 2 needs revision for readability. In panel (b), the red center marker covers part of the "United States" label, and all five boxes leave large blank areas. The whole panel describes click mechanics that are only loosely connected to the paper's argument about judges. Using AI tools to draw figures is fine, but the figure should be revised and reduced to the content the paper relies on.
  • In Figure 3, the y-axis labels ("Pass rate (%)") are very small. The label of panel (c) sits in the gap between panels (b) and (c), so it is unclear which panel it belongs to, and panel (b) has no y-axis label.
  • The captions of Tables 1, 2 and 3 sit almost directly on the top rule of each table, and in Table 2 the rule touches the caption text.
  1. Limited novelty and unclear motivation for the web setting
  • Sensitivity of LLM judges to hedging and to superficial cues is already documented. [1] showed that most tested LLM judges penalize answers containing epistemic markers, most strongly markers of uncertainty, while human judges were essentially unaffected, which closely matches the paper's hedging result. [2] showed that content-free responses, such as a single colon or "Thought process:", elicit false-positive verdicts from reference-based LLM judges. Neither work is cited, and the paper does not explain what its findings add beyond them.
  • It is unclear why the web setting is needed. In the main probe, short template reports are judged on text alone, mostly for simple fact lookups (e.g., the launch date of Apollo 11), and the production judge receives no screenshots or action history. Nothing in this setup is specific to web agents. The one agent-specific aspect, reports that describe execution friction, is represented by a fixed template sentence rather than by real agent reports.
  1. Arbitrary experimental design without justification
  • α and β are condition-specific false-negative and false-positive rates, each tied to one hand-picked framing (α only under DISCLOSE on correct reports, β only under ASSERT on incorrect reports). Beyond motivating them as two errors that push success rates in opposite directions, the paper does not justify this pairing. As a result, gpt-5.4-mini receives α = 0 even though it fails 9 of 65 correct reports under ASSERT. General conclusions drawn from such a heuristic setup are not convincing.
  • The four framings are fixed templates without theoretical grounding. Prior work already offers a linguistically grounded taxonomy of uncertainty expressions [3], and [1] samples such markers according to how often they appear in real LLM outputs, but the paper neither uses nor discusses this line of work. There is also no human check that the templates differ only in framing, and on incorrect units the non-ASSERT templates change the content itself, for example by stating that the task could not be completed.
  • The "production WebJudge" condition does not reflect how WebJudge works. WebJudge judges from the task, the action history, key points and key screenshots, and its authors explicitly chose not to rely on the agent's final response because it is prone to hallucination [4]. The probe instead gives the WebJudge prompt only the task and the final report, so the reported sign reversal describes a use of the prompt outside its design rather than the production judge.
  1. Outdated judge models
  • The three-model panel behind the headline numbers consists of gpt-4o, claude-3-5-sonnet and gemini-1.5-pro, all released in 2024, and the latter two were no longer available from their official APIs at submission time. The paper also does not state what input or prompt this panel received. It uses current models elsewhere (gpt-5.4-mini, gemini-3-flash-preview), and the only current judge it tests reverses the disclosure effect, so it is unclear whether the main finding holds for judges in use today.
  1. Narrow evaluation in the memory study
  • The memory study uses only the authors' own harness and compares its internal variants on 24 tasks, with one run per task and variant. It includes no well-known web agents (e.g., Browser Use, SeeAct or Operator) as baselines, although claims about how judges score web agents require evaluating representative existing agents. Even the 25.0 point WebJudge gap that the study re-examines is not statistically significant (p = 0.146).
  1. Mitigations are not validated
  • The only mitigation actually tested, an appended instruction to ignore tone, was run only on gpt-5.4-mini with the 65 correct units, a judge that does not penalize disclosure in the first place. The claim that checking the page fixes the problem rests on a retrospective note about three fixtures, which the paper itself says is not a paired test, and the paper states that a disclosure-irrelevant rubric and an extract-then-verify pipeline were not run. The recommendations therefore remain untested, and grounding evaluation in environment state is exactly what live-web benchmarks cannot easily do, which is why they use LLM judges.
  1. The released code and logs contradict the paper
  • The Reproducibility Statement says that the raw judge logs are in the repository and that make judgeval-analysis reproduces every bound and contingency table. However, the repository contains no logs for the three-model panel behind Table 2 and no task-level human labels behind Table 3. Its own analysis plan (judgeval/ANALYSIS_PLAN_W2.md) states that these files are not in the repository, and the human totals are hard-coded in judgeval/w3/report.py. The headline numbers therefore cannot be verified.
  • Eight of the 21 units labeled incorrect contain exactly the correct value, only prefixed with Estimated ~ (e.g., Estimated ~$3.85 where the correct answer is $3.85), so these units do not contain wrong values. Five of the six gullibility passes of gpt-5.4-mini come from these units.
  • Appendix E.3 states that every rewrite must pass an automated check that numbers, URLs, entity names and binary answers are preserved exactly. In the released code, the hedge-ladder rewrites behind Figure 3(c) are accepted without any check, and the logs contain rewrites that drop or alter the value (e.g., $4.21 trillion becomes $4.2).
  • The repository's eval/PROTOCOL.md states that the deviations of the memory study are disclosed in the paper (§7.5), but the paper has no such section and does not report them. These include a harness bug in the proxy setup that produced the HTTP 500 errors behind the Fox Sports and Rotten Tomatoes cases, and five task swaps made after runs existed, including the Fox Sports task of Figure 1, which replaced a task that all four arms had failed.

References

[1] Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation, Lee et al., NAACL 2025.

[2] One Token to Fool LLM-as-a-Judge, Zhao et al., arXiv 2025.

[3] Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models, Zhou et al., EMNLP 2023.

[4] An Illusion of Progress? Assessing the Current State of Web Agents, Xue et al., COLM 2025.

Questions for authors

Please check the above weaknesses.

Reviewer_T3zEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper highlights a systematic bias of web judge models used in web navigation evaluation, in that when verified results are reported in different words, LLM judges changes their verdict in unpredictable ways. This paper proposes a agent harness that alleviates this bias through a vision-based pipeline.

Critical strengths
  • highlights a useful bias in some judge models used for web navigation, though it is not clear that this bias continues to persist in modern evaluation frameworks
Critical weaknesses
  • Judge model backbones used are 2-3 model generations behind frontier, not representative of what modern CUAs used for LLM-as-a-judge evals.
  • Paper proposes a new judge with very limited experiments on existing trajectories. The contribution is far too insufficient for a main conference paper for NeurIPS. It would be much more substantial if the authors went on to show useful applications of the judge to other domains (e.g., general CUA, on OSWorld or OSWorld-v2), or that it helps to use such a judge for RLVR on web navigation or computer use.
  • Experiments are limited: subset of tasks on BU and Online-M2W, and the models used are not representative of modern CUA pipelines.
  • Paper ignores existing judge frameworks that are more sophisticated, e.g. OSWorld-v2, CUA-World, Odysseys use more sophisticated judges based on rubrics or a combination of correctness checks with LLM-as-a-judge calls
  • Section 3.1 is quite presented in an unnecessarily complicated manner and difficult to read (also seems largely AI generated)
  • References section is heavily hallucinated. Includes "Anonymous" references that have arxiv IDs, and "et al., Author" instead of "Author, et al.". Large sections of the paper also appear completely AI generated.
Questions for authors
  • Can you demonstrate the efficacy of your judge on trajectories from modern CUA evals and models? For example, running Opus 5.5 or models from the gpt-6 family on OSWorld-v2.
  • Can you compare against more recent judge frameworks, including rubric based judges and hybrid (state checks + LLM as a judge) frameworks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.