- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether an LLM judge changes its verdict when a web agent reports the same result in a different way. It defines Strictness as false negatives associated with disclosing execution friction, and Gullibility as false positives associated with confident claims of completion. The authors test 86 constructed report units: 65 with correct target values and 21 with incorrect ones. Three retrospective judges pass far fewer correct reports under DISCLOSE than ASSERT, whereas the production WebJudge shows the opposite pattern (65/65 versus 56/65). On incorrect units, confident completion claims also receive undeserved passes. The paper then revisits a 240-run Online-Mind2Web memory experiment. The observed Static Ledger versus Full History gap is 25.0 points under WebJudge (11/24 versus 17/24) and 8.3 points under blind human adjudication (18/24 versus 20/24). In the authors' audit, 10 of 12 discordant tasks involve how results were reported or evidenced rather than a difference in the information retrieved.
Critical strengths
The controlled probe makes the concern about report framing concrete. I found the reversal across judges particularly informative: disclosure sharply reduces passes on correct reports for the retrospective panel, but increases them on WebJudge. That makes it difficult to treat framing sensitivity as a single penalty that could be corrected across judges. The memory experiment shows why this matters for a benchmark comparison. Human adjudication changes the observed Ledger versus Full History gap from six tasks to two, and the task audit helps locate the disagreements. I also found the discussion of test-retest agreement useful: a judge can give consistent verdicts while consistently misclassifying particular kinds of reports.
Critical weaknesses
The retrospective disclosure result still has an unresolved length confound. DISCLOSE reports are longer than ASSERT reports; the planned length-matched control did not meet its target, and the repaired control resolves this issue only for WebJudge. I am convinced the retrospective judges respond differently to these constructed conditions, but less certain how much of the reported gap is due specifically to disclosure. The case study shows that automated evaluation can change an observed comparison, but it does not establish the true performance gap between the memory policies. There are 24 tasks per arm, neither pairwise difference is statistically established, and the human results do not establish equivalence within five points. The 10 audited disagreements also include evidence and screenshot-related grading margins; they are not all instances of wording bias
Questions for authors
- Can you score the repaired length-matched ASSERT/DISCLOSE control on all 65 correct units with each retrospective judge? This is the result most likely to change my assessment of the headline disclosure effect.
- How many statements about retries or execution friction in the constructed DISCLOSE reports can be checked against the underlying trajectories? A breakdown by trace-supported versus added disclosure would help me interpret the probe.
- In the memory study, which discordant cases do you regard as report-framing effects specifically, and which depend on screenshots or other evidence available to the judge? This distinction affects how directly the case study supports the probe.