- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces LiveOutcome, a benchmark of 78 human-authored tasks on the live web whose valid outcomes are not fixed in advance and may change over time, and pairs each task with a hidden, property-based success specification instead of a reference answer. Each agent's final output is scored by deterministic constraint checks, LLM verification of its claims against the cited pages, and LLM rubric scoring. The evaluator is validated with human agreement, repeated scoring on fixed evidence and seeded grounding faults. Across 13 agent and model configurations, the best system (Codex with GPT-5.6 Sol) scores 80.6 and passes 61.5% of tasks, and most failures are attributed to incomplete or incorrect content rather than missing evidence.
Critical strengths
- The paper addresses a timely problem. As web agents are increasingly expected to complete real-world tasks on the live web, how to evaluate them is becoming more important. This paper focuses on tasks whose valid answers change over time, which many existing benchmarks avoid by relying on fixed answers or frozen web content.
Critical weaknesses
- No released data or code
- The Reproducibility Statement says the task packages, evaluation code and configurations are provided, but the submission has no link and no supplementary material.
- The paper shows only excerpts (e.g., two success criteria, the Figure 2 anchors, part of one rubric in Figure 9), so neither the tasks nor the evaluation can be checked.
- Limited novelty
- The main ingredients already exist in work the paper cites: rubric-based judging (PaperBench, ResearchRubrics, Agent-as-a-Judge), joint checks of answer correctness and source attribution (Mind2Web 2), and live tasks without fixed ground truth (LiveResearchBench).
- Its main methodological difference, splitting evaluation into three stages instead of one LLM judge, is compared only with the paper's own simplified baselines (Tables 1 and 6), not with existing evaluators such as Mind2Web 2's judge.
- A new dataset could still be a valuable contribution on its own, but since the 78 tasks are not included in the submission and are few in number, the dataset does not offset the limited methodological novelty.
- Unverifiable claims may not be penalized
- An inaccessible page yields an "unverified" finding rather than a rejection (Section 3.2), but the main text does not explain how this affects the score.
- The only hint, in the Figure 9 interface, says not to reduce a score "merely because a criterion is unverified", so an invented value cited to a blocked page may go unpenalized.
- The number of unverified claims in the main runs is not reported.
- Many altered claims may not be penalized
- In the seeded-fault test (Table 6), the score-affecting "unsupported" verdict is given to only 68.3% of value perturbations and 63.9% of entity substitutions. The rest are marked partial (effect unspecified) or not flagged at all (14.2% and 9.2%).
- The grounding checker agrees with humans less than humans agree with each other (κ 0.695 vs. 0.850), and the two did not judge the same version of each page.
- Insufficient description of task creation
- The paper does not say who wrote the tasks, with what expertise, where the task ideas came from, how many candidates were discarded, or which agents were used in the pilot runs. It also does not explicitly state whether LLMs helped draft the criteria or rubrics, or who the human annotators were.
- Figure 2 states that top-score rubric requirements "go beyond the user request", so agents are partly graded on hidden requirements that the request does not state.
- Small benchmark size
- With 78 tasks, the benchmark is smaller than the live-web benchmarks it cites (e.g., Online-Mind2Web 300, Odysseys 200, Mind2Web 2 130).
- Categories have only 11 to 25 tasks and each agent runs once per task, yet category-level leaders are reported without confidence intervals, although the leads are only 0.3 and 0.8 points.
- Limited discussion of sandboxed benchmarks
- Sandboxed environments are covered in only a few sentences (WebArena, OSWorld), and benchmarks such as VisualWebArena and WorkArena are not cited, although state-based checks in a sandbox could also accept multiple valid outcomes.
- The paper does not say whether the 13 configurations were run on the same dates, so it is unclear whether they faced the same web state.
- No guarantee that tasks remain solvable
- The paper states that the set of valid outcomes "evolves with the state of the web", but task validation only checks whether each request "could be represented and evaluated", not whether a valid outcome still exists at run time.
- It does not explain in general how a run is scored when no valid outcome exists, so low scores may partly reflect unsolvable tasks.
- No website selection criteria or access-policy discussion
- Several tasks require specific sites (e.g., Naver, YouTube), but the paper gives no list of such sites and no selection criteria.
- Beyond advising reproductions to respect site policies, it does not state whether its own runs complied with robots.txt and terms of use, even though its Ethics Statement warns against bypassing verification barriers and Figure 5 shows an agent clearing a bot check on google.com.
Questions for authors
Please check the above weaknesses.