acceptodds
Withdrawn from ICLR 2027

Evaluating Agents in Live, User-Facing Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

AI agents are increasingly expected to complete complex, real-world tasks on the live web, where success depends not only on open-ended reasoning but also on information that changes over time. Products fluctuate in price and availability, listings appear and disappear, deadlines pass, and policies are revised. Consequently, many web tasks have dynamic, open outcome spaces: they admit multiple substantively different valid outcomes, and the set of valid outcomes itself evolves with the state of the web. Existing benchmarks typically simplify one or both dimensions, or leave both to a single judge. They may allow agents to choose their own solution paths while evaluating against a pre-authored answer or canonical final state, or preserve reproducibility by freezing web content and avoiding time-sensitive tasks, or grade live-web outcomes with one LLM judge that handles constraints, evidence, and quality together. We introduce LIVEOUTCOME, a benchmark of 78 human-authored, user-facing tasks for evaluating agents against the current state of the web. Its tasks span research, comparison, planning, decision support, data collection, and artifact creation, with outcomes that may change across days, weeks, or months. Each task is paired with a hidden, property-based success specification that remains stable over time without enumerating a fixed answer. LIVEOUTCOME evaluates submissions through deterministic checks of explicit constraints, live retrieval and verification of source-dependent claims against the pages the agent cites, and rubric-based assessment of the resulting deliverable that incorporates the verified findings. We benchmark current agents and validate the evaluation protocol using human judgments of both LLM stages, repeated evaluation under fixed evidence, and controlled grounding failures with unmodified controls. Across 13 agent–model configurations, the strongest system scores 80.6 and passes 61.5% of tasks, and most failures stem from incomplete or incor- rect content rather than missing evidence. By separating stable success criteria from a changing set of web-grounded outcomes, LIVEOUTCOME enables rigorous evaluation of whether agents can solve open-ended tasks as the web exists at execution time.

Primary area:
datasets and benchmarks
Keywords:
benchmark, live web, user-facing, evaluation, open-ended

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wV4iOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LiveOutcome, a benchmark of 78 human-authored tasks on the live web whose valid outcomes are not fixed in advance and may change over time, and pairs each task with a hidden, property-based success specification instead of a reference answer. Each agent's final output is scored by deterministic constraint checks, LLM verification of its claims against the cited pages, and LLM rubric scoring. The evaluator is validated with human agreement, repeated scoring on fixed evidence and seeded grounding faults. Across 13 agent and model configurations, the best system (Codex with GPT-5.6 Sol) scores 80.6 and passes 61.5% of tasks, and most failures are attributed to incomplete or incorrect content rather than missing evidence.

Critical strengths
  • The paper addresses a timely problem. As web agents are increasingly expected to complete real-world tasks on the live web, how to evaluate them is becoming more important. This paper focuses on tasks whose valid answers change over time, which many existing benchmarks avoid by relying on fixed answers or frozen web content.
Critical weaknesses
  1. No released data or code
  • The Reproducibility Statement says the task packages, evaluation code and configurations are provided, but the submission has no link and no supplementary material.
  • The paper shows only excerpts (e.g., two success criteria, the Figure 2 anchors, part of one rubric in Figure 9), so neither the tasks nor the evaluation can be checked.
  1. Limited novelty
  • The main ingredients already exist in work the paper cites: rubric-based judging (PaperBench, ResearchRubrics, Agent-as-a-Judge), joint checks of answer correctness and source attribution (Mind2Web 2), and live tasks without fixed ground truth (LiveResearchBench).
  • Its main methodological difference, splitting evaluation into three stages instead of one LLM judge, is compared only with the paper's own simplified baselines (Tables 1 and 6), not with existing evaluators such as Mind2Web 2's judge.
  • A new dataset could still be a valuable contribution on its own, but since the 78 tasks are not included in the submission and are few in number, the dataset does not offset the limited methodological novelty.
  1. Unverifiable claims may not be penalized
  • An inaccessible page yields an "unverified" finding rather than a rejection (Section 3.2), but the main text does not explain how this affects the score.
  • The only hint, in the Figure 9 interface, says not to reduce a score "merely because a criterion is unverified", so an invented value cited to a blocked page may go unpenalized.
  • The number of unverified claims in the main runs is not reported.
  1. Many altered claims may not be penalized
  • In the seeded-fault test (Table 6), the score-affecting "unsupported" verdict is given to only 68.3% of value perturbations and 63.9% of entity substitutions. The rest are marked partial (effect unspecified) or not flagged at all (14.2% and 9.2%).
  • The grounding checker agrees with humans less than humans agree with each other (κ 0.695 vs. 0.850), and the two did not judge the same version of each page.
  1. Insufficient description of task creation
  • The paper does not say who wrote the tasks, with what expertise, where the task ideas came from, how many candidates were discarded, or which agents were used in the pilot runs. It also does not explicitly state whether LLMs helped draft the criteria or rubrics, or who the human annotators were.
  • Figure 2 states that top-score rubric requirements "go beyond the user request", so agents are partly graded on hidden requirements that the request does not state.
  1. Small benchmark size
  • With 78 tasks, the benchmark is smaller than the live-web benchmarks it cites (e.g., Online-Mind2Web 300, Odysseys 200, Mind2Web 2 130).
  • Categories have only 11 to 25 tasks and each agent runs once per task, yet category-level leaders are reported without confidence intervals, although the leads are only 0.3 and 0.8 points.
  1. Limited discussion of sandboxed benchmarks
  • Sandboxed environments are covered in only a few sentences (WebArena, OSWorld), and benchmarks such as VisualWebArena and WorkArena are not cited, although state-based checks in a sandbox could also accept multiple valid outcomes.
  • The paper does not say whether the 13 configurations were run on the same dates, so it is unclear whether they faced the same web state.
  1. No guarantee that tasks remain solvable
  • The paper states that the set of valid outcomes "evolves with the state of the web", but task validation only checks whether each request "could be represented and evaluated", not whether a valid outcome still exists at run time.
  • It does not explain in general how a run is scored when no valid outcome exists, so low scores may partly reflect unsolvable tasks.
  1. No website selection criteria or access-policy discussion
  • Several tasks require specific sites (e.g., Naver, YouTube), but the paper gives no list of such sites and no selection criteria.
  • Beyond advising reproductions to respect site policies, it does not state whether its own runs complied with robots.txt and terms of use, even though its Ethics Statement warns against bypassing verification barriers and Figure 5 shows an agent clearing a bot check on google.com.
Questions for authors

Please check the above weaknesses.

Reviewer_eAoUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new web navigation benchmark, LiveOutcome, which has 78 human authored tasks for evaluating web navigation agents. LiveOutcome tasks have outcomes that may change as the web evolves, and the benchmark measures the performance through a hidden success specification that remains stable over time, as well as verification of the live results (which varies with the existing state of the live web).

Critical strengths
  • The paper highlights a salient issue with existing web navigation benchmarks. Most existing benchmarks have tasks with dynamic, open ended outcomes. These should be treated according to the status of the current live web (which may differ during evaluation time compared to when the benchmark was initially received).
Critical weaknesses
  • The primary claim of the paper is that LiveOutcome is necessary because existing benchmarks are insufficient because their either only have stable success criteria (that do not change with the web) or judge web-grounded outcomes (the latter may shift with the live web). But there does not seem to be quantitative evidence demonstrating that this is a problem with existing benchmarks, or how frequently/quickly benchmark tasks are invalidated as a result of this.
  • The benchmark at present already appears saturated (gpt-5.6-sol achieves a rubric score of 80.6 with a pass rate of 61.5%). It would be useful to benchmark the latest gpt-6-astra model, to determine if the tasks are already resolved (my guess is that gpt-6-astra would get a pass rate of 90% or more). This does not necessarily invalidate the benchmark, but it is a useful signal that we would need to determine if harder tasks of this nature are required.
  • 78 tasks is too few: web navigation benchmarks typically much more tasks. Online-Mind2Web has 300, WebArena has 800+, and Odysseys has 200 (which are long horizon and more expensive). LiveOutcome would be more valuable if the task set was diversified and expanded by 2-3x to be about as large as these.
  • Related to the above, can the authors quantify the amount of noise or SD in the rubric and pass rate (similar to how it's shown for runtime mean)? This would give a ballpark for how many tasks is sufficient for being able to statistically significantly track models.
Questions for authors
  • Do you have quantitative metrics for how severely existing benchmarks suffer from the issues that LiveOutcome claims to address? Specifically, the relevance of having stable success criteria, or the percentage of existing benchmark tasks that become outdated/invalidated.
  • Can the authors measure the performance of recent frontier models, such as gpt-6-astra or opus 5.5?
  • Could the authors quantify the amount of noise or SD in the rubric and pass rate (similar to how it's shown for runtime mean)?
Reviewer_afPkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LiveOutcome, a benchmark for evaluating agents on live-web tasks where multiple outcomes may satisfy the same user request. It comprises 78 human-authored tasks across eight domains, evaluated through deterministic constraint checks, source-grounded verification, and rubric-based scoring. The authors validate the evaluation protocol through human judgments, repeated scoring, and controlled grounding failures. Across 13 agent–model configurations, the strongest system achieves a score of 80.6 and a pass rate of 61.5%, while the failure analysis identifies content and deliverable deficiencies as major bottlenecks.

Critical strengths
  1. The paper addresses an important limitation of fixed-answer benchmarks by allowing agents to produce different valid outcomes under predefined success criteria. This formulation better reflects realistic user-facing tasks and provides a useful direction for agent benchmarking.

  2. The framework combines deterministic checks, source grounding, and component-level rubrics. The authors also provide human agreement studies, evaluator stability comparisons, and seeded-fault analyses. These experiments offer useful evidence of the framework's feasibility and consistency.

Critical weaknesses
  1. Section 4.2 evaluates agreement between human annotators and automated judges using predefined rubrics and evidence. However, this does not independently establish whether the final scores correctly distinguish successful from unsuccessful user outcomes. Since reliable outcome evaluation is the central contribution, independent human validation of task-level success decisions is important.

  2. Figure 2 explicitly presents score-5 rubric requirements beyond the original user requests. While some additional expectations may be reasonable, it remains unclear how they are justified and whether valid outcomes could be penalized for not satisfying evaluator-imposed preferences. The authors should clarify the criteria construction process and its alignment with user requirements.

  3. Figure 3 illustrates discrepancies between agent outputs and information retrieved one week later. However, such discrepancies may reflect legitimate changes in web content rather than incorrect original outcomes. The relationship between these delayed evaluations and the main evaluation protocol should be clarified, particularly how correctness is anchored to execution-time evidence.

  4. Section 5.2 and Figure 4 report that content and deliverable deficiencies dominate failures, but these categories are assigned by an LLM judge without representative human validation. Appendix D.3 provides manually verified examples, including corrected automatic labels, but does not establish overall classification reliability. Moreover, Table 2 implies 718 non-passing runs, whereas only 687 are analyzed in Figure 4. The authors should explain the exclusions and validate the robustness of the reported distributions.

Questions for authors
  1. Has the benchmark been validated against independent human judgments of overall user-task success, rather than judgments conditioned on predefined rubrics? How often do independently acceptable outcomes receive low scores, particularly due to additional requirements such as those in Figure 2?

  2. How does the evaluator distinguish genuine agent errors from changes in webpage content after execution? Are the delayed evaluations in Figure 3 separate from the main leaderboard, and what temporal reference determines correctness?

  3. Why does the failure analysis include 687 outcomes rather than the 718 non-passing runs implied by Table 2? Is there representative human validation of the failure labels, and how sensitive are the conclusions in Figure 4 to classification errors or excluded cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.