- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
This paper introduces FullStackArena, a benchmark for evaluating browser agents in dynamic web environments where application state can change over time and task success depends on temporal constraints. The benchmark contains six self-hosted websites and 781 tasks, and formalizes the setting using a continuous-time partially observable decision process. The authors evaluate several agents, compare their performance with a human reference, and analyze the causes of failure in temporally dependent tasks.
Critical strengths
Strengths
-
The benchmark targets an important limitation of existing web-agent evaluations. Many browser benchmarks emphasize navigation and action selection over relatively static states, whereas real applications may change while an agent is reasoning, waiting, or interacting with other components. FullStackArena explicitly evaluates temporal dependencies and evolving application states, providing a useful testbed for studying a class of failures that conventional snapshot-based environments may underrepresent.
-
The benchmark design is systematic and supports controlled analysis. The paper covers multiple application types, including auctions, ride-hailing, and social-media-style interactions, and evaluates agents under different waiting and temporal-information configurations. Its deterministic, self-hosted environments support reproducible comparisons, while the task-level evaluation makes it possible to inspect whether agents fail because of incorrect decisions, stale state, or timing violations.
-
The paper provides useful human comparisons and failure diagnostics. The human reference reaches an overall success rate of 83.7%, compared with 30.7% for the best agent configuration in the reported aggregate comparison. The paper also identifies timing violations and inadequate adaptation to state changes as recurring failure modes. These results give the benchmark diagnostic value beyond simply introducing another collection of browser tasks.
Critical weaknesses
Weaknesses
-
The methodological novelty is primarily in benchmark construction and problem formalization. The continuous-time partially observable formulation is a useful way to characterize the task setting, but the paper does not introduce a corresponding agent architecture or learning method that directly addresses the identified failures. The work would benefit from a more explicit demonstration of which capabilities or failure mechanisms cannot be adequately evaluated by existing dynamic web benchmarks, and how the proposed formulation leads to new scientific conclusions beyond lower success rates.
-
The external validity of the benchmark remains uncertain. The benchmark consists of six self-hosted websites with controlled temporal behavior. This design is valuable for reproducibility, but its dynamics may not fully capture real services with changing backend state, concurrent users, network variability, and unanticipated events. It would be useful to establish more clearly which aspects of real-world temporal interaction are faithfully represented and which remain outside the benchmark's scope.
-
The evidence supporting the proposed explanation of agent failures could be strengthened. The large gap between the human and agent success rates establishes that the tasks are difficult for current agents, but it does not by itself isolate temporal reasoning as the sole cause. For example, failures may also arise from planning, instruction following, state tracking, or interface grounding. More controlled experiments that selectively vary temporal demands while holding other task properties fixed would help determine how much of the performance gap is specifically attributable to temporal limitations.
Questions for authors
-
Clarification To what extent does FullStackArena isolate temporal reasoning from general web-navigation and planning difficulty? Do the tasks have matched static or non-temporal counterparts, or are there controlled comparisons that vary temporal constraints while keeping the underlying task otherwise fixed? This distinction is important for interpreting whether the observed failures specifically demonstrate limitations in temporal reasoning rather than broader agent weaknesses.
-
Clarification The paper reports an overall human success rate of 83.7% based on six annotators. Could the authors clarify the human evaluation protocol, including the number of attempts per task, task familiarization, time limits, and available interaction tools? Were the human and agent evaluations conducted under sufficiently comparable conditions? This information is important for interpreting the magnitude of the reported human–agent performance gap.
2 comments
Response to Reviewer wZRz (2/2)
Q2. The paper reports an overall human success rate of 83.7% based on six annotators. Could the authors clarify the human evaluation protocol, including the number of attempts per task, task familiarization, time limits, and available interaction tools? Were the human and agent evaluations conducted under sufficiently comparable conditions? This information is important for interpreting the magnitude of the reported human–agent performance gap.
Thank you for the question. Six participants with backgrounds in computer science or related technical fields and prior experience with modern web applications completed all 781 tasks. They received the same task instructions as the agents, used the same self-hosted environments through a standard browser, and were subject to the same per-task time limits. Their trajectories were graded by the same programmatic evaluators used for the agents. We will add these details to the revision.
Response to Reviewer wZRz (1/2)
Thank you for taking the time to review our work and for the thoughtful and constructive feedback. We have addressed your concerns and questions below.
Thank you for the thoughtful comments. Compared with existing dynamic web benchmarks, we see our contribution as a foundation for future agent designs: the benchmark and formulation expose failures that static or turn-based benchmarks cannot reveal, because there the world waits for the agent. Agents usually know what to do but not when to do it: timing-related violations dominate failures, as a single observe–reason–act step often takes longer than the window allows. Agents also act on page state that has changed while they were deciding, and their deliberation time feeds back into the environment, for example, as dwell time for the recommender. These findings point to concrete directions, such as time-aware execution loops and re-observing the page before acting. We will make these conclusions more explicit in the revision.
Thank you for raising this point. Reproducibility and full realism inherently trade off, as in other self-hosted benchmarks such as WebArena. FullStackArena faithfully represents several key aspects of real temporal interaction: backend state that evolves on its own (e.g., prices, auction countdowns, ride quotes, and traffic-dependent ETAs), simulated concurrent users (e.g., competing bidders), asynchronous page updates, and server-side logic that responds to the agent’s behavior (e.g., dwell-based recommendations), all under a clock that keeps running while the agent decides. The benchmark assumes no network variability, and we verified during evaluation that latency remained stable and comparable across models. We will add a discussion of what the benchmark does and does not capture in the revision.
Thank you for the question. FullStackArena includes time-independent (Type-1) tasks as a control: they use the same websites and workflows as the time-constrained tasks but impose no temporal constraint. Pooled over all models and configurations, agents succeed on 59.0% of Type-1 tasks, compared with 16.1%, 10.7%, and 6.3% on tasks with release times, deadlines, and bounded windows, respectively. On Ride-hailing, for example, success drops from 88.2% on Type-1 tasks to 1.9% on deadline tasks. Our trajectory audit is consistent with this: conventional navigation and interaction errors are less dominant than timing violations in the time-sensitive tasks. We agree that the control is not matched at the template level and that failures can still involve planning or grounding. We will clarify this and discuss template-level matched comparisons in the revision.