acceptodds
Under review as a conference paper at ICLR 2027

FullStackArena: Evaluating Browser Agents in Dynamic and Temporally Evolving Full-Stack Environments

Abstract

Existing browser-use benchmarks largely assume a classical, static web architecture. In contrast, today's web relies heavily on dynamic, stateful, and temporal behaviours driven by asynchronous requests, persistent server communications, and real-time user interactions. Consequently, current agents struggle to handle the non-deterministic, partially observable nature of modern web applications. This exposes fundamental flaws not just in the models themselves, but in current agent architectures and prompting strategies. To address these deficiencies, we introduce FullStackArena, a novel browser-use benchmark that challenges web agents with complex, full-stack environments. FullStackArena encompasses six diverse, fully operational websites spanning online bidding, stock trading, ride-hailing, maps, social media, and video. While emulating the dynamic impact of time-sensitive events, the benchmark remains entirely self-hosted, deterministic, and reproducible. Across all 781 tasks, the four baseline models pass only 16–18% of attempts on average across our evaluation settings.

Primary area:
datasets and benchmarks
Keywords:
benchmark, web agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wZRzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper introduces FullStackArena, a benchmark for evaluating browser agents in dynamic web environments where application state can change over time and task success depends on temporal constraints. The benchmark contains six self-hosted websites and 781 tasks, and formalizes the setting using a continuous-time partially observable decision process. The authors evaluate several agents, compare their performance with a human reference, and analyze the causes of failure in temporally dependent tasks.

Critical strengths

Strengths

  1. The benchmark targets an important limitation of existing web-agent evaluations. Many browser benchmarks emphasize navigation and action selection over relatively static states, whereas real applications may change while an agent is reasoning, waiting, or interacting with other components. FullStackArena explicitly evaluates temporal dependencies and evolving application states, providing a useful testbed for studying a class of failures that conventional snapshot-based environments may underrepresent.

  2. The benchmark design is systematic and supports controlled analysis. The paper covers multiple application types, including auctions, ride-hailing, and social-media-style interactions, and evaluates agents under different waiting and temporal-information configurations. Its deterministic, self-hosted environments support reproducible comparisons, while the task-level evaluation makes it possible to inspect whether agents fail because of incorrect decisions, stale state, or timing violations.

  3. The paper provides useful human comparisons and failure diagnostics. The human reference reaches an overall success rate of 83.7%, compared with 30.7% for the best agent configuration in the reported aggregate comparison. The paper also identifies timing violations and inadequate adaptation to state changes as recurring failure modes. These results give the benchmark diagnostic value beyond simply introducing another collection of browser tasks.

Critical weaknesses

Weaknesses

  1. The methodological novelty is primarily in benchmark construction and problem formalization. The continuous-time partially observable formulation is a useful way to characterize the task setting, but the paper does not introduce a corresponding agent architecture or learning method that directly addresses the identified failures. The work would benefit from a more explicit demonstration of which capabilities or failure mechanisms cannot be adequately evaluated by existing dynamic web benchmarks, and how the proposed formulation leads to new scientific conclusions beyond lower success rates.

  2. The external validity of the benchmark remains uncertain. The benchmark consists of six self-hosted websites with controlled temporal behavior. This design is valuable for reproducibility, but its dynamics may not fully capture real services with changing backend state, concurrent users, network variability, and unanticipated events. It would be useful to establish more clearly which aspects of real-world temporal interaction are faithfully represented and which remain outside the benchmark's scope.

  3. The evidence supporting the proposed explanation of agent failures could be strengthened. The large gap between the human and agent success rates establishes that the tasks are difficult for current agents, but it does not by itself isolate temporal reasoning as the sole cause. For example, failures may also arise from planning, instruction following, state tracking, or interface grounding. More controlled experiments that selectively vary temporal demands while holding other task properties fixed would help determine how much of the performance gap is specifically attributable to temporal limitations.

Questions for authors
  1. Clarification To what extent does FullStackArena isolate temporal reasoning from general web-navigation and planning difficulty? Do the tasks have matched static or non-temporal counterparts, or are there controlled comparisons that vary temporal constraints while keeping the underlying task otherwise fixed? This distinction is important for interpreting whether the observed failures specifically demonstrate limitations in temporal reasoning rather than broader agent weaknesses.

  2. Clarification The paper reports an overall human success rate of 83.7% based on six annotators. Could the authors clarify the human evaluation protocol, including the number of attempts per task, task familiarization, time limits, and available interaction tools? Were the human and agent evaluations conducted under sufficiently comparable conditions? This information is important for interpreting the magnitude of the reported human–agent performance gap.

2 comments
AuthorsOpenReview ↗

Response to Reviewer wZRz (1/2)

Thank you for taking the time to review our work and for the thoughtful and constructive feedback. We have addressed your concerns and questions below.

W1. The methodological novelty is primarily in benchmark construction and problem formalization. The continuous-time partially observable formulation is a useful way to characterize the task setting, but the paper does not introduce a corresponding agent architecture or learning method that directly addresses the identified failures. The work would benefit from a more explicit demonstration of which capabilities or failure mechanisms cannot be adequately evaluated by existing dynamic web benchmarks, and how the proposed formulation leads to new scientific conclusions beyond lower success rates.

Thank you for the thoughtful comments. Compared with existing dynamic web benchmarks, we see our contribution as a foundation for future agent designs: the benchmark and formulation expose failures that static or turn-based benchmarks cannot reveal, because there the world waits for the agent. Agents usually know what to do but not when to do it: timing-related violations dominate failures, as a single observe–reason–act step often takes longer than the window allows. Agents also act on page state that has changed while they were deciding, and their deliberation time feeds back into the environment, for example, as dwell time for the recommender. These findings point to concrete directions, such as time-aware execution loops and re-observing the page before acting. We will make these conclusions more explicit in the revision.

W2. The external validity of the benchmark remains uncertain. The benchmark consists of six self-hosted websites with controlled temporal behavior. This design is valuable for reproducibility, but its dynamics may not fully capture real services with changing backend state, concurrent users, network variability, and unanticipated events. It would be useful to establish more clearly which aspects of real-world temporal interaction are faithfully represented and which remain outside the benchmark's scope.

Thank you for raising this point. Reproducibility and full realism inherently trade off, as in other self-hosted benchmarks such as WebArena. FullStackArena faithfully represents several key aspects of real temporal interaction: backend state that evolves on its own (e.g., prices, auction countdowns, ride quotes, and traffic-dependent ETAs), simulated concurrent users (e.g., competing bidders), asynchronous page updates, and server-side logic that responds to the agent’s behavior (e.g., dwell-based recommendations), all under a clock that keeps running while the agent decides. The benchmark assumes no network variability, and we verified during evaluation that latency remained stable and comparable across models. We will add a discussion of what the benchmark does and does not capture in the revision.

W3. The evidence supporting the proposed explanation of agent failures could be strengthened. The large gap between the human and agent success rates establishes that the tasks are difficult for current agents, but it does not by itself isolate temporal reasoning as the sole cause. For example, failures may also arise from planning, instruction following, state tracking, or interface grounding. More controlled experiments that selectively vary temporal demands while holding other task properties fixed would help determine how much of the performance gap is specifically attributable to temporal limitations.

Q1. To what extent does FullStackArena isolate temporal reasoning from general web-navigation and planning difficulty? Do the tasks have matched static or non-temporal counterparts, or are there controlled comparisons that vary temporal constraints while keeping the underlying task otherwise fixed? This distinction is important for interpreting whether the observed failures specifically demonstrate limitations in temporal reasoning rather than broader agent weaknesses.

Thank you for the question. FullStackArena includes time-independent (Type-1) tasks as a control: they use the same websites and workflows as the time-constrained tasks but impose no temporal constraint. Pooled over all models and configurations, agents succeed on 59.0% of Type-1 tasks, compared with 16.1%, 10.7%, and 6.3% on tasks with release times, deadlines, and bounded windows, respectively. On Ride-hailing, for example, success drops from 88.2% on Type-1 tasks to 1.9% on deadline tasks. Our trajectory audit is consistent with this: conventional navigation and interaction errors are less dominant than timing violations in the time-sensitive tasks. We agree that the control is not matched at the template level and that failures can still involve planning or grounding. We will clarify this and discuss template-level matched comparisons in the revision.

AuthorsOpenReview ↗

Response to Reviewer wZRz (2/2)

Q2. The paper reports an overall human success rate of 83.7% based on six annotators. Could the authors clarify the human evaluation protocol, including the number of attempts per task, task familiarization, time limits, and available interaction tools? Were the human and agent evaluations conducted under sufficiently comparable conditions? This information is important for interpreting the magnitude of the reported human–agent performance gap.

Thank you for the question. Six participants with backgrounds in computer science or related technical fields and prior experience with modern web applications completed all 781 tasks. They received the same task instructions as the agents, used the same self-hosted environments through a standard browser, and were subject to the same per-task time limits. Their trajectories were graded by the same programmatic evaluators used for the agents. We will add these details to the revision.

Reviewer_iFR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FullStackArena is a browser-agent benchmark with six self-hosted websites and 781 tasks. The websites continue changing while the agent reasons or waits, so success depends on both choosing the right action and executing it at the right time. The tasks cover trading, auctions, ride-hailing, maps, social media, and video. Four models are evaluated using the same browser-use scaffold, with variations in waiting limits and access to time information. The best configuration achieves 30.7% success, compared with 83.7% for humans. The authors examine failures involving missed time windows, excessive dwell time, and stale observations.

Critical strengths

The benchmark focuses on a useful problem: an agent can understand a webpage and choose a sensible action, yet still fail because the situation has changed before it acts. The six websites turn this into concrete tasks involving bids, bookings, and recommendation preferences. Checking backend outcomes and their timestamps is a sensible way to evaluate these tasks. The human baseline is also helpful, since low agent scores alone would not tell us whether the tasks are reasonably solvable.

Critical weaknesses
  1. The explanation of the low scores is stronger than the evidence. Section 6 concludes that current LLMs fundamentally lack an accurate sense of time. I am not convinced the experiments establish this. All models use the same scaffold, and failures can reflect poor planning, slow API responses, browser overhead, or the way actions are executed. The waiting-limit and timestamp experiments are useful, but do not separate these factors. Table 5 compares different task groups rather than the same tasks under different timing conditions. A matched-task comparison with controlled decision delays would help distinguish temporal reasoning errors from execution bottlenecks. Measuring the whole system is reasonable; attributing its failures to the model requires more care.

  2. The timing protocol needs more detail. Section 3.2 makes outcomes depend on reasoning and execution delays, but I could not determine from Appendix B.2 exactly how real elapsed time maps to simulated website time. It is also unclear how API queueing, retries, and latency variation are handled. Fixed seeds make the external data streams repeatable, but they do not fix when an agent's actions arrive. This distinction matters for tasks such as skipping an ad within the 3–6 second interval in Table 1. The paper needs a precise description of the clock implementation and some evidence about variation between runs before its reproducibility claim can be assessed.

Questions for authors
  1. How is simulated time advanced during model inference, browser execution, and explicit waits? Are API queueing delays and retries counted?

  2. Do the same tasks remain difficult when decision latency is controlled, or when a simple time-aware executor handles waiting? Such a comparison would help clarify which failures require better reasoning and which require a different execution loop.

  3. Section 5 acknowledges Gaia2, WebGames, and real-time reasoning environments. Could you give a concrete example of a failure mode that requires this full-stack setup and is not adequately tested by those settings? This would help explain what the new benchmark adds beyond broader website coverage.

1 comment
AuthorsOpenReview ↗

Response to Reviewer iFR4

Thank you for taking the time to review our work and for the thoughtful and detailed feedback. We address your concerns and questions below.

W1. The explanation of the low scores is stronger than the evidence. Section 6 concludes that current LLMs fundamentally lack an accurate sense of time. I am not convinced the experiments establish this. All models use the same scaffold, and failures can reflect poor planning, slow API responses, browser overhead, or the way actions are executed. The waiting-limit and timestamp experiments are useful, but do not separate these factors. [...] A matched-task comparison with controlled decision delays would help distinguish temporal reasoning errors from execution bottlenecks.

Q2. Do the same tasks remain difficult when decision latency is controlled, or when a simple time-aware executor handles waiting? Such a comparison would help clarify which failures require better reasoning and which require a different execution loop.

W2. The timing protocol needs more detail. Section 3.2 makes outcomes depend on reasoning and execution delays, but I could not determine from Appendix B.2 exactly how real elapsed time maps to simulated website time. It is also unclear how API queueing, retries, and latency variation are handled. Fixed seeds make the external data streams repeatable, but they do not fix when an agent's actions arrive. [...] The paper needs a precise description of the clock implementation and some evidence about variation between runs before its reproducibility claim can be assessed.

Q1. How is simulated time advanced during model inference, browser execution, and explicit waits? Are API queueing delays and retries counted?

Thank you for the careful reading. The website clock runs at the same rate as the wall clock. What makes it “simulated” is only its starting point: every run of a task begins at the same fixed website time and from the same initial state, regardless of when it is launched, which is what makes runs reproducible. Because the clock never pauses, all real elapsed time counts, including model inference, API queueing and retries, browser execution, and explicit waits. To keep comparisons fair, we verified that API response latency was comparable across all models before running the experiments.

Q3. Section 5 acknowledges Gaia2, WebGames, and real-time reasoning environments. Could you give a concrete example of a failure mode that requires this full-stack setup and is not adequately tested by those settings? This would help explain what the new benchmark adds beyond broader website coverage.

Thank you for the question. Two failure modes we observe in our trajectories require both a real browser and a server whose state keeps changing, and neither arises in Gaia2 or WebGames.

(1) Acting on stale page state. On Auction, an agent identifies the correct listing, waits for the bidding window to open, and then bids without re-reading the page. Meanwhile, other bidders have changed the ranking on the server, and the page has updated asynchronously. The decision was correct when the agent made it but is wrong when it is executed. In Gaia2, agents query app state through tool calls, so there is no rendered page that can go stale between observation and action. In WebGames, the challenges run entirely on the client, with no server state evolving behind the page.

(2) Deliberation time becoming a behavioral signal. On Social Media, the server-side recommender infers interest from how long a post stays on screen, so the agent’s reasoning time is recorded as dwell time (Figure 1). Agents cannot control this: a single observe–reason–act step often keeps a post on screen for 19–29 seconds when the task allows at most 18, and the recommender reads this as interest and shifts the feed. This requires a server that records browser-side interaction and adapts future content to it, which neither Gaia2 nor WebGames models.

Reviewer_FpJnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FULLSTACKARENA, which provides an evaluation framework for complex web services involving asynchronous state changes and temporal dependencies, in addition to the conventional request-response processing assumed by current agents. While reproducing this complex behavior, the framework simultaneously ensures the reproducibility and determinism of experiments by combining pseudorandom numbers with procedural generation. The applications included in FULLSTACKARENA are an online auction, stock trading, a street map, a ride-hailing service, social media, and a video platform with a recommendation system. The experimental results revealed that existing browser agents achieve relatively low performance on this benchmark, with end-to-end task success rates ranging from 10.5% to 24.1%.

Critical strengths
  • S1. The task of performing an action while taking time into account has not been explored in existing research. It is very practical.

  • S2. It is great that they actually built the environment themselves to ensure reproducibility. It is also beneficial to see several other efforts to ensure reproducibility.

  • S3. The experimental results were obtained using SOTA models, and I feel that the error analysis itself is conducted in detail, citing specific failure examples.

Critical weaknesses
  • W1. Insufficient differentiation from existing research: The comparison and discussion with existing research on time-constrained Web Agent tasks [1] are insufficient. After appropriately citing relevant research, it is necessary to clarify how this benchmark differs from existing research and what new challenges it presents.

  • W2. Insufficient analysis of task time constraints: If time constraints are too strict, it may be difficult to complete the task at all given the current agent’s reasoning and operation speeds. For example, in the stock trading task shown in Table 1, multiple operations must be executed at one-minute intervals, and the time constraints themselves significantly increase the difficulty. However, no statistical analysis has been presented regarding the time ranges included in the task or the time remaining until the deadline. To better understand the difficulty and characteristics of the benchmarks, it would be desirable to report the distribution of time constraints and the success rates for each time window.

  • W3. Unclear Methodology for Constructing Task Templates and Their Representativeness: Although this benchmark constructs 781 tasks from 192 templates, the criteria used to select these templates are not sufficiently explained. Furthermore, it is unclear whether the templates were created manually or generated using an LLM. Additionally, there is insufficient discussion regarding the extent to which the selected templates are representative of real-world time-dependent web interactions.

[1] Maldaner+, SentinelBench: A Benchmark for Long-Running Monitoring Agents, arXiv2026

Questions for authors

For more details, please refer to “Critical Weaknesses.” In particular, I would like the authors to clarify the difference from the existing work, add analysis of time constraints, and the details of template and task construction.

1 comment
AuthorsOpenReview ↗

Response to Reviewer FpJn

Thank you for your time and constructive feedback. We address your concerns below.

W1 (Comparison with SentinelBench). Thank you for pointing us to SentinelBench; we will cite and discuss it in the revision.

Specifically, FullStackArena introduces three additional challenges: (1) Temporal planning: agents must coordinate multiple dependent actions under distinct temporal constraints, rather than primarily detecting and responding to a triggering event. (2) Agent-coupled environment dynamics: the agent’s actions and deliberation latency can influence subsequent environment states and observations. For example, prolonged reasoning changes video dwell time and consequently backend recommendations (Figure 1), meaning identical action sequences can produce different outcomes. (3) Time-critical execution: agents must exploit transient opportunities before they disappear, a setting explicitly acknowledged as underexplored in SentinelBench (Section 5).

W2 (Time-constraint analysis). We agree this would help characterize the benchmark. In the revision, we will report the distribution of time constraints across tasks and the success rate for each time-window range.

W3 (Template construction). All templates were written manually by the authors, drawing on the tasks they most commonly perform on such websites (e.g., searching, comparing, booking, bidding, posting, and watching), so that the templates reflect everyday time-dependent web interactions. We will add these details to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.