acceptodds
Under review as a conference paper at ICLR 2027

DigiDeck: Evaluation of Visual Web Agents in Synthetic Interactive Environments

Abstract

Evaluating multimodal web agents on the live web is challenging due to limited reproducibility, bot detection, and the potential for irreversible financial, privacy, or security consequences. Sandboxed environments mitigate these risks, but incur high development overhead and often fail to capture the diversity and complexity of real-world web interactions. To address this issue, we introduce DigiDeck, an automated framework for generating synthetic, interactive web environments for safe and reproducible evaluation of visual web agents. DigiDeck uses coding LLMs as web world models to predict action-conditioned state transitions in the form of browser-renderable environments, enabling interactive synthetic web trajectories without relying on live services. We demonstrate DigiDeck's utility by evaluating how agent performance varies across two dimensions: domain shifts and intent requirements. Using Online-Mind2Web as an anchor, we construct a benchmark of 1,080 tasks that systematically varies both intent (navigational, informational, and transactional) and domain distributions. Across seven agents, models show a consistent performance hierarchy across intents: performance decreases by 20.6 percentage points on average from navigational to informational tasks and by a further 17.8 points from informational to transactional tasks. On the distribution-shift task set, aggregate performance decreases by 5.2 percentage points, with substantial variation across models and a larger average informational-to-transactional gap.

Primary area:
datasets and benchmarks
Keywords:
web-agents, multimodal, synthetic data

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WbrDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DigiDeck constructs synthetic websites and combines native browser actions with generated successor states when an interaction is unsupported. It evaluates seven visual web agents on matched navigational, informational, and transactional tasks, with an additional domain-shift set. The main finding is a decline in workflow completion as the requested intent moves toward transactions, supported by a human plausibility study.

Critical strengths
  1. Combining native browser execution with generated missing interactions is a useful idea. It offers a way to evaluate workflows that would otherwise require building much more application logic into each synthetic website.
  2. Sharing the scaffold and initial page across intent variants is a useful control. It makes the navigation, information, and transaction comparisons more informative than comparing unrelated task collections with different starting websites.
  3. The decline across intents is substantial and appears for every evaluated agent on the benchmark-anchored set. Looking at whether agents finish the requested workflow, including its final action, exposes a limitation that navigation-only success can miss.
  4. The human study examines transition plausibility and consistency alongside appearance. The matched live and synthetic trajectories provide encouraging initial evidence that the generated environments can support realistic interaction, and the source-blinded annotation is a sensible choice.
Critical weaknesses
  1. Plausible screens do not establish that transactions are simulated correctly. The study does not independently check whether generated successors preserve transaction state and constraints across a workflow. A focused audit against known valid and invalid workflows would help establish that the benchmark tests the behavior it is intended to measure.
  2. The success judge sometimes requires a particular procedure, such as sorting to answer a superlative, even when another strategy could be correct. Its agreement with humans also varies by intent. I am concerned that procedure-dependent labels could contribute to the central intent gaps; an independent outcome-label audit is needed.
  3. The live-versus-synthetic validation rests on twenty source contexts and trajectories from one agent. The reported success correlation is encouraging, but cannot establish whether the simulator preserves comparisons between agents with different behavior. A matched subset involving more than one policy would make the benchmark's external validity more convincing.
  4. The denominators in Table 3 need clarification. The rates and intervals for Gemini Pro and GPT-5 look consistent with fewer tasks than those for the other agents, though the resampling scheme could also explain this. Without counts and shared task coverage, the cross-model averages are difficult to assess.
Questions for authors
  1. What fraction of transitions needs generation, and what does a complete evaluation cost? A comparison with fully generated websites such as WebForge on a common subset would help explain the benefit of generating missing behavior during interaction.
  2. Could you provide task counts and the bootstrap resampling unit for Table 3? If task coverage differs across agents, please show the main comparisons on their shared tasks.
  3. Why is Code2World grouped with static generation in the introduction, and CUA-Gym with finite-state search methods in the appendix? Their action-conditioned prediction and joint generation of tasks, states, and reward functions seem relevant to positioning the hybrid contribution.
  4. Are user details fixed per task, and how repeatable are identical action prefixes when transitions are regenerated? Shared starting pages may still lead agents into different continuations before their behavior is compared.
2 comments

We thank the reviewer for the constructive feedback and for recognizing DigiDeck's hybrid design, matched-intent evaluation, and human validation. We address their concerns below.

1. Functional Correctness

We emphasize that our human evaluation assesses complete interaction trajectories, not only individual web states and transitions. Section 5 measures state-level transition plausibility and broken-element severity, alongside trajectory-level realism and consistency, explicitly evaluating whether transitions remain coherent throughout a workflow. Our source-blinded evaluation shows that synthetic trajectories achieve comparable plausibility (94.8% vs. 92.6%), realism (2.74 vs. 2.84), and consistency (2.90 vs. 2.88) relative to real trajectories.

2. Evaluator Reliability

Our evaluator is adapted from Online-Mind2Web and applied consistently across intents. To independently assess its reliability, we conducted a blind human evaluation of 144 trajectories, with two annotators judging task success before and after seeing the evaluator's verdict. Table 2 reports 80.2% human–evaluator agreement (F1: 86.5%), compared with 84.0% human–human agreement before any evaluator verdict is shown. Our additional error analysis reinforces this finding: transactional tasks have lower false-negative rates (18.8% vs. 28.2%) and false-positive rates (14.8% vs. 28.0%) than informational tasks. Thus, although procedural criteria may penalize some valid alternative solutions, the observed evaluator errors do not support disproportionately strict transactional grading as the explanation for the performance hierarchy.

3. Real–Synthetic Validation

Our 20 matched real–synthetic task pairs demonstrate positively correlated human-rated success (Pearson's r = 0.62). Combined with the source-blinded trajectory evaluation, this provides initial evidence that synthetic environments capture meaningful aspects of real-web interactions. We acknowledge that these Gemini 3.1 Pro trajectories do not establish that cross-agent comparisons transfer directly to live websites. However, DigiDeck is specifically motivated by the financial, privacy, and operational risks of evaluating consequential actions on real services. Our framework complements rather than replaces live-web evaluation.

4. Task Coverage and Statistical Comparability

Question for Authors Q2: To reduce inference costs, Gemini 3.1 Pro and GPT-5 were evaluated on 100 benchmark-anchored source-task triplets, compared with 300 for the remaining agents. These 100 triplets are shared across all seven models, and all models use the same 60 distribution-shift triplets. We compute 95% percentile bootstrap confidence intervals using 10,000 replicates and seed 42, jointly resampling complete source-task triplets to preserve the three matched intent outcomes. Each replicate's average is computed from its three intent success rates. To address unequal task coverage, we additionally recomputed results on the identical 100 source contexts for all seven models (300 intent-specific tasks per model):

ModelNav.Info.Trans.Avg.
Gemini 3.1 Pro84.059.051.064.7
GPT-588.059.039.062.0
GLM5V-turbo71.041.027.046.3
Gemini 3.1 Flash Lite77.044.034.051.7
GPT-5-mini77.047.016.046.7
Qwen3-VL 30B43.026.06.025.0
Qwen3-VL 235B52.032.07.030.3

The navigational > informational > transactional performance hierarchy persists for all seven agents on identical tasks, confirming that the central intent-dependent finding is not an artifact of unequal task coverage.

Additional Questions for Authors

Q1. Transition generation frequency and evaluation cost We audited 600 benchmark trajectories from Gemini 3.1 Pro and GPT-5, each evaluated on 100 source contexts across three intents:

PolicyWithout generationGeneration pathway used
Gemini 3.1 Pro67.01%32.99%
GPT-561.50%38.50%
Combined63.27%36.73%

Across 12,104 analyzed browser-action steps, only 21.94% required HTML synthesis or editing. The remaining generation-pathway steps restored cached states or returned no change. We additionally measured recorded next-state-generation and transition-validation token usage per policy step:

PolicyInput tokens/stepOutput tokens/stepTotal tokens/step
Gemini 3.1 Pro14,6313,93418,566
GPT-515,6363,50519,141
Policy average15,1343,72018,853

WebForge does not report end-to-end environment-generation costs and does not release the environment generation pipeline, so direct cost comparisons are difficult to collect.

Q3. Positioning relative to Code2World and CUA-Gym

We thank the reviewer for identifying an imprecise characterization. Code2World performs action-conditioned GUI state prediction through renderable code, while CUA-Gym jointly synthesizes tasks, environment states, and verifiable rewards. Our intention was to highlight DigiDeck's hybrid browser-native/on-demand generation architecture and controlled matched-intent evaluation, rather than claiming that action-conditioned generation itself is novel. We will adapt the phrasing accordingly to make it clearer.

Q4. Reproducibility and Task Prefixes

All agents share the same task-specific scaffold, initial page, generator configuration, and random seed (42). Simulated-user responses are dynamically generated conditioned on interaction history, with instructions to remain consistent with earlier answers, rather than drawn from a guaranteed fixed user profile. We distinguish shared initial conditions from deterministic transition replay: identical initial states do not guarantee identical LLM-generated continuations, even under identical action prefixes. We will clarify the user-simulation mechanism and the limits of repeatability in the revised manuscript.

We thank the reviewer again for the actionable suggestions and giving us the opportunity to respond to their concerns.

Reviewer_gam6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper generates interactive synthetic websites on demand to help evaluate visual web agents safely. Instead of implementing an entire website and its backend, DigiDeck generates the states an agent encounters during interaction. It first generates a website scaffold. A coding model produces HTML/CSS/JavaScript, rendered in a browser. Playwright executes the action. A validator checks whether the resulting transition is plausible. If the page cannot implement the action, an LLM generates the next state page. This paper introduces a benchmark to test whether visual web agents can complete tasks across different user intents and website domains in synthetic, interactive websites. The benchmark contains 1,080 tasks across 64 domains, mainly from Online-Mind2web. It reports that agents perform much worse when asked to complete transactions than when asked to navigate or retrieve information.

Critical strengths
  1. Creating synthetic web environments is certainly interesting (not novel however)
Critical weaknesses

I am not clear about the paper's motivation. Why would someone be interesed in whether agents can successfulyy interact with synthetic websites, some of which might be suboptimal and thus the root cause of poor performance. There is already a lot of work on benchmarking web agents on controlled web environments like WebArena and VisualWebArena (which are manually created for good reason).

Questions for authors

Please clearly explain the motivation for this paper. See Weaknesses above.

1 comment

We thank the reviewer for the feedback and would like to clarify DigiDeck's motivation and contribution.

1. Why DigiDeck instead of WebArena or VisualWebArena?

We agree that manually implemented sandboxes provide valuable, high-fidelity evaluation. However, they require substantial engineering effort, limiting their scalability and diversity. DigiDeck complements these benchmarks by enabling safe, reproducible evaluation across 181 websites and 64 domains, compared with 5 and 3 websites in WebArena and VisualWebArena, respectively. Importantly, DigiDeck includes 246 irreversible transactional tasks, compared with 80 and 164, allowing broader evaluation of consequential workflows without financial or privacy risks.

2. Could synthetic website quality explain poor performance?

We explicitly investigate this concern through source-blinded human evaluation (Section 5). Synthetic trajectories achieve 94.8% transition plausibility versus 92.6% for real trajectories, with comparable realism and consistency. Moreover, transactional trajectories exhibit higher plausibility than informational trajectories (95.8% vs. 89.9%), despite substantially lower agent success. Broken-element rates also do not follow the performance hierarchy. Thus, the observed gaps are unlikely to be explained solely by measured environment quality.

We believe DigiDeck offers a complementary evaluation capability that existing manually engineered benchmarks cannot easily scale to provide.

Reviewer_r2RdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DigiDeck evaluates visual web agents in synthetic environments that combine native browser execution with LLM-generated successor states. The authors construct 1,080 tasks organized into navigational, informational, and transactional triplets, and evaluate seven agents across benchmark-anchored and additional domains. Validation includes human ratings of synthetic and real trajectories and agreement with an automated success evaluator. The experiments report performance differences across intent and domain conditions.

Critical strengths
  1. The target tasks are worth evaluating. Completing transactions and other consequential workflows requires capabilities that navigation and information-retrieval benchmarks do not fully assess.
Critical weaknesses
  1. The connection between the research question and the evaluation instrument is not established. The paper asks how web agents generalize beyond existing benchmark distributions, but measures performance within a single LLM-generated web distribution. Additional live websites could directly test domain robustness for non-consequential tasks; safe transactions motivate a sandbox, but do not establish the validity of this particular simulator. The 20 matched real–synthetic tasks from one policy (§5) do not demonstrate that the reported intent and domain gaps transfer to the target workflows.

  2. Generator behavior is an uncontrolled component of task difficulty. The world model determines available information, workflow requirements, and action consequences, while the same model also validates native transitions. Its predictions are therefore central to the benchmark’s reference dynamics. Yet the paper provides no generator-sensitivity study or independent verification of task-fact preservation, invalid-input handling, and persistent state changes. Consequently, generator-specific errors or domain familiarity could be mistaken for agent weaknesses. Sharing initial pages does not resolve this uncertainty.

  3. The success metric can penalize valid solutions and distort the central comparison. Appendix B.11 requires sorting for superlatives and exact numeric filter ranges, potentially rejecting correct outcomes obtained through alternative interactions. Initial human–evaluator agreement also varies substantially by intent: 71.9% for informational versus 86.5% for navigational tasks (Table 2). The paper does not establish whether outcome-based human adjudication preserves the reported gaps. Higher agreement after revealing the evaluator’s explanation does not provide independent validation.

  4. The empirical analysis offers limited insight into the observed failures. The main finding is a difficulty ordering across task categories. However, intent transformations also change authentication requirements, information needs, workflow length, and exposure to safeguards against consequential actions. Binning observed trajectory lengths and counting HTML elements do not separate these explanations (§7). The paper consequently does not identify which underlying capabilities account for the gaps, or whether failures reflect inability, refusal, or simulator behavior. The descriptive hierarchy alone provides limited scientific insight.

Questions for authors
  1. Do the main intent and domain effects reproduce on matched real websites or independently implemented sandbox workflows, across multiple agents?

  2. How stable are the findings under a different environment generator, and what independent checks establish functional correctness of generated transitions?

  3. What are the evaluator’s false-positive and false-negative rates by intent under blind, outcome-based human adjudication? Do corrected scores preserve the main findings?

1 comment

We thank the reviewer for the detailed feedback and for recognizing the importance of evaluating consequential web workflows. We address their concerns below.

1. Research question and evaluation validity (Q1)

We would like to clarify that our goal is to systematically evaluate visual web agent performance across task requirements that are risky to directly execute on the live-web. DigiDeck aims to provide a scalable, controlled setting to do so, rather than directly predict live-web performance. However, we still provide real-synthetic validation as a sanity check in Section 5. Our source-blinded human evaluation finds comparable transition plausibility (94.8% synthetic vs. 92.6% real), realism, and consistency, with correlated human-rated success across 20 matched pairs (r = 0.62).

Q1, real-world replication: Replicating the evaluation on the live web would involve financial, and privacy risks, which is precisely what DigiDeck is designed to avoid. Our contribution is scalable, reproducible evaluation of these workflows, complementing rather than replacing real-world validation.

2. Generator behavior and functional correctness (Q2)

Q2, generator sensitivity and correctness: We explored alternative coding models, but less capable models struggled to generate functional web-pages. Stronger proprietary alternatives were viable, but incur high costs and are not reproducible. We selected GLM-5.2, an openly available, self-hosted coding model, to balance quality, cost, and reproducibility.

It is important to note that the human evaluation results in Table 1 suggest that measured environment-quality differences do not follow the observed performance hierarchy: transactional trajectories have higher plausibility than informational ones (95.8% vs. 89.9%), despite lower agent success. Major and blocking broken-element rates likewise do not follow the performance ordering.

3. Evaluator reliability (Q3)

We already include independent, blind human evaluation of task success: two annotators assessed each of 144 trajectories, balanced across intents, before seeing the evaluator's judgment or explanation. Q3, false-positive and false-negative analysis: We additionally decomposed evaluator disagreements against the initial blind human judgments:

IntentFPRFNR
Navigational22.2%12.6%
Informational28.0%28.2%
Transactional14.8%18.8%
Overall21.3%19.4%

Importantly, informational tasks exhibit a higher false-negative rate than transactional tasks (28.2% vs. 18.8%), suggesting that the evaluator does not disproportionately penalize transactional success. Our evaluator is adapted from Online-Mind2Web and applied consistently across intents. While procedural requirements may reject valid alternative solutions, the observed error patterns do not support transactional-specific evaluator strictness as the explanation for the performance hierarchy.

4. Scientific insight into failure modes

Beyond aggregate performance gaps, Section 7 provides qualitative error analysis (Figure 4) identifying failures at final consequential actions: agents fill forms without submitting them, select products without completing checkout, or reach purchase workflows without completing transactions. These examples reveal a distinction between successful intermediate navigation and actual workflow completion. Figures 5–6 further investigate trajectory length and interface complexity, showing that the observed performance hierarchy is not readily explained by these coarse measures alone. While these analyses are descriptive rather than causal, they identify actionable failure patterns and motivate evaluating end-to-end task completion rather than navigation success alone.

Again, thank you for your review, and we hope that our response clarifies your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.