- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DigiDeck constructs synthetic websites and combines native browser actions with generated successor states when an interaction is unsupported. It evaluates seven visual web agents on matched navigational, informational, and transactional tasks, with an additional domain-shift set. The main finding is a decline in workflow completion as the requested intent moves toward transactions, supported by a human plausibility study.
Critical strengths
- Combining native browser execution with generated missing interactions is a useful idea. It offers a way to evaluate workflows that would otherwise require building much more application logic into each synthetic website.
- Sharing the scaffold and initial page across intent variants is a useful control. It makes the navigation, information, and transaction comparisons more informative than comparing unrelated task collections with different starting websites.
- The decline across intents is substantial and appears for every evaluated agent on the benchmark-anchored set. Looking at whether agents finish the requested workflow, including its final action, exposes a limitation that navigation-only success can miss.
- The human study examines transition plausibility and consistency alongside appearance. The matched live and synthetic trajectories provide encouraging initial evidence that the generated environments can support realistic interaction, and the source-blinded annotation is a sensible choice.
Critical weaknesses
- Plausible screens do not establish that transactions are simulated correctly. The study does not independently check whether generated successors preserve transaction state and constraints across a workflow. A focused audit against known valid and invalid workflows would help establish that the benchmark tests the behavior it is intended to measure.
- The success judge sometimes requires a particular procedure, such as sorting to answer a superlative, even when another strategy could be correct. Its agreement with humans also varies by intent. I am concerned that procedure-dependent labels could contribute to the central intent gaps; an independent outcome-label audit is needed.
- The live-versus-synthetic validation rests on twenty source contexts and trajectories from one agent. The reported success correlation is encouraging, but cannot establish whether the simulator preserves comparisons between agents with different behavior. A matched subset involving more than one policy would make the benchmark's external validity more convincing.
- The denominators in Table 3 need clarification. The rates and intervals for Gemini Pro and GPT-5 look consistent with fewer tasks than those for the other agents, though the resampling scheme could also explain this. Without counts and shared task coverage, the cross-model averages are difficult to assess.
Questions for authors
- What fraction of transitions needs generation, and what does a complete evaluation cost? A comparison with fully generated websites such as WebForge on a common subset would help explain the benefit of generating missing behavior during interaction.
- Could you provide task counts and the bootstrap resampling unit for Table 3? If task coverage differs across agents, please show the main comparisons on their shared tasks.
- Why is Code2World grouped with static generation in the introduction, and CUA-Gym with finite-state search methods in the appendix? Their action-conditioned prediction and joint generation of tasks, states, and reward functions seem relevant to positioning the hybrid contribution.
- Are user details fixed per task, and how repeatable are identical action prefixes when transitions are regenerated? Shared starting pages may still lead agents into different continuations before their behavior is compared.
2 comments
Additional Questions for Authors
Q1. Transition generation frequency and evaluation cost We audited 600 benchmark trajectories from Gemini 3.1 Pro and GPT-5, each evaluated on 100 source contexts across three intents:
| Policy | Without generation | Generation pathway used |
|---|---|---|
| Gemini 3.1 Pro | 67.01% | 32.99% |
| GPT-5 | 61.50% | 38.50% |
| Combined | 63.27% | 36.73% |
Across 12,104 analyzed browser-action steps, only 21.94% required HTML synthesis or editing. The remaining generation-pathway steps restored cached states or returned no change. We additionally measured recorded next-state-generation and transition-validation token usage per policy step:
| Policy | Input tokens/step | Output tokens/step | Total tokens/step |
|---|---|---|---|
| Gemini 3.1 Pro | 14,631 | 3,934 | 18,566 |
| GPT-5 | 15,636 | 3,505 | 19,141 |
| Policy average | 15,134 | 3,720 | 18,853 |
WebForge does not report end-to-end environment-generation costs and does not release the environment generation pipeline, so direct cost comparisons are difficult to collect.
Q3. Positioning relative to Code2World and CUA-Gym
We thank the reviewer for identifying an imprecise characterization. Code2World performs action-conditioned GUI state prediction through renderable code, while CUA-Gym jointly synthesizes tasks, environment states, and verifiable rewards. Our intention was to highlight DigiDeck's hybrid browser-native/on-demand generation architecture and controlled matched-intent evaluation, rather than claiming that action-conditioned generation itself is novel. We will adapt the phrasing accordingly to make it clearer.
Q4. Reproducibility and Task Prefixes
All agents share the same task-specific scaffold, initial page, generator configuration, and random seed (42). Simulated-user responses are dynamically generated conditioned on interaction history, with instructions to remain consistent with earlier answers, rather than drawn from a guaranteed fixed user profile. We distinguish shared initial conditions from deterministic transition replay: identical initial states do not guarantee identical LLM-generated continuations, even under identical action prefixes. We will clarify the user-simulation mechanism and the limits of repeatability in the revised manuscript.
We thank the reviewer again for the actionable suggestions and giving us the opportunity to respond to their concerns.
We thank the reviewer for the constructive feedback and for recognizing DigiDeck's hybrid design, matched-intent evaluation, and human validation. We address their concerns below.
1. Functional Correctness
We emphasize that our human evaluation assesses complete interaction trajectories, not only individual web states and transitions. Section 5 measures state-level transition plausibility and broken-element severity, alongside trajectory-level realism and consistency, explicitly evaluating whether transitions remain coherent throughout a workflow. Our source-blinded evaluation shows that synthetic trajectories achieve comparable plausibility (94.8% vs. 92.6%), realism (2.74 vs. 2.84), and consistency (2.90 vs. 2.88) relative to real trajectories.
2. Evaluator Reliability
Our evaluator is adapted from Online-Mind2Web and applied consistently across intents. To independently assess its reliability, we conducted a blind human evaluation of 144 trajectories, with two annotators judging task success before and after seeing the evaluator's verdict. Table 2 reports 80.2% human–evaluator agreement (F1: 86.5%), compared with 84.0% human–human agreement before any evaluator verdict is shown. Our additional error analysis reinforces this finding: transactional tasks have lower false-negative rates (18.8% vs. 28.2%) and false-positive rates (14.8% vs. 28.0%) than informational tasks. Thus, although procedural criteria may penalize some valid alternative solutions, the observed evaluator errors do not support disproportionately strict transactional grading as the explanation for the performance hierarchy.
3. Real–Synthetic Validation
Our 20 matched real–synthetic task pairs demonstrate positively correlated human-rated success (Pearson's r = 0.62). Combined with the source-blinded trajectory evaluation, this provides initial evidence that synthetic environments capture meaningful aspects of real-web interactions. We acknowledge that these Gemini 3.1 Pro trajectories do not establish that cross-agent comparisons transfer directly to live websites. However, DigiDeck is specifically motivated by the financial, privacy, and operational risks of evaluating consequential actions on real services. Our framework complements rather than replaces live-web evaluation.
4. Task Coverage and Statistical Comparability
Question for Authors Q2: To reduce inference costs, Gemini 3.1 Pro and GPT-5 were evaluated on 100 benchmark-anchored source-task triplets, compared with 300 for the remaining agents. These 100 triplets are shared across all seven models, and all models use the same 60 distribution-shift triplets. We compute 95% percentile bootstrap confidence intervals using 10,000 replicates and seed 42, jointly resampling complete source-task triplets to preserve the three matched intent outcomes. Each replicate's average is computed from its three intent success rates. To address unequal task coverage, we additionally recomputed results on the identical 100 source contexts for all seven models (300 intent-specific tasks per model):
The navigational > informational > transactional performance hierarchy persists for all seven agents on identical tasks, confirming that the central intent-dependent finding is not an artifact of unequal task coverage.