- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
WebAppBench evaluates whether agent-generated web applications satisfy their functional requirements. It contains 100 bilingual specifications and 3,197 atomic checks covering behaviors such as state transitions, value correctness, persistence, and multi-session interactions. A browser-based agent gathers runtime evidence for each check, producing supported, contradicted, or insufficient-evidence verdicts, with optional VLM review. Experiments across ten generation configurations reveal substantial gaps between interface completeness and behavioral correctness. Human validation shows strong agreement on jointly resolved checks.
Critical strengths
The benchmark targets an important gap in evaluating generated software. The focus on connected workflows, persistent state, and cross-view consistency captures failures that build success and polished screenshots cannot establish. These requirements make the benchmark useful for assessing whether generated applications support their intended use.
Explicit verification conditions make the evaluation interpretable. Each judgment is tied to a concrete requirement and recorded evidence, making failures easier to inspect than an overall quality score. Distinguishing insufficient evidence from observed failure is particularly valuable, even though the final scoring does not fully preserve this distinction.
The paper examines evaluator reliability alongside generator performance. Human review, cross-evaluator comparisons, component ablations, and repeated-generation experiments provide a more informative assessment than a leaderboard alone. The finding that incomplete verification is more common than disagreement on definitive judgments helps identify where the evaluation pipeline needs improvement.
Critical weaknesses
The incremental contribution is insufficiently established. Existing benchmarks, including ViBench and Vibe Code Bench, already evaluate functional workflows and multi-user behavior. Table 1 compares atomic checks against entire workflows/test plans, which contain multiple verification steps. These counts do not establish broader or finer coverage.
The protocol comparisons are not sufficiently controlled. In §6.3 and Appendix C.3, the compared settings differ in requirement access, evaluation granularity, and scoring format; CUA-explore receives only the application URL. A comparison using the same applications, requirements, scoring criteria, and comparable budgets is needed to isolate the benefit of the proposed protocol.
The supplementary checks contain substantive quality issues. S1-011/AC-32 requires P95 to differ from the maximum or mean, although valid samples can make them equal. Conversely, S1-047/AC-36 accepts download initiation or an .otf filename without verifying font validity. Such criteria risk rejecting correct implementations and rewarding superficial ones. The small AC-alignment study does not establish dataset-wide correctness.
Evaluator coverage still affects the reported scores. The 98.8% human agreement is conditional on jointly resolved checks, while coverage is 79.5%. Unresolved checks nevertheless receive zero credit. Together with the reported evaluator and generation variability, this leaves the stability of small leaderboard differences uncertain.
Questions for authors
Were requirement access and interaction budgets matched across protocol comparisons?
How were the identified AC issues audited, and what evidence supports ranking stability when unresolved checks are considered separately?
5 comments
Response to Reviewer aUpM — W2Q1: Protocol comparison and control
Thank you for raising the need to distinguish controlled protocol ablations from open-ended evaluation baselines. Section 6.3 contains two comparison groups with different purposes: the three AC-based settings examine the contributions of verification guidance and VLM Evidence Review under fixed requirements and scoring, while the two open-ended settings examine autonomous exploration with and without task guidance. We clarify the experimental controls and interpretation below.
1. Matched requirements, scoring, and budgets in AC-based ablations
All three AC-based settings evaluate the same 18 applications against identical ACs and verification conditions, use the same criticality-weighted FuncScore, and retain the same CUA and per-type iteration caps (§5.1; Appendix C.3). These caps are 8/12/30/25/10/15/18 for structural, navigation, state-transition, value-correctness, perceptual, persistence, and runtime-integrity checks, respectively.
The guidance and evidence-review components are varied incrementally:
| Setting | Controlled change | FuncScore (§6.3) |
|---|---|---|
| ACs only | ACs and verification conditions, without task-level guidelines or VLM review | 44.24 |
| ACs + Guidelines | Adds navigation, prerequisite, and workflow guidance | 45.50 |
| ACs + Guidelines + VLM | Adds review of recorded evidence, without further browser interaction | 54.68 |
These comparisons hold the acceptance targets, scoring rules, and execution limits fixed. The resulting FuncScore changes characterize the contributions of guidance and evidence review within the same AC-based evaluation protocol. Importantly, VLM review reassesses existing evidence rather than granting additional browser interactions.
2. Controlled conditions for open-ended baselines
Guidelines-only and CUA-explore use the same applications, base system prompt, browser tools, and max_steps=700 limit per application, with early termination permitted. Their intended difference is the availability of task-specific information: Guidelines-only receives the task guidelines, whereas CUA-explore receives only the application URL. Neither uses a predefined AC checklist or per-AC verdicts; both select inspection targets autonomously and return a holistic score between 0 and 100.
All settings access deployed applications through browser URLs. Here, “URL-only” means that no additional task-specific requirements or guidance are supplied, rather than indicating a different application-access mechanism. An unguided CUA must infer both what functionality to examine and how to verify it, potentially introducing omissions or incomplete judgments. Our main protocol therefore provides explicit AC targets and verification guidelines to guide execution, alongside VLM review to reassess uncertain judgments using recorded evidence. The AC-based ablations separately examine these components.
3. Interpretation of the two comparison groups
The open-ended baselines characterize autonomous evaluation behavior rather than provide directly comparable FuncScores. Guidelines-only and CUA-explore report holistic scores of 71.00 and 77.13, respectively, but inspect approximately ten fewer checks per application than the AC-based protocol (§6.3). Because their requirements are self-selected and their scoring formats differ, these nominal scores cannot be interpreted as evidence of superior requirement satisfaction.
Accordingly, our conclusions about the contributions of guidance and VLM review are grounded in the matched AC-based ablations, while the open-ended settings illustrate how autonomous check selection can affect evaluation coverage and interpretation. We have clarified these differences, together with the matched inputs, scoring criteria, and execution budgets, in §6.3 and Appendix C.3.
Response to Reviewer aUpM — W3Q2: Quality of verification conditions
Thank you for examining these verification conditions and providing concrete suggestions. We distinguish the precision of individual acceptance conditions from the broader functional coverage and quality-control mechanisms of WebAppBench.
1. Interpreting the cited ACs in their functional context
The two examples highlight different verification concerns. For S1-011/AC-32, a valid P95 value may coincide with the maximum or mean for certain input distributions; correctness ultimately depends on the specified percentile calculation rather than numerical inequality alone. For S1-047/AC-36, initiating a download or observing an .otf filename does not, by itself, establish that the resulting font is valid. We appreciate these observations regarding the precision of individual verification conditions.
However, the benchmark does not evaluate these applications solely through the cited conditions. For S1-011, other checks verify live HTTP responses (AC-2) and recompute success rates from the underlying data (AC-31). For S1-047, separate checks verify that invalid character mappings and contours prevent export (AC-34/35). These checks assess actual computation, input constraints, and application behavior rather than merely the presence of interface elements. Individual ACs test specific functional obligations, while the complete checklist evaluates complementary aspects of the application's requested functionality.
2. Specification-grounded construction and behavioral verification
WebAppBench uses a multi-stage pipeline to improve AC quality before evaluation (§3.2–3.3). Candidate checks are extracted from the original specification and refined to remove unsupported assumptions, merge redundant requirements, clarify acceptance conditions, and ensure alignment between ACs, verification guidelines, fixtures, and session prerequisites. Each AC specifies an observable functional outcome and an explicit verification condition, rather than merely requiring a particular visual element or click sequence.
This emphasis is reflected in the benchmark's composition. State-transition and value-correctness checks account for 74.1% of all ACs, alongside 205 persistence checks, 142 multi-session checks across 33 tasks, and file fixtures for 37 tasks (§3.3). Verification therefore frequently requires performing operations, inspecting changed states, checking computed values, or confirming behavior across sessions. The CUA collects interaction evidence and assigns Supported, Contradicted, or Insufficient Evidence verdicts according to the applicable condition (§4), preserving the distinction between observed success, observed failure, and unresolved behavior.
3. Independent evidence of AC quality
We also assess the generated checks against independently authored expert ACs (§6.1). Across 10 benchmark specifications, 336 generated ACs were compared with 214 expert ACs, achieving 90.3% per-task macro full coverage under split/merge-aware matching. On 10 additional real-user requests, 75 generated ACs were compared with 58 expert ACs, achieving 70.3% macro full coverage. Full coverage requires that the generated checks jointly capture all conditions and expected outcomes of an expert requirement, not merely share similar wording.
We further audited the 224 generated ACs without strict one-to-one matches in the benchmark sample. 200 (89.3%) were grounded in the original specifications, while 24 (10.7%) introduced additional mandatory constraints. These analyses provide complementary evidence of requirement coverage and specification alignment, while remaining distinct from an exhaustive correctness audit of every verification condition.
Together, the automated quality-control pipeline, behavior-oriented checks, and independent expert comparisons support the reliability of WebAppBench's requirement-based evaluation design. The cited examples underscore the importance of precise acceptance conditions, but do not characterize the benchmark's overall approach to functional verification.
Response to Reviewer aUpM — W1: Contribution relative to prior benchmarks
Thank you for raising this comparison. ViBench and Vibe Code Bench already evaluate functional workflows, including multi-user behavior. We agree that counts of ACs, test plans, and workflow substeps are not directly comparable. WebAppBench contributes a combination of explicit verification status, criticality-aware scoring, and behavioral diagnosis, as summarized below.
1. Distinguishing functional contradictions from incomplete verification
ViBench reports successful test-plan and step execution. Vibe Code Bench records substep pass/fail judgments; a workflow returning no structured evaluation output, including after timeout or termination, is marked failed [2, §3.3]. WebAppBench explicitly separates observed contradictions from unresolved verification at the requirement level (§4).
Each AC receives Supported (S), Contradicted (C), or Insufficient Evidence (U), grounded in recorded execution evidence. For example, observing that a saved item disappears after reload contradicts persistence; failing to complete the reload-and-reopen sequence leaves persistence unresolved. Both receive no positive credit, but their verdicts distinguish an observed application defect from an inconclusive evaluation.
This representation also separates verification completion from judgment reliability. In our human-validation sample, the evaluator resolved 403 of 507 human-resolvable checks (79.5%) and agreed with human judgments on 398 of those 403 checks (98.8%) (§6.2). These statistics make explicit both how much functionality was conclusively evaluated and how reliably the resulting verdicts matched human assessment.
2. Explicit treatment of essential requirements
ViBench reports graded step completion alongside a strict fully-correct criterion [1]. Vibe Code Bench uses a 90% substep-success threshold for workflow passage [2, §3.4]. WebAppBench instead incorporates the importance of individual requirements directly into its aggregate score.
FuncScore assigns Critical/Major/Minor weights of 3/2/1 and multiplies the weighted Supported share by the Supported share among Critical requirements (§4). Thus, equal numbers of Supported checks can yield different scores depending on whether essential functionality has been verified.
For illustration, a ten-substep workflow with nine successful substeps meets a 90% threshold regardless of which substep failed. FuncScore explicitly distinguishes an unmet Critical requirement from an unmet Minor requirement through both weighting and the critical gate. The contribution is this importance-sensitive aggregation, rather than a larger number of evaluation items.
3. Behavioral diagnosis across applications
Test-plan and substep results identify where a particular workflow succeeds or fails. WebAppBench additionally organizes requirements by a common set of behavioral demands: structural, navigation, state transition, value correctness, persistence, perceptual, and runtime integrity (§3.2; Appendix D.1). These seven verification types support comparisons across applications whose workflows and interfaces differ.
The resulting analysis reveals patterns that an aggregate workflow pass rate does not directly express. Across ten configurations, structural and navigation checks average 71.35% and 70.66% Supported, compared with 52.62% for state transitions and 47.82% for value correctness. Under direct generation, GPT-5.6-Sol and Claude Opus 5 have similar structural support rates (75.29% and 77.97%), but their state-transition rates are 40.95% and 68.23% (§5.3). This identifies a concrete behavioral gap behind otherwise similar levels of interface completeness.
We have updated the note to Table 1 to clarify that the reported counts correspond to different evaluation units, including test plans, workflows, and atomic checks, while the comparison above distinguishes judgment representation, aggregation, and diagnostic outputs.
References:
[1] ViBench: A Benchmark on Vibe Coding. ACM CAIS 2026.
[2] Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development. ACM CAIS 2026.