acceptodds
Under review as a conference paper at ICLR 2027

WebAppBench: Benchmarking Functionality of Agent-Generated Web Applications

Abstract

Agentic coding systems increasingly generate complete Web applications from natural-language requirements. Evaluating these applications, however, remains challenging: successful deployment and visually plausible interfaces do not establish that users can complete intended workflows across interactions, pages, reloads, and evolving application state. Fixed test scripts are difficult to transfer across open-ended implementations, while generic browser exploration does not by itself provide a requirement-level functional oracle. We introduce WebAppBench, a bilingual, specification-grounded benchmark and evidence-grounded evaluation framework for functional correctness in agent-generated Web applications. WebAppBench contains 100 Chinese and English specifications decomposed into 3,197 typed atomic functional checkpoints. Each checkpoint defines a behavioral claim, an explicit verification condition, and a criticality level, covering navigation, state transitions, value correctness, persistence, perceptual requirements, and runtime integrity. Given a deployed application, a browser-based computer-use agent executes task-relevant workflows without implementation-specific selectors and grounds each verdict in action trajectories, observed state changes, reload outcomes, and runtime signals. WebAppBench records whether each requirement is supported, contradicted, or left with insufficient evidence, making aggregate scores traceable to individual user requirements and their supporting evidence. Experiments with representative LLM and coding-agent configurations enable fine-grained analyses of functional capabilities and failure patterns beyond build status, screenshots, or holistic application-level scores.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, AI Agents, Web Application Generation, Benchmark and Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_aUpMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WebAppBench evaluates whether agent-generated web applications satisfy their functional requirements. It contains 100 bilingual specifications and 3,197 atomic checks covering behaviors such as state transitions, value correctness, persistence, and multi-session interactions. A browser-based agent gathers runtime evidence for each check, producing supported, contradicted, or insufficient-evidence verdicts, with optional VLM review. Experiments across ten generation configurations reveal substantial gaps between interface completeness and behavioral correctness. Human validation shows strong agreement on jointly resolved checks.

Critical strengths

The benchmark targets an important gap in evaluating generated software. The focus on connected workflows, persistent state, and cross-view consistency captures failures that build success and polished screenshots cannot establish. These requirements make the benchmark useful for assessing whether generated applications support their intended use.

Explicit verification conditions make the evaluation interpretable. Each judgment is tied to a concrete requirement and recorded evidence, making failures easier to inspect than an overall quality score. Distinguishing insufficient evidence from observed failure is particularly valuable, even though the final scoring does not fully preserve this distinction.

The paper examines evaluator reliability alongside generator performance. Human review, cross-evaluator comparisons, component ablations, and repeated-generation experiments provide a more informative assessment than a leaderboard alone. The finding that incomplete verification is more common than disagreement on definitive judgments helps identify where the evaluation pipeline needs improvement.

Critical weaknesses

The incremental contribution is insufficiently established. Existing benchmarks, including ViBench and Vibe Code Bench, already evaluate functional workflows and multi-user behavior. Table 1 compares atomic checks against entire workflows/test plans, which contain multiple verification steps. These counts do not establish broader or finer coverage.

The protocol comparisons are not sufficiently controlled. In §6.3 and Appendix C.3, the compared settings differ in requirement access, evaluation granularity, and scoring format; CUA-explore receives only the application URL. A comparison using the same applications, requirements, scoring criteria, and comparable budgets is needed to isolate the benefit of the proposed protocol.

The supplementary checks contain substantive quality issues. S1-011/AC-32 requires P95 to differ from the maximum or mean, although valid samples can make them equal. Conversely, S1-047/AC-36 accepts download initiation or an .otf filename without verifying font validity. Such criteria risk rejecting correct implementations and rewarding superficial ones. The small AC-alignment study does not establish dataset-wide correctness.

Evaluator coverage still affects the reported scores. The 98.8% human agreement is conditional on jointly resolved checks, while coverage is 79.5%. Unresolved checks nevertheless receive zero credit. Together with the reported evaluator and generation variability, this leaves the stability of small leaderboard differences uncertain.

Questions for authors

Were requirement access and interaction budgets matched across protocol comparisons?

How were the identified AC issues audited, and what evidence supports ranking stability when unresolved checks are considered separately?

5 comments
AuthorsOpenReview ↗

Response to Reviewer aUpM — W1: Contribution relative to prior benchmarks

Thank you for raising this comparison. ViBench and Vibe Code Bench already evaluate functional workflows, including multi-user behavior. We agree that counts of ACs, test plans, and workflow substeps are not directly comparable. WebAppBench contributes a combination of explicit verification status, criticality-aware scoring, and behavioral diagnosis, as summarized below.

DimensionViBench [1]Vibe Code Bench [2]WebAppBench
Judgment representationTest-plan/step successSubstep pass/failSupported, Contradicted, or Insufficient Evidence for each requirement
Score aggregationGraded step completion and fully-correct criterionWorkflow passes if ≥90% of substeps succeed; application score averages workflow passesCriticality-weighted support multiplied by Critical-requirement support
Diagnostic organizationTest-plan and step resultsSubstep/workflow results and error analysisRequirement-level evidence status, criticality, and seven verification types

1. Distinguishing functional contradictions from incomplete verification

ViBench reports successful test-plan and step execution. Vibe Code Bench records substep pass/fail judgments; a workflow returning no structured evaluation output, including after timeout or termination, is marked failed [2, §3.3]. WebAppBench explicitly separates observed contradictions from unresolved verification at the requirement level (§4).

Each AC receives Supported (S), Contradicted (C), or Insufficient Evidence (U), grounded in recorded execution evidence. For example, observing that a saved item disappears after reload contradicts persistence; failing to complete the reload-and-reopen sequence leaves persistence unresolved. Both receive no positive credit, but their verdicts distinguish an observed application defect from an inconclusive evaluation.

This representation also separates verification completion from judgment reliability. In our human-validation sample, the evaluator resolved 403 of 507 human-resolvable checks (79.5%) and agreed with human judgments on 398 of those 403 checks (98.8%) (§6.2). These statistics make explicit both how much functionality was conclusively evaluated and how reliably the resulting verdicts matched human assessment.

2. Explicit treatment of essential requirements

ViBench reports graded step completion alongside a strict fully-correct criterion [1]. Vibe Code Bench uses a 90% substep-success threshold for workflow passage [2, §3.4]. WebAppBench instead incorporates the importance of individual requirements directly into its aggregate score.

FuncScore assigns Critical/Major/Minor weights of 3/2/1 and multiplies the weighted Supported share by the Supported share among Critical requirements (§4). Thus, equal numbers of Supported checks can yield different scores depending on whether essential functionality has been verified.

For illustration, a ten-substep workflow with nine successful substeps meets a 90% threshold regardless of which substep failed. FuncScore explicitly distinguishes an unmet Critical requirement from an unmet Minor requirement through both weighting and the critical gate. The contribution is this importance-sensitive aggregation, rather than a larger number of evaluation items.

3. Behavioral diagnosis across applications

Test-plan and substep results identify where a particular workflow succeeds or fails. WebAppBench additionally organizes requirements by a common set of behavioral demands: structural, navigation, state transition, value correctness, persistence, perceptual, and runtime integrity (§3.2; Appendix D.1). These seven verification types support comparisons across applications whose workflows and interfaces differ.

The resulting analysis reveals patterns that an aggregate workflow pass rate does not directly express. Across ten configurations, structural and navigation checks average 71.35% and 70.66% Supported, compared with 52.62% for state transitions and 47.82% for value correctness. Under direct generation, GPT-5.6-Sol and Claude Opus 5 have similar structural support rates (75.29% and 77.97%), but their state-transition rates are 40.95% and 68.23% (§5.3). This identifies a concrete behavioral gap behind otherwise similar levels of interface completeness.

We have updated the note to Table 1 to clarify that the reported counts correspond to different evaluation units, including test plans, workflows, and atomic checks, while the comparison above distinguishes judgment representation, aggregation, and diagnostic outputs.

References:

[1] ViBench: A Benchmark on Vibe Coding. ACM CAIS 2026.

[2] Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development. ACM CAIS 2026.

AuthorsOpenReview ↗

Response to Reviewer aUpM — W2Q1: Protocol comparison and control

Thank you for raising the need to distinguish controlled protocol ablations from open-ended evaluation baselines. Section 6.3 contains two comparison groups with different purposes: the three AC-based settings examine the contributions of verification guidance and VLM Evidence Review under fixed requirements and scoring, while the two open-ended settings examine autonomous exploration with and without task guidance. We clarify the experimental controls and interpretation below.

1. Matched requirements, scoring, and budgets in AC-based ablations

All three AC-based settings evaluate the same 18 applications against identical ACs and verification conditions, use the same criticality-weighted FuncScore, and retain the same CUA and per-type iteration caps (§5.1; Appendix C.3). These caps are 8/12/30/25/10/15/18 for structural, navigation, state-transition, value-correctness, perceptual, persistence, and runtime-integrity checks, respectively.

The guidance and evidence-review components are varied incrementally:

SettingControlled changeFuncScore (§6.3)
ACs onlyACs and verification conditions, without task-level guidelines or VLM review44.24
ACs + GuidelinesAdds navigation, prerequisite, and workflow guidance45.50
ACs + Guidelines + VLMAdds review of recorded evidence, without further browser interaction54.68

These comparisons hold the acceptance targets, scoring rules, and execution limits fixed. The resulting FuncScore changes characterize the contributions of guidance and evidence review within the same AC-based evaluation protocol. Importantly, VLM review reassesses existing evidence rather than granting additional browser interactions.

2. Controlled conditions for open-ended baselines

Guidelines-only and CUA-explore use the same applications, base system prompt, browser tools, and max_steps=700 limit per application, with early termination permitted. Their intended difference is the availability of task-specific information: Guidelines-only receives the task guidelines, whereas CUA-explore receives only the application URL. Neither uses a predefined AC checklist or per-AC verdicts; both select inspection targets autonomously and return a holistic score between 0 and 100.

All settings access deployed applications through browser URLs. Here, “URL-only” means that no additional task-specific requirements or guidance are supplied, rather than indicating a different application-access mechanism. An unguided CUA must infer both what functionality to examine and how to verify it, potentially introducing omissions or incomplete judgments. Our main protocol therefore provides explicit AC targets and verification guidelines to guide execution, alongside VLM review to reassess uncertain judgments using recorded evidence. The AC-based ablations separately examine these components.

3. Interpretation of the two comparison groups

The open-ended baselines characterize autonomous evaluation behavior rather than provide directly comparable FuncScores. Guidelines-only and CUA-explore report holistic scores of 71.00 and 77.13, respectively, but inspect approximately ten fewer checks per application than the AC-based protocol (§6.3). Because their requirements are self-selected and their scoring formats differ, these nominal scores cannot be interpreted as evidence of superior requirement satisfaction.

Accordingly, our conclusions about the contributions of guidance and VLM review are grounded in the matched AC-based ablations, while the open-ended settings illustrate how autonomous check selection can affect evaluation coverage and interpretation. We have clarified these differences, together with the matched inputs, scoring criteria, and execution budgets, in §6.3 and Appendix C.3.

AuthorsOpenReview ↗

Response to Reviewer aUpM — W3Q2: Quality of verification conditions

Thank you for examining these verification conditions and providing concrete suggestions. We distinguish the precision of individual acceptance conditions from the broader functional coverage and quality-control mechanisms of WebAppBench.

1. Interpreting the cited ACs in their functional context

The two examples highlight different verification concerns. For S1-011/AC-32, a valid P95 value may coincide with the maximum or mean for certain input distributions; correctness ultimately depends on the specified percentile calculation rather than numerical inequality alone. For S1-047/AC-36, initiating a download or observing an .otf filename does not, by itself, establish that the resulting font is valid. We appreciate these observations regarding the precision of individual verification conditions.

However, the benchmark does not evaluate these applications solely through the cited conditions. For S1-011, other checks verify live HTTP responses (AC-2) and recompute success rates from the underlying data (AC-31). For S1-047, separate checks verify that invalid character mappings and contours prevent export (AC-34/35). These checks assess actual computation, input constraints, and application behavior rather than merely the presence of interface elements. Individual ACs test specific functional obligations, while the complete checklist evaluates complementary aspects of the application's requested functionality.

2. Specification-grounded construction and behavioral verification

WebAppBench uses a multi-stage pipeline to improve AC quality before evaluation (§3.2–3.3). Candidate checks are extracted from the original specification and refined to remove unsupported assumptions, merge redundant requirements, clarify acceptance conditions, and ensure alignment between ACs, verification guidelines, fixtures, and session prerequisites. Each AC specifies an observable functional outcome and an explicit verification condition, rather than merely requiring a particular visual element or click sequence.

This emphasis is reflected in the benchmark's composition. State-transition and value-correctness checks account for 74.1% of all ACs, alongside 205 persistence checks, 142 multi-session checks across 33 tasks, and file fixtures for 37 tasks (§3.3). Verification therefore frequently requires performing operations, inspecting changed states, checking computed values, or confirming behavior across sessions. The CUA collects interaction evidence and assigns Supported, Contradicted, or Insufficient Evidence verdicts according to the applicable condition (§4), preserving the distinction between observed success, observed failure, and unresolved behavior.

3. Independent evidence of AC quality

We also assess the generated checks against independently authored expert ACs (§6.1). Across 10 benchmark specifications, 336 generated ACs were compared with 214 expert ACs, achieving 90.3% per-task macro full coverage under split/merge-aware matching. On 10 additional real-user requests, 75 generated ACs were compared with 58 expert ACs, achieving 70.3% macro full coverage. Full coverage requires that the generated checks jointly capture all conditions and expected outcomes of an expert requirement, not merely share similar wording.

We further audited the 224 generated ACs without strict one-to-one matches in the benchmark sample. 200 (89.3%) were grounded in the original specifications, while 24 (10.7%) introduced additional mandatory constraints. These analyses provide complementary evidence of requirement coverage and specification alignment, while remaining distinct from an exhaustive correctness audit of every verification condition.

Together, the automated quality-control pipeline, behavior-oriented checks, and independent expert comparisons support the reliability of WebAppBench's requirement-based evaluation design. The cited examples underscore the importance of precise acceptance conditions, but do not characterize the benchmark's overall approach to functional verification.

Reviewer_8QGvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WebAppBench evaluates whether web applications generated from natural-language specifications actually work. It consists of 100 specifications (50 in Chinese, 50 in English) and 3,197 atomic checks (ACs). Claude Opus 4.8 produced both from social-media posts.

Critical strengths
  1. The evaluation design is auditable and targets real failures. The benchmark includes 205 persistence checks and 142 multi-session checks, which is where generated apps tend to break. A DOM-only baseline resolves only 8.9% of the checks humans can resolve, which shows interaction-based evaluation is needed. This design is the main reason the benchmark is worth building on.

  2. The authors ran the right diagnostics and reported them. These include CUA sensitivity, run-to-run variance and alignment with expert-written checks.

Critical weaknesses
  1. The score counts evaluator uncertainty as app failure, which contradicts the paper's own argument.

  2. The main comparisons fall within noise, and the headline finding is confounded.. both of the paper's main empirical claims, the rankings and direct beating harnesses, depend on these two issues.

  3. Suggest the authors give more evidence about: how do you reconcile treating insufficient evidence as an abstention with FuncScore? Could you report supported / contradicted / insufficient counts per configuration, plus S/(S+C)? Why is GLM-5.3-Flash's unresolved rate 31.9% in §6.2 but 56.8% in Table 3?

  4. How many of the 3,197 ACs did humans review or edit? Against expert-written checks, F1 on critical ACs is only 27.5%. So,. can you run an agent harness starting from the same fixed scaffold used for direct generation?

  5. Where is the anonymous repository? I could not find a link in the PDF.

Questions for authors

see weaknesses

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8QGV — W1: U and FuncScore

Thank you for highlighting this inconsistency. An AC verdict describes what the evaluator can establish about an individual requirement, whereas FuncScore aggregates demonstrated satisfaction across the full specification. These are different levels of assessment.

1. Abstention versus scoring credit

FuncScore measures the extent to which an application demonstrably satisfies its specified functional requirements under a fixed evaluation protocol (§4). Each AC receives one of three evidence-based verdicts: Supported (S) when its verification condition is met, Contradicted (C) when observed behavior violates the condition, and Insufficient Evidence (U) when the evidence does not establish either outcome. Thus, U remains an abstention about functional correctness, not a judgment that the application has failed the check.

At the scoring level, only S earns positive credit. Both C and U contribute zero to the numerator but remain in the denominator because both are requirements whose satisfaction has not been demonstrated. This shared score contribution does not equate their meanings: C records a verified contradiction, whereas U records an unresolved judgment. Excluding U would instead measure the pass rate only among checks the evaluator happened to resolve. For example, with 2 S and 8 U, excluding U would produce a 100% resolved-check pass rate, although only 2 of 10 requirements have been verified as satisfied. FuncScore retains the complete requirement set, with criticality weights and a critical gate emphasizing essential requirements (§4).

2. Human evidence and conservative interpretation

Human review of 190 final automated U verdicts found 63 Supported (33.2%), 41 Contradicted (21.6%), and 86 still unresolved (§6.2). The 63 S cases show that automated uncertainty can withhold credit from working functionality. Yet 127/190 (66.8%) still lacked positive evidence of satisfaction after human inspection: 41 requirements were confirmed unmet, while 86 remained inconclusive. Some unresolved checks may also reflect application-level barriers, such as difficult-to-find features, unreachable prerequisite states, or effects that are hard to observe. These results explain the practical value of a conservative measure of demonstrated functionality without treating every U as a confirmed defect.

3. Evidence recovery and transparent reporting

Our protocol also mitigates evaluator uncertainty rather than merely assigning it zero credit. VLM Evidence Review converted 79 of GLM's 264 initial U verdicts into 62 S and 17 C, all consistent with human judgments; five initial S/C judgments were revised to U, leaving 190 final U verdicts and reducing the U rate from 44.4% to 31.9% (§6.2). In the cross-CUA study, review reduced the mean FuncScore gap from 19.54 to 8.48 points (§6.4). To make the distinction visible in reported results, we additionally provide criticality-weighted S/C/U shares, Verdict Coverage (S+C)/(S+C+U), and Resolved Support Rate S/(S+C) for all ten configurations. The complete verdict breakdown and its metric definitions are also included in §5.2 (Table 3). Together, these measures report both verified satisfaction and the extent of unresolved evidence, while preserving U as a distinct verdict.

ConfigurationFuncScoreS (%)C (%)U (%)Coverage (%)Resolved Support Rate (%)
Direct + Opus 549.6766.889.8023.3276.6887.69
Direct + GPT-5.6-Sol26.5047.6914.7037.6162.3985.32
OpenCode + Opus 536.5756.8610.2332.9167.0988.42
OpenCode + GLM-5.332.7248.9310.9040.1759.8374.33
OpenCode + Qwen 27B32.7850.0412.3037.6662.3475.89
Codex + Opus 547.4563.548.3728.0971.9185.94
Codex + GPT-5.6-Sol25.4344.4815.2740.2559.7578.18
Codex + Kimi-K347.5463.1410.5726.2973.7183.43
Codex + GLM-5.326.0943.1120.9335.9664.0465.44
Codex + Qwen 27B44.4260.1111.6228.2771.7380.54
AuthorsOpenReview ↗

Response to Reviewer 8QGV — W2: Noise, Ranking, and Harness

Thank you for highlighting the uncertainty in the rankings and the role of initialization in the Direct–Harness comparison. We have revised §5.2 and the conclusion to clarify the interpretation of configuration-level performance differences.

1. Generation variability and score interpretation

Table 2 compares ten generation configurations on the same 100 specifications using a common CUA, verification budgets, and evidence-review procedure (§5.1). This provides a shared basis for comparing their observed functional performance.

To assess generation variability, we conducted three independent generation rounds of Codex + Claude Opus 5 on the same 20 specifications. The round-level mean FuncScores were 50.55, 45.38, and 41.35, giving an overall mean of 45.76 with a sample standard deviation of 4.61 (Appendix C.6). These results show that aggregate performance can vary across generations even with the configuration and task set held fixed. A small lead in a single evaluation may therefore reflect generation variability rather than a stable performance advantage.

Accordingly, we interpret the 0.09-point gap between Codex + Kimi-K3 (47.54) and Codex + Claude Opus 5 (47.45), and the 2.22-point gap between Direct + Claude Opus 5 (49.67) and its Codex counterpart, as descriptive differences between closely scored configurations. We do not use their exact ordering to establish superiority. Our workflow comparison instead examines the observed functional outcomes together with generation costs and development settings, as detailed below.

2. End-to-end workflow comparison and the role of the scaffold

The Direct and harness-based settings differ in their generation workflows. Direct generates frontend and backend files within a fixed React/Vite and FastAPI/SQLite scaffold, whereas OpenCode and Codex construct applications through their own development processes. Consequently, the comparison measures the outcomes of these complete configurations; attributing a difference specifically to harness use would require a matched-scaffold experiment (§5.1).

The same-backbone comparison makes the observed outcome concrete:

Claude Opus 5 configurationFuncScoreGeneration cost per appGeneration time per app
Direct49.67$2.2720.69 min
Codex47.45$3.8252.93 min
OpenCode36.57$14.0476.37 min

In this evaluation, both harness-based configurations consumed more time and money than Direct, without obtaining higher observed FuncScores (§5.2, Table 2). The practical finding is that the additional expenditure of these workflows was not accompanied by better measured functional satisfaction. The 13.10-point Direct–OpenCode difference is an observed configuration-level gap; its magnitude alone does not establish statistical significance or identify its cause.

We have revised §5.2 and the conclusion to emphasize the observed functional outcomes of complete generation configurations: across the evaluated configurations, harness-based workflows did not show a consistent advantage in observed functional satisfaction. The revised trajectory discussion describes build-repair and refinement behavior without attributing score differences to iteration alone.

AuthorsOpenReview ↗

Response to Reviewer 8QGV — W3: U Statistics and Reporting Scope

Thank you for requesting a clearer reconciliation of U verdicts, configuration-level outcome statistics, and the two reported GLM unresolved rates. We address each point below.

1. Abstention and FuncScore

At the AC level, Insufficient Evidence (U) is an abstention: the available evidence does not establish whether the requirement is satisfied. Contradicted (C) records an observed violation. FuncScore, however, measures demonstrated satisfaction across the full specification, awarding positive credit only to Supported (S) requirements (§4). U and C thus retain distinct verdict meanings, while neither earns pass credit. Keeping all requirements in the denominator also prevents a high conditional pass rate from being reported when only a small subset has been verified. This makes FuncScore a conservative measure of evidence-verified functionality.

2. Configuration-level S/C/U statistics

We provide S/C/U shares, Verdict Coverage = (S+C)/(S+C+U), and Resolved Support Rate (RSR) = S/(S+C) for all ten generation configurations in the table below. To align these diagnostics with FuncScore, we first calculate them within each application using Critical/Major/Minor weights of 3/2/1, then macro-average across applications. The reported percentages therefore represent criticality-weighted, application-level macro averages, preserving the importance of essential requirements and giving each application equal weight.

ConfigurationFuncScoreS (%)C (%)U (%)Coverage (%)Resolved Support Rate (%)
Direct + Opus 549.6766.889.8023.3276.6887.69
Direct + GPT-5.6-Sol26.5047.6914.7037.6162.3985.32
OpenCode + Opus 536.5756.8610.2332.9167.0988.42
OpenCode + GLM-5.332.7248.9310.9040.1759.8374.33
OpenCode + Qwen 27B32.7850.0412.3037.6662.3475.89
Codex + Opus 547.4563.548.3728.0971.9185.94
Codex + GPT-5.6-Sol25.4344.4815.2740.2559.7578.18
Codex + Kimi-K347.5463.1410.5726.2973.7183.43
Codex + GLM-5.326.0943.1120.9335.9664.0465.44
Codex + Qwen 27B44.4260.1111.6228.2771.7380.54

These statistics make the composition of similar FuncScores interpretable. For example, Codex + GPT-5.6-Sol (FuncScore 25.43) has 40.25% U, 15.27% C, and 78.18% RSR; Codex + GLM-5.3-Flash (26.09) has 35.96% U, 20.93% C, and 65.44% RSR. The close aggregate scores thus reflect different balances of unresolved checks, confirmed contradictions, and verified support among resolved checks. Across all ten configurations, the weighted macro-average U share ranges from 23.32% to 40.25%. We have added the complete table and metric definitions to §5.2 (Table 3).

3. Reconciling the GLM unresolved rates

The two reported unresolved rates correspond to different evaluation stages and application subsets, as summarized below. The cross-CUA sensitivity results appear in Table 4 of the revised manuscript (Table 3 in the original submission).

Setting§6.2 Human Validation§6.4 Table 4
Applications2018
Evaluation stageAfter VLM reviewCUA-only
GLM U rate31.9%56.8%
Statistical basis190/595 ACsCUA-only unresolved rate

The 18-application subset follows §6.3, where two applications were excluded because autonomous exploration repeatedly stalled. Section 6.4 uses this same subset to ensure consistent cross-CUA comparisons. VLM Evidence Review can also convert some initially unresolved U verdicts into definitive S/C judgments. The two figures are therefore consistent with their respective evaluation settings.

Reviewer_FJVQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WebAppBench, a benchmark that tests web apps built by coding agents. It has 100 app specifications (50 Chinese, 50 English) and 3,197 atomic checks. Each check names one behavior the app must have, such as "A saved itinerary reopens with its original stops." Claude Opus 4.8 wrote the specifications from social-media posts, then wrote the checks from the specifications. To grade an app, a browser agent (GLM-5.3-Flash) uses it and marks each check as supported (pass), contradicted (fail), or insufficient evidence (unsure). In an optional evidence-review step, which the main results use, any verdict with confidence below 0.7 goes to a second model (GPT-5.6-Terra) for re-judges.

Critical strengths
  • The paper targets a real gap: whether the generated apps are functional, not just whether they look right (Fig. 1).
  • It measures the grader's limits: how often the grader is unsure, not just how often it agrees with humans (Sec. 6.2).
Critical weaknesses
  • The main score treats "unsure" as "fail": only supported checks earn credit (Sec. 4), though the paper says insufficient evidence does "not establish a functional defect" (Sec. 4). In the human check, humans found 63 of the agent's 190 unsure checks working, yet these scored zero, while only 5 verdicts were the opposite of the humans' (Sec. 6.2). So the score depends on the grader: on the same apps, a grader that is unsure less often (Claude Opus 4.8) gives 63.16 instead of 54.68 (Table 3). Table 2 does not report the unsure rate per setup, so we cannot tell whether a low score means broken apps or apps the grader could not finish checking.

  • One LLM wrote the benchmark, and its checks match the experts' only loosely. On 10 specs, 112 of Claude's 336 checks match one of the experts' 214, and agreement is lowest on critical checks, which count most (F1 27.5%; Table 4). The paper does not say how many unmatched checks go beyond the spec, who the experts and human reviewers were, or whether they tested the apps themselves (Sec. 6.1, 6.2).

  • Some checks need live outside data, e.g., the Louvre's "real current information from an authoritative external source" (Table 9). The prompts shown mention no data source or network access (App. E.1), and the grader "never"uses "third-party services" (Ethics Statement). So the low runtime-integrity scores (11.38–27.05%, Sec. 5.3) may come from this setup, not the models.

  • Claimed features go unanalyzed: there are no results by language for a "bilingual" benchmark, none for the 142 multi-session checks, and no numbers behind the domain claim (Sec. 5.3).

Questions for authors

Refer to weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer FJVQ — W1: Unresolved verdicts and grader dependence

Thank you for these concerns. We clarify scoring with human review, cross-CUA evidence, and per-configuration verdict statistics.

1. Scoring unresolved requirements

FuncScore measures demonstrated satisfaction of specified requirements under a fixed evaluation protocol. It aggregates verified satisfaction across the full set of ACs, with criticality weights (3/2/1) reflecting requirement importance and a critical gate imposing an additional penalty when essential requirements are not verified as satisfied (§4). At the AC level, Supported (S) means observed behavior satisfies the verification condition; Contradicted (C) means the condition is violated; and Insufficient Evidence (U) means the evidence remains inconclusive. U is an abstention, not a confirmed functional failure; neither U nor C earns pass credit. Retaining U in the denominator prevents inflated scores: with 2 S and 8 U, excluding U would yield 100% support among resolved checks, although only 2 of 10 requirements were verified.

2. Human validation and evidence recovery

We manually reviewed all 595 AC verdicts across 20 sampled applications (§6.2). Human review found that 127 of 190 automated U verdicts (66.8%) still lacked evidence of functional satisfaction: 41 were confirmed Contradicted and 86 remained unresolved, while 63 were verified as Supported (§6.2). Unresolved checks may also reflect application barriers, such as hard-to-access functionality or unobservable outcomes, potentially indicating usability issues. These findings support FuncScore as a conservative measure of demonstrated functional satisfaction, rather than excluding unresolved requirements from scoring. Among the 507 ACs resolved by human reviewers, the CUA reached definitive judgments on 403 (79.5%), agreeing with human assessments in 398 cases (98.8%), with only five disagreements. This high agreement provides strong empirical support for the reliability of evidence-grounded CUA judgments (§6.2).

VLM evidence review further recovered definitive judgments from 79 of GLM's 264 initial U verdicts (29.9%), converting 62 to S and 17 to C, all consistent with human judgments. Meanwhile, five initial S/C verdicts were revised to U, leaving 190 final U verdicts (264 − 79 + 5). This reduced the overall U rate from 44.4% to 31.9%, demonstrating the effectiveness of evidence review in improving verification coverage.

3. Cross-CUA sensitivity under controlled conditions

We evaluated the same 18 applications with GLM-5.3-Flash and Claude Opus 4.8, keeping GPT-5.6-Terra fixed as the evidence reviewer (§6.4, Table 4 in the revised manuscript; Table 3 in the original submission). Their CUA-only unresolved rates were 56.8% and 18.9%, respectively, and their mean CUA-only FuncScores were 45.50 and 65.04. After evidence review, the scores were 54.68 and 63.16, narrowing the gap from 19.54 to 8.48 points. The reviewed application-level scores also remained strongly correlated (Spearman ρ = 0.809). Thus, evaluator choice affects absolute scores, but evidence review substantially improves cross-CUA consistency on this subset. Crucially, the main comparison of ten generation configurations in Table 2 uses the same CUA, ACs, verification procedure, and per-type interaction budgets for every configuration (§5.1).

4. Reporting the composition of low scores

To distinguish incomplete verification from observed functional contradictions, we additionally provide criticality-weighted S/C/U shares, Verdict Coverage = (S+C)/(S+C+U), and Resolved Support Rate = S/(S+C), computed within each application and macro-averaged across applications. Across all ten configurations, the weighted U share ranges from 23.32% to 40.25%. For example, Codex + GPT-5.6-Sol (FuncScore 25.43) has U/C shares of 40.25%/15.27%, whereas Codex + GLM-5.3-Flash (26.09) has 35.96%/20.93%. Similar FuncScores can therefore conceal different proportions of inconclusive and contradicted requirements. We have incorporated the full verdict breakdown and its metric definitions into §5.2 (Table 3).

ConfigurationFuncScoreS (%)C (%)U (%)Coverage (%)Resolved Support Rate (%)
Direct + Opus 549.6766.889.8023.3276.6887.69
Direct + GPT-5.6-Sol26.5047.6914.7037.6162.3985.32
OpenCode + Opus 536.5756.8610.2332.9167.0988.42
OpenCode + GLM-5.332.7248.9310.9040.1759.8374.33
OpenCode + Qwen 27B32.7850.0412.3037.6662.3475.89
Codex + Opus 547.4563.548.3728.0971.9185.94
Codex + GPT-5.6-Sol25.4344.4815.2740.2559.7578.18
Codex + Kimi-K347.5463.1410.5726.2973.7183.43
Codex + GLM-5.326.0943.1120.9335.9664.0465.44
Codex + Qwen 27B44.4260.1111.6228.2771.7380.54
AuthorsOpenReview ↗

Response to Reviewer FJVQ — W2: Expert AC alignment and overconstraints

Thank you for highlighting the AC alignment results. We distinguish strict one-to-one alignment from full requirement coverage, including for Critical ACs; separately audit unmatched checks; and clarify expert participation and manual application testing.

1. Overall AC alignment and requirement coverage

The 3,197 benchmark ACs were constructed through automated extraction, refinement, and consistency checks against the original specifications, rather than accepted from a single generation pass (§3.2–3.3). To assess their quality independently, we compared 336 generated ACs against 214 expert-authored ACs on 10 benchmark specifications. A strict match requires agreement on the functional goal, applicable conditions, and expected outcome, with each AC matched at most once. This yielded 112 pairs and a 40.4% macro F1, but one-to-one matching cannot fully capture cases where a single expert requirement is divided into multiple generated checks. We therefore also measure full coverage: an expert AC counts as covered only when one or more generated ACs jointly express all its conditions and expected outcomes; partial overlap is excluded. The per-task macro full coverage reached 90.3% on benchmark specifications and 70.3% on 10 shorter real-user requests (§6.1; Appendix C.4; Table 5 in the revised manuscript, Table 4 in the original submission). These results show that the generated checklists capture most expert-identified requirements despite differences in decomposition.

2. Critical AC alignment and coverage

Because Critical requirements carry the greatest weight, we also examine their alignment under the full-coverage criterion. The strict Critical F1 was 27.5%, reflecting both one-to-one matching and differences in importance labels: 39 of the 112 content-matched pairs differed in criticality labels. Applying the same split/merge-aware full-coverage criterion without requiring identical Critical labels, 93 of 99 expert Critical ACs were fully covered (93.6% per-task macro coverage). This shows that most essential requirements identified by experts are captured despite differences in decomposition and importance labels.

3. Audit of unmatched ACs

We additionally audited all 224 generated ACs without strict one-to-one matches from the 10 benchmark specifications against their original requirements. Of these, 200 (89.3%) were grounded in the specifications, while 24 (10.7%) introduced additional mandatory constraints, mainly involving unspecified UI structure, implementation details, or overly restrictive verification conditions. These 24 cases mainly reflect over-specific acceptance conditions, rather than 24 wholly invented application functions. The audit isolates this narrower overconstraint issue instead of treating every unmatched check as a requirement invented by the model.

4. Expert participation and manual application testing

Five reviewers—three PhD students and two master's students, all with computer-science backgrounds—participated in both human studies. For AC-quality validation (§6.1), they independently authored expert ACs for their assigned specifications and compared them with automatically generated ACs against the original requirements. For evaluator validation (§6.2), the same reviewers directly interacted with the generated applications, performing the necessary operations, observing the resulting behavior, and assigning human S/C/U verdicts according to the AC-specific verification conditions. They manually reviewed all 595 AC verdicts across 20 applications. Thus, the human judgments were grounded in firsthand application testing rather than merely inspecting the CUA's reported results.

AuthorsOpenReview ↗

Response to Reviewer FJVQ — W3: External data and Runtime Integrity

Thank you for pointing out the ambiguity between the external-data requirements and the Ethics Statement. We clarify how such data is available to generated applications and how its authenticity is verified.

1. External-data requirements and generation access

The Louvre check in Table 10 of the revised manuscript (Table 9 in the original submission) comes directly from S1-081 (WorldRoute), whose specification explicitly requires online sources for real destination information, including opening hours, admission prices, ratings, and source links. Appendix E.1 displays shared prompt templates rather than complete task-specific inputs: the full product requirements are inserted for every generator, including the requirements to use online data. Internet access was permitted during application development and at runtime for deployed applications; no particular API or provider was prescribed. The Louvre authenticity check therefore tests a specified product requirement, not an additional data-source constraint introduced by the evaluator.

2. Public-information verification and ethical boundaries

For checks requiring current external facts, the CUA uses its Search tool to retrieve authoritative public information and compares it with the facts displayed by the generated application. For the Louvre example, the verification condition checks the reported location, opening hours, admission price, and rating against real-world sources rather than accepting placeholders or invented values (revised Table 10). The Ethics Statement's restriction concerns direct CUA interaction with real third-party accounts or services; it does not prohibit public-information retrieval for independent verification. We have clarified this distinction in the Ethics Statement, explicitly separating public-information retrieval from direct interaction with third-party accounts or services.

3. Interpretation of Runtime Integrity scores

Runtime Integrity covers more than external-data authenticity: the category includes required data integration, accessibility, and runtime operation (Appendix D.1, Table 8). Accordingly, the 11.38%–27.05% pass-rate range across the ten generation configurations (§5.3) aggregates checks with different runtime requirements, not just live-data lookups. Since external-data requirements were included in the generators' specifications, application access to online data was permitted, and the evaluator could check public sources, these results should not be attributed to a blanket restriction on external data in the experimental setup. They reflect the difficulty of meeting and verifying the specified runtime behavior across generated applications.

Reviewer_Ca8gOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WebAppBench, a benchmark designed to systematically evaluate whether web applications generated by coding agents from natural-language requirements provide complete and reliable functionality. The benchmark contains 100 Chinese and English web application specifications and 3,197 atomic functional checks (ACs) with explicit verification conditions, covering seven capability categories: structure, navigation, state transition, value correctness, persistence, perceptual requirements, and runtime integrity. The authors further develop an adaptive evaluation protocol based on browser-based computer-use agents (CUAs). The agents execute operations within the applications, collect evidence such as screenshots and runtime states, and submit the resulting evidence to a vision-language model for further review when needed.

Critical strengths

1.WebAppBench contains a large and diverse collection of evaluation examples, including various multi-session checks. Its design covers cross-page state synchronization, data persistence, computational logic, and multi-user interactions, all of which are difficult to verify using screenshots or static DOM information alone. It therefore addresses the limited focus of existing web application generation benchmarks on interface appearance or a small number of end-to-end workflows.

  1. The evaluation protocol provides strong auditability and is supported by comprehensive reliability analyses. Each AC has an explicit verification condition, and the agent’s judgments are supported by concrete evidence, including interaction trajectories, screenshots, page states, and runtime signals.
Critical weaknesses
  1. The core FuncScore lacks sufficient measurement validity. The current score is jointly affected by application functionality, AC quality, and the execution capability of the evaluation agent, making it difficult to interpret directly as an application’s functional correctness rate.

  2. There is a clear inconsistency between the definition and scoring of “insufficient evidence.”In the analysis, insufficient evidence is explicitly treated as an abstention by the evaluator. In the calculation of FuncScore, it remains in the denominator and receives a score of zero. Consequently, failure to complete the verification process, exhaustion of the interaction budget, failure to locate the correct interaction entry point, and an actual absence of the required functionality all receive the same score of zero. The human review results further reveal the problems caused by this metric design.

  3. The human review also lacks a clearly defined and consistent evaluation standard. The paper does not clearly explain whether all 3,197 ACs were manually reviewed, how many annotators participated, what proportion of the ACs were modified during review, or how much agreement was achieved among the annotators.

Questions for authors

See the weaknesses If the authors adequately address my concerns, I would be willing to raise my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Ca8g — W1: Measurement validity of FuncScore

Thank you for raising this concern. We agree that the manuscript should more explicitly clarify the measurement target of FuncScore. Specifically, FuncScore measures demonstrated functional satisfaction under a fixed evaluation protocol: the extent to which an application's specified requirements are verified through execution evidence, with greater emphasis on essential functionality. We support this interpretation through two complementary aspects: (1) the validity of the ACs as verification targets, and (2) the reliability of the automated evaluator.

Each AC defines a required behavior and an explicit verification condition. The CUA interacts with the application and records whether the observed behavior supports the condition (Supported), contradicts it (Contradicted), or leaves it unresolved (Insufficient Evidence). FuncScore aggregates Supported checks with Critical/Major/Minor weights of 3/2/1 and applies a critical gate based on the proportion of Critical requirements verified as satisfied (§4). All ten generation configurations are evaluated using the same specifications, ACs, CUA, verification procedure, and per-type interaction budgets (§5.1), providing a consistent basis for comparing their evidence-verified functional outcomes.

1. AC quality: requirement alignment and expert validation

The original specification remains the reference throughout AC extraction and refinement. Candidate checks are examined for requirement alignment, unsupported assumptions, redundancy, and clear acceptance conditions. We also check consistency across ACs, verification guidelines, file fixtures, and session prerequisites (§3.2–3.3).

We independently assessed the generated checks against expert-authored ACs for 10 benchmark specifications and 10 shorter real-user requests. The benchmark comparison included 336 generated ACs and 214 expert ACs. Matching examines the functional goal, applicable conditions, and expected outcome, rather than wording alone. Because a single expert requirement may be decomposed into several generated checks, we additionally measure full coverage: an expert AC is counted as covered only if one or more generated ACs jointly express all its conditions and expected outcomes. Per-task macro full coverage reached 90.3% for benchmark specifications and 70.3% for real-user requests (§6.1; Appendix C.4, Table 5). These results support the pipeline's ability to capture expert-identified functional requirements.

2. Evaluator reliability: human validation and cross-CUA consistency

We assess evaluator reliability through human judgments and sensitivity to the execution CUA, using explicitly identified samples.

Human validation (20 applications). Five human reviewers examined 595 AC verdicts across a stratified sample covering five domains, both languages, and different performance levels (§6.2). Human reviewers resolved 507 checks as S/C; the evaluator also resolved 403 of these (79.5% verification coverage). Among the 403 jointly resolved checks, 398 agreed with human review (98.8%), with five disagreements. These figures distinguish the evaluator's ability to reach a definitive judgment from the reliability of those judgments.

Cross-CUA consistency (18 applications). Separately, §6.4 compares GLM-5.3-Flash and Claude Opus 4.8 on the same 18-application subset used in §6.3; two applications were excluded because autonomous exploration repeatedly stalled. Holding the evidence reviewer fixed, the CUA-only mean FuncScores were 45.50 and 65.04 (a 19.54-point gap). After VLM evidence review, the scores were 54.68 and 63.16 (an 8.48-point gap, 56.6% smaller). Reviewed application-level scores also showed Spearman ρ = 0.809, indicating substantial consistency in their relative assessment of applications on this subset (§6.4, Table 4).

Together, expert AC alignment, human-verdict agreement, and cross-CUA analysis provide complementary validation for interpreting FuncScore as evidence-verified requirement satisfaction under a common protocol. We have clarified the measurement scope of FuncScore in §4 and updated the Table 2 caption to distinguish unresolved judgments from confirmed functional failures.

AuthorsOpenReview ↗

Response to Reviewer Ca8g — W2: Insufficient evidence and scoring

Thank you for raising this concern. FuncScore is designed to measure demonstrated functional satisfaction under a fixed evaluation protocol, rather than an application's latent functional correctness independent of the evaluation process. We therefore distinguish the meaning of an unresolved verdict from its scoring contribution: U does not establish a functional defect, but it cannot earn positive credit without evidence of satisfaction. Beyond clarifying this measurement scope, we explain why retaining U in the full denominator prevents incomplete verification from artificially inflating scores, and support this conservative design with human-review findings and per-configuration verdict statistics.

1. Abstention and positive verification credit

A U verdict means that the recorded evidence leaves the verification condition unresolved. This includes cases where the evaluator exhausts its budget, cannot find the required entry point, or does not complete the necessary interactions. A C verdict records direct evidence that the condition is violated. These outcomes therefore represent different evidence states, while neither supplies the positive evidence required for Supported credit (§4).

The distinction is between judging a requirement and aggregating verified satisfaction. At the judgment level, U is an abstention about whether the requirement is satisfied. At the aggregation level, that requirement remains part of the specification against which demonstrated satisfaction is measured. The same zero contribution therefore means the absence of positive verification credit, not the same diagnosis of application behavior. When budget exhaustion leaves a working feature unresolved, its contribution is zero because its satisfaction was not established within the protocol.

Retaining the full denominator also prevents selective verification from inflating results. Before weighting and the critical gate, an application with 2 Supported and 8 unresolved requirements has demonstrated 20% of its ten requirements; excluding U would report 100% support based on just two resolved checks. The full denominator preserves the extent of demonstrated satisfaction, while Resolved Support Rate separately describes success conditional on reaching a definitive verdict.

2. Human-review findings and score interpretation

Human review of 190 automated U verdicts found 63 Supported, 41 Contradicted, and 86 still unresolved (§6.2). Thus, 127/190 (66.8%) still lacked grounds for pass credit after human inspection: some requirements were confirmed unmet, while others remained unverified. Excluding all original U checks would remove both groups. The 63 Supported cases (33.2%) also show that incomplete automated verification can withhold credit from working functionality. Unresolved outcomes may further involve application-level barriers, such as unclear entry points, difficult-to-reach prerequisite states, or functional effects that are hard to observe; these may reflect discoverability or usability issues as well as evaluator limitations.

To distinguish observed contradictions from incomplete verification, we provide the following complementary statistics across all ten configurations. Verdict Coverage measures the share resolved as S or C, while Resolved Support Rate (RSR) measures the Supported share among resolved checks.

ConfigurationFuncScoreS (%)C (%)U (%)Coverage (%)Resolved Support Rate (%)
Direct + Opus 549.6766.889.8023.3276.6887.69
Direct + GPT-5.6-Sol26.5047.6914.7037.6162.3985.32
OpenCode + Opus 536.5756.8610.2332.9167.0988.42
OpenCode + GLM-5.332.7248.9310.9040.1759.8374.33
OpenCode + Qwen 27B32.7850.0412.3037.6662.3475.89
Codex + Opus 547.4563.548.3728.0971.9185.94
Codex + GPT-5.6-Sol25.4344.4815.2740.2559.7578.18
Codex + Kimi-K347.5463.1410.5726.2973.7183.43
Codex + GLM-5.326.0943.1120.9335.9664.0465.44
Codex + Qwen 27B44.4260.1111.6228.2771.7380.54

All percentages use Critical/Major/Minor weights of 3/2/1 within each application and are then macro-averaged. Coverage = (S+C)/(S+C+U); RSR = S/(S+C), computed per application before averaging. For example, OpenCode + Opus 5 has 88.42% RSR but 67.09% Coverage and 32.91% U: high support among resolved checks coexists with incomplete verification. We have included this breakdown and its metric definitions in §5.2 (Table 3).

AuthorsOpenReview ↗

Response to Reviewer Ca8g — W3: Human review protocol and scope

Thank you for requesting clarification of the human review protocol. We distinguish the automated construction of benchmark ACs from the independent human validation of AC quality and evaluator judgments.

1. AC construction and human validation

All 3,197 benchmark ACs were extracted, refined, and quality-checked by the automated pipeline, without manual rewriting (§3.2–3.3). Human involvement was limited to validation, rather than modification of the benchmark ACs. Five reviewers—three PhD students and two master's students, all with computer-science backgrounds—participated in the human studies. For AC-quality validation, they independently authored expert ACs for 10 benchmark specifications and 10 real-user requests. These expert ACs served as independent references for comparing functional goals, applicable conditions, and expected outcomes, including requirement coverage (§6.1; Appendix C.4). No benchmark ACs were manually modified during this process.

2. Evaluation criteria and reliability

For evaluator validation, the same five reviewers directly tested the 20 selected applications and assessed all 595 ACs (§6.2). Each reviewer evaluated their assigned checks by performing the required interactions and comparing the observed outcomes with the AC-specific verification conditions. All reviewers applied the same Supported (S), Contradicted (C), and Insufficient Evidence (U) criteria (§4).

Human review resolved 507 of the 595 checks as S or C, leaving 88 unresolved. The automated evaluator also resolved 403 of those 507 checks, yielding 79.5% coverage of human-resolvable checks. Among these 403 jointly resolved checks, 398 automated verdicts agreed with human judgments (98.8%), with five disagreements (§6.2). These results separately quantify the extent of automated verification and its agreement with human assessment when both provide definitive judgments.

3. Scope, sampling, and cost of human review

The 20-application sample was selected to cover different observed performance levels, application categories, and both languages. We stratified specifications by their mean FuncScore across four generation configurations (Direct + Claude Opus 5; Codex + Claude Opus 5, Kimi-K3, and Qwen3.8-27B), selecting 6 Low (<30), 7 Mid (30–<70), and 7 High (≥70) specifications. Category allocation was approximately proportional to the full benchmark: 6 Life & Health, 4 Entertainment & Social, 4 Education & Reference, 3 Work & Business, and 3 Creation & Development. The sample also contained 10 Chinese and 10 English specifications (§6; Appendix C.1).

Reviewing one generated application took about 40 minutes on average. Exhaustively reviewing 100 applications would therefore require approximately 67 person-hours per generation configuration, or 667 hours across ten configurations. The stratified sample enabled targeted validation across languages, domains, and performance levels without requiring exhaustive human evaluation. We have clarified the human evaluation procedure and the definitions of verification coverage and agreement with human judgments in §6.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.