- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
WebProgramBench evaluates whether coding agents can reconstruct deployable web applications by exploring a live reference without access to its source repository. Agents receive HTTP endpoint and database contracts, then infer the remaining behavior and implement a frontend, backend, and persistent database.
The benchmark contains 20 open-source applications across four domains, with 1,254 executable tests and 20,851 assertions. Its central contribution is an ordered evaluation protocol that checks each browser interaction across three layers: interface outcomes, HTTP requests and responses, and database effects. An automated pipeline analyzes behavior dependencies, plans stateful executions, generates tests, and iteratively repairs them against the reference application. Candidate implementations are redeployed in clean environments before evaluation.
Experiments evaluate five models under a shared agent harness. Claude Opus 5 achieves the highest cross-layer test pass rate, 56.35%; one application passes every retained test, and three applications, including that one, reach at least 95%. Ablations examine verification layers, live-reference access, and agent harnesses. Additional experiments assess evaluator repeatability and sensitivity to the verifier model.
The results establish a challenging setting for discovering and implementing stateful application behavior. The main contribution is the benchmark and its integrated evaluation infrastructure, with evidence that live interaction provides information beyond the supplied contracts.
Critical strengths
1- The benchmark evaluates meaningful integration across application layers: I find the joint evaluation protocol to be the strongest aspect of the paper. Checking the interface outcome, HTTP exchange, and database effect of the same user interaction provides a more substantive assessment than checking whether an interface appears to work. The layer ablation illustrates the practical importance of this distinction: average test pass rate is 50.01% with interface-only evaluation, compared with 35.12% when all three layers are required.
2 - The construction pipeline addresses practical challenges in stateful evaluation: the test-construction pipeline explicitly handles dependencies between application behaviors, including resources created by earlier actions and consumed by later ones. This is a useful engineering contribution because evaluating workflows such as creating, editing, and deleting an object requires consistent identities and state across executions. Redeploying submissions in a separate evaluation environment with a fresh database further strengthens the assessment of whether the delivered application actually works.
Critical weaknesses
1 - The paper describes implementation choices as unconstrained, but the task requires specified HTTP methods and paths, PostgreSQL, and particular table names, column names, types, and primary keys (Appendix L). These requirements are disclosed, so they are legitimate task constraints. However, they substantially narrow the architectural freedom being evaluated. The task measures reconstruction under explicit API and persistence compatibility requirements.
Questions for authors
How accurate is the evaluator on independently assessed correct and incorrect implementations? Could you evaluate deliberately incomplete backends, faulty persistence implementations, and valid alternative frontends? This would help quantify false acceptance and false rejection beyond repeatability and agreement between judges.
4 comments
Response to [Q1-1]: False acceptance of defective implementations
We thank you for this valuable question. We conduct a mutation experiment to measure the invalid-solution rejection rate of our generated cross-layer test suites.
We conduct this experiment on five selected websites. As defined in Table T, we design 15 mutation operators across three categories: persistence errors, protocol errors, and GUI errors, with five operators per category. For each website, PhD-level experts construct 15 mutant implementations, each created by injecting exactly one defect into a separate copy of the correct reference implementation, and manually verify that each defect is valid.
Table T: Mutation operators used to construct invalid implementations
| ID | Mutation Operator | Description |
|---|---|---|
| Persistence errors | ||
| P1 | Omitted write | Skip a required database write while still returning success. |
| P2 | Incorrect field value | Store a type-valid but semantically incorrect value, such as a different valid status. |
| P3 | Wrong target record | Update or delete another existing record instead of the record selected by the user. |
| P4 | Incorrect ownership or association | Create the correct business record but assign it to the wrong user or project. For example, save a reservation created by user A under user B. |
| P5 | Incomplete related writes | Update the main record but omit required related writes, such as the corresponding history record. |
| Protocol errors | ||
| H1 | Incorrect status code | Return an incorrect HTTP status code after a successful operation, such as returning 400 after successfully cancelling a reservation. |
| H2 | Malformed response structure | Remove or rename a required field, or change the required response structure. |
| H3 | Incorrect response value | Store data correctly but return an incorrect field value, such as returning description as title. |
| H4 | Incorrect redirect target | Modify Location to redirect to an incorrect, existing business page. |
| H5 | Incorrect authorization decision | Reject a legitimate request by incorrectly requiring administrator privileges for an operation available to regular users. |
| GUI errors | ||
| G1 | Missing event handler | Remove an event handler so that a visible, clickable control performs no action. |
| G2 | Incorrect navigation target | Navigate to an incorrect, existing page, such as opening the settings page instead of the edit page. |
| G3 | Incorrect field binding | Bind an input to the wrong field, such as submitting the title input as the description. |
| G4 | Incorrect displayed value | Return the correct service response but display an incorrect value, such as the wrong reservation time. |
| G5 | Stale interface state | Update the backend successfully without synchronizing the interface, such as retaining a deleted record in the list. |
Each of the 15 mutant implementations is evaluated using all cross-layer test suites generated for the corresponding website. A mutant is considered rejected when the injected defect causes a new test failure relative to the correct reference implementation. The invalid-solution rejection rate is the fraction of the 15 mutant implementations in which at least one cross-layer test fails.
Table U: Invalid-solution rejection rates across five websites
| Website | Invalid-Solution Rejection Rate | Undetected Mutants |
|---|---|---|
| Whoami | 80.00% (12/15) | P5,G4,G5 |
| Jawanndenn | 66.67% (10/15) | P4,P5,G2,G3,G4 |
| Hours | 73.33% (11/15) | H3,G1,G4,G5 |
| Booklogr | 86.67% (13/15) | G4,G5 |
| Booking Calendar | 80.00% (12/15) | P5,G2,G4 |
| Mean | 77.33% | — |
As shown in Table U, the generated cross-layer test suites reject 77.33% of the mutant implementations on average across the five websites, demonstrating their ability to detect most injected defects. Missed defects are concentrated in incorrect displayed values (G4), incomplete related writes (P5), and stale interface state (G5), which together account for 11 of the 17 undetected mutants. This suggests that these defects are more subtle and harder for the generated test suites to detect.
This provides independent evidence that the generated cross-layer test suites can distinguish many defective implementations from their correct references, beyond passing reference validation.
Response to [Q1-2]: False rejection of correct alternative implementations
We also conduct a frontend substitution experiment to assess whether our generated cross-layer test suites falsely reject semantically correct alternative implementations.
As defined in Table V, we design nine frontend transformations across four categories: language, controls, visual style, and element order. These transformations preserve application functionality while leaving the backend implementation, database, and API contracts unchanged.
Table V: Alternative frontend implementation categories and definitions
| Category | Transformation | Definition |
|---|---|---|
| Language | Simplified Chinese | Translate all interface text into Simplified Chinese while preserving user data, quotations, and backend protocol values. |
| Language | Arabic | Translate all interface text into Arabic and apply right-to-left text direction and alignment while preserving layout order. |
| Language | Japanese | Translate all interface text into Japanese while preserving user data, quotations, and backend protocol values. |
| Controls | Functional Region Substitution | Replace or reorganize groups of controls that serve the same function, allowing changes to the layout of the affected region while preserving their behavior, action labels, and identifying text. For example, replace a sorting dropdown with a labeled radio group. |
| Controls | Individual Control Substitution | Replace individual controls with functionally equivalent element types while closely preserving their appearance. For example, replace a navigation button with a styled hyperlink. Specialized controls may remain unchanged. |
| Visual Style | Dark and Compact | Apply dark backgrounds, light text, square outlined buttons, and compact spacing across all pages, favoring dense, list-based layouts. |
| Visual Style | Bright and Rounded | Apply light backgrounds, warm accent colors, rounded controls, and generous spacing across all pages, favoring centered, single-column layouts. |
| Element Order | Reversed | Reverse reorderable element groups across all pages while preserving functional dependencies, option values, and default selections. Reorder lists only within the current page. |
| Element Order | Seeded Shuffle | Shuffle reorderable element groups across all pages using a fixed random seed while preserving functional dependencies, option values, and default selections. Reorder lists only within the current page. |
We construct 9 alternative frontend implementations for each of three websites, Booking Calendar, Hours, and Booklogr, yielding 27 alternatives in total.
Unlike the mutation experiment, where each mutant implementation contains only one injected defect, each transformation is applied throughout all applicable pages, functional regions, and controls to cover as much website functionality as possible. After manually verifying functional correctness, we evaluate each alternative using the corresponding cross-layer test suites and compare its results with those of the reference implementation.
Table W: Test results on alternative frontend implementations
| Website | Reference | Chinese | Arabic | Japanese | Region substitution | Control substitution | Dark | Bright | Reversed | Shuffled |
|---|---|---|---|---|---|---|---|---|---|---|
| Booking Calendar | 12/13 | 12/13 | 12/13 | 12/13 | 12/13 | 12/13 | 12/13 | 12/13 | 12/13 | 12/13 |
| Hours | 26/26 | 26/26 | 26/26 | 25/26 | 26/26 | 26/26 | 26/26 | 26/26 | 26/26 | 26/26 |
| Booklogr | 40/40 | 40/40 | 40/40 | 40/40 | 40/40 | 40/40 | 40/40 | 40/40 | 40/40 | 40/40 |
As shown in Table W, only the Japanese version of Hours has one test incorrectly marked as failed among the 27 alternatives. All other versions achieve the same number of passing tests as their reference implementations. The error in Hours arises from a cross-language semantic alignment failure, where DeepSeek fails to correctly match a requirement in the English test workflow to the corresponding Japanese interface message. The one failing test in Booking Calendar also occurs on the reference implementation due to an environmental issue unrelated to the frontend transformations.
These results provide strong evidence that our generated cross-layer test suites reliably accept correct implementations despite substantial differences in language, controls, layout, and visual style, with nearly no false rejections.
Dear Reviewer u9de,
Thank you for your positive assessment and constructive feedback. Following your suggestions, we clarified implementation freedom within the task requirements and added experiments evaluating false acceptance of defective implementations and false rejection of correct alternatives.
We hope these additions address your remaining concerns and appreciate your support for our work. If any questions or concerns remain, please do not hesitate to let us know.
Best regards,
Authors of Submission 42863
Response to [W1]: Task constraints and task definition
We appreciate you raising this concern and recognizing that the disclosed requirements are legitimate task constraints.
WebProgramBench addresses two competing requirements in evaluating coding agents. 1) Preserving implementation freedom. 2) Enabling rigorous, execution-based evaluation. Our task design addresses this challenge by providing structural contracts that expose only the minimum information needed for executable evaluation.
To make our description of implementation freedom more precise, we have clarified that agents are free to choose programming languages, frameworks, and system architecture within the task requirements; see the Introduction and the caption of Table 8 in the revised manuscript.
We have also refined the relevant statements to explicitly state that our results and conclusions reflect reconstruction performance under the supplied endpoint and database contracts; see the Introduction, Section 4.2, and Conclusion.