acceptodds
Under review as a conference paper at ICLR 2027

WebProgramBench: Can Coding Agents Reconstruct Live Applications from Scratch via Black-Box Interaction?

11 views

Abstract

Coding agents have recently achieved substantial progress in fixing bugs and implementing features within existing repositories, yet building complete real-world applications from scratch remains an open challenge. Existing software-development benchmarks assess only partial aspects of this capability through requirement-driven generation and screenshot-based interface cloning. Their evaluations emphasize visual similarity, selected functional checks, or model-based judgments, leaving room for shortcut solutions that pass evaluation. To address these limitations, we introduce WebProgramBench, the first execution-based benchmark for evaluating whether LLMs can reconstruct real-world live applications from scratch through black-box interaction. WebProgramBench requires agents to actively explore a live reference application and reconstruct its behavior in a deployable implementation, without access to its source code. To evaluate task performance, we introduce a novel cross-layer protocol that jointly verifies interface, service, and persistence behavior, preventing frontend-only mocks from passing. The benchmark comprises 20 real-world applications spanning four domains, with reference implementations totaling nearly one million lines of code. To scale this evaluation, our fully automated pipeline generates 1,254 verified executable tests with 20,851 assertions. Our experiments across five leading LLMs show that frontier LLMs remain far from complete real-world application reconstruction. Even the best-performing model achieves near-complete reconstruction on only three applications, only one of which is fully reconstructed. Together, these results establish WebProgramBench as a rigorous benchmark for measuring LLM capabilities in autonomous, closed-loop software engineering.

Primary area:
datasets and benchmarks
Keywords:
Coding agents, software engineering benchmarks, black-box application reconstruction, full-stack evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_u9deOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

WebProgramBench evaluates whether coding agents can reconstruct deployable web applications by exploring a live reference without access to its source repository. Agents receive HTTP endpoint and database contracts, then infer the remaining behavior and implement a frontend, backend, and persistent database.

The benchmark contains 20 open-source applications across four domains, with 1,254 executable tests and 20,851 assertions. Its central contribution is an ordered evaluation protocol that checks each browser interaction across three layers: interface outcomes, HTTP requests and responses, and database effects. An automated pipeline analyzes behavior dependencies, plans stateful executions, generates tests, and iteratively repairs them against the reference application. Candidate implementations are redeployed in clean environments before evaluation.

Experiments evaluate five models under a shared agent harness. Claude Opus 5 achieves the highest cross-layer test pass rate, 56.35%; one application passes every retained test, and three applications, including that one, reach at least 95%. Ablations examine verification layers, live-reference access, and agent harnesses. Additional experiments assess evaluator repeatability and sensitivity to the verifier model.

The results establish a challenging setting for discovering and implementing stateful application behavior. The main contribution is the benchmark and its integrated evaluation infrastructure, with evidence that live interaction provides information beyond the supplied contracts.

Critical strengths

1- The benchmark evaluates meaningful integration across application layers: I find the joint evaluation protocol to be the strongest aspect of the paper. Checking the interface outcome, HTTP exchange, and database effect of the same user interaction provides a more substantive assessment than checking whether an interface appears to work. The layer ablation illustrates the practical importance of this distinction: average test pass rate is 50.01% with interface-only evaluation, compared with 35.12% when all three layers are required.

2 - The construction pipeline addresses practical challenges in stateful evaluation: the test-construction pipeline explicitly handles dependencies between application behaviors, including resources created by earlier actions and consumed by later ones. This is a useful engineering contribution because evaluating workflows such as creating, editing, and deleting an object requires consistent identities and state across executions. Redeploying submissions in a separate evaluation environment with a fresh database further strengthens the assessment of whether the delivered application actually works.

Critical weaknesses

1 - The paper describes implementation choices as unconstrained, but the task requires specified HTTP methods and paths, PostgreSQL, and particular table names, column names, types, and primary keys (Appendix L). These requirements are disclosed, so they are legitimate task constraints. However, they substantially narrow the architectural freedom being evaluated. The task measures reconstruction under explicit API and persistence compatibility requirements.

Questions for authors

How accurate is the evaluator on independently assessed correct and incorrect implementations? Could you evaluate deliberately incomplete backends, faulty persistence implementations, and valid alternative frontends? This would help quantify false acceptance and false rejection beyond repeatability and agreement between judges.

4 comments

Response to [W1]: Task constraints and task definition

We appreciate you raising this concern and recognizing that the disclosed requirements are legitimate task constraints.

WebProgramBench addresses two competing requirements in evaluating coding agents. 1) Preserving implementation freedom. 2) Enabling rigorous, execution-based evaluation. Our task design addresses this challenge by providing structural contracts that expose only the minimum information needed for executable evaluation.

To make our description of implementation freedom more precise, we have clarified that agents are free to choose programming languages, frameworks, and system architecture within the task requirements; see the Introduction and the caption of Table 8 in the revised manuscript.

We have also refined the relevant statements to explicitly state that our results and conclusions reflect reconstruction performance under the supplied endpoint and database contracts; see the Introduction, Section 4.2, and Conclusion.

Response to [Q1-1]: False acceptance of defective implementations

We thank you for this valuable question. We conduct a mutation experiment to measure the invalid-solution rejection rate of our generated cross-layer test suites.

We conduct this experiment on five selected websites. As defined in Table T, we design 15 mutation operators across three categories: persistence errors, protocol errors, and GUI errors, with five operators per category. For each website, PhD-level experts construct 15 mutant implementations, each created by injecting exactly one defect into a separate copy of the correct reference implementation, and manually verify that each defect is valid.

Table T: Mutation operators used to construct invalid implementations

IDMutation OperatorDescription
Persistence errors
P1Omitted writeSkip a required database write while still returning success.
P2Incorrect field valueStore a type-valid but semantically incorrect value, such as a different valid status.
P3Wrong target recordUpdate or delete another existing record instead of the record selected by the user.
P4Incorrect ownership or associationCreate the correct business record but assign it to the wrong user or project. For example, save a reservation created by user A under user B.
P5Incomplete related writesUpdate the main record but omit required related writes, such as the corresponding history record.
Protocol errors
H1Incorrect status codeReturn an incorrect HTTP status code after a successful operation, such as returning 400 after successfully cancelling a reservation.
H2Malformed response structureRemove or rename a required field, or change the required response structure.
H3Incorrect response valueStore data correctly but return an incorrect field value, such as returning description as title.
H4Incorrect redirect targetModify Location to redirect to an incorrect, existing business page.
H5Incorrect authorization decisionReject a legitimate request by incorrectly requiring administrator privileges for an operation available to regular users.
GUI errors
G1Missing event handlerRemove an event handler so that a visible, clickable control performs no action.
G2Incorrect navigation targetNavigate to an incorrect, existing page, such as opening the settings page instead of the edit page.
G3Incorrect field bindingBind an input to the wrong field, such as submitting the title input as the description.
G4Incorrect displayed valueReturn the correct service response but display an incorrect value, such as the wrong reservation time.
G5Stale interface stateUpdate the backend successfully without synchronizing the interface, such as retaining a deleted record in the list.

Each of the 15 mutant implementations is evaluated using all cross-layer test suites generated for the corresponding website. A mutant is considered rejected when the injected defect causes a new test failure relative to the correct reference implementation. The invalid-solution rejection rate is the fraction of the 15 mutant implementations in which at least one cross-layer test fails.

Table U: Invalid-solution rejection rates across five websites

WebsiteInvalid-Solution Rejection RateUndetected Mutants
Whoami80.00% (12/15)P5,G4,G5
Jawanndenn66.67% (10/15)P4,P5,G2,G3,G4
Hours73.33% (11/15)H3,G1,G4,G5
Booklogr86.67% (13/15)G4,G5
Booking Calendar80.00% (12/15)P5,G2,G4
Mean77.33%—

As shown in Table U, the generated cross-layer test suites reject 77.33% of the mutant implementations on average across the five websites, demonstrating their ability to detect most injected defects. Missed defects are concentrated in incorrect displayed values (G4), incomplete related writes (P5), and stale interface state (G5), which together account for 11 of the 17 undetected mutants. This suggests that these defects are more subtle and harder for the generated test suites to detect.

This provides independent evidence that the generated cross-layer test suites can distinguish many defective implementations from their correct references, beyond passing reference validation.

Response to [Q1-2]: False rejection of correct alternative implementations

We also conduct a frontend substitution experiment to assess whether our generated cross-layer test suites falsely reject semantically correct alternative implementations.

As defined in Table V, we design nine frontend transformations across four categories: language, controls, visual style, and element order. These transformations preserve application functionality while leaving the backend implementation, database, and API contracts unchanged.

Table V: Alternative frontend implementation categories and definitions

CategoryTransformationDefinition
LanguageSimplified ChineseTranslate all interface text into Simplified Chinese while preserving user data, quotations, and backend protocol values.
LanguageArabicTranslate all interface text into Arabic and apply right-to-left text direction and alignment while preserving layout order.
LanguageJapaneseTranslate all interface text into Japanese while preserving user data, quotations, and backend protocol values.
ControlsFunctional Region SubstitutionReplace or reorganize groups of controls that serve the same function, allowing changes to the layout of the affected region while preserving their behavior, action labels, and identifying text. For example, replace a sorting dropdown with a labeled radio group.
ControlsIndividual Control SubstitutionReplace individual controls with functionally equivalent element types while closely preserving their appearance. For example, replace a navigation button with a styled hyperlink. Specialized controls may remain unchanged.
Visual StyleDark and CompactApply dark backgrounds, light text, square outlined buttons, and compact spacing across all pages, favoring dense, list-based layouts.
Visual StyleBright and RoundedApply light backgrounds, warm accent colors, rounded controls, and generous spacing across all pages, favoring centered, single-column layouts.
Element OrderReversedReverse reorderable element groups across all pages while preserving functional dependencies, option values, and default selections. Reorder lists only within the current page.
Element OrderSeeded ShuffleShuffle reorderable element groups across all pages using a fixed random seed while preserving functional dependencies, option values, and default selections. Reorder lists only within the current page.

We construct 9 alternative frontend implementations for each of three websites, Booking Calendar, Hours, and Booklogr, yielding 27 alternatives in total.

Unlike the mutation experiment, where each mutant implementation contains only one injected defect, each transformation is applied throughout all applicable pages, functional regions, and controls to cover as much website functionality as possible. After manually verifying functional correctness, we evaluate each alternative using the corresponding cross-layer test suites and compare its results with those of the reference implementation.

Table W: Test results on alternative frontend implementations

WebsiteReferenceChineseArabicJapaneseRegion substitutionControl substitutionDarkBrightReversedShuffled
Booking Calendar12/1312/1312/1312/1312/1312/1312/1312/1312/1312/13
Hours26/2626/2626/2625/2626/2626/2626/2626/2626/2626/26
Booklogr40/4040/4040/4040/4040/4040/4040/4040/4040/4040/40

As shown in Table W, only the Japanese version of Hours has one test incorrectly marked as failed among the 27 alternatives. All other versions achieve the same number of passing tests as their reference implementations. The error in Hours arises from a cross-language semantic alignment failure, where DeepSeek fails to correctly match a requirement in the English test workflow to the corresponding Japanese interface message. The one failing test in Booking Calendar also occurs on the reference implementation due to an environmental issue unrelated to the frontend transformations.

These results provide strong evidence that our generated cross-layer test suites reliably accept correct implementations despite substantial differences in language, controls, layout, and visual style, with nearly no false rejections.

Dear Reviewer u9de,

Thank you for your positive assessment and constructive feedback. Following your suggestions, we clarified implementation freedom within the task requirements and added experiments evaluating false acceptance of defective implementations and false rejection of correct alternatives.

We hope these additions address your remaining concerns and appreciate your support for our work. If any questions or concerns remain, please do not hesitate to let us know.

Best regards,
Authors of Submission 42863

Reviewer_rC3nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WebProgramBench, a benchmark for evaluating coding agents' ability to reconstruct stateful web applications from scratch by exploring live reference websites. Agents are given HTTP endpoint and database schema contracts but cannot access the reference source code or hidden tests. The benchmark includes 20 applications, 1,254 tests, and 20,851 assertions, jointly verifying UI functionality, HTTP requests and responses, and database changes for the same interaction. The paper evaluates five models, with Claude Opus 5 achieving the highest average test pass rate of 56.35%. The experiments also examine the effects of access to live reference websites, agent frameworks, and evaluator stability.

Critical strengths
  1. Combining live behavioral discovery with full-stack reconstruction has practical value. The benchmark requires agents to discover the remaining behavioral specifications through interaction, subject to given endpoint and database schema constraints, and to build independently deployable implementations from empty workspaces. Compared with developing systems from complete requirements documents, this setup introduces an additional requirement for behavioral exploration. In Table 3, removing access to the live reference websites reduces the average pass rate on the evaluated subset from 35.98% to 17.66%, supporting the role of information obtained through interaction in this setting.

  2. The design of state-dependent tests and joint verification is valuable. The benchmark jointly evaluates UI outcomes, HTTP requests and responses, and database changes for the same interaction, while passing required resources between ordered tests. The automated construction pipeline combines behavioral analysis, execution planning, test generation, and validation against the reference applications, reducing the need for manually written tests. Fresh deployments in isolated environments, framework ablations, and evaluator stability analyses further strengthen the completeness of the experimental evaluation.

Critical weaknesses
  1. The distinct contribution and the construction pipeline's added value are insufficiently established. ProgramBench [1] already includes black-box reconstruction and automated testing, SaaSBench [2] includes complete system construction and dependency-graph-based evaluation, and FullStack-Bench [3] includes functional verification across multiple layers. Table 6 omits SaaSBench, and the paper needs to clarify what live behavioral discovery, dependency planning, and joint evaluation each add beyond these settings. The pipeline's empirical benefits also remain unclear: the reported 89.3% validated execution coverage uses only planned executions as its denominator, excluding omitted or unplanned behaviors. Figure 4 demonstrates the effect of iterative repair but does not isolate the additional coverage and defect detection provided by dependency modeling and state planning over direct test generation under the same budget.

  2. The joint evaluator's ability to detect defects has not been independently validated. Figure 3 uses the full three-layer evaluator as the reference for identifying "false acceptances." The lower pass rates therefore primarily indicate stricter constraints and do not independently establish greater judgment accuracy. The paper lacks tests on implementations with known defects and does not distinguish business logic errors, persistence errors, and protocol differences among the additional failures. It thus remains unclear which evaluation blind spots the three-layer joint verification actually addresses.

    [1] ProgramBench: Can Language Models Rebuild Programs From Scratch?

    [2] SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering

    [3] FullStack-Agent: Enhancing Agentic Full-Stack Web Coding via Development-Oriented Testing and Repository Back-Translation

Questions for authors
  1. Positioning of the contribution: Could you add a direct comparison with ProgramBench, SaaSBench, and FullStack-Bench in terms of input information, implementation constraints, dependency handling, and what is verified, and clarify the additional capability dimensions introduced by this work? ProgramDistill [4] also includes complete reconstruction guided by live reference applications; please discuss its relationship to this work in light of its public release date.

  2. Evaluator validation experiment: Could you introduce known defects into representative applications, such as disabling persistence, writing data for the wrong user, or breaking permission checks, and compare the detection rates of single-layer, two-layer, and three-layer evaluators? Please also include control implementations that preserve the specified behavior to check for false rejections, and categorize the existing candidate cases rejected only by the third layer.

  3. Controlled comparison of the construction pipeline: Could you compare the full pipeline with a baseline of "direct test generation + the same iterative repair" using the same model, source code access, and construction budget? In addition to validation pass rates, please report independently defined behavioral/workflow coverage, defect detection rates, and construction costs to distinguish the gains from test repair from those attributable to dependency planning.

    [4] ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

9 comments

Hi, I noticed that you haven’t responded yet, and I’d be happy to discuss these points with you. If addressing all the questions at once would take too much time, please feel free to respond to some of them first, and we can start from there.

Response to [W1-1] and [Q1]: Direct Comparison

We thank you for this valuable question. ProgramDistill was released in September 2026, shortly before our submission; we consider it concurrent work.

Table J: Comparison with related benchmarks

WorkInput informationImplementation constraintsDependency handlingWhat is verified
ProgramBenchExecutable binary + usage documentation + task-specific assetsMust reconstruct a reference executable binary from scratch in an offline environment (Rust, Python, Go, and C/C++), reproducing its documented interfaces and observable behavior.The paper does not explicitly describe inter-test dependency modeling.Observable program outputs, exit codes, and generated files.
SaaSBenchDetailed PRD (avg. 28,304 words): languages, frameworks, middleware, component versions, data models, business rules, APIs, and UI pages + ambiguity-resolution knowledge baseImplementations must conform to the prescribed technology stack, component versions, and detailed data models in the requirements document.Human–agent collaboration to construct dependency DAGs.Functional requirements across UI, APIs, and database state.
FullStack-BenchBrief task description (avg. 72.4 words) + shared development requirementsApplications are built from predefined JS/TS templates, constraining language and framework choices.No explicit modeling of dependencies between test cases; test prerequisites are handled by the evaluation agent.Whether the application satisfies the user instruction.
ProgramDistillPartial-application reconstruction: A masked application repository + feature-specific task description + live reference. Full-application reconstruction: A preconfigured scaffold without a database + product specification + live reference.Partial reconstruction is constrained by the existing language, framework, and surrounding application implementation, with function signatures and route registrations preserved during masking; full reconstruction uses a preconfigured JavaScript scaffold, limiting freedom to choose the implementation language and runtime architecture.LLM-proposed parent-trace dependencies, with prerequisite states established by replaying ancestor traces in order.Whether replayed interaction traces match reference outcomes.
WebProgramBench (Ours)Live stateful reference website + brief task description + endpoint and database contracts + account credentialsMust reconstruct a live reference web application from scratch in an Internet-enabled environment with source-access restrictions (JS/TS, Python, Go, Java, Rust, Ruby, PHP, and C/C++), reproducing its frontend, backend, and database behavior while conforming to endpoint and database contracts.Fully automated construction of an execution-dependency graph without human intervention, with resource sharing through a runtime variable pool.Joint UI outcomes, HTTP exchanges, and database effects for the same user interaction.

Additional capabilities

  • Beyond ProgramBench: WebProgramBench extends reconstruction from executable binaries to live, stateful applications, evaluating agents’ ability to discover behavior through multi-step UI interactions, establish prerequisite states, and infer dependencies between actions. Reconstruction further requires implementing and integrating the frontend, backend services, and persistent database behavior.

  • Beyond FullStack-Bench: WebProgramBench extends brief-instruction-driven development to reconstruction from a live reference, evaluating agents’ ability to discover unspecified behavior through stateful UI exploration and infer the business rules and dependencies needed to reproduce it. Agents must also independently select implementation languages, frameworks, and system architectures rather than develop within predefined templates.

  • Beyond ProgramDistill: WebProgramBench evaluates agents’ ability to build and deploy complete systems from scratch, with independent choices of programming languages and system architectures, rather than work within an existing repository or a preconfigured scaffold.

  • Beyond SaaSBench: WebProgramBench automatically constructs cross-layer test suites and execution-dependency graphs without expert annotation, whereas SaaSBench relies on human–agent collaboration to construct test dependencies. SaaSBench uses validation nodes to check API behavior, database state, authentication, and UI outcomes separately, with each node executing a chain of testing primitives. Such checks do not necessarily establish that UI actions trigger the required backend requests, so an endpoint may pass direct API checks while remaining disconnected from the UI. In contrast, WebProgramBench’s cross-layer test suites capture and verify the HTTP exchange triggered by the UI action, together with its UI outcome and corresponding database effects, detecting such frontend–backend integration failures.

Response to [W2-1] and [Q2-1]: Defect detection and false acceptance(Part1)

We thank you for this valuable question. We conduct a mutation experiment to compare the defect detection rates of single-layer, two-layer, and three-layer verification using our generated cross-layer test suites.

We conduct this experiment on five selected websites. As defined in Table K, we design 15 mutation operators across three categories: persistence errors, protocol errors, and GUI errors, with five operators per category. For each website, PhD-level experts construct 15 mutant implementations, each created by injecting exactly one defect into a separate copy of the correct reference implementation, and manually verify that each defect is valid.

Table K: Mutation operators used to construct invalid implementations

IDMutation OperatorDescription
Persistence errors
P1Omitted writeSkip a required database write while still returning success.
P2Incorrect field valueStore a type-valid but semantically incorrect value, such as a different valid status.
P3Wrong target recordUpdate or delete another existing record instead of the record selected by the user.
P4Incorrect ownership or associationCreate the correct business record but assign it to the wrong user or project. For example, save a reservation created by user A under user B.
P5Incomplete related writesUpdate the main record but omit required related writes, such as the corresponding history record.
Protocol errors
H1Incorrect status codeReturn an incorrect HTTP status code after a successful operation, such as returning 400 after successfully cancelling a reservation.
H2Malformed response structureRemove or rename a required field, or change the required response structure.
H3Incorrect response valueStore data correctly but return an incorrect field value, such as returning description as title.
H4Incorrect redirect targetModify Location to redirect to an incorrect, existing business page.
H5Incorrect authorization decisionReject a legitimate request by incorrectly requiring administrator privileges for an operation available to regular users.
GUI errors
G1Missing event handlerRemove an event handler so that a visible, clickable control performs no action.
G2Incorrect navigation targetNavigate to an incorrect, existing page, such as opening the settings page instead of the edit page.
G3Incorrect field bindingBind an input to the wrong field, such as submitting the title input as the description.
G4Incorrect displayed valueReturn the correct service response but display an incorrect value, such as the wrong reservation time.
G5Stale interface stateUpdate the backend successfully without synchronizing the interface, such as retaining a deleted record in the list.

Each of the 15 mutant implementations is evaluated using all cross-layer test suites generated for the corresponding website. A mutant is considered rejected when the injected defect causes a new test failure relative to the correct reference implementation. The invalid-solution rejection rate is the fraction of the 15 mutant implementations in which at least one cross-layer test fails.

Table L: Defect detection across verification-layer combinations. I: Interface; S: Service; P: Persistence.

WebsiteISPI+SI+PS+PI+S+P
Whoami9/158/158/1511/1511/1511/1512/15
Jawanndenn8/158/157/1510/158/1510/1510/15
Hours9/155/157/1510/1510/159/1511/15
Booklogr10/159/158/1511/1513/1511/1513/15
Booking Calendar6/158/158/1510/1510/1511/1512/15
ALL42/7538/7538/7552/7552/7552/7558/75

As shown in Table L, interface, service, and persistence verification detect 56.00% (42/75), 50.67% (38/75), and 50.67% (38/75) of mutants, respectively. Each two-layer combination detects 69.33% (52/75) of mutants, while full three-layer verification achieves the highest detection rate of 77.33% (58/75), consistent with the cross-layer verification ablation in Figure 3 of the manuscript.

Reviewer_Tp3WOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WebProgramBench, a benchmark for reconstructing functioning web applications from black-box access to a live reference and sanitized HTTP endpoint/database-schema contracts. An automated, source-guided construction pipeline produces ordered tests that invoke functionality through the browser and verify interface behavior, HTTP exchanges, and persistent database effects in conjunction.

The benchmark contains 20 open-source applications, 1,254 executable cross-layer tests, and 20,851 assertions. Experiments with five coding models report substantial failures in end-to-end reconstruction and demonstrate that single-layer verification yields more permissive scores. The paper also studies reference access, agent harness choice, evaluator repeatability, and coding trajectories.

Critical strengths

Meaningful problem setting. Agents must infer stateful behavior from a running reference and produce an independently deployable application; the reference source and hidden tests are isolated from the solver (Secs. 3.1 and 3.3; Appendix F).

Substantive executable evaluation. The ordered, three-layer tests distinguish UI-level imitation from correct service and persistence semantics (Sec. 3.2). The 1,254 tests and 20,851 assertions are a nontrivial engineering effort, and the layer-removal ablation illustrates why joint verification matters (Fig. 3).

Useful complementary analyses. Live-reference removal, harness comparisons, repeated fixed-candidate scoring, and trajectory analysis help characterize what the benchmark measures (Tables 3–5; Sec. 5).

Critical weaknesses

Test validity is not the same as behavioral coverage. The reported 89.3% “validated execution coverage” measures the fraction of pipeline-planned executions that pass on the reference, and the pipeline retains only validated executions (Sec. 5.1; Appendix A, Eq. 5). It does not measure the fraction of real application behaviors covered, nor whether the final oracle rejects plausible incorrect implementations. A test suite that misses a difficult behavior can still achieve high validated execution coverage. Thus, statements about high coverage or complete application reconstruction should be restricted to the evaluated test suite.

Insufficient external validation of the GUI-agent oracle. Interface-layer checking relies on an LLM-based GUI verifier, rather than deterministic browser assertions alone (Appendix E). The five-repeat experiment checks one fixed candidate on each of five websites, with 96.13% agreement on Neith (Table 5; Appendix G). Agreement and cross-model score similarity do not establish correctness against expert judgments or robustness to semantically equivalent interfaces. A blinded human audit and deliberate valid/invalid controls would be valuable.

Strong task assumptions limit the scope of “black-box reconstruction.” Solvers receive all required HTTP methods/paths and the PostgreSQL table/column/type/primary-key contract (Sec. 3.1; Appendix L). This is a reasonable design choice, but it provides privileged structural information and constrains implementation choices. Results should not be generalized to unconstrained reconstruction from browser observation alone.

Limited uncertainty analysis and heterogeneous task difficulty. The primary comparison appears to use one development trajectory per model–application pair, with only 20 applications and 11–154 tests per application (Table 1). The reference-access and harness ablations cover five selected websites. More repeated trajectories, confidence intervals, and per-application results would clarify the stability of model rankings and the solved/almost-solved statistics.

Questions for authors

Can the authors audit a representative sample of endpoints/workflows independently of the test-generation agent and report behavioral recall, invalid-solution rejection rates, and false-rejection rates on semantically correct alternative implementations?

What fraction of interface-verifier decisions agree with blinded human adjudication? How often do different browser paths or semantically equivalent controls cause a valid candidate to fail?

Can the main model comparison be repeated over multiple agent seeds or runs, with uncertainty intervals and per-website performance, especially for the almost-solved threshold?

How does performance change when endpoint/schema contracts are partially or wholly withheld? This would distinguish the effect of exploration from the value of supplied structural contracts.

8 comments

Response to [W1]: Evaluation scope and validated execution coverage

We thank you for raising this valuable question, which helps us make the manuscript more precise. We agree that our evaluation assesses behavioral equivalence within the scope of these cross-layer test suites. Likewise, the reported 89.3% validated execution coverage refers to the proportion of planned executions that pass validation on the reference websites. We have revised the manuscript to explicitly scope our evaluation to the behaviors covered by the generated cross-layer test suites.

Response to [W3]: Task constraints and task definition

We appreciate you raising this concern and agree that providing these contracts is “a reasonable design choice.”

WebProgramBench addresses two competing requirements in evaluating coding agents. 1) Preserving implementation freedom. 2) Enabling rigorous, execution-based evaluation. Our task design addresses this challenge by providing structural contracts that expose only the minimum information needed for executable evaluation. Beyond these external contracts, agents independently make every engineering decision and handle every implementation detail, including their choice of programming languages, module organization, and function names.

To make the scope of our results and conclusions more precise, we have revised the relevant statements in the Introduction, Section 4.2, and Conclusion of the manuscript to explicitly state that our results reflect reconstruction performance under the supplied endpoint and database contracts.

Response to [Q1-1]: Test coverage

We thank you for raising this valuable question.

In the selected websites, endpoints connect frontend interactions to backend logic. Therefore, complex workflows can be decomposed into sequences of endpoint invocations. Based on this observation, we treat endpoints as observable interfaces and basic testable units.

To provide a more comprehensive assessment of test coverage, we conduct an independent human audit of five selected websites and operationalize behavioral recall at the operation level, following prior API-testing work [1]. Three PhD students in computer science independently review the reference source code and interact with five live reference websites to establish an operation inventory for each website, defining each operation as a distinct HTTP method–path pair. Disagreements are resolved by majority vote.

We then measure test coverage as the proportion of audited operations covered by the generated cross-layer test suites. Validated execution coverage measures the proportion of planned executions that pass reference validation, whereas test coverage measures the proportion of independently audited operations covered by the validated tests.

Table A: Test coverage audit

WebsiteOperations Covered by TestsTotal Audited OperationsTest Coverage
whoami396262.90% (39/62)
jawanndenn334082.50% (33/40)
hours346651.52% (34/66)
booklogr264360.47% (26/43)
booking-calendar222588.00% (22/25)

As shown in Table A, the generated cross-layer test suites achieve approximately 70% mean coverage across the five audited websites, providing broad coverage of the reference websites’ basic functionality. These results suggest that generating high-quality cross-layer tests while achieving nearly 100% coverage remains challenging. However, even within the current test scope, the best of the five evaluated models achieves a mean Pass Rate of 63.67% across the five audited websites, suggesting that testing more functional details could reveal further reconstruction failures. To reflect these findings, we have revised the Limitations section of the manuscript in Appendix D.

[1] DeepREST: Automated Test Case Generation for REST APIs Exploiting Deep Reinforcement Learning. ASE 2024.

Response to [Q1-2]: Invalid-solution rejection rate

We further conduct a mutation experiment to measure the invalid-solution rejection rate of our generated cross-layer test suites. We conduct this experiment on five selected websites. As defined in Table B, we design 15 mutation operators across three categories: persistence errors, protocol errors, and GUI errors, with five operators per category. For each website, PhD-level experts construct 15 mutant implementations, each created by injecting exactly one defect into a separate copy of the correct reference implementation, and manually verify that each defect is valid.

Table B: Mutation operators used to construct invalid implementations

IDMutation OperatorDescription
Persistence errors
P1Omitted writeSkip a required database write while still returning success.
P2Incorrect field valueStore a type-valid but semantically incorrect value, such as a different valid status.
P3Wrong target recordUpdate or delete another existing record instead of the record selected by the user.
P4Incorrect ownership or associationCreate the correct business record but assign it to the wrong user or project. For example, save a reservation created by user A under user B.
P5Incomplete related writesUpdate the main record but omit required related writes, such as the corresponding history record.
Protocol errors
H1Incorrect status codeReturn an incorrect HTTP status code after a successful operation, such as returning 400 after successfully cancelling a reservation.
H2Malformed response structureRemove or rename a required field, or change the required response structure.
H3Incorrect response valueStore data correctly but return an incorrect field value, such as returning description as title.
H4Incorrect redirect targetModify Location to redirect to an incorrect, existing business page.
H5Incorrect authorization decisionReject a legitimate request by incorrectly requiring administrator privileges for an operation available to regular users.
GUI errors
G1Missing event handlerRemove an event handler so that a visible, clickable control performs no action.
G2Incorrect navigation targetNavigate to an incorrect, existing page, such as opening the settings page instead of the edit page.
G3Incorrect field bindingBind an input to the wrong field, such as submitting the title input as the description.
G4Incorrect displayed valueReturn the correct service response but display an incorrect value, such as the wrong reservation time.
G5Stale interface stateUpdate the backend successfully without synchronizing the interface, such as retaining a deleted record in the list.

Each of the 15 mutant implementations is evaluated using all cross-layer test suites generated for the corresponding website. A mutant is considered rejected when the injected defect causes a new test failure relative to the correct reference implementation. The invalid-solution rejection rate is the fraction of the 15 mutant implementations in which at least one cross-layer test fails.

Table C: Invalid-solution rejection rates across five websites

WebsiteInvalid-Solution Rejection RateUndetected Mutants
Whoami80.00% (12/15)P5, G4, G5
Jawanndenn66.67% (10/15)P4, P5, G2, G3, G4
Hours73.33% (11/15)H3, G1, G4, G5
Booklogr86.67% (13/15)G4, G5
Booking Calendar80.00% (12/15)P5, G2, G4
Mean77.33%—

As shown in Table C, the generated cross-layer test suites reject 77.33% of the mutant implementations on average across the five websites, demonstrating their ability to detect most injected defects. Missed defects are concentrated in incorrect displayed values (G4), incomplete related writes (P5), and stale interface state (G5), which together account for 11 of the 17 undetected mutants. This suggests that these defects are more subtle and harder for the generated test suites to detect.

This provides independent evidence that the generated cross-layer test suites can distinguish many defective implementations from their correct references, beyond passing reference validation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.