acceptodds
Under review as a conference paper at ICLR 2027

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

Abstract

Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Static benchmarks can credit functionality that exists in source code but is unreachable at runtime. Interactive benchmarks exercise the application, yet incomplete exploration can cause them to miss implemented functionality and confound application defects with agent execution failures. To address these limitations, we propose WebCraftBench, an interactive benchmark for evaluating web application generation from a software testing perspective. WebCraftBench instruments each generated application and uses code coverage to guide an agent in exploring its functionality through user-simulated interactions. It then abstracts the interaction trace into a state-transition graph and evaluates the application along three dimensions: visual aesthetics, usability, and requirement alignment. By separating exploration from scoring, WebCraftBench collects runtime evidence without constraining exploration to predefined acceptance criteria. WebCraftBench comprises 369 real-world user requirements and 5,088 acceptance criteria. Evaluation of 17 frontier LLMs reveals distinct strengths across the three dimensions, with no model leading on every dimension. On 197 validated sessions sampled from an internal arena, WebCraftBench achieves 85.3% agreement with human preferences, with agreement generally increasing as the score difference between paired applications grows. Further experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.

Primary area:
datasets and benchmarks
Keywords:
Web Application Generation, Large Language Model, Benchmark, GUI Testing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZgR8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes WebCraftBench, an automated benchmark for evaluating LLM-generated web applications. The main idea is to use a testing agent to explore generated applications, build a state-transition graph from the interactions, and evaluate aesthetics, usability, and requirements with the help of code-coverage guidance.

Critical strengths
  • Evaluating generated web applications is an important and increasingly relevant problem.

  • Using interactions (instead of static screenshots or code) is reasonable and desirable.

  • The benchmark is evaluated on many models and shows relatively high agreement with human preferences (due to subjectiveness for some of these requirements, it's understandable that we do not have perfect agreement).

Critical weaknesses
  • The motivation and framing are difficult to follow. The abstract and introduction have too many ideas. E.g. human evaluation, static benchmarks, interactive benchmarks, and testing agents are all mentioned before what problem WebCraftBench is solving and what is new.

  • The novelty should be clarified and sharpened. LLM-based GUI testing agents and state-graph creation/exploration have already been studied extensively (e.g. in mobile application testing [1][2]). I believe framing this paper better is an important step that would be needed.

  • It is unclear how much the proposed method improves evaluation itself. There are no comparisons against existing automated evaluation methods on the same human preference data.

  • Similar to the previous point, it would be useful to show whether WebCraftBench actually changes conclusions about model performance. For example, do existing benchmarks overestimate functionality because they fail to explore applications sufficiently?

[1] Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing

[2] Towards Comprehensive Mobile Application Testing for User-centric Feature Coverage via Simulating Real-World Users with Personas

Questions for authors
  • What is the human-preference agreement of existing web-generation evaluation methods on the same 197 pairs?

  • How much do model scores/rankings change compared with existing evaluation approaches?

  • Can the authors better clarify the novelty compared to prior LLM-based GUI testing and state-graph exploration methods?

2 comments

Dear Reviewer, thank you again for your time and constructive feedback. We would appreciate hearing whether our responses have addressed your concerns. If any concerns remain or any points require further clarification, we would be happy to elaborate and continue the discussion. Thank you for your consideration!

I thank the authors for their response. The WebCoderBench comparison is interesting and it helps answer my question about agreement with human preferences and changes in model rankings. However, while the higher agreement is useful, I am still unsure what coverage guidance adds. For example, does it help the testing agent find bugs that the same agent would otherwise miss? I do not think this important analysis is clear.

Additionally, a big concern still remains for the framing and presentation. The abstract and introduction bring up several ideas before it is clear what problem the paper is trying to solve. The authors should explain more clearly what existing evaluators lack, and I believe a discussion for how this work relates to the mobile GUI testing is warranted.

Given these remaining concerns, I will keep my score.

Reviewer_fVCcOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WebCraftBench, a framework for evaluating generated web applications through coverage-guided exploration and runtime evidence. It separates exploration from scoring, organizes observations into a state-transition graph, and assesses visual aesthetics, usability, and requirement alignment. The benchmark contains 369 user requirements and 5,088 acceptance criteria and evaluates 17 models. Validation includes 85.3% agreement with human preferences on 197 validated application pairs, judge-replacement experiments, instrumentation reliability checks, and coverage-guidance ablations.

Critical strengths
  1. Well-motivated and coherent evaluation methodology. The paper clearly identifies the limitations of assessing generated applications through static evidence alone. Agent-based exploration grounded in runtime observations is an appropriate response to this problem. Coverage guidance, state abstraction, and the separation of exploration from scoring have clear purposes and form a coherent evaluation pipeline. This alignment between the problem and the methodology is a central strength.

  2. Substantial empirical evaluation with complementary validation. The evaluation covers 17 models across 369 requirements and examines multiple dimensions of application quality. Human-preference agreement, judge replacement, instrumentation reliability, and coverage-guidance experiments provide complementary evidence supporting the framework. The results are interpretable and reveal meaningful differences between visual quality, usability, and requirement satisfaction.

Critical weaknesses

Limited reproducibility and community accessibility. The decision not to release the dataset or implementation prevents independent reproduction and restricts the benchmark’s use for evaluating future models. This is consequential for a benchmark contribution. However, it does not by itself establish a methodological flaw or invalidate the reported findings. I regard it as a limitation rather than sufficient grounds for rejection.

Questions for authors

refer to weakness.

Reviewer_BwKuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WebCraftBench evaluates generated web applications using coverage-guided exploration and runtime evidence. It scores aesthetics, usability, and requirement alignment. The benchmark has 369 requirements and 5,088 acceptance criteria. The authors evaluate 17 model/harness configurations and report 85.3% agreement with validated human preferences on 197 pairs. Coverage guidance increases median function coverage from 91.9% to 94.3% on 360 applications.

Critical strengths

The method checks whether users can actually reach and use a feature. Combining exploration with recorded runtime evidence is a useful design for evaluating generated applications.

The experiments cover 6,273 applications. Agreement with validated human preferences on 168/197 separately sampled pairs and similar model rankings under two judges give meaningful support for the complete pipeline. Coverage guidance also provides a measured improvement in exploration.

Critical weaknesses

The added evaluation benefit is not clearly established. The coverage gain and 85.3% human agreement are useful results. However, without a comparison on the same applications, it is unclear whether coverage guidance improves judgments over the same pipeline without guidance. This limits the evidence for the main methodological addition. The rules for correcting human votes also need clarification.

Independent use of the benchmark is limited. Neither the dataset nor code will be released, and full prompts and some settings are missing. The methodological design can still be useful, but researchers have no clear way to reproduce this evaluation or test new models. This reduces the value of the benchmark contribution.

Questions for authors
  1. On the same applications, does coverage guidance improve human agreement or criterion-level accuracy compared with no guidance, under comparable budgets?

  2. How were human votes corrected? How many changed, were annotators blinded to automated scores, and what was the agreement among annotators?

  3. How are failed deployments and empty criterion sets scored? Nine Claude-Opus-5 applications were undeployable. If all 369 tasks are included and these failures receive zero, the maximum mean is 97.56%, but Table 6 reports Gemini alignment means of 99.2/98.9/99.5. Please clarify the denominators and failure handling.

  4. Can the authors provide evaluation code, prompts, a sanitized subset, or an evaluation service for independent use?

1 comment

Dear Reviewer, thank you again for your time and constructive feedback. We would appreciate hearing whether our responses have addressed your concerns. If any concerns remain or any points require further clarification, we would be happy to elaborate and continue the discussion. Thank you for your consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.