acceptodds
Under review as a conference paper at ICLR 2027

EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

Abstract

Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily evaluate development toward fixed specifications. We introduce EvoWeb, a novel benchmark comprising 320 cumulative requests over 32 web applications, grounded in real-world product features and structured by functional dependency graphs. Agents iteratively modify a persistent codebase, while a multimodal web agent tests each intermediate application in a live browser, assessing newly requested functionality, cross-request integration, and previously requested functionality. To characterize reliability throughout development, we integrate trajectory performance across user-tolerance thresholds rather than impose a fixed stopping criterion. Experiments across open- and closed-source frontier models show that EvoWeb reveals clear performance differences between models, with cross-request integration emerging as a key challenge. Further analysis shows that later requests with deeper or more numerous dependencies are solved less accurately, while runtime errors become more frequent over successive requests. EvoWeb provides a testbed for evaluating and improving coding agents in long-horizon, multi-request development, with code and data to be released on GitHub.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Long-Horizon Software Development, Web Application Development, Multi-Request Development, Agent Evaluation, Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_B9KbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoWeb evaluates coding agents on 32 web applications containing 320 cumulative requests organized by functional dependency graphs. Agents modify persistent codebases, while browser evaluation checks new functionality, cross-request integration, and sampled earlier functionality. Tolerance-averaged trajectory metrics characterize development performance. Experiments across 14 models reveal integration difficulties and lower accuracy with greater dependency complexity. The primary contribution is an executable benchmark supported by construction checks, human evaluator validation, and robustness analyses.

Critical strengths
  1. The benchmark addresses a realistic setting in which successive user requirements must work together within an evolving application. Persistent codebases and conversation histories support evaluation of cumulative development.
  2. Dependency graphs and separate functional, integration, and backward tests provide an actionable evaluation structure. Explicit integration checks capture a meaningful capability beyond isolated feature implementation.
  3. Construction and validation are reasonably thorough. Product grounding, automated verification, human curation, and a 300-execution human audit support benchmark quality; alternative judges largely preserve broad model ordering.
  4. The multi-model study provides useful capability characterization, including integration gaps, dependency-related difficulty, and distinct build and runtime failures. Repeated closed-source runs help qualify the comparisons.
Critical weaknesses
  1. The empirical findings are useful but mostly intuitive. Dependency complexity, feature difficulty, and prior implementation quality remain partly entangled, limiting explanations of why agents fail.
  2. The trajectory metric’s additional value needs stronger justification. Mean accuracy and Grade produce similar rankings, while uniform tolerance weighting and the assumed abandonment behavior lack empirical user validation.
  3. Evaluator uncertainty is not sufficiently propagated into trajectory scores. Human auditing identifies integration-score bias, and early grading errors can affect subsequent contributions. Quantifying these effects would strengthen confidence in absolute scores and close comparisons.
  4. Retention conclusions are limited by backward-test sampling. One earlier test per follow-up request is not a comprehensive regression audit, and failure may reflect functionality never implemented. The paper acknowledges this tradeoff, but preservation claims should remain correspondingly narrow.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Author response

Thank you for the review. We added experiments addressing integration difficulty, evaluator uncertainty, and backward-test coverage.

W1. Agent failures.

Our main finding is that agents struggle to make new features work with what they have already built. We added two analyses of existing runs to examine this.

1. Compare tests under the same conditions. We compare the integration score with the average of the three functional scores for the same request and app version. The same request fixes dependency complexity and feature difficulty at the request level; the same app version fixes prior implementation quality.

We then account for differences between the tests: steps, exact-value checks, and check category. Integration scores remain lower: −0.045, 95% CI [−0.074, −0.019], with intervals from resampling applications.

2. Check cases where prerequisite tests pass. We select cases where all three new-feature tests and a backward test of a prerequisite pass. Among 1,139 such cases, 18.7% of integration tests still receive PARTIAL or NO. Thus, passing these feature checks does not ensure that the features work together.

Pinpointing the exact root cause of a failure in long agent trajectories remains an open research problem [1, 2, 3]. EvoWeb contributes to it by making these failures measurable, and the paper already characterizes them through app-side failure modes (Table 5) and build vs. runtime errors (Sec. 5.2).

[1] Zhang et al. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. ICML 2025

[2] Chen et al. Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios. 2025

[3] Mehtiyev & Assunção. Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure. 2026

W2. Grade’s additional value.

Grade captures when failures occur, which mean accuracy ignores. The trajectories (0.5, 1, …, 1) and (1, …, 1, 0.5) have the same mean accuracy (0.95). However, any user with tolerance above 0.5 abandons the first after request 1, so its Grade is 0.50 versus 0.95 (Sec. 3.4; App. H). This matters in practice: close models change order (e.g., GPT 5.6 Sol vs. Opus 4.8), and Grade separates models more clearly. Haiku 4.5 reaches 75% of Fable 5's mean accuracy but only 54% of its Grade (Table 13).

Uniform weighting and the abandonment model. These are modeling assumptions, but they do not hide the alternatives. Any other weighting gives a score between mean accuracy Q(0) and the fail-stop score Q(1), both reported in Table 13, and Fig. 3a (Fig. 6 for open-source models) shows results across the full tolerance range. Validating the abandonment model with real users would require humans to build and evaluate applications with each agent, which is very costly at this scale. We will clarify this in Sec. 3.4.

W3. Propagating evaluator uncertainty.

We re-judged 8,190 tests from five model runs twice more on identical snapshots. Grade SD is ≤ 0.009, with unchanged rankings.

We also recomputed complete trajectories in 1,000 audit-calibrated label-resampling draws (using verdict distributions estimated from the 300-test human audit), conditioning on judge verdict and test type. This propagates early verdict changes through later contributions and incorporates the audit’s test-type-specific discrepancies. Grade SDs are 0.023–0.027, with Kendall τ = 0.94 [0.87, 1.00].

Correcting these errors reorders only near-ties: models more than 0.05 apart keep their order in at least 97.5% of draws. Since the judge is lenient on integration tests (+0.050; Table 10), correction would widen the integration gap, not remove it.-

We will add these evaluator-uncertainty analyses to App. D.3 and application-bootstrap confidence intervals to App. F.

W4. Backward-test sampling and evaluation cost.

Each backward test requires a separate browser execution and WebVoager agent steps. We use one earlier test per follow-up request as a sampled estimate of earlier functionality, limiting evaluation time and cost.

To measure the effect of sampling, we re-evaluated all four tests from the same selected earlier request, using three runs per model across all 32 applications.

ModelBackward accuracy: one → fourBackward Grade: one → four
Haiku 4.50.726 → 0.7110.373 → 0.369
Sonnet 4.60.899 → 0.8590.597 → 0.579
Opus 4.80.899 → 0.8620.629 → 0.607

The one-test backward Grades match Table 4. With broader coverage, the model ordering is unchanged and backward Grade changes by at most 0.022. This measures the effect of the sampling choice directly.

We will make the sampling interpretation explicit and add this comparison to App. D.1. Preservation statements concern the earlier functionality tested; a failure counts as an apparent regression only when that same test passed previously.

Thanks for your response, I tend to keep my positive score.

Reviewer_idJ8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a EvoWeb bench to evaluation agents on implementation evolving multi-request web dev tasks. The task creation pipeline is carefully designed and includes verification of human experts. It grounds requirements to real web apps and build a DAG of requirements. Multiple frontier models are evaluated to show their strengths and weaknesses in such tasks.

Critical strengths
  1. This work designs a reasonable pipeline to create multi-request web dev tasks, which grounds requirements organized in DAGs to real web apps/sites and employs WebVoyager to test the dev results in an interactive way as is in WebGen-Bench.
  2. Analysis of failure modes, though being a little bit too simple, are helpful to identify capability drawbacks of existing models. Others like those on languages, user tolerance, etc. are also potentially useful for subsequent research.
Critical weaknesses

The details of the verification part need to be complemented and described more clearly.

  • How a complete test suit covering the three categories is build for each task? The supp provides some information, but it is not easy to build a whole picture for this part.
  • Since there are three categories, what is the distribution of the UTs in terms of the categories, functional points and, say, "failure modes"? The distribution of tests will influence the distribution of failure modes, thus determining the reliability of the corresponding conclusions, i.e. there will be a large portion of a certain failure mode if most of the tests concern this mode.
  • The evaluation results are sampled and checked by human experts. But are the verifiers themselves checked by human experts? The WebVoyager is given an instruction to test a certain functionality, but the accuracy and coverage of the instructions do not seem to be guaranteed. Besides, how many instructions are generated to test one function request? Please inform the reviewer if something is missed.
  • Are the evaluation results given by a judge stable across multiple runs?

The evaluation coverage of harness is limited.

  • It seems only OpenHands is used in evaluation without considering the official harnesses of some evaluated models like Codex, Claude Code, Kimi Code, ZCode, DSH, etc. It would make the experimental results more convincing if more harnesses are included in the evaluation.
  • Does OpenHands allow agents to view images? The OpenHands is tailed for general coding at the very beginning and does not support to supply website snapshots to agents. But snapshots and browser use capabilities are particularly important for agents executing a web dev task. If the harness is limited, the capabilities of models will be correspondingly bounded.

Explanation or analysis of some experimental settings. For example, why limit the turn to be no more than 20?

Currently the pipeline seems to depend on WebGen-Bench to provide an initial task. This work will be endowed with broader impact if it can generate such complex tasks from scratch in a scalable way.

How is the difficulty of each request ensured and measured? Currently the difficulty is mainly considered from request dependency. How about the implementation difficulty of the function itself after eliminating the influence of the dependency. Besides, what are the distributions of failures that are due to prior request failures and that are due to a difficult request itself? The effects of accumulated historical failures need to analyzed separately.

Questions for authors

Is it possible to extend the pipeline to general app dev or coding tasks? Will the generated task useful for RL? Such extension could make this work much more impactful.

2 comments
AuthorsOpenReview ↗

Author response (1/2)

Thank you for the review. We added experiments addressing test coverage, judge stability, and harness choice.

W1. Verification, test coverage, and judge stability.

“How a complete test suit covering the three categories is build for each task? “

As stated in §3.1 and App. D.1, each follow-up has three functional tests, one integration test, and one backward test that repeats an earlier test verbatim. Initial requests retain their WebGen-Bench tests, giving 1,638 tests in total. The construction and verification steps are described in §3.1 and App. D.1–D.2: requests and tests are generated without seeing an implementation, checked by a separate verifier, and reviewed by humans.

“what is the distribution of the UTs in terms of the categories…”

The following breakdown relates failure counts to test coverage. Exposure categories overlap.

Failure modeTests exposing itFable 5 failuresOpus 4.8Sonnet 4.6Haiku 4.5
Missing data/UI1,638334753169
Runtime/build1,6383384274
Incorrect data67829264079
State sync34512162065
Unresponsive control33414171846
Styling17125364453

We will add these exposure counts to Table 5 so failure frequencies can be interpreted against coverage.

“are the verifiers themselves checked by human experts?..”

Yes, the tests themselves are checked, separately from the judge verdicts. App. D.1 reports approximately 200 manually inspected browser executions and confirms that every test passes fully in at least two runs. During this process, we found that the browser agent cannot reliably perform some interactions, so App. D.1 requires that tests must be executable by the click-and-type browser agent (no hover, drag, sliders, or keyboard navigation). Every test therefore stays within what the verifier can execute, so a failed test reflects the generated app rather than a limitation of the verifier. Separately, the 300-execution human audit in App. D.3 checks judge verdicts. Each follow-up request is tested by five instructions: three functional, one integration, and one backward test.

“Are the evaluation results given by a judge stable across multiple runs?”

For stability, we re-judged 8,190 tests twice more on identical snapshots. Pairwise agreement is 91%, Grade SD is ≤ 0.009, and rankings are unchanged. This directly measures evaluator variability at both test and trajectory levels.

W2. Harness choice and visual feedback.

“It seems only OpenHands is used in evaluation without considering the official harnesses..”

We wanted one moving part: the model. Changing both model and harness would make it unclear which caused the performance difference. We therefore fixed OpenHands across all 14 models, choosing it because it is open-source, widely used, and supports them through a common interface.

We then tested the native harnesses separately, keeping the model fixed. Values show OpenHands → alternative harness.

ModelAlternative harnessGradeTurns/request
Opus 4.8Claude Code0.608 → 0.60610.0 → 12.6
GPT 5.6 SolCodex0.622 → 0.6439.0 → 6.9
Kimi K3Kimi Code0.684 → 0.6607.6 → 4.4

Claude Code leaves Opus unchanged, Codex improves Sol while using fewer turns, and Kimi Code uses far fewer turns at a slightly lower grade. These results show that matching the harness to a model can improve performance or reduce turn use, but the gains are within run-to-run variation.

“Does OpenHands allow agents to view images?..”

OpenHands supports images. Our environment currently provides rendered app views to the evaluator, but not to the coding agent. Adding visual feedback during generation requires efficiently serving the evolving apps and capturing screenshots across runs. We will clarify this setup and discuss visual feedback as future work.

AuthorsOpenReview ↗

Author response (2/2)

W3. Turn budget.

We set a 20-turn limit to keep evaluation time and cost manageable. We chose this value using Haiku’s 95th-percentile turn count, measured before imposing the limit. Haiku was a useful reference because it and Qwen had the highest average turn usage.

W4. Initial task generation.

We thank the reviewer for this suggestion. Reusing WebGen-Bench for the first request was a deliberate design choice rather than a requirement. It gives validated starting points and makes the first request directly comparable to WebGen-Bench, which lets us isolate the effect of our interaction protocol (Sec. 3.2). The pipeline itself does not depend on it: the base request is a short application description, which can be generated from the same real-website grounding used for the follow-ups. Extending EvoWeb to new applications therefore requires only an application category as input. We will clarify this in the paper.

W5. Intrinsic difficulty and historical failures.

Ensuring difficulty. We do not score difficulty; it is ensured by construction. App. D.1 requires each follow-up to include a non-trivial problem beyond its dependencies (e.g., combined filters, sorting with ties, exact calculations), targeted by a functional test.

Feature vs. dependency. Functional tests check the new feature alone; the integration test checks it with its prerequisite, in the same request and app state. Integration scores are 0.045 lower (95% CI [−0.074, −0.019]) after adjusting for test properties.

Prior vs. current failures. For the 37% of follow-ups whose backward test re-checks a prerequisite, we can tell whether the prerequisite still works. It passes in 70.5% of integration failures (95% CI [62.7, 77.3]), so most failures come from the new request rather than an inherited one. Even when the prerequisite and all three functional tests pass, 18.7% of integration tests receive PARTIAL or NO, vs. 13.0% for a held-out functional test under the same condition (+5.7 pts, 95% CI [+0.6, +11.7]).

Extension to general coding tasks and RL.

We believe the pipeline extends naturally to both, although the scope of this paper is to provide a benchmark, and we have not evaluated these extensions experimentally.

General app development and coding. The core of the pipeline does not depend on the web domain: grounding features in real products, organizing them into a dependency DAG, and pairing each request with functional, integration, and backward tests. Extending it to other kinds of applications would mainly require replacing the browser-based evaluator with a domain-appropriate verifier, such as unit tests or command-line checks.

Reinforcement learning. Unused WebGen-Bench specifications can be passed through the same pipeline to create a training set, while EvoWeb's applications remain held out for evaluation. The tests then provide intermediate rewards after each request: functional tests reward implementing new features, integration tests reward connecting them to earlier ones, and backward tests reward preserving previously working behavior.

We see both directions as promising and leave them for future work.

Reviewer_w5B8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoWeb evaluates persistent web applications through ten cumulative requests each, totaling 320 requests over 32 apps. Dependency-aware browser tests separate new functionality, integration, and sampled earlier functionality; tolerance-integrated metrics summarize trajectories. The study covers 14 models, with repeated closed-source runs, alternative judges, and a blinded human evaluator audit.

Critical strengths
  1. The benchmark usefully extends web generation to cumulative product development. Dependency-linked integration tests, explicit treatment of backward-test ambiguity, and human/cross-vendor judge validation make it a promising resource for studying reliability across requests.
Critical weaknesses
  1. Evaluator uncertainty can propagate through the prefix-minimum Grade metric. The human audit finds +0.050 integration-score bias and only 38.5% agreement in the ambiguity-selected live-inspection subset. These are transparently reported, but their impact on trajectory scores and backward verdict transitions is not quantified; cross-judge ranking agreement does not resolve shared errors.
  2. The dependency analysis is observational. Restricting to requests 8–10 reduces position confounding but does not match intrinsic feature complexity, archetype, or application difficulty. Lower integration scores and negative depth slopes therefore suggest, rather than isolate, a dependency-specific challenge.
Questions for authors
  1. How much do Grade, model comparisons, and apparent backward regressions change under repeated judging of identical snapshots or human-corrected early verdicts? Please quantify uncertainty over applications as well as agent runs.
  2. Does the dependency effect remain after controlling for feature archetype, request complexity, and application, with application-clustered intervals or matched dependency ablations?
1 comment
AuthorsOpenReview ↗

Author response

Thank you for the review. We ran additional analyses to address both concerns.

W1 / Q1: Propagation of evaluator uncertainty

We measure how evaluator uncertainty affects Grade, model comparisons, and backward regressions, using the two analyses the reviewer suggests:

  • Repeated judging of identical snapshots. We re-judged every test of five runs (8,190 tests) two more times on the same app snapshots, using the paper's judge.
  • Human-corrected verdicts. We used the 300-item human audit to estimate how often each judge verdict disagrees with humans, for each test type. We then replaced every verdict in the paper's runs (early ones included) with a human-calibrated draw and recomputed Grade, repeating this 1,000 times. Because this is anchored to human labels rather than to other judges, it also captures errors that judges share.

We also quantify uncertainty over applications, by resampling the 32 apps 1,000 times, and over agent runs (App. F).

Results. Evaluator uncertainty changes Grade only slightly and leaves the ranking essentially intact. Both evaluator sources are at or below the variation from the choice of applications (0.024–0.046).

Source of uncertaintyStandard deviation of Grade per modelRanking stability
Repeated judging (5 models)≤ 0.009Identical in all 3 judgings
Human-corrected verdicts0.023–0.027Kendall τ = 0.94 [0.87, 1.00]
Agent runs (App. F)0.004–0.043—
Applications0.024–0.046Kendall τ = 0.89 [0.80, 0.98]
  • Model comparisons. Correcting judge errors changes the order only of models that are already nearly tied. Any two models whose Grades differ by more than 0.05 keep the same order in at least 97.5% of the 1,000 human-corrected draws.

  • Integration bias. Integration scores remain below functional scores for every model in every judging. Since the judge is lenient on integration tests (+0.050) but not on functional tests (Table 10), correcting this bias would widen the gap.

  • Backward regressions. 6.4% of earlier-YES tests regress in a single judging, but only 3.2% regress in all three judgings. We will report this consistent rate as the regression estimate.

  • Ambiguous cases. The 38.5% agreement comes from the live-inspection subset, which was selected for ambiguous verdicts, so it is a worst case rather than typical judge accuracy. These items are part of the 300-item audit, so their disagreement is already included in the human-corrected draws above.

We will add these analyses to the appendix and report application-bootstrap confidence intervals alongside Table 2.

W2 / Q2: Does the dependency effect remain after controlling for archetype, complexity, and application?

Yes. We hold these factors fixed by design. Each request has three functional tests, which check the new feature on its own, and one integration test, which checks it together with an earlier feature it depends on. All four are evaluated on the same app snapshot in the same run, so they share the request’s archetype, complexity, application, position, and agent run. The only difference is whether the test exercises a dependency, making this a matched comparison.

We compare the two test types within each request (request × run fixed effects, all 14 models), with 95% confidence intervals from resampling applications:

AnalysisEffect95% CI
Integration − functional (same request)−0.066[−0.097, −0.038]
+ adjustment for the tests' length, steps, exact-value checks, and kind−0.045[−0.074, −0.019]
Not-YES rate when both connected features pass their own tests: integration vs. functional*18.7% vs. 13.0%+5.7 pts [+0.6, +11.7]

*Integration: snapshots where all three functional tests pass and the backward test, which re-checks the request this one depends on, also passes (n = 1,139). Functional baseline (leave-one-out): for each functional test, snapshots where the other two functional tests and the backward test pass (n = 3,928).

What these results show.

The integration gap holds within the same request, after adjusting for test properties, and even when both connected features pass their own tests. Since the judge is lenient on integration tests (Table 10), these estimates are likely conservative. Together with the depth and in-degree trends in Fig. 4, this shows that integration is a distinct challenge beyond implementing each feature in isolation. We will add this analysis to Sec. 5.1 and the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Uploaded Revision

We sincerely thank all three reviewers for their careful reading, thoughtful questions, and constructive suggestions. Their feedback helped us strengthen the experimental evidence, clarify our assumptions, and improve the paper’s presentation. We incorporated the following updates:

  1. Evaluator uncertainty (w5B8, B9Kb, idJ8). Re-judging 8,190 tests twice more yields Grade SD ≤0.009 and unchanged rankings. We added 1,000 human-calibrated resampling draws, with ranking agreement τ=0.94, and quantified application-sampling uncertainty (App. D.4).

  2. Integration difficulty (w5B8, B9Kb, idJ8). Within-request comparisons across all 14 models retain an integration deficit of 0.045 after controlling for test properties. When component checks pass, integration’s not-YES rate is 18.7%, versus 13.0% for the functional baseline (§5.1; App. D.5).

  3. Backward evaluation (B9Kb, w5B8). Expanding from one to four earlier tests preserves the three evaluated models’ ordering, with backward Grade changes of at most 0.022. Repeated judging identifies 3.2% consistent regressions on the re-evaluated subset. We clarify the sampling scope and cost tradeoff (App. D.1).

  4. Verification coverage (idJ8). We added test-exposure counts to Table 5 so failure frequencies can be interpreted against coverage.

  5. Harness choice (idJ8). We added comparisons with Claude Code, Codex, and Kimi Code, and explained why the main evaluation uses a shared harness (§3.2; App. E.1).

  6. Grade’s additional value (B9Kb). We explain how Grade captures failure timing beyond mean accuracy and make the uniform-weighting and abandonment assumptions explicit (§3.4; App. H).

  7. Construction and future directions (idJ8). We clarify the choice of WebGen-Bench seeds and the current visual-feedback setup, and discuss future extensions to visual feedback during generation, new applications, other coding domains, and reinforcement learning (§§3.1–3.2; Conclusion).

We are grateful for the reviewers’ time and care. Their suggestions have made the paper stronger and clearer.

Thank you,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.