acceptodds
Under review as a conference paper at ICLR 2027

WebDevHorizon: Evaluating Coding Agents on Complete Web Feature Delivery

Abstract

Driven by rapid advances in large language models, autonomous coding agents are increasingly being applied to full-stack Web development. Existing benchmarks, however, typically evaluate either the resolution of isolated repository-level issues or the greenfield development of Web applications from scratch. In practice, developers regularly implement user-facing features within established Web codebases. Delivering such features is inherently a long-horizon, cross-layer endeavor: an agent must understand and modify frontend interfaces, backend logic, and often database schemas, while keeping states synchronized across these layers through iterative test and runtime feedback. Yet, it remains uncertain how reliably autonomous agents can complete such full-stack feature implementation tasks. In this paper, we introduce WebDevHorizon, a benchmark for long-horizon, full-stack Web feature delivery. It comprises 114 tasks obtained from 52 releases across 24 public repositories. Each task corresponds to a product feature shipped by the repository maintainers, with its behavioral requirements reconstructed from the project's development history. To evaluate feature-level behavioral completeness, WebDevHorizon maps these requirements to executable checks of component behavior, API contracts, persistent state, and browser workflows as needed. The environments and test suites are human-verified, and a task is resolved only when all target and regression checks pass. Evaluating eight frontier LLMs with mini-swe-agent highlights the difficulty of long-horizon delivery: across trajectories averaging 56–161 agent steps per task, models resolve 40.35%–63.16% of tasks, led jointly by Opus-5 and GPT-5.6-Sol.

Primary area:
datasets and benchmarks
Keywords:
Language models, Natural language processing, Software engineering, Full-Stack Web Development

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qfBWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WebDevHorizon, a benchmark of 114 tasks in which an agent must deliver a complete user-facing feature inside an existing full-stack Web application. Tasks come from 52 releases published in 2026 across 24 public repositories. Features are reconstructed from release notes and linked pull requests by extracting behavioural claims and clustering them; each task gives the agent a stakeholder-style request and the repository at the base commit with history removed. Evaluation uses fail-to-pass and pass-to-pass checks that combine repository-native tests with LLM-drafted supplemental checks at the component, API, database and browser level; three experts review each task, and admission requires deterministic differential execution over three clean replays. Eight models are evaluated with mini-swe-agent in a network-isolated container. Resolved rates range from 40.35% to 63.16%, with Opus-5 and GPT-5.6-Sol tied at 72 of 114. The analysis reports that resolution drops with implementation size and with schema migrations, that a passing final local test run often does not imply resolution, that the two leading models score lower in their native harnesses, and that incomplete implementations and logic errors account for most failures.

Critical strengths
  1. The benchmark fills a real gap and is built with attention to contamination and evaluation surface. Existing feature-level benchmarks are mostly Python with unit tests, and greenfield Web benchmarks often depend on LLM or GUI-agent judges. Here every task modifies both frontend and backend code in a maintained application, all source releases are from January–July 2026, requests are rewritten rather than copied, and checks are deterministic and span the layers the feature touches. Tasks are admitted only after three-expert review and three clean replays. This is what gives the benchmark a chance of remaining useful for current models.

  2. Reporting is transparent and internally consistent. I recomputed per-model resolved counts from every stratified result (size strata, layer strata, local-test groups, failure categories); all of them sum to the Table 3 totals (512 resolved, 400 unresolved over 912 model–task pairs), and the dataset distributions (domains, intents, release months, claims per task) sum to 114 tasks and 254 claims. The paper is also explicit about what its trajectory detector can miss. This makes the measured results trustworthy as measurements, which is the basis for everything else.

Critical weaknesses
  1. Whether the hidden checks measure "feature delivered" is not validated, in either direction.

    • Every task relies on supplemental checks drafted by an LLM from the feature's claims and admitted when they pass on the reference implementation. Requests name public interfaces, but checks may still depend on internal identifiers or on the structure of the reference patch. The paper's own Cal.com case (Appendix B.3) notes that the verifier reported a missing RRSkipFieldRuleAction.IGNORE export and an undefined expected function, and that treating these as real defects "additionally requires linking those expectations to existing callers or framework conventions", i.e. this was not established. Differential execution (fail on base, pass on reference) cannot detect over-specification, and the expert review checked that claims are covered, not that alternative correct implementations pass.
    • In the other direction, the median task has one regression check (interquartile range 1–2) on a codebase with a median of 429K lines, so "no regression" is weakly tested.
    • The failure audit is LLM-assisted with the reference patch and tests in context, which presupposes that a failed check reflects a defect. Opus-5 drafted the requests and supplemental checks and is also one of the two top-scoring models; DeepSeek-V4-Pro labels the failures and is also evaluated.
    • The task packages are not yet available to reviewers, so none of this can be examined externally.

    This is consequential because the headline finding (the best models leave more than a third of tasks unresolved) and the failure taxonomy both depend on it. The OpenAI audit of SWE-bench Verified found overly narrow tests in 35.5% of the problems it examined, so this is a known risk for this benchmark style. A manual audit of a random sample of unresolved runs for functionally correct but rejected patches, and results split by repository-native versus supplemental checks, would largely resolve it.

  2. The comparative and behavioural conclusions are stronger than a 114-task, single-run evaluation supports.

    • The top six models span 63.2% to 56.1%. With 114 tasks, one resolved rate has a 95% half-width of about nine points. Repeated runs (Appendix A.5) cover only four lower-cost models on 20 tasks and vary by one task per round; the leading models are not repeated. Four repositories contribute 44 of the 114 tasks.

    • Local tests and false confidence. Figure 5 with Table 3 also determines the resolved rate of trajectories whose last local test run failed. In six of eight models it is equal to or higher than for trajectories whose last local run passed:

      ModelLast local run passedLast local run failed
      Opus-553/84 (63.1%)13/20 (65.0%)
      GPT-5.6-Sol52/77 (67.5%)18/31 (58.1%)
      DeepSeek-V4-Pro53/86 (61.6%)14/22 (63.6%)
      Kimi-K349/79 (62.0%)16/24 (66.7%)
      Qwen3.8-Max50/79 (63.3%)12/24 (50.0%)
      DeepSeek-V4-Flash47/86 (54.7%)15/26 (57.7%)
      GLM-5.231/70 (44.3%)19/37 (51.4%)
      Hy318/58 (31.0%)23/46 (50.0%)

      The last recognized local result therefore carries little information about resolution either way, which fits a noisy proxy better than agents being misled by passing tests.

    • Cost. Opus-5 is priced with no cache rate (all input at 5 USD per million tokens, giving 84.71 USD per task), while GPT-5.6-Sol and others use cached-input rates. Opus-5 has a public cache-read price of 0.50 USD per million tokens, so the "84.71 USD versus 4.00 USD" comparison is largely an accounting difference.

    • Harness. The finding that both leading models score 8–11 points lower in their native harnesses rests on one run each; the harness configuration and the effect of the 5,400-second wall-clock budget are not reported.

    This is consequential because the model ranking and the behavioural findings are what readers will cite from the paper.

Questions for authors
  1. In a manually audited random sample of unresolved runs, how many patches implement the requested behaviour but fail a check that depends on identifiers or structure not stated in the request? Can the task packages be made available for review? (See Weakness 1.)
  2. How do per-model resolved rates differ when computed on repository-native checks only versus supplemental checks, and does Opus-5's relative standing change on checks it did not draft? (See Weakness 1, third point.)
  3. Can repeated runs or confidence intervals be reported for the leading models on the full set, or at least on the 20-task subset? (See Weakness 2, first point.)
  4. How should the local-test finding be interpreted given that failed-last-run trajectories resolve at similar or higher rates in six of eight models? (See Weakness 2, local-tests point.)
  5. How many runs per model, and per native harness, ended by reaching the 5,400-second budget, and what would Opus-5 cost with cache-read pricing applied? (See Weakness 2, cost and harness points.)
1 comment

Thank you for the response. I appreciate the work by the authors, and I am looking forward to see the whole benchmark. I think the original score is good for the paper. Good luck :)

Reviewer_AxXHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a benchmark for long-horizon and fullstack web development. The authors design carefully to guarantee the quality of the benchmark tests with various checks by both AIs and human experts and complement of repo native UTs. The tasks span both frontend and backend as well as databases. Evaluations are performed on several frontier models and controlled harnesses. The failure cases are statistically analyzed.

Critical strengths
  1. The benchmark is carefully created and validated, involving multiple checks by both AIs and human experts to ensure task quality.
  2. The tasks cover both frontend and backend as well as database, featuring fullstack development.
  3. The authors complements the repo native UTs to enhance the test coverage.
Critical weaknesses

The experimental analysis are expected to be improved.

  • For the harness comparison, it is counter-intuitive that the official harnesses like Codex and Claude Code obtain much lower performance. The results are presented without further analysis of concrete reasons.
  • The incomplete implementation occupies a large portion of failures. But the causes and reasons are not examined and analyzed. Do those failures come from factors like instruction ambiguities? Do the models miss some requirements at the very beginning or forget some during the development? Subsequent researches could barely benefit from the evaluation unless more concrete analysis are presented.
  • Similar problems as above also exist in other experiments for test generation, application layers, etc. The evaluation is expected to guide subsequent enhancement of models and be more than summarized statistics on the surface. It would be more helpful if concrete drawbacks of model capabilities could be identified and explained.

The difficulty of the tasks seem to need to be enhanced.

  • In current tasks, the median of claims is 2, the median of the number of edited lines in gold patches is 139, and 111 out of the 114 tasks correspond to only 1 PR. At the current state, the value of such tasks is limited. In fact, the resolution rates have already reach 63% for the "last" generation of models. The benchmark should more look into the future of the capability of frontier models.

Are there any GUI tests like in WebGen-Bench? Since the tasks involves both frontend and backend, it remains unclear whether the frontend features can be comprehensively validated without browser use.

Questions for authors

How can the current pipeline be enhanced to generate tasks in a scalable way? What are the bottlenecks?

Reviewer_vzt9OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper describes an approach for evaluating coding agents for full-stack feature delivery. It analyzes the challenges and existing solutions. It describes the key design consideration of the problem, as well as how the dataset reflects the design considerations. The dataset is evaluated using a list of models, showing the measured capability and common trend in model capabilities.

Critical strengths

This paper is well written and easy to read. The paper made a nice and clear problem analysis, highlighting the technical challenges and demonstrating the authors understanding of the domain knowledge. I like the discussions the authors made about the design rationale. The technical solution is also well connected to the problem analysis.

In the evaluation, the authors showed the model’s capability. More importantly, they also summarize the comment trend and give an explanation.

Critical weaknesses

The evaluation can shed more light on the models' capability beyond the numbers. For example, Opus-5 and GPT-5.6-Sol are similar in results of Table 3. Is it possible to decide which one is better based on the interactions and actions carried out by the models? For example, which one behaves more professionally with better domain knowledge and skills? In some cases, the model capability is reflected by how they approach a problem, rather than whether the solve the problem.

Questions for authors

When the benchmark is refreshed or expanded with future projects, how to decide which samples to include and which samples to replace?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.