- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces WebDevHorizon, a benchmark of 114 tasks in which an agent must deliver a complete user-facing feature inside an existing full-stack Web application. Tasks come from 52 releases published in 2026 across 24 public repositories. Features are reconstructed from release notes and linked pull requests by extracting behavioural claims and clustering them; each task gives the agent a stakeholder-style request and the repository at the base commit with history removed. Evaluation uses fail-to-pass and pass-to-pass checks that combine repository-native tests with LLM-drafted supplemental checks at the component, API, database and browser level; three experts review each task, and admission requires deterministic differential execution over three clean replays. Eight models are evaluated with mini-swe-agent in a network-isolated container. Resolved rates range from 40.35% to 63.16%, with Opus-5 and GPT-5.6-Sol tied at 72 of 114. The analysis reports that resolution drops with implementation size and with schema migrations, that a passing final local test run often does not imply resolution, that the two leading models score lower in their native harnesses, and that incomplete implementations and logic errors account for most failures.
Critical strengths
-
The benchmark fills a real gap and is built with attention to contamination and evaluation surface. Existing feature-level benchmarks are mostly Python with unit tests, and greenfield Web benchmarks often depend on LLM or GUI-agent judges. Here every task modifies both frontend and backend code in a maintained application, all source releases are from January–July 2026, requests are rewritten rather than copied, and checks are deterministic and span the layers the feature touches. Tasks are admitted only after three-expert review and three clean replays. This is what gives the benchmark a chance of remaining useful for current models.
-
Reporting is transparent and internally consistent. I recomputed per-model resolved counts from every stratified result (size strata, layer strata, local-test groups, failure categories); all of them sum to the Table 3 totals (512 resolved, 400 unresolved over 912 model–task pairs), and the dataset distributions (domains, intents, release months, claims per task) sum to 114 tasks and 254 claims. The paper is also explicit about what its trajectory detector can miss. This makes the measured results trustworthy as measurements, which is the basis for everything else.
Critical weaknesses
-
Whether the hidden checks measure "feature delivered" is not validated, in either direction.
- Every task relies on supplemental checks drafted by an LLM from the feature's claims and admitted when they pass on the reference implementation. Requests name public interfaces, but checks may still depend on internal identifiers or on the structure of the reference patch. The paper's own Cal.com case (Appendix B.3) notes that the verifier reported a missing
RRSkipFieldRuleAction.IGNOREexport and an undefined expected function, and that treating these as real defects "additionally requires linking those expectations to existing callers or framework conventions", i.e. this was not established. Differential execution (fail on base, pass on reference) cannot detect over-specification, and the expert review checked that claims are covered, not that alternative correct implementations pass. - In the other direction, the median task has one regression check (interquartile range 1–2) on a codebase with a median of 429K lines, so "no regression" is weakly tested.
- The failure audit is LLM-assisted with the reference patch and tests in context, which presupposes that a failed check reflects a defect. Opus-5 drafted the requests and supplemental checks and is also one of the two top-scoring models; DeepSeek-V4-Pro labels the failures and is also evaluated.
- The task packages are not yet available to reviewers, so none of this can be examined externally.
This is consequential because the headline finding (the best models leave more than a third of tasks unresolved) and the failure taxonomy both depend on it. The OpenAI audit of SWE-bench Verified found overly narrow tests in 35.5% of the problems it examined, so this is a known risk for this benchmark style. A manual audit of a random sample of unresolved runs for functionally correct but rejected patches, and results split by repository-native versus supplemental checks, would largely resolve it.
- Every task relies on supplemental checks drafted by an LLM from the feature's claims and admitted when they pass on the reference implementation. Requests name public interfaces, but checks may still depend on internal identifiers or on the structure of the reference patch. The paper's own Cal.com case (Appendix B.3) notes that the verifier reported a missing
-
The comparative and behavioural conclusions are stronger than a 114-task, single-run evaluation supports.
-
The top six models span 63.2% to 56.1%. With 114 tasks, one resolved rate has a 95% half-width of about nine points. Repeated runs (Appendix A.5) cover only four lower-cost models on 20 tasks and vary by one task per round; the leading models are not repeated. Four repositories contribute 44 of the 114 tasks.
-
Local tests and false confidence. Figure 5 with Table 3 also determines the resolved rate of trajectories whose last local test run failed. In six of eight models it is equal to or higher than for trajectories whose last local run passed:
Model Last local run passed Last local run failed Opus-5 53/84 (63.1%) 13/20 (65.0%) GPT-5.6-Sol 52/77 (67.5%) 18/31 (58.1%) DeepSeek-V4-Pro 53/86 (61.6%) 14/22 (63.6%) Kimi-K3 49/79 (62.0%) 16/24 (66.7%) Qwen3.8-Max 50/79 (63.3%) 12/24 (50.0%) DeepSeek-V4-Flash 47/86 (54.7%) 15/26 (57.7%) GLM-5.2 31/70 (44.3%) 19/37 (51.4%) Hy3 18/58 (31.0%) 23/46 (50.0%) The last recognized local result therefore carries little information about resolution either way, which fits a noisy proxy better than agents being misled by passing tests.
-
Cost. Opus-5 is priced with no cache rate (all input at 5 USD per million tokens, giving 84.71 USD per task), while GPT-5.6-Sol and others use cached-input rates. Opus-5 has a public cache-read price of 0.50 USD per million tokens, so the "84.71 USD versus 4.00 USD" comparison is largely an accounting difference.
-
Harness. The finding that both leading models score 8–11 points lower in their native harnesses rests on one run each; the harness configuration and the effect of the 5,400-second wall-clock budget are not reported.
This is consequential because the model ranking and the behavioural findings are what readers will cite from the paper.
-
Questions for authors
- In a manually audited random sample of unresolved runs, how many patches implement the requested behaviour but fail a check that depends on identifiers or structure not stated in the request? Can the task packages be made available for review? (See Weakness 1.)
- How do per-model resolved rates differ when computed on repository-native checks only versus supplemental checks, and does Opus-5's relative standing change on checks it did not draft? (See Weakness 1, third point.)
- Can repeated runs or confidence intervals be reported for the leading models on the full set, or at least on the 20-task subset? (See Weakness 2, first point.)
- How should the local-test finding be interpreted given that failed-last-run trajectories resolve at similar or higher rates in six of eight models? (See Weakness 2, local-tests point.)
- How many runs per model, and per native harness, ended by reaching the 5,400-second budget, and what would Opus-5 cost with cache-read pricing applied? (See Weakness 2, cost and harness points.)
Thank you for the response. I appreciate the work by the authors, and I am looking forward to see the whole benchmark. I think the original score is good for the paper. Good luck :)