- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EvoWeb evaluates coding agents on 32 web applications containing 320 cumulative requests organized by functional dependency graphs. Agents modify persistent codebases, while browser evaluation checks new functionality, cross-request integration, and sampled earlier functionality. Tolerance-averaged trajectory metrics characterize development performance. Experiments across 14 models reveal integration difficulties and lower accuracy with greater dependency complexity. The primary contribution is an executable benchmark supported by construction checks, human evaluator validation, and robustness analyses.
Critical strengths
- The benchmark addresses a realistic setting in which successive user requirements must work together within an evolving application. Persistent codebases and conversation histories support evaluation of cumulative development.
- Dependency graphs and separate functional, integration, and backward tests provide an actionable evaluation structure. Explicit integration checks capture a meaningful capability beyond isolated feature implementation.
- Construction and validation are reasonably thorough. Product grounding, automated verification, human curation, and a 300-execution human audit support benchmark quality; alternative judges largely preserve broad model ordering.
- The multi-model study provides useful capability characterization, including integration gaps, dependency-related difficulty, and distinct build and runtime failures. Repeated closed-source runs help qualify the comparisons.
Critical weaknesses
- The empirical findings are useful but mostly intuitive. Dependency complexity, feature difficulty, and prior implementation quality remain partly entangled, limiting explanations of why agents fail.
- The trajectory metric’s additional value needs stronger justification. Mean accuracy and Grade produce similar rankings, while uniform tolerance weighting and the assumed abandonment behavior lack empirical user validation.
- Evaluator uncertainty is not sufficiently propagated into trajectory scores. Human auditing identifies integration-score bias, and early grading errors can affect subsequent contributions. Quantifying these effects would strengthen confidence in absolute scores and close comparisons.
- Retention conclusions are limited by backward-test sampling. One earlier test per follow-up request is not a comprehensive regression audit, and failure may reflect functionality never implemented. The paper acknowledges this tradeoff, but preservation claims should remain correspondingly narrow.
Questions for authors
See Weaknesses.
2 comments
Thanks for your response, I tend to keep my positive score.
Author response
Thank you for the review. We added experiments addressing integration difficulty, evaluator uncertainty, and backward-test coverage.
W1. Agent failures.
Our main finding is that agents struggle to make new features work with what they have already built. We added two analyses of existing runs to examine this.
1. Compare tests under the same conditions. We compare the integration score with the average of the three functional scores for the same request and app version. The same request fixes dependency complexity and feature difficulty at the request level; the same app version fixes prior implementation quality.
We then account for differences between the tests: steps, exact-value checks, and check category. Integration scores remain lower: −0.045, 95% CI [−0.074, −0.019], with intervals from resampling applications.
2. Check cases where prerequisite tests pass. We select cases where all three new-feature tests and a backward test of a prerequisite pass. Among 1,139 such cases, 18.7% of integration tests still receive PARTIAL or NO. Thus, passing these feature checks does not ensure that the features work together.
Pinpointing the exact root cause of a failure in long agent trajectories remains an open research problem [1, 2, 3]. EvoWeb contributes to it by making these failures measurable, and the paper already characterizes them through app-side failure modes (Table 5) and build vs. runtime errors (Sec. 5.2).
[1] Zhang et al. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. ICML 2025
[2] Chen et al. Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios. 2025
[3] Mehtiyev & Assunção. Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure. 2026
W2. Grade’s additional value.
Grade captures when failures occur, which mean accuracy ignores. The trajectories (0.5, 1, …, 1) and (1, …, 1, 0.5) have the same mean accuracy (0.95). However, any user with tolerance above 0.5 abandons the first after request 1, so its Grade is 0.50 versus 0.95 (Sec. 3.4; App. H). This matters in practice: close models change order (e.g., GPT 5.6 Sol vs. Opus 4.8), and Grade separates models more clearly. Haiku 4.5 reaches 75% of Fable 5's mean accuracy but only 54% of its Grade (Table 13).
Uniform weighting and the abandonment model. These are modeling assumptions, but they do not hide the alternatives. Any other weighting gives a score between mean accuracy Q(0) and the fail-stop score Q(1), both reported in Table 13, and Fig. 3a (Fig. 6 for open-source models) shows results across the full tolerance range. Validating the abandonment model with real users would require humans to build and evaluate applications with each agent, which is very costly at this scale. We will clarify this in Sec. 3.4.
W3. Propagating evaluator uncertainty.
We re-judged 8,190 tests from five model runs twice more on identical snapshots. Grade SD is ≤ 0.009, with unchanged rankings.
We also recomputed complete trajectories in 1,000 audit-calibrated label-resampling draws (using verdict distributions estimated from the 300-test human audit), conditioning on judge verdict and test type. This propagates early verdict changes through later contributions and incorporates the audit’s test-type-specific discrepancies. Grade SDs are 0.023–0.027, with Kendall τ = 0.94 [0.87, 1.00].
Correcting these errors reorders only near-ties: models more than 0.05 apart keep their order in at least 97.5% of draws. Since the judge is lenient on integration tests (+0.050; Table 10), correction would widen the integration gap, not remove it.-
We will add these evaluator-uncertainty analyses to App. D.3 and application-bootstrap confidence intervals to App. F.
W4. Backward-test sampling and evaluation cost.
Each backward test requires a separate browser execution and WebVoager agent steps. We use one earlier test per follow-up request as a sampled estimate of earlier functionality, limiting evaluation time and cost.
To measure the effect of sampling, we re-evaluated all four tests from the same selected earlier request, using three runs per model across all 32 applications.
The one-test backward Grades match Table 4. With broader coverage, the model ordering is unchanged and backward Grade changes by at most 0.022. This measures the effect of the sampling choice directly.
We will make the sampling interpretation explicit and add this comparison to App. D.1. Preservation statements concern the earlier functionality tested; a failure counts as an apparent regression only when that same test passed previously.