acceptodds
Under review as a conference paper at ICLR 2027

ProductWebBench: Evaluating Website Continuity Beyond Task Completion

Abstract

Frontend coding agents are usually graded on one question: did the requested target get built. Real engineering changes carry a second obligation, because the rest of the site has to keep working. We call that second obligation *website continuity*, and introduce ProductWebBench, a benchmark that scores both on living frontend repositories: 320 Change tasks over existing code and 80 long-horizon Build tasks. Every task freezes a repository snapshot, staged requirements, browser evidence and deterministic checks before evaluation, and enters only once a reference passes every check and plausible-wrong ones fail. A simulated user reveals the requirements one stage at a time, and every check cleared earlier stays armed for the rest of the run. Across 8,737 runs from 13 models the two come apart: of the Change runs passing every requested-work check, **28.2%** still fail a continuity check — a blank route, horizontal overflow, console errors — and **10.8%** push a browser state they had already passed back to failing. Because the same checks also run on the untouched repository, each regression carries exact provenance, not a post-hoc label: on the 586 runs where both readings exist, **18.3%** are caught breaking a verified state. Interaction is a further blind spot, failing in **29.5%** of runs whose static DOM already passes. ProductWebBench therefore scores a patch against the living system it lands in, not the target alone.

Primary area:
datasets and benchmarks
Keywords:
coding agents, frontend development, benchmarks, long-horizon evaluation, website continuity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ukr6OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper evaluates whether, when given a sequence of queries from a simulated user, an LLM agent is able to continuously update the code without making breaking changes. The paper then quantifies the continuity and whether the LLM agent breaks the code after having subsequently fixed it.

Critical strengths
  • The underlying idea is practically relevant: can the same model change a website without breaking it?
  • The results are interesting, illustrating that agents break 53% of existing web apps after having already fixed them ( I assume that is what incumbent property refers to)
Critical weaknesses
  • Presentation needs some work. A lot of the abstract/main contributions in the introduction sound impressive but don't mean much to me e.g. " ProductWebBench therefore scores a patch against the living system it lands in, not the target alone." is the final sentence of the abstract, it's unclear unless you dig deeper into the paper what this truly means, something that the final sentence of an abstract shouldn't do. Many of the concepts introduced in the introduction lack a clear definition and sound very AI-generated... I think it would be simpler to say something like "ProductWebBench therefore evaluates code changes in live systems, unlike prior systems
  • It would be nice if bullet points in the introduction started on a new line.
  • Overall, the method is too confusingly written for me to fully evaluate this paper.
  • It is unclear how the user simulator is specified
Questions for authors

Where is the simulated user specified and evaluated? How does this process take existing tasks and break them into multi-turn chunks?

Reviewer_K2wPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a benchmark for agents that modify existing websites. It contains 400 tasks on frozen front-end repositories across React, Vue, Svelte, Astro, Gatsby, Hugo, PHP and static sites: 320 ask for a change to a running site while preserving everything that already works, and 80 build a site through five to fourteen ordered milestones. Each task freezes the repository, a stage plan, captured browser states, 19 deterministic checks (does the page render at the route, does the console stay clean, does the layout overflow, does the earlier feature still respond), a reference patch and a plausible wrong solution. A task is admitted only if the reference passes every check and the wrong solution fails. Evaluation is multi-turn: a scripted persona reveals requirements one stage at a time, and every check that passed at an earlier stage is re-run at every later one, so a later failure is recorded as a regression of protected state. A second channel runs the checks on the untouched repository and flags anything the agent broke that was previously working. No LLM decides pass or fail. Over 8,700 runs from 13 models and 8 scaffolds, whole-task pass rates sit between 1% and 4%; among runs that complete the requested change, 28% fail a preservation check, 11% regress a protected state, and 18% trip either regression channel.

Critical strengths

Every verdict comes from frozen checks on captured browser state, every task is admitted only when the reference passes and the plausible wrong solution fails, and cleared checks are re-run at every later checkpoint. Compared with WebGen-Bench, ArtifactsBench and WebCompass, which rely on LLM or agent judges, this removes judge leniency and self-preference as explanations for the completion-versus-preservation gap.

Critical weaknesses
  • My main worry is that the regression numbers are noise. The authors rerun the same cell three times and get different verdicts on about 2.6 of the 19 checks. They say on p.15 they never ran the verifier alone on frozen artifacts to see how often a check flips on its own. But a “protected-state regression” is exactly a check that passed and then failed later, so a flaky check produces one for free. And the results behave like flakiness: no correlation with model strength, the best model has the most regressions, and the authors’ own explanation on p.17 (stronger models get checked at more checkpoints) is what you’d expect if every checkpoint had the same small chance of flipping. The untouched-repository channel has the same problem in a worse form. All 66 events come from two checks, overflow and console errors. The baseline was captured on a different machine from the final states, and Appendix I says that when they recapture the baseline it only agrees with the old one 85% of the time, mostly on those same two checks. This regression claim is the thing that makes the paper different from pass-to-pass tests, so it decides my score. Rerun the verifier on the frozen artifacts, report the flip rate, and capture baseline and final on the same machine.

  • Nobody human checked the verifier. The two raters in Appendix D are an auto-rater and an LLM. The false-fail rate is 19% overall, and for the preservation checks it’s 0 out of 7, which tells us almost nothing, and every false fail there goes straight into the 28% headline. I also couldn’t find out who wrote the 479 tasks, which models were the persona, rater and judge, or which repositories were used. The case-study sites look like templates. A human audit of a hundred preservation failures, a repository list, one complete task package and the model names would fix this.

Questions for authors
  • What is the per-check flip rate when the verifier is re-run on the same captured artifacts, and how many of the 73 protected-state regressions and 66 untouched-repository events survive after netting it out? (Clarification)
  • Were the 479 specifications, reference patches and wrong solutions written by humans, generated by a model, or generated and human-checked? If a model, which one, and does it overlap with the evaluated solvers? (Clarification)
  • Which models serve as the simulated user, the blind rater and the LLM judge? (Clarification)
  • How does the protected-state regression rate change when normalised by the number of (checkpoint, protected check) pairs a run was exposed to? (Clarification)
  • Please position the untouched-repository channel against SWE-bench/ Pro pass-to-pass tests and EvoCode-Bench, and the staged-reveal protocol against Web-Bench (arXiv 2505.07473) and MT-Web2Code (arXiv 2608.03474), none of which are cited. (Suggestion)
  • In Table 1 the Sonnet 4.6 row shows 10 passes out of 800 but a 1.62% rate, and its two sub-columns sum to 13; with 13 the column total matches the stated 198. Please correct. (Suggestion)
Reviewer_SrvTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

this paper introduces Product webbench, a benchmark for evaluating front end agents on tasks related to editing or modifying websites preserving existing functionality and previously established behaviour . I think the key difference with other benchmarks is that in this case the concept of "website continuity" is defined , which means any modification should not break any unrelated part of the existing application. overall, the evaluation uses staged requirements, simulated users, browser state verification. this over ~400 tasks. The paper presents evaluations of 13 models across ~8K trajectories, showing that around 30% of the change runs, while mostly satisfying work checks, still fail continuity checks , around 10% actually regress to a previously seen state in their trajectory

overall the point of the paper is that when coding, task completion and the preservation of existing behaviour could not be aligned, so it is necessary to evaluate them separately. this distinction is not present in current front end development benchmarks,

Critical strengths

in the first place i think the motivation of the paper comes from a real problem and it tackles an important limitation of coding agents. it is in some sense the essence of regression testing, which is probably the most common type of testing conducted in real world apps . in that sense i think the problem formulation is convincing

in the second place, the benchmark itself was construction from real repos with executable browser evidence. in particular the criteria to accept a task is sound (reference solution passes acceptance conditions why intentionally buggy implementation still fail)

again, benchmark construction has some very interesting decisions: for example, in my understanding , it is possible to distinguish between regressions generated by the target agent and regressions that where already working in the original repo

the evaluation is extensive: 13 models, across several frontend frameworks, and more importantly: modality (text based, vs screenshot enabled config )

Critical weaknesses

after reading the paper and also the appendix, i think the main weak point is related to the verifier validity, which is something the author already know / acklnowedge. this means although the benchmark relies on deterministic verification, we know "deterministic" checks are not necessarily "semantically correct" , we can see this on appendix D -> 19% false fail rate. this number is not minor we all agree it can impact on both the task success numbers and also the actual regression estimates .

another issue, not only in ML benchmarks, but in general in the software engineering research community (ICSE, ISSTA, FSE) is that any claims depend heavily on correctly identifying the transitions between "passing" and "failing" states. here this paper unfortunately does not provide evidence, even samples, about examining the before and after browser evidence. I understand this is a hard task and people have been trying to get a working dataset for years

another element that can be seen as a weakness is that there is a small set of verification signals, 19, ranging from DOM properties, runtime health, etc. That is fine, but it is not representative of possible behaviours we can see from real world applications, some of them difficult to encode as a deterministic assertion. So the "continuity" this paper measures is relative to that specific set , so i would careful about the claims on this paper about "preserving website functionality"

next, maybe im being a bit picky but looking at the weighting used on the website continuity score, the weights looks quite arbitrary and it is a bit difficult to think we can generalize . Naturally, we can consider this as "" design choices" , but it would be useful to have some discussion at least on the decision and the ranges these values could take

Questions for authors
  • is it possible to independently inspect a statistically meaningful sample of detected regressions, and compare browser before-and -after states after that "alleged" regression?

  • sensitivity of the website continuity -> how sensitive are model ranking to the score weights ?

  • how were the preserved browser states selected and , more importantly, what evidence can we have on how representative are those states of important, real world application behaviour?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.