acceptodds
Under review as a conference paper at ICLR 2027

WebsiteBench: Can AI Agents Rebuild Websites through Browser Exploration?

Abstract

Humans learn how unfamiliar environments work by observing them, trying different actions, and revising their understanding based on what happens. Websites provide a concrete setting for studying this process: they combine interfaces, application logic, and persistent data, while each browser interaction reveals only part of their behavior. Can AI agents decide what to explore and use what they learn to rebuild a website faithfully? We introduce WebsiteBench, which contains 74 expert-built and expert-reviewed website reconstruction tasks and 50,356 hidden tests. In each task, an agent explores a locally hosted website through a browser and builds a runnable offline site that looks and behaves the same way. The agent decides which pages to visit and which interactions to try, and may revisit the reference website while coding and testing. The hidden test suite consists of three groups: Request checks page responses and form submissions; Browser checks what users see and can do; and System checks application requirements. Our experiments show that WebsiteBench is challenging for frontier models: even the top model, Claude Opus 5.5, scores only 22.5%; GPT 5.6 Sol scores 15.6%, and Kimi K3 scores 8.9%. We systematically analyze agent trajectories and identify three failure modes: (1) underexploration, (2) failure to implement observed behavior, and (3) early stopping. We also find that agents given human exploration traces achieve higher reconstruction scores. These results show that agents struggle to learn how an unfamiliar system works through interaction and to use that understanding to rebuild it faithfully.

Primary area:
datasets and benchmarks
Keywords:
browser agents, coding agents, full-stack website construction, interactive evaluation, agent benchmark, agent behavior analysis, active exploration

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_pkkvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new benchmark called WebsiteBench. It is designed to evaluate whether AI agents can explore a locally hosted reference website and reconstruct it. The benchmark has 74 tasks across multiple domains and has been peer-reviewed by experts. The authors have also provided 50k hidden tests and evaluated 9 frontier models. The authors also have many interesting findings on analyzing 17 trajectories, such as shallow workflow exploration and failure to implement the observed behavior.

Critical strengths
  • The problem is genuinely interesting and timely. I do see the motivation of the problem in the current era, where AI coding agents are becoming widespread. The task of website reconstruction via exploration can have a ton of practical usability.
  • I think the overall engineering of the benchmark and evaluation is really good. The experts have built self-contained full-stack applications with persistent states. This is much better than scraping live webpages.
  • I also like the metrics used in the paper. The normalized SSIM is a neat trick and pretty intuitive.
  • There has been a good amount of experimental depth, with nine frontier models being benchmarked.
Critical weaknesses

The paper has a direct naming and concept collision with another paper from COLM 2026 (https://openreview.net/pdf?id=oX9AW8cXMK). I checked that this work has been public before ICLR's September deadline. Hence, this work should have been cited and compared. As this is a relatively newer work, it is highly likely that the authors missed it. However, this does question the novelty of the paper. Both papers have a lot of similarities, e.g., the problem formulation and core workflow, information isolation and anti-cheating block, and the offline reproducibility. The papers also have a similar conclusion that reconstruction remains largely unsolved by frontier models, and agents are much better at capturing surface-level pages or visual appearances without being able to recover or understand the underlying structure. I do agree that there are a lot of dissimilarities in benchmark construction, the agent's observation, and the evaluation, but I would still not consider that to be sufficient. I also find the current submission weaker in certain claims, such as the fact that the findings are based on only 17 inspected trajectories (Line 115), which is a bit on the lower end, and could be noisy.

Questions for authors
  • Could the authors please differentiate their work from WebsiteBench, and would you consider renaming the current benchmark as well?
  • Can the authors provide new findings that haven't been discussed in previous literature?
  • I would really appreciate it if more trajectories were analyzed for more grounded conclusions.
3 comments
AuthorsOpenReview ↗

(1/2) Author Responses

Dear reviewer pkkv,

We sincerely thank you for recognizing our paper as interesting, timely, and well motivated, and for emphasizing its interesting findings and practical utility. We also appreciate your positive assessment of our dataset construction process, in which experts build self-contained full-stack applications, as well as our evaluation design, evaluation metrics, and the depth and breadth of our experiments.

  1. Potential overlap (response to Weaknesses and Questions Q1 and Q2)

Thank you also for raising the concern about potential overlap. We checked the provided link (https://openreview.net/pdf?id=oX9AW8cXMK) and found that it is not publicly accessible. We also reviewed the COLM 2026 accepted papers list (https://colm.eventhosts.cc/Conferences/2026/AcceptedPapers) and found no paper titled "WebsiteBench." The referenced paper has not been published and does not appear in any proceedings; therefore, it is neither prior work nor contemporaneous published work.

  1. Analysis of additional trajectories

We appreciate your suggestion to analyze more trajectories. We have analyzed all trajectories from nine models across 74 tasks. The results are summarized below.

ModelOverallTurnsStateful workflows completedMedian run time (h)
Claude-Opus-5.522.52107.136.5%9.7
Claude-Opus-518.7980.530.5%8.0
GPT-5.6-Sol15.6212.328.3%0.6
GPT-5.6-Luna13.2174.09.5%0.7
Grok-4.611.6403.915.5%2.6
GLM-5.3-Flash9.7318.811.5%1.0
Kimi-K38.9604.327.0%3.1
DeepSeek-V4.1-Flash8.5572.619.2%2.8
Qwen-3.8-Max5.7252.119.4%1.8

"Stateful workflows completed" is the share of the hidden-test workflows involving changes to server state (for example, form submissions) that the agent carried out on the reference website. The time limit is 16 hours.

As shown in the table, agents complete only 9.5% to 36.5% of the stateful workflows, and no model completes more than half. Except for the two Claude models, median run times range from 0.6 to 3.1 hours, despite the 16-hour time limit.

AuthorsOpenReview ↗

(2/2) Author Responses

We further use these trajectories to support the findings of the paper. Each hidden browser test is a scripted user workflow with a defined end state, and the hidden tests list the workflows each website supports. A workflow is completed when the agent has carried out every step and then read the resulting page. In WebsiteBench, the agent must discover the website behavior itself by proactive exploration. This differenciate out work from other benchmarks, such as Vision2Web [1] (input: UI prototypes and assets, textual specifications, structured requirement documents), UI2App [2] (input: A set of static screenshots) and IWR-Bench [3] (input: User interaction video).

ModelRead-only workflows completedStateful workflows completedStateful, before first code editRuns ended by the agent
Claude-Opus-5.574.7%36.5%12.6%72%
Claude-Opus-566.3%30.5%13.4%67%
GPT-5.6-Sol38.6%28.3%27.9%100%
GPT-5.6-Luna12.8%9.5%9.1%99%
Grok-4.655.2%15.5%11.8%81%
GLM-5.3-Flash53.9%11.5%11.4%92%
Kimi-K354.4%27.0%23.9%83%
DeepSeek-V4.1-Flash55.5%19.2%13.0%85%
Qwen-3.8-Max68.4%19.4%15.7%84%

Finding 1. Underexploration. Agents read pages but rarely carry out the workflows that change state. Every model completes fewer stateful workflows than read-only ones, and before its first code edit for writing the website, no model has completed more than 27.9% of the stateful workflows. Agents start building before they have explored most of the website behavior.

Finding 2. Failure to implement observed behavior. Completing a workflow on the reference website during exploration is not enough for correct implementation. Claude-Opus-5 completed 147 of the 150 stateful workflows of site 022 (Example 15), including registration, sign-in and search, and none of them work in its submitted code.

Finding 3. Early stopping. The instruction states the time budget and tells the agent to use it ("You have up to 16 hours for this task"). Except for the two Claude models, agents stop after a median of 0.6 to 3.1 hours (table in part 1/2), while 84% to 94% of the hidden tests fail, and 81% to 100% of their runs end with the agent's own final message. GPT-5.6-Sol ended its run on site 058 after 0.7 hours with only 6.0% of its browser tests passing (Example 22).

We provide 30 examples, ten per finding and covering all nine models, together with per-model tables and figures, in an anonymous link: https://anonymous.4open.science/r/analysis-6E0C/.

We hope this clarification resolves the overlap concern and that our extended analyses strengthen our claims and findings. We would appreciate your reconsideration of our work, given that there is no overlap with previous publications and that we have conducted comprehensive analyses.

[1] He, Zehai, et al. "Vision2web: A hierarchical benchmark for visual website development with agent verification." arXiv preprint arXiv:2603.26648 (2026).

[2] Chen, Grace Man, et al. "UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation." arXiv preprint arXiv:2607.06306 (2026).

[3] Chen, Yang, et al. "IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?." International Conference on Learning Representations. Vol. 2026. 2026.

Dear Reviewer pkkv,

Thank you again for your constructive review. As the discussion period is drawing to a close, we would be grateful if you could take a look at our responses, specifically our expanded trajectory analysis across all trajectories. If these resolve your concerns, we would sincerely appreciate it if you could consider raising your rating. Happy to clarify anything further!

Reviewer_oKLiOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

WebsiteBench evaluates whether an agent can explore a reference website through a browser and reconstruct it as a runnable offline application without accessing the reference source code. The benchmark contains 74 locally hosted reference sites built and cross-checked by 16 developers, with 50,356 hidden tests covering request behavior, browser workflows, and system requirements. Nine models are evaluated, and the highest overall test pass rate is 22.5%. The paper also analyzes where exploration and implementation fail. On 18 paired sites, providing human exploration traces raises the mean reconstruction score from 2.8% to 9.8%.

Critical strengths
  1. The benchmark tests behaviors that are easy to miss when evaluating generated websites from screenshots alone. The checks include reading back edited data, keeping users' state separate, and preserving changes after a restart. These requirements make the task depend on a working application rather than a collection of plausible-looking pages. Having another developer review each reference and write its hidden tests is also a useful quality-control step. I see the main contribution as this concrete test resource for evaluating whether exploration can be turned into a functioning implementation.

  2. The human-trace experiment adds useful evidence about a possible source of failure. It keeps the model and agent configuration matched and changes the exploration information available to the agent. Across 18 pairs, the assisted condition improves on ten sites, worsens on two, and ties on six, with a seven-point increase in the site-average score. The trajectory analysis also separates visiting a page, completing its workflow, and reproducing the observed behavior. That distinction is useful: a long browsing trace does not by itself show that the agent discovered the requirements it needed to implement.

Critical weaknesses
  1. The low overall scores are difficult to interpret without a breakdown of runnable and failed submissions. Appendix E.2 assigns zero to compilation/runtime failures and absent candidates, but I could not find the per-model counts for these outcomes. Counting delivery failures is reasonable for the complete task. However, if many candidates never reach functional testing, the aggregate cannot tell us how much of the difficulty comes from reconstructing behavior. The interpretation of model rankings also needs more support. The main score pools tests, giving more weight to sites with larger test suites, while the bootstrap analysis in Appendix E.3 covers only five configurations on 36 sites rather than the full nine-model, 74-site comparison. Per-model outcome counts, site-level results, and uncertainty for the full table would make this benchmark much easier to use.

  2. Some details of the exploration analysis are not yet consistent enough to support the explanation of why models fail. Section 5.1 reports that independent runs completed roughly 40% of workflows before the first implementation edit, whereas Section 5.2 reports 4/122, or 3.3%. Appendix I.1 distinguishes the analysis subsets, so these need not be contradictory, but the counts and aggregation behind the 40% figure are still unclear. The trace input is also described as action records without page content in the text, while Figure 4 labels it as actions plus responses. That difference matters: replay guidance and supplied observations are different interventions. Finally, most of the paired score gain comes from sites whose independent run scored zero. Without separating build failures from behavior failures, I am not sure how much of the gain can be attributed specifically to more complete exploration.

Questions for authors
  1. For each model, how many scheduled trials produced runnable candidates, compilation/runtime failures, absent submissions, and excluded infrastructure failures? Please provide scores on runnable candidates alongside, rather than instead of, the existing end-to-end score.

  2. What are the numerator, denominator, and aggregation rule behind Figure 4c? Please explain how these observations relate to the 4/122 result, and specify exactly which action outputs or responses the supplied human trace contains.

  3. Can you report site-level results and paired uncertainty for the full 74-site comparison? For the 18-site trace experiment, how many gains reflect a transition from a non-runnable candidate to a runnable one, and how many improve the behavior of candidates that already ran?

4.Could you briefly compare WebsiteBench with RecreationWorld? Both involve exploring a running reference and rebuilding its behavior.

4 comments
AuthorsOpenReview ↗

(1/3) Author Responses

Dear reviewer oKLi,

We sincerely thank you for recognizing WebsiteBench as a concrete resource for testing whether exploration can lead to functional implementation, and for your positive assessment of the design of the hidden tests, user isolation and persistence across restarts, the independent review of each reference website, and the human-trace experiment. We also provide detailed analysis of agent behavior (https://anonymous.4open.science/r/analysis-6E0C/), including the finding that agents do not fully understand the concrete requirements of the websites. We directly address the weaknesses and questions below.

W1. Site-level results and uncertainty

1. Per-model outcome counts

ModelScheduledRunnableBuild failureRuntime-contract failureZero scoreExcluded infrastructure failureNot runnable
Claude-Opus-5.5747000040
Claude-Opus-5747200110
GPT-5.6-Sol7447003024
GPT-5.6-Luna747300001
Grok-4.67434139000
GLM-5.3-Flash747200200
Kimi-K3747200020
DeepSeek-V4.1-Flash746700160
Qwen-3.8-Max7459201030

Across 9 models, runnable submitted websites range from 34 to 73.

2. Uncertainty

We compared each pair of models site by site on the sites both completed, using the per-site scores from the per-run evaluation records and a paired site-cluster bootstrap (10,000 resamples) of the mean per-site difference. Claude-Opus-5.5 scores higher than each of the other eight models on 83% to 96% of shared sites (mean per-site difference +8.6 to +18.0 points). Claude-Opus-5 scores higher than each of the seven models below it on 63% to 89% of shared sites (+3.0 to +9.4 points). Every other model scores higher than Qwen-3.8-Max on 61% to 94% of shared sites (+3.4 to +18.0 points).

3. Site-level results and test-count weighting

Table 1 uses raw pass rate across all tests, so a site with a larger test suite counts more. The comparison in point 2 Uncertainty above does not. For each pair of models it takes the two models' scores on each shared site, subtracts them site by site, and averages the differences, so every website counts once. The percentages in point 2 are the share of shared sites on which the higher-ranked model scores higher.

AuthorsOpenReview ↗

(2/3) Author Responses

W2. Exploration analysis

1. Counts and aggregation behind Figure 4c

The two numbers use different run sets and definitions.

  • Figure 4c: for each site, the numerator is the number of state-changing workflows whose final route appears in the trajectory before the first implementation edit (the first time the agent is writing the code for website implementation), and the denominator is the site's number of state-changing workflows. The figure reports the median of these per-site shares: 40.0% (4 of 10 at the median site) for independent runs and 77.8% (7 of 9 at the median site).
  • 4/122 (3.3%): 122 core workflows taken from the 17 Grok-4.6 trajectories that reached a first implementation edit; a workflow counts only if its terminal outcome was observed before that edit.

The 4/122 criterion is stricter: seeing a workflow's route is weaker than observing its terminal outcome. Our trajectory analysis of all nine models applies one criterion throughout (every step performed and the result page read): before the first code edit, agents complete 9.1% to 27.9% of the stateful workflows (per-model table below).

ModelRead-only workflows completedStateful workflows completedStateful, before first code editRuns ended by the agent
Claude-Opus-5.574.7%36.5%12.6%72%
Claude-Opus-566.3%30.5%13.4%67%
GPT-5.6-Sol38.6%28.3%27.9%100%
GPT-5.6-Luna12.8%9.5%9.1%99%
Grok-4.655.2%15.5%11.8%81%
GLM-5.3-Flash53.9%11.5%11.4%92%
Kimi-K354.4%27.0%23.9%83%
DeepSeek-V4.1-Flash55.5%19.2%13.0%85%
Qwen-3.8-Max68.4%19.4%15.7%84%

2. What the human trace contains

In the original 18-pair experiment, the human trace listed every action the expert took in the browser: each click, each piece of text typed, each page opened. Next to the actions, it gave the title and text of each page. Figure 4's "actions plus responses" describes this correctly.

3. Build failures versus behavior failures

When adding the human trace, the score was higher on 10 of the 18 sites, lower on 2 and equal on 6. We separate two kinds of gain. On 7 of the 10 websites, the run without the human trace did not produce a website that could start (it failed the build or the startup check), so its score was 0. With the human trace, the website started and passed tests, and on 5 of these 7 it also passed tests that click through multi-step workflows in the browser. On the other 3 sites, the run without the human trace already produced a working website, and the human trace made it better: on two of them, the passed browser-workflow tests rose from 0 to 51 and from 0 to 25.

AuthorsOpenReview ↗

(3/3) Author Responses

Q1. Outcome counts per model

The table in W1 shows, for each of the nine models, how many scheduled runs produced a website that ran, failed to build, failed the startup check, scored 0, were excluded because of infrastructure failures, or were not started. If we compute the score only on the websites that started, the score is at most 1.3 points higher than our main score, which counts a website that fails to start as 0. So the low scores are not caused by websites that never started.

Q2. Figure 4c

The two numbers use different run sets and definitions.

  • Figure 4c: for each site, the numerator is the number of state-changing workflows whose final route appears in the trajectory before the first implementation edit (first time the agent is writing the code for website implementation), and the denominator is the site's number of state-changing workflows. The figure reports the median of these per-site shares: 40.0% (4 of 10 at the median site) for independent runs and 77.8% (7 of 9 at the median site).
  • 4/122 (3.3%): 122 core workflows taken from the 17 Grok-4.6 trajectories that reached a first implementation edit; a workflow counts only if its terminal outcome was observed before that edit.

The 4/122 criterion is stricter: seeing a workflow's route is weaker than observing its terminal outcome. Our trajectory analysis of all nine models applies one criterion throughout (every step performed and the result page read): before the first code edit, agents complete 9.1% to 27.9% of the stateful workflows (per-model table in W2).

Q3. Uncertainty and the source of the human trace gains

  1. Comparing models website by website, with every website counting once: Claude-Opus-5.5 scores higher than every other model on 83% to 96% of these websites. Claude-Opus-5 scores higher than every model below it on 63% to 89%. Every other model scores higher than Qwen-3.8-Max on 61% to 94%.
  2. The human trace raised the score on 10 sites. On 7 of them, the run without the human trace could not start. On the other 3, the agent's website already worked without the human trace, and the trace made it better.

Q4. WebsiteBench and RecreationWorld

RecreationWorld also gives an agent a running reference and asks it to rebuild it. The two benchmarks differ in what is rebuilt, how the data were constructed, how it is tested, and what it is used for.

  • What is rebuilt. RecreationWorld covers five platforms (Ubuntu, macOS, Windows, Android and Web), with 50 tasks collected from GitHub on each. Its web references are frozen static snapshots of pages. WebsiteBench focuses on production-level websites, and its 74 references are full-stack websites with a server, user accounts and stored data, so the agent must rebuild the server side as well.
  • Data construction process. RecreationWorld constructs its data from public open-source applications on resource websites such as GitHub. Its authors note that public reference implementations may have appeared in the pretraining data of models, and they list newly constructed references as future work. WebsiteBench builds every reference from scratch. Website development experts wrote all 74 as new full-stack websites for this benchmark, and a second developer reviewed each reference website and wrote its hidden tests. The server code and the tests did not exist in any public repository before release, so a model is unlikely to have seen them on GitHub.
  • How it is tested. RecreationWorld checks what a page shows and does through its accessibility tree, plus screenshots that a vision-language model judges. WebsiteBench checks what the server returns, multi-step workflows, whether data survives a restart, and whether one user's data stays hidden from other users. All tests are fixed programs, with no LLM judge. This design makes the evaluation process of WebsiteBench more stable.
  • What it is used for. RecreationWorld also uses its environments to generate trajectories for training models. WebsiteBench is built to evaluate frontier agents.

Additionally, RecreationWorld was posted to arXiv on September 18, 2026 (revised September 21), the day of the ICLR 2027 abstract deadline and a week before the paper deadline, so it is contemporaneous with our submission.

We sincerely appreciate your careful, thoughtful, and constructive suggestions. Your feedback has helped us clarify the scope of our claims, and strengthen our analysis of agent behavior. We hope that the additional analyses and clarifications address your concerns.

[1] Bai, Shuai, et al. "RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents." arXiv preprint arXiv:2609.22000 (2026).

Dear Reviewer oKLi,

Thank you again for your thoughtful review. As the discussion period wraps up, we look forward to your thoughts on our responses, specifically the per-model runnable breakdown, the equal-weight site-level analysis, and our clarifications on Figure 4c and RecreationWorld. If they address your concerns, we would deeply appreciate your consideration in raising your score.

Reviewer_b5CbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WebsiteBench asks agents to browse a local website and build a copy that works on its own without internet access. It includes 74 websites built by experts and 50,356 hidden tests for appearance, page responses, user actions, and system requirements. The paper tests nine models and studies what agents miss while browsing, what they fail to build, whether human browsing records help, and whether forcing agents to keep working helps.

Critical strengths
  1. The task is useful because agents must both explore the website and make its features work. Tests check whether data survives a restart, users can access only what they should, and actions work across several steps.
  2. The authors put considerable work into building and reviewing the websites and tests. Agents cannot read the reference code or hidden tests, and the tests run automatically without an LLM or VLM judge, which keeps the evaluation fair and makes it possible to evaluate many outputs. The human-browsing study also shows that giving agents guidance can help them build better copies.
Critical weaknesses
  1. It is not clear enough what new ability this benchmark measures, and the claims about agent failures go beyond the runs studied. The main difference from the generation benchmarks in Table 7 is that agents explore through a browser, and comparing this with giving agents the same observations or requirements would help show why that difference matters. The broad failure claims mainly come from 20 Grok runs and one Kimi continuation example. How the sites and requirements were chosen is unclear, especially for the finding that only 0.9% of observed requirements were correctly built. The share of workflows whose final outcome was observed before the first implementation edit is also about 40% in Section 5.1 but 3.3% in Section 5.2.
  2. More evidence is needed that the tests give reliable scores. Appendix D.5 describes checks with deliberately broken websites, repeated screenshots, and two test runners, but gives no numbers for the first two. For the two runners, the only result is a Claude-Opus-5 rescoring in which the overall score does not change. The reference passing does not show that other correct solutions pass or broken ones fail. The same rescoring shows that removing visual checks leaves the Claude-Opus-5 score unchanged, so the effect of the visual threshold on the other models is unknown. Websites with more tests also count more toward Overall, so adding similar tests can give some features more weight, and the full model comparison has no confidence intervals. Failures to build or start a website are not reported separately from features that do not work.
Questions for authors

How would agents score if they were given the same observations or requirements directly instead of exploring through the browser? Please report the results of the broken-website, repeated-screenshot and two-runner checks in Appendix D.5, how the visual threshold affects scores for all models, and how long evaluation takes. Please add scores with equal weight per website and confidence intervals for all nine models in Table 1, and report build or startup failures separately. Please explain how the 20 analyzed sites and the observed requirements were selected, and reconcile the ~40% and 3.3% workflow figures.

6 comments
AuthorsOpenReview ↗

(1/5) Author Responses

Dear reviewer b5Cb,

We sincerely thank you for your thoughtful review. We appreciate your recognition that WebsiteBench requires agents to both explore a website and make its features work, as well as your positive assessment of our expert construction and review process, the isolation of reference code and hidden tests, our fair and automated evaluation without an LLM or VLM judge, and our human-browsing study. We address each weakness and question below.

W1. What WebsiteBench measures

The website-generation benchmarks in Table 7 give the agent its evidence up front, as screenshots, designs, prototypes with written requirements, or a video. In WebsiteBench, the agent receives the task instructions and browser access to a running reference website with its own server, and nothing lists the pages, workflows or stored data it has to rebuild. A WebsiteBench run therefore tests three abilities together:

  • Finding the website behavior through active exploration. The agent decides which pages to open and which workflows to carry out, including multi-step workflows that change data on the server.
  • Keeping what it found over a long-horizon run. Agents take 174 to 2,107 turns per task on average (Table 1), so behavior observed early must still be accessible during code written stage many turns later.
  • Rebuilding the server side. The hidden tests check data stored on the server, whether it survives a restart, and whether one user's data stays hidden from other users. Such abilities are important for agents to build production-level websites that work in the real world.

Q1. Scores when agents receive the same observations or requirements directly

We ran GPT-5.6-Luna on 8 of the websites analyzed in Section 5.2, under three conditions. All three conditions have the same time limit. Conditions (b) and (c) only add one read-only folder to the agent's workspace and one instruction sentence that points to it.

  • (a) Browser exploration only.
  • (b) Browser plus a human trace: the expert's recorded actions, the page text at each page load or state change, and screenshots.
  • (c) Browser plus a written requirement list, provided by the experts who annotated the benchmark. It lists pages, server requests and their effects, multi-step workflows, and different subpages' functionalities.
ConditionOverallRequestBrowserSystem
(a) Browser only14.219.88.833.3
(b) + human trace14.821.08.935.2
(c) + requirement list16.529.310.135.9

With the requirement list, the score is higher on 6 of 8 websites, lower on none, equal on 2. The gain is +2.3 points (95% confidence interval [0.7, 4.6], resampling websites), and the largest change is on Request tests (19.8% to 29.3%). So discovering the requirements is part of the difficulty of WebsiteBench, and related benchmarks listed in Table 7 that provide the requirements directly do not measure it. Even with the requirement list, the agent fails 83.5% of all tests and 89.9% of the Browser workflow tests. This is because the implementation of production-level websites is also challenging. WebsiteBench therefore measures both discovering the behavior and implementing it.

In Section 5.1, the human trace raised Grok-4.6's score on 10 of 18 websites and lowered it on 2. On 7 of these 10 websites, the run without the human trace could not start (there are bugs in the submitted code). On the other 3, the agent's website already worked without the human trace, and the human trace made it better and pass more hidden tests.

AuthorsOpenReview ↗

(2/5) Author Responses

W1. Scope of the failure claims

We extended the trajectory analysis to trajectories of all nine models. Examples are provided at https://anonymous.4open.science/r/analysis-6E0C/.

ModelRead-only workflows completedStateful workflows completedStateful, before first code editRuns ended by the agent
Claude-Opus-5.574.7%36.5%12.6%72%
Claude-Opus-566.3%30.5%13.4%67%
GPT-5.6-Sol38.6%28.3%27.9%100%
GPT-5.6-Luna12.8%9.5%9.1%99%
Grok-4.655.2%15.5%11.8%81%
GLM-5.3-Flash53.9%11.5%11.4%92%
Kimi-K354.4%27.0%23.9%83%
DeepSeek-V4.1-Flash55.5%19.2%13.0%85%
Qwen-3.8-Max68.4%19.4%15.7%84%
ModelOverallTurnsStateful workflows completedMedian run time (h)
Claude-Opus-5.522.52107.136.5%9.7
Claude-Opus-518.7980.530.5%8.0
GPT-5.6-Sol15.6212.328.3%0.6
GPT-5.6-Luna13.2174.09.5%0.7
Grok-4.611.6403.915.5%2.6
GLM-5.3-Flash9.7318.811.5%1.0
Kimi-K38.9604.327.0%3.1
DeepSeek-V4.1-Flash8.5572.619.2%2.8
Qwen-3.8-Max5.7252.119.4%1.8
  • Underexploration. Agents complete only 9.5% to 36.5% of the stateful workflows (workflows that change server state) on the reference website over the whole run, and every model completes fewer stateful workflows than read-only workflows.
  • Failure to implement observed behavior. 96.1% of the stateful workflows that agents completed on the reference website during the exploration stage fail in the website they submitted.
  • Early stopping. Except for the two Claude models, the median run lasts 0.6 to 3.1 hours despite the 16-hour limit, and 81% to 100% of runs end because the agent stops on its own. When we asked GPT-5.6-Luna to run 2 more hours, the score rose in 6 of 8 resumed runs.
  • Browser tests are the most difficult. Browser tests, which contain multi-step workflows, have the lowest pass rate for every model in Table 1 (1.8% to 18.0%, against 20.4% to 63.5% for Request tests).

Additionally, tests have a higher pass rate when the agent explored their related workflows on the reference website. A test's workflow counts as explored when the agent performed every step of it and then read the final page. In the submitted websites, 20.0% of the tests related to an explored workflow pass, against 9.1% of the other tests. Within the same run, the pass rate of tests related to an explored workflow is 9.5 points higher (95% confidence interval [7.1, 12.2]), and this difference is positive for each of the nine models.

W1 and Q4. Selection of the 20 websites and the observed requirements

Websites. We randomly select 20 websites from the total 74 websites for further analysis. Averaged over the nine models, their score falls within the middle half of randomly drawn sets of 20 websites (73rd percentile), so their difficulty is typical of the benchmark.

Requirements. We listed 316 requirements on the 20 websites. Each is a group of pages, a stateful workflow or a server behavior, for example sign-up with email verification, catalog filtering and sorting, or data kept after a restart. A requirement counts as one of the 111 observed requirements when three conditions hold: the trajectory shows that the agent observed it, the run made a code edit, and the hidden tests linked to it ran on the submitted website. The other 205 miss at least one condition, for example because the submitted server did not start, so their tests could not run, or because the run never made a code edit. A requirement counts as correctly implemented only when all of its linked tests pass. 96.1% of the stateful workflows that agents completed on the reference website fail in their submitted website. Therefore, there's gap between exploration and implementation.

W1 and Q4. ~40% and 3.3%

The two numbers use different run sets and definitions.

  • Figure 4c (about 40%): for each website, we count the stateful workflows whose final page address appears in the agent's trajectory before its first code edit, and divide by the website's number of stateful workflows. The figure shows the median of this share across websites: 40.0% for independent runs (4 of 10 workflows at the median website) and 77.8% with the human trace (7 of 9).
  • 4/122 (3.3%): the 122 core workflows of Section 5.2, taken from the Grok-4.6 trajectories that reached a first code edit. A workflow counts only if the agent saw its final result before that edit.

The 4/122 criterion is stricter: it requires seeing a workflow's final result, while Figure 4c only requires seeing the address of its final page. Our comprehensive analysis applies one criterion throughout (every step performed and the result page read): before the first code edit, agents complete 9.1% to 27.9% of the stateful workflows.

AuthorsOpenReview ↗

(3/5) Author Responses

Q2. Appendix D.5 results

Other correct implementations pass. On two websites, Confluence and Simplify-Jobs, we chose a set of 70 tests per website and built 4 other correct versions that change how the website works inside but keep what users see. Each version passes every test that exercises the part it changed, and all 70 tests in its set.

WebsiteChangeTests on the changed part passedExtra failures
ConfluenceJSON-file storage instead of SQLite27/270
ConfluenceOverview table built in the browser instead of on the server12/120
Simplify-Jobs12 CSS classes renamed27/270
Simplify-JobsAPI endpoints moved to /api/v221/210

Broken implementations fail. On the same two websites, we built 10 broken versions, each breaking one behavior. Before running the tests, we chose the tests that check the broken behavior (targeted tests) and tests that the change should not affect (control tests). All 107 targeted tests fail. All control tests still pass.

WebsiteBroken behaviorTargeted tests failed
ConfluenceStatic copy, nothing is saved23/23
ConfluenceSubmit button disabled11/11
ConfluenceTitle lost after a page edit3/3
ConfluenceWorkspace shared across accounts2/2
ConfluencePage section hidden (visual)3/3
ConfluenceFonts changed (visual)13/13
Simplify-JobsData lost after a server restart8/8
Simplify-JobsCompany page returns HTTP 50022/22
Simplify-JobsNote lost after creating an item16/16
Simplify-JobsData shared across accounts6/6

Checks fail when the behavior is wrong or missing. While writing the tests, we took 1,795 checks on 42 websites and ran each one again with one change: either the result that the check examines was altered (for example, a cart quantity above the website's limit), or the step that produces that result was removed. Every check failed after the change.

Repeated screenshots. We ran the reference implementation 176 times on 70 websites, producing 23,706 visual similarity scores. 97.3% of them are exactly 1.0, and the lowest is 0.886, above the 0.75 threshold. No visual test changes between pass and fail across repeated runs.

Two runners. Both runners of Equation 3 (Playwright and Browser Use) executed the browser workflow tests. On the tests that both runners executed, they agree on 99.90% of results (Cohen's kappa 0.998; per model 0.994 to 1.0). Most of the disagreements come from tests where Browser Use could not locate a page element.

Re-evaluation on a second machine. We re-ran the hidden evaluation (GLM-5.3-Flash, Kimi-K3 and Qwen-3.8-Max) on a second machine. 7,508 of the 7,510 test results (99.97%) are identical, and 17 of the 18 submitted websites receive exactly the same score; the 2 differing results come from a timeout of the reference website on the first machine. On both machines, the two runners agree on all 1,447 browser workflow tests, and visual similarity scores differ between machines by less than 0.0001.

Reviewer_CsxdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WebsiteBench, a benchmark of 74 expert-built and independently reviewed website reconstruction tasks with 50,356 hidden tests. Agents explore locally hosted reference websites through a restricted browser interface and build runnable offline implementations without access to the reference source code or hidden evaluation. The evaluation uses Request, Browser, and System tests to assess functional behavior, visual fidelity, persistent state, and runtime requirements. Experiments with nine models yield a best overall test pass rate of 22.5%. The paper further analyzes incomplete workflow exploration, failures to implement observed behavior, and early stopping, and reports improved reconstruction scores when agents receive human exploration traces.

Critical strengths
  1. A concrete task combining discovery and implementation. Agents must discover implicit website behavior through interaction and translate their observations into executable software. Allowing agents to revisit the reference during development provides a setting for studying how exploration, implementation, and validation interact.

  2. Expert construction and quality control. The benchmark spans 74 websites across nine categories, with 16 website development experts involved in construction and review. Independent review of the reference implementations and validation of the tests against those implementations provide quality control for the corpus.

  3. Evaluation of multiple aspects of reconstruction. The hidden tests cover page responses, browser workflows, runtime constraints, and persistent state. The protocol combines deterministic functional checks with specified visual checkpoints and evaluates submissions in an isolated environment, providing explicit criteria for assessing reconstructed websites.

Critical weaknesses
  1. The Introduction contains too much methodological detail, and its current structure is not sufficiently clear and does not adequately highlight the novelty of the work. The subsequent sections also lack concision. The Related Work section is placed entirely in the appendix. I consider reorganizing the manuscript particularly important, as the current writing and presentation are not sufficiently polished.

  2. The paper substantially overlaps with existing benchmarks for full-stack website generation and interactive reconstruction, with autonomous exploration of the reference website as its main additional dimension. Although this setting has potential value, the paper does not sufficiently demonstrate which capability gaps it reveals that existing evaluations do not adequately expose. Consequently, the benchmark's incremental contribution and distinct research value remain unclear.

Questions for authors
  1. What is the benchmark's incremental value? Compared with Vision2Web, UI2App, and IWR-Bench, the main additional dimension appears to be autonomous requirements discovery. Could the authors clarify which capability gaps this setting reveals that existing website construction evaluations do not adequately capture? Is there controlled evidence on the same tasks that separates difficulties in discovering requirements from general implementation difficulties?

  2. At which stage do failures to implement observed requirements occur? Section 5.2 reports that only one of 111 observed requirements was correctly implemented. Does "observed" mean that the relevant content appeared in a screenshot or tool output, or that the trajectory provides evidence that the agent correctly understood the behavior? Could the authors distinguish misunderstanding, forgetting, planning omissions, and coding errors, rather than grouping these cases as implementation failures?

  3. Does exploration coverage before the first edit sufficiently support the conclusion of overall underexploration? The task allows exploration, implementation, and validation to be interleaved, but the core coverage metrics only consider activity before the first source edit. Would the conclusion still hold when considering complete runs or the relevant exploration preceding the implementation of each feature? Can missing observations be linked to specific tests that ultimately fail?

  4. Through what mechanism do human exploration traces improve performance? In Section 5.1, the mean score increases from 2.8% to 9.8%. Does the gain mainly come from discovering additional behavior, reducing search costs, improving implementation planning, or reducing build and runtime failures? Could the authors link newly observed behavior to newly passed tests to explain this result more precisely?

  5. How broadly do the continuation experiments support the conclusions? The key improvement in Section 5.3 comes from a single website and model case, while the intervention both increases computation and introduces completion review. How do the authors distinguish the effects of additional computation from those of review feedback? Is there evidence across tasks that premature stopping is a widespread and important bottleneck?

4 comments
AuthorsOpenReview ↗

(1/3) Author Responses

Dear reviewer Csxd,

We sincerely thank you for your thoughtful review and for recognizing WebsiteBench as a concrete task that combines discovery and implementation, and for your positive assessment of our expert construction and quality control and of how WebsiteBench evaluates several aspects of reconstructing a website. We address each concern below.

Q1. Incremental value and controlled evidence

Vision2Web, UI2App and IWR-Bench give the model its evidence up front. Vision2Web gives UI prototypes and assets, textual specifications and structured requirement documents. UI2App gives a set of static screenshots. IWR-Bench gives a user interaction video. WebsiteBench is different from these related papers. In WebsiteBench, the agent receives the task instructions and browser access to a running reference website with its own server, and nothing lists the pages, workflows or stored data it has to rebuild. The agent must find the website behavior itself, keep what it found over long runs (174 to 2,107 turns per task on average, Table 1), and rebuild the server side as well. The hidden tests check data stored on the server, whether data survives a restart, and whether one user's data stays hidden from other users.

Our analysis of the trajectories of all nine models shows three capability gaps. The first gap appears only when the agent must find the behavior itself. Examples are provided at https://anonymous.4open.science/r/analysis-6E0C/.

  1. Agents rarely carry out the workflows that change data. Across all runs, they complete 56.6% of the read-only workflows on the reference website but only 21.4% of the stateful workflows (workflows that change server state, for example submitting a form), and each of the nine models completes fewer stateful than read-only workflows. Completing the workflow is associated with passing its test. Tests whose workflow the agent completed before its first code edit for that page pass 22.5% of the time, against 8.6% of the others. Within the same run, the difference is +10.6 points (95% confidence interval [7.8, 13.8]) and positive for each of the nine models. For 62.1% of failed tests, the agent never completed the test's workflow during exploration. Benchmarks that give the evidence up front do not measure whether the agent carries out these workflows and exhibit active exploration abilities.
  2. Agents rarely turn what they observed into working code. Of the stateful workflows that agents completed on the reference website, 96.1% fail in the website they submitted. Claude-Opus-5 completed 147 of the 150 stateful workflows of website 022, and none of them works in its submitted code (Example 15 at https://anonymous.4open.science/r/analysis-6E0C/).
  3. Agents stop early. Except for the two Claude models, agents end their runs after a median of 0.6 to 3.1 hours of the 16-hour limit.

Across Table 1, Browser tests, which contain multi-step workflows, have the lowest pass rate for every model (1.8% to 18.0%, against 20.4% to 63.5% for Request tests).

To separate discovery from implementation on the same tasks, we ran GPT-5.6-Luna on 8 websites under two conditions: browser access only, and browser access plus a written requirement list. Both conditions use the same agent setup, tasks and time limit. The list was provided by annotators who contribute to the consturction of WebsiteBench. It contains pages design, server requests and their effects, multi-step workflows. With the list, the score was higher on 6 of 8 websites and lower on none, and Request tests rose from 19.8% to 29.3%. The overall score rose from 14.2% to 16.5% (+2.3 points, 95% confidence interval [0.7, 4.6]). So discovering the requirements is part of the difficulty, and benchmarks that supply the requirements do not measure it. Even with the list, the agent failed 83.5% of all tests and 89.9% of the Browser workflow tests, so implementing multi-step workflows stays hard after the requirements are given. WebsiteBench measures both requirements discovery and code implementation abilities.

AuthorsOpenReview ↗

(2/3) Author Responses

Q2. Where implementation failures occur

What "observed" means. In Section 5.2, a requirement counts as observed when three conditions hold: (1) the agent acted on the reference website, for example by opening a page or submitting a form; (2) its tool returned a screenshot, page content or interface state; and (3) that output contains the information the requirement needs. For example, opening the sign-up page and receiving a screenshot of its fields counts as observing the form; submitting it and receiving the email-verification code counts as observing the next step. The stage analysis below checks the agent's messages, notes and code for signs of misunderstanding.

In the 20 Grok-4.6 runs of Section 5.2, code exists for almost every failed requirement. Of the 110 observed requirements that were not correctly implemented, 109 have code for the behavior in the submission, and 1 has no corresponding code. In almost every case, the agent wrote code for what it observed, but the code does not correctly reproduce the behavior.

Two examples.

  • AMC Theatres: the agent opened /movies?tab=coming-soon and captured the Coming Soon list, then implemented tab switching on the page. Its Movies page does not read the tab parameter in the address, so opening that address does not select the tab, and the tests for this interface fail.
  • PetSmart: the agent's screenshot of the reference homepage showed the Fish, Bird and Small Pet cards with matching animal images. Its code connects each label to the right category but assigns dog images to all three cards.

In both cases the agent had observed the behavior, and the error still occurred when writing the code.

Q3. Complete runs, exploration before each feature

The finding that agents explore too little also holds over complete runs. Each hidden browser test describes one workflow as a list of steps on the reference website. A workflow is stateful when at least one step changes data on the server, for example submitting a form. We matched each test's steps against the agent's recorded browser actions and HTTP requests to the reference website. A workflow counts as completed when the agent carried out every step and then read the final page.

ModelOverallTurnsStateful workflows completedMedian run time (h)
Claude-Opus-5.522.52107.136.5%9.7
Claude-Opus-518.7980.530.5%8.0
GPT-5.6-Sol15.6212.328.3%0.6
GPT-5.6-Luna13.2174.09.5%0.7
Grok-4.611.6403.915.5%2.6
GLM-5.3-Flash9.7318.811.5%1.0
Kimi-K38.9604.327.0%3.1
DeepSeek-V4.1-Flash8.5572.619.2%2.8
Qwen-3.8-Max5.7252.119.4%1.8

According to the table above, models only completed 9.5% to 36.5% stateful workflows.

We then compared completed workflows with the hidden test results.

  • Complete runs. Tests have a higher pass rate when the agent explored their related workflows on the reference website. In the submitted websites, 20.0% of the tests related to an explored workflow pass, against 9.1% of the other tests. Within the same run, the pass rate of tests related to an explored workflow is 9.5 points higher (95% confidence interval [7.1, 12.2]), and this difference is positive for each of the nine models.
  • Exploration before each feature. Of the tests whose workflow was completed during exploration before the first code edit, 22.5% pass, against 8.6% of the other tests. Comparing tests within the same run, the difference is +10.6 points (95% CI [7.8, 13.8]), positive for each of the nine models.
  • Merely visiting a page is not enough. Within the same run, tests whose page the agent opened on the reference website and tests whose page it never opened have almost the same pass rate (difference -0.7 points, 95% CI [-3.1, 1.4]).
  • Failed tests without a completed workflow. For 62.1% of failed tests, the agent never completed the test's workflow on the reference website during the exploration.

Q4. Mechanism of the human-trace gain

In Section 5.1, the trace raised the score on 10 of the 18 websites, lowered it on 2 and left 6 unchanged, and the number of websites scoring above zero rose from 6 to 12.

  • Reaching a working website. On 7 of the 10 websites where the human trace raised the score, the run without the human trace did not produce a website that could be built or started, so it scored 0. With the help of human trace, the website started, and on 5 of these 7 it also passed more browser workflow tests: 165 on Backcountry, 84 on Samsung, 62 on Meetup, 34 on Target and 28 on L.L.Bean, against none without the trace.
  • Seeing and implementing more behavior. On the other 3 of the 10 websites, the run without the human trace already produced a working website. With the human trace, the passed browser workflow tests rose from 0 to 51 on Apple and from 0 to 25 on Tripadvisor.
AuthorsOpenReview ↗

(3/3) Author Responses

Q5. Premature stopping across tasks

Premature stopping is common across models. 82% of the runs ended before 8 hours, half of the 16-hour limit. For the seven models other than the two Claude models, 81% to 100% of runs end with the agent's own final message. Except for the two Claude models, the median run lasts 0.6 to 3.1 hours while 84% to 94% of the hidden tests fail.

The Lowe's case in Section 5.3 combines extra time with a completion review by the /goal hook. To separate the two, we resumed the GPT-5.6-Luna runs from Q1 on 4 websites in a fresh container providing each last run's final code. Each time the agent stopped, the continuing group received "Continue working on the task." The review group was asked to go through the reference website page by page and workflow by workflow, list what is missing or different in its own implemented website, and fix it. The continuing group raised the score on all 4 of its websites. The review group raised it on 2 of its 4 websites and left the other 2 unchanged. Simply asking agents to work longer was enough to raise the score, indicating that stopping early is one reason agents underperform on WebsiteBench.

We sincerely appreciate your careful and constructive review. The additional results separate discovery from implementation, link missing exploration to failed tests across all nine models, locate where implementation failures occur, and show that continuing stopped runs raised the score. We hope these results address your concerns, and we would appreciate your reconsideration of our work.

Dear Reviewer Csxd,

Thank you again for your review. As the discussion period draws to a close, we would appreciate your thoughts on our responses, especially the controlled study separating discovery from implementation and the full nine-model exploration analysis. If these address your concerns, we would be very grateful if you could consider raising your score. Happy to clarify anything further!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.