- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a new benchmark called WebsiteBench. It is designed to evaluate whether AI agents can explore a locally hosted reference website and reconstruct it. The benchmark has 74 tasks across multiple domains and has been peer-reviewed by experts. The authors have also provided 50k hidden tests and evaluated 9 frontier models. The authors also have many interesting findings on analyzing 17 trajectories, such as shallow workflow exploration and failure to implement the observed behavior.
Critical strengths
- The problem is genuinely interesting and timely. I do see the motivation of the problem in the current era, where AI coding agents are becoming widespread. The task of website reconstruction via exploration can have a ton of practical usability.
- I think the overall engineering of the benchmark and evaluation is really good. The experts have built self-contained full-stack applications with persistent states. This is much better than scraping live webpages.
- I also like the metrics used in the paper. The normalized SSIM is a neat trick and pretty intuitive.
- There has been a good amount of experimental depth, with nine frontier models being benchmarked.
Critical weaknesses
The paper has a direct naming and concept collision with another paper from COLM 2026 (https://openreview.net/pdf?id=oX9AW8cXMK). I checked that this work has been public before ICLR's September deadline. Hence, this work should have been cited and compared. As this is a relatively newer work, it is highly likely that the authors missed it. However, this does question the novelty of the paper. Both papers have a lot of similarities, e.g., the problem formulation and core workflow, information isolation and anti-cheating block, and the offline reproducibility. The papers also have a similar conclusion that reconstruction remains largely unsolved by frontier models, and agents are much better at capturing surface-level pages or visual appearances without being able to recover or understand the underlying structure. I do agree that there are a lot of dissimilarities in benchmark construction, the agent's observation, and the evaluation, but I would still not consider that to be sufficient. I also find the current submission weaker in certain claims, such as the fact that the findings are based on only 17 inspected trajectories (Line 115), which is a bit on the lower end, and could be noisy.
Questions for authors
- Could the authors please differentiate their work from WebsiteBench, and would you consider renaming the current benchmark as well?
- Can the authors provide new findings that haven't been discussed in previous literature?
- I would really appreciate it if more trajectories were analyzed for more grounded conclusions.
3 comments
(2/2) Author Responses
We further use these trajectories to support the findings of the paper. Each hidden browser test is a scripted user workflow with a defined end state, and the hidden tests list the workflows each website supports. A workflow is completed when the agent has carried out every step and then read the resulting page. In WebsiteBench, the agent must discover the website behavior itself by proactive exploration. This differenciate out work from other benchmarks, such as Vision2Web [1] (input: UI prototypes and assets, textual specifications, structured requirement documents), UI2App [2] (input: A set of static screenshots) and IWR-Bench [3] (input: User interaction video).
| Model | Read-only workflows completed | Stateful workflows completed | Stateful, before first code edit | Runs ended by the agent |
|---|---|---|---|---|
| Claude-Opus-5.5 | 74.7% | 36.5% | 12.6% | 72% |
| Claude-Opus-5 | 66.3% | 30.5% | 13.4% | 67% |
| GPT-5.6-Sol | 38.6% | 28.3% | 27.9% | 100% |
| GPT-5.6-Luna | 12.8% | 9.5% | 9.1% | 99% |
| Grok-4.6 | 55.2% | 15.5% | 11.8% | 81% |
| GLM-5.3-Flash | 53.9% | 11.5% | 11.4% | 92% |
| Kimi-K3 | 54.4% | 27.0% | 23.9% | 83% |
| DeepSeek-V4.1-Flash | 55.5% | 19.2% | 13.0% | 85% |
| Qwen-3.8-Max | 68.4% | 19.4% | 15.7% | 84% |
Finding 1. Underexploration. Agents read pages but rarely carry out the workflows that change state. Every model completes fewer stateful workflows than read-only ones, and before its first code edit for writing the website, no model has completed more than 27.9% of the stateful workflows. Agents start building before they have explored most of the website behavior.
Finding 2. Failure to implement observed behavior. Completing a workflow on the reference website during exploration is not enough for correct implementation. Claude-Opus-5 completed 147 of the 150 stateful workflows of site 022 (Example 15), including registration, sign-in and search, and none of them work in its submitted code.
Finding 3. Early stopping. The instruction states the time budget and tells the agent to use it ("You have up to 16 hours for this task"). Except for the two Claude models, agents stop after a median of 0.6 to 3.1 hours (table in part 1/2), while 84% to 94% of the hidden tests fail, and 81% to 100% of their runs end with the agent's own final message. GPT-5.6-Sol ended its run on site 058 after 0.7 hours with only 6.0% of its browser tests passing (Example 22).
We provide 30 examples, ten per finding and covering all nine models, together with per-model tables and figures, in an anonymous link: https://anonymous.4open.science/r/analysis-6E0C/.
We hope this clarification resolves the overlap concern and that our extended analyses strengthen our claims and findings. We would appreciate your reconsideration of our work, given that there is no overlap with previous publications and that we have conducted comprehensive analyses.
[1] He, Zehai, et al. "Vision2web: A hierarchical benchmark for visual website development with agent verification." arXiv preprint arXiv:2603.26648 (2026).
[2] Chen, Grace Man, et al. "UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation." arXiv preprint arXiv:2607.06306 (2026).
[3] Chen, Yang, et al. "IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?." International Conference on Learning Representations. Vol. 2026. 2026.
Dear Reviewer pkkv,
Thank you again for your constructive review. As the discussion period is drawing to a close, we would be grateful if you could take a look at our responses, specifically our expanded trajectory analysis across all trajectories. If these resolve your concerns, we would sincerely appreciate it if you could consider raising your rating. Happy to clarify anything further!
(1/2) Author Responses
Dear reviewer pkkv,
We sincerely thank you for recognizing our paper as interesting, timely, and well motivated, and for emphasizing its interesting findings and practical utility. We also appreciate your positive assessment of our dataset construction process, in which experts build self-contained full-stack applications, as well as our evaluation design, evaluation metrics, and the depth and breadth of our experiments.
Thank you also for raising the concern about potential overlap. We checked the provided link (https://openreview.net/pdf?id=oX9AW8cXMK) and found that it is not publicly accessible. We also reviewed the COLM 2026 accepted papers list (https://colm.eventhosts.cc/Conferences/2026/AcceptedPapers) and found no paper titled "WebsiteBench." The referenced paper has not been published and does not appear in any proceedings; therefore, it is neither prior work nor contemporaneous published work.
We appreciate your suggestion to analyze more trajectories. We have analyzed all trajectories from nine models across 74 tasks. The results are summarized below.
"Stateful workflows completed" is the share of the hidden-test workflows involving changes to server state (for example, form submissions) that the agent carried out on the reference website. The time limit is 16 hours.
As shown in the table, agents complete only 9.5% to 36.5% of the stateful workflows, and no model completes more than half. Except for the two Claude models, median run times range from 0.6 to 3.1 hours, despite the 16-hour time limit.