acceptodds
Under review as a conference paper at ICLR 2027

WebMockGym: Benchmarking Web Reconstruction with Autonomous Information Gathering

Abstract

Autonomous software development requires agents to determine what they need to know, acquire and organize relevant information, and use it to produce reliable implementations. Evaluating end-to-end development calls for tasks that leave exploration and implementation decisions to agents while providing clear criteria for judging the final result. Existing web-development benchmarks often supply screenshots or interaction recordings, offering limited evidence about development when agents must gather reference information themselves. We introduce WebMockGym, a benchmark of 50 websites and 50 web apps for studying autonomous development through website reconstruction. Agents explore a frozen reference website and build a separately runnable reconstruction of its entry page and captured first-hop pages. The reference provides an observable target, while agents decide what to inspect and how to implement and check their work. Fixed, hidden tests assess content, images, visual appearance, and cross-page navigation. The same reference-defined requirements apply to every reconstruction of a task, including requirements an agent may overlook. Experiments with six coding-agent systems reveal frequent navigation failures, and no system passes every applicable test on any scored task in the website track. Trajectory analysis associates earlier writing and more frequent inspection of the reconstruction followed by further writing with higher Web scores. WebMockGym provides a concrete setting for studying autonomous development through the quality of independently runnable reconstructions.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Web Reconstrction, Web Discovery, LLM Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QewsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WebMockGym, a benchmark of 50 websites and 50 web apps in which coding agents explore a frozen reference site and build a separately runnable reconstruction of its entry page and the pages one link away. Reconstructions are scored with about 500K hidden tests derived from the reference, covering content, images, visual appearance and navigation, and the evaluator is validated with expert rankings and controlled defects. Across six coding-agent configurations, the best Web AUC is 41.7, no Web reconstruction passes every test, and navigation is the weakest test family for most systems. Trajectory analysis associates earlier file writes and more frequent inspection followed by further writing with higher Web scores.

Critical strengths
  • The paper is clearly written and easy to follow, with concrete examples of the tests and short takeaway boxes that summarize the main results.
  • The setting differs from prior web-generation benchmarks, which supply screenshots, videos or text requirements. Agents must gather the reference information themselves by browsing, including links hidden behind menus, and are scored with fixed hidden tests that do not rely on LLM judges.
  • The experiments cover six coding-agent configurations built on recent models from several providers, and report per-task cost, time and token usage.
Critical weaknesses
  1. Benchmark data, tests and scoring code are not released
  • The repository contains only test-generation and test-freezing code. Its README states that no captured pages, page configurations, network recordings, generated tests or run reports are included, and its documentation describes the included fixture as "not a complete candidate-scoring runner".
  • The 100 tasks, the roughly 500K tests, the agent trajectories, the scores and the human rankings are therefore not available in the released materials, and none of the reported numbers can be reproduced or checked. The paper only states that the reference captures will be released later.
  1. The stated scoring condition is inconsistent with the released code
  • Section 3.1 states that reconstructions are scored "with JavaScript enabled". The released generator code, which the README describes as extracted from the test pipeline, repeatedly describes the scorer as JavaScript-disabled, for example "the frozen, JavaScript-disabled scorer" and "the scorer renders JS-off". Reference observations are also extracted with JavaScript disabled, with comments stating that this must match the scoring runtime.
  • If scoring was JavaScript-disabled, reconstructions whose menus or content rely on JavaScript would have been penalized, although the agent instruction (Appendix A.1) never mentions this. If scoring was JavaScript-enabled, the reference observations and the reconstructions were rendered under different conditions, contrary to the code's own requirement that reference extraction match the scoring runtime. Either way the meaning of the scores, especially on App, is unclear, and the released materials cannot resolve it.
  1. Weak alignment between the motivation and what is measured
  • The paper motivates the benchmark as evaluating autonomous software development, which it describes as obtaining the needed information, deciding how to implement, checking the result and "delivering a working implementation". The benchmark mainly measures how closely the visible content, appearance and links of an observable website are reproduced. The guidance study (Table 5) supports the information-gathering part, but beyond noting that the reference provides an observable target, the paper does not discuss how well website reconstruction represents autonomous development as a whole.
  • Functionality is explicitly excluded from scoring (Section 4, Appendix D.2), although the task selection, the agent instruction and the rating instructions all refer to it. The App tasks are selected "to cover different functions and interaction structures" (Section 2.2), the agent instruction asks agents to preserve the "interactions you judge important" (Appendix A.1), and the rating interface tells raters to "Compare menus, links, forms, and core functionality" (Figure 8). Whether a reconstructed calculator or form actually works is therefore not evaluated by any reported test family.
  1. No justification for the test thresholds
  • The pass conditions determine every score, but the paper gives no rationale, reference or sensitivity analysis for the chosen values. Examples include an image hash distance of at most 8 of 64 bits, a position tolerance of 128 by 80 pixels, a 16×12 layout grid with a per-band similarity of 0.78, color-share bands whose rule is not stated, a screenshot tolerance whose value is not reported, and a 1.5 to 5 second check timeout (Appendices A.3, A.4, B.1). Section 2.3 mentions calibrating the suite on the saved reference but does not explain what this involves or how the values were set.
  • The controlled studies in Appendix C.1 test responses only at the chosen values. The released code notes that the image threshold was "tightened from 12", but the paper does not say how or when the thresholds were chosen.
  1. Limited human evaluation
  • The human study ranks only the six agent reconstructions within each task, so it supports relative ordering but not the absolute meaning of the scores. The reference is shown but not ranked, and no blank or wget outputs are included as anchors.
  • The rating criteria differ from the automated tests. The rating interface in Figure 8 tells raters to compare forms and core functionality, which the tests do not cover, so it is unclear what the reported agreement (Kendall's τb of 0.705) validates.
  • Beyond describing them as three PhD participants, the paper gives no information on the raters' expertise, recruitment or compensation.
Questions for authors
  • Five configurations run in Claude Code and only GPT-5.6-sol runs in Codex, yet every table and figure that reports per-configuration results labels the configurations by model name, and Appendix B.7 attributes behavior to "GPT". Shouldn't the results be attributed to agent systems (harness plus model) rather than to models?
  • Shouldn't the same model be run in both harnesses, or all models in one harness, to separate harness effects from model effects?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Qwes

Thank you for the constructive feedback. We address the weaknesses and questions you raised as follows.

W1. Benchmark data, tests and scoring code are not released.

Response: With the camera-ready version, we will release the scoring runner, the tests, and the outcome of every test for all 600 reconstructions, with scripts that recompute every table.

W2. The stated scoring condition is inconsistent with the released code.

Response: We thank the reviewer for pointing out this inconsistency. Reconstructions were scored with JavaScript enabled, as Sec. 3.1 states, while the test generator records the expected values from the reference with JavaScript off. To check whether this mismatch affects the results, we first re-ran every test on the saved reference with JavaScript enabled and kept only the tests that the reference passes and a blank page fails, so that the reference receives a full score under the scoring condition.

We then rescored all six systems with these tests on the entire benchmark, and the comparison remains essentially unchanged. On Web the six systems keep the same order (Opus 5 53.1, GLM 43.2, DeepSeek 43.0, Kimi 31.4, GPT 22.6, Sonnet 5 16.2), and 12 of the 15 pairwise differences are significant, against 10 on all tests. On App no pair differs significantly under either scoring.

W3. Weak alignment between the motivation and what is measured.

Response: The benchmark measures autonomous development ability through website reconstruction, which is one form of autonomous development. We do not claim that it represents autonomous development as a whole, and the revised version will state this.

For functionality, our goal is a programmatic verifier, so every test takes its expected value from the frozen reference and runs automatically. On Web, the tasks are mostly content sites whose function is what they display and where their links lead, so these tests cover what the sites do.

On App, the selected sites expose their functions largely through navigation, and the tests replay the recorded hover and click actions before following each link (Sec. 2.3). A reconstruction that looks like the reference and navigates correctly thus reproduces most of its functionality. Responses to new input are not scored, and the revised version will state this boundary. On the rating instruction in Figure 8, please see W5.

W4. No justification for the test thresholds.

Response: We agree that the specific threshold values are design choices. What matters is whether the conclusions depend on them, so we re-scored the systems under alternative values. Scores shift by a few points, while the order of the systems stays largely stable. Relaxing the image distance from 8 to 12 raises the Image pass rate on Web by 1.7–2.4 points for every system without changing their order. On 20 randomly drawn tasks (10 Web, 10 App), doubling every timeout changes AUC by at most 3.5 points without changing the Web order, and halving or doubling the position tolerance on the App tasks changes AUC by at most 2.7 points without changing the App order.

The revised version will report all values and these analyses, including the two not stated in the paper: each color covering at least 0.5% of the viewport must keep between half and twice its reference share, and a screenshot check passes when fewer than 8% of pixels differ. The image distance was tightened from 12 to 8 so that a different image cannot pass as a match.

W5. Limited human evaluation.

Response: The human study is designed to check whether the automated scores order reconstructions as people do, since the benchmark is used to compare systems. Raters ranked the six reconstructions of each task against the shown reference. The automated ranking agrees with theirs at Kendall's τb of 0.705. The absolute scale is anchored on the automated side instead. A blank page scores near zero on both tracks (Figure 3a), and the wget mirror lies between the blank page and the agents (Table 4).

The raters judged overall fidelity to the reference. Their criteria also cover aspects that the tests do not check, such as forms. The agreement therefore shows that the automated ranking matches people's overall judgment even under broader criteria. It does not validate each test family on its own. What each family measures is checked separately by the controlled defects in Sec. 2.5. Removing images lowers the Image score by 33.0 points, and redirecting links lowers the Navigation score by 33.3 points.

The three raters are PhD students experienced in web development. The revised version will report these details.

Reviewer_r6nnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces WebMockGym, which uses website reconstruction as a task for evaluating end-to-end development. WebMockGym consists of 50 websites and 50 web apps.

During evaluation, agents explore a frozen reference and write and run code to build an independently runnable reconstruction. The agents can continually interact with the websites and revise the written website. The main takeaway is that more frequent interaction and revisits of the websites written can lead to better performance.

There are several concerns: AOC is not properly justified as a good metric for this benchmark; wget is not restricted from use; and the paper does not thoroughly discuss the infrastructure design that can be used to try to restrict the agents from cheating.

Critical strengths
  1. This paper provides a detailed analysis, for example, in Figure 7. This makes the tasks more concrete and easier to understand.

  2. This paper's evaluation pipeline is reasonable and clearly designed. It is a little similar to how we evaluate reverse-engineered software.

  3. This paper provided a wget baseline, which helps the reader better understand what score an agent will achieve if it only wget the website resources.

  4. This paper has a clear and interesting design. Learning from environments is an important ability and a sign of intelligence. This paper provided a concrete step towards letting agents learn from an unknown environment.

Critical weaknesses
  1. AUC is not a very good metric for these tasks. Evidence | Image | Perception | Navigation | AUC Kimi K3 | 76.5 | 64.5 | 77.2 | 47.2 | 48.2 Opus 5 | 74.2 | 61.8 | 75.4 | 44.9 | 49.7 Kimi K3 outperforms Opus 5 across four domains, but AUC is lower. Also, according to Table 3, Kimi K3 has better performance compared with GPT-5.6-sol in web tasks, and DeepSeekV4Flash has the same performance as GPT-5.6-sol. This is not aligned with https://www.kimi.ai/blog/kimi-k3 and https://arxiv.org/pdf/2606.19348. Using AUC for these tasks is not reasonable to me.

  2. In Figure GLM-5.3-Flash, DeepSeek V4 Flash and Claude Opus 5 use the same figures.

  3. Wget and agent cheating. According to Table 4, app-50, wget is better than GLM-5.3-Flash, Navigation. Entry is better than Opus 5. This is not very reasonable: if 43.7% of the tasks are solvable by wget, why should this benchmark be used to test frontier models? Also, agents have access to bash tools, and they can just wget themselves.

  4. In Table 6, Claude Sonnet 5's running time is missing. Kimi K3 Web token and cost estimates use six runs with complete request accounting. Timing and trajectory measurements use a separate set of recorded runs. This is a little confusing.

Questions for authors
  1. AUC is not a very good metric for these tasks. Evidence | Image | Perception | Navigation | AUC Kimi K3 | 76.5 | 64.5 | 77.2 | 47.2 | 48.2 Opus 5 | 74.2 | 61.8 | 75.4 | 44.9 | 49.7 Kimi K3 outperforms Opus 5 across four domains, but AUC is lower. Also, according to Table 3, Kimi K3 has better performance compared with GPT-5.6-sol in web tasks, and DeepSeekV4Flash has the same performance as GPT-5.6-sol. This is not aligned with https://www.kimi.ai/blog/kimi-k3 and https://arxiv.org/pdf/2606.19348. Using AUC for these tasks is not reasonable to me. Question: justify why AUC is important and required to use.

  2. In Figure GLM-5.3-Flash, DeepSeek V4 Flash and Claude Opus 5 use the same figures. Question: Why do three models have the same figure?

  3. Wget and agent cheating. According to Table 4, app-50, wget is better than GLM-5.3-Flash, Navigation. Entry is better than Opus 5. This is not very reasonable: if 43.7% of the tasks are solvable by wget, why should this benchmark be used to test frontier models? Also, agents have access to bash tools, and they can just wget themselves. Question: Does this paper use an anti-cheating strategy? Why does wget outperform frontier models?

  4. In Table 6, Claude Sonnet 5's running time is missing. Kimi K3 Web token and cost estimates use six runs with complete request accounting. Timing and trajectory measurements use a separate set of recorded runs. This is a little confusing. Question: Why is Claude Sonnet 5's running time missing? Why do Kimi K3 Web token and cost estimates use six runs with complete request accounting?

2 comments
AuthorsOpenReview ↗

Response to Reviewer r6nn

Q1. Why is AUC a suitable metric?

Response: AUC scores each task by its lowest test family and averages over tasks, so a site with accurate pages but broken links does not score well. A system can therefore lead on every family mean while fewer of its sites are good in every family: Opus 5 passes every test on more App tasks than Kimi K3 (16.0% vs. 14.0%), and their AUCs differ by 1.5 points. On Web the six systems keep the same order under AUC, the mean of family means, the per-task mean and the geometric mean. The cited reports do not evaluate this task.

Q2. Why do GLM-5.3-Flash, DeepSeek V4 Flash and Claude Opus 5 show the same figure in Figure 9?

Response: Figure 9 shows only the first viewport of the entry page, which all three reconstructions render the same as the reference. The scores cover every page and link of the task, and there the three differ: AUC 78.8 (GLM), 75.2 (DeepSeek) and 71.7 (Opus 5).

Q3. Why does wget outperform some frontier models, and is there an anti-cheating strategy?

Response: 43.7 is wget's Navigation score on App, not the share of tasks it solves: wget passes every test on 0% of Web tasks and 4% of App tasks, and on Web the best system's AUC is more than twice wget's. wget saves each page as served, so on small App tasks it can match agents on single columns; it performs no browser interaction and cannot reach links behind menus.

For anti-cheating, the tests are hidden, a hook blocks and logs access to benchmark-private files, and each reconstruction is scored from its own local files. Agents may download reference pages (Sec. 3.1), but mirroring alone meets the same limits as wget.

Q4. Why is Sonnet 5's running time missing from Table 6, and why are Kimi K3 Web token and cost estimates based on six runs?

Response: Sonnet 5's Web time is 37.4 minutes per task. For Kimi K3 Web, the original runs did not record cache details, so we use six to estimate the cache hit rate, while time and trajectory length use the original runs.

Thanks for the response. "For anti-cheating, the tests are hidden, a hook blocks and logs access to benchmark-private files, and each reconstruction is scored from its own local files." This does not answer the question. Agents have access to bash tools, and they can just wget themselves. In this case, why can wget still outperform some frontier models? Also, Kimi K3 outperforms Opus 5 across four domains, but AUC is lower. This question is not addressed.

Reviewer_KZtEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WebMockGym asks agents to browse and copy 50 websites and 50 web apps from saved versions. Agents must find and rebuild the starting page and saved pages reachable in one navigation step. About 504,000 hidden tests check content, images, appearance, and links. The paper compares six agent systems with wget and checks its scores using human rankings, deliberately broken pages, and code changes. It also tests extra browsing guidance.

Critical strengths
  1. The authors make a useful effort to check their scores. Automatic rankings agree with human rankings about as much as people agree with each other, and deliberately breaking images and links lowers the relevant scores. Saved reference sites and automatic tests also allow many outputs to be evaluated without an LLM or VLM judge, and the code-change study shows that some simple changes to the implementation leave scores mostly unchanged.
  2. Giving agents page and link guidance improves navigation scores for all six systems on five websites. This supports the view that finding the required pages and hidden links is part of the difficulty.
Critical weaknesses
  1. It is unclear how much actual development the scores measure, and the paper needs to explain more clearly what it adds to earlier website-rebuilding benchmarks. Agents gather information themselves, but they may also reuse content, and the wget setup is unclear. A stronger baseline that saves pages through a browser and fixes local links would help show how much development is needed. The App tests check pages and links, not whether the app's main functions work. AUC takes each task's lowest test-category score and then averages across tasks, but not every task has the same categories, and Appendix B.5 restricts only the page-count analysis to the 25 App tasks with all four categories.
  2. The model comparisons are harder to interpret than the paper suggests. Automatic tests can still favor certain implementations. Changing images to CSS backgrounds makes 7.16% of previously passing checks fail on average across the selected tasks, and matching elements by role and name with short timeouts also needs more checking. The main comparison does not show how much scores change across repeated runs or how certain the differences are. Models use different agent tools, and time, token, and continuation limits are unclear. Generation costs are reported, but evaluation time and evaluation compute are not.
Questions for authors

How was wget configured, and how would a browser-based copying baseline that fixes local links score? How do the Table 4 results change on tasks where all four test categories apply, or under other ways of combining category scores? Please report run-to-run variance on a subset of tasks, the time, token and continuation limits for each system, and the time and compute needed for evaluation.

1 comment
AuthorsOpenReview ↗

Response to Reviewer KZtE

Thank you for the precise review. We answer each point below.

W1. How much actual development do the scores measure?

Response: Agents may download reference resources and reuse them as they work (Sec. 3.1). Even so, correctly organizing these resources into a site that runs on its own is itself a form of autonomous development: the agent must find the pages one navigation away, including those reachable only through menus, and make the links between them work locally. We do not claim that this represents autonomous development as a whole, only that it is one way to verify an agent's capability.

We additionally ran the stronger baseline you suggest, in which headless Chromium saves the entry page and every page it links to and rewrites the links to the saved files. It reaches an AUC of 40.4 on Web and 58.5 on App. In the revised version, we will state the boundaries of what the benchmark measures more explicitly, including that the tests check pages and navigation rather than application functions.

W2. The model comparisons are hard to interpret.

Response: First, we acknowledge that the Image tests are sensitive to CSS background images. In the actual reconstructions, however, only 0.2% of failed Image tests match a CSS background image, so this sensitivity has little effect on the comparison. Similarly, renaming identifiers or adding wrappers changes only 0.48% of passing checks (Sec. 2.5), and doubling every timeout leaves the Web order unchanged. The revised version will state this sensitivity explicitly.

Second, we resample the 50 tasks to obtain 95% bootstrap intervals. On Web, Opus 5 leads every other system by 7.9–25.9 points, and all five differences are significant. Each task is a separate run, so these intervals include run-to-run variation as well as variation across tasks.

Third, the revised version will report evaluation time and compute. Scoring uses 4 browser workers per reconstruction and no LLM calls. One reconstruction takes 3.1 minutes on App and 60 minutes on Web on average, about 209 worker-hours for all 100 tasks.

Finally, for the generation configuration, we do not restrict the agents' tool calls. Both harnesses (Sec. 3.1) provide Playwright and shell tools, and the only restriction is the anti-cheating hook, which blocks and logs access to benchmark-private files. Runs also had no time, token, or continuation limit, so each agent could keep working until it finished.

Q1. How was wget configured, and how would a browser-based copying baseline that fixes local links score?

Response: wget starts from each task's entry URL and follows links one level deep. It saves each page with the resources it needs, including those on other hosts, and rewrites links to the local copies. We additionally ran the stronger baseline you suggest, in which headless Chromium saves the entry page and every page it links to and rewrites the links to the saved files. It reaches an AUC of 40.4 on Web and 58.5 on App.

Q2. How do the Table 4 results change on tasks with all four categories, or under other ways of combining category scores?

Response: We additionally report the average of the four category scores for each task, weighting every category equally. On Web the six systems keep the same order as under AUC; on App they lie within 2.8 points of each other (67.2–70.0). Every Web task has all four categories, so restricting to such tasks only affects App: on the 25 App tasks with all four categories, AUC is 33.3–45.7, against 45.0–51.9 on all 50 tasks.

Q3. What are the run-to-run variance, the time/token/continuation limits, and the evaluation compute?

Response: One final reconstruction per system and task is scored (Sec. 3.1). Each task is a separate run, so the 95% bootstrap intervals over tasks include run-to-run variation as well as variation across tasks. For the limits and evaluation compute, please see W2 (Third and Finally).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.