- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WebMockGym, a benchmark of 50 websites and 50 web apps in which coding agents explore a frozen reference site and build a separately runnable reconstruction of its entry page and the pages one link away. Reconstructions are scored with about 500K hidden tests derived from the reference, covering content, images, visual appearance and navigation, and the evaluator is validated with expert rankings and controlled defects. Across six coding-agent configurations, the best Web AUC is 41.7, no Web reconstruction passes every test, and navigation is the weakest test family for most systems. Trajectory analysis associates earlier file writes and more frequent inspection followed by further writing with higher Web scores.
Critical strengths
- The paper is clearly written and easy to follow, with concrete examples of the tests and short takeaway boxes that summarize the main results.
- The setting differs from prior web-generation benchmarks, which supply screenshots, videos or text requirements. Agents must gather the reference information themselves by browsing, including links hidden behind menus, and are scored with fixed hidden tests that do not rely on LLM judges.
- The experiments cover six coding-agent configurations built on recent models from several providers, and report per-task cost, time and token usage.
Critical weaknesses
- Benchmark data, tests and scoring code are not released
- The repository contains only test-generation and test-freezing code. Its README states that no captured pages, page configurations, network recordings, generated tests or run reports are included, and its documentation describes the included fixture as "not a complete candidate-scoring runner".
- The 100 tasks, the roughly 500K tests, the agent trajectories, the scores and the human rankings are therefore not available in the released materials, and none of the reported numbers can be reproduced or checked. The paper only states that the reference captures will be released later.
- The stated scoring condition is inconsistent with the released code
- Section 3.1 states that reconstructions are scored "with JavaScript enabled". The released generator code, which the README describes as extracted from the test pipeline, repeatedly describes the scorer as JavaScript-disabled, for example "the frozen, JavaScript-disabled scorer" and "the scorer renders JS-off". Reference observations are also extracted with JavaScript disabled, with comments stating that this must match the scoring runtime.
- If scoring was JavaScript-disabled, reconstructions whose menus or content rely on JavaScript would have been penalized, although the agent instruction (Appendix A.1) never mentions this. If scoring was JavaScript-enabled, the reference observations and the reconstructions were rendered under different conditions, contrary to the code's own requirement that reference extraction match the scoring runtime. Either way the meaning of the scores, especially on App, is unclear, and the released materials cannot resolve it.
- Weak alignment between the motivation and what is measured
- The paper motivates the benchmark as evaluating autonomous software development, which it describes as obtaining the needed information, deciding how to implement, checking the result and "delivering a working implementation". The benchmark mainly measures how closely the visible content, appearance and links of an observable website are reproduced. The guidance study (Table 5) supports the information-gathering part, but beyond noting that the reference provides an observable target, the paper does not discuss how well website reconstruction represents autonomous development as a whole.
- Functionality is explicitly excluded from scoring (Section 4, Appendix D.2), although the task selection, the agent instruction and the rating instructions all refer to it. The App tasks are selected "to cover different functions and interaction structures" (Section 2.2), the agent instruction asks agents to preserve the "interactions you judge important" (Appendix A.1), and the rating interface tells raters to "Compare menus, links, forms, and core functionality" (Figure 8). Whether a reconstructed calculator or form actually works is therefore not evaluated by any reported test family.
- No justification for the test thresholds
- The pass conditions determine every score, but the paper gives no rationale, reference or sensitivity analysis for the chosen values. Examples include an image hash distance of at most 8 of 64 bits, a position tolerance of 128 by 80 pixels, a 16×12 layout grid with a per-band similarity of 0.78, color-share bands whose rule is not stated, a screenshot tolerance whose value is not reported, and a 1.5 to 5 second check timeout (Appendices A.3, A.4, B.1). Section 2.3 mentions calibrating the suite on the saved reference but does not explain what this involves or how the values were set.
- The controlled studies in Appendix C.1 test responses only at the chosen values. The released code notes that the image threshold was "tightened from 12", but the paper does not say how or when the thresholds were chosen.
- Limited human evaluation
- The human study ranks only the six agent reconstructions within each task, so it supports relative ordering but not the absolute meaning of the scores. The reference is shown but not ranked, and no blank or wget outputs are included as anchors.
- The rating criteria differ from the automated tests. The rating interface in Figure 8 tells raters to compare forms and core functionality, which the tests do not cover, so it is unclear what the reported agreement (Kendall's τb of 0.705) validates.
- Beyond describing them as three PhD participants, the paper gives no information on the raters' expertise, recruitment or compensation.
Questions for authors
- Five configurations run in Claude Code and only GPT-5.6-sol runs in Codex, yet every table and figure that reports per-configuration results labels the configurations by model name, and Appendix B.7 attributes behavior to "GPT". Shouldn't the results be attributed to agent systems (harness plus model) rather than to models?
- Shouldn't the same model be run in both harnesses, or all models in one harness, to separate harness effects from model effects?
Response to Reviewer Qwes
Thank you for the constructive feedback. We address the weaknesses and questions you raised as follows.
W1. Benchmark data, tests and scoring code are not released.
Response: With the camera-ready version, we will release the scoring runner, the tests, and the outcome of every test for all 600 reconstructions, with scripts that recompute every table.
W2. The stated scoring condition is inconsistent with the released code.
Response: We thank the reviewer for pointing out this inconsistency. Reconstructions were scored with JavaScript enabled, as Sec. 3.1 states, while the test generator records the expected values from the reference with JavaScript off. To check whether this mismatch affects the results, we first re-ran every test on the saved reference with JavaScript enabled and kept only the tests that the reference passes and a blank page fails, so that the reference receives a full score under the scoring condition.
We then rescored all six systems with these tests on the entire benchmark, and the comparison remains essentially unchanged. On Web the six systems keep the same order (Opus 5 53.1, GLM 43.2, DeepSeek 43.0, Kimi 31.4, GPT 22.6, Sonnet 5 16.2), and 12 of the 15 pairwise differences are significant, against 10 on all tests. On App no pair differs significantly under either scoring.
W3. Weak alignment between the motivation and what is measured.
Response: The benchmark measures autonomous development ability through website reconstruction, which is one form of autonomous development. We do not claim that it represents autonomous development as a whole, and the revised version will state this.
For functionality, our goal is a programmatic verifier, so every test takes its expected value from the frozen reference and runs automatically. On Web, the tasks are mostly content sites whose function is what they display and where their links lead, so these tests cover what the sites do.
On App, the selected sites expose their functions largely through navigation, and the tests replay the recorded hover and click actions before following each link (Sec. 2.3). A reconstruction that looks like the reference and navigates correctly thus reproduces most of its functionality. Responses to new input are not scored, and the revised version will state this boundary. On the rating instruction in Figure 8, please see W5.
W4. No justification for the test thresholds.
Response: We agree that the specific threshold values are design choices. What matters is whether the conclusions depend on them, so we re-scored the systems under alternative values. Scores shift by a few points, while the order of the systems stays largely stable. Relaxing the image distance from 8 to 12 raises the Image pass rate on Web by 1.7–2.4 points for every system without changing their order. On 20 randomly drawn tasks (10 Web, 10 App), doubling every timeout changes AUC by at most 3.5 points without changing the Web order, and halving or doubling the position tolerance on the App tasks changes AUC by at most 2.7 points without changing the App order.
The revised version will report all values and these analyses, including the two not stated in the paper: each color covering at least 0.5% of the viewport must keep between half and twice its reference share, and a screenshot check passes when fewer than 8% of pixels differ. The image distance was tightened from 12 to 8 so that a different image cannot pass as a match.
W5. Limited human evaluation.
Response: The human study is designed to check whether the automated scores order reconstructions as people do, since the benchmark is used to compare systems. Raters ranked the six reconstructions of each task against the shown reference. The automated ranking agrees with theirs at Kendall's τb of 0.705. The absolute scale is anchored on the automated side instead. A blank page scores near zero on both tracks (Figure 3a), and the wget mirror lies between the blank page and the agents (Table 4).
The raters judged overall fidelity to the reference. Their criteria also cover aspects that the tests do not check, such as forms. The agreement therefore shows that the automated ranking matches people's overall judgment even under broader criteria. It does not validate each test family on its own. What each family measures is checked separately by the controlled defects in Sec. 2.5. Removing images lowers the Image score by 33.0 points, and redirecting links lowers the Navigation score by 33.3 points.
The three raters are PhD students experienced in web development. The revised version will report these details.