- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces WebMetaBench, a benchmark for evaluating the reliability of agentic judges in complex web development tasks. Unlike existing evaluation approaches that primarily rely on static screenshots or predefined checks, WebMetaBench requires agents to interact with deployed applications, collect video evidence, and identify requirement-level defects. The benchmark contains 120 single-defect and 19 multi-defect instances constructed from 42 frontend projects. The authors evaluate 10 frontier models and introduce a three-stage failure analysis covering requirement coverage, evidence acquisition, and video reasoning. The results show that current models struggle with this task, with the best model achieving 25.8% accuracy in the single-defect setting.
Critical strengths
- This paper addresses an important challenge in evaluating agent-generated web applications. Existing static evaluation approaches can miss dynamic behaviors, and the proposed interactive evaluation setting is relevant to practical web development.
- WebMetaBench covers six categories of defects involving different types of interactive and dynamic behaviors. The use of real frontend projects, manually verified defects, and acceptance checklists provides a useful foundation for systematic evaluation.
- This paper evaluates 10 models and reports results across different defect categories, including both single-defect and multi-defect settings. The additional analysis of agent behavior, interaction costs, and harness configurations provides useful empirical observations.
Critical weaknesses
- The main contribution is a new benchmark and its empirical evaluation, rather than a new evaluation algorithm or methodological framework. The proposed three-stage failure paradigm is intuitive and closely follows the natural workflow of interactive testing. It remains unclear what fundamentally new insights this decomposition offers beyond a structured error taxonomy.
- Although the benchmark contains 139 instances, they are derived from only 42 frontend projects, and the multi-defect setting includes just 19 instances. Multiple instances originating from the same projects may share similar interaction patterns and implementation characteristics.
- All benchmark instances appear to contain at least one intentionally introduced defect. In practical web testing, evaluators must also distinguish correctly implemented functionality from actual defects. Without defect-free examples, the benchmark provides limited insight into false-alarm behavior and an agent's ability to recognize satisfied requirements.
- The defects are manually designed and implemented with AI assistance, and the filtering stage removes instances that can be solved without video evidence. Although this supports the intended benchmark focus, it may produce a distribution of defects that is substantially different from naturally occurring web development errors. Moreover, filtering with specific models could introduce selection bias.
- The single-defect evaluation uses an LLM judge to compare final reports with annotated defects. While the authors report high agreement with human judgments, the scoring protocol allows up to four additional defect claims while still marking an answer as correct. This relatively permissive criterion may not adequately reflect the reliability required of an automated testing system.
Questions for authors
- How representative are the constructed defects of real-world web development failures?
- How sensitive are the main findings to the scoring protocol and harness configuration?
2 comments
Thank you for reviewing our submission! Following our rebuttal, we have uploaded a revised manuscript with the following additions addressing your main concerns:
- In Section 4.4, we extend the three-stage failure analysis to quantify the limits of individual capability improvements and the potential gains from optimizing different stages, demonstrating its value beyond a simple error taxonomy.
- In Appendix B, we provide further details on human annotation and cross-checking procedures.
- We also completed a defect-free control experiment, reporting results for the GPT-5.6 series in Section 4.2. In Appendix D, we add sensitivity analyses of false-positive tolerance and false-positive statistics across models to provide a more comprehensive assessment of evaluator reliability.
We hope these additions help you better appreciate the contributions of our work. We would greatly appreciate it if you could take a moment to review our responses or the revised manuscript and share any further feedback.
Follow-Up on Our Rebuttal Responses
Dear Reviewer,
Thank you again for your careful review and insightful comments on our paper. We have now provided detailed responses and additional analyses in the rebuttal to address your concerns. If you have a moment to look over our replies and let us know whether there are any remaining points that would benefit from further clarification, we would be very grateful.
Best regards, The Authors