acceptodds
Under review as a conference paper at ICLR 2027

WebMetaBench: Interactive Black-Box Testing as Multimodal Meta-Evaluation for Web Development Agents

Abstract

Web development agents are increasingly capable of end-to-end application development beyond isolated code generation. Their expanding scope makes reliable evaluation critical and challenging, motivating systematic assessment of agentic judges. Existing evaluation relies mainly on structured page information or static screenshots, while many frontend requirements involve dynamic visual behaviors such as particles, animations, and interactive responses. Evaluating such behaviors requires a judge to actively test the application, acquire diagnostic video evidence, and reason over its dynamics. We introduce WebMetaBench, an interactive black-box benchmark for multimodal meta-evaluation of web development agents. It contains 120 single-defect and 19 multi-defect instances, with a gold-video setting that decouples active testing from video understanding and enables fine-grained evaluation of judge reliability. We further propose a three-stage failure paradigm consisting of requirement coverage, evidence acquisition, and video reasoning for fine-grained diagnosis of agent capabilities. Our experiments reveal substantial limitations in current models, with the best model achieving only 25.8% accuracy on single-defect instances and 26.9% defect recall in the multi-defect setting. WebMetaBench exposes critical limitations in both active testing and video understanding, providing a foundation for developing reliable judges for complex web development.

Primary area:
datasets and benchmarks
Keywords:
MLLM, Agent-as-a-Judge, Video Reasoning, Web Development Agents, Meta-Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UdMUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WebMetaBench, a benchmark for evaluating the reliability of agentic judges in complex web development tasks. Unlike existing evaluation approaches that primarily rely on static screenshots or predefined checks, WebMetaBench requires agents to interact with deployed applications, collect video evidence, and identify requirement-level defects. The benchmark contains 120 single-defect and 19 multi-defect instances constructed from 42 frontend projects. The authors evaluate 10 frontier models and introduce a three-stage failure analysis covering requirement coverage, evidence acquisition, and video reasoning. The results show that current models struggle with this task, with the best model achieving 25.8% accuracy in the single-defect setting.

Critical strengths
  1. This paper addresses an important challenge in evaluating agent-generated web applications. Existing static evaluation approaches can miss dynamic behaviors, and the proposed interactive evaluation setting is relevant to practical web development.
  2. WebMetaBench covers six categories of defects involving different types of interactive and dynamic behaviors. The use of real frontend projects, manually verified defects, and acceptance checklists provides a useful foundation for systematic evaluation.
  3. This paper evaluates 10 models and reports results across different defect categories, including both single-defect and multi-defect settings. The additional analysis of agent behavior, interaction costs, and harness configurations provides useful empirical observations.
Critical weaknesses
  1. The main contribution is a new benchmark and its empirical evaluation, rather than a new evaluation algorithm or methodological framework. The proposed three-stage failure paradigm is intuitive and closely follows the natural workflow of interactive testing. It remains unclear what fundamentally new insights this decomposition offers beyond a structured error taxonomy.
  2. Although the benchmark contains 139 instances, they are derived from only 42 frontend projects, and the multi-defect setting includes just 19 instances. Multiple instances originating from the same projects may share similar interaction patterns and implementation characteristics.
  3. All benchmark instances appear to contain at least one intentionally introduced defect. In practical web testing, evaluators must also distinguish correctly implemented functionality from actual defects. Without defect-free examples, the benchmark provides limited insight into false-alarm behavior and an agent's ability to recognize satisfied requirements.
  4. The defects are manually designed and implemented with AI assistance, and the filtering stage removes instances that can be solved without video evidence. Although this supports the intended benchmark focus, it may produce a distribution of defects that is substantially different from naturally occurring web development errors. Moreover, filtering with specific models could introduce selection bias.
  5. The single-defect evaluation uses an LLM judge to compare final reports with annotated defects. While the authors report high agreement with human judgments, the scoring protocol allows up to four additional defect claims while still marking an answer as correct. This relatively permissive criterion may not adequately reflect the reliability required of an automated testing system.
Questions for authors
  1. How representative are the constructed defects of real-world web development failures?
  2. How sensitive are the main findings to the scoring protocol and harness configuration?
2 comments
AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Responses

Dear Reviewer,

Thank you again for your careful review and insightful comments on our paper. We have now provided detailed responses and additional analyses in the rebuttal to address your concerns. If you have a moment to look over our replies and let us know whether there are any remaining points that would benefit from further clarification, we would be very grateful.

Best regards, The Authors

Thank you for reviewing our submission! Following our rebuttal, we have uploaded a revised manuscript with the following additions addressing your main concerns:

  1. In Section 4.4, we extend the three-stage failure analysis to quantify the limits of individual capability improvements and the potential gains from optimizing different stages, demonstrating its value beyond a simple error taxonomy.
  2. In Appendix B, we provide further details on human annotation and cross-checking procedures.
  3. We also completed a defect-free control experiment, reporting results for the GPT-5.6 series in Section 4.2. In Appendix D, we add sensitivity analyses of false-positive tolerance and false-positive statistics across models to provide a more comprehensive assessment of evaluator reliability.

We hope these additions help you better appreciate the contributions of our work. We would greatly appreciate it if you could take a moment to review our responses or the revised manuscript and share any further feedback.

Reviewer_KFcDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose WebMetaBench, an interactive black-box benchmark for evaluating agent's judges in web development. The benchmark contains 120 single-defect and 19 multi-defect instances derived from 42 projects. Its core contributions includes: controlled dynamic defects, a gold-video evaluation setting, and a diagnostic decomposition into requirement coverage, evidence acquisition, and video reasoning.

Critical strengths
  1. The gold-video setting and three-stage failure framework enable precise failure diagnosis and provide actionable insights for agent design and training.
  2. Manual verification of defects and high agreement with human judgments (Appendix D.4) make the evaluation more convincing.
  3. The paper is clearly presented and easy to follow.
Critical weaknesses
  1. The dataset size is modest (139 total instances and 42 real-world projects) and limited diversity (more than 50% of website application domains from (Creative; Business & Marketing)). This raise a concern about the breadth of claims about judge reliability?
  2. The model comparison is partly confounded by harness differences. Section 4.5 acknowledges impacts of harness design. Table 4 shows Kimi increasing from 1.7% with Claude Code to 12.5% with enhanced Codex, and Qwen increasing from 1.7% to 10.0%. These changes are substantial relative to the reported performance differences. How stable are results across harnesses?
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Responses

Dear Reviewer,

Thank you again for your careful review and insightful comments on our paper. We have now provided detailed responses and additional analyses in the rebuttal to address your concerns. If you have a moment to look over our replies and let us know whether there are any remaining points that would benefit from further clarification, we would be very grateful.

Best regards, The Authors

Thank you for reviewing our submission! Following our rebuttal, we have uploaded a revised manuscript with the following additions addressing your main concerns:

  1. In Appendix D.6, we report results with equal weighting across projects.
  2. In Appendix D.6, we provide domain distribution and exclusion analyses. The two domains with the largest shares of projects account for a smaller proportion of instances, resulting in a more balanced dataset distribution. Moreover, the main results and conclusions remain consistent even after completely excluding these two domains.

We hope these additions address your concerns. We would greatly appreciate it if you could take a moment to review our rebuttal or the revised manuscript and share any further feedback.

Reviewer_KrbpOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WebMetaBench, a black-box benchmark for multimodal meta-evaluation of web development agents, containing 120 single-defect and 19 multi-defect instances from 42 projects, focusing on dynamic behaviors. Agents receive an application and an acceptance checklist, and are asked to identify requirement violations and report evidence, such as videos, by interacting with the application. The authors argue that current evaluation methods rely on static screenshots and programmatic checks, lacking in evaluating dynamic behaviors. This paper investigates how agents perform as evaluators in this setting. The authors also introduce a gold video ablation that provides diagnostic videos containing the target defects, to evaluate whether agents can identify failures when useful evidence is already provided. The results show limitations in both active testing and video understanding. WebMetaBench acts as a starting point for studying agent-as-a-judge reliability in dynamic frontend evaluation.

Critical strengths
  1. The paper is overall clear. The benchmark introduced is a good contribution for automation of WebDev testing through autonomous agents, especially for dynamic behaviors that are difficult to assess through static observations.
  2. The benchmark curation is well described and adequate, with projects taken from real-world applications and deliberately introduced defects. This provides controlled ground truth, and the authors manually verify that the defects are reproducible and consistent with the requirements. Checklist construction is assisted by GPT-5.6 Sol, and gold video evidence is also collected. There are a few concerns, described in the section below.
  3. The evaluation includes different models and useful error analysis. The final scoring is done with Qwen, but the authors provide evidence in the appendix about agreement with human judgments and human verification of the main results. The harness analysis, as well as the gold video analysis, also provides interesting insights that are useful for both benchmark creation and also for training models.
Critical weaknesses
  1. Three major concerns on the benchmark creation process:
  • all instances contain defects. Adding error-free samples would strengthen the benchmark (I understand it would add work, since the clean applications would come from GitHub, they could not be used as is, because of potential data leakage).
  • The paper provides little explanation of the annotators, and there is no cross-check review among humans to confirm the naturalness of the introduced bugs. Naturalness is important, and more detail on the annotation process is needed.
  • GPT-5.6 Sol is involved in defect and checklist construction and in filtering, while also being one of the evaluated models. This could introduce model-specific bias in the construction or selection of instances. It would require discussion from the authors.
  1. As someone not directly working on web development evaluation, I struggle to understand how difficult these tasks are compared to human performance. A human comparison would help contextualize the low agent performance.
  2. For the single-defect setting, the main paper does not clearly explain how false positives affect accuracy. This is described in more detail only in the appendix, and it should be clearly explained in the main text.
Questions for authors

Suggestions for future work

It took me a while to understand the benchmark contains only samples related to dynamic behaviors. It became clear only when it was described that defects that do not require a video to be detected are dropped: make this more clear in the abstract/appending

Questions for the authors:

Regarding weakness #2: How do humans perform on this failure-detection task, given the same checklist and comparable tools and time?

1 comment
AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Responses

Dear Reviewer,

Thank you again for your careful review and insightful comments on our paper. We have now provided detailed responses and additional analyses in the rebuttal to address your concerns. If you have a moment to look over our replies and let us know whether there are any remaining points that would benefit from further clarification, we would be very grateful.

Best regards, The Authors

Comments on the paper

1 comment

Dear reviewers, we are approaching the end of the author-reviewer discussion period. We have attempted to address all the concerns pointed out in your review. If there is any additional information we can provide before the deadline, please let us know!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.