- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes WebCraftBench, an automated benchmark for evaluating LLM-generated web applications. The main idea is to use a testing agent to explore generated applications, build a state-transition graph from the interactions, and evaluate aesthetics, usability, and requirements with the help of code-coverage guidance.
Critical strengths
-
Evaluating generated web applications is an important and increasingly relevant problem.
-
Using interactions (instead of static screenshots or code) is reasonable and desirable.
-
The benchmark is evaluated on many models and shows relatively high agreement with human preferences (due to subjectiveness for some of these requirements, it's understandable that we do not have perfect agreement).
Critical weaknesses
-
The motivation and framing are difficult to follow. The abstract and introduction have too many ideas. E.g. human evaluation, static benchmarks, interactive benchmarks, and testing agents are all mentioned before what problem WebCraftBench is solving and what is new.
-
The novelty should be clarified and sharpened. LLM-based GUI testing agents and state-graph creation/exploration have already been studied extensively (e.g. in mobile application testing [1][2]). I believe framing this paper better is an important step that would be needed.
-
It is unclear how much the proposed method improves evaluation itself. There are no comparisons against existing automated evaluation methods on the same human preference data.
-
Similar to the previous point, it would be useful to show whether WebCraftBench actually changes conclusions about model performance. For example, do existing benchmarks overestimate functionality because they fail to explore applications sufficiently?
[1] Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing
[2] Towards Comprehensive Mobile Application Testing for User-centric Feature Coverage via Simulating Real-World Users with Personas
Questions for authors
-
What is the human-preference agreement of existing web-generation evaluation methods on the same 197 pairs?
-
How much do model scores/rankings change compared with existing evaluation approaches?
-
Can the authors better clarify the novelty compared to prior LLM-based GUI testing and state-graph exploration methods?
2 comments
I thank the authors for their response. The WebCoderBench comparison is interesting and it helps answer my question about agreement with human preferences and changes in model rankings. However, while the higher agreement is useful, I am still unsure what coverage guidance adds. For example, does it help the testing agent find bugs that the same agent would otherwise miss? I do not think this important analysis is clear.
Additionally, a big concern still remains for the framing and presentation. The abstract and introduction bring up several ideas before it is clear what problem the paper is trying to solve. The authors should explain more clearly what existing evaluators lack, and I believe a discussion for how this work relates to the mobile GUI testing is warranted.
Given these remaining concerns, I will keep my score.
Dear Reviewer, thank you again for your time and constructive feedback. We would appreciate hearing whether our responses have addressed your concerns. If any concerns remain or any points require further clarification, we would be happy to elaborate and continue the discussion. Thank you for your consideration!