- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Co-Design Bench, a benchmark for evaluating coding agents on real-world web application editing tasks. It contains 273 tasks collected from real user coding sessions, covering multimodal editing and repository-level modifications. The authors also propose a judge-then-refute evaluation pipeline that combines source code inspection, browser interaction, and independent LLM verification. The experiments show that even the strongest model achieves only a 26.4% full-pass rate, suggesting that current coding agents still struggle to fully satisfy complex user requirements.
Critical strengths
-
Realistic task setting. The tasks are collected from real user coding sessions rather than purely synthetic instructions, making the benchmark practically relevant.
-
Comprehensive evaluation. The combination of code inspection, browser interaction, and criterion-level verification is useful for evaluating functional web applications.
-
Extensive experiments. The paper evaluates ten model endpoints and provides additional analyses of input modalities, editing complexity, and execution harnesses.
Critical weaknesses
-
Limited novelty compared with existing benchmarks. The task setting overlaps considerably with existing work on feature-level coding and interactive web application evaluation, such as SWE-Lancer, WebCompass, and MiniAppBench. In particular, MiniAppBench already evaluates interactive web applications using browser-based agentic evaluation, covering both static quality and dynamic functionality. While Co-Design Bench focuses on editing existing applications rather than generating them from scratch, it is unclear what fundamentally new challenges this introduces beyond existing benchmarks.
-
The unique challenges of repository editing are not sufficiently demonstrated. One potential distinction is that editing existing applications requires preserving original functionality while implementing new features. However, the experiments mainly report overall completion rates and performance across different task categories. It would be helpful to analyze editing-specific failures, such as unintended regressions, failure to preserve existing behavior, and inconsistent updates across components.
-
The benefit of judge-then-refute is not clearly isolated. The proposed evaluation pipeline is reasonable, but its additional value over existing agent-based browser evaluation methods remains unclear. Although the paper includes evaluator sensitivity experiments and some human verification, a direct ablation comparing the primary judge with and without the refutation stage would make the contribution more convincing.
Questions for authors
-
How does Co-Design Bench differ from MiniAppBench beyond the distinction between application generation and editing?
-
What proportion of task failures are specifically caused by regressions or incorrect propagation of changes, rather than general implementation failures?
-
How much does the independent refutation stage improve evaluation accuracy compared with the primary judge alone?