acceptodds
Withdrawn from ICLR 2027

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating human–AI co-design requires testing whether coding agents can translate real users' multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Its two tracks, Multimodal Editing and Repository Editing, cover localized edits with optional image, audio, video, or document inputs and coordinated repository revisions. Tasks span three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The tasks pair normalized requests and runnable pre-edit applications with 2,331 hidden expert-authored acceptance criteria. Our judge-then-refute protocol grades criteria using source and browser evidence, challenges positive verdicts with a separate LLM verifier, and audits visual and interaction claims without gold code or target screenshots. Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.

Primary area:
datasets and benchmarks
Keywords:
coding agents, multimodal evaluation, human-AI collaboration, code editing, web development, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Bv7DOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Co-Design Bench, a benchmark for evaluating coding agents on real-world web application editing tasks. It contains 273 tasks collected from real user coding sessions, covering multimodal editing and repository-level modifications. The authors also propose a judge-then-refute evaluation pipeline that combines source code inspection, browser interaction, and independent LLM verification. The experiments show that even the strongest model achieves only a 26.4% full-pass rate, suggesting that current coding agents still struggle to fully satisfy complex user requirements.

Critical strengths
  1. Realistic task setting. The tasks are collected from real user coding sessions rather than purely synthetic instructions, making the benchmark practically relevant.

  2. Comprehensive evaluation. The combination of code inspection, browser interaction, and criterion-level verification is useful for evaluating functional web applications.

  3. Extensive experiments. The paper evaluates ten model endpoints and provides additional analyses of input modalities, editing complexity, and execution harnesses.

Critical weaknesses
  1. Limited novelty compared with existing benchmarks. The task setting overlaps considerably with existing work on feature-level coding and interactive web application evaluation, such as SWE-Lancer, WebCompass, and MiniAppBench. In particular, MiniAppBench already evaluates interactive web applications using browser-based agentic evaluation, covering both static quality and dynamic functionality. While Co-Design Bench focuses on editing existing applications rather than generating them from scratch, it is unclear what fundamentally new challenges this introduces beyond existing benchmarks.

  2. The unique challenges of repository editing are not sufficiently demonstrated. One potential distinction is that editing existing applications requires preserving original functionality while implementing new features. However, the experiments mainly report overall completion rates and performance across different task categories. It would be helpful to analyze editing-specific failures, such as unintended regressions, failure to preserve existing behavior, and inconsistent updates across components.

  3. The benefit of judge-then-refute is not clearly isolated. The proposed evaluation pipeline is reasonable, but its additional value over existing agent-based browser evaluation methods remains unclear. Although the paper includes evaluator sensitivity experiments and some human verification, a direct ablation comparing the primary judge with and without the refutation stage would make the contribution more convincing.

Questions for authors
  1. How does Co-Design Bench differ from MiniAppBench beyond the distinction between application generation and editing?

  2. What proportion of task failures are specifically caused by regressions or incorrect propagation of changes, rather than general implementation failures?

  3. How much does the independent refutation stage improve evaluation accuracy compared with the primary judge alone?

Reviewer_cQsoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Co-Design Bench, comprising 273 web-application revision tasks curated from real user coding sessions, with Multimodal Editing (ME) and Repository Editing (RE) tracks and 2,331 hidden acceptance criteria. Evaluation combines source and browser evidence with an LLM judge and a refutation pass, reporting strict completion, graded completion, and full-task success. Ten model endpoints are evaluated under a shared Claude Code harness; the strongest achieves 69.9% mean strict completion and a 26.4% full-pass rate.

Critical strengths
  1. The task setting is practically relevant. Revising existing applications based on user needs, including consistent views and proper interactions, is more useful for maintenance than just reviewing a final screenshot.
  2. The evaluation objectives and empirical coverage are useful. Requiring browser evidence for visual and interaction claims, and distinguishing criterion completion from complete delivery, are sensible design choices The ten model endpoints and task breakdowns provide informative descriptive results.
Critical weaknesses

The reliability and auditability of the central scores are insufficiently established. Section 4.1 and Appendix J mention that two frontend experts checked 200 task-model outputs, but report no results on human-evaluator agreement or error analysis. Because the refutation pass only challenges positive verdicts, it does not address potentially incorrect FAIL judgments. Additionally, I could not locate a benchmark access link. For a benchmark relying on LLM judgments, these gaps prevent independent verification of its scores and full-pass rates. The cross-harness results are not clearly reconciled with the main performance conclusions. Table 12 reports ME completion of 0.828-0.849 for kimi-k3, whereas Table 2 reports 0.662 strict/ 0.718 graded under Claude Code, however, Table 12 does not specify whether completion refers to strict or graded completion. Harness-dependent performance is plausible, but the comparability of these results and their implications for the reported headroom are unclear. Moreover, Section 5.4 attributes a decisive role to closed-loop verification, while Appendix H explicitly states that changing entire harnesses cannot isolate individual tool practices. These limits conclusions about model capabilities and remaining headroom. The analysis does not sufficiently separate coordination difficulty from requirement count. The gap between criterion completion and full-pass rate is informative, but full-pass success also depends on the number and granularity of acceptance criteria. ME averages 5.5 criteria per task, while RE averages 10.3 and reaches 46. Equal task weighting does not remove this dependence. Controlling for criterion count and examining failures involving dependent updates would help establish how much incomplete delivery reflects the coordination challenges emphasized by the benchmark.

Questions for authors

What were the findings of the 200-output human audit? Please identify the judge and verifier configurations and report criterion-level agreement and errors, including false FAIL judgments and the correctness of refutation-induced changes. Please provide a directly comparable Claude Code baseline and clarify whether the task set, execution budgets, and scoring pipeline match the main experiment. Are anonymized task packages and evaluation artifacts available, including complete example rubrics and evaluator prompts? Inconsistencies in the reported results raise concerns about their accuracy and reliability. Appendix B.2 identifies document tasks as having the highest Graded Completion, whereas Table 6(a) shows image tasks scoring higher (0.731 vs. 0.682). In Table 7, two reported gains (+0.018 and +0.013) also differ from the displayed score differences (+0.019 and +0.014).

Reviewer_YpBvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Co-Design Bench, a benchmark curated from real-user coding agent sessions involving web application editing, covering both localized multimodal edits and repository-level revisions. The authors evaluate a comprehensive suite of frontier models, all paired with the Claude Code harness. Their evaluation reports both Strict Completion, which measures the average proportion of fully satisfied criteria, and Full-Pass Rate, which requires satisfying all criteria within a task. They observe a substantial gap between these metrics, highlighting complete requirement satisfaction as an important challenge in web development tasks, and provide detailed performance breakdowns across different capabilities.

Critical strengths
  1. The benchmark is curated from real-user sessions and covers a wider range of input modalities than prior work, which is a valuable strength.
  2. The work provides a comprehensive evaluation across frontier models, along with helpful additional robustness analyses, such as evaluator robustness studies.
Critical weaknesses
  1. The gap between Strict Completion and Full-Pass Rate, presented as one of the main findings of the paper, is somewhat superficial and has been observed in prior work. I would expect more in-depth analysis of its underlying causes. For example, is the gap mainly driven by differences in difficulty across individual criteria, or does satisfying a larger number of requirements introduce additional challenges for the models? Under what types of tasks or scenarios does this gap become more pronounced? More qualitative examples of failure cases would also be helpful.
  2. I think the benchmark could support more in-depth analyses and task-specific insights. For example, which aspects of web coding tasks remain particularly challenging for frontier coding agents? Do additional modalities, such as audio, introduce distinct challenges? Which capabilities are particularly challenging for open models to catch up on? Do real-user sessions reveal behaviors or capability gaps that prior benchmarks relying largely on synthetic data fail to capture?
  3. The paper lacks sufficient clarity regarding the collection and curation of user sessions. In particular, how were the original sessions collected, how exactly were user requests rewritten, and to what extent does the resulting benchmark remain faithful to and representative of the original real-user data?
  4. Evaluating all models under the Claude Code harness may be suboptimal, given that many models have their own native harnesses. It would be valuable to see evaluations with each model's native harness, such as GPT with Codex.
Questions for authors

As discussed in Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.