- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Design Anything, a harness for long-horizon multimodal design agents that preserves design intent through a task contract, a workflow graph, and a deterministic controller for tracking deliverables and preventing premature termination. The authors also introduce DesignAnything-Bench, comprising 228 tasks across 19 design domains, including 38 project families evaluated at scales of 6, 14, and 24 artifacts. Experiments with DeepSeek V4.1 Flash High and GPT-5.6 Sol High show improvements in constraint satisfaction, reference fidelity, and cross-artifact consistency over OpenDesign, JarvisHub, and Codex CLI.
Critical strengths
- The research topic is meaningful and practical. It also introduces a comprehensive benchmark comprising 228 tasks across 19 design domains, covering different task scales, reference-material requirements, and diverse design scenarios.
- The work includes a human evaluation with professional designers, providing valuable evidence of the practical quality of the generated artifacts beyond automatic evaluation metrics.
- The paper provides evaluation across multiple dimensions, including constraint satisfaction, artifact quality, and cross-artifact consistency, offering a more holistic assessment of long-horizon multimodal design agents.
Critical weaknesses
- The paper lacks a dedicated related work section.
- The evaluation covers only two language models.
- The methodological novelty of the proposed harness is limited. Its three core components (Task Contract, Workflow Graph, and Delivery Controller) largely build on existing ideas.
- The evaluation of Constraint Satisfaction is limited. According to Section 3.1 and Appendix B.3, it primarily relies on exact text matching of factual constraints, which overlooks visual requirements and more subjective or semantically complex design requirements. Although the paper includes additional metrics for visual quality and consistency, these do not directly assess whether the generated artifacts satisfy the specific requirements of the design brief.
Questions for authors
- How many tasks were run in parallel, and how long did it take to complete the full 228-task benchmark evaluation?
- Although the paper includes a human evaluation, it mainly reports overall preferences. Can you further analyze the correlation between human Fit to Intent judgments and automatic evaluation metrics to assess whether these metrics accurately reflect human perceptions of design requirement satisfaction?
- How are the evaluation specifications constructed from the design briefs? The paper does not clarify this point.
4 comments
Response to Reviewer JVQy (2/2): W4, Q1–Q3
W4. Constraint satisfaction relies on exact text
You are right that exact text matching misses meaning, so we added a check judged by designers. Judged by meaning, DA still meets far more requirements: 94–100%, against 48–67% for the baselines. For each of the 38 briefs at 24 outputs, we wrote about four visual requirements (e.g., "the privacy shutter covers the camera"), each traced to the brief. Two designers, blind to the method, judged 130 cases on delivered outputs and agreed on 93.1% (κ=0.83).
| DA (GPT-5.6 Sol) | DA (DeepSeek) | JarvisHub | OpenDesign | Codex |
|---|---|---|---|---|
| 100% | 94.2% | 67.3% | 55.8% | 48.1% |
Every same-model gap is significant (Fisher exact p ≤ 0.023). Revision: Section 3.3, lines 303–310 (Table 2b, now in the main text); Appendix B.7, lines 1213–1252.
Q1. Parallelism and total time
Each method ran 10 tasks in parallel. All 228 took about 9 hours for DA on DeepSeek and 25 on GPT-5.6 Sol; OpenDesign, JarvisHub and Codex took about 13, 8 and 7. Revision: Appendix C.7, lines 1687–1705 (Table 16).
Q2. Human judgments and automatic measures
The judge's domain score tracks the designers' judgments; constraint satisfaction measures wording only, so the designer check in W4 covers meaning. DA vs Codex, 211 tasks: the domain score has Spearman ρ=0.26 with domain correctness and 0.27 with fit to intent (95% intervals above 0). Constraint satisfaction checks required wording and is nearly unrelated to fit to intent (intervals include 0): wording and intent are separate properties. This agrees with your W4, and it is why we added the designer check. Revision: Appendix C.6, lines 1654–1663; Section 3.1, lines 229–230.
Q3. Building the evaluation specifications
Thank you for asking. Appendix B.1 (lines 1010–1030) now gives the four steps, following one task from brief to checks: (1) an LLM, as a client, writes the brief; (2) a second call, as the evaluator, records each requirement with its sentence and target outputs; (3) hidden scoring rules are compiled from this record; (4) specifications are frozen once every requirement maps to a rule.
Thank you again for the detailed comments. We are glad to clarify anything further during the discussion.
Thank you for the clarifications and additional analysis. I appreciate the authors' responses to my questions. I'd like to raise my final score to 3.
Thank you
Thank you for reading our responses and for raising your score. Your questions improved the paper: the new Related Work section, the GPT-6 Astra comparison, the designer check of requirements judged by meaning, and the description of how specifications are built (Appendix B.1) all came from them. We appreciate the time you spent on our paper.
Response to Reviewer JVQy (1/2): W1–W3
Thank you for the thorough review. We answer in your order: W1–W3 here, W4 and Q1–Q3 in Part 2. In the revised PDF, changes made for you alone are in orange, and passages that also answer other reviewers are in blue; each answer gives the line numbers.
W1. No dedicated related work section
Thank you for pointing this out. The revision adds a Related Work section (Section 4, lines 481–531) and an extended version (Appendix G, lines 2810–2877). They cover agentic generation and design, long-horizon harnesses and benchmarks, and position Design Anything (DA) against them. Table 19 in Appendix G compares DA with six harnesses and frameworks, including Magentic-One, LangGraph and JarvisHub, on where requirements are kept, who decides completion, and who builds the execution graph.
W2. Only two language models
We added GPT-6 Astra, and the gap remains: on 19 paired 24-output tasks, constraint satisfaction is 97.3% for DA vs 72.6% for Codex (p=0.001). Revision: Section 3.3, lines 266–301 (Table 2a, now in the main text); Appendix C.8, lines 1708–1744.
W3. Novelty of the harness
DA solves a problem that existing harnesses leave to the model or to the developer: in long generation tasks, nothing reports a missing output or a dropped requirement, so agents stop early and outputs drift from the brief. DA moves both jobs into the harness: a rule checks completion against the brief, and a graph carries each requirement into every generation call that needs it (Section 2.2).
The two problems. Agents fail more often as tasks get longer [1]: they stop early [2] and lose information [3, 4]. Harness design alone changes agent performance [5], and generation makes both failures harder to catch:
From 6 to 24 outputs, constraint satisfaction drops 21.9 points for JarvisHub and 16.1 for Codex, against at most 5.5 for DA. The components build on known ideas, as you say; what differs is who does each job:
Evidence. On 19 paired 24-output tasks we removed the controller, then the contract. When the agent stops, the benchmark prompts it to continue; such an external prompt marks an early stop.
Neither half is enough alone: Codex with a list and a check still loses requirements, and DA without the controller needs 24 external prompts instead of 2. Long generation tasks need both.
The benchmark is a second contribution, as you note: each of 38 project families is posed at 6, 14 and 24 outputs, which no benchmark we survey does (Appendix G).
Revision: Section 1, lines 62–78 (the two problems); Section 3.4.4, lines 433–461 (removal tests, Table 3b).
References