acceptodds
Under review as a conference paper at ICLR 2027

Design Anything, Design at Scale

Abstract

As multimodal resources are transformed into media outputs, agentic AI is expected to align with design priors. Recent work dynamically reuses prior design experience in long-horizon agentic design, yet struggles to keep to the brief over extended workflows. In contrast, designers typically work from an initial brief that specifies project goals, constraints, and expected deliverables, which provides persistent guidance throughout the design process. This highlights the need for long-horizon agents that can preserve and consistently apply design intent throughout the entire workflow. To address this, we propose Design Anything, a harness that connects persistent requirements and design decisions with an executable workflow and delivery control. We further introduce Design Anything-Bench, a benchmark of 228 tasks across 19 design domains, including 38 project families each posed at 6, 14, and 24 artifacts with shared context and inputs. On this bench, Design Anything satisfies 85.0% of the brief's constraints with DeepSeek V4.1 Flash High and 92.5% with GPT-5.6 Sol High, 15.4 and 14.4 percentage points above the strongest baseline using the same model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Horizon Agent, Agentic AI, Agentic Design, Harness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JVQyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Design Anything, a harness for long-horizon multimodal design agents that preserves design intent through a task contract, a workflow graph, and a deterministic controller for tracking deliverables and preventing premature termination. The authors also introduce DesignAnything-Bench, comprising 228 tasks across 19 design domains, including 38 project families evaluated at scales of 6, 14, and 24 artifacts. Experiments with DeepSeek V4.1 Flash High and GPT-5.6 Sol High show improvements in constraint satisfaction, reference fidelity, and cross-artifact consistency over OpenDesign, JarvisHub, and Codex CLI.

Critical strengths
  1. The research topic is meaningful and practical. It also introduces a comprehensive benchmark comprising 228 tasks across 19 design domains, covering different task scales, reference-material requirements, and diverse design scenarios.
  2. The work includes a human evaluation with professional designers, providing valuable evidence of the practical quality of the generated artifacts beyond automatic evaluation metrics.
  3. The paper provides evaluation across multiple dimensions, including constraint satisfaction, artifact quality, and cross-artifact consistency, offering a more holistic assessment of long-horizon multimodal design agents.
Critical weaknesses
  1. The paper lacks a dedicated related work section.
  2. The evaluation covers only two language models.
  3. The methodological novelty of the proposed harness is limited. Its three core components (Task Contract, Workflow Graph, and Delivery Controller) largely build on existing ideas.
  4. The evaluation of Constraint Satisfaction is limited. According to Section 3.1 and Appendix B.3, it primarily relies on exact text matching of factual constraints, which overlooks visual requirements and more subjective or semantically complex design requirements. Although the paper includes additional metrics for visual quality and consistency, these do not directly assess whether the generated artifacts satisfy the specific requirements of the design brief.
Questions for authors
  1. How many tasks were run in parallel, and how long did it take to complete the full 228-task benchmark evaluation?
  2. Although the paper includes a human evaluation, it mainly reports overall preferences. Can you further analyze the correlation between human Fit to Intent judgments and automatic evaluation metrics to assess whether these metrics accurately reflect human perceptions of design requirement satisfaction?
  3. How are the evaluation specifications constructed from the design briefs? The paper does not clarify this point.
4 comments
AuthorsOpenReview ↗

Response to Reviewer JVQy (1/2): W1–W3

Thank you for the thorough review. We answer in your order: W1–W3 here, W4 and Q1–Q3 in Part 2. In the revised PDF, changes made for you alone are in orange, and passages that also answer other reviewers are in blue; each answer gives the line numbers.

W1. No dedicated related work section

Thank you for pointing this out. The revision adds a Related Work section (Section 4, lines 481–531) and an extended version (Appendix G, lines 2810–2877). They cover agentic generation and design, long-horizon harnesses and benchmarks, and position Design Anything (DA) against them. Table 19 in Appendix G compares DA with six harnesses and frameworks, including Magentic-One, LangGraph and JarvisHub, on where requirements are kept, who decides completion, and who builds the execution graph.

W2. Only two language models

We added GPT-6 Astra, and the gap remains: on 19 paired 24-output tasks, constraint satisfaction is 97.3% for DA vs 72.6% for Codex (p=0.001). Revision: Section 3.3, lines 266–301 (Table 2a, now in the main text); Appendix C.8, lines 1708–1744.

W3. Novelty of the harness

DA solves a problem that existing harnesses leave to the model or to the developer: in long generation tasks, nothing reports a missing output or a dropped requirement, so agents stop early and outputs drift from the brief. DA moves both jobs into the harness: a rule checks completion against the brief, and a graph carries each requirement into every generation call that needs it (Section 2.2).

The two problems. Agents fail more often as tasks get longer [1]: they stop early [2] and lose information [3, 4]. Harness design alone changes agent performance [5], and generation makes both failures harder to catch:

  1. Stopping early. A missing output raises no error, so the model alone decides when the task is done, and it stops too soon.
  2. Losing information. A generation model has no memory of the task: it sees only the prompt and inputs of one call. If the brief fixes a product name and the agent chooses a logo early on, it must repeat both in every later call. After one omission, that output no longer matches the rest, again without an error.

From 6 to 24 outputs, constraint satisfaction drops 21.9 points for JarvisHub and 16.1 for Codex, against at most 5.5 for DA. The components build on known ideas, as you say; what differs is who does each job:

DimensionExisting harnessesDA
Requirement trackingThe model keeps a plan and judges its progress (Magentic-One [6])Task contract: the agent records every output the brief requires when it plans. It holds no progress, so the model cannot mark one done.
Completion controlThe model (Magentic-One) or developer code (LangGraph)Controller: after each turn, a rule checks that every listed output was produced, finished, and of the right kind and count.
Information passingThe agent repeats requirements and choices in each call (Codex; JarvisHub [7])Graph: the agent connects a source once. Every later call receives it as input.

Evidence. On 19 paired 24-output tasks we removed the controller, then the contract. When the agent stops, the benchmark prompts it to continue; such an external prompt marks an early stop.

  • Controller: without it, external prompts rise from 2 to 24, and three runs end with outputs missing.
  • Contract: without it as well, external prompts rise to 47; the agent no longer plans the whole task up front.
  • Graph: a list in the prompt does not replace it. On GPT-6 Astra, Codex given the full requirement list and a completion check delivers 99.8% of outputs, but satisfies 75.3% of constraints, against 97.3% for DA (p=0.027).

Neither half is enough alone: Codex with a list and a check still loses requirements, and DA without the controller needs 24 external prompts instead of 2. Long generation tasks need both.

The benchmark is a second contribution, as you note: each of 38 project families is posed at 6, 14 and 24 outputs, which no benchmark we survey does (Appendix G).

Revision: Section 1, lines 62–78 (the two problems); Section 3.4.4, lines 433–461 (removal tests, Table 3b).

References

  • [1] Measuring AI Ability to Complete Long Software Tasks, NeurIPS 2025
  • [2] Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces, ICLR 2026
  • [3] Lost in the Middle: How Language Models Use Long Contexts, TACL 2024
  • [4] LLMs Get Lost in Multi-Turn Conversation, ICLR 2026
  • [5] SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024
  • [6] Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks, arXiv 2024
  • [7] JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents, arXiv 2026
AuthorsOpenReview ↗

Response to Reviewer JVQy (2/2): W4, Q1–Q3

W4. Constraint satisfaction relies on exact text

You are right that exact text matching misses meaning, so we added a check judged by designers. Judged by meaning, DA still meets far more requirements: 94–100%, against 48–67% for the baselines. For each of the 38 briefs at 24 outputs, we wrote about four visual requirements (e.g., "the privacy shutter covers the camera"), each traced to the brief. Two designers, blind to the method, judged 130 cases on delivered outputs and agreed on 93.1% (κ=0.83).

DA (GPT-5.6 Sol)DA (DeepSeek)JarvisHubOpenDesignCodex
100%94.2%67.3%55.8%48.1%

Every same-model gap is significant (Fisher exact p ≤ 0.023). Revision: Section 3.3, lines 303–310 (Table 2b, now in the main text); Appendix B.7, lines 1213–1252.

Q1. Parallelism and total time

Each method ran 10 tasks in parallel. All 228 took about 9 hours for DA on DeepSeek and 25 on GPT-5.6 Sol; OpenDesign, JarvisHub and Codex took about 13, 8 and 7. Revision: Appendix C.7, lines 1687–1705 (Table 16).

Q2. Human judgments and automatic measures

The judge's domain score tracks the designers' judgments; constraint satisfaction measures wording only, so the designer check in W4 covers meaning. DA vs Codex, 211 tasks: the domain score has Spearman ρ=0.26 with domain correctness and 0.27 with fit to intent (95% intervals above 0). Constraint satisfaction checks required wording and is nearly unrelated to fit to intent (intervals include 0): wording and intent are separate properties. This agrees with your W4, and it is why we added the designer check. Revision: Appendix C.6, lines 1654–1663; Section 3.1, lines 229–230.

Q3. Building the evaluation specifications

Thank you for asking. Appendix B.1 (lines 1010–1030) now gives the four steps, following one task from brief to checks: (1) an LLM, as a client, writes the brief; (2) a second call, as the evaluator, records each requirement with its sentence and target outputs; (3) hidden scoring rules are compiled from this record; (4) specifications are frozen once every requirement maps to a rule.

Thank you again for the detailed comments. We are glad to clarify anything further during the discussion.

Thank you for the clarifications and additional analysis. I appreciate the authors' responses to my questions. I'd like to raise my final score to 3.

Thank you for reading our responses and for raising your score. Your questions improved the paper: the new Related Work section, the GPT-6 Astra comparison, the designer check of requirements judged by meaning, and the description of how specifications are built (Appendix B.1) all came from them. We appreciate the time you spent on our paper.

Reviewer_VhqJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Design Anything which is an agentic framework for managing complex design projects that require multiple coordinated artifacts. The main idea is to maintain design requirements, decisions and deliverables via a persistent task contract, a workflow graph and a deterministic controller that tracks completion. Also introduced is a benchmark named DesignAnything Bench, containing 228 tasks across 19 design domains.

Critical strengths
  • I think it is an interesting benchmark for evaluating long horizon design consistency

  • Strong experimental results are displayed on complex design tasks

Critical weaknesses
  • The necessity of the proposed harness is a bit unclear in the presence of stronger autonomous agents. The proposed framework relies on explicit task contracts, workflow graphs and deterministic completion control to address failures in long horizon design tasks. However, as frontier agents become increasingly capable of planning, maintaining task context, verifying outputs and continuing execution autonomously (e.g., GPT 6 Astra), it is unclear whether the additional complexity of this specialized harness remains necessary. Although the paper evaluates strong language models and includes continuation ablations, it does not compare against a sufficiently strong lightweight orchestration baseline that provides explicit requirements and completion criteria without the full proposed architecture. As a consequence, the experiments do not establish whether the observed improvements reflect a fundamental advantage of the proposed design or compensate for limitations that more capable agents can already overcome.

  • I could not find interrater agreement analysis in the human evaluation with 9 designers. Even though the paper conducts a human evaluation involving 9 professional designers, it does not report interrater agreement or analyze the consistency of preferences across evaluators. This is particularly important because design quality and aesthetic preferences can be subjective and may vary across professional backgrounds. While provide bootstrap confidence intervals for the aggregated preference estimates are provided, these do not directly establish whether individual evaluators consistently agree. Reporting interrater agreement and the distribution of preferences across designers would strengthen the reliability of the human evaluation, especially for comparisons where the preference margins are relatively small.

  • The main components of Design Anything including persistent task state, explicit requirement tracking, workflow graphs and progress based execution control are closely related to established agent orchestration techniques. For instance, Magentic One maintains task and progress ledgers to guide execution and monitor completion, while LangGraph supports persistent graph state and durable workflow execution. Even though Design Anything integrates these mechanisms specifically for complex design workflows, the paper does not sufficiently establish which aspects represent a fundamentally new technical contribution rather than an application specific combination of existing orchestration principles. The proposed benchmark is a useful contribution, but the methodological novelty of the harness itself remains less convincing. A clearer comparison with these existing frameworks, particularly regarding requirement tracking and completion control would strengthen the novelty claim.

Questions for authors
  • How does Design Anything differ fundamentally from existing agent orchestration frameworks such as Magentic One and LangGraph, which already support persistent state, task tracking, and workflow execution? Could the authors clarify which components are technically novel rather than adaptations of established orchestration mechanisms?

  • What is the impact of progressively removing the task contract, workflow graph and deterministic controller?

  • The constraint satisfaction metric in Eqn 6 primarily checks whether factual information appears in all relevant artifacts. How would the results change under a semantic evaluation that verifies whether the information is correctly interpreted and applied, rather than merely present?

  • Could the authors provide a more comprehensive comparison of total execution time, number of tool calls and generation costs against the main baselines?

  • Can the authors compare Design Anything against a frontier agent given the same tools, persistent requirements and explicit completion instructions, but without the proposed workflow graph and deterministic controller? Currently it is difficult to establish whether the full harness provides benefits beyond those achievable through stronger agent capabilities and simpler orchestration.

8 comments
AuthorsOpenReview ↗

Response to Reviewer VhqJ (1/3): W1 stronger agents, W2 interrater agreement

Thank you for the careful review and for the concrete experiments you proposed; we ran each of them. We answer in your order: W1 and W2 here, W3 in Part 2, and Q1–Q5 in Part 3. In the revised PDF, changes made for you alone are in red, and passages that also answer other reviewers are in blue; each answer gives the line numbers.

W1. Necessity of the harness with stronger agents

We ran the baseline you describe on GPT-6 Astra, and the harness is still needed. Given the requirement list and a completion check, Codex delivers almost every output, but one requirement in four still does not reach every output that needs it. We ran Design Anything (DA), Codex and Codex + list on 19 paired 24-output tasks. Codex + list adds to Codex's instructions the brief's requirement list, its list of outputs, and a check before finishing.

GPT-6 Astra, 19 tasksDACodexCodex + listp, DA vs Codex + list
Delivery (%)98.595.099.80.11
Constraint satisfaction (%)97.372.675.30.027
Identity consistency (%)96.888.988.10.031
Reference fidelity (%)84.774.078.30.76
Set consistency (0–100)78.684.993.8<0.001

Identity (9 tasks) / reference fidelity (15 tasks): share of outputs showing the same object as the anchor output / a supplied reference. DA also scores highest on all four per-image criteria, significantly on domain criteria (p=0.032).

  • The list fixes delivery, not the loss of requirements. Delivery rises from 95.0% to 99.8%, but constraint satisfaction moves only from 72.6% to 75.3%: the agent must still repeat each requirement in each call.
  • A newer model does not close the gap. On the same 19 tasks with GPT-5.6 Sol, constraint satisfaction is 76.7% for Codex and 95.9% for DA. With GPT-6 Astra it is 72.6% and 97.3%: the gap is 19.2 points, then 24.7.
  • The image tool does not explain the gap. Codex uses its built-in image tool and DA a separate image model. Their text-rendering scores are close (95.0 for Codex + list, 95.9 for DA; p=0.37), so the gap in constraint satisfaction is unlikely to come from the image tool.

Revision: Section 3.3, lines 266–301 (Table 2a, now in the main text); Appendix C.8, lines 1708–1744 (Table 17).

W2. Interrater agreement in the human evaluation

Thank you for asking for this analysis. Designers agree well above chance, and against OpenDesign and Codex nearly every designer prefers DA.

A correction. While preparing it we found a counting error: the 1,019 whole-set judgments come from twelve designers, not nine. The revision corrects the count (lines 464 and 1132).

Agreement. On 41 tasks, two designers of the same discipline rated the same comparisons. They made the same choice in 60.0% of comparisons (95% CI 52.2–66.3%, median Cohen's κ 0.40), and repeated their own choice in 76.5–82.4% of cases when shown a comparison again with sides swapped.

Preference per designer (table below; scored as in Figure 6, a tie counts as half). Against JarvisHub, the closest pair, designers are split, and DA leads when all judgments are pooled. Each designer rated 13–27 comparisons per pair, so single-designer shares are noisy.

PairDesigners preferring DAAll judgments pooled (Figure 6)
vs OpenDesign12 of 12DA preferred in 82%
vs Codex11 of 12DA preferred in 70%
vs JarvisHub5 of 12 (6 prefer JarvisHub, 1 even)DA 51%, JarvisHub 37%

Revision: Appendix B.5, lines 1154–1162; Section 3.5, lines 476–478.

AuthorsOpenReview ↗

Response to Reviewer VhqJ (2/3): W3 novelty

W3. How DA differs from Magentic-One and LangGraph

Thank you for naming the closest frameworks; the comparison below is now in the paper.

DA solves a problem that existing harnesses leave to the model or to the developer: in long generation tasks, nothing reports a missing output or a dropped requirement, so agents stop early and outputs drift from the brief. DA moves both jobs into the harness: a rule checks completion against the brief, and a graph carries each requirement into every generation call that needs it (Section 2.2).

The two problems. Agents fail more often as tasks get longer [1]: they stop early [2] and lose information [3, 4]. Harness design alone changes agent performance [5], and generation makes both failures harder to catch:

  1. Stopping early. A missing output raises no error, so the model alone decides when the task is done, and it stops too soon.
  2. Losing information. A generation model has no memory of the task: it sees only the prompt and inputs of one call. If the brief fixes a product name and the agent chooses a logo early on, it must repeat both in every later call. After one omission, that output no longer matches the rest, again without an error.

On our benchmark, both get worse with length: from 6 to 24 outputs, JarvisHub's delivery falls from 93.4% to 81.8%, and constraint satisfaction drops 21.9 points for JarvisHub and 16.1 for Codex, against at most 1.7 and 5.5 for DA.

How existing harnesses and DA handle them. The frameworks you name do keep persistent state and track requirements. The difference is who decides completion and who delivers the information:

DimensionExisting harnessesDA
Requirement trackingThe model keeps a plan and judges its progress (Magentic-One [6])Task contract: the agent records every output the brief requires when it plans. It holds no progress, so the model cannot mark one done.
Completion controlThe model (Magentic-One) or developer code (LangGraph)Controller: after each turn, a rule checks that every listed output was produced, finished, and of the right kind and count.
Information passingThe agent repeats requirements and choices in each call (Codex; JarvisHub [7]). LangGraph nodes share a state, but developer code decides what each node reads.Graph: the agent connects a source once. Every later call receives it as input.

Evidence. Without the controller or the contract, the agent stops early more often (Q2, Part 3), and a requirement list in the prompt does not replace the graph (W1, Part 1).

In short, the contract and the controller keep the agent working until every required output exists, and the graph carries each requirement into every output.

The benchmark is a second contribution, as you note: each of 38 project families is posed at 6, 14 and 24 outputs, which no benchmark we survey does (Appendix G).

Revision: Section 1 states the two problems and how existing harnesses handle them (lines 62–78). Section 4 (lines 481–531) and Appendix G (lines 2810–2877, Table 19) compare DA with them.

References

  • [1] Measuring AI Ability to Complete Long Software Tasks, NeurIPS 2025
  • [2] Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces, ICLR 2026
  • [3] Lost in the Middle: How Language Models Use Long Contexts, TACL 2024
  • [4] LLMs Get Lost in Multi-Turn Conversation, ICLR 2026
  • [5] SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024
  • [6] Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks, arXiv 2024
  • [7] JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents, arXiv 2026
AuthorsOpenReview ↗

Response to Reviewer VhqJ (3/3): Q1–Q5

Q1. How does DA differ fundamentally from Magentic-One and LangGraph?

Please see W3 (Part 2): existing frameworks leave the completion decision and the passing of requirements to the model or to developer code, and DA moves both into the harness.

Q2. Removing the components one at a time

Each removal makes the agent stop early more often: the 19 runs need 2, then 24, then 47 external prompts. On the 19 paired 24-output tasks (DeepSeek V4.1 Flash High, one run each), we removed the controller, then the contract as well. When the agent stops, the benchmark prompts it to continue; an external prompt is counted when the agent did more work after it, so it marks an early stop. Later outputs are those made after the first external prompt.

19 tasksFullNo controllerNo controller, no contract
External prompts22447
Later outputs5.8%5.2%40.1%
  • Without the controller, external prompts rise on 18 of 19 tasks (p<0.001), the main agent takes 23% more steps and 21% more LLM cost (both p=0.014), and three runs end with outputs missing (delivery 97.8% → 95.2%, p=0.031). Five of the six missing outputs had never entered the agent's plan; the controller checks against the contract, not the plan.
  • Without the contract too, external prompts double again (p<0.001), and later outputs rise to 40.1% (p=0.008): without the list, the agent no longer plans the whole task up front.
  • The graph runs every generation call, so its effect is measured against harnesses without one: Codex + list (W1) and JarvisHub's per-call logs (Appendix C.5).

Why external prompts matter. In these tests a prompt answers every stop, so the damage shows as extra prompts. With nobody to prompt, an early stop ends the run: in Table 3a, with the contract, the controller and the prompts all removed, delivery falls from 96.1% to 73.9%, and 6 of 19 runs deliver fewer than half of their outputs.

In short, the contract makes the agent plan every output, the controller keeps it working until all exist, and the graph carries requirements into each call; the run finishes on its own only with all three.

Revision: the table is now in the main text (Section 3.4.4, lines 433–461, Table 3b); setup in Appendix D.3, lines 1863–1904.

Q3. A more semantic evaluation

Judged by meaning, DA still meets far more requirements: 94–100%, against 48–67% for the baselines. For each of the 38 briefs at 24 outputs, we wrote about four visual requirements (e.g., "the privacy shutter covers the camera"), each traced to the brief. Two designers, blind to the method, judged 130 cases on delivered outputs and agreed on 93.1% (κ=0.83).

DA (GPT-5.6 Sol)DA (DeepSeek)JarvisHubOpenDesignCodex
100%94.2%67.3%55.8%48.1%

Every same-model gap is significant (Fisher exact p ≤ 0.023). Revision: Section 3.3, lines 303–310 (Table 2b, now in the main text); Appendix B.7, lines 1213–1252.

Q4. Time, tool calls and cost

On DeepSeek, DA makes the fewest tool calls and costs $0.95 per task, against at least $3.30 for JarvisHub. On GPT-5.6 Sol, DA takes longer than Codex. All 228 tasks, 10 in parallel:

MethodHours, all 228Tool callsCost per task
DA (DeepSeek)911,207$0.95
OpenDesign (DeepSeek)1353,337$0.44
JarvisHub (DeepSeek)815,211≥ $3.30
DA (GPT-5.6 Sol)2518,946–
Codex (GPT-5.6 Sol)74,865–

Costs: API bills for media, list prices for the LLM; JarvisHub's is a lower bound, as a few of its records carry no price. The GPT runs used subscriptions, so they have no cost. On GPT-5.6 Sol, DA makes more tool calls than Codex and satisfies 14.4 points more of the constraints (92.5% vs 78.1%). Revision: Appendix C.7, lines 1687–1705 (Table 16).

Q5. A frontier agent with the same requirements and completion instructions, but without the harness

Please see W1 (Part 1): this is the Codex + list run on GPT-6 Astra. It delivers 99.8% of outputs but satisfies 75.3% of constraints, against 97.3% for DA (p=0.027).

Thank you again for the detailed questions. We hope the new experiments address your concerns about soundness, and we are glad to run further checks during the discussion.

Reviewer_AnyQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Design Anything. Existing agents can produce multiple deliverables in a single run, but as the number of deliverables increases, they tend to forget the original requirements and lose consistency, or even terminate prematurely. The authors integrate a Task Contract to preserve design requirements, a Workflow Graph to propagate previously selected designs and reference materials, and a Deterministic Controller to determine which deliverables remain incomplete based on recorded states and decide whether execution should continue. The authors evaluate the system across multiple domains and tasks and introduce DesignAnything-Bench. In addition to outperforming the baselines, Design Anything's outputs are also generally preferred by professional designers in their evaluations.

Critical strengths

The paper integrates elements such as the Task Contract and Graph into a unified, trackable mechanism. It evaluates the system's capabilities in long-running workflows by testing different scales of the same project, together with blind evaluations by professional designers. The experiments already provide strong support for the research questions the paper aims to address.

Critical weaknesses

This is more of a suggestion for future work and a concern regarding practicality: Since Design Anything's main improvement lies in Constraint Satisfaction, but this metric relies on textual matching of specified facts across all relevant files, I think it is somewhat insufficient for comprehensively evaluating more complex design requirements (for example, if a client wants a product to be "ergonomic," even if all design drawings correctly label the product name and dimensions, this does not necessarily mean that the requirement has actually been satisfied).

Although the paper additionally uses a VLM judge and professional designers to partially cover design intent and visual requirements, incorporating more comprehensive evaluations of semantic and functional requirements in future work (for example, having a coding agent use domain knowledge to generate validation programs) could provide a more complete assessment of the system's actual design capabilities.

Questions for authors

JarvisHub already has a typed graph and persistent state, as well as mechanisms for reference and dependency connections.

Although I can understand that JarvisHub mainly relies on the agent to provide references during generation calls, while Design Anything connects references to downstream nodes through the Graph before execution so that subsequent generation steps can directly access these inputs, I am still somewhat uncertain about the differences between the two in actual execution, since Design Anything's Graph and reference connections are also constructed by the agent. If the same references are ultimately passed to the generation tools, what is the practical difference between establishing Graph connections in advance and directly specifying references in generation calls? What makes Design Anything more reliable in its use of references?

Would it be correct to understand that the mechanisms proposed in Design Anything could be incrementally added to the existing JarvisHub framework? If this is technically feasible (possibly requiring some adaptations), would it be possible to conduct the following experiments under the same model, generation tools, and execution budget?

  1. JarvisHub
  2. JarvisHub + Task Contract
  3. JarvisHub + Task Contract + Design Anything's reference connections and deliverable bindings
  4. JarvisHub + Task Contract + Design Anything's reference connections and deliverable bindings + Deterministic Controller

This should make it clearer how much improvement each mechanism contributes individually and help clarify the actual technical contributions of Design Anything relative to JarvisHub.

This is just my suggestion. If these mechanisms cannot be directly incorporated into JarvisHub, I would also appreciate it if the authors could further explain what substantial architectural differences between the two systems prevent such integration.

3 comments
AuthorsOpenReview ↗

Response to Reviewer AnyQ: W1, Q1, Q2

Thank you for the constructive review and the concrete experimental design. We answer the weakness first, then your two questions. In the revised PDF, changes made for you alone are in green, and passages that also answer other reviewers are in blue; each answer gives the line numbers.

W1. Evaluating semantic and functional requirements

We agree that text matching cannot judge requirements such as "ergonomic". As a first step, we added a designer-judged check of visible requirements that text matching cannot score: DA met 94–100% of them and the baselines 48–67%, with every same-model gap significant (Section 3.3, lines 303–310 (Table 2b, now in the main text); Appendix B.7, lines 1213–1252). Requirements such as ergonomics need domain checks beyond inspection. Generating validation programs from domain knowledge, as you suggest, would give generated outputs the kind of signal that tests give code; we add it to future work (Appendix F, lines 2773–2774).

Q1. "What is the practical difference between establishing Graph connections in advance and directly specifying references in generation calls?"

A connection is made and checked once; a call-time reference must be named correctly in every call, and each call is a chance to lose it. As you say, the agent picks the references in both harnesses. The difference is how often it must state them.

  • In Design Anything (DA) the agent draws one edge. The referenced node runs first, so its image exists, and every later generation and retry of the downstream node receives it.
  • In JarvisHub the agent must name each reference again in every call, in an accepted form, pointing to a node that already has an image.

JarvisHub's per-call logs on the same 228 tasks show how often this fails:

DAJarvisHub
Image calls naming references that were rejected–2,006 of 4,000 (50%)
Nodes generated with none of the references they named–22%
Regenerated nodes that kept their references37 of 3730 of 78
Image generations that carry a reference95.3%39.4%

–: DA names no references in its calls; they arrive through edges. DA's regenerations are counted on its 19-task runs.

Most rejections came from a reference in the wrong form or pointing to a node with no image yet. After a rejection the agent usually changed the references, and 434 calls succeeded only by dropping all of them. Connections remove these failure points, which is why DA's references survive. Revision: Appendix C.5, lines 1490–1502.

Q2. "Could the mechanisms be incrementally added to JarvisHub?"

The task contract can be added; the other two depend on DA's execution layer, so we ran your four-step ladder inside DA. Reference connections need graph-based execution, and the controller counts remaining work from contract items bound to graph nodes. Adding both to JarvisHub would rebuild DA inside it and measure our port, not the mechanisms. On the 19 paired 24-output tasks of Table 3 (same model, tools, tasks and stopping rule), we started from DA with neither contract nor controller and added one mechanism per step, in your order. When the agent stops, the benchmark prompts it to continue; an external prompt is counted when the agent did more work after it, so it marks an early stop.

StepExternal promptsOther effects
1 → 2, task contract47 → 24 (p<0.001)Later outputs (made after the first external prompt): 40.1% → 5.2% (p=0.008)
2 → 3, connections and bindings–Bindings come with the contract; connections cannot be switched off, so JarvisHub (Q1) is the call-time alternative
3 → 4, controller24 → 2 (p<0.001)Runs with missing outputs: 3 → 0; without it, the main agent takes 23% more steps and 21% more LLM cost (both p=0.014)

With the contract, the agent plans every output up front; with the controller, it rarely stops before the task is done. The closest test of the contract on another harness is Codex given the brief's requirement list and a completion check (GPT-6 Astra, 19 tasks): it delivers 99.8% of outputs but satisfies 75.3% of constraints, against 97.3% for DA (p=0.027). Revision: Section 3.4.4, lines 433–461 (Table 3b); Appendix D.3, lines 1863–1904.

Thank you again for the suggestions. We are glad to discuss further.

Thank you for the detailed response and additional experiments.

The reference usage statistics are particularly helpful and clarify the practical differences between Design Anything and JarvisHub.

I also appreciate the additional ablation studies and the explanation of why directly incorporating these mechanisms into JarvisHub would be difficult. While the contribution of reference connections is still not fully isolated through ablation, I understand the architectural limitations.

Overall, the response has addressed most of my concerns. Thank you for the effort put into the revision. I maintain my original rating.

Thank you for reading our response and the revision so carefully, and for the kind words about the reference statistics and the ablations. Your two questions led directly to both: the per-call analysis of references (Appendix C.5) and the four-step ladder (Table 3b) are now part of the paper, and it is clearer for them.

Thank you again for the constructive suggestions throughout the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and revisions

Dear Area Chair,

Thank you for your time and effort in handling our submission. We are also grateful to the three reviewers for their careful and constructive discussion; their suggestions led to new experiments and a clearer paper.

Summary of our work

Long-horizon generation asks an agent to deliver many interdependent outputs in one run, with no test or error signal to reveal a missing output or a dropped requirement. Our paper makes three contributions.

(1) A harness for long-horizon generation. In a standard agent loop, the model decides from its own context whether the task is finished and what the next call receives. Design Anything (DA) computes both from an explicit task state: termination is one fixed rule over the outputs the brief requires, and the inputs of each generation call are read from a workflow graph. On the same model, DA satisfies 85.0% of constraints against 68.0% for JarvisHub, and 92.5% against 78.1% for Codex.

(2) A benchmark for long-horizon generation. DesignAnything-Bench contains 228 tasks in 19 domains, built from professional design briefs, where one brief requires many outputs that must stay consistent. Each of 38 briefs is posed at 6, 14 and 24 outputs, so task length varies within a brief. It is hard for current agents: from 6 to 24 outputs, constraint satisfaction falls by 21.9 points for JarvisHub and 16.1 for Codex, and at most 5.5 for DA.

(3) A comprehensive evaluation. We evaluate the whole set of outputs of a task, not a single output as in generation benchmarks such as [1, 2]. Automatic measures cover delivery, constraint satisfaction, output quality, consistency across outputs and fidelity to references, on three language models and three baselines. Professional designers add requirement checks by meaning and a blind preference study (12 designers).

Strengths highlighted by the reviewers

  • An important and practical problem. JVQy described the research topic as “meaningful and practical.” AnyQ's summary restates the failure we target: agents “forget the original requirements and lose consistency, or even terminate prematurely.”
  • A unified method, distinct from the closest harness. AnyQ noted that the paper integrates the task contract and the graph into “a unified, trackable mechanism,” and later wrote that the added statistics “clarify the practical differences between Design Anything and JarvisHub.”
  • A benchmark valued by all three reviewers. JVQy called it “a comprehensive benchmark.” VhqJ called it “an interesting benchmark for evaluating long horizon design consistency” and “a useful contribution.” AnyQ highlighted that it tests “different scales of the same project.”
  • Evaluation beyond automatic metrics. JVQy wrote that the evaluation offers “a more holistic assessment,” and that the study with professional designers provides “valuable evidence of the practical quality” of the outputs. AnyQ judged that the experiments “already provide strong support for the research questions the paper aims to address.”

Main concerns and how we resolved them

  1. Necessity with stronger agents (VhqJ, JVQy). In the experiment VhqJ proposed, on the stronger GPT-6 Astra, Codex given the full requirement list and a completion check satisfied 75.3% of constraints; DA satisfied 97.3% (p = 0.027, Table 2a).
  2. Methodological novelty (VhqJ, JVQy). We added a Related Work section and a comparison with six frameworks (Section 4, Table 19). Magentic-One leaves completion to the model and LangGraph to routing written per application; none of the six is described as checking completion against the outputs the brief requires, as DA's rule does (Eqs. 4–5).
  3. Contribution of each component (AnyQ, VhqJ). In a step-by-step removal study (Table 3b), the external prompts needed after early stops rose from 2 to 24 without the controller, and to 47 without the contract as well. AnyQ found the added per-call reference statistics (Appendix C.5) “particularly helpful.”
  4. Semantic evaluation of requirements (all reviewers). Two designers, blind to the method, judged whether 130 requirements were met (κ = 0.83). DA met 94–100%, the baselines 48–67% (Table 2b).

Outcome of the discussion

Two of the three reviewers now recommend acceptance. JVQy raised their rating from 2 to 3 (weak acceptance). AnyQ maintained a 3 and wrote that the response “has addressed most of my concerns.” VhqJ maintained a 2, wrote that the responses “address a lot of my concerns” and appreciated “the substantial additional experimental effort”; we answered each remaining point (novelty, cost and rater agreement) in a two-part follow-up.

All changes are highlighted in color in the revision.

Sincerely, Authors of Submission 25711

References

  • [1] Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering, NAACL 2025
  • [2] Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers, NeurIPS 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.