acceptodds
Under review as a conference paper at ICLR 2027

AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design

Abstract

Transforming multimodal sources into condensed and structured media outputs can be fundamentally conceptualized as a long-horizon agentic process centered on a model-harness system. While an ideal harness system should align with human design priors and accumulate reusable experience through empirical exploration to drive recursive self-improvement, existing paradigms remain static and fall short of this capability. In this paper, we present AutoDesign, a framework that aligns with human design priors, where a meta-harness optimizer guides a code agent to recursively improve harness based on rollout feedback. To instantiate and evaluate this framework, we focus on the academic paper-to-poster generation task and introduce PosterBench, comprising a 100-paper Main Track spanning five disciplines and PosterBench-mini, a shared 10-paper subset for controlled evaluation. On the PosterBench Main Track, AutoDesign achieves the highest score of 78.32, surpassing the closed-source commercial system Claude Design by 7.45 points. Across seven controlled code agent–model configurations, integrating the learned DesignHarness consistently improves performance, increasing the average PosterBench Score from 54.99 to 67.39 (+12.4). In a fully autonomous long-horizon loop, it executes 253 tool calls and 11 editing turns within 40 minutes for under $3, reaching average conference-poster quality in human evaluation. A system-blind human study further demonstrates that AutoDesign achieves the highest human preference among evaluated systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
meta-harness optimization, self-improving agents, long-horizon agentic design, multimodal generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p9VwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoDesign optimizes the executable harness around a fixed language model. An outer coding agent revises the harness using rollout feedback and a development-set acceptance gate, with optional human guidance. The resulting DesignHarness generates editable scientific posters and is evaluated on the 100-paper PosterBench, a 10-paper controlled subset, and a human preference study.

Critical strengths
  • With Claude Code and Claude 4.8 fixed, AutoDesign scores 78.32 versus 70.01 for the standalone agent, 70.87 for Claude Design, and 69.45 for OpenDesign (Table 1). These results support the final harness's usefulness.
  • Attaching the harness improves all seven model-agent configurations by 5.01-19.56 points (Table 2). The gains extend beyond one model.
  • The human study includes 933 judgments from 11 reviewers and reports a 64.0% Bradley-Terry preference estimate, with paper-reviewer bootstrap intervals (Appendix A.8). It also discloses low reviewer agreement and imperfect metric alignment.
  • Editable HTML and source provenance suit scientific poster production. The distinction between the inner critic, optimization evaluator, and frozen benchmark scorer is clearly explained.
Critical weaknesses
  1. training/development set sizes, identities, and overlap with PosterBench and evaluator reference posters are unspecified (Sections 2.3 and A.3). A frozen scorer does not guarantee an untouched test set. This is a reporting gap, not evidence of contamination.
  2. the human-level claim exceeds the evidence: the human study compares automated systems, not human-designed posters. Automatic scores correlate with preference at r = 0.34, with only 51.9% agreement for 0-3-point gaps. Neither result establishes a human-quality threshold.
  3. designer-only API cost excludes harness development: seven days, 224 subagents, at least 123 iterations, 54 updates, evaluator calls, and human guidance. These costs should be reported separately from per-poster inference and amortized over realistic usage.
Questions for authors
  1. how do the initial, autonomously optimized, human-guided, and fixed handcrafted harnesses compare on untouched papers at equal generation budgets?
  2. what were the total search costs and human effort, separate from per-poster inference?
Reviewer_KthpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AutoDesign uses a coding-agent loop to modify a reusable design harness from poster-generation trajectories and evaluator feedback; a train-improvement/dev-nonregression gate selects updates. Its final DesignHarness creates editable HTML academic posters with source grounding, rule-based checks, visual critique, and iterative revision. On the 100-paper PosterBench, the Claude scores 78.32 versus 70.87 for Claude Design. A system-blind study with 11 volunteers and 933 comparisons gives AutoDesign a Bradley–Terry estimated win probability of 64.0% against a uniformly sampled alternative; seven harness-attachment comparisons on the 10-paper PosterBench-mini show gains of 5.01–19.56 points.

Critical strengths
  1. The paper addresses a practically important long-horizon design problem with an integrated, editable workflow. DesignHarness combines source grounding, validation, visual critique, and localized revision in editable HTML, making the system relevant to practical scientific communication.
  2. The evaluation is broad and well supported by controlled and human studies. PosterBench covers 100 papers across five disciplines, with matched comparisons under a frozen evaluator. DesignHarness also improves performance across seven model–agent configurations, and the paper further validates the results with runtime/cost analysis and a system-blind human study involving 11 reviewers and 933 non-skip judgments.
Critical weaknesses
  1. The experiments provide stronger evidence for the final DesignHarness than for the meta-harness optimization procedure itself. The final harness comes from a single seven-day optimization record with optional human guidance, but the paper does not report independent optimization runs or matched-budget comparisons against manual or simpler iterative harness revision. The harness-attachment ablation also does not establish matched tool access, critic feedback, validation, or revision budgets between conditions.
  2. The evidence for strong human-alignment and “human-level” claims is limited. PosterBench has only moderate poster-level correlation with human preference (r=0.34r=0.34), 51.9% agreement for 0–3-point score gaps, close to chance, and a low nominal Krippendorff coefficient of 0.101, indicating very low reviewer agreement. The human study uses 11 volunteers and compares AutoDesign only with automated systems, without a human-designed poster baseline. In addition, the exact VLM and version used by the frozen PosterBench evaluator are not clearly specified, and the robustness of its manually specified weights and score ceilings is not established.
Questions for authors
  1. What are the exact sizes and identities of DtrainD_{\mathrm{train}} and DdevD_{\mathrm{dev}}, and how do they relate to the final PosterBench evaluation sets? Given that DdevD_{\mathrm{dev}} is repeatedly used for acceptance decisions across 123+ adaptive iterations, how do the authors mitigate potential selection overfitting?
  2. Can the authors report multiple independent AutoDesign optimization runs and compare the proposed outer loop with simpler or matched-budget alternatives, such as optimization without human guidance or iterative coding-agent revision? In the harness-attachment ablation, can the original and DesignHarness conditions be compared under matched inference budgets, for example maximum revision attempts, tool calls, critic usage, token/API cost, or wall-clock time?
  3. Which exact VLM and version are used by the frozen PosterBench evaluator? Are the reported rankings and score differences stable across repeated evaluations, an alternative evaluator, and reasonable variations of the dimension weights and record-level ceilings?
  4. Given the modest agreement between PosterBench and human judgments, how should the reported benchmark–human alignment be interpreted? Could the authors clarify how the “human-level academic posters” claim should be interpreted, given that the human study does not include human-designed posters as a reference?
  5. The broader multimodal-design framing is only partially validated. The paper presents AutoDesign as a general framework for long-horizon multimodal design, but the formal evaluation is limited to paper-to-poster generation. Slides, webpages, and videos are only demonstrated as pilot outputs rather than systematically evaluated.

Convincing answers to these questions would make me consider raising my score.

Reviewer_H1BVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AutoDesign, a Meta-Harness Optimization framework. Unlike agent-based methods that only revise the results of individual generations, AutoDesign analyzes execution records across multiple tasks and modifies modules within the Harness, allowing improvements to be reused in subsequent tasks. The paper focuses primarily on academic poster generation as its main application and experimental setting, and introduces PosterBench. Experimental results show that AutoDesign outperforms several existing systems.

Critical strengths
  • The system itself is complete and practical, covering content extraction from papers, HTML generation, validation, and iterative improvements accumulated through multi-round experience. The experiments also involve different models and agents.
Critical weaknesses
  • My main concern is the lack of comparisons with meta-harness optimization baselines. The experiments mainly compare existing poster generation frameworks (Paper2Poster, PosterGen). I think it would be necessary to compare against existing general-purpose Meta-Harness Optimization methods (e.g., [1]) and investigate whether modifying the entire Harness is actually more effective than simple Prompt Evolution [2].

  • Although the paper focuses on Agentic Design, the quantitative evaluation is limited to academic poster generation. Other tasks, such as slide decks, webpages, and conference videos, are not quantitatively evaluated to the same extent as posters (the results presented in Appendix B look promising, and I understand that quantitative evaluation of different tasks may require additional benchmarks and substantial API costs, but I hope the authors can further explain these limitations).


[1] Meta-Harness: End-to-End Optimization of Model Harnesses.

[2] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, ICLR 2026.

Questions for authors

Q1. Could the authors provide a fair comparison with existing Meta-Harness Optimization methods? (Even if AutoDesign performs better, is this because it is allowed to modify tools, runtime, and orchestration, or because its update algorithm is better?) If such a comparison is not feasible, please explain why. I would also appreciate clarification if I have misunderstood anything.

Q2. Could AutoDesign be evaluated using existing poster benchmarks? If not, please explain why, and I would appreciate clarification if I have misunderstood anything.

This is because the correlation between PosterBench Score and human preference is r = 0.34, indicating that there is still a gap between the metric and human preferences. Therefore, including existing human-designed posters under the same evaluation protocol, or directly using previous benchmarks for academic poster generation [1][2][3], would help provide a more meaningful reference point for assessing AutoDesign's output quality.

Q3. The authors include very specific HTML and DOM structure requirements in the shared generation prompt. I would like to ask whether Paper2Poster was evaluated using the approach from its original paper (as far as I can see, its GitHub repository outputs PPTX files), or whether its output specification was also changed to HTML. Otherwise, there may be concerns regarding the fairness of the comparison.

Q4. This is a more open-ended question: The paper reports the cost of AutoDesign. If we hired a designer to directly create a reasonably high-quality, reusable Static Skill or Harness, could it achieve comparable results? Or could AutoDesign discover an even better solution? (In future work, comparing the cost of evolution itself with the cost of hiring a designer could further help demonstrate the practical value of AutoDesign.)


[1] P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark, ICLR 2026.

[2] Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers, NeurIPS 2025.

[3] PosterForest: Hierarchical Multi-Agent Collaboration for Scientific Poster Generation, ACL 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.