- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AutoDesign optimizes the executable harness around a fixed language model. An outer coding agent revises the harness using rollout feedback and a development-set acceptance gate, with optional human guidance. The resulting DesignHarness generates editable scientific posters and is evaluated on the 100-paper PosterBench, a 10-paper controlled subset, and a human preference study.
Critical strengths
- With Claude Code and Claude 4.8 fixed, AutoDesign scores 78.32 versus 70.01 for the standalone agent, 70.87 for Claude Design, and 69.45 for OpenDesign (Table 1). These results support the final harness's usefulness.
- Attaching the harness improves all seven model-agent configurations by 5.01-19.56 points (Table 2). The gains extend beyond one model.
- The human study includes 933 judgments from 11 reviewers and reports a 64.0% Bradley-Terry preference estimate, with paper-reviewer bootstrap intervals (Appendix A.8). It also discloses low reviewer agreement and imperfect metric alignment.
- Editable HTML and source provenance suit scientific poster production. The distinction between the inner critic, optimization evaluator, and frozen benchmark scorer is clearly explained.
Critical weaknesses
- training/development set sizes, identities, and overlap with PosterBench and evaluator reference posters are unspecified (Sections 2.3 and A.3). A frozen scorer does not guarantee an untouched test set. This is a reporting gap, not evidence of contamination.
- the human-level claim exceeds the evidence: the human study compares automated systems, not human-designed posters. Automatic scores correlate with preference at r = 0.34, with only 51.9% agreement for 0-3-point gaps. Neither result establishes a human-quality threshold.
- designer-only API cost excludes harness development: seven days, 224 subagents, at least 123 iterations, 54 updates, evaluator calls, and human guidance. These costs should be reported separately from per-poster inference and amortized over realistic usage.
Questions for authors
- how do the initial, autonomously optimized, human-guided, and fixed handcrafted harnesses compare on untouched papers at equal generation budgets?
- what were the total search costs and human effort, separate from per-poster inference?