acceptodds
Under review as a conference paper at ICLR 2027

CREATIVE DESIGN OF AGENTIC SYSTEMS

Abstract

Frontier models are increasingly contributing to mathematical problem solving and scientific discovery, making creativity a central capability to evaluate beyond established task performance. Existing agent benchmarks keep the agent architecture fixed and measure execution ability, while creativity studies assess generated content rather than the executable systems agents build. Therefore, we present Agents Creating Agents (ACA), the first benchmark for evaluating the creativity of agents from an architectural perspective: an architect transforms real development tasks into an executable architecture satisfying utility, novelty, and cost awareness, which together form the Creativity score. We further propose Evidence-Grounded Mechanism Search (EGMS), guides architecture search using the causal structure of failures rather than a scalar development score. EGMS improves mean utility by 6.4 points over the strongest of five baselines and raises the Creativity score to 0.272 . Further analysis shows that the harness influences architectural creativity as much as the backbone model, and a small local model with a compiled domain architecture outperforms on the Codex harness with runtime skill injection, suggesting that architecture can substitute for model scale at execution time. Our code is open at https://anonymous.4open.science/r/ACA-BENCHMARK-2775/.

Primary area:
datasets and benchmarks
Keywords:
Agents Creating Agents (ACA), agent architectural creativity, Evidence-Grounded Mechanism Search (EGMS), agent benchmark, architecture search, model scale substitution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QKcPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a benchmark for evaluating LLM agents on optimizing agent harnesses. They adapted tasks from STATE-bench and defined a "Creativity Score" which is a combination of task performance, LLM-judged harness novelty, and token cost. They also propose a method for harness optimization that learns from failure execution examples. They report improved success rate over baselines.

Critical strengths
  1. Harness optimization has gained a lot of attention recently and a dedicated benchmark would be useful to make fair comparison in this field.

  2. The proposed approach to automate harness engineering achieves improvement of average performance. It leverages lessons from failures and preserves success.

Critical weaknesses
  1. The paper claims to evaluate agent creativity yet its core metric "novelty" is not validated. I'd like to see more evidence like: agreement between agent-as-a-judge and human experts, how sensitive the judge is to repeated sampling, prompt variation, and agreement between different models as judge.
Questions for authors
  1. The paper uses a 100-template library to compute novelty score. Is the coverage sufficient enough?
Reviewer_72sgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ACA, a benchmark for evaluating the creativeness of agents generating harnesses (in their paper they call them "architecture"). The paper also proposes EGMS, a method to improve agents harness optimization using agent failures. Experiment show that the method (EGMS) is both effective in terms of utility (quality) and creativity.

Critical strengths
  • The paper did research on important topic: on creativity of agents optimizing harnesses. evaluating agent diversity (or the lack thereof) should be of interest of many researchers building agentic systems and optimizing them.

  • The experiment results are good: proposed method outperform other baselines.

  • The benchmark itself originates from practical settings, which is unlike certain benchmark that is on fully synthetic settings.

Critical weaknesses
  • The paper does not have solid evaluation of their method. For example, if I read it correctly, the methods all have very large variances across multiple runs, therefore it is hard to claim that their method is successful. Moreover, all evaluation is done on their own proposed benchmark with zero standardized benchmark evaluation for harness optimization. Furthermore, the authors only have ran their harness optimization on one single model (gpt 5.5) on one single benchmark proposed by themselves, so the claims might not be generalizable. The experiment part is my main concern.

  • The paper is hard to understand, with many important details hidden in appendix and many not so important details in the main paper. For example, one of the core contributions is the evaluation of creativity, in this case the authors mentioned that they used a "agent-as-judge" but didn't mention which agent was used and how in the main text. There is not evaluation of the judge here - what if the judge makes bad judgement of creativity? there are also far too many variables, some of them are not even defined (e.g. what does T mean in line 334?).

  • Ablations: for a benchmark paper, the authors should report what setting they are using for getting the numbers. However, I couldn't find any documentation of temperature, reasoning effort, max generated tokens. It makes people hard to understand and reproduce the results.

Questions for authors
  • What is the difference between a harness and an architecture? it seems to me that there is not so why not use the more commonly used word "harness".
Reviewer_bRdcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A designer agent receives a task-domain description, tool specifications, and task examples of different domains. It generates executable code defining the target agent’s workflow, tests that agent on the examples, and revises the design. The proposed method, Evidence-Grounded Mechanism Search (EGMS), organizes revision around shared failure causes. It diagnoses where executions first go wrong, repairs the responsible mechanism, generates counterexamples, and checks for regressions. It also supplies architectural principles such as explicit goal tracking, grounding actions in observations, and verifying completion. The resulting agent is evaluated on 152 unseen tasks across customer support, shopping, and travel. EGMS reports 49.6% success versus 43.2% for the strongest baseline adaptation.

Critical strengths

The paper addresses an important and emerging problem: automatically designing reusable agent architectures from limited development evidence. This setting is challenging because the designer must search a large architectural space, diagnose failures, and develop mechanisms that generalize beyond observed examples without introducing regressions. The proposed benchmark and refinement framework provide a useful experimental setting for studying this problem.

Critical weaknesses

The evaluation does not convincingly establish a general advantage for EGMS. Its overall improvement is driven by customer support, while it slightly underperforms the strongest overall baseline on shopping and travel.

Questions for authors

Could you analyze cases where ADAS succeeds but EGMS fails, particularly in shopping and travel? What do these cases reveal about the limitations of the proposed search procedure and architectural guidance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.