acceptodds
Under review as a conference paper at ICLR 2027

AgentEvolve: LLM-Guided Multi-Objective Evolutionary Optimization for Expensive Black-Box Design Problems

Abstract

Many engineering workflows rest on an expensive multi-objective design problem: a designer sets a few named parameters, runs a slow tool to score each candidate, and trades off conflicting objectives such as the area, speed and power of a chip, within a budget of tens to hundreds of evaluations. Tuning chipimplementation flows, sizing analog circuits and searching neural architectures are such problems. Their parameters have names, units and a short description that tell an expert what good designs look like. Evolutionary and Bayesian optimizers ignore this information; language-model optimizers use it but fix the model’s role in advance, and budget-matched studies often trace their improvements to the surrounding loop rather than the model. We propose AGENTEVOLVE, a drop-in replacement for evolutionary optimizers in which the model is one more source of candidate designs and earns its budget share by measurement: the initial population it writes, periodic proposals and a sampler program it authors are pooled with evolutionary offspring, and each source receives evaluations in proportion to the hypervolume its past candidates added to the front. We evaluate one configuration on nine benchmarks (OpenROAD, an open-source flow that turns a hardware design into a chip layout, on three chip technologies; the NAS-Bench-301 architecture-search benchmark; XGBoost hyperparameter tuning; a knapsack; the sizing of two analog circuits) at validation seeds kept apart from development: AGENTEVOLVE needs 3.09× fewer evaluations than the evolutionary optimizer NSGA-II to reach a target hypervolume on NAS-Bench-301 and 2.90× fewer on a low-dropout regulator, ending with 28% and 50% more hypervolume, and outperforms AGENTEVOLVE without LLM on three benchmarks. On the knapsack a program the model writes from the item descriptions brings the median run within 10^−4 of the exact front’s hypervolume, using the cheapest model tier at under $0.33 per run.

Primary area:
optimization
Keywords:
LLM guided optimization, accelerator design, multi-objective search, agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YfXnOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AgentEvolve, a drop‑in LLM‑guided multi‑objective black‑box optimizer for costly engineering design problems including chip tuning, analog circuit sizing, architecture search and hyperparameter tuning. Rather than fixing the LLM’s functional role, AgentEvolve treats the LLM as one candidate source alongside classical evolutionary operators. Candidates from diverse sources are pooled, filtered by a surrogate model, and assigned evaluation budget via discounted Thompson sampling governed by per‑source hypervolume improvement credit. A Semantic Adapter resolves objective‑direction mismatches between the optimization engine and LLM prompts. The authors adopt a strict validation protocol with separated development and validation seeds and pre‑defined comparison rules. Across nine benchmarks, AgentEvolve decisively outperforms NSGA‑II on five tasks. It benefits from descriptive problem information that helps the LLM infer promising designs but fails on tasks with hidden narrow feasible regions or low‑quality LLM initial populations that subsequent optimization cannot recover.

Critical strengths
  1. Carefully controlled validation protocol for LLM‑based optimization. Separating development and validation seeds, locking evaluation targets in advance, and deploying targeted ablations (AgentEvolve without LLM, LLM‑as‑optimizer baseline) avoids post‑hoc result selection, a common flaw in related literature. This increases trust in empirical conclusions and offers a usable template for future work in the field.
  2. System design that treats LLM outputs as competing candidate sources. Budget share for LLM‑generated proposals is determined empirically by hypervolume gains instead of static rules. Its drop‑in compatibility with existing evolutionary tooling improves practical applicability for expensive real‑world black‑box design workflows.
Critical weaknesses
  1. Inadequate benchmarking against prior adaptive operator selection (AOS) methods. The proposed credit‑driven budget allocation builds on established bandit‑enabled AOS techniques. Direct comparisons against AOS, UCB and fixed‑allocation baselines are missing. It remains difficult to disentangle performance gains brought by the new allocation mechanism from gains stemming solely from high‑quality LLM proposals, blurring the paper’s methodological novelty.
  2. Under‑specified mathematical formulations and incomplete diagnosis of failure modes. Key equations carry unstated statistical assumptions, including risks of hypervolume improvement double‑counting and ambiguous probabilistic interpretations. For benchmarks degraded by poor LLM initial populations, there are no targeted ablations to test whether the credit allocation can mitigate incorrect LLM priors. These issues weaken theoretical foundations and understanding of observed empirical failures.
Questions for authors
  1. Using identical candidate source pools, can you supply comparative results against established AOS, UCB and fixed‑allocation baselines to isolate the performance contribution of your discounted‑Thompson‑sampling credit allocation?
  2. What statistical assumptions underpin multiplying expected hypervolume improvement by feasibility probability in Equation (1)? How does Equation (2) prevent double‑counting hypervolume gains from overlapping dominated regions across sources, and how is zero‑normalization implemented for credit computation?
  3. For cases with poor LLM‑generated initial populations such as SKY130HD and XGB‑40981, is your credit allocation mechanism intended to compensate for inaccurate LLM priors in theory and practice, and why do performance gaps persist in later generations?
  4. What formal probabilistic justification exists for using continuous credit rates as fractional pseudo‑counts in the Beta posterior updates of Equation (3)? How are the multiplicative advice weights and clipping bounds in Equation (4) justified without supporting sensitivity analysis?
1 comment

W1, Q1. "Direct comparisons against AOS, UCB and fixed-allocation baselines are missing."

All are run on identical source pools. With a correct description on the problem card, no test separates our rule from fixed shares, UCB1 or FRRMAB in final hypervolume. However, under a misleading problem description, ours is the only rule with a significant lead over fixed shares at these seeds (knapsack). We claim the method as a whole.

Change in final hypervolume of AgentEvolve against each rule: 16 seeds for fixed uniform shares (8 on XGB-40981), 8 for the others (marks: comment to all reviewers):

Benchmarkvs fixed uniform sharesvs UCB1vs FRRMAB (AOS)vs Adaptive Pursuit (AOS)
Knapsack0.0% (7-9)0.0% (1-7)0.0% (5-3)-0.02% (2-6)
NB-301+0.8% (9-7)+3.0% (5-3)+4.2% (7-1)-0.1% (3-5)
LDO+15% (12-4)+37% (5-3)+18% (5-3)+45% (6-2)*
Nangate45-0.1% (7-9)-0.4% (4-4)+1.3% (7-1)+0.6% (5-3)
XGB-40981-3.0% (2-6)-2.5% (2-6)-2.8% (1-7)+0.3% (5-3)

A difference of several percent is not excluded on the LDO, NB-301 and XGB-40981. On SKY130HD fixed uniform shares reach the target sooner (0.82x, 0-8, an estimate). The paper places the rule within adaptive operator selection (Sections 2 and 6, Appendix E).

W2, Q3. "... whether the credit allocation can mitigate incorrect LLM priors." "... why do performance gaps persist in later generations?"

Yes for the later budget: under a wrong description the rule takes it from the model sources and ends significantly ahead of fixed shares on the knapsack. What a poor initial population cost is already spent, and that gap closes in part. (Section 6, Appendices D to G)

  • A wrong description (labels permuted): the model sources' share of the later budget falls on the knapsack (86.5% to 15.0%) and Nangate45 (36.8% to 19.4%), not on NB-301. On the knapsack the rule ends significantly ahead of fixed uniform shares (+1.7%, 14-2, a family registered after some of its seeds were read). UCB1 against fixed shares: +2.0% (13-3), not separated, so we do not call our rule better than other adaptive rules.
  • A poor initial population: the credit compares sources within a run and never sees the run without the model, so it cannot return spent evaluations. On XGB-40981 the median gap in hypervolume ratio to AgentEvolve without LLM goes from -0.069 after the initial population to -0.048 at the end (14 of 16 behind), the model sources keep a median 34% of the later budget, and runs where they keep more end lower (rank correlation -0.66, post hoc). On SKY130HD the gap goes from -0.134 to -0.027. With a quarter of the initial population from the model, not half, the loss on XGB-40981 is not found (+0.05%, 8-8), at a cost against the half of 2.5% on NB-301 (an estimate) and 9.1% on the LDO (not separated, Appendix F).

W2, Q2. "What statistical assumptions underpin ... Equation (1)? How does Equation (2) prevent double-counting ...?"

Section 3 states what each equation assumes. Equation 2 allows double counting: in the recorded runs the median overcount is 0% to 4%. (Appendix D)

  • Equation 1 needs no independence assumption: with the improvement of an infeasible candidate counted as zero, its expectation is P(feasible) x E[improvement | feasible] by the chain rule. It ranks candidates and is not calibrated: the feasibility probability's calibration error is 0.04 to 0.08 on the chip flows and 0.22 to 0.28 on the LDO.
  • Equation 2 does not prevent double counting: credits are compared between sources and never added up. Over the 2,407 generations with a joint gain, the sources' gains sum to a median 1.00 to 1.04 times it (90th percentile up to 1.46). A Shapley split (nothing counted twice) changes final hypervolume by 0.0% on NB-301 and, as an estimate, -0.1% on XGB-40981.
  • Zero normalization divides a source's gain per candidate by the best such rate of any source over the last four generations (zero credit if that rate is zero).

Q4. "What formal probabilistic justification exists for ... fractional pseudo-counts ...? ... without supporting sensitivity analysis?"

Equation 3 is an expected update, not an exact posterior. Sensitivity analyses of Equations 3 and 4 are run, and no test separates the variants (one estimate on the LDO favours Equation 4's advice). (Appendix E)

  • Equation 3: adding r successes and 1 - r failures is the expected update of Thompson sampling with a 0-or-1 draw of success probability r (Agrawal and Goyal, 2011). With that draw instead of the fraction, AgentEvolve without LLM ends -2.3% to +2.5% in final hypervolume on five benchmarks, not separated.
  • Equation 4: the clip never binds in the submitted runs (0 of 2,000 values outside [1/4, 4]). Against the rule without the advice, the default ends -1.7% to +0.4% in final hypervolume on five benchmarks, not separated, and +20% on the LDO (7-1, an estimate).
Reviewer_qY2GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AgentEvolve replaces an evolutionary optimizer for expensive multi-objective design problems by treating an LLM as one more candidate source alongside NSGA-II offspring and local search: the model reads a problem card, writes part of the initial population, proposes designs periodically, and authors a sampler program, while a surrogate filter screens candidates and Thompson sampling gives each source budget in proportion to its measured hypervolume gain. The contributions are this credit-based arbitration, a sealed development/validation protocol with pre-registered targets, and an honest map of where LLM guidance fails. On nine benchmarks (three OpenROAD chip flows, NAS-Bench-301, two XGBoost tasks, a knapsack, two analog circuits) at under $0.33 per run, it is decisive over NSGA-II on 5/9 and over HEBO/qNEHVI on Nangate45, beats its own no-LLM ablation on 4/9 but trails on 2, and loses to qNEHVI on the op-amp where only measurements reveal feasibility.

Critical strengths

The paper is built to answer the question most LLM-optimizer papers avoid: does the model actually contribute, or is it the loop? The baseline design (NSGA-II, AgentEvolve without LLM, LLM as optimizer, Bayesian specialists, the earlier design) separates engine from model, the credit mechanism makes the model’s budget share a measured quantity rather than a design choice, and Figure 5a shows that share ranging from 92% on the knapsack to under 20% on the circuits. The paper then reports the losses as plainly as the wins: behind its own no-LLM ablation on two benchmarks, beaten by qNEHVI on the op-amp, no separation on XGBoost. I trust the positive results more because of how the negative ones are handled.

Critical weaknesses
  1. The paper cannot separate “the model reads the description” from “the model remembers the benchmark.” The dev/validation split is by seed, not by problem, so the engine and its dozen hand-set constants were tuned on the same nine benchmarks. The LLM helps exactly where public optima exist (OpenROAD gcd, NB-301, a knapsack whose card contains the whole instance) and fails on the op-amp, which has none. The standard control, obfuscated parameter names or a problem absent from training data, is not run, even though the paper cites Rodrigues et al. (2026) for this very confound. The no-LLM ablation already shows the engine carries most of the gain (4.65× → 1.51× on Nangate45, 3.09× → 1.65× on NB-301).
  2. The statistics cannot support the verdicts. Four validation seeds on the four headline benchmarks: a two-sided sign-flip test on four pairs bottoms out at p = 0.125, and a BCa bootstrap on four values has no usable coverage, so “significant” and “decisive” there are labels, not inferences. Roughly fifty paired verdicts are issued at p < 0.05 with no multiplicity correction. For a paper whose second contribution is its validation protocol, running the key benchmarks at a sample size the protocol cannot evaluate is the weakness that most affects my recommendation.
Questions for authors
  1. Was any benchmark held out entirely during development? If not, can you report one problem that was never seen while tuning the engine’s constants, run with the frozen configuration?
  2. Please give the raw per-seed ETT and final hypervolume for every pairing marked “significant” or “decisive” on the four-seed benchmarks, and state which test produced each p-value. With n = 4, a two-sided sign-flip test cannot reach 0.05, so I need to see what did.
  3. How many paired verdicts were issued in total, and what fraction remain significant under a Holm or BH correction across the nine benchmarks?
  4. What are NSGA-II’s population and offspring settings for the 100-evaluation chip flows, and were they tuned for that budget? AgentEvolve-without-LLM beating NSGA-II by 0.871 vs 0.698 on Nangate45 suggests either a strong surrogate-assisted engine or an under-configured baseline; a comparison to a standard surrogate-assisted EA would settle which.
  5. For LLM-as-optimizer, how many model calls and candidates per call were allowed, and was the total number of evaluated designs matched to AgentEvolve’s budget?
1 comment

W2. "The statistics cannot support the verdicts."

You are right. With 16 seeds, exact tests and Holm correction, 32 of the 54 primary tests are significant, 30 for AgentEvolve and 2 against it, both on XGB-40981. (Section 4, Appendix C)

  • Q2: on the four-seed benchmarks no test produced the 18 labels: all came from a bootstrap (BCa) interval, which with four pairs excludes zero exactly when all four agree. Appendix C lists the per-seed values.
  • Q3: the submission gave 52 paired verdicts on 103 tests, of which 15 have a raw p below 0.05, 0 pass Holm, 0 Benjamini-Hochberg.

W1. "The paper cannot separate 'the model reads the description' from 'the model remembers the benchmark.'"

The model's gain is not recall of the benchmark: pooled over two chip designs published nowhere it is significant (+3.8%, 26-6, 1.7x in evaluations), and it is not found once their names and text are blinded. On the public NB-301, uart, aes and truss recall is not excluded.

16 seeds (Sections 5, 6, Appendices G, H, marks: comment to all reviewers). The blinded card renames parameters and objectives and removes the text.

ProblemTrue card vs blinded cardBlinded card vs AgentEvolve without LLM
pmac16 (Nangate45)1.6x (14-2)* · +2.3% (14-2)*0.79x (7-9) · -0.9% (8-8)
crot16 (Nangate45)2.6x (14-1)† · +15% (16-0)†1.0x (5-7) · -6.0% (5-11)
Nangate452.4x (16-0)† · +14% (16-0)†0.62x (2-14)* · -4.8% (1-15)*
NB-3011.7x (11-4) · +3.9% (12-4)2.0x (13-3)* · +4.7% (12-4)*
  • Absent from training data: we wrote pmac16 and crot16 for this study, and the pooled test was registered before crot16 was written. The gain is significant on each alone.
  • Obfuscated names: the last column is the blinded card. Under a misleading card (labels permuted) the gain is not found either (-1.3%, 6-10, and -2.1%, 6-10).
  • What carries the gain: against the blinded card, the text alone is significantly ahead on pmac16 (+2.9%), the names alone are not (+1.7%). crot16 does not replicate it: the names alone are (+8.5%), the text alone is not (+9.3%, 12-4), so there the gain comes with the names and is not shown to come from the text.

Q1. "... can you report one problem that was never seen while tuning the engine's constants, run with the frozen configuration?"

Yes, six: four chip designs beyond gcd (same flow parameters) and two design problems of another domain with their own parameters. On the car side impact and the truss the frozen method is significantly ahead of NSGA-II on both measures (2.5x, +17% and 2.3x, +0.5%).

A rule fixed before any run chose them: the engineering design problems of pymoo's registry, less a development pilot (10 of 12 registered tests significant, all for AgentEvolve, Appendix H). The model's gain is significant on the truss (2.1x, +0.03%, a textbook problem: recall not excluded) and not separated on the car side impact (+0.2%, 8-8), where AgentEvolve without LLM has most of the lead (an estimate).

W1. "The no-LLM ablation already shows the engine carries most of the gain"

Yes, and both parts are measured: without the model the method keeps most of its lead over NSGA-II in final hypervolume (median share over seeds 57% to 106%), and the model adds a significant gain on 4 of 9 benchmarks and on the unpublished designs.

Post hoc (Appendix J). In evaluations to target the share without the model is 70% on Nangate45 and above half on both circuits, and 35% and 35% on NB-301 and the knapsack, where the model has most of the lead. On the two benchmarks you name the model's factor is 1.5x of 3.2x and 2.7x of 4.4x, both significant.

Q4. "What are NSGA-II's population and offspring settings ... a comparison to a standard surrogate-assisted EA would settle which."

The baseline is not under-configured: against the best of eight NSGA-II configurations in hindsight AgentEvolve without LLM stays ahead on five of six benchmarks (estimates). A standard surrogate-assisted EA ends behind it on Nangate45 (an estimate) and NB-301 and is significantly ahead on XGBoost in one measure each.

Population and offspring: 12 on the chip flows, untuned. AgentEvolve without LLM, 16 seeds (Appendix I):

  • Against the best NSGA-II configuration in hindsight: ahead on Nangate45 (2.0x, +13%) and four of five others, behind on XGB-1489 (-4.9%), as estimates.
  • Against the standard surrogate-assisted EA: ahead on Nangate45 (1.8x, +24%, an estimate) and NB-301 (+35%, significant), behind on XGB-1489 (0.71x) and XGB-40981 (-2.4%), both significant.

Q5. "For LLM-as-optimizer, how many model calls and candidates per call were allowed ...?"

Model calls are not limited and the evaluation budget is the same, with 10 or 22 designs per call after the first: the LLM as optimizer makes 17 calls per run on Nangate45 (AgentEvolve: 7) and stops 1 to 4 evaluations early in 33 of its 80 chip-flow runs. (Appendix I)

Reviewer_6iUEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AgentEvolve, a framework for expensive multi-objective black-box optimization that combines LLM-generated initial solutions, candidate proposals, and sampling programs with evolutionary search. It uses surrogate filtering and discounted Thompson sampling to allocate evaluations based on each source’s hypervolume gains. Experiments cover nine benchmarks across five domains, showing improvements on several tasks, although the gains over the same framework without LLMs are mixed.

Critical strengths
  • The framework offers a practical way to incorporate LLM suggestions into evolutionary search, with measured optimization progress guiding how much evaluation budget each source receives.
  • Improvements on Nangate45 and NAS-Bench-301 persist against the no-LLM version, providing evidence that the LLM contributes beyond the underlying search engine. The reporting of negative results and wall-clock costs also helps assess when the method is useful.
Critical weaknesses
  • The method combines established components, but the current ablations do not isolate the benefit of its adaptive allocation. Comparisons with fixed source proportions and an LLM warm-start-only variant are needed to establish what the proposed coordination adds.
  • The engineering evidence is limited: all three OpenROAD tasks use the same small gcd design, and LDO lacks a strong Bayesian optimization baseline. Direct comparisons with closely related LLM-assisted evolutionary methods are also missing.
Questions for authors
  • With the same initialization, candidate sources, and evaluation budget, how does adaptive allocation compare with fixed source proportions?
  • How does the full method compare with using the LLM only for initialization and running the classical search engine afterward?
  • Can you provide a comparison with a closely related LLM-assisted evolutionary method under a matched evaluation budget?
  • How does AgentEvolve compare with qNEHVI on LDO, where the results appear promising?
1 comment

W1, Q1. "... how does adaptive allocation compare with fixed source proportions?"

We ran it. With a correct description the adaptive rule and fixed uniform shares are not separated. However, with a "misleading problem description" on the problem card, the rule takes the later budget from the model and ends significantly ahead on the knapsack. The paper's claim is the whole method.

The adaptive rule against fixed uniform shares, all else equal (Section 6, Appendices E, G, marks: comment to all reviewers):

Benchmark (paired seeds)Adaptive rule vs fixed uniform shares
Nangate45 (16)1.1x (9-5) · -0.1% (7-9)
NB-301 (16)0.98x (7-8) · +0.8% (9-7)
Knapsack (16)1.0x (7-0) · 0.0% (7-9)
LDO (16)1.5x (12-4) · +15% (12-4)
XGB-40981 (8)0.91x (1-5) · -3.0% (2-6)
SKY130HD (8)0.82x (0-8)* · -6.2% (2-6)

No test separates the two (one estimate on SKY130HD favours fixed shares in evaluations), nor the rule from a fixed mix tuned in hindsight (8 seeds, -20% on the LDO to +0.9% on NB-301, Appendix E). Under a misleading description (labels permuted) the rule ends significantly ahead on the knapsack (+1.7%, 14-2, a family registered after some of its seeds were read), not on NB-301 or Nangate45. The whole method, in final hypervolume, is significantly ahead of AgentEvolve without LLM on 4 of 9 benchmarks (behind on 1), of the LLM as optimizer on 4 of 7 and of an LLM-initialized NSGA-II on 3 of 6.

W1, Q2. "How does the full method compare with using the LLM only for initialization and running the classical search engine afterward?"

Against the LLM used only for initialization with NSGA-II afterwards, the full method ends significantly higher on three of six benchmarks (+6.1% to +14%) and is not separated on three. With AgentEvolve's search afterwards, the later model sources add +2.3% and +1.9% on two benchmarks, significantly.

Both start from the full method's initial population and call no model afterwards (Section 6, Appendix F):

Benchmark (paired seeds)vs initial population + NSGA-IIvs initial population + AgentEvolve without model sources
Knapsack (16)1.4x (8-0) · +7.1% (16-0)†1.1x (8-0) · +2.3% (15-1)†
NB-301 (16)3.2x (15-1)† · +14% (16-0)†0.97x (7-9) · +1.6% (9-7)
LDO (16)1.5x (13-2) · +16% (11-5)1.2x (10-6) · -2.9% (7-9)
Nangate45 (16)2.0x (13-2)† · +6.1% (16-0)†1.2x (13-2) · +1.9% (16-0)†
SKY130HD (8 to 16)1.3x (12-4) · +12% (13-3)0.78x (0-8)* · -5.4% (3-5)
XGB-40981 (16)1.0x (5-6) · -1.5% (7-9)0.93x (4-8)* · -4.6% (2-14)*

The first column changes the whole later search, the second only the later model sources, which as estimates cost 4.6% on XGB-40981 and evaluations on SKY130HD (0.78x). The first column's family was registered after eight of its sixteen seeds were read: its three significant tests of final hypervolume hold on the other eight.

W2. "all three OpenROAD tasks use the same small gcd design"

We added four chip designs beyond gcd (same flow parameters, two unpublished) and two design problems of another domain. AgentEvolve is ahead of NSGA-II on all six, by 2.3x to 3.2x in evaluations to target, significantly on pmac16, uart, aes, the truss and the car side impact.

No constant was changed (comment to all reviewers, Section 5, Appendix H). On pmac16 and uart AgentEvolve is ahead of qNEHVI and HEBO, by 2.0x to 2.5x and +7.6% to +12% (at least 13 of 16 seeds, estimates). The model's gain is significant on all four chip designs (+2.3%, +5.2%, +4.1%, +3.5%) and on the truss (2.1x, +0.03%), not on the car side impact.

W2, Q4. "LDO lacks a strong Bayesian optimization baseline."

On the LDO AgentEvolve is significantly ahead of qNEHVI on both measures.

qNEHVI (BoTorch defaults), 16 seeds: 2.2x (16-0)† and +127% (16-0)†. It reaches the target in 1/16 runs, NSGA-II in 6/16 and AgentEvolve in 16/16 (Appendix I).

W2, Q3. "Can you provide a comparison with a closely related LLM-assisted evolutionary method under a matched evaluation budget?"

Against MOEA/D-LLM at the same evaluation budget, AgentEvolve ends higher in 23 of 24 paired runs on three benchmarks (estimates, 8 seeds each) with 16 to 33 times fewer model calls.

MOEA/D-LLM (Liu et al., 2023: the model is MOEA/D's reproduction operator), same model, description and budget (Appendix I):

Benchmark (paired seeds)AgentEvolve vs MOEA/D-LLM
Nangate45 (8)2.5x (8-0)* · +16% (8-0)*
NB-301 (8)2.3x (6-2) · +8.1% (7-1)*
LDO (8)3.2x (8-0)* · +109% (8-0)*

Median model calls per run: 7, 10, 11 against 110, 188, 364 for MOEA/D-LLM. Against the LLM as optimizer (16 seeds), AgentEvolve ends significantly higher on ASAP7, NB-301, XGB-40981 and the LDO. We reach the target significantly later on Nangate45 (0.42x).

Comments on the paper

2 comments

We thank the reviewers for their careful inspection and their attention to detail. Their reviews helped us uncover important insights around our method and helped us sharpen our claims.

  • We have conducted 5096 new runs under a pre-registration committed before the first of them, 16 validation seeds on all nine benchmarks, exact two-sided sign-flip tests, Holm correction within registered families.

  • In the table below, "3.2x (15-1)† · +27% (16-0)†" means that AgentEvolve reaches the target (a hypervolume level fixed in advance) in 3.2 times fewer evaluations (ahead in 15 seeds, behind in 1, ties not counted) and ends with 27% more hypervolume. † is a Holm-adjusted p below 0.05, indicating significance. * is an unadjusted p below 0.05. No mark means "not separated".

BenchmarkAgentEvolve vs NSGA-IIAgentEvolve vs AgentEvolve without LLM
Nangate453.2x (15-1)† · +27% (16-0)†1.5x (16-0)† · +6.1% (14-2)†
ASAP71.0x (8-0) · +19% (16-0)†1.0x (8-0) · +3.1% (15-1)†
SKY130HD1.1x (9-7) · +6.0% (10-6)0.61x (3-13) · -3.1% (3-13)
NB-3014.4x (16-0)† · +24% (16-0)†2.7x (16-0)† · +6.4% (16-0)†
XGB-14890.86x (1-10) · -2.2% (1-15)1.0x (3-2) · +3.0% (12-4)
XGB-409811.0x (4-5) · -1.3% (5-11)0.52x (1-12)† · -5.3% (2-14)†
Knapsack59x (16-0)† · +18% (16-0)†10x (16-0)† · +3.8% (16-0)†
Op-amp2.5x (15-1)† · +58% (14-2)†1.1x (9-7) · +7.2% (11-5)
LDO2.0x (15-1)† · +44% (16-0)†1.3x (13-3) · -2.7% (8-8)
Added chip designs
pmac16 (Nangate45)2.6x (15-0)† · +11% (16-0)†1.6x (14-1)† · +2.3% (13-3)†
crot16 (Nangate45)3.2x (15-0)* · +15% (15-1)*2.1x (12-3)† · +5.2% (13-3)†
Both designs pooled—1.7x (26-4)† · +3.8% (26-6)†
uart (ASAP7)2.3x (15-1)† · +12% (15-1)†1.6x (12-3)† · +4.1% (15-1)†
aes (ASAP7)3.2x (8-0)† · +19% (8-0)†1.7x (8-0)† · +3.5% (7-1)†
Added design problems of another domain
Two-bar truss2.3x (13-3)† · +0.5% (16-0)†2.1x (13-3)† · +0.03% (12-4)†
Car side impact2.5x (16-0)† · +17% (16-0)†1.2x (13-3) · +0.2% (8-8)

Rows have 16 paired seeds (the pooled row 32, aes 8). The comparison of crot16 with NSGA-II is in no registered family, hence its asterisks. pmac16 and crot16 are chip designs written for this study and published nowhere (uart and aes are public, as are the flow and its parameters). The two design problems have parameters of their own and were chosen by a rule fixed before any run.

Main findings:

  • Primary tests: 32 of 54 significant after Holm, 30 for AgentEvolve, which is significantly ahead of NSGA-II in final hypervolume on 6 of 9 benchmarks and behind it on none.
  • Six added problems, no constant changed: AgentEvolve is ahead of NSGA-II on all six, by 2.3x to 3.2x in evaluations to target, significantly on pmac16, uart, aes, the truss and the car side impact. The model's gain is significant on all four chip designs (+2.3% to +5.2%) and on the truss (2.1x in evaluations).
  • Not recall: the model's gain is significant on two chip designs published nowhere (+3.8%, 26-6, the registered test) and is not found once their names and text are blinded.
  • Requested baselines: significantly ahead of qNEHVI on the LDO (2.2x, +127%) and of the LLM used only for initialization on three of six benchmarks (+6.1% to +14%), above MOEA/D-LLM in 23 of 24 paired runs with 16 to 33 times fewer model calls (estimates).
  • Allocation: under a misleading problem description, the allocation rule takes the later budget from the model (86.5% to 15.0% on the knapsack) and ends significantly ahead of fixed shares there.

Limits (all in the paper): AgentEvolve does not lead on the two XGBoost benchmarks: the model costs significantly on XGB-40981 (-5.3%, not found with a quarter of the initial population from the model, at a cost of 2.5% on NB-301 as an estimate), the surrogate filter on XGB-1489 (-4.5%), and a standard surrogate-assisted EA ends ahead. qNEHVI is significantly ahead on the op-amp, where only measurements reveal the feasible region. The LLM as optimizer reaches the target significantly sooner on Nangate45 (0.42x, final hypervolume not separated). SKY130HD is not separated from NSGA-II, qNEHVI or HEBO, nor the allocation rule from fixed shares, UCB1 or FRRMAB in final hypervolume under a correct problem description. Recall is not excluded on the public NB-301, uart, aes and truss.

What changed in the paper: The method and the nine benchmarks are as submitted. Tests at 16 seeds replace the labels (Section 4). New: six problems outside the development set (Section 5, Table 2), controls of the problem description, allocation baselines, roles of the model and the surrogate filter (Section 6), assumptions of Equations 1 to 4 (Section 3). The submission's statements on allocation, wrong priors and a budget floor are replaced by what was measured.

Update: the revised paper is uploaded and every run is complete.

  • The section, appendix and table numbers in our comments reflect the rebuttal revision version. To make room for the new material, the integration example, the timing of the model advantage, wall-clock time, reasoning effort and the recorded model calls moved from the main text to Appendices A, B, K and L. The supplementary material opens with a list of every change.
  • The last run has ended (the eighth seed of aes). It completes the registered family of 18 tests on the added chip designs pmac16, uart and aes: all 18 are significant after Holm, all for AgentEvolve. aes at 8 seeds: 3.2x (8-0)† · +19% (8-0)† against NSGA-II and 1.7x (8-0)† · +3.5% (7-1)† against AgentEvolve without LLM.
  • AgentEvolve is now significantly ahead of NSGA-II on both measures on five of the six added problems, and the model's gain is significant on all four added chip designs (+2.3% to +5.2%).
  • Our four comments of 14 November are edited to match: the aes row, the marks of pmac16 and uart (now †), the sentences on the seed that was still running, and a few clarifications of wording.

No other result changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.