- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents AgentEvolve, a drop‑in LLM‑guided multi‑objective black‑box optimizer for costly engineering design problems including chip tuning, analog circuit sizing, architecture search and hyperparameter tuning. Rather than fixing the LLM’s functional role, AgentEvolve treats the LLM as one candidate source alongside classical evolutionary operators. Candidates from diverse sources are pooled, filtered by a surrogate model, and assigned evaluation budget via discounted Thompson sampling governed by per‑source hypervolume improvement credit. A Semantic Adapter resolves objective‑direction mismatches between the optimization engine and LLM prompts. The authors adopt a strict validation protocol with separated development and validation seeds and pre‑defined comparison rules. Across nine benchmarks, AgentEvolve decisively outperforms NSGA‑II on five tasks. It benefits from descriptive problem information that helps the LLM infer promising designs but fails on tasks with hidden narrow feasible regions or low‑quality LLM initial populations that subsequent optimization cannot recover.
Critical strengths
- Carefully controlled validation protocol for LLM‑based optimization. Separating development and validation seeds, locking evaluation targets in advance, and deploying targeted ablations (AgentEvolve without LLM, LLM‑as‑optimizer baseline) avoids post‑hoc result selection, a common flaw in related literature. This increases trust in empirical conclusions and offers a usable template for future work in the field.
- System design that treats LLM outputs as competing candidate sources. Budget share for LLM‑generated proposals is determined empirically by hypervolume gains instead of static rules. Its drop‑in compatibility with existing evolutionary tooling improves practical applicability for expensive real‑world black‑box design workflows.
Critical weaknesses
- Inadequate benchmarking against prior adaptive operator selection (AOS) methods. The proposed credit‑driven budget allocation builds on established bandit‑enabled AOS techniques. Direct comparisons against AOS, UCB and fixed‑allocation baselines are missing. It remains difficult to disentangle performance gains brought by the new allocation mechanism from gains stemming solely from high‑quality LLM proposals, blurring the paper’s methodological novelty.
- Under‑specified mathematical formulations and incomplete diagnosis of failure modes. Key equations carry unstated statistical assumptions, including risks of hypervolume improvement double‑counting and ambiguous probabilistic interpretations. For benchmarks degraded by poor LLM initial populations, there are no targeted ablations to test whether the credit allocation can mitigate incorrect LLM priors. These issues weaken theoretical foundations and understanding of observed empirical failures.
Questions for authors
- Using identical candidate source pools, can you supply comparative results against established AOS, UCB and fixed‑allocation baselines to isolate the performance contribution of your discounted‑Thompson‑sampling credit allocation?
- What statistical assumptions underpin multiplying expected hypervolume improvement by feasibility probability in Equation (1)? How does Equation (2) prevent double‑counting hypervolume gains from overlapping dominated regions across sources, and how is zero‑normalization implemented for credit computation?
- For cases with poor LLM‑generated initial populations such as SKY130HD and XGB‑40981, is your credit allocation mechanism intended to compensate for inaccurate LLM priors in theory and practice, and why do performance gaps persist in later generations?
- What formal probabilistic justification exists for using continuous credit rates as fractional pseudo‑counts in the Beta posterior updates of Equation (3)? How are the multiplicative advice weights and clipping bounds in Equation (4) justified without supporting sensitivity analysis?
All are run on identical source pools. With a correct description on the problem card, no test separates our rule from fixed shares, UCB1 or FRRMAB in final hypervolume. However, under a misleading problem description, ours is the only rule with a significant lead over fixed shares at these seeds (knapsack). We claim the method as a whole.
Change in final hypervolume of AgentEvolve against each rule: 16 seeds for fixed uniform shares (8 on XGB-40981), 8 for the others (marks: comment to all reviewers):
A difference of several percent is not excluded on the LDO, NB-301 and XGB-40981. On SKY130HD fixed uniform shares reach the target sooner (0.82x, 0-8, an estimate). The paper places the rule within adaptive operator selection (Sections 2 and 6, Appendix E).
Yes for the later budget: under a wrong description the rule takes it from the model sources and ends significantly ahead of fixed shares on the knapsack. What a poor initial population cost is already spent, and that gap closes in part. (Section 6, Appendices D to G)
Section 3 states what each equation assumes. Equation 2 allows double counting: in the recorded runs the median overcount is 0% to 4%. (Appendix D)
Equation 3 is an expected update, not an exact posterior. Sensitivity analyses of Equations 3 and 4 are run, and no test separates the variants (one estimate on the LDO favours Equation 4's advice). (Appendix E)