- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes EKO, an LLM-based framework for expensive multi-objective optimization in which natural-language reasoning is treated as inheritable knowledge. Knowledge is defined as a record of the reasoning behind a candidate and its validation outcome. Three agents replace parts of the evolutionary loop: a Selector chooses parent knowledge and an exploit/explore directive, a Generator retrieves domain background, parent knowledge, and global knowledge to produce an offspring with an unvalidated reasoning record, and a Reflector compares the registered hypotheses with observed objective changes, updates global knowledge and the evolution state, and provides feedback for the next round. The protocol is intended to be independent of variable representation. Experiments use six continuous real-world multi-objective tasks with 50 real evaluations, comparing against NSGA-II, MOEA/D, EHVI, qLogEHVI, HOLLM, and LLAMBO; additional binary, categorical, and permutation tasks test cross-representation behavior.
Critical strengths
-
Treating an LLM’s reasoning as an inheritable, validated natural-language record, rather than as a one-shot prompt or an ever-growing context, gives the evolutionary loop a persistent object to select, retrieve, and revise.
-
Selector, Generator, and Reflector have distinct inputs, outputs, and responsibilities, and the loop closes with real evaluations rather than LLM self-critique. The Reflector’s deterministic validation of registered atomic hypotheses is a sensible way to prevent unvalidated LLM assertions from entering the global archive. The supplementary execution trace makes this concrete.
-
The paper does not only test continuous benchmarks; it changes the problem interface to binary, categorical, and permutation spaces without representation-specific numerical operators, and EKO remains best on the two binary and two permutation tasks.
Critical weaknesses
-
The comparison does not control the LLM-call budget, which is consequential because EKO invokes three agents per round while the LLM-based baselines may use different numbers of calls. All methods share the same real-evaluation budget, backbone, temperature, and token limit, but not the same number of LLM queries, tokens consumed, or wall-clock cost. Under a 50-evaluation budget, extra LLM calls can be a major advantage, especially for the Selector and Reflector steps. Without a matched LLM-call or token budget, the reported gains over HOLLM and LLAMBO are difficult to attribute to the knowledge-evolution mechanism rather than to more LLM computation. This is a central fairness issue.
-
The main comparison uses six continuous tasks with 50 real evaluations, and the search spaces are small to medium (3–7 variables in the continuous tasks; 5–52 in the non-continuous tasks). The paper itself notes that large-scale problems are unresolved. Several reported gains are small—0.4% on CrossBarrel_MO and 0.7% on Fullerene_MO—and the CrossBarrel result is not marked significant in Table 1. The categorical tasks are a further limitation: qLogEHVI is better on both NB201 and LNP3. A claim of broad superiority across representations is therefore not supported; the honest claim is narrower.
-
Each atomic hypothesis is validated by a binary sign comparison between the registered direction and the observed parent-relative change. This checks direction only, not magnitude, uncertainty, or whether the observed change is distinguishable from noise. A “consistent” record can be a coincidental sign match, and a “contradicted” record can reflect a noisy evaluation rather than a wrong mechanism. The audit in Figure 4 reports aggregate proportions of consistent and contradicted outcomes, but it does not relate knowledge quality to later performance, nor does it show that validated knowledge actually changes subsequent decisions in a beneficial way. Since the paper’s main selling point is evidence-linked, interpretable knowledge, this is a material gap.
-
Three-agent LLM evolutionary loops, LLM-guided parent selection, and LLM-generated candidates are close to existing work such as LLAMBO, HOLLM, MOLLEO, and LLEMA; the paper’s distinguishing claim is knowledge inheritance plus validation. That distinction is plausible but not yet sharply demonstrated against those methods, especially because the global-knowledge ablation has limited effect on most tasks. In addition, the main text does not report LLM calls per run, token consumption, latency, knowledge-archive size, retrieval/compression rules, or how prompt length is controlled as the archive grows. The supplementary material helps, but the main paper does not give enough cost information to judge whether EKO is practical under an expensive-optimization budget.
Questions for authors
-
Can you report the number of LLM calls, tokens consumed, and wall-clock time for EKO and each LLM-based baseline, and rerun the comparison under a matched LLM-call or token budget? This would directly test whether the gains come from the knowledge-evolution loop rather than from additional LLM computation.
-
Can you strengthen knowledge validation beyond binary sign agreement. For example, by repeating evaluations, using confidence intervals, or testing whether the observed change exceeds a noise threshold, and show a causal link between validated knowledge quality and subsequent search performance? The current audit reports aggregate consistency but not whether the archive improves decisions.
-
Can you evaluate EKO on more tasks, larger search spaces, and larger evaluation budgets, and explain why qLogEHVI outperforms EKO on the categorical tasks? The current evidence supports a narrow claim on small/medium tasks, not a general cross-representation advantage.
-
How is the global knowledge archive retrieved and compressed as it grows? What are the prompt-length and token-cost implications for longer runs, and does the LLM-backbone sensitivity result hold on tasks other than CrossBarrel_MO?