- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The work proposes an LLM-guided framework for materials discovery under a limited evaluation budget. The method uses a coordination of 4 search mechanisms: Open, which proposes new candidates; Element, which looks for substitution directions; Explore, looks for underexplored alternatives; and Refine, which looks for parent structures. The objective, as highlighted by the authors is to discover as many distinct feasible compositions as possible within a fixed evaluation budget.
The empirical results are good. C4 outperforms the reported LLM-based baselines, as well as MCTS and simulated annealing, across the 14 materials tasks. The ablations are also good and indicate that the different components are complementary and that adaptive coordination performs better than fixing the allocation and scoring parameters.
Overall, I think the problem is relevant and the results are interesting. However, I have two main concerns that currently make me lean towards rejection: I was not really able to find a methodological novelty to the work, there is a very broad literature which seems either missing, or just simply that the authors are unaware that this problem has been addressed by whole other communities, that of sequential search/optimisation literature, and I am not fully convinced that the paper establishes why an LLM is needed for the adaptive search-control decisions.
Critical strengths
- The problem is relevant. Searching a very large materials space under a limited evaluation budget is important.
- The empirical advantages over the reported baselines are good and over several tasks.
- The Open/Element/Explore/Refine decomposition is intuitive, and an ablation was done over them.
- The paper is well written and the framework is relatively easy to follow.
Critical weaknesses
- I find the methodological novelty and positioning relative to well known and established sequential search and optimisation methods unclear.
C4 combines several ideas that are individually quite well established: exploration based on visitation, adaptive allocation between different search operators, sharing information across related branches, broad generation, and local refinement. There is substantial literature around these ideas in bandits, evolutionary optimisation, adaptive operator selection, dynamic algorithm configuration, and more generally sequential optimisation.
The paper does cite UCB, MCTS, simulated annealing, and some work on adaptive operator selection. However, I was quite surprised by how little discussion there is of the much broader literature on Bayesian optimisation, active learning / experimental design, categorical and combinatorial optimisation, and related sample-efficient optimisation methods. Even some LLM-BO combinations.
I appreciate that the materials space here is discrete, structured, and can also be dynamically expanded through Open, so I agree that applying a standard GP-BO might not be straightforward. However, I do not think this makes the broader literature irrelevant. For example, one could be using an LLM or another generative model to construct or expand the candidate space, while using an uncertainty-aware optimisation method to decide which of these candidates should actually consume the evaluation budget.
Because of this, I still struggle somewhat to understand what the fundamentally new search principle in C4 is, beyond the particular decomposition into Open/Element/Explore/Refine and the way information is shared between them. I think the paper needs to position this considerably more carefully against the existing sequential optimisation literature and, in particular, explain what limitation of these approaches C4 is addressing that cannot be dealt with by existing methods.
- I am not fully convinced is why the LLM should be responsible for the adaptive search-control decisions. I see a motivation for using an LLM for candidate generation, particularly early in a search, as several LLM-BO papers have recently suggested that is a good idea.
I find the motivation much less clear for using the LLM to decide how the budget should be allocated between Open/Element/Explore/Refine, or to adapt the numerical branch-scoring weights. Once the problem has been reduced to allocating budget between four operators and adapting a small number of weights based on previous search statistics, this starts to look much more like an online algorithm-selection or meta-optimisation problem. There are several standard approaches for doing this using bandits, online learning, adaptive operator selection, or related methods.
The Fixed-C4 ablation is useful, but I think what it shows is that adaptive coordination is better than fixed coordination. It does not show that LLM-based adaptive coordination is better than algorithmic adaptive coordination. To me this distinction is quite important. The current results could equally be interpreted as showing that the C4 search architecture itself is strong, with the LLM being one possible mechanism for adapting its control parameters.
I think a particularly useful, and relatively limited, additional experiment would therefore be to keep exactly the same C4 architecture and the same LLM-based candidate-generation mechanism, but replace the LLM controller with a simple adaptive non-LLM controller that receives exactly the same search statistics. This would much more directly test whether the LLM is contributing something specific to the search-control problem.
Questions for authors
- What do the authors consider to be the closest existing sequential optimisation formulation to C4? In particular, why would structured/categorical Bayesian optimisation, Bayesian active learning, bandit methods, or adaptive operator-selection methods not address essentially the same problem, potentially together with an LLM or generative model for constructing the candidate space?
- Could the authors compare the LLM-based adaptation of the tool allocation and branch-scoring weights against an adaptive non-LLM controller that receives the same observations? As far as I can tell, the current Fixed-C4 comparison demonstrates the benefit of adaptation, but not the benefit of LLM-based adaptation specifically.
- More generally, can the authors disentangle how much of the improvement comes from the chemical/semantic prior of the LLM, and how much comes from the search architecture and adaptive-control mechanism itself?