- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 3(+2)
Read review
Paper summary
This paper looks at exploratory landscape analysis (ELA) for black box optimization when using LLM-driven meta-black-box optimizers. It proposes AdaELA, a method to adaptively select ELA indicators for use by LLM meta-optimizers. It claims to see the learned settings for AdaELA generalize to out of distribution optimization tasks.
The paper suffers from lack of readability, general vague terminology and lack of concrete descriptions for its methodology.
Critical strengths
The paper does substantive analysis of exactly what role the ELA plays in in the ELA + LLM metaBBO, extracting exactly what benefit it provides.
Critical weaknesses
The paper generally reads as unclear, and is generally difficult to understand what exactly is being tested and how. I find it vague, dense, and confusing to read.
I find it difficult to ascertain how LLMs are being used in the BBO problems. There is no indication of what a prompt to this LLM would look like, how it would respond, what the harness looks like, what tools it may or may not have access too. The only mention of a prompt is in the appendix and it is vague. Additionally, given this is black box optimization, it is missing one comparison I think would be necessary for this type of analysis, which is simply providing codex or claude code with the optimization problem and measuring the effectiveness with the /goal command.
Revisions
See below comment for my revised interpretation of the paper.
Questions for authors
What is the role of the LLM in the BBO setups? What does the harness look like? What tool calls or methods does the LLM have for interacting with this system? In figure 1, where does the LLM fit in? What do LLM rollouts look like? What is the information passed to the MetaBBO?
3 comments
I would like to apologize to the authors on an earlier reading placing too much focus on the lack of details of the BBO methodology, given the focus of this paper on the ELA. Not understanding how the LMM-based MetaBBO worked and not seeing an explanation within the paper was functioning led to confusion in my reading of the paper - but I see there is already existing literature on these. I have followed through the cited LLM-based MetaBBO papers, and while this paper does not need to have an extensive description of their setup, I do believe it still requires more description than is given now, in the appendix - and would like to see example LLM interactions (prompt/ responses). I also think possibly a figure and/or prompt/response in the related works section of how the LLM-based MetaBBO works could be helpful, though my score does not depend on this.
I was initially concerned that results would struggle to generalize to MetaBBO's with newer models. For example, autoEP runs with Qwen-3-30B and gpt-4, these models are well behind the frontier. Newer models tend to be able to take advantage of more information - I had not caught the first time that late in the appendix there is a chart covering this with newer models. This was an immediate concern on reading the paper, and either moving that section up, or providing a small table with commentary on this in the main paper would be helpful.
I am revising my score to a 3.
We sincerely thank you for your recognition of our work and for taking the time to consider our responses.
We wish you all the best!
We sincerely apologize that our work has caused confusion. We have responded to your questions in detail and provided further explanations of our method. We sincerely hope you will reconsider our work in light of our response.
Clarification: What AdaELA Does
We would first like to clarify the role of AdaELA.
How LLMs Are Used in MetaBBO
Different optimization problems and stages may require different search strategies. MetaBBO uses a meta-policy to observe the search process and guide the optimizer accordingly[1,2,3]. In LLM-based MetaBBO, the LLM serves as this meta-policy. To make the process easier to understand, we provide an illustration in https://anonymous.4open.science/r/ICLR2027-EBB5/LLM-based%20MetaBBO.pdf.
Comparison with Codex or Claude Code
Summary
We apologize again for the lack of clarity and sincerely hope you will reconsider our paper in light of our response. We wish you all the best!