- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces three new Bayesian optimization methods designed to perform competitively on datasets where use of global GP posterior acquisition is problematic. The performance of these and several other Bayesian optimization methods is explored on a range of real-world and synthetic black-box optimization problems. The results show competitive performance of the newly proposed ATLAS/SAGE methods on one of the datasets outperforming 8 out of 10 alternative methods. The analysis provides useful insights into when alternatives to GP posterior acquisition, such as the approaches adopted by the new methods, can be advantageous. A key insight is that global posterior acquisition is appropriate on smooth calibrated landscapes but methods exploiting locality, dissonance and diversity should be used when those modeling assumptions are violated. The paper also identifies deficiencies in some of the public datasets, calling into question their usefulness for evaluating Bayesian optimization techniques,
Critical strengths
The paper is clearly written. The experiments cover an extensive range of methods and datasets with much additional experimental detail in the appendices and provide some useful insights. The designs of the new methods seem heuristically reasonable. An objective and honest account of the performance of the newly introduced methods is provided.
Critical weaknesses
The paper is addressing two things at once:
(a) introducing, explaining and showcasing the new Bayesian optimization methods
(b) understanding how performance of these and other Bayesian optimization methodologies relate to data landscape and identifying deficiencies in some of the benchmark datasets.
Perhaps as a result of covering both aspects at once the central messages of the paper have become somewhat mixed-up/blurred
A more rigorous showcasing of the new methods would need to include: expanded justification of the construction and choice of components in the models e.g. through more extensive ablation studies, justifying parameter/weight choices together with sensitivity analysis etc. The reader would also expect the new methods to be shown competitive against others on several relevant datasets. The paper in its current form doesn't achieve this combination.
If a primary aim is a scientific understanding of how methodologies need to match dataset landscape then the simultaneous introduction of these new methods may be seen as more confusing than helpful. The methods are quite complex and as yet not fully understood making scientific diagnostic conclusions from them difficult.
I feel that the paper in its current form has failed to cover both (a) and (b) adequately (particularly (a)). At least in part this may be due to the challenge of attempting to cover both at once in a short conference paper.
Questions for authors
(1) Can you please clarify whether ProSe, SAGE and ATLAS are intended for diagnostic purposes or intended to be competitive optimization methods.
(2) If intended as competitive optimization methods can you provide some more detail on how you went about designing the new methods and to what extent they've yet been tuned or subjected to ablation studies.
4 comments
Thank you for responding to my review. I'm having difficulty seeing how my concerns are being addressed as you've not adopted the usual rebuttal convention of addressing each of my points individually and in turn. It will be challenging for me to look at the changed emphasis plus substantial paper-rewrite within the remaining time-limit but I will do my best. Meanwhile I maintain my score.
Point-by-point clarification
Thank you for explaining that our response was hard to follow. We apologize. Here are direct answers to your two questions and the related concerns:
-
Purpose. ProSe and ATLAS are proposed as search policies to be evaluated against established methods; SAGE is a separately parameterized reference. They are not presented as a diagnostic for GP failure. We now ask how these policies perform on target-region retrieval and best observed response on fixed retrospective emulators (Sections 1 and 3).
-
Design and ablation. The methods and hand-chosen weights are specified in Sections 3.2–3.4. Section 4.2 lists the fixed controls, and Section 5.2 reports their results: only two of ten ATLAS component/sensitivity comparisons support a benefit after correction. They do not establish that individual components are necessary. The weights were developed with historical tasks in view; broad tuning robustness is not established.
-
Competitiveness across datasets. On the two primary tasks, ATLAS’s observed retrieval was higher than MAP-Elites’, but neither difference was significant after multiple-comparison correction. ProSe retrieved neither target in 100 runs per task. On the exploratory band-gap emulator, the tested GP variants had higher target retrieval than ATLAS and MAP-Elites. These fixed retrospective tasks do not establish general superiority or performance on unseen materials (Sections 5 and 6).
We have also rewritten the conclusion in Section 8 to state the question, main findings, and limits more plainly. We appreciate your taking the time to review the revision.
I have looked quickly at the revised manuscript in an attempt to comment usefully before final discussion closure. At the moment I think I will at best be able to increase my rating to 2 (weak reject) as I feel more has to be done to justify the value of the new methods and to show how the parameters and combination of components were arrived at (as per my earlier comments on ablation etc). I will look some more after discussion-closure to see whether I can justify raising this to 2 or whether I retain my 1 marking.
Response to Reviewer dEmn
Revision note. We revised the title to “Recall-Oriented Search on Retrospective Materials Emulators” and updated the abstract to reflect the narrower claims and additional evidence. The original TL;DR no longer reflects the revised conclusions; please refer to the revised abstract and manuscript.
Thank you for asking us to clarify the contribution and its relationship to competitive optimization. We now ask a narrower empirical question: how do specified search policies behave on target-region retrieval and best observed response in fixed retrospective materials emulators? We do not claim a general GP-failure diagnostic, a regret guarantee, or universal superiority.
ProSe combines local value estimation with anti-clustering; SAGE is a separately parameterized anchored reference; ATLAS combines residual-guided proposals, coverage/dissonance scores and a model hierarchy. The revised methods specify the scoring, geometry, proposal allocation and transitions. These combine known ingredients, and their coefficients were hand-designed with historical tasks in view; we do not claim each ingredient is new or that broad tuning robustness has been demonstrated.
The quantitative limits are central to the revision: ProSe records 0/100 retrievals on each primary task, and ATLAS versus MAP-Elites is not significant after the primary Holm28 correction. Only two of ten specified ATLAS component/sensitivity contrasts support corrected benefits. The later joint residual-pathway deletion was selected after earlier results and changes several pathways at once; its results do not establish the necessity of any individual component. The available weight controls and post-hoc target/radius re-scoring do not establish general sensitivity robustness.
An additional exploratory band-gap emulator likewise gives ATLAS/MAP target hit rates of .36/.28 without a corrected significant difference, while the four tested GP variants achieve .88–.99. The mean best observed normalized responses are .887719 for ATLAS and .861510 for MAP; a corrected sign test favors ATLAS in direction, and the GP variants are also favored over both. The sign test does not test the mean difference itself. The previously studied subset includes the target points in the emulator support. We therefore report this counterevidence and restrict conclusions to the specified retrospective tasks, without claiming prospective discovery or generalization to unseen materials.