- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies active search with pool-dependent utility, where querying a candidate reveals a local observation while its terminal reward depends on the latent states of the entire candidate pool. The authors formulate this problem and propose Delta-Hull, a greedy acquisition rule that selects candidates according to their posterior probability of belonging to the completed convex hull. They further analyze the value of nonmyopic planning through an exact two-step decomposition, a worst-case counterexample for greedy search, and a bound under weak posterior movement. Experiments on 230 MatPES chemical systems show that aligning the acquisition objective with complete-pool stability substantially improves terminal utility over history-visible hull selection, while more sophisticated scoring and two-step planning provide relatively limited additional benefits.
Critical strengths
-
The paper highlights a meaningful distinction between query feedback and terminal reward in active search, particularly for materials stability screening.
-
The proposed formulation provides a clear decision-theoretic perspective on pool-dependent utility and the potential value of nonmyopic planning.
-
The experiments use controlled comparisons to separate the effects of terminal-event definition, acquisition objective, scoring rule, and planning depth.
-
The evaluation includes multiple chemical systems, posterior calibration diagnostics, paired statistical analyses, and additional planning experiments, providing useful empirical insights.
Critical weaknesses
1. Limited methodological and theoretical novelty.
The central methodological contribution appears to be a relatively straightforward application of existing Bayesian decision-theoretic principles to a particular utility definition. The one-step Delta-Hull acquisition rule simply selects the candidate with the highest posterior probability of terminal success, which follows directly from the linearity of expectation under additive utility and equal query costs. Moreover, probabilistic convex-hull modeling and posterior hull-membership estimation have already been investigated in CAL [R1], while nonmyopic active search and observation-dependent decision value are closely related to ENS [R2] and classical Bayesian active search [R3]. The two-step gain decomposition largely reformulates a standard Bellman-style decision problem, and several theoretical results follow from established probabilistic arguments. Although explicitly distinguishing pool-dependent reward from query feedback is conceptually meaningful, the paper does not sufficiently establish what fundamentally new methodological or theoretical capability is obtained beyond these existing frameworks.
2. The largest empirical improvements arise from an objective-mismatched baseline.
The most prominent result is the substantial improvement of Delta-Hull over Local-Hull Probability. However, Local-Hull estimates membership in the history-visible hull, whereas the evaluation metric directly rewards membership in the completed candidate-pool hull. Since Delta-Hull explicitly optimizes the latter event, its advantage is largely expected and primarily demonstrates the importance of objective alignment rather than the effectiveness of a novel acquisition mechanism. This concern becomes more apparent when considering stronger alternatives. At B = 6, Delta-Hull improves terminal utility over Target Margin by only 0.043 confirmations per system, while Complete-Pool Mean Margin achieves exactly the same mean terminal utility (3.809). Their budget-curve areas are also statistically indistinguishable. Although the large Local-Hull contrast is a valid demonstration of objective mismatch, the limited advantage over simpler terminal-aligned scores weakens the practical significance of the proposed acquisition rule.
3. The theoretical analysis provides limited new insight into when nonmyopic planning is beneficial.
The proposed two-step decomposition is closely related to established value-of-information formulations in Bayesian active search [R2, R3]. More importantly, Proposition 2 bounds the suboptimality of greedy search under a strong uniform weak-posterior-movement assumption over all reachable observation continuations. The empirical analysis, however, measures only average posterior movement along realized trajectories, which does not establish the required uniform condition. Indeed, Appendix A.12 reports a weak association between observed-path posterior movement and realized lookahead gains, limiting the diagnostic value of this quantity. Consequently, the theoretical result does not provide a practically validated criterion for determining when nonmyopic planning is worthwhile. The paper would benefit from a clearer distinction between general decision-theoretic observations, sufficient theoretical conditions, and empirically testable predictions.
4. The conclusions about the limited value of lookahead are not fully supported.
The paper reports that an anchored two-step rollout changes actions frequently but provides only a small terminal-utility improvement over repeated Delta-Hull, despite substantially higher computational cost. While this is an interesting observation, the evaluation uses a finite-Monte-Carlo approximation rather than an exact nonmyopic policy. Appendix A.10 shows that the selected action is sensitive to the number of posterior samples, with only 64% agreement between the 128- and 256-sample configurations in the evaluated states. Moreover, the analysis is limited to a two-step planning horizon and query budgets of at most six. Therefore, the small observed improvement may reflect limitations of the approximate planner rather than an inherently low value of information. Comparisons with exact or substantially more accurate lookahead on smaller candidate pools would help establish whether the empirical conclusions reflect the underlying decision problem.
5. Limited generality of the problem formulation and experimental validation.
Although the paper introduces pool-dependent active search as a general framework, the empirical evaluation is restricted to materials stability screening using the MatPES dataset. The experiments assume finite closed candidate pools, equal query costs, additive terminal utility, and a specific PBE-to-r2SCAN posterior model. The secondary evaluation panel is drawn from the same underlying dataset and does not establish generalization to other forms of pool-dependent utility. Furthermore, the convex-hull evaluation includes elemental references but excludes lower-order compound competitors, meaning that the terminal stability labels are defined relative to the constructed candidate pools rather than a more comprehensive thermodynamic phase diagram. These choices are consistent with the stated benchmark, but they limit the broader materials-discovery implications. Evaluation on a different pool-dependent search problem or under alternative posterior and candidate-pool configurations would strengthen the claimed generality.
6. Limited reproducibility due to the absence of source code.
Despite providing an extensive experimental appendix, the submission does not include source code. This is particularly relevant because the reported results depend on several implementation-sensitive components, including posterior conditioning, covariance construction, convex-hull membership calculations, duplicate-composition handling, and Monte Carlo lookahead estimation. The paper also refers to a reproducibility artifact for additional implementation details and random seeds. Without access to the implementation, independently reproducing the matched acquisition baselines and verifying the reported planning behavior is difficult. Given the small performance differences among the stronger terminal-aligned methods and the demonstrated sensitivity of rollout decisions to Monte Carlo sampling, reproducibility is especially important for assessing the empirical conclusions.
[References]
[R1] "Probabilistic Prediction of Material Stability: Integrating Convex Hulls into Active Learning," Materials Horizons, 2024.
[R2] "Efficient Nonmyopic Active Search," ICML, 2017.
[R3] "Bayesian Optimal Active Search and Surveying," ICML, 2012.
[R4] "Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information," ICML, 2021.
I also have some minor concerns as follows:
-
There is a mathematical inconsistency in the proof of Proposition 3 (Appendix A.14, p. 24). Given reference points r_i = (i, 10i^2), the local reference chord should contain 10(2v-1)t rather than (2v-1)t. The stated formula is inconsistent with the subsequent hull-geometry identity. This appears potentially correctable, but the proof should be checked and revised accordingly.
-
The reachability condition in Proposition 2 is phrased using observation continuations with positive posterior probability, which requires clarification for continuously distributed energy observations.
-
Posterior calibration is considerably worse for the selected top-ranked candidates than for the full candidate set, suggesting that decision-relevant uncertainty estimates deserve further analysis.
-
The evaluated CAL- and BAX-inspired baselines optimize reconstruction-oriented objectives rather than the selected-item terminal utility. Their lower performance should not be interpreted as general superiority over the original methods on their intended tasks.
-
The reported 46.9x planning-runtime increase excludes the target-query cost and is specific to the evaluated implementations. It should not be interpreted as the end-to-end computational cost of materials discovery.
-
The experiments cover relatively small candidate pools and short query budgets, limiting conclusions about computational scalability and long-horizon search.
-
The relationship between posterior calibration quality and actual acquisition performance is not sufficiently investigated, particularly for high-probability candidates that dominate greedy decisions.
Questions for authors
Please refer to the Weaknesses section for detailed explanations of the concerns below.
-
What is the fundamental methodological or theoretical contribution beyond existing posterior convex-hull membership estimation [R1] and Bayesian nonmyopic active search [R2, R3]?
-
Given that Complete-Pool Mean Margin achieves essentially the same terminal utility as Delta-Hull, what practical advantage does the proposed posterior-probability acquisition rule provide over simpler terminal-aligned alternatives?
-
How can the weak-posterior-movement condition in Proposition 2 be meaningfully assessed in practice, given that the observed-path statistics do not establish the required uniform bound?
-
Can the authors distinguish the inherently limited value of nonmyopic planning from the approximation error of the finite-Monte-Carlo rollout, particularly on small problems where more accurate planning is feasible?
-
Can the authors correct the convex-hull chord equation in the proof of Proposition 3 and confirm that the subsequent construction and guarantee remain valid?
-
To what extent do the conclusions generalize beyond the MatPES-specific posterior, finite-pool convex-hull utility, and short query horizons?
Thank you for the thorough rebuttal and the substantial effort devoted to revising the paper. I also apologize for providing my follow-up comments so close to the end of the discussion period. Given the extensive additional experiments and theoretical analyses, I needed some time to carefully examine the revisions and reconsider my assessment.
I appreciate that the authors have addressed many of my technical and empirical concerns. In particular, the corrected proof of Proposition 3, the clarification of Proposition 2, the exact-planning and Monte Carlo analyses, and the additional Airfoil and reference-sensitivity experiments significantly strengthen the paper. These additions have largely resolved my concerns regarding technical correctness, planning evaluation, and experimental generality.
However, my main concerns regarding methodological novelty and the significance of the contribution remain only partially addressed.
First, while the distinction between query feedback and pool-dependent terminal utility is meaningful, the proposed acquisition rule still appears to be a relatively straightforward application of existing Bayesian decision-theoretic principles. The rebuttal clarifies the differences from CAL, ENS, and BAX, but I remain unconvinced that the formulation leads to sufficiently novel methodological or theoretical insights beyond defining and optimizing the appropriate downstream utility. The additional theoretical analyses validate the proposed results but do not fundamentally change this assessment.
Second, the empirical results continue to raise questions about the practical significance of Delta-Hull itself. In particular, Complete-Pool Mean Margin achieves essentially identical terminal utility on MatPES, and the new Airfoil experiments similarly show near parity between membership-probability selection and posterior-mean ranking. These findings support the importance of objective alignment, but they also suggest that relatively simple terminal-aligned strategies can achieve comparable results. Moreover, the substantial improvements over Local-Hull primarily reflect the difference in the events being optimized, rather than a clear advantage of the proposed acquisition mechanism.
Overall, I acknowledge that the revised manuscript is substantially stronger and that many of my original concerns have been carefully addressed. Nevertheless, I still find the methodological and theoretical novelty, as well as the incremental practical contribution beyond simpler terminal-aligned approaches, somewhat limited for ICLR.
I sincerely appreciate the authors' thoughtful responses and the considerable effort invested in strengthening the paper.