- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
I understand ARC-SC as a batch acquisition method for finding the first valid design above a target threshold. It combines marginal-score anchors with greedy coverage of sampled predictive scenarios, subject to a validity/support risk rule. The paper establishes submodularity for the fixed coverage objective and compares against marginal ranking on two simulated materials benchmarks.
Critical strengths
-
I find the first-valid-hit objective practically relevant when experiments are expensive and can fail. Preserving strong marginal candidates while diversifying the remaining batch is a reasonable idea.
-
I appreciate the paired evaluation across several failure mechanisms and the reporting of effect sizes and uncertainty. The superconductivity results provide evidence of improvement over the chosen baseline.
Critical weaknesses
-
I see a mismatch between the motivation and the implemented scenarios. The method aims to avoid candidates sharing the same uncertain predictive explanation, but Eq. (6) samples candidates independently. Appendix A.1.5 confirms that every experiment discards cross-candidate posterior covariance. I therefore do not see how these scenarios reliably identify the predictive redundancy motivating the method. Submodularity alone does not establish this connection, and the greedy guarantee does not cover the implemented dynamic risk rule and fallback.
-
The experiments do not isolate the proposed contribution. POF is the only baseline and lacks ARC-SC’s support-based risk filter, so the gains cannot be attributed specifically to scenario coverage. The anchor ablation omits the key control: selecting all five candidates by marginal score under the same risk rule. Comparisons with established batch-diversification approaches, such as local penalization or DPP-based selection, are also missing. Evidence is further limited to two reduced-dimensional frozen-oracle simulations, with inconclusive first-hit gains on JARVIS.
Questions for authors
-
How do independent marginal scenarios capture shared predictive explanations between candidates? Can you demonstrate this on a controlled example with known posterior correlations?
-
Does ARC-SC still improve first-hit discovery over marginal selection with the identical risk/support rule, and over a standard batch-diversification baseline adapted to the same valid-hit objective?
-
Some JARVIS initial datasets already contain valid target hits (Appendix A.5.1). How often does this occur, and what are the results when initialization contains no valid target hits?
3 comments
Response to Reviewer 59i5 - part 2
b) We agree that comparing only against ordinary POF could not attribute the improvement to scenario coverage, because POF lacks ARC-SC's additional support-aware risk selection. We therefore implemented precisely the requested Risk-POF control, using for all five slots with the identical risk rule and fallback. Over 160 clean-stream paired superconductivity cases, ARC-SC versus Risk-POF achieved 16/134/10 wins/ties/losses and −0.100 first-hit rounds [−0.250, 0.038]. Thus, the incremental first-hit effect of the completion module remains unresolved, despite a secondary exploratory gain of 0.338 cumulative valid hits per run [0.088, 0.613].
To isolate the putative overlap mechanism more tightly, we also replaced the last two slots with scale-matched factorized-mean completion, retaining the shared uncovered-response multiplier in the fallback desirability scale but removing the empirical covariance term. This produced 8/146/6 wins/ties/losses and −0.019 first-hit rounds [−0.119, 0.081] for ARC-SC versus the factorized control. Direct soft-marginal ranking yielded the same first-hit comparison, while its difference from Risk-POF was −0.081 [−0.206, 0.031]. These results do not establish a first-hit benefit for the overlap term or for soft rescoring separately. We will not attribute the complete policy's improvement over POF to a mechanism that these controls have not isolated.
We additionally adapted standard diversification methods to the same objective and selection setting. With identical first-three marginal anchors, support-aware risk, candidate pools, and budget, a quality-weighted greedy DPP used determinant gains from a descriptor-space RBF kernel for the last two slots. On 80 matched superconductivity cases, ARC-SC versus this DPP obtained 9/69/2 wins/ties/losses and −0.175 rounds [−0.338, −0.025]. The analogous three-anchor distance penalty gave −0.363 [−0.575, −0.175], whereas applying distance penalization from the second slot gave an inconclusive −0.150 [−0.425, 0.125]. DPP length-scale sensitivity also weakened the certainty of the comparison at alternative scales. We conclude only that ARC-SC is competitive with and sometimes earlier than these matched adaptations in the superconductivity setting—not that it universally outperforms established diversification methods.
The JARVIS transfer check is an important counterexample to broad attribution. In 40 shared-uniform, no-initial-hit cases, Risk-POF was 0.275 rounds later than ordinary POF [0.075, 0.525]; ARC-SC versus Risk-POF was −0.375 [−0.800, 0.025]. Full-risk fallback occurred at roughly three quarters of positions, and changing to validity-only risk eliminated fallback with little change to observed first-hit timing. The original support-aware design can therefore become restrictive without reliably changing decisions in these candidate pools. We will report this behavior and avoid generalizing the superconductivity risk attribution to JARVIS.
In short, the new controls substantiate the overall superconductivity first-hit result and identify the operating-risk rule as a substantial source of the observed improvement at the original threshold. They also support your criticism that scenario completion's independent first-hit contribution, and its relationship to real posterior redundancy, remain unproven. We believe the more precise characterization is important for assessing the contribution fairly.
c) Thank you for catching this ambiguity. In the original mixed-pool JARVIS study, 115/360 paired initializations (31.9%) already contained at least one valid target hit; all belonged to the stratum. Because the reported endpoint begins with the first newly queried batch, those cases measure discovery of an additional valid hit rather than the literal first-ever discovery. We agree that this qualification should have been prominent in the manuscript.
Among the 245 initializations with no valid target hit, ARC-SC versus POF yielded 33/185/27 wins/ties/losses and a mean capped first-hit difference of −0.106 rounds. By round five, 97 ARC-SC and 90 POF runs had found a hit, but ARC-SC had 19 fewer cumulative valid hits across these runs. This is a post hoc descriptive subgroup with mixed endpoint evidence, not a new confirmatory finding. We will report it transparently beside the full JARVIS result.
For further protocol clarity, the original uniform-pool comparisons use the same pool at corresponding paired rounds. In the two trust-region settings, each method generates its next pool from its own observed trajectory under the same rule; those comparisons evaluate complete closed-loop policies, not two acquisition functions over identical pools. The new 40-case shared-uniform JARVIS study, with no initial target hits, was designed separately to examine matched-risk attribution. We will distinguish these protocols explicitly.
I appreciate the authors' responses. After reading the replies to my questions and responses to other reviewers' questions, I'd like to update my rating from 1 to 2. However, I still see the weakness of this paper outweight its strength. I still lean to reject.
Response to Reviewer 59i5 - part 1
Thank you for pointing out the gap between independent scenario construction and shared posterior explanations, and for identifying the particularly important missing matched-risk and established-diversification controls. We agree that these concerns bear directly on the interpretation of our methodological contribution, and we addressed them through an implementation audit and several new targeted experiments. Here are our responses to your questions:
a) You are correct that independent marginal predictive draws do not capture cross-candidate GP posterior dependence. The original motivation suggesting that they identify shared predictive explanations was too strong, and we will revise it. The implemented soft responses can nevertheless interact because each scenario applies a common pool-wise relaxed threshold. If Um denotes the response mass uncovered by the selected prefix and μx=rm(x), the exact empirical gain is
Umrm(x)=Umμx+CovM(Um,rm(x)).
This covariance is between constructed soft responses, not candidate predictions under the GP joint posterior. It includes shared-threshold and finite-sample effects. Our code audit confirms that the 10,000-candidate material pools used independent Gaussian marginal draws; the shared random-number generator does not create a joint posterior sample. We therefore agree that the implemented method cannot reliably identify shared posterior errors merely from those draws.
A simple Gaussian counterexample makes the missing information explicit. Let A be a fixed anchor; let A and B have standard-normal predictive scores with correlation ρ, and let independent C have marginal hit probability 0.40. With a zero threshold, the marginal hit probabilities of (A,B,C) are (0.50,0.50,0.40), but P(A∪B)=43−2πarcsin(ρ),P(A∪C)=0.70.
At ρ=0, choosing B is better (0.75 versus 0.70); at ρ=0.8, choosing C is better (0.70 versus approximately 0.602). Because the marginals are unchanged, independent marginal draws cannot represent this reversal. We accept this limitation rather than invoke submodularity as a justification for posterior-dependence modeling.
We then tested the potential value of the missing information separately. In controlled known-joint Gaussian experiments, joint-soft sampling increased the true first-batch valid-hit probability by 1.27 percentage points [0.87, 1.70] relative to independent-soft sampling at within-group correlation 0.8. In a five-round synthetic closed loop with 48 new fixtures and five paired repeats per fixture, however, the mean first-hit difference was only −0.042 rounds [−0.129, 0.046], and each arm found a valid target in 178/240 runs by round five. At zero correlation the two soft policies selected identical trajectories. The experiments use known covariance and universally admissible synthetic candidates; they demonstrate that dependence information can matter to a single batch but do not establish a multi-round first-hit gain or validate joint-posterior coverage in our material benchmarks.