acceptodds
Under review as a conference paper at ICLR 2027

Anchored Scenario Coverage for Failure-Aware First-Hit Batch Inverse Design

Abstract

Early discovery of at least one valid design satisfying a target requirement is a central objective in failure-prone closed-loop inverse design. A natural batch strategy ranks candidate designs by a product-form marginal valid-hit score, but selecting the highest-ranked candidates independently within a batch can produce redundant recommendations under predictive uncertainty. We introduce ARC-SC (Anchored Risk-Constrained Scenario Coverage), which preserves strong marginal candidates as anchors and allocates the remaining batch positions by maximizing complementary coverage over predictive target scenarios under a risk-support constraint. For fixed anchors and a fixed admissible pool, the scenario-coverage objective is monotone submodular and admits the standard greedy-completion guarantee. In frozen-oracle closed-loop simulations, ARC-SC yields a statistically supported improvement in first-hit discovery, while remaining competitive with directionally favorable performance on higher-dimensional tasks. These results support scenario-aware batch diversification as a principled complement to strong marginal acquisition for early valid-target discovery under experimental failure.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
inverse design, uncertainty quantification, Bayesian optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_59i5OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

I understand ARC-SC as a batch acquisition method for finding the first valid design above a target threshold. It combines marginal-score anchors with greedy coverage of sampled predictive scenarios, subject to a validity/support risk rule. The paper establishes submodularity for the fixed coverage objective and compares against marginal ranking on two simulated materials benchmarks.

Critical strengths
  1. I find the first-valid-hit objective practically relevant when experiments are expensive and can fail. Preserving strong marginal candidates while diversifying the remaining batch is a reasonable idea.

  2. I appreciate the paired evaluation across several failure mechanisms and the reporting of effect sizes and uncertainty. The superconductivity results provide evidence of improvement over the chosen baseline.

Critical weaknesses
  1. I see a mismatch between the motivation and the implemented scenarios. The method aims to avoid candidates sharing the same uncertain predictive explanation, but Eq. (6) samples candidates independently. Appendix A.1.5 confirms that every experiment discards cross-candidate posterior covariance. I therefore do not see how these scenarios reliably identify the predictive redundancy motivating the method. Submodularity alone does not establish this connection, and the greedy guarantee does not cover the implemented dynamic risk rule and fallback.

  2. The experiments do not isolate the proposed contribution. POF is the only baseline and lacks ARC-SC’s support-based risk filter, so the gains cannot be attributed specifically to scenario coverage. The anchor ablation omits the key control: selecting all five candidates by marginal score under the same risk rule. Comparisons with established batch-diversification approaches, such as local penalization or DPP-based selection, are also missing. Evidence is further limited to two reduced-dimensional frozen-oracle simulations, with inconclusive first-hit gains on JARVIS.

Questions for authors
  1. How do independent marginal scenarios capture shared predictive explanations between candidates? Can you demonstrate this on a controlled example with known posterior correlations?

  2. Does ARC-SC still improve first-hit discovery over marginal selection with the identical risk/support rule, and over a standard batch-diversification baseline adapted to the same valid-hit objective?

  3. Some JARVIS initial datasets already contain valid target hits (Appendix A.5.1). How often does this occur, and what are the results when initialization contains no valid target hits?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 59i5 - part 1

Thank you for pointing out the gap between independent scenario construction and shared posterior explanations, and for identifying the particularly important missing matched-risk and established-diversification controls. We agree that these concerns bear directly on the interpretation of our methodological contribution, and we addressed them through an implementation audit and several new targeted experiments. Here are our responses to your questions:

a) You are correct that independent marginal predictive draws do not capture cross-candidate GP posterior dependence. The original motivation suggesting that they identify shared predictive explanations was too strong, and we will revise it. The implemented soft responses can nevertheless interact because each scenario applies a common pool-wise relaxed threshold. If UmU_m denotes the response mass uncovered by the selected prefix and μx=rm(x)‾\mu_x=\overline{r_m(x)}, the exact empirical gain is

Umrm(x)‾=Um‾μx+Cov^M(Um,rm(x)).\overline{U_mr_m(x)}=\overline{U_m}\mu_x+\widehat{\mathrm{Cov}}_M(U_m,r_m(x)).

This covariance is between constructed soft responses, not candidate predictions under the GP joint posterior. It includes shared-threshold and finite-sample effects. Our code audit confirms that the 10,000-candidate material pools used independent Gaussian marginal draws; the shared random-number generator does not create a joint posterior sample. We therefore agree that the implemented method cannot reliably identify shared posterior errors merely from those draws.

A simple Gaussian counterexample makes the missing information explicit. Let AA be a fixed anchor; let AA and BB have standard-normal predictive scores with correlation ρ\rho, and let independent CC have marginal hit probability 0.40. With a zero threshold, the marginal hit probabilities of (A,B,C)(A,B,C) are (0.50,0.50,0.40)(0.50,0.50,0.40), but P(A∪B)=34−arcsin⁡(ρ)2π,P(A∪C)=0.70.P(A\cup B)=\tfrac34-\frac{\arcsin(\rho)}{2\pi},\quad P(A\cup C)=0.70.

At ρ=0\rho=0, choosing BB is better (0.75 versus 0.70); at ρ=0.8\rho=0.8, choosing CC is better (0.70 versus approximately 0.602). Because the marginals are unchanged, independent marginal draws cannot represent this reversal. We accept this limitation rather than invoke submodularity as a justification for posterior-dependence modeling.

We then tested the potential value of the missing information separately. In controlled known-joint Gaussian experiments, joint-soft sampling increased the true first-batch valid-hit probability by 1.27 percentage points [0.87, 1.70] relative to independent-soft sampling at within-group correlation 0.80.8. In a five-round synthetic closed loop with 48 new fixtures and five paired repeats per fixture, however, the mean first-hit difference was only −0.042 rounds [−0.129, 0.046], and each arm found a valid target in 178/240 runs by round five. At zero correlation the two soft policies selected identical trajectories. The experiments use known covariance and universally admissible synthetic candidates; they demonstrate that dependence information can matter to a single batch but do not establish a multi-round first-hit gain or validate joint-posterior coverage in our material benchmarks.

AuthorsOpenReview ↗

Response to Reviewer 59i5 - part 2

b) We agree that comparing only against ordinary POF could not attribute the improvement to scenario coverage, because POF lacks ARC-SC's additional support-aware risk selection. We therefore implemented precisely the requested Risk-POF control, using h(x)=py(x)pv(x)h(x)=p_y(x)p_v(x) for all five slots with the identical risk rule and fallback. Over 160 clean-stream paired superconductivity cases, ARC-SC versus Risk-POF achieved 16/134/10 wins/ties/losses and −0.100 first-hit rounds [−0.250, 0.038]. Thus, the incremental first-hit effect of the completion module remains unresolved, despite a secondary exploratory gain of 0.338 cumulative valid hits per run [0.088, 0.613].

To isolate the putative overlap mechanism more tightly, we also replaced the last two slots with scale-matched factorized-mean completion, retaining the shared uncovered-response multiplier in the fallback desirability scale but removing the empirical covariance term. This produced 8/146/6 wins/ties/losses and −0.019 first-hit rounds [−0.119, 0.081] for ARC-SC versus the factorized control. Direct soft-marginal ranking yielded the same first-hit comparison, while its difference from Risk-POF was −0.081 [−0.206, 0.031]. These results do not establish a first-hit benefit for the overlap term or for soft rescoring separately. We will not attribute the complete policy's improvement over POF to a mechanism that these controls have not isolated.

We additionally adapted standard diversification methods to the same objective and selection setting. With identical first-three marginal anchors, support-aware risk, candidate pools, and budget, a quality-weighted greedy DPP used determinant gains from a descriptor-space RBF kernel for the last two slots. On 80 matched superconductivity cases, ARC-SC versus this DPP obtained 9/69/2 wins/ties/losses and −0.175 rounds [−0.338, −0.025]. The analogous three-anchor distance penalty gave −0.363 [−0.575, −0.175], whereas applying distance penalization from the second slot gave an inconclusive −0.150 [−0.425, 0.125]. DPP length-scale sensitivity also weakened the certainty of the comparison at alternative scales. We conclude only that ARC-SC is competitive with and sometimes earlier than these matched adaptations in the superconductivity setting—not that it universally outperforms established diversification methods.

The JARVIS transfer check is an important counterexample to broad attribution. In 40 shared-uniform, no-initial-hit cases, Risk-POF was 0.275 rounds later than ordinary POF [0.075, 0.525]; ARC-SC versus Risk-POF was −0.375 [−0.800, 0.025]. Full-risk fallback occurred at roughly three quarters of positions, and changing to validity-only risk eliminated fallback with little change to observed first-hit timing. The original support-aware design can therefore become restrictive without reliably changing decisions in these candidate pools. We will report this behavior and avoid generalizing the superconductivity risk attribution to JARVIS.

In short, the new controls substantiate the overall superconductivity first-hit result and identify the operating-risk rule as a substantial source of the observed improvement at the original threshold. They also support your criticism that scenario completion's independent first-hit contribution, and its relationship to real posterior redundancy, remain unproven. We believe the more precise characterization is important for assessing the contribution fairly.

c) Thank you for catching this ambiguity. In the original mixed-pool JARVIS study, 115/360 paired initializations (31.9%) already contained at least one valid target hit; all belonged to the qinit=0.85q_{\mathrm{init}}=0.85 stratum. Because the reported endpoint begins with the first newly queried batch, those cases measure discovery of an additional valid hit rather than the literal first-ever discovery. We agree that this qualification should have been prominent in the manuscript.

Among the 245 initializations with no valid target hit, ARC-SC versus POF yielded 33/185/27 wins/ties/losses and a mean capped first-hit difference of −0.106 rounds. By round five, 97 ARC-SC and 90 POF runs had found a hit, but ARC-SC had 19 fewer cumulative valid hits across these runs. This is a post hoc descriptive subgroup with mixed endpoint evidence, not a new confirmatory finding. We will report it transparently beside the full JARVIS result.

For further protocol clarity, the original uniform-pool comparisons use the same pool at corresponding paired rounds. In the two trust-region settings, each method generates its next pool from its own observed trajectory under the same rule; those comparisons evaluate complete closed-loop policies, not two acquisition functions over identical pools. The new 40-case shared-uniform JARVIS study, with no initial target hits, was designed separately to examine matched-risk attribution. We will distinguish these protocols explicitly.

I appreciate the authors' responses. After reading the replies to my questions and responses to other reviewers' questions, I'd like to update my rating from 1 to 2. However, I still see the weakness of this paper outweight its strength. I still lean to reject.

Reviewer_Jbe5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ARC-SC, a batch-selection method for inverse design when experimental evaluations can fail. Its goal is to discover at least one valid design satisfying a target requirement as early as possible, rather than necessarily finding the global optimum.

Critical strengths

First-hit discovery directly addresses limited-budget experimentation where finding one successful design matters more than maximizing the eventual objective.

The noisy-OR coverage function has a straightforward submodularity proof and a standard greedy-completion guarantee.

Critical weaknesses

Only two frozen oracle materials benchmarks are reported in the provided text. Real closed loop experimental validation and broader baseline comparisons remain future work.

The relaxed threshold, sigmoid response, percentile-rank gates, and tuning parameters introduce design choices. The coverage score is not established as a calibrated probability of first hit success.

Questions for authors

How sensitive is ARC-SC to these hyperparameters, especially the anchor count $K$ and risk budget $R$? An ablation or sensitivity analysis would help establish whether the reported improvement is robust or depends on a particular parameter configuration.

Multiple existing strategies can achieve similar performance. Could the authors clarify the novelty of the proposed method and explain why people should prefer it over existing approaches?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Jbe5 - part 1

Thank you for recognizing the practical first-valid-hit objective! We appreciate your requests for broader comparisons, sensitivity checks, calibration clarity, and a more actionable account of when the method should be preferred. Here are our responses to your questions:

a) Regarding the weakness: We agree that the two materials tasks use frozen response oracles and synthetic failure mechanisms in reduced descriptor spaces. Although the policies sequentially refit on their own observed feedback, these are simulated closed loops, not prospective laboratory experiments. We will distinguish these settings clearly in the abstract, experimental protocol, and conclusion.

As for the relaxed threshold, sigmoid, rank gates, and coverage utility, no calibration claim is intended, and the original text should have made that clearer. ARC-SC uses a noisy-OR utility of soft, gated responses, not an estimated, calibrated probability that the batch will contain a valid target hit. The relaxed target keeps responses informative when few predictive draws meet the true target; sigmoid smoothing and the percentile-rank gates change the utility used for selecting candidates. Evaluation always uses the unrelaxed target and a valid observed outcome, and invalid queries still consume budget. The submodularity result requires a fixed response matrix with entries in [0,1]; it implies neither predictive calibration nor real-world reliability of the risk score.

We also tested whether empirical support provides information beyond predicted validity. In a separate case-held-out candidate-pool diagnostic, adding support to a binned validity predictor improved JARVIS feature-interaction failure Brier loss by 0.00349 (loss difference −0.00349 [−0.00695, −0.00010]), but worsened JARVIS hidden-nonlinear failure Brier loss by 0.00130 ([0.00028, 0.00215]). This mechanism dependence is why we use psp_s as an operational support proxy, not a universally calibrated or independently validated failure probability. Likewise, the fixed-budget risk ablation does not control for equal filtering strength.

b) Question for sensitivity analysis: We ran controlled, one-factor-at-a-time sensitivity checks with the same 25-query budget. On 80 paired superconductivity cases, relative to the original K=3K=3, R=0.40R=0.40 policy, decreasing the anchor count to K=2K=2 changed the capped first-hit round by +0.150 [−0.025, 0.338], tightening risk to R=0.30R=0.30 by +0.075 [−0.113, 0.250], and relaxing it to R=0.50R=0.50 by +0.288 [0.013, 0.575]. The tighter risk setting reduced invalid evaluations by 1.30 percentage points but reduced cumulative valid hits by 0.625 per run; the looser setting increased invalid evaluations by 2.35 percentage points. Thus, we observe a genuine operating trade-off, not parameter invariance. The K=2K=2 effect remains uncertain, and JARVIS used a separately explored R=0.80R=0.80, which we will disclose rather than imply transferability.

On a locked 48-pair superconductivity subset, raising the relaxation fraction from 0.05 to 0.10 delayed first hit by +0.167 rounds [0.042, 0.292]. Doubling the effective sigmoid temperature produced −0.063 [−0.271, 0.083]; removing all three rank gates produced 0.000 [−0.208, 0.188]; and raising the number of scenarios from 64 to 128 produced 0.000 [−0.188, 0.208]. The latter settings show no resolved first-hit differences in this sample, not evidence of global insensitivity. We will describe these as local sensitivity analyses and retain the originally reported settings.

AuthorsOpenReview ↗

Response to Reviewer Jbe5 - part 2

c) Question for comparison with established diversification and when to prefer ARC-SC: Our contribution is most accurately described as an objective-specific construction and controlled empirical study of first-valid-hit batch selection, rather than a new submodular theorem or a universally superior diversification rule. The design preserves strong marginal valid-hit candidates, applies an explicit support/validity operating criterion, and uses soft-response completion for the remaining slots. This is a meaningful design distinction from the tested descriptor-kernel DPP and spatial distance-penalty adaptations, although the first-hit benefit of the completion component itself is still uncertain.

Under an identical three-anchor structure, risk rule, and 25-query budget, ARC-SC found the first hit 0.175 rounds earlier on average than the tested quality-weighted DPP adaptation [−0.338, −0.025] on 80 superconductivity pairs (9 wins, 69 ties, 2 losses). The DPP result was not robustly separated at all tested kernel scales, and the matched Risk-POF comparison did not establish an independent completion advantage. We therefore do not claim a general reason to choose ARC-SC over DPP or over simpler risk-matched marginal ranking.

The new risk studies provide a more useful conditional interpretation. On the superconductivity grid, the full support-aware rule outperformed validity-only risk at the original risk threshold, even for Risk-POF; this supports retaining support-aware filtering in that tested setting. In the JARVIS uniform candidate pools, empirical support often lay near its numerical floor, causing frequent fallback. Relaxing the rule to validity-only nearly eliminated admissibility failures but did not establish an improvement in first-hit timing. Thus, our evidence favors monitoring the admissible-pool size, fallback frequency, and realized first-hit performance, rather than using ARC-SC unconditionally. The present experiments do not justify a transferable switching threshold or claim that support is always predictive of failure. These limitations will be explicit in the revision.

We appreciate this opportunity to state more carefully what the method achieves, which choices matter, and where its evidence ends.

Hi Jbe5, please take a look at the authors' responses and leave a comment acknowledging that you've read their comments, whether those changee your assessment, and if you have additional questions. Thanks!

I appreciate the author's response and the additional experimental results. Based on my concern in Question (b), I believe the reliance on these heuristic settings may limit the method's applicability to previously unseen optimization problems. In practical applications, evaluation budgets are often limited, making it difficult to afford extensive tuning to identify the most suitable settings for an unknown problem. Therefore, I maintain my original score.

Reviewer_DWJ4OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes ARC-SC to find a valid candidate that meets a target under a limited experimental budget. It combines marginal valid-hit selection, scenario coverage, and risk filtering. The paper establishes submodularity for the fixed-coverage objective and reports improved first-hit timing in superconductivity.

Critical strengths

a) Finding the first valid candidate that meets a target quickly is a useful objective for costly experiments.

b) The superconductivity experiments show an improvement in first-hit timing, supported by paired comparisons and confidence intervals (Table 2).

Critical weaknesses

a) It is not clear why the proposed method is needed. The introduction discusses first-hit discovery but does not clearly explain what existing batch Bayesian and feasibility-aware methods cannot address. Identifying related issues does not automatically establish the need for the proposed combination.

b) The contribution beyond combining existing ideas is unclear. ARC-SC combines marginal selection, scenario coverage, and risk filtering. Its theoretical guarantee follows the standard submodular greedy result and applies only to fixed anchors and a fixed admissible pool. It does not cover the dynamic risk rule used by the method. The combination may be useful, but the paper needs stronger evidence of what it adds beyond these established components. Acknowledging this limitation does not resolve the disconnect between the theoretical guarantee and the implementation.

c) The scenario construction does not address a central part of the motivation. The introduction says that several candidates may share the same uncertain predictive explanation. However, the proposed scenario construction uses independent marginal draws and omits cross-candidate posterior covariance. I believe modeling this shared dependence is a practically important challenge, and addressing it would have been a better research contribution.

d) The experiments use only one baseline and do not show which component actually produces the improvement. A clearer ablation is needed to separate the effects of the combined componenets.

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer DWJ4 - part 1

We sincerely thank you for the constructive criticism and recognizing the practical importance of finding the first valid target-satisfying candidate under a limited experimental budget! We have re-examined the implementation and conducted some new tests. Here are our responses to your questions:

a) We agree that existing batch BO and feasibility-aware methods can be adapted to first-valid-hit search; our contribution is not that they are unable to address this task. In fact, if a candidate's marginal valid-hit probability is accurately represented by h(x)=py(x)pv(x)h(x)=p_y(x)p_v(x) and candidate hit events are conditionally independent, selecting the top B values of h(x) maximizes the probability of at least one hit in that single batch. Our POF baseline already accounts for predicted validity through p_v, and we will state this explicitly.

The design rationale for ARC-SC is to separate three decisions that can behave differently in failure-prone searches: marginal candidate quality (h), operational reliability (a support-aware prefix risk score, c(x)=1−pv(x)ps(x)c(x)=1-p_v(x)p_s(x), and completion of the remaining batch (a soft target-response coverage utility). The first three slots preserve strong marginal candidates; the remaining two permit response-based completion under the same risk selection mechanism. The empirical support term p_s measures proximity to the observed design region, rather than a calibrated failure probability. This is a structured, objective-specific policy, not a claim of universal optimality or an exact solution to the multi-round first-hit problem.

New controls also clarify which part of this rationale is supported. On 160 paired superconductivity closed-loop runs, changing only the risk formula from 1−pvps1-p_vp_s to 1−pv1-p_v at the original R=0.40R=0.40 delayed the mean capped first hit by 0.463 rounds for ARC-SC (95% paired interval [0.212, 0.719]) and 0.531 rounds for Risk-POF ([0.275, 0.781]); invalid-evaluation rates rose by 5.45 and 6.48 percentage points, respectively. Thus, the implemented support-aware operating rule has demonstrable value at this operating point. However, removing p_s also substantially enlarges the admissible set; without matching filtering strength, these experiments do not isolate the predictive information in p_s from the effect of stricter selection. Conversely, the incremental first-hit gain from scenario completion remains uncertain (details in part (d)). We will position the contribution around the complete, empirically evaluated first-hit policy and its measured component effects, rather than claiming that each component independently improves first-hit timing.

b) Your concern is correct. Our mathematical statement is the standard cardinality-constrained greedy guarantee for residual noisy-OR coverage with fixed anchors and a fixed admissible completion pool. It is not a new general submodular-optimization result, and it does not establish an approximation guarantee for the complete dynamically risk-filtered ARC-SC policy. The implementation recomputes admissibility after each selected prefix using an average-risk operating budget. Such admissibility is not hereditary: with budget 0.5, candidates with costs 0.9 and 0.1 form a pair with average cost 0.5, even though the singleton of cost 0.9 violates the budget. ARC-SC greedily maximizes the coverage gain among currently admissible extensions when any exist; otherwise, it uses a distinct risk-penalized fallback. Therefore, the fixed-domain greedy bound cannot be transferred to this full selection procedure. In the original superconductivity logs, 545/1800 batches ended above the operating risk threshold after fallback. The threshold is consequently an operating rule, not a hard safety guarantee. We will place this qualification beside the theorem and Algorithm 1, and report the realized exceedance frequency.

AuthorsOpenReview ↗

Response to Reviewer DWJ4 - part 2

c) We agree that our original motivation overstated this connection. The materials experiments use independent candidate-wise marginal predictive draws (Eq. (6) and Appendix A.1.5), not joint GP posterior-function samples. They therefore cannot identify shared cross-candidate GP posterior errors. We appreciate the reviewer for drawing attention to this substantive distinction.

Our implementation audit adds an important clarification about what the algorithm does compute. Each sampled scenario uses a shared, pool-wise relaxed threshold, which couples the resulting soft responses even though the underlying predictive draws are independent. For a fixed selected prefix, writing U_m for the remaining uncovered response mass and μx=rm(x)‾\mu_x=\overline{r_m(x)}, the empirical completion gain has the exact decomposition

Umrm(x)‾=Um‾μx+Cov^M(Um,rm(x)).\overline{U_m r_m(x)}=\overline{U_m} \mu_x+\widehat{\mathrm{Cov}}_M(U_m,r_m(x)).

The second term represents interaction in the constructed response matrix, including shared-threshold and finite-sample effects; it is not joint GP posterior covariance. The scale-matched factorized control in part (d) directly removes this empirical interaction. We will replace the claim about shared predictive explanations with the narrower description coverage of constructed soft marginal responses, and identify true joint-posterior completion as a separate methodological extension. Our controlled Gaussian analysis in the response to Reviewer 59i5 shows why the distinction can change which candidate should complete a batch.

D) Thank you for identifying this missing comparison. We added Risk-POF, which selects all five slots by the same marginal score h(x) but uses ARC-SC's identical support-aware prefix risk rule and fallback. For the 160 paired superconductivity cases (16 fixed conditions, 10 repeats per condition), all arms share the starting observations, oracle, simulated failure setting, model classes, candidate-pool protocol, and five batches of five queries. Each arm subsequently updates from its own feedback trajectory. The endpoint is the first newly queried batch containing a valid target hit; runs without a hit in five rounds are assigned 6. Negative differences favor the first method.

Comparison (160 paired cases)Wins / ties / lossesDifference in capped first-hit rounds (95% paired interval)
Risk-POF − POF38 / 107 / 15−0.481 [−0.744, −0.219]
ARC-SC − POF42 / 104 / 14−0.581 [−0.844, −0.319]
ARC-SC − Risk-POF16 / 134 / 10−0.100 [−0.250, 0.038]

These controls change our interpretation of the original result. On this grid, adding the risk/support operating rule accounts for the larger observed timing difference relative to ordinary POF; the added first-hit benefit of replacing the last two risk-matched marginal slots with scenario completion is not established. ARC-SC had an exploratory advantage of 0.338 cumulative valid hits per run over Risk-POF [0.088, 0.613], which we do not substitute for the primary first-hit outcome.

We then separated soft-response rescoring from the scenario-overlap term. With the same anchors, response construction, risk solver, and budget, ARC-SC versus scale-matched factorized-mean completion (U‾μx\overline{U}\mu_x) gave 8/146/6 wins/ties/losses and −0.019 rounds [−0.119, 0.081] over 160 pairs. The direct soft-marginal completion (\mu_x) produced the same first-hit comparison with ARC-SC. Relative to Risk-POF, direct soft-marginal completion gave −0.081 rounds [−0.206, 0.031]. Neither the empirical overlap term nor soft-response rescoring is thereby established as an independent first-hit improvement. The scale-matched and direct-mean variants are distinct during fallback because the latter changes the desirability-to-risk-penalty scale; we preserve this distinction in the revised account.

We also compared matched diversification alternatives. On 80 superconductivity pairs, ARC-SC was 0.175 rounds earlier than a three-anchor, quality-weighted greedy DPP completion [−0.338, −0.025] and 0.363 rounds earlier than a three-anchor distance-penalization adaptation [−0.575, −0.175]. The full-batch distance-penalty comparison was inconclusive (−0.150 [−0.425, 0.125]), and DPP length-scale changes weakened the certainty of its comparison. These results demonstrate competitiveness with the tested, objective-matched adaptations, not superiority over all batch diversification methods.

AuthorsOpenReview ↗

Response to Reviewer DWJ4 - part 3

Finally, the attribution is not universal across response surfaces. In 40 new JARVIS paired cases without initial hits and with shared uniform candidate pools, Risk-POF was 0.275 rounds later than POF [0.075, 0.525], while ARC-SC versus Risk-POF was −0.375 rounds [−0.800, 0.025]. The full risk rule triggered fallback in 73.4% of Risk-POF and 75.3% of ARC-SC selection positions. Replacing it with validity-only risk eliminated fallback in both policies, but changed first-hit timing little (ARC-SC: −0.025 [−0.125, 0.050]; Risk-POF: −0.050 [−0.125, 0.000]). This provides a concrete boundary condition: support-based admissibility can become overly restrictive for a candidate pool with very low empirical support, without necessarily altering the selected queries or improving timing. We will report this counterexample rather than generalize the superconductivity decomposition.

In summary, your concerns led us to isolate the complete-policy effect, the risk-rule effect at the original operating point, and the unresolved incremental completion effect. We hope this clearer and more limited attribution addresses the principal empirical and conceptual weaknesses you identified.

Thanks for your comprehensive responses. Appreciate your willingness to acknowledge the limitations and revise the claims accordingly. The added experiments provide further evidence and clarify the paper’s claims. In view of this, I’m raising my score from 1 to 2 (weak reject).

However, the new experiments also support much of my initial concerns. For exp. as you acknowledge in your response to point (d), “the added first-hit benefit of replacing the last two risk-matched marginal slots with scenario completion is not established.” I believe the methodology still needs substantial simplification, and the motivation for combining these components needs to be made clearer. I therefore recommend reject in its current form.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Update

We sincerely thank all three reviewers for their careful reading and constructive criticism. Your comments identified important gaps in our explanation of the first-valid-hit objective, ARC-SC's components, the scope of its theoretical guarantee, and the strength of the experimental evidence.

We have revised the manuscript and added matched-policy ablations, diversification baselines, sensitivity analyses, independent-random-stream replication, and controlled correlation studies. The new results strengthen the complete-policy finding on superconductivity while requiring more cautious claims about scenario coverage and generalization. The main changes are:

  1. Motivation and positioning (§1; §§2.1–2.2, 3.1). We clarify valid versus invalid outcomes and the first newly queried hit. We explain when marginal valid-hit ranking (POF) is optimal for a single batch and position ARC-SC as an objective-specific policy, not a method that uniquely solves first-hit search.
  2. Scenarios and theoretical scope (§§2.5–2.7, 3; Appendices A.2, B.2, B.5). We state explicitly that the materials experiments draw independent predictive marginals, not joint GP posterior samples. Constructed soft-response overlap does not imply shared posterior errors. The standard submodular greedy guarantee covers fixed anchors and a fixed candidate pool, not dynamic risk filtering, fallback, or first-hit time. A Gaussian counterexample and joint-sampling experiments examine the missing dependence information.
  3. Matched component attribution (§4.5, Table 2; Appendix B.3). We add Risk-POF, which uses ARC-SC's risk/support rule and fallback but ranks all five slots by marginal score. Across 160 paired superconductivity cases, Risk-POF improves on POF by 0.481 capped rounds, while ARC-SC improves by 0.581. ARC-SC's additional 0.100-round improvement over Risk-POF has a confidence interval crossing zero. Factorized- and direct-mean controls likewise do not establish an independent first-hit benefit from scenario overlap.
  4. Additional baselines and sensitivity (§§4.5–4.6; Appendices B.3–B.4, B.6). We compare matched DPP and distance-based diversification, examine anchor count, risk budget, response relaxation, smoothing, rank gates, and scenario count, and diagnose support-aware admissibility and fallback. These results expose operating trade-offs rather than universal superiority.
  5. Independent replication and statistics (§§4.1–4.3, Table 1, Figure 1; Appendix B.1). After identifying seed reuse in historical grids, we ran 360 new paired superconductivity cases with independent random streams. ARC-SC achieves 143/360 discoveries by round five versus 112/360 for POF, with a mean first-hit difference of −0.353 capped rounds (95% CI [−0.620, −0.097]). Historical inference is now qualified.
  6. JARVIS checks and evaluation limits (§§4.1–4.2, 4.6; Appendices A.3, A.6, B.4). We report that 115/360 historical JARVIS starts already contained a valid hit, analyze the hit-free subset, and distinguish shared-uniform from trajectory-dependent trust-region pools. The historical mixed-pool comparison is explicitly exploratory. A further 40-pair shared-uniform study reveals frequent fallback and uncertain transfer of the superconductivity benefit. We also clarify the frozen-oracle, synthetic-failure setting and its limitations for physical validation and out-of-distribution predictions.

In summary, the revision supports earlier first-hit discovery for the complete policy in the tested superconductivity simulation, while acknowledging that scenario completion's incremental benefit, joint-posterior redundancy reduction, and real-laboratory generalization remain unestablished. We have revised our claims accordingly. We appreciate the feedback and hope the revised manuscript makes these distinctions easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.