acceptodds
Under review as a conference paper at ICLR 2027

One-Hit Detection in Generative Sampling: Optimal Stopping and Conformal Calibration

Abstract

Generative models can produce candidates for scientific discovery, but most lack the desired property, and determining which ones qualify often requires costly experiments. In practice, predictive models help assess whether at least one hit—a candidate satisfying the target property—has been generated, thereby guiding when to stop generation and reducing unnecessary experimental validation. We formulate this task as a finite-horizon optimal stopping problem that maximizes expected reward for early detection while controlling the probability of false alarms, i.e., stopping before any hit appears. Given a sequence of independent -values, we first construct a monotone family of stopping times based on Lagrangian duality, each optimal for the Lagrangian problem at its corresponding multiplier. We then propose a novel method Conformalized Optimal Stopping, applying the monotone family to conformal \(p\)-values constructed from screening scores and selecting the multiplier by calibration on simulated global-null trajectories. Under suitable assumptions, the resulting stopping strategy achieves nearly optimal expected reward with exact finite-sample false-alarm control. Experiments on QED–HGraph and TargetDiff candidates demonstrate substantial improvements over competing methods with the same finite-sample false-alarm guarantees.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Optimal stopping, Conformal inference, Generative sampling, One-hit detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_NGjuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a problem: one-hit detection in sequential generative sampling candidates arrive online. The goal of the problem is to stop as soon as there is enough evidence that at least one hit, saving costly experimental feedback. It is formulated as a finite-horizon constrained optimal stopping problem, maximizing expected reward for early correct detection under a global-null false-alarm constraint. With independent p-values and a two-group working alternative, Lagrangian duality yields a monotone family of Bellman-optimal rules (Thm. 1). For real scores, the proposed Conformalized Optimal Stopping applies this family to conformal p-values and calibrates the multiplier on simulated null trajectories. This gives finite-sample false-alarm control under exchangeability, even if the working alternative is misspecified (Thm. 2), and a reward gap to the oracle of O(√(log n/n) + 1/√B) (Thm. 3).

Contributions: The paper introduces a new, well-motivated problem that is distinct from prior work, and gives an exact characterization of its optimal stopping rule together with a tractable finite-partition implementation. It also proposes a calibration method that uses pathwise monotonicity to handle the dependence induced by the shared reference sample, so that validity does not depend on the working alternative being correct. Finally, it provides efficiency guarantees that separate the effects of the reference sample size and the number of calibration trajectories.

Experiments on synthetic data, QED–HGraph, and TargetDiff show COS with the highest reward in all settings (gains of roughly 8%–29% over the best baseline), with false-alarm rates below nominal levels as guaranteed.

Critical strengths

1. The problem setting is interesting and well-motivated.

The problem is well-motivated and can have real impact. Online one-hit detection ties directly to a real cost in scientific discovery (stopping generation and triggering costly validation).

2. The solution is principled and with a clean theory.

Lagrangian duality gives a monotone family of Bellman-optimal rules and exact constrained optimality (Thm. 1). The finite-partition implementation comes with an explicit approximation-loss bound, the method is both optimal in principle and computable in practice.

3. Experimental results have significant improvement on real world settings.

COS achieves the highest mean reward in every setting tested: all 16 synthetic settings and all 12 settings on each of QED–HGraph and TargetDiff. The margins over the strongest baseline are substantial (about 8%–29%). The results hold across two reward schedules, multiple horizons, and multiple α levels, and the gap tends to widen at larger T. All methods are recalibrated on complete null trajectories, so the comparison is made under the same finite-sample false-alarm guarantee, and the real-data benchmarks span both generative molecular design and docking.

Critical weaknesses

1. Baseline strength is unclear.

The baselines are the authors' own adaptations of existing methods (e.g., O-CONFHIT is an online variant of an offline procedure). How much tuning each received is not very clear, part of COS's reported gain may reflect how the baselines were adapted.

2. The false-alarm constraint appears conservative.

Realized false-alarm rates are well below the nominal level (e.g., roughly 0.06 at α = 0.10 and 0.25 at α = 0.40 in Table 1), which the authors acknowledge. Because the global-null constraint is loose for any particular generation mechanism, it is hard to separate the benefit of the optimal stopping design from more efficient use of unused budget. It would help to compare rewards at matched realized false-alarm rates, or to explore ways to reduce the conservatism.

Questions for authors
  1. The reduction from false-alarm control to the global-null stopping probability (Prop. A.1) requires candidates to be independent across time. Could the authors comment on how the guarantee degrades for generators that produce correlated or iteratively refined samples?

  2. See weakness 1.

  3. See weakness 2.

2 comments
AuthorsOpenReview ↗

Response to Reviewer NGju

W1. The baselines are the authors' own adaptations of existing methods (e.g., O-CONFHIT is an online variant of an offline procedure). How much tuning each received is not very clear, part of COS's reported gain may reflect how the baselines were adapted.

Response. We thank the reviewer for raising this concern. We have clarified the tuning and calibration protocol as follows.

  • Common calibration. Because conformal pp-values share a reference set and are dependent, we recalibrate the final rejection threshold of every method, including COS, LR-e, OneHit, Stouffer, IMT-Alt, and O-CONFHIT, using the same independent conformal-null trajectories.

  • Global-null diagnostic. The table reports held-out global-null false-alarm rates for the Beta(1.5,3)\mathrm{Beta}(1.5,3) experiment with T=30T=30 and linear reward. Each entry is averaged over 100 repeats, with 9,999 held-out null trajectories per repeat.

α\alphaCOSOneHitO-CONFHITLR-eStoufferIMT-Alt
0.10.1000.0990.1000.1000.0990.094
0.20.2000.1990.2010.1990.2000.191
0.30.3000.2980.3000.2990.3000.296
0.40.3990.3980.4000.3990.4000.373

Most baselines, including the strongest baseline LR-e, are no more conservative than COS. Therefore, COS's advantage cannot be attributed to a more permissive calibration or unfavorable baseline tuning. It arises from its reward-aware optimal stopping rule under the two-group working model.

W2. Realized false-alarm rates are well below the nominal level (e.g., roughly 0.06 at α\alpha = 0.10 and 0.25 at α\alpha = 0.40 in Table 1), which the authors acknowledge. Because the global-null constraint is loose for any particular generation mechanism, it is hard to separate the benefit of the optimal stopping design from more efficient use of unused budget. It would help to compare rewards at matched realized false-alarm rates, or to explore ways to reduce the conservatism.

Response. Thank you for this suggestion. The methods are compared at approximately matched global-null alarm rates, making the reward comparison meaningful under our common constraint.

We use this conservative constraint because the hit-generation mechanism is generally unknown. It ensures false-alarm control under our assumptions without requiring accurate specification of that mechanism.

Direct calibration of the realized false-alarm rate is a promising direction for future work. A setting with labeled calibration trajectories, similar to Conformal Language Modeling (Quach et al., 2024), could support such control, but requires a different data structure from our current setting.

Q1. The reduction from false-alarm control to the global-null stopping probability (Prop. A.1) requires candidates to be independent across time. Could the authors comment on how the guarantee degrades for generators that produce correlated or iteratively refined samples?

Response. Thank you for this question. Appendix A now quantifies the degradation under temporal dependence. If P0(τ≤T)≤α\mathbb P_0(\tau\le T)\le\alpha, then

P(τ<ν∗)≤α+∑t=1TP(Ht)δt,Ht=A1=⋯=At=0,\mathbb P(\tau < \nu^*) \le \alpha+\sum_{t=1}^T\mathbb P(H_t)\delta_t, \qquad H_t=\\{A_1=\cdots=A_t=0\\},

where δt=dTV!(LP(Y1:t∣Ht),LP0(Y1:t)).\delta_t= d_{\mathrm{TV}}\\!\left( \mathcal L_{\mathbb P}(Y_{1:t}\mid H_t), \mathcal L_{\mathbb P_0}(Y_{1:t}) \right). The additional term measures the discrepancy between non-hit prefix distributions under the actual process and the global null. For correlated or iteratively refined samples, conditioning on future samples also being non-hits can alter earlier prefix distributions. Under independence, these discrepancies vanish, recovering Proposition A.1.

References

[1] Laghuvarapu, S., Jin, Y., & Sun, J. (2026). ConfHit: Conformal Generative Design with Oracle-Free Guarantees. International Conference on Learning Representations.

[2] Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T., & Barzilay, R. (2024). Conformal Language Modeling. International Conference on Learning Representations.

I appreciate the authors’ response, I continue to support acceptance.

Reviewer_jpEZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when to stop sampling from a generative model so that at least one generated candidate is a hit, while keeping the probability of stopping too early below α\alpha. The authors first derive the optimal stopping rule assuming that each candidate comes with an independent p-value and that hits follow a simple working model, using a Lagrangian relaxation and dynamic programming. In practice, only screening scores are available, so they propose Conformalized Optimal Stopping (COS), which converts the scores into conformal p-values and calibrates the rule on simulated trajectories with no hits. They show that COS controls the false-alarm rate in finite samples even if the working model is wrong, and that its reward approaches the oracle reward as the reference set and the number of simulated trajectories grow. Experiments on synthetic data and on HGraph and TargetDiff molecules show that COS achieves higher reward than the baselines while keeping the false-alarm rate below α\alpha.

Critical strengths
  1. The calibration idea is sound and easy to understand: because the stopping rules are monotone in the multiplier, calibration reduces to taking a quantile over simulated trajectories, which gives finite-sample false-alarm control without relying on the working model.

  2. The theoretical analysis is complete, and the proofs look correct on a quick read.

Critical weaknesses
  1. The practical benefit of stopping generation early is not demonstrated. COS decides when to stop generating, but in practice the costly step is validation, and COS does not say which candidates to validate. If validation is sequential, one can validate candidates in order of their scores until a hit is found. If validation is done in batches, one can generate a larger pool and validate the top-scoring candidates. In both cases, a simple score-based strategy seems more natural, since it spends the expensive validations on the most promising candidates, and the paper neither describes a setting where COS is preferable nor compares against such a strategy.

  2. The experiments do not reflect realistic use cases. The synthetic data are generated from the same model under which COS is optimal, so its advantage there is expected. The real-data experiments pool candidates from many pockets (or starting molecules) into one stream, rather than evaluating the single-pocket setting that motivates the paper, and all experiments use hit rates of 10–16%. It would be more informative to test a misspecified working model, per-pocket trajectories, and rare hits. The gains over the strongest baseline (LR-e) are also small relative to the variation across repeats.

Questions for authors
  1. In what practical setting is stopping generation early more useful than generating a larger pool and validating the top-scoring candidates? How does COS compare with this strategy on the existing data?

  2. How do the results change when trajectories are formed within a single pocket (or starting molecule), especially for contexts with few hits?

  3. How does COS perform when the hit rate is around 1% or lower?

5 comments
AuthorsOpenReview ↗

Response to Reviewer jpEZ (part I)

W1(A). The practical benefit of stopping generation early is not demonstrated. COS decides when to stop generating, but in practice the costly step is validation, and COS does not say which candidates to validate. If validation is sequential, one can validate candidates in order of their scores until a hit is found. If validation is done in batches, one can generate a larger pool and validate the top-scoring candidates.

Response. We thank the reviewer for this question. COS and score-based validation address complementary decisions: COS determines how many candidates to generate, while score ranking determines which generated candidates to validate first.

Generation itself can be expensive: for example, TargetDiff uses 1,000 reverse-diffusion steps per molecule (Guan et al., 2023). Early stopping can therefore reduce generation cost. Moreover, scores are imperfect proxies for hit labels, and a larger pool may introduce extreme-score non-hits that are validated before genuine hits. COS can complement ranking by selecting a smaller prefix for downstream validation. The comparison in W1(B) illustrates this potential benefit.

W1(B). In both cases, a simple score-based strategy seems more natural, since it spends the expensive validations on the most promising candidates, and the paper neither describes a setting where COS is preferable nor compares against such a strategy.

Response. We added a simulation comparing (i) generating all TT candidates and validating them in increasing-score order and (ii) using COS to select a prefix and applying the same ranking strategy within that prefix.

We use T=40T=40, P(A=1)=0.1\mathbb P(A=1)=0.1, S∣A=0∼Unif(0,1)S\mid A=0\sim\mathrm{Unif}(0,1), and S∣A=1∼Unif(0.05,0.30)S\mid A=1\sim\mathrm{Unif}(0.05,0.30), with 10610^6 simulated trajectories. Costs from unsuccessful attempts are included.

Average cost per discovered hit.

Methodα\alphaGenerations/hitValidations/hit
Generate-all + ranking--40.605.50
COS + ranking0.133.465.28
COS + ranking0.229.445.23
COS + ranking0.326.375.27

At α=0.1\alpha=0.1, COS reduces generations per discovered hit by 17.617.6\\% and validations by 4.04.0\\%. This example shows that adaptive stopping can reduce generation cost while also improving the efficiency of the same score-based validation strategy. In particular, COS adaptively narrows the candidate set before applying the same score-based ranking, thereby requiring fewer validations per discovered hit.

W2(A). The experiments do not reflect realistic use cases. The synthetic data are generated from the same model under which COS is optimal, so its advantage there is expected. The real-data experiments pool candidates from many pockets (or starting molecules) into one stream, rather than evaluating the single-pocket setting that motivates the paper, and all experiments use hit rates of 10–16%.

Response. We thank the reviewer for this suggestion. Appendix I now forms every test trajectory within a single starting-molecule seed or protein pocket, with disjoint historical and test contexts. Weighted COS attains the largest mean utility in 8 of the 12 High-QED settings and the best average utility rank across all 12 docking settings. Its global-null false-stopping rates are broadly controlled, with modest inflation in some settings. For other related discussion, please see our response to Reviewer hGES's W1 and W2.

W2(B). It would be more informative to test a misspecified working model, per-pocket trajectories, and rare hits. The gains over the strongest baseline (LR-e) are also small relative to the variation across repeats.

Response. We thank the reviewer for this suggestion. In the real-data experiments, the working alternative is estimated from finite data and is already subject to estimation error and potential misspecification. Appendix G.2.1 additionally provides a controlled simulation study, fixing the truth at b=0.10b=0.10 and S∣A=1∼Unif(0,0.2)S\mid A=1\sim\mathrm{Unif}(0,0.2).

With misspecified hit probabilities b~∈0.06,0.20\widetilde b\in\\{0.06,0.20\\}, COS achieves the highest utility in all evaluated settings. When the working hit-score distribution is changed to Unif(0,0.1)\mathrm{Unif}(0,0.1) or Unif(0,0.3)\mathrm{Unif}(0,0.3), COS no longer consistently ranks first but remains competitive.

Results for trajectories formed within individual pockets or starting-molecule contexts are provided in Appendix I; see our response to W2(A). Although the gains over LR-e are modest in some settings, they are substantial in others, notably the QED-HGraph experiment in Section 5.2 (Figure 3); we discuss the rare-hit regime separately in our response to Q3.

AuthorsOpenReview ↗

Response to Reviewer jpEZ (part II)

Q1. In what practical setting is stopping generation early more useful than generating a larger pool and validating the top-scoring candidates? How does COS compare with this strategy on the existing data?

Response. Please see our response to W1(A) and W1(B).

Q2. How do the results change when trajectories are formed within a single pocket (or starting molecule), especially for contexts with few hits?

Response. Please see our response to W2(A) and W2(B).

Q3. How does COS perform when the hit rate is around 1% or lower?

Response.

  • In the ConfHit experiments (Laghuvarapu et al., 2026), the hit proportions are approximately 4545\\%--6262\\%. To investigate more challenging tasks, our synthetic, High-QED, and molecular-docking experiments use lower hit proportions of 1010\\%, 15.8715.87\\%, and 15.5015.50\\%, respectively.

  • Our theoretical optimality result applies to any hit proportion when the working alternative is correctly specified. Nevertheless, a hit rate of 11\\% or lower is considerably more challenging for all methods, since most finite-horizon trajectories contain no hits and the alternative is harder to estimate. Preliminary screening or improved generation may be needed in this regime, which we leave for future work.

References

[1] Guan, J., Qian, W. W., Peng, X., Su, Y., Peng, J., & Ma, J. (2023). 3D Equivariant Diffusion for Target-Aware Molecule Generation and Affinity Prediction. International Conference on Learning Representations.

[2] Laghuvarapu, S., Jin, Y., & Sun, J. (2026). ConfHit: Conformal Generative Design with Oracle-Free Guarantees. International Conference on Learning Representations.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. When you have a chance, we would appreciate your thoughts on our rebuttal and whether any concerns remain unresolved. We would also be grateful if you could reassess the paper in light of our responses. We would be happy to provide further clarification during the discussion period.

Reviewer_wkaqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The work introduces Conformalized Optimal Stopping (COS) that Imas to pick an optimal candidate from a pool of generated samples based on their true/estimated p-values.

Critical strengths

Strengths

  • Solid idea: The paper presents a straightforward and natural optimization technique for selecting an optimal candidate from a pool of generated samples. This idea has potential applications in many scientific domains.

  • Good contribution: I found the included theorems and results to be non-trivial and interesting.

  • Experimental results: The paper includes both synthetic and real-world experiments, providing empirical evidence to support its claims.

Critical weaknesses

Weaknesses

  • Writing: I found the writing somewhat confusing and convoluted, particularly in Section 3.2. Authors should simplify language, refer less to Appendix to understand critical parts of the paper, and dedicate a full section on used terminology. The narration jumps between definitions and proposed methods too much. Additionally, authors should shorten discussions and include a proper ‘Conclusion’ section. This is not a critical weakness but rather a recommendation to improve the readability of the work.

  • Reference set choice: The authors mention that 20% of samples are used as references to estimate p-values. What is the guideline behind this number? At what point does COS collapse due to a small reference set? This is important, since in many domains, getting a solid reference set can be somewhat challenging.

Questions for authors
  • What is the guideline behind "20% for reference set"? Is there an ablation of sorts that could help establish a healthy size of the reference set?
  • At what point does COS collapse due to insufficient size of a reference set?
2 comments
AuthorsOpenReview ↗

Response to Reviewer wkaq

W2. The authors mention that 20% of samples are used as references to estimate p-values. What is the guideline behind this number? At what point does COS collapse due to a small reference set? This is important, since in many domains, getting a solid reference set can be somewhat challenging.

Response. Please see our response to Q1 and Q2.

Q1. What is the guideline behind "20% for reference set"? Is there an ablation of sorts that could help establish a healthy size of the reference set?

Response. We thank the reviewer for raising this question. We clarify the role of the reference split and add a reference-size ablation.

  • What the 2020\\% split means. The 20% split defines a reference pool, from which only 200 observations are randomly sampled to construct conformal (p)-values. Thus, 20% is a practical data-splitting choice rather than a theoretical requirement of COS.

  • Reference-size ablation. We added an ablation in Appendix G.2.2 with nref∈10,20,30,40,50,100,150,200n_{\rm ref}\in\\{10,20,30,40,50,100,150,200\\} and T∈20,30,40,50T\in\\{20,30,40,50\\}, while keeping all other settings fixed. The following table reports mean utilities for the Unif(0,0.2)\text{Unif}(0,0.2) hit-score distribution, linear reward, and α=0.20\alpha=0.20.

n_refT = 20T = 30T = 40T = 50
2004.1057.65711.71316.224
503.7766.85010.30213.897
303.7066.5589.76013.229
103.0415.2707.55310.174

Overall, performance is reasonably robust across reference sizes, and choosing nref=CTn_{\rm ref}=CT for a suitable constant CC appears to be a practical rule of thumb.

Q2. At what point does COS collapse due to insufficient size of a reference set?

Response. We observe no abrupt validity breakdown: the global-null false-alarm rate remains controlled even at nref=10n_{\rm ref}=10. Smaller reference sets mainly reduce utility, with a substantial decline when nrefn_{\rm ref} decreases from 30 to 10. Appendix H also extends COS to covariate shift through density-ratio weighting, allowing reference data from a related distribution to be used under the stated assumptions. This can help alleviate the scarcity of in-distribution reference data.

References

[1] Laghuvarapu, S., Jin, Y., & Sun, J. (2026). ConfHit: Conformal Generative Design with Oracle-Free Guarantees. International Conference on Learning Representations.

I thank the authors for their response. I maintain my initial positive assessment.

Reviewer_hGESOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sequential one-hit detection, where the goal is to stop generation once there is sufficient evidence that at least one generated candidate satisfies a desired property. The authors formulate this as a constrained optimal stopping problem and derive stopping rules under a working alternative distribution. They then calibrate the stopping penalty using simulated conformal null trajectories to obtain finite-sample control. Experiments compare the method against sequential-testing baselines on synthetic data and molecular candidates released with prior work, ConfHit.

Critical strengths
  • The paper addresses a relevant problem. Deciding when to stop generating candidates while controlling false alarms is useful when generation and subsequent validation are expensive.

  • The method explicitly accounts for the dependence between conformal p-values introduced by a shared reference panel. Calibrating complete null trajectories is a useful component of the approach.

  • COS consistently achieves higher mean reward across the reported synthetic and molecular settings. The experiments include multiple reward schedules, horizons, and error levels, with repeated data splits.

Critical weaknesses
  1. The experimental setup assumes access to labeled generated candidates which is the primary bottleneck the motivating application tries to address (ConfHit). The motivation is to identify whether generation has produced a hit when evaluating candidates is costly. However, the experiments begin with an already evaluated pool and randomly split it into training, reference, and test sets. This provides labeled reference non-hits from the same population as the test candidates. It leaves unresolved how the method would operate using only historical labeled data, without evaluating an additional sample from the deployment distribution. Covariate shift can invalidate the required exchangeability, and the uniform simulator does not correct this mismatch.

  2. The pooled experiments do not establish performance for individual generation tasks. TargetDiff candidates from 83 protein–pocket contexts are pooled before splitting, so test trajectories can mix candidates from different pockets. The QED experiment similarly pools candidates across starting molecules. In practice, generation typically targets a particular pocket or optimization task. Hit rates and non-hit score distributions may differ substantially across these contexts, making the pooled results insufficient to establish validity or efficiency in the motivating setting.

  3. Relevant conformal generative sampling methods are missing from the comparison. Conformal Language Modeling already calibrates sequential sampling to obtain a set containing at least one acceptable output. The labeled calibration data available in these experiments make it a directly relevant baseline, with appropriate alignment of error criteria and budget handling. The paper should clarify its contribution relative to this literature and establish the benefit of its optimal stopping formulation over existing calibrated sampling rules.

  4. The reward metric makes the practical benefit difficult to assess. The experiments use linear and exponential rewards that favor early stopping, but their connection to actual generation or validation costs is not established. Since COS directly optimizes these rewards, the reported gains do not reveal whether it detects hits more frequently, stops earlier, or both. Successful-detection probability, stopping times, and the fraction of trajectories with no alarm should be reported separately.

  5. The reported error rates do not directly evaluate the calibrated constraint. Calibration controls alarms under the global null, while the experiments report stopping before the first hit on mixed hit/non-hit trajectories. The latter can be substantially smaller because any alarm after a hit is counted as correct. Held-out global-null alarm rates are needed to assess calibration conservatism and interpret reward differences across methods.

  6. The baseline adaptations need stronger justification. O-CONFHIT uses prefix maxima, prespecified temporal weights, and additional trajectory calibration, while omitting density-ratio weighting and backward monotonization. The comparison therefore concerns this particular adaptation.

Questions for authors
  1. How would calibration work when only historical labeled data are available, without evaluating additional candidates from the deployment distribution? What happens to false-alarm control under covariate shift between the reference non-hits and newly generated candidates?

  2. Can the authors evaluate trajectories within individual protein pockets or starting-molecule contexts? Pooling candidates across contexts does not establish performance for the fixed-context generation setting used to motivate the method.

  3. Given the labeled calibration data available in these experiments, can the authors compare against Conformal Language Modeling or related calibrated generative sampling methods? What additional benefit does the proposed stopping formulation provide?

  4. Can the authors separately report successful-detection probability, stopping times, and the fraction of trajectories with no alarm? How sensitive are the conclusions to the reward schedule, including the constant-reward setting wt=1w_t=1?

  5. Can the authors report held-out global-null alarm rates alongside the mixed-trajectory false-alarm rates? This would help determine whether differences in reward are partly explained by calibration conservatism.

5 comments
AuthorsOpenReview ↗

Response to Reviewer hGES (part I)

W1(A). The experimental setup assumes access to labeled generated candidates which is the primary bottleneck the motivating application tries to address (ConfHit). The motivation is to identify whether generation has produced a hit when evaluating candidates is costly. However, the experiments begin with an already evaluated pool and randomly split it into training, reference, and test sets. This provides labeled reference non-hits from the same population as the test candidates. It leaves unresolved how the method would operate using only historical labeled data, without evaluating an additional sample from the deployment distribution.

Response. We thank the reviewer for highlighting this limitation. We agree that the original experiments use reference and test candidates from the same population and therefore do not fully address historical-to-deployment distribution shift.

To address this concern, Appendix H.1 extends COS using density-ratio weighting, following ConfHit (Laghuvarapu et al., 2026). The weights are estimated using only historical labeled data and used to construct weighted conformal pp-values for deployment candidates:

P~t,w=∑i=1nv^(Yi0)1Si0<St+Utv^(Yt)∑i=1nv^(Yi0)+v^(Yt),\widetilde P_t^{\\,\mathrm w} = \frac{ \sum_{i=1}^n \widehat v(Y_i^0)\mathbf{1}\\{S_i^0 < S_t\\} +U_t\widehat v(Y_t) }{ \sum_{i=1}^n\widehat v(Y_i^0) +\widehat v(Y_t) },

where Yi0Y_i^0 are historical non-hit references, Si0=s(Yi0)S_i^0=s(Y_i^0), St=s(Yt)S_t=s(Y_t), and Ut∼Unif⁡(0,1)U_t\sim\operatorname{Unif}(0,1) are drawn independently. Here, v^\widehat v estimates the density ratio between the deployment and historical populations to account for distribution shift. We then apply the stopping rule τλ\tau_\lambda to the weighted pp-value sequence to determine when to stop generation.

Appendix I adds experiments on the High-QED and molecular-docking tasks with disjoint historical and deployment seeds. Model fitting, weight estimation, and reference construction use only historical data; deployment labels are used only for evaluation. Weighted COS achieves competitive utility compared with ConfHit and the other baselines in this setting.

W1(B). Covariate shift can invalidate the required exchangeability, and the uniform simulator does not correct this mismatch.

Response. We agree that the trajectory simulator in the main text does not itself correct covariate shift. Under suitable conditions, however, weighted conformal pp-values are asymptotically independent and uniform under the global null, motivating independent uniform trajectories as an approximate calibration law. Appendix H.2 quantifies the approximation error: under its stated assumptions, the false-alarm rate is bounded with probability at least 1−δ1-\delta by

\alpha+C\left\\{ \varepsilon_v +\sqrt{\frac{\log(n/\delta)}{n}} +\sqrt{\frac{\log(1/\delta)}{B}} +\frac{1}{B} \right\\}.

Here, εv\varepsilon_v measures density-ratio estimation error; the nn-dependent term captures finite-reference approximation error; and the BB-dependent terms capture Monte Carlo calibration error from BB simulated trajectories.

W2. The pooled experiments do not establish performance for individual generation tasks. TargetDiff candidates from 83 protein–pocket contexts are pooled before splitting, so test trajectories can mix candidates from different pockets. The QED experiment similarly pools candidates across starting molecules. In practice, generation typically targets a particular pocket or optimization task. Hit rates and non-hit score distributions may differ substantially across these contexts, making the pooled results insufficient to establish validity or efficiency in the motivating setting.

Response. We thank the reviewer for highlighting this issue. Appendix H develops weighted COS to address distribution shifts across contexts through density-ratio weighting and context-specific working alternatives. Appendix I evaluates this extension in the requested fixed-context setting.

  • Within-context trajectories. Historical and test contexts are disjoint. Within each test context, candidates are divided into length-TT trajectories, so no trajectory mixes candidates from different pockets or starting molecules.

  • Results. Weighted COS attains the largest mean utility in 8 of the 12 High-QED settings and the best average utility rank across all 12 docking settings. Its global-null false-stopping rates are broadly controlled, with modest inflation in some settings.

AuthorsOpenReview ↗

Response to Reviewer hGES (part II)

W3. Relevant conformal generative sampling methods are missing from the comparison. Conformal Language Modeling already calibrates sequential sampling to obtain a set containing at least one acceptable output. The labeled calibration data available in these experiments make it a directly relevant baseline, with appropriate alignment of error criteria and budget handling. The paper should clarify its contribution relative to this literature and establish the benefit of its optimal stopping formulation over existing calibrated sampling rules.

Response. We thank the reviewer for this suggestion. We have added a discussion and comparison with CLM.

  • Difference from CLM. Conformal Language Modeling (CLM) considers prespecified stopping rules, including count-, maximum-, and sum-based rules, and calibrates their thresholds to obtain coverage guarantees (Quach et al., 2024). In contrast, under the independent two-group working model, we optimize the one-hit reward over all adapted stopping rules subject to the global-null constraint. This explicit reward optimization (Theorem 1) and the optimality guarantee (Theorem 3), also distinguish our framework from the other baselines considered.

  • Experimental comparison. We adapt the sum-based CLM rule and calibrate its threshold using global-null trajectories, yielding CLM-GN. The table reports molecular-docking results under the same budget T=10T=10, averaged over 100 complete repeats.

Utility and global-null false-stopping rate in the molecular-docking experiment at T=10T=10.

Rewardα\alphaUtility: Weighted COSUtility: CLM-GNGlobal-null rate: Weighted COSGlobal-null rate: CLM-GN
Linear0.11.4750.8740.1110.115
Linear0.21.6391.5010.1450.214
Exponential0.11.1050.5350.1010.115
Exponential0.21.2600.9630.1320.214

Weighted COS achieves higher utility in all four settings with similar or lower global-null false-stopping rates, supporting the benefit of reward-aware stopping-rule design. Additional results are provided in Appendix I.

W4. The reward metric makes the practical benefit difficult to assess. The experiments use linear and exponential rewards that favor early stopping, but their connection to actual generation or validation costs is not established. Since COS directly optimizes these rewards, the reported gains do not reveal whether it detects hits more frequently, stops earlier, or both. Successful-detection probability, stopping times, and the fraction of trajectories with no alarm should be reported separately.

Response. We thank the reviewer for this suggestion. We agree that utility alone does not distinguish between more frequent successful detection and earlier stopping. Appendix I now separately reports successful-detection probabilities and mean stopping times conditional on successful detection.

These results also clarify the practical role of the reward schedule. Compared with the linear reward, the exponential reward places greater emphasis on early detection, generally leading to earlier successful stops and a smaller candidate pool for downstream validation. However, its lower successful-detection probability may require additional generation runs to obtain a successful alarm. The reward schedule can therefore be interpreted as reflecting a trade-off between generation effort and downstream validation costs. For other related discussion, please see our response to Reviewer jpEZ’s W1.

W5. The reported error rates do not directly evaluate the calibrated constraint. Calibration controls alarms under the global null, while the experiments report stopping before the first hit on mixed hit/non-hit trajectories. The latter can be substantially smaller because any alarm after a hit is counted as correct. Held-out global-null alarm rates are needed to assess calibration conservatism and interpret reward differences across methods.

Response. We thank the reviewer for highlighting this distinction. We agree that the mixed-trajectory false-alarm rate does not directly assess the global-null constraint.

Appendix I now reports global-null alarm rates alongside mixed-trajectory false-alarm rates. In several settings, weighted COS achieves the highest utility despite having a lower global-null alarm rate than competing methods. These results support the benefit of reward-aware stopping beyond differences in calibration conservatism. For other related discussion, please see our response to Reviewer NGju’s W1.

AuthorsOpenReview ↗

Response to Reviewer hGES (part III)

Q1. How would calibration work when only historical labeled data are available, without evaluating additional candidates from the deployment distribution? What happens to false-alarm control under covariate shift between the reference non-hits and newly generated candidates?

Response. Please see our responses to W1(A) and W1(B), which address calibration using historical labeled data and false-alarm control under covariate shift, respectively.

Q2. Can the authors evaluate trajectories within individual protein pockets or starting-molecule contexts? Pooling candidates across contexts does not establish performance for the fixed-context generation setting used to motivate the method.

Response. Please see our response to W2.

Q3. Given the labeled calibration data available in these experiments, can the authors compare against Conformal Language Modeling or related calibrated generative sampling methods? What additional benefit does the proposed stopping formulation provide?

Response. Please see our response to W3.

Q4. Can the authors separately report successful-detection probability, stopping times, and the fraction of trajectories with no alarm? How sensitive are the conclusions to the reward schedule, including the constant-reward setting wt=1w_t=1?

Response. Please see our response to W4 for the additional metrics and comparison of reward schedules.

The constant-reward case wt=1w_t=1 is degenerate from an early-stopping perspective. Thus, an optimal policy can be chosen to output either τ=T\tau=T, indicating that a hit is believed to have appeared by the horizon, or τ=∞\tau=\infty, indicating no alarm.

Q5. Can the authors report held-out global-null alarm rates alongside the mixed-trajectory false-alarm rates? This would help determine whether differences in reward are partly explained by calibration conservatism.

Response. Please see our response to W5.

Q6. The baseline adaptations need stronger justification. O-CONFHIT uses prefix maxima, prespecified temporal weights, and additional trajectory calibration, while omitting density-ratio weighting and backward monotonization. The comparison therefore concerns this particular adaptation.

Response. We thank the reviewer for this clarification. ConfHit is an offline method, so our original comparison uses its online adaptation, O-ConfHit. To complement this comparison, Appendix I also evaluates the original ConfHit procedure following Laghuvarapu et al. (2026), including density-ratio weighting and backward monotonization.

References

[1] Laghuvarapu, S., Jin, Y., & Sun, J. (2026). ConfHit: Conformal Generative Design with Oracle-Free Guarantees. International Conference on Learning Representations.

[2] Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T., & Barzilay, R. (2024). Conformal Language Modeling. International Conference on Learning Representations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive comments. We have revised the manuscript, with additions marked in blue, and provided detailed responses to each reviewer. We summarize the main revisions and clarifications below.

  • Single-context evaluation and covariate shift (Appendices H and I). Appendix H extends COS to covariate shift using density-ratio weighting and establishes false-alarm and utility bounds accounting for weight-estimation and calibration errors. Appendix I evaluates trajectories within individual protein pockets or starting-molecule seeds, using disjoint historical and deployment contexts. Weighted COS remains competitive against the original ConfHit procedure and other baselines, including the added CLM-GN baseline. It achieves the highest mean utility in 8 of 12 High-QED settings and the best average utility rank in molecular docking. Please see our responses to Reviewer hGES: W1–W3, W6; Reviewer jpEZ: W2(A).

  • Practical value of early stopping and reward schedules. We clarify that COS determines how many candidates to generate, while score ranking determines which candidates to validate first. Early stopping can reduce generation effort and produce a smaller candidate pool for downstream validation. Reward schedules can therefore be interpreted as reflecting a trade-off between generation effort and validation costs. Our response to Reviewer jpEZ includes a simulation comparing COS followed by ranking with generating the full pool followed by the same ranking strategy. At α\alpha = 0.1, COS reduces generations per discovered hit by 17.6% and validations by 4.0%. Please see our responses to Reviewer hGES: W4; Reviewer jpEZ: W1(A)–W1(B).

  • Separate performance and calibration metrics (Appendix I). We now report successful-detection probabilities, mean stopping times conditional on successful detection, and global-null alarm rates alongside mixed-trajectory false-alarm rates. These metrics distinguish detection frequency from stopping speed and directly assess the global-null constraint, helping interpret utility differences across methods. Please see our responses to Reviewer hGES: W4–W5.

  • Working-model misspecification and reference-size ablations (Appendices G.2.1–G.2.2). We added experiments varying the assumed hit probability, working hit-score distribution, and reference-set size. COS remains competitive under the tested misspecifications, although it does not always rank first. Smaller reference sets mainly reduce utility; global-null false-alarm control remains stable in the tested settings, including a reference size of 10. Please see our responses to Reviewer wkaq: Q1–Q2; Reviewer jpEZ: W2(B).

  • Temporal dependence (Appendix A). We added a discussion of correlated and iteratively refined candidates and derived a bound quantifying degradation of the global-null reduction. Assuming the global-null stopping probability is controlled, the additional false-alarm inflation depends on discrepancies between actual non-hit prefix distributions and their global-null counterparts. These discrepancies vanish under independence, recovering the original guarantee. Please see our responses to Reviewer NGju: Q1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.