- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a problem: one-hit detection in sequential generative sampling candidates arrive online. The goal of the problem is to stop as soon as there is enough evidence that at least one hit, saving costly experimental feedback. It is formulated as a finite-horizon constrained optimal stopping problem, maximizing expected reward for early correct detection under a global-null false-alarm constraint. With independent p-values and a two-group working alternative, Lagrangian duality yields a monotone family of Bellman-optimal rules (Thm. 1). For real scores, the proposed Conformalized Optimal Stopping applies this family to conformal p-values and calibrates the multiplier on simulated null trajectories. This gives finite-sample false-alarm control under exchangeability, even if the working alternative is misspecified (Thm. 2), and a reward gap to the oracle of O(√(log n/n) + 1/√B) (Thm. 3).
Contributions: The paper introduces a new, well-motivated problem that is distinct from prior work, and gives an exact characterization of its optimal stopping rule together with a tractable finite-partition implementation. It also proposes a calibration method that uses pathwise monotonicity to handle the dependence induced by the shared reference sample, so that validity does not depend on the working alternative being correct. Finally, it provides efficiency guarantees that separate the effects of the reference sample size and the number of calibration trajectories.
Experiments on synthetic data, QED–HGraph, and TargetDiff show COS with the highest reward in all settings (gains of roughly 8%–29% over the best baseline), with false-alarm rates below nominal levels as guaranteed.
Critical strengths
1. The problem setting is interesting and well-motivated.
The problem is well-motivated and can have real impact. Online one-hit detection ties directly to a real cost in scientific discovery (stopping generation and triggering costly validation).
2. The solution is principled and with a clean theory.
Lagrangian duality gives a monotone family of Bellman-optimal rules and exact constrained optimality (Thm. 1). The finite-partition implementation comes with an explicit approximation-loss bound, the method is both optimal in principle and computable in practice.
3. Experimental results have significant improvement on real world settings.
COS achieves the highest mean reward in every setting tested: all 16 synthetic settings and all 12 settings on each of QED–HGraph and TargetDiff. The margins over the strongest baseline are substantial (about 8%–29%). The results hold across two reward schedules, multiple horizons, and multiple α levels, and the gap tends to widen at larger T. All methods are recalibrated on complete null trajectories, so the comparison is made under the same finite-sample false-alarm guarantee, and the real-data benchmarks span both generative molecular design and docking.
Critical weaknesses
1. Baseline strength is unclear.
The baselines are the authors' own adaptations of existing methods (e.g., O-CONFHIT is an online variant of an offline procedure). How much tuning each received is not very clear, part of COS's reported gain may reflect how the baselines were adapted.
2. The false-alarm constraint appears conservative.
Realized false-alarm rates are well below the nominal level (e.g., roughly 0.06 at α = 0.10 and 0.25 at α = 0.40 in Table 1), which the authors acknowledge. Because the global-null constraint is loose for any particular generation mechanism, it is hard to separate the benefit of the optimal stopping design from more efficient use of unused budget. It would help to compare rewards at matched realized false-alarm rates, or to explore ways to reduce the conservatism.
Questions for authors
-
The reduction from false-alarm control to the global-null stopping probability (Prop. A.1) requires candidates to be independent across time. Could the authors comment on how the guarantee degrades for generators that produce correlated or iteratively refined samples?
-
See weakness 1.
-
See weakness 2.
2 comments
I appreciate the authors’ response, I continue to support acceptance.
Response to Reviewer NGju
Response. We thank the reviewer for raising this concern. We have clarified the tuning and calibration protocol as follows.
Common calibration. Because conformal p-values share a reference set and are dependent, we recalibrate the final rejection threshold of every method, including COS, LR-e, OneHit, Stouffer, IMT-Alt, and O-CONFHIT, using the same independent conformal-null trajectories.
Global-null diagnostic. The table reports held-out global-null false-alarm rates for the Beta(1.5,3) experiment with T=30 and linear reward. Each entry is averaged over 100 repeats, with 9,999 held-out null trajectories per repeat.
Most baselines, including the strongest baseline LR-e, are no more conservative than COS. Therefore, COS's advantage cannot be attributed to a more permissive calibration or unfavorable baseline tuning. It arises from its reward-aware optimal stopping rule under the two-group working model.
Response. Thank you for this suggestion. The methods are compared at approximately matched global-null alarm rates, making the reward comparison meaningful under our common constraint.
We use this conservative constraint because the hit-generation mechanism is generally unknown. It ensures false-alarm control under our assumptions without requiring accurate specification of that mechanism.
Direct calibration of the realized false-alarm rate is a promising direction for future work. A setting with labeled calibration trajectories, similar to Conformal Language Modeling (Quach et al., 2024), could support such control, but requires a different data structure from our current setting.
Response. Thank you for this question. Appendix A now quantifies the degradation under temporal dependence. If P0(τ≤T)≤α, then
P(τ<ν∗)≤α+∑t=1TP(Ht)δt,Ht=A1=⋯=At=0,
where δt=dTV!(LP(Y1:t∣Ht),LP0(Y1:t)). The additional term measures the discrepancy between non-hit prefix distributions under the actual process and the global null. For correlated or iteratively refined samples, conditioning on future samples also being non-hits can alter earlier prefix distributions. Under independence, these discrepancies vanish, recovering Proposition A.1.
References
[1] Laghuvarapu, S., Jin, Y., & Sun, J. (2026). ConfHit: Conformal Generative Design with Oracle-Free Guarantees. International Conference on Learning Representations.
[2] Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T., & Barzilay, R. (2024). Conformal Language Modeling. International Conference on Learning Representations.