- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies safe stopping in sequential data acquisition. Measurements arrive progressively, and the system must decide when to stop acquiring data and commit to a prediction. The paper argues that existing acquisition policies and heuristic stopping rules generally optimize average accuracy, cost, or reward, without controlling the risk among the samples on which the system actually stops.
The authors propose Risk-Controlled Stopping (RCS), a post-hoc, distribution-free procedure based on the Learn-Then-Test framework. Each candidate stopping rule is treated as a trajectory-level selective predictor. RCS evaluates the risk only on samples for which the rule stops, applies multiple-testing correction with Holm’s step-down procedure, and selects a certified rule according to a user-specified utility. The method can control overall misclassification risk or class-conditional FNR/FPR, and can be applied to fixed, random, or learned acquisition policies without retraining.
The paper evaluates RCS on sequential CIFAR-10 patch acquisition, FAST MRI knee k-space acquisition, an internal prostate MRI cohort, and a learned JAFA acquisition policy.
Critical strengths
-
The paper identifies an important distinction between controlling marginal risk over the full population and controlling risk among the samples on which a system actually commits to an early decision.
-
RCS is a model-agnostic post-hoc wrapper that can certify fixed or learned acquisition policies without retraining and can control class-conditional FNR/FPR.
Critical weaknesses
-
Limited positioning and comparison with the closest related work.
The paper’s main distinction from prior work on risk-controlled early stopping and sequential prediction is not fully established. In particular, Ringel et al. (2024), Gauthier et al. (2025), and Jazbec et al. (2024) are discussed but not systematically compared, and the experiments do not include the most closely related risk-control baselines. -
Theoretical scope and practical assumptions are insufficiently clarified.
The main guarantee is a direct application of the Learn-Then-Test framework rather than a new theoretical result. Its validity depends on the candidate policy family being fixed before calibration and on calibration and test trajectories following the same induced distribution. The relationship between the theorem and the fallback behavior is also unclear: when no policy is certified, full acquisition is used, but full acquisition itself is not shown to satisfy the target conditional risk. -
Insufficient empirical validation of the statistical guarantee.
The experiments show empirical risk curves, but do not directly evaluate the probability of successful certification associated with the guarantee. The effects of calibration-set size, , the number and spacing of candidate policies, and the number of stopped samples are not systematically studied. -
Limited analysis of practical behavior and design choices.
The evaluation covers several datasets and acquisition settings, but the experimental analysis remains largely descriptive. It does not sufficiently examine empty certified sets, low-stopping-rate regimes, distribution shift, adaptive-policy trajectory mismatch, or the contributions of the main statistical design choices such as Holm correction, p-value construction, policy-family parameterization, and fallback strategy.
Questions for authors
Please see Weaknesses.
4 comments
The rebuttal clarifies the theoretical scope and fallback behavior, and the reported sensitivity experiments address several concerns about calibration and statistical power. However, direct comparisons with the closest risk-control methods remain missing, and the acknowledged within-patient slice dependence limits the formal certification claims supported by the FastMRI experiments. These responses improve the paper but do not fully resolve the main concerns. I would maintain the rating of 2 (Weak rejection).
- As we state in our response to reviewer gWJT, we have added multiple calibration-based stopping baselines for all three datasets.
We added two calibration-based stopping baselines across all three datasets (Appendix A.3, Figures 6–8). At each sampling rate, Operating Point selects a threshold targeting empirical risk over all calibration examples, while Selective Inference targets empirical risk among the stopped samples. Both use the same nominal risk target as RCS, but at a per-sampling rate.
On FastMRI Knee, RCS stops 13% of slices at 5% sampling with zero observed ACL FNR and meniscus FNR at most 0.012. Selective Inference also satisfies the empirical risk target but stops no slices at 5% or 8%, while achieving greater stopping coverage at later stages. On prostate MRI, RCS stops 36–53% of examinations with observed FNR 0.026–0.064, whereas Operating Point (0.14–0.17) and Selective Inference (0.105–0.14) exceed the target of 0.1. These comparisons evaluate the empirical risk–efficiency trade-off under a common nominal calibration target; they do not establish universal efficiency superiority or exact matching of held-out FNR.
-
Additionally, we have added comparisons to Ringel et al as well, see figures 12, 13, and 14 in the appendix. We observe that RCS controls the risk over the stopped samples while Ringel et al does not. Their method is constructed to control the risk relative to the model with the fully-sampled data as input, which may or may not control the risk at the desired level, as opposed to RCS which controls the risk at a user-specified level not relative to the model being used.
-
On the CIFAR-10 and Prostate MR experiments, we do have exchangeability as the samples are independent and the results show that RCS controls the risk at a desired risk level, while the baselines do not. Moreover, in the FastMRI knee experiments, despite exchangeability potentially being violated, therefore a more challenging setting, the risk is controlled at the desired level.
Thank you for your valuable feedback, which has greatly improved our paper. As the discussion phase wraps up, we believe we have addressed your final comment. Please let us know if any further clarification is needed.
We thank the reviewer for the thoughtful comments on novelty, theoretical scope, and empirical validation.
Positioning relative to prior work. We agree that the finite-sample guarantee follows directly from LTT and do not claim a new concentration inequality. Our contribution is formulating sequential acquisition stopping as a trajectory-level selective prediction problem with absolute ground-truth risk conditional on stopping. Unlike Ringel et al., which control disagreement with the full-data prediction, RCS controls the prediction error itself. Unlike computational early-exit methods such as Jazbec et al., RCS addresses external measurement acquisition. This distinction matters because absolute risk admits no automatically safe full-acquisition configuration and need not be monotone across stopping policies. The paper also situates Gauthier et al. in the broader sequential risk-control literature.
Theoretical scope and fallback. We agree that validity requires the candidate policy family to be fixed independently of risk calibration and calibration/deployment trajectories to follow the same induced distribution. An empty certified set indicates that no candidate could be certified from the available data, not that no safe policy exists. Full acquisition is an operational fallback rather than an automatically certified predictor. We have corrected these distinctions throughout the manuscript.
Calibration sensitivity and statistical power. We added systematic sensitivity experiments across all three datasets (Appendix A.4, Figures 9–11), varying significance level δ, candidate-grid size ∣Λ∣, and calibration-set size ncal. The main finding is that certification is primarily limited by effective calibration sample size, particularly for rare positive classes:
These results characterize the statistical power required for certification, sensitivity to grid discretization, and behavior when no candidate can be certified. They demonstrate that an empty certified set can result from insufficient calibration evidence rather than the absence of safe policies.
Empirical validation of the guarantee. We agree that the sensitivity experiments do not directly estimate the repeated-calibration probability of incorrectly certifying an unsafe policy. However, the finite-sample guarantee is established analytically through valid policy-level hypothesis tests and Holm's family-wise error correction under the stated independence assumptions. Repeated-calibration trials would provide an additional empirical diagnostic rather than a prerequisite for the theoretical result. We have clarified this distinction and do not claim that the new experiments constitute direct empirical verification of the 1−δ probability statement.
Practical design choices. Holm's procedure provides family-wise error control without requiring a monotone ordering of candidate risks, while exact binomial testing is tighter than general bounded-loss Hoeffding bounds for binary FNR/FPR. The new experiments systematically assess sensitivity to significance level, candidate-grid density, and calibration sample size. We acknowledge that these analyses do not independently compare alternative multiple-testing corrections, p-value constructions, or policy-family parameterizations. The latter remain interchangeable components of the RCS framework, with their choice influencing statistical power and achievable stopping efficiency.
Distribution shift and adaptive-policy mismatch. RCS requires calibration trajectories representative of the deployed acquisition process. Arbitrary distribution shift or changes to the adaptive acquisition backbone can invalidate its certificate. We now state these assumptions explicitly and identify representative recalibration or shift-aware procedures as potential approaches for deployment under distribution shift.