- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes how score scaling, checkpoint selection, threshold flexibility, and training-failure constraints affect comparisons of early-stopping systems. It compares shared and stage-specific threshold policies and establishes a greedy optimality result under zero training failures. The empirical evaluation uses repeated diffusion-weighted MRI acquisitions, where early stopping reduces the number of repetitions averaged per diffusion direction. The results demonstrate that savings comparisons depend on both policy design and fitting choices. A held-out evaluation also shows that a fixed-budget baseline achieves nearly the same mean savings as the primary adaptive policy.
Critical strengths
The idea of explaining the gain of early-stopping is interesting. The paper isolates the effects of checkpoint availability, threshold flexibility, and stage-wise score scaling while keeping the observation trajectories fixed. This provides examples of why resource savings cannot always be attributed to improved within-stage ranking.
Critical weaknesses
Insight beyond an intuitive observation is unclear. The general observation that early-stopping savings depend on the score, stopping policy, and fitting procedure is intuitive. The paper formalizes specific mechanisms and recommends explicit policy classes, matched fitting constraints, and strong fixed-budget baselines. However, it does not sufficiently demonstrate what additional insight these results provide beyond established evaluation principles, or how they lead to better policy-selection decisions in practice.
The dependence of the observed policy gap on score quality and risk calibration remains unclear. The analyses demonstrate disadvantages of shared thresholds for fixed scores, but do not examine whether these gaps persist with more accurate prediction of downstream loss or failure probability. It also remains untested whether substantial efficiency gaps persist when both policy classes are calibrated to a common downstream risk requirement. Such comparisons would clarify the practical importance of threshold flexibility beyond improved scores and risk-calibrated policy selection.
Theoretical depth and practical significance are limited. Proposition 2 follows directly from monotonicity and order preservation. Proposition 1 provides a more substantive pointwise training-set optimum, but only under zero empirical failures. Its practical significance beyond this restricted fitting regime remains unclear.
Limited precision in policy-level risk comparisons. Differences involving only a few failures among 107 development patients provide limited evidence of population-risk differences between policies. Zero failures among 45 held-out patients likewise does not establish risk equivalence; the reported one-sided 95% risk upper bound is 6.44%. Although appropriately acknowledged, this uncertainty limits conclusions about efficiency at comparable population risk.
Questions for authors
Please refer to my comments on the weaknesses above.