acceptodds
Withdrawn from ICLR 2027

What Drives Early-Stopping Gains? Threshold Policies and an MRI Case Study

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Early-stopping systems use intermediate scores to decide when further computation or measurement is unnecessary. Yet greater resource savings can reflect better within-stage ranking, changes in cross-stage numerical scale, more flexible threshold policies, or a different fitting constraint, obscuring what improved. We study these dependencies for threshold-based stopping along fixed sequences, using repeated MRI as a case study. Sharing one threshold couples decisions across stages, so adding checkpoints can reduce attainable savings. Stage-specific thresholds can reproduce any shared-threshold policy that uses only a subset of checkpoints; with zero training failures, a greedy construction stops every training case no later than any feasible policy in this class. Strictly increasing stage-wise score transformations preserve stopping decisions when thresholds are transformed accordingly, distinguishing within-stage ranking from cross-stage numerical scale. Mechanism analyses in repeated prostate MRI show that these dependencies can change method comparisons: rescaling scores preserves within-stage ranking yet can reverse mean savings comparisons between scoring methods, while relaxing the training failure budget largely removes the advantage of fewer checkpoints. Brain MRI also exhibits scale sensitivity. The choice of baseline further changes the interpretation: in a separate test of prespecified policies, a strong fixed-budget control captures nearly all the primary adaptive policy’s mean saving, leaving its additional advantage unresolved. Early-stopping savings thus reflect the score, stopping policy, and fitting procedure jointly; comparisons require explicit policy classes, matched fitting constraints, and strong fixed-budget controls.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
early stopping, threshold policies, evaluation methodology, sequential decision-making, adaptive acquisition, magnetic resonance imaging

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K24rOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes how score scaling, checkpoint selection, threshold flexibility, and training-failure constraints affect comparisons of early-stopping systems. It compares shared and stage-specific threshold policies and establishes a greedy optimality result under zero training failures. The empirical evaluation uses repeated diffusion-weighted MRI acquisitions, where early stopping reduces the number of repetitions averaged per diffusion direction. The results demonstrate that savings comparisons depend on both policy design and fitting choices. A held-out evaluation also shows that a fixed-budget baseline achieves nearly the same mean savings as the primary adaptive policy.

Critical strengths

The idea of explaining the gain of early-stopping is interesting. The paper isolates the effects of checkpoint availability, threshold flexibility, and stage-wise score scaling while keeping the observation trajectories fixed. This provides examples of why resource savings cannot always be attributed to improved within-stage ranking.

Critical weaknesses

Insight beyond an intuitive observation is unclear. The general observation that early-stopping savings depend on the score, stopping policy, and fitting procedure is intuitive. The paper formalizes specific mechanisms and recommends explicit policy classes, matched fitting constraints, and strong fixed-budget baselines. However, it does not sufficiently demonstrate what additional insight these results provide beyond established evaluation principles, or how they lead to better policy-selection decisions in practice.

The dependence of the observed policy gap on score quality and risk calibration remains unclear. The analyses demonstrate disadvantages of shared thresholds for fixed scores, but do not examine whether these gaps persist with more accurate prediction of downstream loss or failure probability. It also remains untested whether substantial efficiency gaps persist when both policy classes are calibrated to a common downstream risk requirement. Such comparisons would clarify the practical importance of threshold flexibility beyond improved scores and risk-calibrated policy selection.

Theoretical depth and practical significance are limited. Proposition 2 follows directly from monotonicity and order preservation. Proposition 1 provides a more substantive pointwise training-set optimum, but only under zero empirical failures. Its practical significance beyond this restricted fitting regime remains unclear.

Limited precision in policy-level risk comparisons. Differences involving only a few failures among 107 development patients provide limited evidence of population-risk differences between policies. Zero failures among 45 held-out patients likewise does not establish risk equivalence; the reported one-sided 95% risk upper bound is 6.44%. Although appropriately acknowledged, this uncertainty limits conclusions about efficiency at comparable population risk.

Questions for authors

Please refer to my comments on the weaknesses above.

Reviewer_pu5DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The present paper studies how score scaling, checkpoint selection, and fitting constraints affect resource savings in early-stopping systems along fixed sequences. The authors show that adding checkpoints under a shared threshold can reduce attainable savings. They provide a greedy procedure that gives the earliest feasible training stops within the stage-specific threshold class under zero training failures. They also show that strictly increasing score transformations at each stage preserve stopping decisions when the thresholds are transformed accordingly. Prostate and brain MRI experiments demonstrate scale sensitivity despite unchanged rankings within each stage. A separate frozen prostate evaluation shows that a fixed-budget baseline fitted under a more permissive training failure budget captures nearly all the primary adaptive policy's mean saving.

Critical strengths
  1. The greedy fitting result provides a useful reference for comparing threshold policies. Under the stated assumptions, it gives the earliest feasible stopping time for every training case among stage-specific threshold policies with zero training failures. This establishes the maximum attainable training savings in that setting and helps explain the effect of threshold restrictions.

  2. The controlled score-rescaling experiments directly demonstrate sensitivity to cross-stage score scaling. They preserve the rankings within each stage while changing shared-threshold savings on both prostate and brain MRI. This helps distinguish the effect of score scaling from improvements in ranking when interpreting reported gains.

Critical weaknesses
  1. The methodological advance over prior work is not sufficiently clear. Fast yet Safe by Jazbec et al. (2024) already identifies shared thresholds as a restriction and discusses relaxing it. Therefore, the motivation for allowing different thresholds at different stages is already present in prior work. I find the policy-class inclusion and score-transformation arguments relatively straightforward once the threshold classes are specified, with the greedy fitting guarantee being the more substantive formal result. The controlled rescaling experiments also provide useful evidence, but the additional methodological insight should be explained more clearly. The authors should distinguish the new fitting guarantee and empirical diagnosis from the previously recognized restriction. This matters because the contribution depends on what these results add to our understanding of early stopping beyond confirming that threshold flexibility affects efficiency.

  2. The practical implications of the fixed-budget comparison remain limited. The primary adaptive policy is fitted with zero training failures, whereas the fixed-eight-repeat baseline is allowed a 5 percent training failure budget. Their mean savings after measured image-domain computation are 24.39 and 24.00 percent, respectively. The difference is 0.39 percentage points, with a paired 95 percent interval from -2.01 to 2.80 percentage points, leaving the additional adaptive advantage uncertain. Both policies have zero observed failures among 45 test patients, but this does not establish comparable population risk. The authors acknowledge these limitations. The development comparisons under matched training failure budgets also retain an adaptive advantage in saving: at a 2.5 percent budget, dense and two-checkpoint residual policies save 29.51 and 29.38 percent, compared with 24.00 percent for the fixed policy, with three versus two out-of-fold failures. Taken together, these results show that baseline choice and fitting constraints affect the comparison. They leave the benefit of adaptation at matched population risk unresolved, which limits the practical conclusion about choosing between fixed and adaptive stopping.

Questions for authors
  1. How do the greedy fitting guarantee and rescaling analysis extend the shared-threshold limitation discussed in Fast yet Safe? In particular, which conclusions depend on the exact greedy result, and which follow directly from allowing separate thresholds at each stage? A direct comparison of the settings, fitting objectives, and guarantees would help clarify the contribution. It would also help to explain what additional insight the controlled rescaling analysis provides beyond the previously identified threshold restriction. This clarification would affect my assessment of the methodological novelty.

  2. Are the frozen protocol, policy-fitting code, and per-patient outputs described in the paper available for review? The paper provides rounded threshold values and describes a saved protocol containing the full-precision settings and code hashes. Access to these materials would help reproduce the stopping decisions and paired savings comparisons. It would also allow the implementation of the original quantile-grid fits and the later exact-search analyses to be checked against the described procedures. This would improve the ability to assess reproducibility, since the comparisons depend on the fitting constraints and threshold-search procedures.

Reviewer_FYuZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks where the savings of threshold-based early stopping actually come from: a better score ranking, a different cross-stage score scale, a more flexible threshold class, or a different fitting constraint. It formalises stopping with checkpoints and thresholds fitted under a training failure budget, and proves three short results: stage-specific thresholds contain shared ones (Lemma 1), a greedy rule is optimal at zero training failures (Prop. 1), and decisions are invariant to matched monotone maps (Prop. 2). The case study uses repeated-average prostate DWI from fastMRI (107 development patients, a frozen 12-policy test on 45 patients). The main findings are that a synthetic stage-dependent rescaling changes shared-threshold savings and can flip a small score ranking, that the sparse-checkpoint advantage mostly disappears at a 2.5% budget, and that the primary adaptive policy saves only 0.39 pp [−2.01, 2.80] more than a fixed k = 8 control.

Critical strengths
  1. The protocol is unusually rigorous and transparent. Policies are frozen before testing, thresholds and exclusions are documented, and exploratory analyses are labelled as such. The statistics are reported honestly, including contrasts that go against the primary policy and an explicit note that 0/45 failures does not establish a 5% risk guarantee. This is the main reason I take the empirical findings at face value.

  2. The fixed-budget control reframes the headline result. Showing that a fixed k = 8 policy captures nearly all of the 12.39 pp gain is a sharp, useful observation, and it illustrates the paper's own point about attributing early-stopping gains correctly.

Critical weaknesses
  1. The question it asks (where early-stopping savings come from) is a good one, but the answers are close to definitional. Stage-specific thresholds contain shared ones, a greedy rule is optimal when no failures are allowed, and decisions don't change under matched monotone maps; each follows in a few lines. The practical lessons, that shared thresholds are sensitive to score scale and that comparisons should be made at matched error against a fixed-budget baseline, are already standard in early-exit work such as MSDNet, Shallow-Deep Networks, BranchyNet and CALM, none of which are cited. The paper would need either theory that goes beyond the zero-failure case (e.g., positive budgets or out-of-sample guarantees), or evidence that these pitfalls have actually led published comparisons to the wrong conclusion.

  2. The paper is framed around early-exit methods in general, but all the evidence comes from a single prostate DWI setting with 45 test patients, and the authors admit effects outside MRI are untested. This setting is also a poor test of adaptivity. Almost every patient can stop at stage 6 or 7 (100/107, Fig. 2a), so it is unsurprising that a fixed budget does as well as the adaptive policy. All 12 policies have zero test failures, so the test can't separate them on risk. The headline scale "reversal" comes from a rescaling the authors chose and rests on a 0.26 pp gap with no interval. The findings may therefore say more about this dataset than about early stopping. At least one standard early-exit benchmark (e.g., BERT on GLUE or a multi-exit CNN) is needed to support the general claims.

Questions for authors
  1. Can you show a published early-exit comparison whose conclusion changes under your controls?
  2. What are the paired CIs for the OAS vs. Gaussian MC reversal at β = 0 and β = 1?
  3. On test, how do the primary policy and fixed k = 7/8 compare when fitted at the same budget (0%, 2.5%, 5%)?
  4. How much of the "fixed control captures nearly everything" result do you attribute to low patient heterogeneity?
  5. Do Props. 1–2 extend to positive failure budgets?
  6. How do LTT-calibrated stage-specific thresholds compare at matched nominal risk, given that the greedy policy has zero margin and overfits (1–3 out-of-fold failures)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.