acceptodds
Under review as a conference paper at ICLR 2027

The Curvature Floor: A Geometric Criterion for Early Stopping

Abstract

Early stopping needs a clean validation set, which self-supervised pretraining and training with noisy labels often lack. Rules without one watch the loss, the gradient or the prediction changes, which keep changing after the model stops improving. We instead monitor the effective learning rate in units of the stability threshold of gradient descent, , where is the top eigenvalue of the Fisher information, preconditioned for adaptive methods and estimated with labels sampled from the model. In seven training regimes, from ResNet-50 on ImageNet to MAE pretraining, BERT fine-tuning and 40% label noise, falls by at least a factor of five and settles at a floor. It crosses 0.05 at 5% to 15% of the budget before the best checkpoint, a lead that varies almost 700-fold in updates, so a rule should wait for a time that scales with the budget. We prove that when the statistic depends only on the budget fraction, a threshold rule on a linearly smoothed statistic stops at the same fraction of every budget, for every profile of the statistic, if and only if its kernel is rescaled with the budget at every lag within the run. We also bound the delay of the exponential moving average in closed form. The resulting rule, Curvature Floor Stopping (CFS), uses no labels while it runs and adds about 5% memory on models up to 340M parameters. With four constants set per regime on development labels of the same runs, it matches or exceeds validation-based early stopping in five of seven regimes, stays within 0.02 points in the other two and cuts the total compute by 20% to 59%, monitoring included. With one budget-scaled time constant on held-out runs, it is on par with a stronger validation rule that returns its best checkpoint, while costing less than it in six of the seven regimes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Early Stopping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aGNKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of stopping training without access to reliable validation data, as in self-supervised pretraining or when many training labels are wrong. The proposed signal is the product of effective learning rate and the largest eigenvalue of the Fisher information matrix. This metric uses the model's own predicted labels, so no real labels are needed. The authors observe that this metric decreases sharply in the course of training, and the flattens out at a low level, called the curvature floor. It drops below a chosen threshold when about 5–15% of the training budget still remain before the optimal stopping point is reached. To capture this effect, the authors suggest to exponentially smooth the metric, with exponential broadening on a time scale which is proportional to the training budget. The paper also contains theory to back up this protocol.

Critical strengths

The theory predicts that a rule whose smoothing window is fixed in absolute steps must fail when the training budget is short , which is confirmed by experiments. An ablation confirms that the stop position depends on the smoothing settings only through the single combination the theory says it should. This shows that the method was derived from a principle, and does not just rely on parameter tuning.

Every setup is run with five random seeds and per-seed numbers are published . The results barely change when the rule is rerun with 40 fresh samples of the curvature estimate. All costs include the monitoring overhead; and fresh untuned runs come with proper statistical intervals.

Critical weaknesses

The opening part of the manuscript is hard to read: key terms like "curvature floor", "best checkpoint", "crossing" are used before being defined, sentences are short and stacked without connecting words that show the logical links, and many rule parameters appear before the reader learns which ones matter. The abstract is too dense.

The paper's headline result (Table 1) is that its stopping rule, matches or beats early stopping based on a validation set in five of seven settings. The problem is how the settings were picked: for each task, the authors tried 35 to 48 combinations of four parameters, and picked the best combination using performance measured on the very same five runs that appear in Table 1. Choosing the settings by looking at the test performance of those runs tends to make the reported numbers look better than the method would perform on a fresh run. In additiion, for the SST-2 task (the BERT column), no public test labels exist, so the score shown is on the same data used for tuning. The tuning objective itself (Eq. 39) rewards settings that stop near the best checkpoint, i.e., it uses the answers to pick the settings.

The rule that the "stop signal arrives 5–15% before the best moment" holds only for training schedules in which the learning rate only decreases over time. The paper's own failure experiments (Section J) show what with a cyclic schedule (the learning rate periodically rises again), the rule stops far too early.

The theoretical assumption that the monitored quantity depends only on the fraction of the budget spent is supported by comparing different tasks at different budgets, not by re-running one identical recipe at several budgets. From a theoretical point of view, this relation can only hold for "well chosen budgets" which are adapted to the problem at hand, since in principle the budget could be increased tenfold, and when keeping all other hyperparameters unchanged this would not influence the delay between crossing of the metric and optimal stopping.

Questions for authors

Can Table 1 be regenerated with constants calibrated on runs disjoint from those reported, for instance by using results from appendix G? This would resolve whether the "five out of seven" margin survives independent calibration.

How was the single held-out value c = 0.012 chosen (Sec. C.3, App. G)? Did the authors compare alternatives in [0.012, 0.017] ?

Will the implementation of monitoring (forward-mode Fisher products, warm starts, SNR test) be released? The ~5% memory (vs. 232% for Hessian products, Table 30) and per-check costs underpin the compute-saving claims and cannot currently be checked.

Reviewer_GnQeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an early stopping rule based on a stability condition involving the eigenvalue of a Fisher matrix and an effective learning rate. The authors analyze the stopping rule to show that it stops at the same fraction of a given training budget under some conditions. They provide extensive experiments showing it to be effective across architectures and datasets.

Critical strengths
  • The experiments cover a broad range, involving many different architectures and datasets, and appear relevant to training models at relevant scales.
Critical weaknesses
  • The presentation significantly impedes clarity, making the contribution are hard to evaluate. The effective learning rate is repeatedly used before it is defined. The text has many paragraphs that list precise numerical values without explaining their significance. Some broad claims are made that aren't obviously supported, such as the opening line "Early stopping needs a clean validation set" which I have not seen demonstrated. Moreover, the terms used without clear introduction (curvature matrix is Hessian or Fisher?), only defined much later (development vs validation) or undefined (recipe in assumption 1). As these issues add up, the merits of the paper become less and less certain.
  • The connection between learning and the stability statistic that is derived is not established well enough. Lemma 1 explains that the Fisher can become small as predictions become confident regardless of whether they are correct. Later, Theorem 1 does not obviously explain why the stopping point would be a good one. The theoretical sections are very dense, making it difficult to understand or connect with existing work on neural network geometry.
  • The empirical comparisons and computational costs are difficult to interpret: In Table 1, the proposed method is evaluated on the runs used to tune its constants. Only later are results presented on held-out runs. For a run in a new setting that requires tuning,it is not clear if there would be any computational savings after accounting for tuning.
Questions for authors
  • What evidence supports Assumption 1 when the "recipe" is held fixed with the budget changes? It's unclear to me how Figure 1d is meant to support this, given that it is said to vary the "recipe"
  • How does curvature floor stopping compare on held-out runs to a fixed stopping fraction of a given training budget selected using the same calibration runs and labels?
  • What is the total computational cost of tuning the constants, and is it included in the reported savings?
  • With the architecture fixed, how well do the hyperparameters tuned on one dataset transfer to a new dataset without retuning?
Reviewer_KL3uOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies early stopping in neural networks and proposes a practical measure of early stopping that doesn’t require any labeled set via a geometric view. The metric is computed using the eigenvalue of the sampled Fisher matrix and the effective learning rate. The paper then shows empirically that the proposed algorithm outperforms baseline early stopping methods and shows its performance is very close to the oracle optimum performance. They also provide ablation studies in order to fully analyse the effectiveness of each parameter in their proposed metric.

Critical strengths
  1. The idea presented in the paper that the smoothened average would simply give us the lag required for the optimum early stopping point is very interesting. This gives also the tailoring required for different experimental settings. This makes the metric simple, intuitive and of practical value. The metric also doesn’t introduce too much of a computational overhead (+5% memory is raised which is rather affordable).
  2. The proposed metric and the use of the model predictions in the Fisher computations instead of using the training set labels is, to the best of my knowledge, novel.
  3. The empirical results show that the proposed metric outperforms almost all the baselines. Moreover, the results are very close to the optimum oracle access case.
Critical weaknesses
  1. While the authors present their comparison to their closest baselines, the proposed early stopping criteria needs to be compared with other baseline metrics as well. Why were only IES and Label Wave baselines selected? The authors mention a few other baselines metrics in the related work sections which all work without requiring any labeled validation set and are also practically very cheap to compute. So it would be important to complete the empirical results with including a full comparison with those other baseline metrics as well.
  2. The paper doesn’t provide an empirical result for the regime where the classification dataset is imbalanced. Particularly because Lemma 1 is related to how much the model focuses on a specific class. It is hence important to also show how the metric works where the tail classes have much less samples in the training set.
Questions for authors

Figure 1(d) has 6 settings, but in the introduction text the authors mention 7 different regimes: “No single wait fixed in updates fits all seven regimes”. Is the regime here the 6 settings presented in the Figure or is it something else?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response 2: What the paper does and what is new, in plain terms

Dear AC and reviewers,

Thank you for the reviews and the discussion. Two reviews found the opening hard to read, and the first revised version addresses part of this. As the discussion closes, we restate in plain terms what the paper does and what is new.

The problem. Early stopping usually watches a clean validation set and stops when it stops improving. Such a set is missing in self-supervised pretraining, under noisy labels and in federated training. Without it, one trains for the full budget or guesses when to stop.

The idea. CFS watches one number that needs no labels: the top eigenvalue of the Fisher information, computed with labels sampled from the model itself, times the effective step size. During training this number first stays near the stability limit of the optimizer, then falls sharply and flattens at a low level, the curvature floor. Its crossing of a fixed threshold marks the end of fast learning along the sharpest direction. The best checkpoint comes after a slower refinement phase, 5.3% to 15.0% of the budget later. CFS waits for this phase by smoothing the signal over a window that grows with the budget, and then stops.

Why it works without labels. Under label noise, networks fit the clean patterns first and the wrong labels later. Signals built on the given labels, such as the gradient, stay high while the network memorizes wrong labels. The Fisher with labels sampled from the model carries none of these wrong labels, so it falls once the clean patterns are fit. On CIFAR-10 with 40% label noise, CFS stops at epoch 75.1 on average, next to the best checkpoint on the clean test set at epoch 74.9.

What is new.

  1. A stopping signal that reads no labels. The paper monitors the Fisher computed with the model's own predicted labels, scaled by the effective step size. Reviewer KL3u notes that this use of the model's predictions is, "to the best of my knowledge, novel".
  2. A wait that scales with the budget, derived rather than tuned. Theorem 1 proves that if the crossing leads the optimum by a share of the budget, a smoothing window proportional to the budget is necessary and sufficient to stop at the same share of every budget. A window fixed in steps must then fail on short budgets, and the experiments confirm it. Reviewer aGNK notes that the method "was derived from a principle, and does not just rely on parameter tuning".
  3. A cheap monitor. Products with the Fisher need no second derivatives, so monitoring adds about 5% memory, against 232% for Hessian products.

What the evidence shows. On runs that took no part in calibration, with every selection made on development data and every number reported on a sealed test split, CFS beats the validation rule of Table 1 in 7 of 7 settings in one direction and in 5 of 7 in the other, with the other two within 0.02 points. One shared constant matches or beats the stronger validation rule that returns its best checkpoint in 7 of 7 settings (Table R1). CFS also beats every applicable rule without validation data that Related Work names, in all seven regimes (Table R6), and a fixed stopping fraction tuned per task on the same runs and labels in 64 of 84 held-out comparisons (Table R5). On held-out runs it saves 19.5% to 60.3% of the compute per run (Table 25), and on long-tailed CIFAR it saves 30% to 32% while keeping the few-shot classes 2.60 points above the full run (Table R7).

What the reviews credit. Reviewer KL3u finds the signal novel and its cost affordable, Reviewer aGNK finds the rule principled, and Reviewer GnQe finds that the experiments "appear relevant to training models at relevant scales". In the terms of the ICLR reviewer guidelines, the paper contributes new knowledge, theoretical and empirical, with a direct use for practitioners: a label-free signal for when learning has run its course, a proof of how long to wait after it, and a rule that stops near the best checkpoint at a fraction of the cost.

We thank the AC and the three reviewers again for their time and careful reading.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.