- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tackles the problem of stopping training without access to reliable validation data, as in self-supervised pretraining or when many training labels are wrong. The proposed signal is the product of effective learning rate and the largest eigenvalue of the Fisher information matrix. This metric uses the model's own predicted labels, so no real labels are needed. The authors observe that this metric decreases sharply in the course of training, and the flattens out at a low level, called the curvature floor. It drops below a chosen threshold when about 5–15% of the training budget still remain before the optimal stopping point is reached. To capture this effect, the authors suggest to exponentially smooth the metric, with exponential broadening on a time scale which is proportional to the training budget. The paper also contains theory to back up this protocol.
Critical strengths
The theory predicts that a rule whose smoothing window is fixed in absolute steps must fail when the training budget is short , which is confirmed by experiments. An ablation confirms that the stop position depends on the smoothing settings only through the single combination the theory says it should. This shows that the method was derived from a principle, and does not just rely on parameter tuning.
Every setup is run with five random seeds and per-seed numbers are published . The results barely change when the rule is rerun with 40 fresh samples of the curvature estimate. All costs include the monitoring overhead; and fresh untuned runs come with proper statistical intervals.
Critical weaknesses
The opening part of the manuscript is hard to read: key terms like "curvature floor", "best checkpoint", "crossing" are used before being defined, sentences are short and stacked without connecting words that show the logical links, and many rule parameters appear before the reader learns which ones matter. The abstract is too dense.
The paper's headline result (Table 1) is that its stopping rule, matches or beats early stopping based on a validation set in five of seven settings. The problem is how the settings were picked: for each task, the authors tried 35 to 48 combinations of four parameters, and picked the best combination using performance measured on the very same five runs that appear in Table 1. Choosing the settings by looking at the test performance of those runs tends to make the reported numbers look better than the method would perform on a fresh run. In additiion, for the SST-2 task (the BERT column), no public test labels exist, so the score shown is on the same data used for tuning. The tuning objective itself (Eq. 39) rewards settings that stop near the best checkpoint, i.e., it uses the answers to pick the settings.
The rule that the "stop signal arrives 5–15% before the best moment" holds only for training schedules in which the learning rate only decreases over time. The paper's own failure experiments (Section J) show what with a cyclic schedule (the learning rate periodically rises again), the rule stops far too early.
The theoretical assumption that the monitored quantity depends only on the fraction of the budget spent is supported by comparing different tasks at different budgets, not by re-running one identical recipe at several budgets. From a theoretical point of view, this relation can only hold for "well chosen budgets" which are adapted to the problem at hand, since in principle the budget could be increased tenfold, and when keeping all other hyperparameters unchanged this would not influence the delay between crossing of the metric and optimal stopping.
Questions for authors
Can Table 1 be regenerated with constants calibrated on runs disjoint from those reported, for instance by using results from appendix G? This would resolve whether the "five out of seven" margin survives independent calibration.
How was the single held-out value c = 0.012 chosen (Sec. C.3, App. G)? Did the authors compare alternatives in [0.012, 0.017] ?
Will the implementation of monitoring (forward-mode Fisher products, warm starts, SNR test) be released? The ~5% memory (vs. 232% for Hessian products, Table 30) and per-check costs underpin the compute-saving claims and cannot currently be checked.
General Response 2: What the paper does and what is new, in plain terms
Dear AC and reviewers,
Thank you for the reviews and the discussion. Two reviews found the opening hard to read, and the first revised version addresses part of this. As the discussion closes, we restate in plain terms what the paper does and what is new.
The problem. Early stopping usually watches a clean validation set and stops when it stops improving. Such a set is missing in self-supervised pretraining, under noisy labels and in federated training. Without it, one trains for the full budget or guesses when to stop.
The idea. CFS watches one number that needs no labels: the top eigenvalue of the Fisher information, computed with labels sampled from the model itself, times the effective step size. During training this number first stays near the stability limit of the optimizer, then falls sharply and flattens at a low level, the curvature floor. Its crossing of a fixed threshold marks the end of fast learning along the sharpest direction. The best checkpoint comes after a slower refinement phase, 5.3% to 15.0% of the budget later. CFS waits for this phase by smoothing the signal over a window that grows with the budget, and then stops.
Why it works without labels. Under label noise, networks fit the clean patterns first and the wrong labels later. Signals built on the given labels, such as the gradient, stay high while the network memorizes wrong labels. The Fisher with labels sampled from the model carries none of these wrong labels, so it falls once the clean patterns are fit. On CIFAR-10 with 40% label noise, CFS stops at epoch 75.1 on average, next to the best checkpoint on the clean test set at epoch 74.9.
What is new.
What the evidence shows. On runs that took no part in calibration, with every selection made on development data and every number reported on a sealed test split, CFS beats the validation rule of Table 1 in 7 of 7 settings in one direction and in 5 of 7 in the other, with the other two within 0.02 points. One shared constant matches or beats the stronger validation rule that returns its best checkpoint in 7 of 7 settings (Table R1). CFS also beats every applicable rule without validation data that Related Work names, in all seven regimes (Table R6), and a fixed stopping fraction tuned per task on the same runs and labels in 64 of 84 held-out comparisons (Table R5). On held-out runs it saves 19.5% to 60.3% of the compute per run (Table 25), and on long-tailed CIFAR it saves 30% to 32% while keeping the few-shot classes 2.60 points above the full run (Table R7).
What the reviews credit. Reviewer KL3u finds the signal novel and its cost affordable, Reviewer aGNK finds the rule principled, and Reviewer GnQe finds that the experiments "appear relevant to training models at relevant scales". In the terms of the ICLR reviewer guidelines, the paper contributes new knowledge, theoretical and empirical, with a direct use for practitioners: a label-free signal for when learning has run its course, a proof of how long to wait after it, and a rule that stops near the best checkpoint at a fraction of the cost.
We thank the AC and the three reviewers again for their time and careful reading.