- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper provides a theory-guided extension of Instance-dependent Early Stopping (IES) by (i) analysing when second-order loss differencing offers a favourable signal-to-noise trade-off, (ii) deriving a noise-calibrated stopping threshold, and (iii) establishing conditional stability bounds based on the aggregate gradient of excluded instances. Experiments demonstrate substantial computational savings on CIFAR under decaying learning-rate schedules, with limited degradation in predictive accuracy.
Critical strengths
-
The motivation and method are clear.
-
The paper is technically sound, with rigorous mathematical formulations and proofs provided for key theorems that underpin the main claims.
-
The authors provide sufficient evidence, including extensive experiments and ablation studies, to support the effectiveness of their proposed method.
-
This paper is well-written, making it easy to follow.
Critical weaknesses
-
Gap between theory and the practical algorithm. The connection between the theoretical analysis and the actual algorithm remains incomplete. The main stability results rely on controlling the omitted-gradient residual , but TG-IES neither measures nor explicitly controls this quantity. Moreover, the paper’s own experiments suggest that the stronger per-instance gradient condition is frequently violated.
-
Sensitivity to the learning-rate schedule. The benefits of TG-IES appear strongly dependent on the learning-rate schedule. While TG-IES improves over IES under exponential learning-rate decay, it achieves substantially lower computational savings than IES under fixed-learning-rate SGD and Adam.
-
Comparison with IES+ and generalization beyond vision. It would be useful to include a comparison with IES+ [1] and discuss whether TG-IES can generalize to NLP tasks, as demonstrated by IES+.
-
Missing zeroth-order analysis. The theoretical motivation for using the second-order loss difference mainly compares N=2 against N=1 and N=3, while the zeroth-order signal (N=0, loss itself) proposed in DOM [2] is not included in the theoretical comparison. Since loss magnitude is arguably the most natural baseline for determining whether an instance has been sufficiently learned, can the analysis be extended to characterize when second-order differencing is preferable to directly using the loss?
[1] Yuan, S., Lin, R., Azian, F., Feng, L., Han, B., Niu, G., ... & Liu, T. (2026). Instance-dependent Early Stopping for Adaptive Data Pruning. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[2] Lin, R., Yu, C., Han, B., & Liu, T. (2024, May). On the over-memorization during natural, robust and catastrophic overfitting. In International Conference on Learning Representations (Vol. 2024, pp. 43814-43831).
Questions for authors
Refer to weaknesses.
4 comments
Detailed Revisions Update
Continued Revision
W3. IES+ [1] and beyond vision
IES+. We reimplemented IES+. Below is the implementation result provided.
| Method | Acc. best (final) | Backward saved | FLOPs saved |
|---|---|---|---|
| C10: IES | (94.55) | 32.7 | |
| C10: IES+ | (94.53) | 47.9 | |
| C10: TG-IES | (94.47) | 35.8 | |
| C100: IES | (76.37) | 12.3 | |
| C100: IES+ | (76.18) | 21.3 | |
| C100: TG-IES | (75.34) | 38.5 |
On CIFAR-10, accuracy is equal (CI ). TG-IES skips 3.4 points more backward passes, but IES+ saves 12.2 points more FLOPs because it also skips forward passes, while TG-IES forwards everything to track its signal. On CIFAR-100, TG-IES saves 34.5 more backward and 17.3 more FLOPs, but at 0.80 lower accuracy (CI ). They're different points on the trade-off.
Text: AG News. We trained a 6-layer transformer from scratch (120k articles, 4 classes), the text counterpart of our ViT-Tiny: same AdamW recipe, 100 epochs, same constants, protocol fixed up front.
| Method | Acc. best (final) | Backward saved | FLOPs saved |
|---|---|---|---|
| Full | (91.28) | 0.00 | 0.0 |
| IES | (91.17) | 52.3 | |
| IES+ | (91.16) | 73.5 | |
| InfoBatch | (91.18) | 41.7 | |
| TG-IES | (90.96) | 52.4 |
TG-IES carries over unchanged: 78.6% skipped, 0.11 below full training (CI ). It ties IES and IES+ on backward passes (both CIs include zero); IES+ saves 21.1 more FLOP points with 0.20 higher final accuracy, and InfoBatch saves the least. Two caveats. Best accuracy is reached at epoch 2–3, before any exclusion, and then every method loses 0.4–0.7 as training loss goes to , so we report final accuracy too. Also, IES+'s SGD-defined LR boundaries give a period of 5 for nearly the whole AdamW run, and we left them unchanged. This is smaller than LLM fine-tuning, and we don't claim it covers that.
Bottom line. IES+ is cheaper in FLOPs at equal accuracy on C10 and AG News, while TG-IES skips more backward passes on C10 and saves more of both on C100. The paper no longer claims the highest FLOP savings. Combining IES+'s forward schedule with our calibration is untested future work.
W4. The loss itself (, as in DOM [2])
Good point. We added to the theory and experiments (Sec. 4.2, App. A.6, citing DOM).
Theory. We model (plateau, drift, transient, noise), and every rule gets the same false-retention rate, .
- Immunity: the loss isn't immune to a per-instance plateau, is, and also ignores linear drift. A non-immune rule may never flag an instance whose loss settles above zero.
- Power: among equally immune rules, lower order wins when (0.41 / 0.37 / 0.35 for ). The measured mean is 0.53 early and 0.76 late.
So the loss is best when everything plateaus at zero, and when you need robustness to plateaus and drift. In our logs 10.6% of instances plateau above and 3.7% drift.
Unbiased comparison. At eight fixed epochs, with future losses used only for targets:
| Target | ||||
|---|---|---|---|---|
| "converged" | .987 | .862 | .811 | .757 |
| "stable" | .710 | .759 | .779 | .782 |
The loss wins "converged", which contains the loss by construction, and higher orders win "stable". Both hold in every SNR tercile.
| Setting | acc. / saved | acc. / saved |
|---|---|---|
| Clean | / | / |
| 20% noise (full ) | / | / |
These are within seed spread, so we claim no empirical edge for ; its case is robustness. We also withdrew Fig. 9, which favoured through a selection-biased evaluation epoch.
Thanks again. This review got us to measure , sharpen the schedule limitation, add IES+ and a text task, and put the plain loss into both theory and experiments.
References
[1] Yuan, S., Lin, R., Azian, F., Feng, L., Han, B., Niu, G., ... & Liu, T. (2026). Instance-dependent Early Stopping for Adaptive Data Pruning. IEEE TPAMI.
[2] Lin, R., Yu, C., Han, B., & Liu, T. (2024). On over-memorization during natural, robust and catastrophic overfitting. ICLR.
I thank the authors for their detailed and thoughtful rebuttal. I appreciate the substantial efforts they have made to address my concerns, particularly through the additional theoretical analyses and extensive experimental evaluations. While these additions have improved the paper, I believe that certain inherent limitations remain, particularly regarding the theoretical assumptions and the empirical advantages. Therefore, I will maintain my original rating. Best of luck with your submission!
Thank you so much for your kind review again. We appreciate your time and comments.
Issues addressed by the reviewer addressed ; Additional results are added.
Response to Reviewer 3
Thank you for the constructive review and for pointing us to IES+ [1] and DOM [2]. Both are now in the paper, with new experiments. Unless noted, numbers are ResNet-18 / CIFAR-10, 3 seeds, mean ± s.d.; "saved" = share of backward passes skipped.
Reviews are addressed in Green for reviewer 3 in:
https://anonymous.4open.science/r/TG_IES_Results-8EA0/rebuttal_report.pdf
W1. Theory–algorithm gap: the omitted-gradient residual
We agree that TG-IES doesn't control ωt, and the paper now says plainly that the stability results are conditional and don't certify the reported runs. To close part of the gap, we measured ωt directly (App. A.5).
Why the aggregate condition matters. You're right that the per-instance condition fails: about half of the probed passive instances sit above ε0 at every check (0.39–0.52). But the deviation and descent results only need the aggregate ωt=N1∑i∈Ptgi to be small, which is a much weaker requirement.
Measured. Every 10 epochs we computed ωt exactly, νt=∥ωt∥/∥g^(t)∥, and the top Hessian eigenvalue β^t, giving 60 probes. The probe doesn't perturb the run.
So the hypotheses hold for most of training but not at the start and end. The deviation bound's (1+βη)H factor makes it vacuous over these horizons, so we don't compare it to measured distances. Using ωt as a control signal (e.g. residual-aware reactivation) is the main open step.
W2. Sensitivity to the LR schedule
Agreed. This remains a stated limitation in the abstract, results and limitations (App. A.7).
Likely mechanism. δ is the noise level at tc. Under decay the noise keeps falling (≈41× by epoch 200), so many signals drop below δ. With a constant LR it falls only ≈12× (and Adam hits a numerical floor), so fewer instances string together 15 passes. The logs fit this: at epoch 31, δ is 0.01597 (decay), 0.00247 (fixed) and 0.000257 (Adam), and final exclusion is 87.4% vs 42.0% and 45.6%. An open-loop replay already predicts the ordering (58.3 / 23.6 / 28.7% vs 53.6 / 29.3 / 30.2% in training). This supports the mechanism without proving it. The drift term is only weakly correlated with the signal (r=0.23–0.37), and we don't claim a different tc or K∗ would close the gap.
It's the decay, not the optimizer. With a decaying schedule TG-IES helps under adaptive optimizers too: ViT-Tiny + AdamW + cosine saves 25.52 vs 16.68% (C10) and 26.28 vs 12.51% (C100), and it matches IES on AG News (below).