acceptodds
Under review as a conference paper at ICLR 2027

Theory-Guided Instance-Dependent Early Stopping via Second-Order Loss Dynamics normal

Abstract

Instance-dependent early stopping (IES) reduces training computation by skipping backpropagation for examples whose losses have stabilized, but the second-order loss-difference criterion used by IES lacks a theoretical justification. We develop Theory-Guided Instance-Dependent Early Stopping (TG-IES), which analyzes this criterion under explicit assumptions and derives a noise-calibrated mastery threshold and a conditional stability bound for instance exclusion. We show that second-order differencing has a favorable bias–variance trade-off over first- and third-order differences in a specific signal-to-noise regime, and that its signal contains both gradient-drift and curvature terms. TG-IES derives its stopping threshold and patience from training-run statistics rather than using the manually selected threshold of IES. Across CIFAR-10 and CIFAR-100, TG-IES saves 53–58% of backpropagation instances in its main convolutional settings while keeping accuracy within 0.1–0.9 percentage points of full training; in the primary ResNet-18/CIFAR-10 setting, this corresponds to 35.7% FLOP-normalized savings. The method also transfers to ViT-Tiny, where it achieves smaller savings and incurs a modest accuracy cost. These results provide a theoretical basis for second-order instance-dependent early stopping and show that theory-guided calibration can substantially reduce training computation under the assumptions and experimental settings studied. The code is available at https://anonymous.4open.science/r/TG_IES_Results-8EA0 https://anonymous.4open.science/r/TG_IES_Results-8EA0

Primary area:
optimization
Keywords:
Instance-Dependent Early Stopping, Training Efficiency, Loss Dynamics, Optimization, Learning Theory, Compute-Efficient Training, Dynamic Data Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iZxQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides a theory-guided extension of Instance-dependent Early Stopping (IES) by (i) analysing when second-order loss differencing offers a favourable signal-to-noise trade-off, (ii) deriving a noise-calibrated stopping threshold, and (iii) establishing conditional stability bounds based on the aggregate gradient of excluded instances. Experiments demonstrate substantial computational savings on CIFAR under decaying learning-rate schedules, with limited degradation in predictive accuracy.

Critical strengths
  1. The motivation and method are clear.

  2. The paper is technically sound, with rigorous mathematical formulations and proofs provided for key theorems that underpin the main claims.


  3. The authors provide sufficient evidence, including extensive experiments and ablation studies, to support the effectiveness of their proposed method.


  4. This paper is well-written, making it easy to follow.

Critical weaknesses
  1. Gap between theory and the practical algorithm. The connection between the theoretical analysis and the actual algorithm remains incomplete. The main stability results rely on controlling the omitted-gradient residual ωt\omega_t, but TG-IES neither measures nor explicitly controls this quantity. Moreover, the paper’s own experiments suggest that the stronger per-instance gradient condition is frequently violated.

  2. Sensitivity to the learning-rate schedule. The benefits of TG-IES appear strongly dependent on the learning-rate schedule. While TG-IES improves over IES under exponential learning-rate decay, it achieves substantially lower computational savings than IES under fixed-learning-rate SGD and Adam.

  3. Comparison with IES+ and generalization beyond vision. It would be useful to include a comparison with IES+ [1] and discuss whether TG-IES can generalize to NLP tasks, as demonstrated by IES+.

  4. Missing zeroth-order analysis. The theoretical motivation for using the second-order loss difference mainly compares N=2 against N=1 and N=3, while the zeroth-order signal (N=0, loss itself) proposed in DOM [2] is not included in the theoretical comparison. Since loss magnitude is arguably the most natural baseline for determining whether an instance has been sufficiently learned, can the analysis be extended to characterize when second-order differencing is preferable to directly using the loss?

[1] Yuan, S., Lin, R., Azian, F., Feng, L., Han, B., Niu, G., ... & Liu, T. (2026). Instance-dependent Early Stopping for Adaptive Data Pruning. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[2] Lin, R., Yu, C., Han, B., & Liu, T. (2024, May). On the over-memorization during natural, robust and catastrophic overfitting. In International Conference on Learning Representations (Vol. 2024, pp. 43814-43831).

Questions for authors

Refer to weaknesses.

4 comments
AuthorsOpenReview ↗

Issues addressed by the reviewer addressed ; Additional results are added.

Response to Reviewer 3

Thank you for the constructive review and for pointing us to IES+ [1] and DOM [2]. Both are now in the paper, with new experiments. Unless noted, numbers are ResNet-18 / CIFAR-10, 3 seeds, mean ±\pm s.d.; "saved" = share of backward passes skipped.

Reviews are addressed in Green for reviewer 3 in:

https://anonymous.4open.science/r/TG_IES_Results-8EA0/rebuttal_report.pdf

W1. Theory–algorithm gap: the omitted-gradient residual

We agree that TG-IES doesn't control ωt\omega_t, and the paper now says plainly that the stability results are conditional and don't certify the reported runs. To close part of the gap, we measured ωt\omega_t directly (App. A.5).

Why the aggregate condition matters. You're right that the per-instance condition fails: about half of the probed passive instances sit above ε0\varepsilon_0 at every check (0.39–0.52). But the deviation and descent results only need the aggregate ωt=1N∑i∈Ptgi\omega_t=\frac1N\sum_{i\in P_t}g_i to be small, which is a much weaker requirement.

Measured. Every 10 epochs we computed ωt\omega_t exactly, νt=∥ωt∥/∥g^(t)∥\nu_t=\|\omega_t\|/\|\hat g^{(t)}\|, and the top Hessian eigenvalue β^t\hat\beta_t, giving 60 probes. The probe doesn't perturb the run.

Epoch406080100120140160180200
Frac. >ε0>\varepsilon_0.52.48.39.46.48.46.47.50.50
∥ωt∥\|\omega_t\|.001.013.016.021.013.013.014.014.013
νt\nu_t.000.009.033.173.565.737.814.849.944
  • ∥ωt∥≤0.038\|\omega_t\|\le0.038 even though many individual gradients aren't small. They partly cancel, to 26% of the triangle bound on average.
  • νt<1\nu_t<1 at 58/60 probes. It stays under 0.05 until epoch 80 and rises later only because the active gradient shrinks.
  • ηt<2(1−νt)/β^t\eta_t<2(1-\nu_t)/\hat\beta_t holds at 52/60, failing early (epochs 10–20, large LR) and at the very end (νt≈1\nu_t\approx1). With momentum 0.9, which the theorem doesn't cover, it holds at 34.

So the hypotheses hold for most of training but not at the start and end. The deviation bound's (1+βη)H(1+\beta\eta)^H factor makes it vacuous over these horizons, so we don't compare it to measured distances. Using ωt\omega_t as a control signal (e.g. residual-aware reactivation) is the main open step.


W2. Sensitivity to the LR schedule

Agreed. This remains a stated limitation in the abstract, results and limitations (App. A.7).

ResNet-18 / C10BaselineIES acc. / savedTG-IES acc. / saved
Exp. decay95.03±0.1395.03\pm0.1394.85±0.1794.85\pm0.17 / 49.02±0.4449.02\pm0.4494.85±0.2894.85\pm0.28 / 53.63±0.3053.63\pm0.30
SGD, fixed92.21±0.0892.21\pm0.0891.77±0.1191.77\pm0.11 / 49.42±0.1749.42\pm0.1791.91±0.1891.91\pm0.18 / 29.27±0.3029.27\pm0.30
Adam92.91±0.0692.91\pm0.0692.42±0.1092.42\pm0.10 / 60.09±0.2960.09\pm0.2992.44±0.1092.44\pm0.10 / 30.18±0.6230.18\pm0.62

Likely mechanism. δ\delta is the noise level at tct_c. Under decay the noise keeps falling (≈41× by epoch 200), so many signals drop below δ\delta. With a constant LR it falls only ≈12× (and Adam hits a numerical floor), so fewer instances string together 15 passes. The logs fit this: at epoch 31, δ\delta is 0.01597 (decay), 0.00247 (fixed) and 0.000257 (Adam), and final exclusion is 87.4% vs 42.0% and 45.6%. An open-loop replay already predicts the ordering (58.3 / 23.6 / 28.7% vs 53.6 / 29.3 / 30.2% in training). This supports the mechanism without proving it. The drift term is only weakly correlated with the signal (r=0.23r=0.23–0.370.37), and we don't claim a different tct_c or K∗K^* would close the gap.

It's the decay, not the optimizer. With a decaying schedule TG-IES helps under adaptive optimizers too: ViT-Tiny + AdamW + cosine saves 25.52 vs 16.68% (C10) and 26.28 vs 12.51% (C100), and it matches IES on AG News (below).


AuthorsOpenReview ↗

Detailed Revisions Update

Continued Revision

W3. IES+ [1] and beyond vision

IES+. We reimplemented IES+. Below is the implementation result provided.

MethodAcc. best (final)Backward savedFLOPs saved
C10: IES94.85±0.1794.85\pm0.17 (94.55)49.02±0.4449.02\pm0.4432.7
C10: IES+94.84±0.0994.84\pm0.09 (94.53)50.25±0.3750.25\pm0.3747.9
C10: TG-IES94.85±0.2894.85\pm0.28 (94.47)53.63±0.3053.63\pm0.3035.8
C100: IES76.78±0.4076.78\pm0.40 (76.37)18.49±0.4318.49\pm0.4312.3
C100: IES+76.82±0.2076.82\pm0.20 (76.18)23.28±0.7323.28\pm0.7321.3
C100: TG-IES76.01±0.0876.01\pm0.08 (75.34)57.76±0.0457.76\pm0.0438.5

On CIFAR-10, accuracy is equal (CI [−0.88,0.89][-0.88, 0.89]). TG-IES skips 3.4 points more backward passes, but IES+ saves 12.2 points more FLOPs because it also skips forward passes, while TG-IES forwards everything to track its signal. On CIFAR-100, TG-IES saves 34.5 more backward and 17.3 more FLOPs, but at 0.80 lower accuracy (CI [−1.26,−0.34][-1.26, -0.34]). They're different points on the trade-off.

Text: AG News. We trained a 6-layer transformer from scratch (120k articles, 4 classes), the text counterpart of our ViT-Tiny: same AdamW recipe, 100 epochs, same constants, protocol fixed up front.

MethodAcc. best (final)Backward savedFLOPs saved
Full91.75±0.0891.75\pm0.08 (91.28)0.000.0
IES91.66±0.1191.66\pm0.11 (91.17)78.47±0.5278.47\pm0.5252.3
IES+91.67±0.0991.67\pm0.09 (91.16)78.93±0.4078.93\pm0.4073.5
InfoBatch91.55±0.1191.55\pm0.11 (91.18)41.70±0.0241.70\pm0.0241.7
TG-IES91.64±0.1391.64\pm0.13 (90.96)78.63±0.7778.63\pm0.7752.4

TG-IES carries over unchanged: 78.6% skipped, 0.11 below full training (CI [−0.35,0.13][-0.35, 0.13]). It ties IES and IES+ on backward passes (both CIs include zero); IES+ saves 21.1 more FLOP points with 0.20 higher final accuracy, and InfoBatch saves the least. Two caveats. Best accuracy is reached at epoch 2–3, before any exclusion, and then every method loses 0.4–0.7 as training loss goes to 3×10−43\times10^{-4}, so we report final accuracy too. Also, IES+'s SGD-defined LR boundaries give a period of 5 for nearly the whole AdamW run, and we left them unchanged. This is smaller than LLM fine-tuning, and we don't claim it covers that.

Bottom line. IES+ is cheaper in FLOPs at equal accuracy on C10 and AG News, while TG-IES skips more backward passes on C10 and saves more of both on C100. The paper no longer claims the highest FLOP savings. Combining IES+'s forward schedule with our calibration is untested future work.


W4. The loss itself (N=0N=0, as in DOM [2])

Good point. We added N=0N=0 to the theory and experiments (Sec. 4.2, App. A.6, citing DOM).

Theory. We model L~i(s)=ℓi+ais+ciris+ξi,s\tilde L_i^{(s)}=\ell_i+a_i s+c_i r_i^s+\xi_{i,s} (plateau, drift, transient, noise), and every rule gets the same false-retention rate, ∣ΔNL~∣<kσ(2NN)|\Delta^N\tilde L|<k\sigma\sqrt{\binom{2N}{N}}.

  • Immunity: the loss isn't immune to a per-instance plateau, N=1N=1 is, and N=2N=2 also ignores linear drift. A non-immune rule may never flag an instance whose loss settles above zero.
  • Power: among equally immune rules, lower order wins when ri>(1+2(2N+1)/(N+1))−1r_i>(1+\sqrt{2(2N+1)/(N+1)})^{-1} (0.41 / 0.37 / 0.35 for N=0,1,2N=0,1,2). The measured mean rr is 0.53 early and 0.76 late.

So the loss is best when everything plateaus at zero, and N=2N=2 when you need robustness to plateaus and drift. In our logs 10.6% of instances plateau above δ\delta and 3.7% drift.

Unbiased comparison. At eight fixed epochs, with future losses used only for targets:

TargetN=0N=0N=1N=1N=2N=2N=3N=3
"converged".987.862.811.757
"stable".710.759.779.782

The loss wins "converged", which contains the loss by construction, and higher orders win "stable". Both hold in every SNR tercile.

SettingN=0N=0 acc. / savedN=2N=2 acc. / saved
Clean94.77±0.0894.77\pm0.08 / 53.47±0.1253.47\pm0.1294.85±0.2894.85\pm0.28 / 53.63±0.3053.63\pm0.30
20% noise (full 86.85±0.7486.85\pm0.74)86.77±0.3786.77\pm0.37 / 37.29±0.2837.29\pm0.2886.63±0.4386.63\pm0.43 / 39.21±0.2239.21\pm0.22

These are within seed spread, so we claim no empirical edge for N=2N=2; its case is robustness. We also withdrew Fig. 9, which favoured N=2N=2 through a selection-biased evaluation epoch.


Thanks again. This review got us to measure ωt\omega_t, sharpen the schedule limitation, add IES+ and a text task, and put the plain loss into both theory and experiments.

References

[1] Yuan, S., Lin, R., Azian, F., Feng, L., Han, B., Niu, G., ... & Liu, T. (2026). Instance-dependent Early Stopping for Adaptive Data Pruning. IEEE TPAMI.

[2] Lin, R., Yu, C., Han, B., & Liu, T. (2024). On over-memorization during natural, robust and catastrophic overfitting. ICLR.

I thank the authors for their detailed and thoughtful rebuttal. I appreciate the substantial efforts they have made to address my concerns, particularly through the additional theoretical analyses and extensive experimental evaluations. While these additions have improved the paper, I believe that certain inherent limitations remain, particularly regarding the theoretical assumptions and the empirical advantages. Therefore, I will maintain my original rating. Best of luck with your submission!

Thank you so much for your kind review again. We appreciate your time and comments.

Reviewer_vy6COpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates instance-dependent early stopping based on second-order differences of per-example training losses. TG-IES calibrates the stopping threshold using an estimated noise scale, suspends backpropagation for examples that repeatedly satisfy the criterion, and reactivates them when the signal exceeds the threshold. The paper analyzes the signal–noise behavior of different finite-difference orders and characterizes selective training through the aggregate gradient omitted by passive examples.

Critical strengths

1.A simple and reversible mechanism. The algorithm uses existing loss histories without an auxiliary model. Passive examples remain observable and can re-enter backpropagation when their behavior changes.

2.An appropriate formulation of optimization error. Expressing selective updates as the full gradient minus the aggregate omitted gradient correctly identifies the quantity relevant to stability, without requiring every passive example to have a small gradient.

3.Transparent reporting of limitations. The paper acknowledges that second-order differences are not universally optimal, benefits depend on optimization settings, individual-gradient assumptions frequently fail, and default patience saturates at its configured cap.

Critical weaknesses

1.A smaller difference signal does not establish a better stopping criterion. The theoretical comparison concerns expected squared outputs of different finite-difference operators, rather than error for a common estimation target or erroneous-exclusion risk. Evaluating examples when the second-order signal first crosses its threshold also introduces selection bias favoring that signal. (Equations 3–4; Figure 9)

2.The exclusion certificate does not derive the operational patience rule. The certificate bounds an exclusion duration, whereas the algorithm uses it to initialize the waiting period before exclusion. Moreover, all default calibration runs yield (K=15), so the experiments do not demonstrate an adaptive benefit from the formula. (Appendices E and H)

3.Noise calibration lacks sufficient specification and operational validation. The estimator’s window, detrending procedure, and aggregation are not fully described. Pointwise bounds are not extended to repeated adaptive decisions, and the aggregate omitted gradient is not measured. The conditional stability results therefore cannot directly certify the reported runs.

4.The diagnostic timeline is inconsistent with the stated algorithm unless additional details are supplied. Counting begins at epoch 30, implying the first passive examples at epoch 44 for patience 15, yet Table 10 reports passive-set diagnostics at epoch 40. Figure 11 also shows substantial pre-selection divergence from the shared-seed baseline. These observations require clarification rather than an assumption that the experiments follow Algorithm 1 exactly.

Questions for authors

1.How is the noise scale estimated, including the observations, temporal window, detrending, and aggregation? How sensitive is the threshold to calibration timing, and what supports keeping it fixed subsequently?

2.How does the Figure 9 protocol exclude selection bias from evaluating at the second-order threshold-crossing event? Can the comparison use independently selected epochs or an independently estimated signal-to-noise regime?

3.What justifies using the conditional exclusion horizon as pre-exclusion patience? Since default runs yield (K=15), what does the formula add beyond fixing this value, and have you measured the aggregate omitted-gradient residual?

4.How should the epoch-40 passive-set probe and epoch-20 parameter divergence be reconciled with Algorithm 1 and the shared-seed protocol? Please specify any differences in diagnostic configurations, initialization, or execution.

4 comments
AuthorsOpenReview ↗

Weakness and Questions addressed with vast analysis and experiments

Response to Reviewer 2

https://anonymous.4open.science/r/TG_IES_Results-8EA0/rebuttal_report.pdf

We thank the reviewer for the detailed and precise comments. We have gone through the issues raised by the reviewer and addressed them accordingly. Unless stated otherwise, results are for ResNet-18 on CIFAR-10, 3 seeds, mean ±\pm s.d.; "saved" means the share of backward passes skipped.

Experimental details are updated in the main manuscript in red .


W1 / Q2. A smaller difference signal is not a better stopping criterion; selection bias in Figure 9

Concern. Equations 3–4 compare the expected squared outputs of different operators, not the error for a common target or the risk of an erroneous exclusion. Evaluating at the epoch where the second-order signal first crosses its threshold favours that signal.

Response. We agree with both points and revised the analysis (Appendix A.6).

(a) Selection bias. We repeated the Figure 9 protocol, letting each order N=1,2,3N=1, 2, 3 choose the evaluation epoch (its own first crossing). In all 9 of 9 cases (3 orders ×\times 3 schedules), the order that chose the epoch has the smallest error at that epoch. The protocol is biased, so we withdraw the Figure 9 evidence for N=2N=2. Equations 3–4 are now described as a comparison of output sizes only, and the paper points to the decision-level comparison below.

(b) A criterion on a common target. We model the recorded loss as L~i(s)=ℓi+ais+ciris+ξi,s\tilde L_i^{(s)}=\ell_i+a_i s+c_i r_i^s+\xi_{i,s}, with a plateau ℓi\ell_i, a linear drift aia_i, a geometric transient with rate rir_i, and noise. Every rule, including the loss itself (N=0N=0), uses a threshold set to the same false-retention rate at convergence, ∣ΔNL~i(t)∣<kσ(2NN)|\Delta^N\tilde L_i^{(t)}|<k\sigma\sqrt{\binom{2N}{N}}. A new proposition then compares the rules by erroneous exclusion:

  • ΔN\Delta^N removes polynomial trends of degree below NN. So N=1N=1 is immune to a plateau, and N=2N=2 to both a plateau and a linear drift. A rule that is not immune can fail to ever flag a converged instance.
  • Among immune rules, order NN makes fewer erroneous exclusions than order N+1N+1 when ri>(1+2(2N+1)/(N+1))−1r_i>\left(1+\sqrt{2(2N+1)/(N+1)}\right)^{-1}, which is 0.410.41, 0.370.37 and 0.350.35 for N=0,1,2N=0, 1, 2.

So the loss itself is the best rule when instances share a zero plateau, and N=2N=2 is the best rule that is immune to plateaus and drift. In our logged losses, 10.6% of instances plateau above δ\delta and 3.7% show drift.

(c) Independently selected epochs. We evaluated all orders at eight fixed epochs (40,60,…,18040, 60, \dots, 180), using future losses only to define the targets. The mean AUROC is:

TargetN=0N=0N=1N=1N=2N=2N=3N=3
"converged".987.987.862.862.811.811.757.757
"stable".710.710.759.759.779.779.782.782

The "converged" target contains the loss, so it favours N=0N=0 by construction.

(d) An independently estimated signal-to-noise regime. We split instances into terciles by their mean loss over epochs t−12,…,t−4t-12,\dots,t-4 relative to σ^\hat\sigma. This measure depends neither on the signals being tested nor on the targets.

  • Against "converged", the loss is best in every tercile (AUROC .980.980–.997.997).
  • Against "stable", N=2N=2 and N=3N=3 are within 0.0040.004 of each other (.676.676–.752.752), and both are above N=0N=0 and N=1N=1. No tercile separates N=2N=2 from N=3N=3.

(e) End to end. We ran TG-IES with N=0N=0 (the loss) against N=2N=2, calibrating the threshold separately for each (new rows in Table 3):

SettingN=0N=0 acc. / saved (%)N=2N=2 acc. / saved (%)
Clean labels94.77±0.0894.77\pm0.08 / 53.47±0.1253.47\pm0.1294.85±0.2894.85\pm0.28 / 53.63±0.3053.63\pm0.30
20% label noise (full training 86.85±0.7486.85\pm0.74)86.77±0.3786.77\pm0.37 / 37.29±0.2837.29\pm0.2886.63±0.4386.63\pm0.43 / 39.21±0.2239.21\pm0.22

The two orders perform slightly better than zeroth order but not worse, and two orders provide stability. We therefore claim not too large an empirical advantage of N=2N=2 over the loss itself; its justification is immunity to plateaus and linear drift.


Continued revision explanations

W2 / Q3. The exclusion certificate does not give the patience rule; all runs use K=15K=15

Concern. The certificate bounds how long an excluded instance may stay excluded, but the algorithm uses it as the waiting period before exclusion; and every default run gives K=15K=15.

Response. We agree on both points (Appendix A.4).

(a) What the formula adds. In our primary setting Ksafe≈1.15×106K_{\mathrm{safe}}\approx1.15\times10^6, so the cap Kmax⁡=15K_{\max}=15 sets K∗K^* in every run, and the formula adds nothing beyond fixing the value. It bounds how long one excluded instance can stay excluded, not how long to wait before excluding it. We withdraw it as the justification of the waiting period. The abstract, introduction, Figure 1 and contributions now say that patience is a fixed cap, and that KsafeK_{\mathrm{safe}} is reported but does not set it.

(b) What patience actually controls. Patience controls erroneous exclusion under repeated noisy checks, and we now prove a bound for exactly that (new proposition, repeated adaptive checks). Under the stated independent, σ2\sigma^2-sub-Gaussian noise, the probability that an instance whose noise-free signal stays above δ+x6 σ\delta+x\sqrt6\,\sigma still passes all KK consecutive checks is at most (2e−x2/2)⌈K/3⌉\left(2e^{-x^2/2}\right)^{\lceil K/3\rceil}, however the trajectory and δ\delta depend on the past. With the margin set by the error budget p=0.3p=0.3, the bound depends on KK alone:

KK881010121215151616
Bound1.8×10−21.8\times10^{-2}2.2×10−32.2\times10^{-3}1.2×10−31.2\times10^{-3}8.6×10−58.6\times10^{-5}9.5×10−69.5\times10^{-6}

Kmax⁡=15K_{\max}=15 is the smallest KK with a bound below 10−410^{-4}. Budgets from 10−310^{-3} to 10−510^{-5} give KK between 13 and 16, so the value is not sensitive to the budget.

(c) The cost of the choice. The patience sweep (Table 9) shows the trade:

K∗K^*Acc. (%)Saved (%)
8894.80±0.0994.80\pm0.0958.85±0.5758.85\pm0.57
151594.80±0.0994.80\pm0.0953.38±0.5853.38\pm0.58
303094.87±0.0994.87\pm0.0944.13±0.4044.13\pm0.40

K∗=8K^*=8 has the same mean accuracy and saves 5.5 points more, but with a weaker guarantee. We keep 1515 as a conservative default, not an optimum, and report 88 as a sensitivity result. Choosing the default again after seeing test accuracy would be selection on the test set.

(d) Limits of the bound. It assumes noise independent across epochs. Our independence check used epochs 101–200, not epochs 31–50, where the first checks happen; we state this in the paper.


W3 / Q1. Noise calibration: specification, repeated decisions, and the omitted gradient

(a) How the noise scale is estimated (Q1)

At the calibration epoch tct_c, for every instance with three consecutive loss observations:

Xi=∣L~i(tc)−2L~i(tc−1)+L~i(tc−2)∣.X_i=\left|\tilde L_i^{(t_c)}-2\tilde L_i^{(t_c-1)}+\tilde L_i^{(t_c-2)}\right|.

  • Observations: the per-instance losses recorded in the forward pass of epochs tc−2t_c-2, tc−1t_c-1, tct_c.
  • Temporal window: none; the estimate is across instances at one epoch.
  • Detrending: the second difference itself, which removes a locally linear trend.
  • Aggregation: σ^2=16 median{Xi2:Xi≤medianjXj}\hat\sigma^2=\frac16\,\mathrm{median}\{X_i^2: X_i\le\mathrm{median}_j X_j\}, the median over the lower half of instances, so instances that still carry signal have little influence.
  • Threshold: δ=k6 σ^\delta=k\sqrt6\,\hat\sigma with k=Φ−1(1−pcheck/2)k=\Phi^{-1}(1-p_{\mathrm{check}}/2) and pcheck=1−(1−p)1/K∗p_{\mathrm{check}}=1-(1-p)^{1/K^\ast}, for a compound budget p=0.3p=0.3 over the K∗K^* checks.

While checking this, we found that under Gaussian noise σ^→Φ−1(5/8) σ≈0.32 σ\hat\sigma\to\Phi^{-1}(5/8)\,\sigma\approx0.32\,\sigma (new lemma). So in all reported runs the threshold is about 3.1×3.1\times below its nominal value, which makes the rule more cautious. Monte Carlo agrees: with σ=1\sigma=1 the estimate is 0.3190.319, while a consistency-corrected quantile estimate gives 1.0021.002 and a temporal estimate gives 0.9980.998. We state this in the paper and did not re-tune. A variant using the corrected estimate (run after the ImageNet result and labelled as post hoc) moves TG-IES along its trade-off without improving it: 60.30% saved at 94.62±0.1394.62\pm0.13 on CIFAR-10.

Continued revisions

(b) Sensitivity to calibration timing, and why δ\delta stays fixed (Q1)

tct_cAcc. (%)Saved (%)δ\delta
202094.74±0.2194.74\pm0.2158.57±0.2558.57\pm0.250.03770.0377
3030 (reported)94.85±0.2894.85\pm0.2853.63±0.3053.63\pm0.300.01660.0166
505094.84±0.0494.84\pm0.0439.98±0.4639.98\pm0.460.00370.0037

The threshold is sensitive to timing because the noise shrinks during training: σ^\hat\sigma falls about 41×41\times from epoch 31 to 200. Accuracy is flat over tc∈[20,50]t_c\in[20,50] on CIFAR-10, so tct_c acts as a savings dial there (on CIFAR-100 a later tct_c also raises accuracy). We keep δ\delta fixed after tct_c because recalibrating would follow the shrinking noise and keep tightening the threshold. A configuration that recalibrated repeatedly collapsed to 1.8% savings on VGG-16, against 54.4% with one calibration. tct_c is a fixed rule, tc=0.15 Tt_c=0.15\,T, used unchanged in all settings.

(c) Repeated adaptive decisions

The pointwise bound now extends to repeated adaptive checks through the proposition in W2 (b), which holds however δ\delta and the trajectory depend on the past.

(d) The aggregate omitted gradient, measured (also Q3)

Every 10 epochs of the reported configuration, we computed ωt=1N∑i∈Ptgi\omega_t=\frac1N\sum_{i\in P_t}g_i exactly, the ratio νt=∥ωt∥/∥g^(t)∥\nu_t=\|\omega_t\|/\|\hat g^{(t)}\|, and the top Hessian eigenvalue β^t\hat\beta_t. That is 60 probes (20 per seed). The probe does not change the run: the epoch logs are identical with and without it under deterministic cuDNN. New rows of Table 10:

Epoch406080100120140160180200
∥ωt∥\|\omega_t\|0.0010.0010.0130.0130.0160.0160.0210.0210.0130.0130.0130.0130.0140.0140.0140.0140.0130.013
νt\nu_t0.0000.0000.0090.0090.0330.0330.1730.1730.5650.5650.7370.7370.8140.8140.8490.8490.9440.944
  • The omitted gradient stays small (∥ωt∥≤0.038\|\omega_t\|\le0.038), because the passive gradients partly cancel: on average 26% of the triangle bound.
  • The descent condition νt<1\nu_t<1 holds at 58 of 60 probes. νt\nu_t rises late because the active gradient shrinks, not because ωt\omega_t grows.
  • The step-size condition ηt<2(1−νt)/β^t\eta_t<2(1-\nu_t)/\hat\beta_t holds at 52 of 60 probes. It fails at epochs 10–20, where ηt\eta_t is large, and at the last probes, where νt≈1\nu_t\approx1. With momentum 0.90.9, which our theorem does not cover, it holds at 34.

So the conditions of the descent result hold through most of training but not at its start and very end. TG-IES does not control ωt\omega_t, and the stability results stay conditional; they do not certify the reported runs, and the paper now says so.


W4 / Q4. The timeline (passives at epoch 40) and the early divergence in Figure 11

Concern. Counting from epoch 30 with patience 15 implies the first passive instances at epoch 44, yet Table 10 reports passives at epoch 40; and Figure 11 shows divergence before any exclusion.

Response. The reviewer is right that the experiments did not follow Algorithm 1 as written (Appendix A.1).

(a) The missing warm-up. All reported TG-IES runs use a warm-up that the submitted Algorithm 1 left out. From epoch 3 until tct_c, the mastery check runs with the IES defaults (δ=10−3\delta=10^{-3}, K∗=1K^\ast = 1). Exclusions and pass counters carry over into the calibrated phase. At the first calibrated check, a passive instance stays passive if it passes the new δ\delta, and instances that have been passing keep their counters. So K∗=15K^*=15 does not delay every exclusion to epoch 44, which explains the passive instances at epoch 40. Algorithm 1 and its text now include the warm-up.

(b) The run without the warm-up. We ran Algorithm 1 exactly as written (new rows of Tables 3 and 10):

ConfigurationAcc. (%)Saved (%)δ\delta
Reported (with warm-up)94.85±0.2894.85\pm0.2853.63±0.3053.63\pm0.300.01660.0166
No warm-up94.86±0.2094.86\pm0.2050.67±0.1150.67\pm0.110.01230.0123
Epoch406080100120140160180200
Passive (%), with warm-up1.81.820.720.745.745.772.972.984.084.086.186.187.087.087.487.487.387.3
Passive (%), without warm-up0.00.017.217.241.641.668.768.781.081.083.883.884.784.785.085.085.185.1

Without the warm-up nothing is passive before epoch 45, as the reviewer derived. Accuracy is unchanged, and savings fall by 3 points.

(c) The early divergence in Figure 11 is GPU nondeterminism. The runs use default cuDNN autotuning, which is nondeterministic, so two runs with the same seed separate even when nothing is excluded. We measured this floor with a run that excludes nothing (δ=0\delta=0, one seed):

Parameter distance to the same-seed baselineEpoch 20Epoch 200
No-exclusion control (δ=0\delta=0)53.653.626.526.5
TG-IES, with warm-up53.653.631.931.9
TG-IES, without warm-up53.853.831.131.1

We thank the reviewer again.

Thank you for the clarification. Considering the paper’s novelty, efficiency, and simplicity, I remain unconvinced that it meets the acceptance bar for ICLR. I therefore maintain my score of 2.

Reviewer_YiaXOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends Instance-dependent early stopping (IES) with a noise-based stopping threshold, different gradient-update scaling, and theoretical analysis. It temporarily skips backward updates for selected examples while continuing to check their losses. Strictly speaking, this is selective backpropagation. Experiments on CIFAR-10 and CIFAR-100 show computation savings with some accuracy loss.

Critical strengths

The method addresses a practical problem: reducing training computation while largely preserving accuracy. The paper evaluates several architectures and reports both favorable and unfavorable settings.

Critical weaknesses

I have three major concerns about the paper:

  1. TG-IES keeps the same basic skipping rule and adds noise-based calibration, different update scaling, and theoretical analysis. The algorithmic change therefore seems incremental. In Table 2, it matches IES literal’s mean accuracy while increasing backward savings from 49.02% to 53.63%, a modest improvement. Larger savings appear in other settings, i.e., CIFAR-100, but come with lower accuracy. Under fixed-rate SGD and Adam, it saves substantially less computation than IES. The paper does not separately test the different ways the methods average gradients. IES divides the sum of active examples’ gradients by the number of active examples, whereas TG-IES divides it by the full dataset size. This difference may also affect the results, so its contribution needs clarification.
  2. Appendix H explains why calibration is performed only once, but not why epoch 30 is chosen or how sensitive the results are to this choice. The authors use a theoretical rule to choose how many consecutive epochs an example must pass the check before its backward update is skipped. However, this rule gives a value above 15 in every standard run, and they always limit it to 15. Although the paper proposes a theory-based way to choose patience, all standard experiments use the fixed limit of 15. These experiments therefore do not show whether the proposed calculation is useful in practice. Appendix R tests several patience values, but it is still unclear why the authors chose 15. Waiting eight (8) epochs gives the same average accuracy while saving more computation. Also, the main experiments run for 200 epochs, so the paper offers little guidance on choosing these settings for shorter runs.
  3. One of my major concerns is that the authors do not evaluate their method on ImageNet. Evaluations on CIFAR-10 and CIFAR-100 alone are insufficient to establish the effectiveness and scalability of the proposed approach. Relevant prior work, such as InfoBatch, has evaluated its method on ImageNet. The authors should include experiments on ImageNet to enable a more meaningful comparison with existing approaches and demonstrate the effectiveness of their method on a larger, more challenging dataset.
Questions for authors
  1. How much of the improvement comes from the noise-based threshold? Could you compare IES and TG-IES using the same update scaling and patience to make this clearer?
  2. Why did you choose epoch 30 for calibration? Would calibrating at epoch 20 or 50 change the results? How would you choose the calibration time and patience for a shorter training run?
5 comments
AuthorsOpenReview ↗

Updated Experiments and Findings asked from reviewers addressed

Response to Reviewer 1

We thank the reviewer for the careful reading and the clear questions. We ran new experiments for each concern and added the results to the existing tables of the paper. Below, we answer every point in order. Unless stated otherwise, results are for ResNet-18 on CIFAR-10, 3 seeds, mean ±\pm s.d.; "saved" means the share of backward passes skipped.

The review is addressed in blue for reviewer 1, including in the manuscript: https://anonymous.4open.science/r/TG_IES_Results-8EA0/rebuttal_report.pdf


W1. "The algorithmic change seems incremental" (and the update scaling)

Concern. TG-IES keeps the IES skipping rule; on CIFAR-10 it raises savings only from 49.02% to 53.63%; on CIFAR-100 the larger savings come with lower accuracy; under fixed-rate SGD and Adam it saves less than IES; and IES and TG-IES may average gradients differently (1/∣At∣1/|A_t| against 1/N1/N).

Response.

(a) Where the gain comes from (also Q1). We calibrated one component at a time, with everything else, including the update scaling, kept the same (Table 3 of the paper):

CalibratedAcc. (%)Saved (%)δ\deltaK∗K^*
IES (literal), neither94.85±0.1794.85\pm0.1749.02±0.4449.02\pm0.440.0010.00111
δ\delta only94.73±0.0994.73\pm0.0967.02±0.3667.02\pm0.360.00760.007611
K∗K^* only95.03±0.0295.03\pm0.023.46±0.273.46\pm0.270.0010.0011515
Both (TG-IES)94.85±0.2894.85\pm0.2853.63±0.3053.63\pm0.300.01660.01661515

The savings above IES come from the noise-calibrated threshold δ\delta. The patience K∗K^\ast makes the rule more conservative and gives back 13.413.4 points of that saving. With the patience alone, IES's strict threshold leaves almost nothing to exclude. CIFAR-100 shows the same pattern: δ\delta only saves 70.12%, K∗K^\ast only 1.01%, both 57.76%, against 18.49% for IES (literal).

(b) Update scaling. We checked the released IES code: it computes losses.mean() per mini-batch with drop_last=True, so IES and TG-IES both average over the mini-batch and take the same step size. The "1/∣At∣1/|A_t|" description holds only for full-batch descent, and we have corrected it in the paper. A new lemma (Appendix A.2) shows the two scalings agree at the epoch level to first order. We also ran TG-IES with every step rescaled by N/∣At∣N/|A_t|:

ScalingAcc. (%)Saved (%)
1/N1/N (TG-IES)94.85±0.2894.85\pm0.2853.63±0.3053.63\pm0.30
1/∥At∥1/\|A_t\|94.78±0.0994.78\pm0.0952.17±0.1552.17\pm0.15
IES (literal), for reference94.85±0.1794.85\pm0.1749.02±0.4449.02\pm0.44

The two scalings differ by 0.070.07 points of accuracy and 1.461.46 points of savings, so the gain over IES does not come from the scaling. We keep 1/N1/N because it keeps the stability identity of the theory exact.

(c) Size of the gain. On CIFAR-10 the gain is 4.64.6 points of backward passes at the same accuracy. It is modest, but it has the same sign in all three paired seeds (95% interval [4.00,5.22][4.00, 5.22]). With the same constants, never re-tuned, TG-IES saves more than IES (literal) in every setting with a decaying learning rate:

SettingIES (literal) saved (%)TG-IES saved (%)
VGG-16 / CIFAR-1041.0241.0254.4354.43
DenseNet-121 / CIFAR-1051.6051.6054.5354.53
ResNet-18 / CIFAR-10018.4918.4957.7657.76
ViT-Tiny / CIFAR-1016.6816.6825.5225.52
ViT-Tiny / CIFAR-10012.5112.5126.2826.28
ResNet-18 / CIFAR-10, T=100T=10041.9241.9244.2344.23
ResNet-50 / ImageNet-1002.762.765.925.92

We agree that on CIFAR-100 the larger saving is a trade: TG-IES is 0.770.77 points less accurate than IES (literal). We state this in the paper rather than calling it a pure win. Beyond the savings numbers, the contribution is that the threshold is set from each run's own noise instead of by hand, together with the theory that explains the signal and the stability of exclusion.

(d) Fixed-rate SGD and Adam. We agree, and the paper keeps this as a stated limitation. TG-IES saves 29.27% against 49.42% under fixed-rate SGD and 30.18% against 60.09% under Adam, at equal accuracy. A candidate reason: δ\delta is the noise level at tct_c. With a decaying step the noise keeps shrinking after tct_c (about 41×41\times from epoch 31 to 200), so many signals fall below the fixed δ\delta. With a constant step it shrinks only about 12×12\times, so fewer instances complete K∗K^* passing checks. Replaying the rule on the logged losses already predicts this ordering (58.3%, 23.6%, 28.7% open loop against 53.6%, 29.3%, 30.2% in training). This supports the explanation but does not prove it.

Changes in the paper. New rows in Table 3 (component, scaling); the scaling lemma and corrected description in Appendix A.2; the abstract, introduction, Figure 1 and contributions now say that the gain comes from δ\delta and state the schedule limitation.


AuthorsOpenReview ↗

Continued Revisions

##Continued Revisions

W2. Calibration epoch, patience K∗=15K^*=15, and shorter runs

(a) Why epoch 30, and is it sensitive? (also Q2)

tct_c is not tuned. It is a fixed fraction of the run, tc=0.15 Tt_c=0.15\,T, the same in every setting. Under the exponential decay it marks the same point of the schedule for every run length: η(tc)=0.29 η0\eta(t_c)=0.29\,\eta_0 for both T=200T=200 (tc=30t_c=30) and T=100T=100 (tc=15t_c=15).

We ran tc=20,30,50t_c=20, 30, 50 (3 seeds, new rows of Table 3):

tct_cCIFAR-10 acc. (%)CIFAR-10 saved (%)δ\deltaCIFAR-100 acc. (%)CIFAR-100 saved (%)
202094.74±0.2194.74\pm0.2158.57±0.2558.57\pm0.250.03770.037775.76±0.4075.76\pm0.4062.42±0.1062.42\pm0.10
3030 (reported)94.85±0.2894.85\pm0.2853.63±0.3053.63\pm0.300.01660.016676.01±0.0876.01\pm0.0857.76±0.0457.76\pm0.04
505094.84±0.0494.84\pm0.0439.98±0.4639.98\pm0.460.00370.003776.69±0.7376.69\pm0.7332.94±0.2432.94\pm0.24
  • Savings depend strongly on tct_c. Later calibration gives a smaller δ\delta, because the noise shrinks during training, so fewer instances are skipped.
  • On CIFAR-10, accuracy is flat over tc∈[20,50]t_c\in[20,50], so tct_c acts as a savings dial.
  • On CIFAR-100, accuracy rises with tct_c (+0.93+0.93 points from 20 to 50), so there tct_c trades savings for accuracy.
  • Calibrating too early is risky. Replaying the rule on the logged losses, more exclusions still have learnable progress left at tc=10t_c=10 (24%) than at tc=30t_c=30 (19%).

We chose 30 as a middle value between these two effects, not as an optimum: tc=20t_c=20 saves 5 points more at 0.110.11 points lower accuracy on CIFAR-10. We calibrate once rather than repeatedly, because recalibrating keeps tightening δ\delta as the noise shrinks; a configuration that recalibrated collapsed to 1.8% savings on VGG-16, against 54.4% with a single calibration.

(b) Patience: the formula always gives more than 15

We agree. In our primary setting the reference horizon is Ksafe≈1.15×106K_{\mathrm{safe}}\approx1.15\times10^6, so the cap Kmax⁡=15K_{\max}=15 sets K∗K^* in every run, and the formula does not choose the patience in practice. We have removed the claim that the patience is derived from it. The bound limits how long an excluded instance can stay excluded; it does not decide how many checks are needed before exclusion.

We now justify the value of 15 as error control over repeated noisy checks. With the error budget p=0.3p=0.3 and a margin of k(K)6 σk(K)\sqrt6\,\sigma, the probability bound of an erroneous exclusion depends on KK alone:

KK881010121215151616
Bound1.8×10−21.8\times10^{-2}2.2×10−32.2\times10^{-3}1.2×10−31.2\times10^{-3}8.6×10−58.6\times10^{-5}9.5×10−69.5\times10^{-6}

Kmax⁡=15K_{\max}=15 is the smallest KK whose bound is below 10−410^{-4}. The bound falls so steeply that the budget matters little: budgets from 10−310^{-3} to 10−510^{-5} give KK between 13 and 16.

(c) Why 15 and not 8?

The reviewer is right that K∗=8K^*=8 gives the same mean accuracy with more savings (patience sweep, Table 9):

K∗K^*Acc. (%)Saved (%)
8894.80±0.0994.80\pm0.0958.85±0.5758.85\pm0.57
151594.80±0.0994.80\pm0.0953.38±0.5853.38\pm0.58
303094.87±0.0994.87\pm0.0944.13±0.4044.13\pm0.40

The difference between 8 and 15 is the risk, not the accuracy: the error bound at K∗=8K^\ast=8 is about 200×200\times larger (1.8×10−21.8\times10^{-2} against 8.6×10−58.6\times10^{-5}). In a replay on the logged losses, the share of exclusions that the rule later has to reverse falls from 46% (K=8K=8) to 26% (K=15K=15) and 9.7% (K=30K=30). There is no clear knee, so we present 15 as a conservative default, not an optimum, and report K∗=8K^*=8 as the cheaper alternative with a weaker guarantee.

(d) Shorter runs (also Q2)

For a run of TT epochs we use tc=⌈0.15 T⌉t_c=\lceil0.15\,T\rceil, rescale the decay to the same final learning rate (γ=0.96200/T\gamma=0.96^{200/T}), and keep Kmax⁡=15K_{\max}=15, since the patience does not depend on TT. We tested this at T=100T=100 (3 seeds, new row of Table 4):

T=100T=100Acc. (%)Saved (%)
Full training94.11±0.0794.11\pm0.07–
IES (literal)93.76±0.1093.76\pm0.1041.92±0.7841.92\pm0.78
TG-IES93.69±0.2293.69\pm0.2244.23±0.6244.23\pm0.62

TG-IES keeps its advantage (more savings in all three paired seeds), although the gain is smaller than at T=200T=200 (2.32.3 against 4.64.6 points). The ImageNet-100 runs below also use T=100T=100 with the same rule and no change of constants.

Changes in the paper. Calibration-epoch rows in Table 3 (CIFAR-10 and CIFAR-100); the T=100T=100 row in Table 4; the choice of tct_c and Kmax⁡K_{\max} explained in Appendix A.3 (calibration epoch, shorter runs) and Appendix A.4 (patience); the claim that patience is derived from KsafeK_{\mathrm{safe}} removed from the abstract, introduction, Figure 1 and contributions.


W3. ImageNet

We added ImageNet-100 with ResNet-50 (100 epochs, 3 seeds; new block of Table 2). The protocol and all TG-IES constants were fixed before the first run, and nothing was tuned on the validation images.

MethodBest (final) top-1 (%)Saved (%)
Full training72.49±0.7872.49\pm0.78 (72.15)0.16±0.000.16\pm0.00
IES (literal)70.33±0.7170.33\pm0.71 (69.85)2.76±0.442.76\pm0.44
InfoBatch70.95±0.5170.95\pm0.51 (70.62)30.80±0.2630.80\pm0.26
TG-IES70.33±0.5470.33\pm0.54 (69.71)5.92±0.025.92\pm0.02

(Full training shows 0.16% because the last incomplete mini-batch of each epoch is dropped.)

We report this result honestly:

  • Against IES (literal). TG-IES saves 3.23.2 points more, in every seed, at the same mean accuracy.
  • Against InfoBatch. InfoBatch saves much more (30.8%30.8\% against 5.9%5.9\%). The savings seen on CIFAR do not transfer to this setting, and the paper states this as a limitation.
  • Likely reason. After 100 epochs the mean training loss is still 0.620.62 on ImageNet-100, against 0.0020.002 on CIFAR-10 and 0.0100.010 on CIFAR-100, so few instances reach a stable loss. For seed 0, at most 8.5% of instances are passive at any epoch, against 87% on CIFAR-10. InfoBatch's rule is relative (below-mean loss) and does not need a stable loss. We did not log per-instance losses on ImageNet, so we cannot separate the effect of the threshold from that of the task.
  • The constants are not the reason. On seed 0, tc=10t_c=10 and 2525 save 2.2% and 1.5%, and K∗=8K^*=8 saves 3.5%, against 5.9% for the reported configuration.
  • A post-hoc variant. A variant with the consistency-corrected noise estimate, run after the first seed was seen and labelled as post hoc, saves 4.1% at 71.19±1.3171.19\pm1.31 and does not change the conclusion.

ImageNet-1K and longer schedules, where the training loss gets closer to zero, are untested and listed as future work.


Q1. How much of the improvement comes from the noise-based threshold?

Almost all of it. With the update scaling shared and patience K∗=1K^*=1 as in IES, calibrating only δ\delta raises savings from 49.0% to 67.0% (CIFAR-10) and from 18.5% to 70.1% (CIFAR-100). Calibrating only the patience saves 3.5% and 1.0%. The reported TG-IES (both) lies in between: 53.6% and 57.8%. See W1 (a) and (b).

Q2. Why epoch 30? Would 20 or 50 change the results? How to choose tct_c and patience for a shorter run?

tc=0.15 Tt_c=0.15\,T is a fixed rule, not a tuned value. Calibrating at 20 or 50 changes the savings a lot (58.6% and 40.0% against 53.6% on CIFAR-10) but not the CIFAR-10 accuracy; on CIFAR-100 a later calibration gives higher accuracy and lower savings. For a shorter run, use tc=⌈0.15 T⌉t_c=\lceil0.15\,T\rceil with the decay rescaled to the same final rate, and keep Kmax⁡=15K_{\max}=15; at T=100T=100 this still beats IES (literal) in all three seeds. See W2 (a) and (d).


We thank the reviewer again. These experiments made the paper clearer about where the gain comes from, how the two constants are set, and where the method does not help (fixed-rate schedules and ImageNet-100 at 100 epochs).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to the Reviewers

We sincerely thank all three reviewers for their thoughtful comments and constructive suggestions. We have addressed the issues and found stronger findings in our experiments.

Here is the rebuttal paper: https://anonymous.4open.science/r/TG_IES_Results-8EA0/rebuttal_report.pdf

Here is the fully implemented code: https://anonymous.4open.science/r/TG_IES_Results-8EA0

We have carefully revised the manuscript and conducted additional experiments to address the concerns raised. Below, we briefly summarize the major revisions and explain how the reviewers' comments have been incorporated.

1. Color-coded changes in the revised manuscript

To make the revisions easier to follow, we have highlighted the changes using the following colors:

  • Blue – Reviewer 1
  • Red – Reviewer 2
  • Green – Reviewer 3
  • Orange – Comments shared by multiple reviewers

When a particular revision addresses concerns from multiple reviewers, we have used the corresponding reviewer colors to identify the changes.

2. Point-by-point responses to reviewer comments

We have addressed the reviewers' comments individually and identified each response using the original comment references (e.g., R1-W3, R2-Q1, R3-W2). The corresponding changes can be located in the revised manuscript using these identifiers.

Reviewer 1 – Blue

  • R1-W1, R1-Q1: We added component-wise ablation experiments to clarify the individual contributions of threshold calibration and patience. We also examined alternative update-scaling strategies.
  • R1-W2, R1-Q2: We conducted additional experiments on calibration timing, patience selection, and shorter training schedules. We also evaluated these choices on CIFAR-100 and ImageNet-100.
  • R1-W3: We added ImageNet-100 experiments and expanded comparisons with IES, InfoBatch, and other relevant baselines, including comparisons at matched computational savings.

Reviewer 2 – Red

  • R2-W1, R2-Q2: We revisited the comparison between different difference orders, investigated the selection-bias concern, and added fixed-epoch and SNR-stratified evaluations.
  • R2-W2, R2-Q3: We clarified the role of the patience parameter, explained the choice of (K∗=15)(K^\ast=15), and provided an additional analysis of repeated noisy decisions.
  • R2-W3, R2-Q1: We provided a more detailed explanation of the noise-scale estimator and added validation and sensitivity analyses.
  • R2-W4, R2-Q4: We clarified the warm-up procedure, training timeline, and the effects of implementation nondeterminism.

Reviewer 3 – Green

  • R3-W1: We added empirical measurements of the omitted-gradient residual and examined the conditions required by our stability analysis.
  • R3-W2: We expanded the evaluation across learning-rate schedules and clarified the limitations observed with fixed-rate SGD and Adam.
  • R3-W3: We added comparisons with IES+ and evaluated TG-IES on AG News to investigate its applicability beyond image classification.
  • R3-W4: We expanded the theoretical and empirical comparison of different loss-difference orders, including a direct comparison between N=0N=0 and N=2N=2.

3. Additional experiments and analysis

Beyond clarifying the manuscript, we conducted additional experiments to directly investigate the reviewers' questions and concerns. These include component ablations, calibration and patience sensitivity analyses, noisy-label experiments, shorter training schedules, ImageNet-100 evaluations, and experiments on AG News.

We also expanded our comparisons with existing methods, including IES+, InfoBatch, BLS-InfoBatch, AlignPrune, and Selective Backprop. The corresponding results and analyses have been incorporated into the revised manuscript and supplementary materials.

4. Updated experimental codebase

We have updated the experimental codebase to include the revised implementations and additional experiments. We have also incorporated the necessary implementation and reproducibility-related updates.

5. Supplementary rebuttal report

We have included a separate supplementary document titled Rebuttals in Points. It provides a point-by-point mapping of reviewer comments to the corresponding experiments, results, and explanations.

We request that the reviewers consult this document alongside the revised manuscript. It provides additional details about the experimental protocols and results clearly.

Once again, we sincerely appreciate the reviewers' time and constructive feedback. We believe these revisions have made the manuscript more rigorous, transparent, and comprehensive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.