- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Epoch Pruning, replacing selected late-stage training epochs with synthetic gradient updates that avoid forward and backward passes. SGR samples from the preceding active epoch's gradient moments with progressive weighting; MGR averages moments across active epochs. Experiments cover supervised classification, self-supervised learning, and combinations with data pruning, reporting approximately 1.2–1.4× estimated speedups. Conditional one-step descent bounds support the analysis.
Critical strengths
-
The method offers a simple, inexpensive way to exploit historical gradients, requiring only per-parameter moments. Its compatibility with data pruning suggests useful complementary savings.
-
The evaluation spans several architectures, optimizers, and learning settings. Table 7 provides useful evidence that replay contributes beyond skipping updates: on CIFAR-100, SGR/MGR improve over the no-replay ablation by 0.91/1.10 percentage points.
Critical weaknesses
-
The accuracy–time advantage needs stronger validation. Appendix E extrapolates speedups using a common overhead ratio of 0.1, with component timings limited to ResNet-18/CIFAR-10. Table 7 includes skipping and early-stopping controls, but does not establish an advantage over shorter training with a schedule adapted to its budget, as motivated by Budgeted Training (ICLR 2020), at matched wall-clock time. This matters especially on ImageNet: SGD-Cosine accuracy falls from 72.99% to 71.52%/71.50%, making the claim of preserved performance too broad. Appendix D also uses half the test set for stopping decisions before reporting results on the full test set, compromising that comparison's held-out evaluation.
-
The theoretical stability interpretation is overstated. Appendix H.3 claims that MGR's variance satisfies . Averaging raw first and second moments also introduces variance between epoch means, so this inequality generally fails. Two scalar epochs with means , variance each, and EMA weights yield replay variance , even for arbitrarily small . The conditional descent inequality does not establish the claimed variance suppression. Moreover, negligible drift between historical and current gradients is assumed; Appendix F's within-epoch normality tests do not establish this central assumption.
Questions for authors
-
Empirical validation: Can you provide synchronized end-to-end timing and accuracy against budget-adapted shorter training, particularly on ImageNet, with independent validation and test splits for stopping decisions?
-
Theoretical clarification: How should the variance claim change when epoch means differ, and what evidence bounds historical-gradient drift throughout a replay epoch?
-
Implementation clarification and ablation: The supplied SGR/MGR code directly subtracts synthetic gradients, bypassing optimizer momentum/adaptive-state updates and weight decay during replay. Is this intended across the reported optimizer settings? Under this update rule, how much does Gaussian sampling improve over replaying only the mean?
2 comments
Response to Reviewer sTwi (2/2)
Theoretical clarification on MGR variance and gradient drift during replay (Weakness 2 and Question 2)
Response: We agree that the variance upper bound in Appendix H.3 is incorrect in general. Let be the normalized EMA weight of recorded active epoch , and let and be its gradient mean and variance trace. With , Eqs. (6)–(7) give
In the reviewer’s example, the within-epoch and between-epoch terms are and , respectively, yielding . We will replace the claimed upper bound with this decomposition. At the same step size, MGR reduces the variance penalty relative to unweighted replay from the latest recorded epoch only when the reduction in the weighted within-epoch variance exceeds the between-epoch term. Theorem 2 uses the actual replay variance .
For drift, the submitted experiments measure within-epoch marginal Gaussian fit in Appendix F. The magnitude of replay-mean drift along a replay trajectory remains unmeasured in the submission. Let and let be the first step of a replay epoch. Since the stored mean is fixed throughout the epoch, -smoothness gives
This relates the drift at every replay step to the initial mismatch and the parameter displacement. We will clarify in Section 3.4 that a uniform bound on across replay steps is an assumption, and that Theorem 2 gives a conditional one-step expected-descent bound under this assumption.
Implementation clarification about optimizer and ablation on replaying only the mean (Question 3)
Response: The direct update in Eqs. (5) and (8) is intentional and is shared across optimizer choices in the supplied scripts. The configured SGD, Adam, or AdamW optimizer is used in active epochs. In replay epochs, the implementation subtracts the synthetic gradient scaled by the current learning rate. We quantify the incremental benefit of Gaussian sampling compared to replaying only the mean under SGD-Cosine confirguration on CIFAR-10. The results support that Gaussian sampling improve over replaying only the mean.
| Method | Mean-Only Replay (%) | Gaussian Replay (%) |
|---|---|---|
| SGR | 93.12 ± 0.11 | 94.19 ± 0.12 |
| MGR | 93.59 ± 0.15 | 94.16 ± 0.14 |
Response to Reviewer sTwi (1/2)
Thank you for your careful review and constructive comments. We have carefully addressed all of the issues raised. We have also revised the corresponding parts in the manuscript and marked them in blue. Below is our response:
Response: We have re-conducted the runtime breakdown measurements across the all used datasets as well as multiple network architectures. The experimental results are presented in Tables 10–15 at the end of the manuscript. The new results include detailed component-level timing and speedup for all experiments reported in Tables 1–6.
Response: We carefully reviewed the Budget Training paper and cited it in the Related Work section. Due to limited computational resources, we conducted experiments on Tiny ImageNet using the SGD-Cosine configuration. Under the same budget, Budget Training achieved slightly lower performance than SGR or MGR. In addition, SGR and MGR are compatible with Budget Training. If Budget Training and Epoch Pruning are used simultaneously, further acceleration can be achieved at a negligible performance cost. The results are presented below.
Response: We acknowledge that enabling SGR or MGR during the latter half of ImageNet training results in a performance drop of approximately 1.5% under the SGD-Cosine configuration. However, there is no significant performance gap, indicating that the proposed methods preserve performance during the gradient-replay stage and achieves an effect close to that of normal training. Moreover, model training is accelerated by about 1.3 ×, greatly reducing training time. Meanwhile, under other methods or models, the performance loss caused by acceleration is even smaller, demonstrating the effectiveness of the proposed method in most scenarios. The performance drop on ImageNet is related to the scale and difficulty of this dataset, and we expect that future epoch pruning methods can further improve this. We have revised the wording in the ImageNet experimental section to state that our method achieves comparable performance while also acknowledging the performance drop raised by the reviewer, so as to avoid ambiguity.
Response: Thank you for pointing out this issue. Our original experimental setup was intended to ensure that early stopping and the main experiments were evaluated on the same test set. We have revised the experimental setup and re-run the experiments, partitioning an independent validation set from the training set by sampling 10% of the samples per class. The results are as follows and still demonstrate the performance improvement of our method. We have also revised the corresponding statements and experimental results in the Appendix D.