acceptodds
Under review as a conference paper at ICLR 2027

Epoch Pruning: Accelerating Model Training via Gradient Replay

Abstract

Training deep neural networks typically requires a prolonged training process. However, as training proceeds and the network gradually converges, the rate of performance improvement tends to diminish. While performance growth slows down, the training overhead stays the same. Thus, a natural question arises: can we adopt a differentiated strategy for the later stages of training, accelerating training and saving resources while maintaining effectiveness? Motivated by this, we introduce a novel concept termed Epoch Pruning. By reusing previously computed gradients, we can bypass the computationally expensive forward and backward propagation of certain later-stage epochs and update the model directly. Based on gradient replay, we devise two effective strategies to preserve the validity of the reused gradients. Extensive experiments conducted on multiple datasets, under both supervised and unsupervised learning settings, demonstrate that the proposed methods achieve approximately 1.2× to 1.4× speedup while maintaining the original model performance. **The code is available in the Supplementary Material.**

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model Training Acceleration, Efficient Training, Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sTwiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Epoch Pruning, replacing selected late-stage training epochs with synthetic gradient updates that avoid forward and backward passes. SGR samples from the preceding active epoch's gradient moments with progressive weighting; MGR averages moments across active epochs. Experiments cover supervised classification, self-supervised learning, and combinations with data pruning, reporting approximately 1.2–1.4× estimated speedups. Conditional one-step descent bounds support the analysis.

Critical strengths
  1. The method offers a simple, inexpensive way to exploit historical gradients, requiring only per-parameter moments. Its compatibility with data pruning suggests useful complementary savings.

  2. The evaluation spans several architectures, optimizers, and learning settings. Table 7 provides useful evidence that replay contributes beyond skipping updates: on CIFAR-100, SGR/MGR improve over the no-replay ablation by 0.91/1.10 percentage points.

Critical weaknesses
  1. The accuracy–time advantage needs stronger validation. Appendix E extrapolates speedups using a common overhead ratio of 0.1, with component timings limited to ResNet-18/CIFAR-10. Table 7 includes skipping and early-stopping controls, but does not establish an advantage over shorter training with a schedule adapted to its budget, as motivated by Budgeted Training (ICLR 2020), at matched wall-clock time. This matters especially on ImageNet: SGD-Cosine accuracy falls from 72.99% to 71.52%/71.50%, making the claim of preserved performance too broad. Appendix D also uses half the test set for stopping decisions before reporting results on the full test set, compromising that comparison's held-out evaluation.

  2. The theoretical stability interpretation is overstated. Appendix H.3 claims that MGR's variance satisfies Vˉt≤max⁡kVk\bar V_t\leq\max_k V_k. Averaging raw first and second moments also introduces variance between epoch means, so this inequality generally fails. Two scalar epochs with means δ,−δ\delta,-\delta, variance 0.01δ20.01\delta^2 each, and EMA weights 0.9,0.10.9,0.1 yield replay variance 0.37δ20.37\delta^2, even for arbitrarily small δ>0\delta>0. The conditional descent inequality does not establish the claimed variance suppression. Moreover, negligible drift between historical and current gradients is assumed; Appendix F's within-epoch normality tests do not establish this central assumption.

Questions for authors
  1. Empirical validation: Can you provide synchronized end-to-end timing and accuracy against budget-adapted shorter training, particularly on ImageNet, with independent validation and test splits for stopping decisions?

  2. Theoretical clarification: How should the variance claim change when epoch means differ, and what evidence bounds historical-gradient drift throughout a replay epoch?

  3. Implementation clarification and ablation: The supplied SGR/MGR code directly subtracts synthetic gradients, bypassing optimizer momentum/adaptive-state updates and weight decay during replay. Is this intended across the reported optimizer settings? Under this update rule, how much does Gaussian sampling improve over replaying only the mean?

2 comments
AuthorsOpenReview ↗

Response to Reviewer sTwi (1/2)

Thank you for your careful review and constructive comments. We have carefully addressed all of the issues raised. We have also revised the corresponding parts in the manuscript and marked them in blue. Below is our response:

Component timings limited to ResNet-18/CIFAR-10 (Weakness 1)

Response: We have re-conducted the runtime breakdown measurements across the all used datasets as well as multiple network architectures. The experimental results are presented in Tables 10–15 at the end of the manuscript. The new results include detailed component-level timing and speedup for all experiments reported in Tables 1–6.

Comparison with Budget Training (Weakness 1 and Question 1)

Response: We carefully reviewed the Budget Training paper and cited it in the Related Work section. Due to limited computational resources, we conducted experiments on Tiny ImageNet using the SGD-Cosine configuration. Under the same budget, Budget Training achieved slightly lower performance than SGR or MGR. In addition, SGR and MGR are compatible with Budget Training. If Budget Training and Epoch Pruning are used simultaneously, further acceleration can be achieved at a negligible performance cost. The results are presented below.

MethodAccuracy (%)
No Pruning63.95 ± 0.37
SGR63.80 ± 0.21
MGR63.70 ± 0.50
Budgeted Training63.10 ± 0.22
Budgeted Training + SGR62.71 ± 0.27
Budgeted Training + MGR62.84 ± 0.30

Concerns on performance drop on ImageNet SGD-Cosine in Table 2 (Weakness 1)

Response: We acknowledge that enabling SGR or MGR during the latter half of ImageNet training results in a performance drop of approximately 1.5% under the SGD-Cosine configuration. However, there is no significant performance gap, indicating that the proposed methods preserve performance during the gradient-replay stage and achieves an effect close to that of normal training. Moreover, model training is accelerated by about 1.3 ×\times, greatly reducing training time. Meanwhile, under other methods or models, the performance loss caused by acceleration is even smaller, demonstrating the effectiveness of the proposed method in most scenarios. The performance drop on ImageNet is related to the scale and difficulty of this dataset, and we expect that future epoch pruning methods can further improve this. We have revised the wording in the ImageNet experimental section to state that our method achieves comparable performance while also acknowledging the performance drop raised by the reviewer, so as to avoid ambiguity.

Appendix D also uses half the test set for stopping decisions before reporting results on the full test set (Weakness 1 and Question 1)

Response: Thank you for pointing out this issue. Our original experimental setup was intended to ensure that early stopping and the main experiments were evaluated on the same test set. We have revised the experimental setup and re-run the experiments, partitioning an independent validation set from the training set by sampling 10% of the samples per class. The results are as follows and still demonstrate the performance improvement of our method. We have also revised the corresponding statements and experimental results in the Appendix D.

MethodCIFAR-10 Accuracy (%)CIFAR-100 Accuracy (%)
Early Stopping (ES)87.28 ± 4.3663.18 ± 8.39
SGR93.81 ± 0.0673.72 ± 0.09
MGR93.63 ± 0.3173.09 ± 0.54
AuthorsOpenReview ↗

Response to Reviewer sTwi (2/2)

Theoretical clarification on MGR variance and gradient drift during replay (Weakness 2 and Question 2)

Response: We agree that the variance upper bound in Appendix H.3 is incorrect in general. Let aka_k be the normalized EMA weight of recorded active epoch kk, and let μk\mu_k and VkV_k be its gradient mean and variance trace. With μˉ=∑kakμk\bar\mu=\sum_k a_k\mu_k, Eqs. (6)–(7) give

Vˉt=∑kakVk+∑kak∥μk−μˉ∥2.\bar V_t=\sum_k a_k V_k+\sum_k a_k\lVert\mu_k-\bar\mu\rVert^2.

In the reviewer’s example, the within-epoch and between-epoch terms are 0.01δ20.01\delta^2 and 0.36δ20.36\delta^2, respectively, yielding Vˉt=0.37δ2\bar V_t=0.37\delta^2. We will replace the claimed upper bound with this decomposition. At the same step size, MGR reduces the variance penalty relative to unweighted replay from the latest recorded epoch only when the reduction in the weighted within-epoch variance exceeds the between-epoch term. Theorem 2 uses the actual replay variance Vˉt\bar V_t.

For drift, the submitted experiments measure within-epoch marginal Gaussian fit in Appendix F. The magnitude of replay-mean drift along a replay trajectory remains unmeasured in the submission. Let dt=∥μˉ−∇L(θt)∥d_t=\lVert\bar\mu-\nabla\mathcal L(\theta_t)\rVert and let t0t_0 be the first step of a replay epoch. Since the stored mean is fixed throughout the epoch, LL-smoothness gives

dt≤dt0+L∥θt−θt0∥.d_t\le d_{t_0}+L\lVert\theta_t-\theta_{t_0}\rVert.

This relates the drift at every replay step to the initial mismatch and the parameter displacement. We will clarify in Section 3.4 that a uniform bound on dtd_t across replay steps is an assumption, and that Theorem 2 gives a conditional one-step expected-descent bound under this assumption.

Implementation clarification about optimizer and ablation on replaying only the mean (Question 3)

Response: The direct update in Eqs. (5) and (8) is intentional and is shared across optimizer choices in the supplied scripts. The configured SGD, Adam, or AdamW optimizer is used in active epochs. In replay epochs, the implementation subtracts the synthetic gradient scaled by the current learning rate. We quantify the incremental benefit of Gaussian sampling compared to replaying only the mean under SGD-Cosine confirguration on CIFAR-10. The results support that Gaussian sampling improve over replaying only the mean.

MethodMean-Only Replay (%)Gaussian Replay (%)
SGR93.12 ± 0.1194.19 ± 0.12
MGR93.59 ± 0.1594.16 ± 0.14
Reviewer_sdPiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Epoch Pruning, a framework for accelerating model training by reusing gradients computed in previous epochs rather than repeatedly performing expensive forward and backward passes. It introduces two strategies: Single-step Gradient Replay (SGR), which reuses gradients from the previous epoch, and Momentum Gradient Replay (MGR), which incorporates gradients from multiple past epochs using momentum-based updates. The methods are evaluated in both supervised and unsupervised learning scenarios, achieving reported speedups of approximately 1.2×–1.4× while preserving performance.

Critical strengths
  1. The paper addresses an important practical problem: reducing the computational cost of neural network training. Its gradient replay approach is simple and intuitive, based on the observation that training progress may slow in later epochs while computational costs remain high. The method also avoids major architectural changes and can potentially be integrated into existing training pipelines. This research aspect is important because the computational cost of deep learning training is a longstanding issue.
Critical weaknesses
  1. The observation that training progress often slows in later epochs is well established in the optimization and training-efficiency literature. Although the paper presents epoch pruning as a new framework, the broader idea of reducing training costs by reusing historical optimization information or skipping expensive computations has been explored in prior work. The paper does not sufficiently clarify how its approach differs from closely related previous methods or what fundamental new insight it contributes, making its novelty and significance difficult to assess.
  2. The proposed SGR and MGR strategies provide relatively modest speedups, while performance degradation occurs in several settings, leaving their practical benefits uncertain. Moreover, the method depends on hyperparameters such as the pruning epoch (E) and replay interval (K), but the rationale for selecting these values across models and tasks is unclear. A systematic sensitivity analysis is needed to establish the robustness of the approach and its dependence on these choices. Finally, the evaluation lacks comparisons with relevant state-of-the-art training-acceleration and gradient-reuse methods. Without comparisons, it is difficult to determine whether the proposed framework offers a meaningful advantage over existing alternatives.
Questions for authors
  1. Can the author(s) explain how the proposed framework fundamentally differs from existing methods that reuse past information or reduce training computation?
  2. Can the author(s) answer: How are the pruning epoch (E) and replay interval (K) hyperparameter values selected for different models and tasks, and how sensitive are the results to these choices?
  3. What role does the replay interval (K) play, and how does it affect gradient reliability, model convergence, and training efficiency?
  4. Can the authors provide a clearer analysis of the trade-off between accuracy and speedup, including explicit speedup results for all experimental settings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer sdPi (1/2)

How Epoch Pruning differs from prior work (Weakness 1 and Question 1)

Response: Most methods that utilize stale gradients or historical information tend to correct or estimate the required gradients. They often still require the forward propagation process, while backpropagation is estimated or replaced. Compared with these algorithms, we directly generate replay gradients from the mean and standard deviation of stale gradients, avoiding both the forward and backward propagation training processes and thereby achieving faster acceleration. Experimental results demonstrate that the proposed methods preserve the performance of the original model well.

How EE and KK were selected and how sensitive they are (Weakness 2 and Question 2)

Response: EE determines when replay becomes eligible. We used shared defaults across the main experiments. For supervised learning, replay begins after the halfway point, with E=100E=100 of 200 epochs on CIFAR, E=45E=45 of 90 epochs on ImageNet and Tiny-ImageNet, and E=10E=10 of 20 epochs on the text tasks. For self-supervised learning, E=400E=400 of 600 leaves a longer active phase. We set K=2K=2, the most frequent replay schedule that still places an active epoch before every replay epoch.

EE and KK primarily determine the degree of acceleration and are user-selectable in our framework. In Appendix C, we have analyzed the sensitivity of EE and KK. As shown in Figure 3, different values of EE do not significantly affect performance; even when EE is reduced from the experimental setting of 100 epochs to 60 epochs, performance close to the No Pruning baseline can still be achieved, thereby completing training more quickly. In Figure 4, increasing KK does not yield greater performance gains; therefore, we choose KK = 2, the setting most conducive to acceleration.

What role does the replay interval (K) play, and how does it affect gradient reliability, model convergence, and training efficiency? (Question 3)

Response: Once replay is enabled, one of every KK epochs uses replay and the other K−1K-1 compute gradients as normal training. A larger KK therefore reduces how often the model relies on approximate updates. Every replay epoch uses statistics recorded in the immediately preceding active epoch, so increasing KK does not make SGR’s source statistics older. MGR additionally incorporates earlier statistics through its moving average. As shown in Figure 4 in Appendix C, Model performance is comparable across different values of KK, indicating similar reliability and convergence. Generally, increasing KK reduces the degree of model acceleration; however, in all experiments, KK = 2, the most accelerated setting, achieved good performance. Therefore, we can set KK = 2 by default rather than treating it as a hyperparameter tuned across different datasets.

Provide a clearer analysis of the trade-off between accuracy and speedup, including explicit speedup results for all experimental settings (Weakness 2 and Question 4)

Response: Our method achieves performance close to the baseline under different settings. The speedup is mainly determined by the experimental setup and is also related to the properties of the dataset. A detailed discussion is provided in Appendix E. We present the runtime breakdown and speedup for all experiments in Tables 10–15 at the end of the manuscript.

AuthorsOpenReview ↗

Response to Reviewer sdPi (2/2)

Comparisons with relevant state-of-the-art (Weakness 2)

Response: The reason we originally did not compare with state-of-the-art methods is that, for model acceleration methods, the performance upper bound is generally regarded as the original unaccelerated model. Since the performance of our methods is already close to this baseline, this is sufficient to demonstrate their performance advantage. In addition, because our methods adopt a new setting, comparisons with other methods are not easy to standardize. We currently provide performance comparisons with three state-of-the-art methods, including GradSamp[1], InfoBatch[2] and W-ReSprop[3], using a unified budget identical to that of our methods at EE = 100 and 60. The experimental results below demonstrate the superiority of our methods.

EE = 100 Budget:

CIFAR-10 (%)CIFAR-100 (%)
No Pruning94.30 ± 0.0774.03 ± 0.13
SGR94.19 ± 0.1273.99 ± 0.29
MGR94.16 ± 0.1474.18 ± 0.35
GradSamp91.41 ± 0.3365.59 ± 0.49
InfoBatch94.04 ± 0.0973.61 ± 0.36
W-ReSprop94.10 ± 0.0873.94 ± 0.41

EE = 60 Budget:

CIFAR-10 (%)CIFAR-100 (%)
No Pruning94.30 ± 0.0774.03 ± 0.13
SGR93.89 ± 0.1374.16 ± 0.37
MGR94.21 ± 0.1874.32 ± 0.13
GradSamp86.10 ± 0.9665.40 ± 1.12
InfoBatch94.04 ± 0.1471.42 ± 3.13
W-ReSprop94.04 ± 0.3272.95 ± 0.21

[1] Minimizing energy costs in deep learning model training: The Gaussian sampling approach

[2] Infobatch: Lossless training speed up by unbiased dynamic data pruning

[3] Resprop: Reuse sparsified backpropagation

I thank the author(s) for the responses. I am satisfied with the answers regarding the role of the replay interval and the speedup analysis. However, my main concerns remain.

Epoch Pruning is not a new concept. Ahmad et al. (2024) and Alam et al. (2026) already use a similar idea for early stopping of epochs and data pruning.

Second, how the hyperparameters are selected is still unclear. Only what values are selected for E and K is mentioned for each dataset. Why and how they are selected is not defined. Also, the sensitivity analyses are for the CIFAR family only, and it is still unclear. How a practitioner would select these values for a new task is not mentioned.

Additionally, the author(s) mentioned that they originally did not compare with state-of-the-art methods because the result is close to baseline. I would like to note that several pruning techniques match the whole-dataset (baseline) accuracy, including the one the author(s) compared against, InfoBatch. It is important to compare against the recent state-of-the-art methods to determine how the proposed method is different and efficient, and why practitioners will use the new method. Also, the state-of-the-art comparison is with the CIFAR family dataset, and the difference in accuracy is very minor. The limited increment could be due to noise. In addition, the comparison is missing any savings index, which is very important in an efficiency/pruning method. Without the savings, it is not possible to determine how good the method is explained by accuracy only.

Since the main concerns are still present, I am keeping my score.

References

Alam, M. M., Islam, S., & Islam, A. (2026). Batch pruning by activation stability. In The Fourteenth International Conference on Learning Representations.

Ahmad, S., Trahan, G., & Islam, A. (2024). When do Convolutional Neural Networks Stop Learning?. arXiv preprint arXiv:2403.02473.

Reviewer_WGiTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose "Epoch Pruning", which divides the training process into two stages. In the early stage, the model is trained normally. In the later stage, epochs are further divided into "Active Epochs" and "Replay Epochs". Active Epochs are also trained normally, and Replay Epochs skip both forward and backward propagation. Instead, synthetic gradients are generated based on the mean and standard deviation of historical gradients and then used to update the model parameters, thereby accelerating training. The authors further introduce two historical gradient statistics and replay strategies, Single-step Gradient Replay (SGR) and Momentum Gradient Replay (MGR). Their experiments demonstrate performance comparable to standard training while achieving a certain degree of speedup.

Critical strengths

The paper introduces an interesting perspective on Epoch Pruning and adapts ideas from classical gradient replay methods in continual learning to this setting. The experimental results are relatively promising.

Critical weaknesses
  1. The hyperparameter configurations in the experiments appear potentially suboptimal, or the models may not have fully converged. For example, the results in Table 2 are approximately three percentage points lower than the official baseline without additional training tricks [1][2]. In addition, although the authors state that the training procedure is identical during the first 100 epochs, the curves in Appendices B and G do not overlap in their early stages.
  2. The authors observe that performance improvements diminish during the later stages of training. But this observation does not necessarily imply that later training epochs are less important.
  3. The related work section appears somewhat outdated and does not sufficiently cover other closely related studies, such as [3]. I believe there is a considerable body of research on gradient reuse, and the authors may need to provide a more comprehensive discussion of the existing literature and clarify how their approach differs from previous methods.
  4. Moreover, Table2 experiment about ImageNet--SGD-Cosine can't support the proposed method.
    

References [1] https://docs.pytorch.org/vision/stable/models/generated/torchvision.models.resnet50 [2] https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/ [3] The Lingering of Gradients: How to Reuse Gradients over Time

Questions for authors
  1. Could the authors explain why the curves in Appendix G do not overlap during the first half of training?
  2. Why would diminishing performance improvements during later training epochs imply that these epochs are less important?
3 comments
AuthorsOpenReview ↗

Response to Reviewer WGiT (1/2)

Thank you for your constructive comments. We have carefully considered your feedback and provided the response as below. We have also revised the corresponding parts in the manuscript and marked them in blue.

Concerns on hyperparameter configurations of ImageNet-1K baseline (Weakness 1)

Response: Thank you for pointing out this gap between our ImageNet-1K SGD-Cosine No Pruning baseline and TorchVision ResNet-50 V1. The primary goal of the proposed method is to accelerate training under different experimental settings while maintaining final performance comparable to that of the baseline. Accordingly, whether the baseline achieves the best absolute performance is not the central focus of this work; rather, the key is to evaluate, under consistent experimental settings, the training-efficiency gains and the performance-preservation capability of the proposed method. Although our model achieves lower performance, the baseline and the proposed SGR and MGR are trained and evaluated under exactly the same experimental settings. Therefore, the comparisons within this work are fair. The performance gap relative to TorchVision ResNet-50 V1 is attributable to differences in experimental settings. As reported in Appendix A, our supervised SGD experiments on CIFAR-10/100, Tiny-ImageNet, and ImageNet-1K use the same base learning rate of 0.1 and weight decay of 5×10−45\times10^{-4}, with the learning-rate schedule set according to the training duration. TorchVision V1 also uses 90 epochs and a base learning rate of 0.1, with StepLR decays every 30 epochs and a weight decay of 10−410^{-4}. Due to limited computational resources, our ImageNet-1K runs also use automatic mixed precision (AMP), which executes selected operations in FP16. The reduced numerical precision can affect the optimization trajectory and may have contributed to the lower accuracy. Also, we ensure that all reported results are faithfully recorded and that the comparisons are conducted fairly.

Early differences in Appendices B and G (Weakness 1 and Question 1)

Response: We apologize for this confusion. Each curve in Appendices B and G comes from a separate training run initialized from scratch. All runs follow the same experimental setting during the first 100 epochs. All reported results are authentic and faithfully recorded. This setup lets us evaluate each method end to end without making the comparison depend on a particular pre-replay checkpoint. Despite differences in the curves, the trends they reflect are consistent. We clarify this in the revised manuscript.

### The role of late training (Weakness 2 and Question 2)

Response: We apologize for the misunderstanding. In the paper, we do not claim that later-stage epochs are unimportant. On the contrary, our method still updates the network weights during these epochs, but accelerates model training via gradient replay. Inspired by our experimental observations, as shown in Figure 1 of the manuscript, the performance gains from later-stage training are noticeably smaller than those from early-stage training. Therefore, methods specifically designed to accelerate later-stage training are important, and we propose two gradient replay methods to address this issue.

Prior work on gradient reuse and how our approach differs from previous methods (Weakness 3)

Response: Thank you for pointing us to The Lingering of Gradients. We have carefully read the article and discussed it in Related Work section. We have rewritten the Related Work section to disucss other closely related studies. Specifically, we have expanded the Gradient Replay subsection to cover selective gradient refresh in finite-sum and distributed optimization, variance-reduced estimators, and predicted or delayed gradients for decoupling backpropagation. We also discuss recent methods that reuse a previous gradient to avoid an extra gradient evaluation or cache regularization-gradient components to avoid some auxiliary backward passes.

Existing gradient reuse methods often cache gradients and refresh them selectively, or combine historical gradients with fresh evaluations to improve the current estimate. In contrast, we use the mini-batch gradients actually computed during normal training to estimate the first and second moments separately for each parameter along the batch dimension, and then sample and replay gradients from the distributions determined by these statistics for the next epoch. Our approach retains distinct and timely gradient distributions for different parameters, so as to skip the forward and backward propagation processes, which is a new paradigm.

AuthorsOpenReview ↗

Response to Reviewer WGiT (2/2)

Concerns on ImageNet SGD-Cosine in Table 2 (Weakness 4)

Response: We acknowledge that enabling SGR or MGR during the latter half of ImageNet training results in a performance drop of approximately 1.5% under the SGD-Cosine configuration. However, there is no significant performance gap, indicating that the proposed methods preserve performance during the gradient-replay stage and achieves an effect close to that of normal training. Moreover, model training is accelerated by about 1.3 ×\times, greatly reducing training time. Meanwhile, under other methods or models, the performance loss caused by acceleration is even smaller, demonstrating the effectiveness of the proposed method in most scenarios. The performance drop on ImageNet is related to the scale and difficulty of this dataset, and we expect that future epoch pruning methods can further improve this. We have revised the wording in the ImageNet experimental section to state that our method achieves comparable performance while also acknowledging the performance drop raised by the reviewer, so as to avoid ambiguity.

Reviewer_WGiTOpenReview ↗

Reply to authors

Thank you for your response. However, I remain concerned that the proposed gradient replay may limit the model's convergence performance.

I would encourage the authors to reproduce the official ResNet-50 baseline in [1], using its complete training recipe and hyperparameter configurations, and then apply SGR and MGR to this baseline without changing the other training settings.

At a minimum, please consider both training recipes provided in [1]: the standard ResNet-50 V1 recipe (without additional training tricks) and the improved V2 recipe (with the full training tricks). For each setting, please report the accuracy and training speedup of the original baseline, baseline + SGR, and baseline + MGR.

This would help verify whether the proposed method remains effective under both standard and well-optimized training configurations, making the experimental evidence more convincing.

[1] TorchVision ResNet-50

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.