Epoch Pruning: Accelerating Model Training via Gradient Replay

Training deep neural networks typically requires a prolonged training process. However, as training proceeds and the network gradually converges, the rate of performance improvement tends to diminish. While performance growth slows down, the training overhead stays the same. Thus, a natural question arises: can we adopt a differentiated strategy for the later stages of training, accelerating training and saving resources while maintaining effectiveness? Motivated by this, we introduce a novel concept termed Epoch Pruning. By reusing previously computed gradients, we can bypass the computationally expensive forward and backward propagation of certain later-stage epochs and update the model directly. Based on gradient replay, we devise two effective strategies to preserve the validity of the reused gradients. Extensive experiments conducted on multiple datasets, under both supervised and unsupervised learning settings, demonstrate that the proposed methods achieve approximately 1.2× to 1.4× speedup while maintaining the original model performance. **The code is available in the Supplementary Material.**