- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper attempts to reduce the memory consumption of AdamW by not tracking the first moment in AdamW. In doing so, the authors first compute the batch-averaged gradients of the final output layer for a classification task, where the gradient for each sample in the batch is the predicted probability vector minus the one-hot vector. The authors then compute a moving average of batch-averaged gradients across iterations, which is referred to as 'backpropagated output momentum (BOM)'. BOM is then utilized when performing backpropagation for the current batch. The new method still computes and maintains the second moment by using the computed gradient as in AdamW. Finally, the model is updated following the AdamW style, replacing the first momentum in AdamW with only the computed gradient. As a result, the new method saves memory consumption and slightly reduces computational complexity. Experiments in natural language processing show that the new training method produces better validation performance.
,
Critical strengths
In brief, the paper is easy to follow. The authors propose an innovative method for computing backpropagated output momentum (BOM). BOM may carry additional output error information missing from AdamW, which could be helpful in training DNN models for classification tasks.
Critical weaknesses
While it is clear that the proposed training method reduces memory usage, it remains unclear why it also improves validation performance. Is this improvement genuinely driven by BOM carrying additional output error information absent from AdamW? The authors performed an ablation study in Table 2, which is helpful. However, I think further analysis or additional ablation studies are needed to clarify the exact cause of this performance gain.
Questions for authors
(1) When plugging BOM into AdamW, the framework uses three moving average factors (denoted as beta0, beta1, and beta2), where beta0 corresponds to BOM and (beta1, beta2) to AdamW. Could the authors provide additional ablation experiments to show how different values of beta0 affect performance while keeping (beta1, beta2) fixed? Furthermore, can they evaluate combinations of beta0 and beta1 to confirm whether BOM consistently yields performance gains across various settings of beta1 for different clasification tasks?
(3) Does the batch size affect the results discussed in question (1)?
(2) In this paper, the authors primarily focus on classification tasks. How well does the BOM method extend to regression tasks, or are there limitations in that setting?