- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that difficult temporal regions in sliding-window forecasting can provide useful current gradients while their accumulated Adam momentum harms later training. It introduces RAMP, which suppresses these regions’ gradients, and -short, which retains every training window but replaces the selected regions’ momentum history with their current gradient contribution. The central contribution is controlling optimizer memory according to the source of the gradients.
Critical strengths
- The paper proposes a well-motivated, technically coherent optimizer intervention. Distinguishing a sample’s immediate usefulness from its accumulated influence is a valuable perspective. -short implements this distinction cleanly through Adam’s additive first-moment decomposition, preserving data participation and the full-gradient second-moment computation.
- The experimental setup is generally broad and includes control studies.
Critical weaknesses
My main concern is about the method. For instance, the proposed memory replacement does not distinguish harmful temporal drift from useful stochastic averaging.
replaces the selected region’s accumulated first moment with its current minibatch gradient contribution. However, the discrepancy between these quantities reflects both parameter-induced drift and minibatch sampling noise, as Appendix E.4 explicitly acknowledges. The algorithm does not distinguish these sources of variation before removing the regional history. Consequently, it can discard useful averaging even when the underlying regional gradient is stationary.
Furthermore, applying different temporal filters to the two regions can disrupt beneficial cancellation between their gradients. A stable full gradient can arise from changing regional contributions that cancel each other; selectively removing one region’s history need not preserve that stability. Keeping the full-data second moment and matching the update norm does not prevent deterioration of the update direction. The proposed intervention therefore introduces a substantive tradeoff between reduced historical lag, increased sampling noise, and altered regional cancellation. The paper does not characterize when this tradeoff is favorable. This limits the justification for describing -short as selectively eliminating harmful historical influence while preserving useful learning signals. A stronger methodological account should identify conditions under which regional memory replacement improves the update despite these costs.
Questions for authors
See weaknesses. I'm happy to raise my score if it's resolved.