- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work investigates the role of depth in a diagonal linear network when training with gradient descent with momentum. The work characterizes the dependence of the early stopping window and time of occurrence on the depth, initialization and learning rate to guarantee sparse recovery. The analysis leverages a Gauss-Newton approximation to derive this. Furthermore, the work draws connections to incremental learning and kernel regime experimentally in toy recovery problems.
Critical strengths
Investigating early stopping is an important topic even in simple models. The work makes clear progress on this by integrating momentum and the interplay with depth. The Gauss-Newton approximation is well-found and an interesting approach to this problem.
Critical weaknesses
-
The analysis confounds initialization scale and learning rates with depth. The derived bounds are depending on the initialization scale and learning rate, however, to compare fairly in the experiments the same learning rate (the smaller one is fine) to ensure a fair comparison. Similarly, the effective update in the predictor scales with and depth . Therefore using the same across depths inherently leads to smaller updates and thus affects the early stopping window. For instance, Fig. 3 should use the same initialization scale for each considered depth.
-
The exposition can be significantly improved. The current theoretical results involve quite a lot of mathematical variables which makes it hard to parse the intended takeaway. For example, a condition on the initialization in Equation 6 is written out over two lines without direct guidance. This is persistent trough the theoretical section. Moreover, this makes it hard to compare similar expressions as it looks like that Init() at lines 209-215 is exactly Equation 6.
Questions for authors
Can the authors discuss the relationship between the manuscript and related work [1] as it directly involves flow analysis of momentum in diagonal linear networks?
[1] Papazov, Hristo et al. “Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks.” International Conference on Artificial Intelligence and Statistics (2024).
2 comments
I thank the authors for their response. The experiment resolves my first concern, however the current manuscript remains rather dense which is also in line with the responses of the other reviewers. I will maintain my score.
Thank you for pointing us to Papazov et al. (2024). Both papers study how heavy-ball momentum influences sparsity in diagonal linear networks. Their analysis concerns the solution selected at convergence, while ours concerns the iterations at which early stopping permits sparse recovery.
We will add this comparison to the related-work section and clarify that our contribution is a finite-time recovery guarantee for the modified iteration in noisy sparse regression.
Furthermore, we fix the initialization to a constant α=0.0005 for all cases, instead of using the depth-dependent scheme αN=αN. The experimental results demonstrate that the trend of the early stopping window persists (see Figure 14 in Appendix M).