- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors develop a cross-curvature based theory which allows to compare Jacobi and Gauss–Seidel block-gradient updates, extending one-step loss comparisons to multiple iterations. This allows to estimate a priori which iterate is a better choice up to third order in the learning rate. The work is complemented with a small series of numerical experiments
Critical strengths
- The problem studied by the authors is of interest for the community and well-motivated.
- The multi-step theoretical analysis is supported by positive empirical results
Critical weaknesses
- The proposed analysis, while being informative from a theoretical perspective, are not clearly applicable as this would require computing gradient Hessian products from both observed trajectories, which can be fairly expensive in large-scale optimization problems.
- The theoretical analysis is strongly based on known theory, and the overall key idea relies on a Taylor expansion and a standard trajectory perturbation analysis.
- In the main paper there is no empirical validation of the result presented in Theorem 3.
Questions for authors
Do the authors believe this criterion can be used or adapted to a scalable criterion for online update selection in block coordinate methods? What would be the computational overhead?