- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates generalized smoothness from a Riemannian optimization perspective. The authors introduce two families of Riemannian metrics, conformal metrics and warped product metrics, to establish global Riemannian smoothness for objective functions that may not satisfy classical Euclidean smoothness conditions. They further derive sufficient conditions for Riemannian smoothness, analyze their applicability to neural network training, and establish convergence guarantees for Riemannian SGD and Adagrad-Norm. Additionally, the paper develops low-dimensional formulations for computing geodesics efficiently. Experiments on regression and image classification tasks demonstrate the potential benefits of the proposed metrics.
Critical strengths
1. The paper presents an interesting research perspective with theoretical value.
Classical Euclidean smoothness assumptions are difficult to satisfy for many non-convex neural network optimization problems. Rather than further relaxing smoothness assumptions in Euclidean space, this paper explores the geometry of the parameter space and constructs Riemannian metrics to establish new smoothness conditions. This perspective is conceptually interesting and provides a promising direction for further research.
2. The paper provides a relatively comprehensive theoretical analysis that could serve as a foundation for future research.
Beyond introducing two families of Riemannian metrics, the authors derive sufficient conditions for Riemannian smoothness, investigate their applicability to neural network optimization, establish convergence guarantees for the corresponding optimization methods, and develop efficient approaches for computing geodesics. Overall, these contributions form a relatively complete theoretical framework.
Critical weaknesses
1. The practical benefits of the proposed Riemannian metrics remain unclear.
Although the authors demonstrate that the proposed metrics can establish Riemannian smoothness for certain objective functions that do not satisfy classical Euclidean smoothness conditions, and provide corresponding convergence guarantees, the paper does not sufficiently explain the concrete benefits of these geometric designs for practical optimization.
For instance, it remains unclear if the proposed metrics primarily improve optimization stability, accelerate convergence, or lead to higher-quality final solutions. While the paper provides some experimental results, these aspects have not been systematically investigated.
Furthermore, the convergence guarantees are mainly established in terms of the Riemannian gradient norm, without sufficiently clarifying their implications for the solution quality of the original Euclidean optimization problem. Therefore, if the proposed geometric framework only establishes new smoothness conditions and convergence guarantees without demonstrating its practical advantages in solving the original optimization problem, the significance of these theoretical results remains limited.
2. The presentation and articulation of the contributions could be further improved.
The current abstract and introduction are relatively brief and mainly focus on describing the proposed metrics and their theoretical properties, without sufficiently highlighting their practical advantages or the additional value they provide compared with existing approaches.
Considering that the ICLR audience includes researchers from diverse backgrounds beyond optimization theory, the paper would benefit from more intuitive explanations of the research motivation, the key advantages of the proposed approach, and the practical implications of the theoretical results for machine learning optimization. The current presentation makes it difficult for readers to immediately understand the significance of the main contributions.
Questions for authors
1. Applicability and limitations of the proposed metrics
The paper mainly establishes Riemannian smoothness conditions and convergence guarantees for SGD and Adagrad-Norm based on conformal and warped product metrics. However, the extent to which these metrics and their theoretical guarantees can be extended to other optimization algorithms, different classes of Riemannian manifolds, and more general objective functions remains unclear. Could the authors further clarify the applicability, necessary conditions, and main limitations of the proposed metrics, and discuss their potential extensions to more general optimization settings?
2. Sufficiency of experimental validation
The current experiments mainly focus on regression tasks and image classification using shallow MLP and CNN models, resulting in a relatively limited evaluation scale. Although the authors provide computational overhead analysis for models with larger parameter sizes, corresponding evaluations of optimization performance are still lacking. It would be valuable to conduct additional experiments on deeper or larger-scale neural networks to further validate the effectiveness and scalability of the proposed metrics in terms of optimization stability, convergence speed, and final model performance.
3. Advantages over existing optimization strategies
From an algorithmic perspective, the paper mainly employs existing Riemannian SGD and Adagrad-Norm methods, with its primary contribution lying in the design of Riemannian metrics. However, the gradient scaling mechanisms introduced by the conformal and warped product metrics share certain similarities with existing optimization strategies, such as adaptive learning rates and gradient clipping. Could the authors further clarify the distinctive advantages of the proposed metrics over these existing strategies? Additional comparisons and analyses would help demonstrate if the geometric designs provide practical optimization benefits beyond establishing new smoothness conditions and convergence guarantees.
3 comments
Reponse to reviewer vSxH
Comment: Applicability and limitations of the proposed metrics. The paper mainly establishes Riemannian smoothness conditions and convergence guarantees for SGD and Adagrad-Norm based on conformal and warped product metrics. However, the extent to which these metrics and their theoretical guarantees can be extended to other optimization algorithms, different classes of Riemannian manifolds, and more general objective functions remains unclear. Could the authors further clarify the applicability, necessary conditions, and main limitations of the proposed metrics, and discuss their potential extensions to more general optimization settings?
Response: We agree that the paper could benefit from an extended discussion of potential limitations and extensions, and have added a new paragraph in the concluding section to address this. We clarify that applicability includes stochastic optimization settings where derivative estimates are available and classical smoothness is violated. Roughly speaking, our approach is most easily applied when a function’s Hessian norm is bounded by a polynomial in the parameter. In this case, the geodesic calculation simplifies significantly. Necessary conditions include the availability of unbiased stochastic gradients at each stage. These gradients should be either uniformly bounded in the Riemannian metrics (for Adagrad) or have uniformly bounded variance (SGD).
Comment: Sufficiency of experimental validation. The current experiments mainly focus on regression tasks and image classification using shallow MLP and CNN models, resulting in a relatively limited evaluation scale. Although the authors provide computational overhead analysis for models with larger parameter sizes, corresponding evaluations of optimization performance are still lacking. It would be valuable to conduct additional experiments on deeper or larger-scale neural networks to further validate the effectiveness and scalability of the proposed metrics in terms of optimization stability, convergence speed, and final model performance.
Response: In order to explore the performance on a diverse set of tasks and model sizes, we carried out a set of experiments on a new task and model. This task focuses on text classification using the AG News dataset, and we employ a classifier with multi-head self-attention. This model is not only larger in terms of parameter count (it has 741K parameters in its larger configuration) than our existing experiments, but it has a more complex structure, incorporating attention matrices in addition to a shared value matrix and a final linear classifier. We include the details in Section F of the appendix. Overall, the experiments on this dataset further confirm that the Riemannian approach is more stable across varying initialization scales, compared to the Euclidean approach.
Comment: Advantages over existing optimization strategies. From an algorithmic perspective, the paper mainly employs existing Riemannian SGD and Adagrad-Norm methods, with its primary contribution lying in the design of Riemannian metrics. However, the gradient scaling mechanisms introduced by the conformal and warped product metrics share certain similarities with existing optimization strategies, such as adaptive learning rates and gradient clipping. Could the authors further clarify the distinctive advantages of the proposed metrics over these existing strategies? Additional comparisons and analyses would help demonstrate if the geometric designs provide practical optimization benefits beyond establishing new smoothness conditions and convergence guarantees.
Response: We thank the reviewer for the suggestion and we have significantly expanded the discussion at the end of our paper, including how our approach relates to other forms of adaptive learning rates. In short, while we focused on a simple form of adaptivity (adagrad-norm), our approach is potentially compatible with other adaptation mechanisms, such as per-parameter adaptation as used in Adam, and the full version of adagrad. We believe the resulting algorithms would need certain other features to be implemented such as a parallel transport, and this could be a fruitful direction for future work. Furthermore, we’ve highlighted in the introduction that, overall, we believe our approach is a way to encode problem information into geometry in order to alleviate some of the challenges with traditional Euclidean approaches to analysis.
Thank you for the authors' response. However, I still have concerns about the clarity and overall quality of the paper's writing. Therefore, I will maintain my original rating.
Response to reviewer vSxH
We thank the reviewer for their thoughtful analysis. We agree that the presentation of contributions could be improved, and that it is important to clarify the benefits of the Riemannian approach. In response to this, we’ve made the following changes and carried out further experiments.
The reviewer notes that convergence guarantees are focused on the Riemannian gradient norm without explaining the implications for the original Euclidean criteria. This echoes similar points raised by the other reviewers. To add insight to this matter, we have included a new result that applies in the deterministic regime of Adagrad-Norm, i.e., when using exact gradients. In this scenario, we can prove that 1) the Riemannian gradient norms tend to zero and 2) any accumulation point of the Riemannian algorithm is also a standard Euclidean stationary point. We believe that these observations shed some light on the question the reviewer is asking. This discussion is in Section 5.
The reviewer also noted that the intro could benefit from a higher level discussion aimed at a broader ICLR audience; we agree and are grateful for this suggestion. We have reframed the abstract with two new sentences that we believe place the work within the broader optimization for machine learning context and expand the accessibility of our paper. Furthermore, we have also revised the introduction to this effect. In brief, our motivation stems from the large amount of time spent on hyperparameter tuning for models, and how the current theory for SGD does not enable one to connect problem structure to provable hyperparameters like batch size, step size, and number of epochs. We hope that this approach can potentially provide insight into these issues.
We respond to the reviewer’s questions point-by-point below in a separate comment.