acceptodds
Under review as a conference paper at ICLR 2027

A Riemannian perspective on generalized smoothness

Abstract

In this work we develop Riemannian optimization formulations for optimization problems posed on Euclidean spaces, and that may not satisfy classical smoothness criteria. We introduce a family of metrics that can confirm global Riemannian smoothness in function classes including certain neural-network training problems, enabling convergence guarantees for Riemannian optimization schemes applied to these objectives. The metrics are conformal scalings and warped products defined using radial scaling functions, and we further derive low-dimensional geodesic equations for these metrics, which can lead to efficient implementations. Finally, we consider connections with -smoothness and -smoothness criteria.

Primary area:
optimization
Keywords:
optimization, riemannian optimization, stochastic optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vSxHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates generalized smoothness from a Riemannian optimization perspective. The authors introduce two families of Riemannian metrics, conformal metrics and warped product metrics, to establish global Riemannian smoothness for objective functions that may not satisfy classical Euclidean smoothness conditions. They further derive sufficient conditions for Riemannian smoothness, analyze their applicability to neural network training, and establish convergence guarantees for Riemannian SGD and Adagrad-Norm. Additionally, the paper develops low-dimensional formulations for computing geodesics efficiently. Experiments on regression and image classification tasks demonstrate the potential benefits of the proposed metrics.

Critical strengths

1. The paper presents an interesting research perspective with theoretical value.

Classical Euclidean smoothness assumptions are difficult to satisfy for many non-convex neural network optimization problems. Rather than further relaxing smoothness assumptions in Euclidean space, this paper explores the geometry of the parameter space and constructs Riemannian metrics to establish new smoothness conditions. This perspective is conceptually interesting and provides a promising direction for further research.

2. The paper provides a relatively comprehensive theoretical analysis that could serve as a foundation for future research.

Beyond introducing two families of Riemannian metrics, the authors derive sufficient conditions for Riemannian smoothness, investigate their applicability to neural network optimization, establish convergence guarantees for the corresponding optimization methods, and develop efficient approaches for computing geodesics. Overall, these contributions form a relatively complete theoretical framework.

Critical weaknesses

1. The practical benefits of the proposed Riemannian metrics remain unclear.

Although the authors demonstrate that the proposed metrics can establish Riemannian smoothness for certain objective functions that do not satisfy classical Euclidean smoothness conditions, and provide corresponding convergence guarantees, the paper does not sufficiently explain the concrete benefits of these geometric designs for practical optimization.

For instance, it remains unclear if the proposed metrics primarily improve optimization stability, accelerate convergence, or lead to higher-quality final solutions. While the paper provides some experimental results, these aspects have not been systematically investigated.

Furthermore, the convergence guarantees are mainly established in terms of the Riemannian gradient norm, without sufficiently clarifying their implications for the solution quality of the original Euclidean optimization problem. Therefore, if the proposed geometric framework only establishes new smoothness conditions and convergence guarantees without demonstrating its practical advantages in solving the original optimization problem, the significance of these theoretical results remains limited.

2. The presentation and articulation of the contributions could be further improved.

The current abstract and introduction are relatively brief and mainly focus on describing the proposed metrics and their theoretical properties, without sufficiently highlighting their practical advantages or the additional value they provide compared with existing approaches.

Considering that the ICLR audience includes researchers from diverse backgrounds beyond optimization theory, the paper would benefit from more intuitive explanations of the research motivation, the key advantages of the proposed approach, and the practical implications of the theoretical results for machine learning optimization. The current presentation makes it difficult for readers to immediately understand the significance of the main contributions.

Questions for authors

1. Applicability and limitations of the proposed metrics

The paper mainly establishes Riemannian smoothness conditions and convergence guarantees for SGD and Adagrad-Norm based on conformal and warped product metrics. However, the extent to which these metrics and their theoretical guarantees can be extended to other optimization algorithms, different classes of Riemannian manifolds, and more general objective functions remains unclear. Could the authors further clarify the applicability, necessary conditions, and main limitations of the proposed metrics, and discuss their potential extensions to more general optimization settings?

2. Sufficiency of experimental validation

The current experiments mainly focus on regression tasks and image classification using shallow MLP and CNN models, resulting in a relatively limited evaluation scale. Although the authors provide computational overhead analysis for models with larger parameter sizes, corresponding evaluations of optimization performance are still lacking. It would be valuable to conduct additional experiments on deeper or larger-scale neural networks to further validate the effectiveness and scalability of the proposed metrics in terms of optimization stability, convergence speed, and final model performance.

3. Advantages over existing optimization strategies

From an algorithmic perspective, the paper mainly employs existing Riemannian SGD and Adagrad-Norm methods, with its primary contribution lying in the design of Riemannian metrics. However, the gradient scaling mechanisms introduced by the conformal and warped product metrics share certain similarities with existing optimization strategies, such as adaptive learning rates and gradient clipping. Could the authors further clarify the distinctive advantages of the proposed metrics over these existing strategies? Additional comparisons and analyses would help demonstrate if the geometric designs provide practical optimization benefits beyond establishing new smoothness conditions and convergence guarantees.

3 comments
AuthorsOpenReview ↗

Response to reviewer vSxH

We thank the reviewer for their thoughtful analysis. We agree that the presentation of contributions could be improved, and that it is important to clarify the benefits of the Riemannian approach. In response to this, we’ve made the following changes and carried out further experiments.

The reviewer notes that convergence guarantees are focused on the Riemannian gradient norm without explaining the implications for the original Euclidean criteria. This echoes similar points raised by the other reviewers. To add insight to this matter, we have included a new result that applies in the deterministic regime of Adagrad-Norm, i.e., when using exact gradients. In this scenario, we can prove that 1) the Riemannian gradient norms tend to zero and 2) any accumulation point of the Riemannian algorithm is also a standard Euclidean stationary point. We believe that these observations shed some light on the question the reviewer is asking. This discussion is in Section 5.

The reviewer also noted that the intro could benefit from a higher level discussion aimed at a broader ICLR audience; we agree and are grateful for this suggestion. We have reframed the abstract with two new sentences that we believe place the work within the broader optimization for machine learning context and expand the accessibility of our paper. Furthermore, we have also revised the introduction to this effect. In brief, our motivation stems from the large amount of time spent on hyperparameter tuning for models, and how the current theory for SGD does not enable one to connect problem structure to provable hyperparameters like batch size, step size, and number of epochs. We hope that this approach can potentially provide insight into these issues.

We respond to the reviewer’s questions point-by-point below in a separate comment.

AuthorsOpenReview ↗

Reponse to reviewer vSxH

Comment: Applicability and limitations of the proposed metrics. The paper mainly establishes Riemannian smoothness conditions and convergence guarantees for SGD and Adagrad-Norm based on conformal and warped product metrics. However, the extent to which these metrics and their theoretical guarantees can be extended to other optimization algorithms, different classes of Riemannian manifolds, and more general objective functions remains unclear. Could the authors further clarify the applicability, necessary conditions, and main limitations of the proposed metrics, and discuss their potential extensions to more general optimization settings?

Response: We agree that the paper could benefit from an extended discussion of potential limitations and extensions, and have added a new paragraph in the concluding section to address this. We clarify that applicability includes stochastic optimization settings where derivative estimates are available and classical smoothness is violated. Roughly speaking, our approach is most easily applied when a function’s Hessian norm is bounded by a polynomial in the parameter. In this case, the geodesic calculation simplifies significantly. Necessary conditions include the availability of unbiased stochastic gradients at each stage. These gradients should be either uniformly bounded in the Riemannian metrics (for Adagrad) or have uniformly bounded variance (SGD).

Comment: Sufficiency of experimental validation. The current experiments mainly focus on regression tasks and image classification using shallow MLP and CNN models, resulting in a relatively limited evaluation scale. Although the authors provide computational overhead analysis for models with larger parameter sizes, corresponding evaluations of optimization performance are still lacking. It would be valuable to conduct additional experiments on deeper or larger-scale neural networks to further validate the effectiveness and scalability of the proposed metrics in terms of optimization stability, convergence speed, and final model performance.

Response: In order to explore the performance on a diverse set of tasks and model sizes, we carried out a set of experiments on a new task and model. This task focuses on text classification using the AG News dataset, and we employ a classifier with multi-head self-attention. This model is not only larger in terms of parameter count (it has 741K parameters in its larger configuration) than our existing experiments, but it has a more complex structure, incorporating attention matrices in addition to a shared value matrix and a final linear classifier. We include the details in Section F of the appendix. Overall, the experiments on this dataset further confirm that the Riemannian approach is more stable across varying initialization scales, compared to the Euclidean approach.

Comment: Advantages over existing optimization strategies. From an algorithmic perspective, the paper mainly employs existing Riemannian SGD and Adagrad-Norm methods, with its primary contribution lying in the design of Riemannian metrics. However, the gradient scaling mechanisms introduced by the conformal and warped product metrics share certain similarities with existing optimization strategies, such as adaptive learning rates and gradient clipping. Could the authors further clarify the distinctive advantages of the proposed metrics over these existing strategies? Additional comparisons and analyses would help demonstrate if the geometric designs provide practical optimization benefits beyond establishing new smoothness conditions and convergence guarantees.

Response: We thank the reviewer for the suggestion and we have significantly expanded the discussion at the end of our paper, including how our approach relates to other forms of adaptive learning rates. In short, while we focused on a simple form of adaptivity (adagrad-norm), our approach is potentially compatible with other adaptation mechanisms, such as per-parameter adaptation as used in Adam, and the full version of adagrad. We believe the resulting algorithms would need certain other features to be implemented such as a parallel transport, and this could be a fruitful direction for future work. Furthermore, we’ve highlighted in the introduction that, overall, we believe our approach is a way to encode problem information into geometry in order to alleviate some of the challenges with traditional Euclidean approaches to analysis.

Thank you for the authors' response. However, I still have concerns about the clarity and overall quality of the paper's writing. Therefore, I will maintain my original rating.

Reviewer_ZvP5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper main idea is that euclidean optimization problem uder a generalized smoothness condition can be made globally smooth under some riemannian metric.

Critical strengths

I like the originality of the idea and I find the proposed framework quite interesting. The construction of a Riemannian metric which is adapted to the growth of the function is quite elegant. I also like the fact that the theoritical results are quite general, in particular polynomial growing Hessians are covered by the theory.

Critical weaknesses

My main concerns are about the practicality of the proposed framework:

First I believe that some algorithm pseudocode should be included in the appendix. It would be very usefull to explain how the proposed metric and corresponding exponential map are computed in practice.

I think the author should also add a general discussion about the computational cost for constructing the metric and computing the exponential map. Also since the manifold is still R^n can we use simpler retraction ? (like addition)

I also wonder under which additional assumption can the riemannian stationarity can be translated into euclidean stationarity.

Questions for authors

See the weakness paragraph

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZvP5

The reviewer pointed out several interesting opportunities to improve the clarity of our paper including the relation between our approach and euclidean stationarity, and the importance of using the exponential retraction. We address the comments individually below.

Comment: First I believe that some algorithm pseudocode should be included in the appendix. It would be very useful to explain how the proposed metric and corresponding exponential map are computed in practice.

Response: We agree that this is a great idea and thank the reviewer for suggesting it. We now include pseudocode in the appendix (Algorithm 1) showing how the Riemannian methods are integrated with a standard model training loop, starting from euclidean gradients, computing the riemannian gradient, performing adaptive scaling, and finally through to the riemannian update step. This is presented for the conformal setting, the warped case being analogous.

Comment: I think the author should also add a general discussion about the computational cost for constructing the metric and computing the exponential map.

Response: We agree that this should be expanded. Now instead of deferring the topic to the appendix, we have a Computational Overhead discussion at the end of our Section 6 which specifically describes the cost of computing the Riemannian gradients and computing the exponential.

Comment: Also since the manifold is still Rn can we use simpler retraction ? (like addition)

Response: From a mechanical point of view, addition is an option, but it’s not clear if the resulting step would still satisfy the smoothness condition needed for convergence analysis. We feel this point is valuable as it allows for an intuitive way to justify the warped and conformal methods. We’ve added a sentence about this at the beginning of Section 6.

Comment: I also wonder under which additional assumption can the riemannian stationarity can be translated into euclidean stationarity.

Response: We agree that this is an important point to explore in the paper, as it was brought up by several reviewers. We now include in Section 5 a potentially interesting result where Riemannian convergence can be translated into Euclidean convergence. We show that, in the deterministic setting, Adagrad-Norm causes the Riemannian gradient norm to tend to zero, and furthermore, any accumulation point of the Riemannian algorithm is a Euclidean stationary point. We think this result can provide some clarity into the relation between Riemannian and Euclidean convergence.

I thank the authors for the clarification and the additions to the paper, I have no further concerns

Reviewer_96bCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a theoretical framework for optimization problems based on Riemannian geometry, extending beyond the classical Euclidean notion of smoothness. By employing conformal and warped-product metrics, the authors derive bounded Riemannian Hessian operators for specific classes of functions, including those encountered in neural network training. The paper establishes convergence guarantees for the stochastic AdaGrad-Norm algorithm and simplifies the computation of geodesics by reducing them to a low-dimensional system of ODEs. Experiments involving regression and image classification tasks clearly demonstrate the optimization characteristics of the proposed geometric structures.

Critical strengths
  1. The paper is well-written and generally easy to follow.
  2. The analysis of smoothness equivalence, supplemented by a counterexample involving neural networks, provides valuable theoretical insights.
  3. The reduction of the geodesic calculation ensures that the numerical integration process remains independent of the model's dimensionality.
  4. The concept of designing metrics specifically to achieve smoothness and convergence guarantees represents an intriguing approach.
Critical weaknesses
  1. The definition of a stationary point in Riemannian geometry does not coincide with that in Euclidean space, which limits the practical significance of the convergence guarantees.
  2. The convergence guarantees rely on the assumption that exponential maps are computed exactly; however, the paper fails to analyze the errors introduced by using the Runge-Kutta approximation method during practical implementation. 3.The neural-network guarantees require bounded smooth activations and do not directly cover standard ReLU networks.
Questions for authors

Please see the weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 96bC

The reviewer highlighted several interesting questions on the nature of the convergence guarantees, the impact of using approximate exponential calculation, and the scope of activation functions covered by our theory. We’ve made the following updates to address these issues.

Comment: The definition of a stationary point in Riemannian geometry does not coincide with that in Euclidean space, which limits the practical significance of the convergence guarantees.

Response: We now include an expanded discussion on the relation between Euclidean stationarity and the guarantees obtained in the Riemannian setting. Primarily we show that, in the deterministic setting, accumulation points of the algorithm are guaranteed to be ordinary Euclidean stationary points. We highlight this in Section 5, with a proof in the appendix.

Comment: The convergence guarantees rely on the assumption that exponential maps are computed exactly; however, the paper fails to analyze the errors introduced by using the Runge-Kutta approximation method during practical implementation

Response: Thank you for pointing this out. Indeed our current analysis assumes that the exponential map is computed exactly. We believe that an extended analysis which can take into account these errors would be an interesting avenue for future work, and we discuss the details in the Discussion and Conclusion section. Briefly, there are several criteria in the literature that can quantify the effect of using approximate retractions within convergence guarantees, including [1][2]. Secondly, these criteria could potentially be combined with existing analyses of integrator errors on Riemannian manifolds [3]. We expand on this discussion in Section 8.

  1. Riemannian Zeroth-Order Gradient Estimation with Structure Preserving Metrics for Geodesically Incomplete Manifolds. Ma and Huang. ICLR 2026.
  2. Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms. Li et al. ICML 2024.
  3. Numerical Accuracy of Ladder Schemes for Parallel Transport on Manifolds. Guigui and Pennec. Foundations of Computational Mathematics 2022.

Comment: The neural-network guarantees require bounded smooth activations and do not directly cover standard ReLU networks.

Response: The example we considered in the paper involved a network with bounded and smooth activations which we recognize determines only a limited subset of activation functions, excluding ReLU. We see two routes to generalizing to more activation functions. Part of the reason for using a restrictive activation function is to simplify the analysis needed to determine the degree of the polynomial scaling function. Strictly speaking, at least for the conformal case, we can certainly use activation functions whose values and derivatives are bounded by polynomials in their input; it simply requires more book keeping to identify the degree of the scaling function. This is now highlighted immediately after Assumption 5.1. Secondly, to handle the non-smooth structure of the ReLU function, we believe that smoothing approaches used for zeroth order optimization are particularly relevant. We highlight this aspect in our discussion in Section 8 and thank the reviewer for highlighting an important opportunity to discuss future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We thank the reviewers for their thoughtful and constructive comments. We are encouraged by the consensus of the reviewers that the idea of designing a metric to achieve a smoothness guarantee is a novel and intriguing approach that can potentially generate follow up research. Based on the feedback, we have revised the manuscript to include further discussion, theoretical analysis, and experimental results. We outline the major edits here and provide more details in our responses below.

  • Narrative and analysis

    • Abstract: We added key sentences at the beginning and end of the abstract on the higher level motivation and significance of our work.
    • 1 Introduction: We further clarify the practical implications of current theoretical limitations, and frame our work at a higher, more general level.
    • 5 Model smoothness: We discuss extending our method to other activation functions.
    • 5.1 Optimization analysis: We discuss when accumulation points of the Riemannian algorithm are Euclidean stationary points, providing a link between Riemannian and Euclidean guarantees.
    • 6 Geodesics: We provide motivation for this section by considering whether we can use the simple retraction of addition on Euclidean space. We also expand the discussion on computational overhead of Riemannian approaches.
    • 8 Conclusion: We have significantly expanded our conclusion with discussion on extensions, limitations, and the generality of this work. We discuss how other problems with tractable Hessian growth rate bounds are amenable to the techniques introduced, and we mention extensions to non-smooth activations, combination with other adaptation schemes, and future work on analyzing exponential integration errors.
    • Appendix: Added details to several proofs to make the paper easier to follow, as well as minor formatting changes for clarity. We also introduced Pseudocode to give an intuitive feel for the algorithm steps.
  • Further experiments

    • We carried out experiments for a new task. AGNews is an NLP text classification dataset, and we train a model to classify the news articles in that dataset into 4 categories using a multi-head self attention classifier, demonstrating our approach with a more complex model structure compared to the prior experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.