acceptodds
Under review as a conference paper at ICLR 2027

Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks

Abstract

A pervasive structural pattern in modern deep learning is the linear factorization block: a submodule of the form in which two parameter matrices are multiplied directly, with no intervening nonlinearity. Such blocks appear in LoRA adapters, low-rank compressed layers, query-key products of self-attention, and share a common pathology: the factorization is non-unique, which can destabilize training and limit usable learning rates. Despite this, factorization blocks are typically optimized with standard Euclidean methods that ignore the underlying geometry. We introduce Stiefel-AdamW, a near drop-in replacement for AdamW for use wherever such blocks appear. By constraining one factor on the Stiefel manifold while leaving the other Euclidean, Stiefel-AdamW relaxes the full gauge symmetry to a compact orthogonal symmetry, ruling out factor blow-up while retaining the coordinate-wise diagonal preconditioning that gives AdamW its practical strength. Moment estimation is performed in the ambient Euclidean space, with geometry entering only through a tangent-space projection and a manifold retraction. The implementation overhead over AdamW is minimal, and we show that the resulting optimizer inherits both the stability benefits of Riemannian methods and standard convergence guarantees. We validate Stiefel-AdamW on LoRA-style fine-tuning of GPT2, ViT, and Mistral 7B and on full pretraining of GPT2 on OpenWebText and Qwen2 on Wikitext-103, showing consistent improvements over strong baselines at essentially no additional cost over AdamW.

Primary area:
optimization
Keywords:
Riemannian optimization, Adam, low-rank

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2KA7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Stiefel-AdamW, a replacement of AdamW designed for networks with factorization blocks W=ABW=AB, where the two matrices are directly multiplied without any nonlinearity between them. Stiefel-AdamW relaxes the symmetry to orthogonal invariance by constraining one factor to the Stiefel manifold while leaving B unconstrained. It leverages the compactness of manifold constraints to prevent factor value explosions and stabilize training, while enabling all adaptive moment estimates. Empirical validation across LoRA fine-tuning (GPT2, ViT, Mistral 7B) and full LLM pretraining (GPT2 on OpenWebText) demonstrates the performance.

Critical strengths
  • Low overhead: The momentum estimation of the algorithm is consistent with AdamW, and geometric operations are only involved during updates, resulting in costs and GPU memory usage that are on par with AdamW. In GPT2 pre-training experiments, the peak GPU memory usage was identical (13.8 GB).
  • Performance improvements are consistent: Across LoRA fine-tuning of GPT2, ViT, and Mistral 7B, as well as full pretraining of GPT2, Stiefel-AdamW outperforms baselines, though the advantages are modest in magnitude.
  • Retraction features a modular design: The retraction mechanism is designed as a replaceable component, allowing the use of methods such as Cayley transform, QR, polar decomposition, Newton-Schulz. Experiments demonstrate that the choice among these methods has minimal impact on final performance, thereby lowering the threshold for practical deployment.
Critical weaknesses

Performance

  • For the performance, on one hand, the performance improvement of this method is relatively small, and its advantages are not significant in some tasks. For example, the performance difference of fine-tuning between Stiefel-AdamW and Scaled AdamW in Table 1 is very small. On the other hand, most of experiments are fine-tuning. The largest scale experiment, Mistral 7B, was fine-tuning rather than full parameter training. Moreover, the full amount of pre training only had small models such as GPT2 and Qwen2 (100M), and GPT2 only ran 7000 steps. The scale and training length are also insufficient.
  • There have been similar works on using Stiefel manifolds for LoRA optimization, such as Park J's proposal to apply Stiefel orthogonal constraints on the B matrix of LoRA to improve parameter efficiency. However, i think the uniqueness of Stiefel-AdamW lies in its specific relaxation strategy, constraining only one factor, and emphasis on fully preserving the diagonal preconditions of AdamW. Therefore, there are some concerns as follows. Regarding the baseline, it is appreciated to compare with the same period of work that applied Stiefel constraints on LoRA . Furthermore, why only constrain A and not constrain B. It is recommended to dissolve the comparison between "only constrain A and not constrain B" and "only constrain B and not constrain A" or "both constrain" or "neither constrain".

[1] Park J, Kang M, Lee S, et al. Riemannian Optimization for LoRA on the Stiefel Manifold[C]//EMNLP (Findings). 2025: 20971-20985.

  • Theoretical analysis is based on convex regret analysis, while experiments are all non convex deep learning tasks. The explanatory power of theory on experiments is limited. Can conclusions be transferred or guided, and is there a gap between assumptions and experiments.
  • The stepsize stability experiment (Figure 1 right) only performed a learning rate scan of ViT-Base fine-tuning on CIFAR-10, without operating it on LLM fine-tuning or pretraining, which are the main application scenarios. Also, why do the curves of Cayley Adam and Stiefel Adam coincide?
  • In the experiment of GPT2 LoRA fine tuning on E2E. Why does Stiefel-AdamW use (0.98,0.98) while AdamW uses (0.9, 0.999), Scaled AdamW uses (0.7, 0.8), and GeoLoRA uses (0.98,0.98) for hyperparameter settings of beta1,beta2beta_1, beta_2? (Table 4). Does the use of significantly different hyperparameters in different methods mean that some of the performance differences be due to tuning.

Inconsistencies

There are multiple instances in the article where the preceding statements do not match the content of the main text, resulting in inconsistencies. It makes readers feel like authors have not read the main text carefully, or like AI generated content. Specifically, for example:

  • The abstract states that full pretraining was performed on Qwen2, but in Experiment 5.3, it is mentioned that there are present numerical results for Qwen2 LoRA pretraining.

  • Contribution 6 in the Introduction states, “consistently improving stability and final performance.”, But in Experiment 5.2, it is stated that the GPT2 pretraining results `produce results comparable to standard AdamW'.

  • The introduction states that state space models (SSMs) are listed as applicable scenarios, but there is neither citation nor experimentation.

  • The introduction states that the proposed method can inherits the principal benefits of geometric optimization,i.e., parameter invariance to orthogonal reparametrization. However, in conclusion, it states that one limitation of the current method is that it still retains a set of orthogonal invariants, which stems from the entropy nature of the Adam algorithm.

  • Regarding the GPT pretraining section, Section 5.2 states that both AdamW and Stiefel AdamW were trained for 7000 iterations, but the caption in Table 3 states that, for 6000 iterations, which is a contradiction.

  • The title of Lemma A.3 is Second order accuracy, but the actual content is X+αξ+O(α2)+O(ρs)X + \alpha\xi + \mathcal{O}(\alpha^2) + \mathcal{O}(\rho^s). Is it the same thing as the first order retrieval condition in line 893?

  • Are the three parts 864-866,888-891,892-896 trying to express the same meaning? In line 891, is it necessary to add reference after "see, e.g., standard analyses of imperfect retraction schemes".

  • Is Theorem A.2 proving linear convergence? Why is it mentioned in the text as convergences quadratically in line 917?

  • The title of Figure 1 is Speed vs Accuracy Trade off, but the y-axis of this graph represents the training loss rather than accuracy.

Questions for authors
  • Would it be possible to consider whether the current gains might hold at larger model sizes, or that the fine-tuning setting may transfer to pretraining. Since the scale and training length are insufficient to prove the performance improvement, would full-parameter pretraining on a larger model, or longer training runs, be feasible additions?
  • Regarding the baseline, it is appreciated to compare with the same period of work that applied Stiefel constraints on LoRA . Furthermore, why only constrain A and not constrain B?
  • Can theoretical analysis be transferred or guided on real experiments, and is there a gap between assumptions and experiments?
  • Would it be possible to operate learning rate sweeps on LLM fine-tuning or pretraining? Why do the curves of Cayley Adam and Stiefel Adam coincide?
  • Why does Stiefel-AdamW use (0.98,0.98) while AdamW uses (0.9, 0.999), Scaled AdamW uses (0.7, 0.8), and GeoLoRA uses (0.98,0.98) for hyperparameter settings of beta1,beta2beta_1, beta_2? Does the use of significantly different hyperparameters in different methods mean that some of the performance differences be due to tuning?
  • Please explain the inconsistencies of this paper mentioned above.
2 comments

We thank the reviewer the low memory overhead, modular retractions, and performance across several settings. Replies 1-5 address the weaknesses and Questions 1-5; the corrections address Question 6.

Critical Weaknesses and Questions For Authors

  1. We acknowledge that the gains are modest in some settings. The current pretraining horizon leaves open whether the observed behavior persists with longer training or larger models. A longer full-parameter GPT2 comparison would address training duration, but we lack the resources for these additional experiments during the rebuttal.
  2. Park et al. [1] use ambient Adam moments, tangent projection and QR retraction on BB; we will acknowledge this overlap and clarify our broader factorization-block formulation and analysis. Constraining AA allows B0=0B_0=0 with row-orthonormal A0A_0, leaving BB free to learn scale. Constraining BB permits the analogous initialization through A0=0A_0=0; we neither establish nor claim that constraining AA is better. Leaving both unconstrained gives standard factorization; constraining both without a free core forces the nonzero singular values of BABA to one. A free core preserves expressivity, as in PoLAR [2]. These comparisons are relevant but infeasible during rebuttal. See also our reply to cHpJ's Questions 1, 2 and 4.
  3. Thm. 4.2 is conditional on Ass. 4.1: convexity, boundedness, decaying momentum, AMSGrad and alignment, with decreasing stepsizes and no weight decay. It does not establish convergence for the nonconvex networks. Sec. 4.2 motivates stability independently of convexity: constraining one factor removes noncompact rescaling freedom and prevents unbounded factor growth at a bounded product. We will clarify this gap.
  4. Fig. 1 only tests learning-rate sensitivity on ViT/CIFAR-10; LLM sweeps are unreported and cannot be added during the rebuttal. Cayley Adam and Stiefel-AdamW share Stiefel constraints but use scalar and diagonal preconditioning, respectively. The two curves in fact only look identical because of scale, but the raw data confirms they are not equal indeed.
LRStiefel-AdamWCayley Adam
0.010.3832525659580620.3833367514367006
0.0080.31300456000834090.3121205914993675
0.0050.10392688792579020.1028324960141765
0.0030.04327298164823830.045130223644023
0.0010.01162391690993490.0126363391208709

5.For GPT2/E2E, AdamW's (0.9,0.999) and Scaled AdamW's (0.7,0.8) follow Zhang and Pilanci [3], App. E.1.1, Tab. 3. Our (0.98,0.98) setting (Tab. 4) is a separate choice; we will clarify its selection. As we discussed in subsection 5.1, GPT paragraph, we observe consistently with [Zhang & Pilanci, 2024] that a more aggressive moving average seems to be more effective for Riemannian based methods. The methods is stable but we observe performance to be consistently better with a more aggressive choice of (β1,β2)(\beta_1,\beta_2). We will further clarify in the revised version.

Inconsistencies

We would first like to point out that the scientific content of this paper is entirely the authors’ own work. AI was used only to improve the English in a few passages. We take full responsibility for the manuscript and the accuracy of this statement. We consider the suggestion that our scientific work is AI-generated a serious allegation, which we do not take lightly. We respectfully ask the reviewer to clarify the specific basis for this concern so that we can address it directly.

  1. We will delete "full" in the abstract.
  2. We will replace the claim in Contribution 6 with task-specific statements. In particular, Sec. 5.2 reports GPT2 pretraining performance comparable to AdamW. We will therefore describe the results as competitive performance across the tested settings, with improvements in several cases, rather than claim consistent improvements in stability and final performance.
  3. This is a fair point and we will emphasize the SSM setting in more details. A concrete example is Mamba's time-step projection, Δt=softplus⁡(BAut+b)\Delta_t=\operatorname{softplus}(B A u_t+b), where utu_t denotes the input features to that projection. See for example [4] or the original Mamba paper [5] (sec 3.6).
  4. We will distinguish representation symmetry, BA=(BQ)(Q⊤A)BA=(BQ)(Q^\top A) for Q∈O(r)Q\in O(r), from optimizer equivariance. Entrywise Adam generally depends on the basis. We will correct the introduction and conclusion: the residual symmetry belongs to the factorization; basis dependence comes from the adaptive update.
  5. This is a typo, the correct is 7000 iterations.
  6. We will rename Lemma A.3 "First-order consistency and approximation error" and retain ∣∣Ys−Y⋆∣∣≤Cρsα2||Y_s-Y^\star||\le C\rho^s\alpha^2. The finite update approximates the exact retraction: first-order consistency does not imply exact manifold feasibility.
  7. We will consolidate the repeated passages in App. A.2.2 and remove the vague reference at line 891 and the associated general convergence-preservation claim.
  8. Thm. A.2 establishes linear (geometric) convergence, ∣∣Ys−Y⋆∣∣≤ρs∣∣Y0−Y⋆∣∣||Y_s-Y^\star||\le\rho^s||Y_0-Y^\star|| for ρ<1\rho<1. We will correct "quadratically" at line 917 and distinguish this rate from approximation order in the stepsize.
  9. We will change the left-panel title of Fig. 1 from "Speed vs Accuracy Trade-off" to "Speed vs Training Loss".

[1] Park et al. 2025, Stiefel-LoRA

[2] Lion et al. 2025, PoLAR

[3] Zhang and Pilanci 2024, Riemannian Preconditioned LoRA

[4] Tri Dao and Albert Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, ArXiv 2024.

[5] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2024

Reviewer_cHpJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new Riemannian inspired optimizer (“Stiefel-AdamW") for linear factorization blocks of the form W=BAW=BA. It constrains the rows of AA to be orthonormal and BB is unconstrained. For AA, the Adam moments are computed in the ambient Euclidean space, and after which the update is projected onto the tangent space and retracted to the Stiefel manifold. The paper's primary motivation is that this constraint allows you to reduce the symmetry from the full, non-compact GL(Rr)GL(\mathbb{R}^r) to leaving only rotational/reflectional symmetry O(r)O(r). They theoretically show this prevents either factor from exploding and proves boundedness of gradients allowing larger learning rates. Experiments cover LoRA fine-tuning for language and vision models, as well as GPT-2 and Qwen2 pretraining.

Critical strengths
  • The geometric motivation is sound and theoretically justified. Relating the instability in factorized layers to their non-compact scaling symmetry explains why the factors may become unbalanced even though their product remains constant and bounded. This is an important contribution as it motivates the need for the Stiefel constraint rather than just an empirical heuristic.
  • The method is simple and easy to replace AdamW where required especially for the LoRA adapters and query-key factorizations.
Critical weaknesses
  • My primary weakness of this work is regarding the novelty aspect. In [1] they propose an optimizer “Stiefel-LoRA” which discusses constraining one LoRA factor to the Stiefel manifold and the other factor Euclidean. They similarly compute Adam moments in the ambient space, project the update onto the tangent space and apply a retraction. In [2], they also use Stiefel-constrained factors for low-rank adapters, so I’m curious how Stiefel-AdamW compare to these methods. This is my main weakness for this work as I’m unable to precisely identify the algorithmic novelty and practical advantages over these methods as comparison in the experiments is lacking.

[1] Riemannian Optimization for LoRA on the Stiefel Manifold. Park et al. 2025.

[2] PoLAR: Polar-Decomposed Low-Rank Adapter Representation. Lion et al. 2025.

Questions for authors
  • More concretely as already described in the weaknesses, can the authors explain in more detail how Stiefel-AdamW differs from Stiefel-LoRA [1], and maybe even formally at the algorithmic level?
  • Can the authors directly compare with Stiefel-LoRA and/or PoLAR experimentally, or explain why this is not an appropriate comparison? This comparison would help show that when constraining only one factor is better than constraining both factors.
  • Also can the authors explain why Muon was not added as a baseline for pretraining? Maybe a useful controlled comparison would be to first try Muon on all parameters and then Muon only on the query and key matrices with AdamW for the remaining parameters. If already tried, I would be curious to see how it performs and where it fails. Because this would help isolate the Stiefel-AdamW effect, as explicitly controlling the factorization geometry helps more than simply using a different optimizer on those same matrices.
  • So Stiefel-AdamW puts the Stiefel constraint on AA while leaving BB unconstrained, whereas Stiefel-LoRA does the opposite. Can the authors justify their choice as to why they constraint AA and are there any benefits they can comment on?

I would be inclined to increase my score if the questions are adequately addressed.

2 comments

We thank the reviewer for the comments. Unfortunately, we did not have the means to run additional experiments. We nonetheless provide our perspective in the responses below:

Weaknesses

  • Regarding the novelty aspect: We thank the reviewer for highlighting the closely related works [1] and [2]. Regarding Stiefel-LoRA, we apologize for overlooking the work by Park et al. [1] and thank the reviewer for bringing it to our attention. We will cite it and explicitly discuss its relationship to Stiefel-AdamW. We acknowledge that our algorithm also uses ambient Adam moments, tangent projection, and retraction. However, Stiefel-LoRA constrains BB and uses QR retraction; we constrain AA and support different retractions, with Cayley as the default. Our main contribution is a unified formulation and analysis for general direct factorization blocks W=BAW=BA, relating their noncompact scaling symmetry to instability, with applications to adapter fine-tuning and attention blocks during pretraining. In contrast, PoLAR [2] clearly differs from our method, since it uses two Stiefel direction factors and a free core, ΔW=XΘY⊤\Delta W=X\Theta Y^\top; see eq. (2) in [2]. Our method retains the factorization ΔW=BA\Delta W=BA, constraining only AA and leaving BB unconstrained. Moreover, PoLAR's practical optimizer uses a landing field, whereas ours uses tangent projection and retraction. Thus, PoLAR and Stiefel-AdamW differ in both parameterization and update mechanism: ours requires geometric operations on only one factor and introduces no additional core matrix. We agree that a direct empirical comparison is relevant; the submitted experiments do not establish performance or runtime superiority over PoLAR. Please also cf. the answer to Reviewer cHpJ.

Questions For Authors

  1. For the algorithmic comparison with Stiefel-LoRA, both methods share the constrained update template Dt=MtVt+εD_t=\frac{M_t}{\sqrt{V_t}+\varepsilon}, Xt+1=RXt!(−ηtPXt(Dt))X_{t+1}=\mathrm{R}_{X_t}\\!\left(-\eta_t \mathrm{P}_{X_t}(D_t)\right), where Mt,VtM_t,V_t are ambient Adam moments, and division is elementwise. Stiefel-LoRA applies this update to X=BX=B using QR retraction. Our method applies it to X=A⊤X=A^\top, supports different retractions with Cayley as the default, and updates the unconstrained factor BB using AdamW. We will explicitly acknowledge this shared update template and clarify that our broader contribution concerns the formulation and analysis of general factorization blocks, including applications beyond adapters. See above for the differences between Stiefel-LoRA and Stiefel-Adam.
  2. We agree that direct comparisons with Stiefel-LoRA and PoLAR would strengthen our empirical evaluation. These comparisons are not included in the submission, and time constraints prevent us from adding them during the rebuttal. We plan to include them in a future revision; the current results do not establish superiority over either method.
  3. We agree that adding Muon as a baseline would be informative. Our GPT2 experiment compares AdamW and Stiefel-AdamW using identical hyperparameters, retaining AdamW outside the factorization blocks (Sec. 5.2). A fair comparison with Muon would require additional tuning and pretraining runs, which we cannot complete within the rebuttal period. As our current submission reports no Muon results, we cannot draw any conclusions about its relative performance or failures.
  4. We thank the Reviewer for this question. From the perspective of representability, the choice is completely arbitrary, since im⁡(Φ∣St(n,r)×Rm×r)=im⁡(Φ∣Rn×r×St(m,r))=W∈Rn×m∣rank⁡(W)≤r.\operatorname{im}(\Phi|_{St(n,r) \times \mathbb{R}^{m \times r}}) = \operatorname{im}(\Phi|_{\mathbb{R}^{n \times r} \times St(m,r)}) = \\{ W \in \mathbb{R}^{n \times m}|\operatorname{rank}(W)\leq r \\}. Therefore, constraining either factor represents exactly the same family of low-rank matrices. The two formulations may, however, induce different optimization dynamics. Our presentation chooses to constrain AA (and leave BB unconstrained) primarily as a convention. In addition, recent empirical work [3] suggests that allocating the unconstrained factor to BB can be advantageous during fine-tuning, although this observation is not required by our theory and is not fundamental to the proposed method. We will clarify this point in the revised manuscript.

[1] Park et al. Riemannian Optimization for LoRA on the Stiefel Manifold. Findings of EMNLP, 2025.

[2] Lion et al. PoLAR: Polar-Decomposed Low-Rank Adapter Representation. NeurIPS, 2025.

[3] Zhu et al. Asymmetry in Low-Rank Adapters of Foundation Models. ICML, 2024.

Thank you to the authors for their responses and helpful clarifications. While I appreciate their efforts, the rebuttal does not include the requested comparisons with closely related methods such as Stiefel-LoRA and PoLAR, which I believe are important for clearly establishing the paper’s novelty; I will therefore maintain my original score, while encouraging the authors to strengthen these comparisons and distinctions in a revised manuscript.

Reviewer_3C4aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The manuscript is focused on blocks consisting of two consecutive linear transformations. They constrain the system to require one of the matrices to be full rank, i.e., lie of Stiefel manifold, and show how to adapt AdamW optimizer to this system.

Critical strengths

The matrix product block is ubiquitous, and the paper provides a rigorous theoretical analysis of the proposed method.

Critical weaknesses

The paper is of limited theoretical novelty, with some prior work on similar topics not cited, for example Adam-Stiefel optimizer (Kong et al., ICLR 2023).

The empirical results show that the improvement compared to other methods is rarely substantial.

Questions for authors

How does stability (Fig. 1) of Stiefel-AdamW compare to other methods on other tasks? It is analyzed only on ViT/CIFAR10, which is the only task in which Stiefel-AdamW shows consistent improvement over baselines.

2 comments

We thank the reviewer for recognizing the relevance of factorization blocks and the theoretical motivation. We respond in order:

Weaknesses

  • Thank you for pointing out the work by Kong et al. (ICLR 2023); we will make sure to cite this work and clarify the distinction of our algorithm with respect to their work. Kong et al.'s Adam/Stiefel optimizer uses a structure-preserving discretization based on operator splitting; our method retains standard Adam moment estimation in the ambient space and applies tangent projection and retraction to the resulting update.
  • We agree that the improvements compared to other methods are not uniformly substantial. What we wanted to show with our experiments is that the numerical results for Stiefel-AdamW support a simple method with competitive performance with the state-of-the-art, and improvements in several settings, rather than a consistently large advantage.

Questions For Authors This raises a fair point. Fig. 1 evaluates learning-rate sensitivity only on ViT/CIFAR-10, measured by the best training loss attained at each tested learning rate. We do not report analogous comparisons on other tasks, as unfortunately we lack the resources to add them at the moment. Although the geometric motivation applies to factorization blocks across tasks, similar learning-rate robustness elsewhere remains to be verified. We will clarify that our empirical stability claim is limited to the tested setting.

Thank you for the clarifications. I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.