- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Stiefel-AdamW, a replacement of AdamW designed for networks with factorization blocks , where the two matrices are directly multiplied without any nonlinearity between them. Stiefel-AdamW relaxes the symmetry to orthogonal invariance by constraining one factor to the Stiefel manifold while leaving B unconstrained. It leverages the compactness of manifold constraints to prevent factor value explosions and stabilize training, while enabling all adaptive moment estimates. Empirical validation across LoRA fine-tuning (GPT2, ViT, Mistral 7B) and full LLM pretraining (GPT2 on OpenWebText) demonstrates the performance.
Critical strengths
- Low overhead: The momentum estimation of the algorithm is consistent with AdamW, and geometric operations are only involved during updates, resulting in costs and GPU memory usage that are on par with AdamW. In GPT2 pre-training experiments, the peak GPU memory usage was identical (13.8 GB).
- Performance improvements are consistent: Across LoRA fine-tuning of GPT2, ViT, and Mistral 7B, as well as full pretraining of GPT2, Stiefel-AdamW outperforms baselines, though the advantages are modest in magnitude.
- Retraction features a modular design: The retraction mechanism is designed as a replaceable component, allowing the use of methods such as Cayley transform, QR, polar decomposition, Newton-Schulz. Experiments demonstrate that the choice among these methods has minimal impact on final performance, thereby lowering the threshold for practical deployment.
Critical weaknesses
Performance
- For the performance, on one hand, the performance improvement of this method is relatively small, and its advantages are not significant in some tasks. For example, the performance difference of fine-tuning between Stiefel-AdamW and Scaled AdamW in Table 1 is very small. On the other hand, most of experiments are fine-tuning. The largest scale experiment, Mistral 7B, was fine-tuning rather than full parameter training. Moreover, the full amount of pre training only had small models such as GPT2 and Qwen2 (100M), and GPT2 only ran 7000 steps. The scale and training length are also insufficient.
- There have been similar works on using Stiefel manifolds for LoRA optimization, such as Park J's proposal to apply Stiefel orthogonal constraints on the B matrix of LoRA to improve parameter efficiency. However, i think the uniqueness of Stiefel-AdamW lies in its specific relaxation strategy, constraining only one factor, and emphasis on fully preserving the diagonal preconditions of AdamW. Therefore, there are some concerns as follows. Regarding the baseline, it is appreciated to compare with the same period of work that applied Stiefel constraints on LoRA . Furthermore, why only constrain A and not constrain B. It is recommended to dissolve the comparison between "only constrain A and not constrain B" and "only constrain B and not constrain A" or "both constrain" or "neither constrain".
[1] Park J, Kang M, Lee S, et al. Riemannian Optimization for LoRA on the Stiefel Manifold[C]//EMNLP (Findings). 2025: 20971-20985.
- Theoretical analysis is based on convex regret analysis, while experiments are all non convex deep learning tasks. The explanatory power of theory on experiments is limited. Can conclusions be transferred or guided, and is there a gap between assumptions and experiments.
- The stepsize stability experiment (Figure 1 right) only performed a learning rate scan of ViT-Base fine-tuning on CIFAR-10, without operating it on LLM fine-tuning or pretraining, which are the main application scenarios. Also, why do the curves of Cayley Adam and Stiefel Adam coincide?
- In the experiment of GPT2 LoRA fine tuning on E2E. Why does Stiefel-AdamW use (0.98,0.98) while AdamW uses (0.9, 0.999), Scaled AdamW uses (0.7, 0.8), and GeoLoRA uses (0.98,0.98) for hyperparameter settings of ? (Table 4). Does the use of significantly different hyperparameters in different methods mean that some of the performance differences be due to tuning.
Inconsistencies
There are multiple instances in the article where the preceding statements do not match the content of the main text, resulting in inconsistencies. It makes readers feel like authors have not read the main text carefully, or like AI generated content. Specifically, for example:
-
The abstract states that full pretraining was performed on Qwen2, but in Experiment 5.3, it is mentioned that there are present numerical results for Qwen2 LoRA pretraining.
-
Contribution 6 in the Introduction states, “consistently improving stability and final performance.”, But in Experiment 5.2, it is stated that the GPT2 pretraining results `produce results comparable to standard AdamW'.
-
The introduction states that state space models (SSMs) are listed as applicable scenarios, but there is neither citation nor experimentation.
-
The introduction states that the proposed method can inherits the principal benefits of geometric optimization,i.e., parameter invariance to orthogonal reparametrization. However, in conclusion, it states that one limitation of the current method is that it still retains a set of orthogonal invariants, which stems from the entropy nature of the Adam algorithm.
-
Regarding the GPT pretraining section, Section 5.2 states that both AdamW and Stiefel AdamW were trained for 7000 iterations, but the caption in Table 3 states that, for 6000 iterations, which is a contradiction.
-
The title of Lemma A.3 is Second order accuracy, but the actual content is . Is it the same thing as the first order retrieval condition in line 893?
-
Are the three parts 864-866,888-891,892-896 trying to express the same meaning? In line 891, is it necessary to add reference after "see, e.g., standard analyses of imperfect retraction schemes".
-
Is Theorem A.2 proving linear convergence? Why is it mentioned in the text as convergences quadratically in line 917?
-
The title of Figure 1 is Speed vs Accuracy Trade off, but the y-axis of this graph represents the training loss rather than accuracy.
Questions for authors
- Would it be possible to consider whether the current gains might hold at larger model sizes, or that the fine-tuning setting may transfer to pretraining. Since the scale and training length are insufficient to prove the performance improvement, would full-parameter pretraining on a larger model, or longer training runs, be feasible additions?
- Regarding the baseline, it is appreciated to compare with the same period of work that applied Stiefel constraints on LoRA . Furthermore, why only constrain A and not constrain B?
- Can theoretical analysis be transferred or guided on real experiments, and is there a gap between assumptions and experiments?
- Would it be possible to operate learning rate sweeps on LLM fine-tuning or pretraining? Why do the curves of Cayley Adam and Stiefel Adam coincide?
- Why does Stiefel-AdamW use (0.98,0.98) while AdamW uses (0.9, 0.999), Scaled AdamW uses (0.7, 0.8), and GeoLoRA uses (0.98,0.98) for hyperparameter settings of ? Does the use of significantly different hyperparameters in different methods mean that some of the performance differences be due to tuning?
- Please explain the inconsistencies of this paper mentioned above.
2 comments
Inconsistencies
We would first like to point out that the scientific content of this paper is entirely the authors’ own work. AI was used only to improve the English in a few passages. We take full responsibility for the manuscript and the accuracy of this statement. We consider the suggestion that our scientific work is AI-generated a serious allegation, which we do not take lightly. We respectfully ask the reviewer to clarify the specific basis for this concern so that we can address it directly.
- We will delete "full" in the abstract.
- We will replace the claim in Contribution 6 with task-specific statements. In particular, Sec. 5.2 reports GPT2 pretraining performance comparable to AdamW. We will therefore describe the results as competitive performance across the tested settings, with improvements in several cases, rather than claim consistent improvements in stability and final performance.
- This is a fair point and we will emphasize the SSM setting in more details. A concrete example is Mamba's time-step projection, , where denotes the input features to that projection. See for example [4] or the original Mamba paper [5] (sec 3.6).
- We will distinguish representation symmetry, for , from optimizer equivariance. Entrywise Adam generally depends on the basis. We will correct the introduction and conclusion: the residual symmetry belongs to the factorization; basis dependence comes from the adaptive update.
- This is a typo, the correct is 7000 iterations.
- We will rename Lemma A.3 "First-order consistency and approximation error" and retain . The finite update approximates the exact retraction: first-order consistency does not imply exact manifold feasibility.
- We will consolidate the repeated passages in App. A.2.2 and remove the vague reference at line 891 and the associated general convergence-preservation claim.
- Thm. A.2 establishes linear (geometric) convergence, for . We will correct "quadratically" at line 917 and distinguish this rate from approximation order in the stepsize.
- We will change the left-panel title of Fig. 1 from "Speed vs Accuracy Trade-off" to "Speed vs Training Loss".
[1] Park et al. 2025, Stiefel-LoRA
[3] Zhang and Pilanci 2024, Riemannian Preconditioned LoRA
[4] Tri Dao and Albert Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, ArXiv 2024.
[5] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2024
We thank the reviewer the low memory overhead, modular retractions, and performance across several settings. Replies 1-5 address the weaknesses and Questions 1-5; the corrections address Question 6.
Critical Weaknesses and Questions For Authors
5.For GPT2/E2E, AdamW's (0.9,0.999) and Scaled AdamW's (0.7,0.8) follow Zhang and Pilanci [3], App. E.1.1, Tab. 3. Our (0.98,0.98) setting (Tab. 4) is a separate choice; we will clarify its selection. As we discussed in subsection 5.1, GPT paragraph, we observe consistently with [Zhang & Pilanci, 2024] that a more aggressive moving average seems to be more effective for Riemannian based methods. The methods is stable but we observe performance to be consistently better with a more aggressive choice of (β1,β2). We will further clarify in the revised version.