acceptodds
Under review as a conference paper at ICLR 2027

Do Orthogonality Constraints Help Language-Model Training Beyond Sphere Optimization?

Abstract

A growing body of work reports benefits from orthogonality constraints in fine-tuning large language models (LLMs). These results raise an open question: do orthogonality constraints also help LLM pre-training? We conduct a systematic empirical study of decoder-only LLM pre-training with models ranging from 360M to 3.1B parameters, trained on nearly 200B tokens. To explore the potential of orthogonality in pre-training, we examine seventeen variants of orthogonality training: some constrain the weight matrix itself, while others use orthogonal factors to reparameterize the weight. Given that requiring a square weight matrix to be strictly orthogonal results in a loss of half of its parameter degrees of freedom, we also investigate block-wise orthogonality, where every rows of the weight matrix are required to form a row-orthonormal (Stiefel) matrix, with the block size serving as a hyperparameter. Among block sizes ranging from 1 to the shorter side of the matrix, no reliably outperforms , and large blocks are clearly worse. However, when , the orthogonality constraint degenerates into sphere optimization, meaning each row direction is optimized on its own unit sphere, which by itself already improves on tuned Muon optimizers. Our conclusion is that, across the realizations we tried, orthogonality constraints do not help language-model pre-training beyond sphere optimization. Since this appears to conflict with several prior results, we re-examine five prior methods in pre-training and fine-tuning that report orthogonality improvements. In each case the reported gain is attributable to numerical precision, training horizon, optimizer realization, or per-row norm control rather than to orthogonality between rows.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM pretraining, Orthogonality, Manifold Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6zQqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether orthogonality constraints provide additional benefits for large language model (LLM) training beyond sphere optimization. To disentangle the effects of inter-row orthogonality and per-row norm control, the authors introduce a unified framework of block-wise orthogonality constraints, where the block size controls the degree of orthogonality while preserving learnable per-row magnitudes. They also develop OrthMuon, which combines Muon update directions with tangent-space projection and polynomial retraction, and explore different reparameterization strategies to recover the degrees of freedom lost under orthogonality constraints.

The authors systematically evaluate 17 training variants on decoder-only language models ranging from 360M to 3.1B parameters, with training budgets reaching nearly 200B tokens. The experimental results show that sphere optimization (block size d = 1) consistently outperforms tuned Muon baselines, while imposing additional inter-row orthogonality (d > 1) does not yield consistent improvements and can even degrade performance for larger block sizes. Furthermore, the authors revisit five previously proposed orthogonality-related methods in pre-training and fine-tuning, suggesting that their reported gains may be attributed to factors such as numerical precision, training duration, optimizer implementation, or per-row norm control rather than inter-row orthogonality itself.

Overall, the paper provides a systematic empirical investigation of the actual contribution of orthogonality constraints to LLM training and highlights the importance of carefully controlled baselines when evaluating geometry-constrained optimization methods.

Critical strengths

1. Well-motivated problem and carefully controlled experimental design. The paper addresses an important question regarding whether orthogonality itself provides benefits beyond per-row norm control in LLM training. The proposed block-wise framework provides a systematic way to isolate the contribution of inter-row orthogonality while keeping per-row magnitude control consistent. I find this experimental design particularly valuable, as it helps disentangle factors that have often been considered together in previous studies.

2. Comprehensive empirical investigation with meaningful findings. The paper evaluates 17 training variants across multiple model scales and training budgets, and further revisits several existing orthogonality-related methods. The results provide useful evidence that the benefits previously attributed to orthogonality may instead arise from norm control, numerical precision, or optimizer implementation. I believe these findings are informative and could help guide future research on geometry-aware optimization for LLMs.

Critical weaknesses

1. Limited validation at larger model scales. Although the experiments are extensive, the largest pretrained model contains approximately 3.1B parameters. More importantly, the performance gap between sphere optimization and stronger block-wise orthogonality appears to decrease as model size increases. This raises the question of whether the observed conclusions would continue to hold for substantially larger LLMs. I consider this a limitation of the generalizability of the findings rather than a fundamental flaw, especially given the computational cost of such controlled experiments.

2. Limited novelty in the underlying optimization techniques. Several components, including Stiefel manifold optimization, tangent-space projection, and magnitude-direction decomposition, have been explored in previous work. The proposed OrthMuon largely builds upon these established ideas. Therefore, I view the primary contribution as the systematic experimental framework and empirical analysis rather than a fundamentally new optimization algorithm. Nevertheless, I believe the empirical insights remain valuable.

Questions for authors

Q1. The results suggest that the performance gap between sphere optimization and block-wise orthogonality decreases as model size increases. Could the authors provide further analysis or discussion of this scaling trend? In particular, is there evidence supporting the extrapolation of the main conclusions to substantially larger models?

Q2. Section 4.5 reports that different block sizes achieve the lowest validation loss on different evaluation corpora. Could the authors further clarify how these results affect the generality of the conclusion that inter-row orthogonality provides no consistent benefit? Would a more detailed analysis across training and evaluation distributions help identify settings where stronger orthogonality constraints might be beneficial?

Reviewer_fEBaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

the work asks whether hard orthogonality constraints on weight matrices help LLM pretraining, and the authors benchmark different orthogonalization approaches. they introduce a clear measure of how to control different types of orthogonality in weight matrices from spherical constraints to row-orthonormal. to decide on the best approach, the authors run different variants of muon for models of different parameterizations, and conduct extensive experiments. the authors find a simple spherical constraints to be the best approach. i find it to be a useful study, very relevant to current advances, however, the i notice some inconsistency in reports (see my weaknessess) and the work requires more polishing.

Critical strengths

Strengths.

S1: a clean design of experiments. only one hyperparameter controls the entire sweep of different model orthogonalization approaches. of course it does not cover all different scenarios, but the authors state it clearly in the limitations. this kind of unification is easy to follow, and the setup overall is quite easy to follow. it would be even better if the authors open-source the efficient implementation of the OrthMuon optimizer (which is quite easy to implement).

Critical weaknesses

Weaknesess.

W1: paper does not give any understanding about the actual training dynamics under different norm constraints. showing just the final loss is no enough. i would be curious to see the actual loss (steps or tokens) plots and the gradient norm dynamic. also, its useful for the authors to track the model norm so in different constrained settings, by sweeping d, they would be able to verify correctness of the setup. as i see, the norm for matrix params should be constant for the spherical case, but the model still has gains that contribute towards the grow of the total norm. observing how different optimizers affect this is definitely a bonus for the paper.

W2: inconsistent description of weight decay. Appendix B (Line 707) states that decoupled weight decay is applied to Muon-updated matrices; Appendix C (Lines 841–842) states that row gains and constrained directional factors receive no weight decay, which is in line with the limitations. but Appendix N (Lines1286 and 1290) reports weight decay 0.1 for the spherical case (d=1) and a row-orthonormal (d=32) one. it must be some confusion here, i would love the authors to clarify it. additionally, it would enhance the paper if the authors mention tables with hyperparameter sweeps for each experiment in Appendix

Questions for authors

basically my weaknesses points

Q1: could the authors show plots with dynamic of the training and/or validation loss throughout the training?

Q2: also could the authors show the training dynamics plots such as: model norm (total norm, only matrix params, only gains), gradient norm for different d?

Q3: could the authors clarify the mismatch about weight decay (W2), and show tables with clear hyperparameter sweeps for main runs?

all points are critical for assessment and i intent to increase the score towards Acceptance if answered properly. if the authors cannot edit PDFs, i would be happy to see a table with loss/norm/gradient norm at X tokens with around X=0.5 B tokens for short runs and X=1 B tokens for 50B+ long runs. in my opinion looking at the training dynamics gives much more understanding: you could easily see whether or not the weight decay has been used, which scheduler, whether the training is stable. overall, paper claims very interesting things, but implementation could be substantially improved.

Reviewer_bHtoOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether pairwise orthogonality improves language-model training beyond controlling individual row norms. It varies the size of orthogonal blocks while keeping learned per-row magnitudes, using sphere optimization at block size one as the main reference. The principal C4 experiments find no consistent benefit from adding orthogonality, with a measurable penalty for 32-row blocks at 360M scale. Additional parameterization and optimizer controls examine this result, and the paper revisits five prior methods to investigate other explanations for their reported gains.

Critical strengths
  1. The block-size comparison uses an appropriate control: learned per-row magnitudes are retained as pairwise orthogonality is added. Appendix F also reports paired-seed uncertainty. At 360M, the 32-row constraint increases loss at both tested budgets, while the eight-row result remains unresolved. The matrix-gain and optimizer controls make this much more informative than simply comparing an orthogonal model with an unconstrained baseline.

  2. The StelLA study provides a separate, direct test of the same question. Across four paired seeds, sphere-constrained factors improve accuracy by 2.116 points over Euclidean factors, while adding Stiefel constraints changes it by −0.089 points (Appendix R). The factor-decay and learning-rate checks are useful here: they support the interpretation that norm control accounts for much of the gain in this setting.

Critical weaknesses
  1. The abstract goes too far in attributing the gains of all five prior methods to other factors. The longer POET runs use a separate attention-only implementation, and the Pion comparison changes the original context length from 256 to 4096 (Appendices P–Q). These experiments show sensitivity to precision and training horizon under the tested protocols, but do not fully establish the cause of the original improvements. Section 5 makes this distinction more carefully than the abstract.

  2. Some auxiliary negative comparisons are not fully bracketed by the learning-rate search. All four 1.3B manifold-Muon arms prefer the smallest tested rate, 0.001, while Pion prefers its largest tested rate, also 0.001 (Tables 29 and 39). Extending those searches could matter when ruling out alternative optimizer realizations. This concern does not apply to the main matched block-size ladder, whose selected rate is inside its tested range.

Questions for authors
  1. Is there evidence under the original POET and Pion protocols that supports the stronger attribution claim in the abstract? Otherwise, could that claim be restricted to what the modified comparisons establish?

  2. Does extending the learning-rate range below 0.001 for 1.3B manifold Muon and above 0.001 for Pion change their ordering?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.