- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether orthogonality constraints provide additional benefits for large language model (LLM) training beyond sphere optimization. To disentangle the effects of inter-row orthogonality and per-row norm control, the authors introduce a unified framework of block-wise orthogonality constraints, where the block size controls the degree of orthogonality while preserving learnable per-row magnitudes. They also develop OrthMuon, which combines Muon update directions with tangent-space projection and polynomial retraction, and explore different reparameterization strategies to recover the degrees of freedom lost under orthogonality constraints.
The authors systematically evaluate 17 training variants on decoder-only language models ranging from 360M to 3.1B parameters, with training budgets reaching nearly 200B tokens. The experimental results show that sphere optimization (block size d = 1) consistently outperforms tuned Muon baselines, while imposing additional inter-row orthogonality (d > 1) does not yield consistent improvements and can even degrade performance for larger block sizes. Furthermore, the authors revisit five previously proposed orthogonality-related methods in pre-training and fine-tuning, suggesting that their reported gains may be attributed to factors such as numerical precision, training duration, optimizer implementation, or per-row norm control rather than inter-row orthogonality itself.
Overall, the paper provides a systematic empirical investigation of the actual contribution of orthogonality constraints to LLM training and highlights the importance of carefully controlled baselines when evaluating geometry-constrained optimization methods.
Critical strengths
1. Well-motivated problem and carefully controlled experimental design. The paper addresses an important question regarding whether orthogonality itself provides benefits beyond per-row norm control in LLM training. The proposed block-wise framework provides a systematic way to isolate the contribution of inter-row orthogonality while keeping per-row magnitude control consistent. I find this experimental design particularly valuable, as it helps disentangle factors that have often been considered together in previous studies.
2. Comprehensive empirical investigation with meaningful findings. The paper evaluates 17 training variants across multiple model scales and training budgets, and further revisits several existing orthogonality-related methods. The results provide useful evidence that the benefits previously attributed to orthogonality may instead arise from norm control, numerical precision, or optimizer implementation. I believe these findings are informative and could help guide future research on geometry-aware optimization for LLMs.
Critical weaknesses
1. Limited validation at larger model scales. Although the experiments are extensive, the largest pretrained model contains approximately 3.1B parameters. More importantly, the performance gap between sphere optimization and stronger block-wise orthogonality appears to decrease as model size increases. This raises the question of whether the observed conclusions would continue to hold for substantially larger LLMs. I consider this a limitation of the generalizability of the findings rather than a fundamental flaw, especially given the computational cost of such controlled experiments.
2. Limited novelty in the underlying optimization techniques. Several components, including Stiefel manifold optimization, tangent-space projection, and magnitude-direction decomposition, have been explored in previous work. The proposed OrthMuon largely builds upon these established ideas. Therefore, I view the primary contribution as the systematic experimental framework and empirical analysis rather than a fundamentally new optimization algorithm. Nevertheless, I believe the empirical insights remain valuable.
Questions for authors
Q1. The results suggest that the performance gap between sphere optimization and block-wise orthogonality decreases as model size increases. Could the authors provide further analysis or discussion of this scaling trend? In particular, is there evidence supporting the extrapolation of the main conclusions to substantially larger models?
Q2. Section 4.5 reports that different block sizes achieve the lowest validation loss on different evaluation corpora. Could the authors further clarify how these results affect the generality of the conclusion that inter-row orthogonality provides no consistent benefit? Would a more detailed analysis across training and evaluation distributions help identify settings where stronger orthogonality constraints might be beneficial?