acceptodds
Under review as a conference paper at ICLR 2027

Biax: Fixed-Radius Matrix Optimization with Factored Preconditioning for Transformers

Abstract

Matrix optimizers for Transformers must reconcile spectral direction, anisotropic gradient scale, and parameter-norm control. We present Biax, which combines Muon²-Factorized-style temporal row/column profiling before polar extraction with Hyperball’s per-matrix fixed-initial-radius projection, adding (O(m+n)) persistent state. The profile selects a spectrally balanced direction, while sphere projection controls parameter scale. We show that the idealized update is a profiled-covector spectral linear oracle followed by exact sphere projection, separating anisotropic direction selection from parameter-scale control. We evaluate Biax in a paired five-tier, 2B-token language-model study spanning 10M to 1B parameters, with additional 500M and 1B experiments trained for 20B tokens. We also conduct an adaptive four-condition study at 10M, comparing full and row/column-factored profiles both with and without fixed-radius projection. Factoring lowers mean NLL in both comparisons, but with four seeds neither its overall effect nor its interaction with fixed-radius projection is statistically resolved. In the 2B-token panel, Biax lowers mean held-out NLL against its fixed-radius parent at every scale and wins all four paired seeds at 500M and 1B. Additional tangent-history and final-direction projection are inferior at both large 2B-token tiers. These results identify Biax as a simple, low-state optimizer and demonstrate why optimizer mechanisms must be evaluated against their nearest parents across scale.

Primary area:
optimization
Keywords:
matrix optimization, language-model pretraining, Muon optimizer, factored preconditioning, fixed-radius optimization, polar decomposition, Transformer optimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_jLLwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Biax, a matrix optimizer that combines Muon2-Factorized-style temporal row/column second-moment profiling before polar direction extraction with Hyperball’s fixed-initial-Frobenius-radius parameter update. The paper provides an operator-level characterization of the method, including a profiled spectral linear-oracle interpretation and fixed-radius angular dynamics. Empirically, Biax is compared with Hyperball-Muon and Muon in 20B-token pretraining runs at 500M and 1B parameters using four paired seeds. At 500M, Biax obtains a small but statistically resolved improvement over Hyperball-Muon, although the two configurations use different inherited learning rates. At 1B, where the learning rate is matched, the comparison is effectively null. The paper appropriately frames these experiments as comparisons between complete optimizer configurations rather than as causal evidence for the profiling component.

Critical strengths
  1. Careful experimental accounting and appropriately calibrated claims. The paper clearly distinguishes matched and unmatched comparisons, reports paired-seed confidence intervals, discloses the learning-rate mismatch at 500M, and explicitly treats the 1B result as null. This substantially increases confidence in the claims that are actually made.
  2. Clean structural formulation of the optimizer. The decomposition of Biax into factored temporal profiling, spectral direction extraction, and fixed-radius projection is technically clear. The profiled-oracle and angular-dynamics viewpoints provide a useful conceptual description of how the combined optimizer behaves.
Critical weaknesses
  1. The experiments do not establish that the proposed profiling mechanism improves Hyperball-Muon. The only positive primary result is the 500M comparison, but Biax and Hyperball-Muon use different global learning rates, and this difference also propagates to the AdamW fallback parameters. At 1B, where the learning rates are matched, the difference is essentially zero. Consequently, there is currently no controlled empirical evidence that the defining profiling component of Biax provides an advantage. The authors themselves acknowledge this limitation.
  2. The methodological novelty appears incremental relative to the closest components. Biax is primarily an instantiation that combines Muon2-Factorized-style profiling with Hyperball-style radius control. Since the factored estimator, pre-polar adaptation, and fixed-radius projection are inherited ideas, the empirical burden for demonstrating that this particular composition is important is relatively high. The current scale-limited and mixed-result experiments do not yet meet that burden for me.
Questions for authors
  1. Clarification materially affecting my score: Can the authors report a learning-rate-matched 500M comparison between Biax and Hyperball-Muon, ideally with the same small learning-rate sweep for both methods? If the 500M advantage persists under this control, this would substantially strengthen the paper.
  2. Clarification materially affecting my score: Can the authors provide a direct mechanistic ablation that isolates the pre-polar profiling component, rather than comparing complete configurations? In particular, it would be useful to understand whether the null 1B result reflects scale, learning-rate sensitivity, or simply the absence of a profiling benefit.
Reviewer_2YkkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a Transformer matrix optimiser, Biax, that combines Muon2-Factorised-Style temporal row/column profiling before polar extraction with Hyperball’s per-matrix fixed-initial-radius projection. The method is evaluated on 500M/1B decoder-only Llama-style transformers, and Biax achieves a small gain on 500M but no difference on 1B.

Critical strengths

None

Critical weaknesses
  • The presentation is low quality and confusing. The authors do not provide any motivation for combining the Muon2-Factorised-Style profiling and Hyperball’s fixed-radius projection. The paper has almost no citations, except in the related works section (which is still insufficient), making it difficult for non-expert readers to trace related work or experimental settings. Multiple paragraphs are confusing and hard to follow. The authors also used only 6 of the 9 available pages, leaving many questions unanswered, including motivations and ablations.

  • The novelty is limited. Biax essentially combines two existing techniques and adds little novelty. The theory section is written in a very careful style but mostly repeats what is already known.

  • The experiments are only conducted on a single model architecture and dataset. The improvement shown for 500M is marginal, and because the authors used different learning rates, it is unclear whether this improvement comes from the varied LR or the method itself. For 1B, where learning rates are identical, no improvement is shown. The baseline settings are weak, and more optimisers (AdamW and other momentum-based methods) should be added.

Questions for authors
  • Can you clarify Biax's motivation and novelty?
  • Can you rerun the experiments for 500M with the same LRs to confirm where the improvement comes from?
  • How does Bias compare with other optimisers?
Reviewer_nU7bOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The author proposes Biax, which combines Muon^2-Factorized and Hyperball.

Critical strengths

They combine two ideas.

Critical weaknesses
  1. It's not novel.
  2. The result is weak. At 1B scale, Biax performs on-par with Hyperball-Muon
Questions for authors

The main text writes "The two scales give different outcomes: Biax is lower on all four paired seeds at 500M, while the 1B comparison shows no resolved difference." What do you think is the main contribution then?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.