- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes TUON, a tensor-aware optimizer for large language model pretraining that exploits correlations across Transformer layers. TUON uses a data-dependent unitary transform (DecorQ) to capture cross-layer correlations, followed by slice-wise polarization and an inverse transform to recover parameter updates. Experiments on different models show that TUON generally achieves lower validation loss than Muon and TEON with modest computational overhead.
Critical strengths
- TUON exploits correlations across Transformer layers while preserving layer-specific characteristics, addressing limitations of existing matrix- and tensor-aware optimizers
Critical weaknesses
-
TUON achieves only modest improvements over existing optimizers. For example, on Qwen3-4B, its validation loss is only 0.0006 lower than TEON's. It also underperforms AdamW on GPT-2 124M. These results raise questions about the practical significance of the reported gains, particularly given the additional computational overhead.
-
There are several potentially incorrect theoretical claims and implementation inconsistencies. Appendix E claims that Frobenius orthogonality between transformed momentum matrices is preserved after polarization (Equation 95), which does not hold in general. For example, two Frobenius-orthogonal matrices can have nonorthogonal polar factors. Additionally, Algorithm 1 and Equation 101 use different normalization factors, potentially resulting in substantially different update magnitudes. The experimental configurations are also unclear: Appendix I.2 specifies a 1024-token context window for GPT-2 but later states that all runs use a sequence length of 2048. These issues raise concerns about the correctness of some theoretical claims and the reproducibility of the experiments.
Questions for authors
See the above weakness