acceptodds
Withdrawn from ICLR 2027

TUON: A Tensor-Aware Optimizer Utilizing Tensor-SVD and Polarization for Large Language Model Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large scale language model (LLM) pretraining is increasingly bottlenecked by optimizer efficiency, driving optimizer design from vector-aware methods such as AdamW to matrix-aware methods. Matrix-aware optimizers such as Muon exploit matrix structure but treat layers independently, neglecting cross-layer correlations. Tensor-aware optimizers are therefore needed, yet existing ones rely on a matricization-based tensor algebra and remain restricted to shallow stacking and specific modules. In this paper, we propose TUON, a novel optimizer built on a transform-based tensor algebra derived from the Tensor-SVD (T-SVD). Unlike methods that polarize a globally unfolded matrix, TUON mixes layers via a unitary transform and then polarizes each transformed slice, and inverts the transform to recover the update. This mix-then-separate strategy captures cross-layer correlations while preserving per-layer diversity, avoiding the structural dimension imbalance and cross-layer misalignment of unfolding. Thus it successfully scales to more layers and diverse modules beyond QKV. Theoretically, we prove that TUON performs steepest descent under a -transformed tensor spectral norm, and identify its update as the tensor polar factor of the T-SVD. We further establish transform non-redundancy, norm and smoothness comparisons with Muon and TEON, and a convergence guarantee. Empirically, TUON improves consistently over Muon and TEON on GPT-124M/350M and Qwen-0.6B/4B, with ablations confirming the transformation mechanism, cross-layer structure beyond two-layer stacking, adaptability beyond QKV, and robustness to the polarization method.

Primary area:
optimization
Keywords:
Optimization, optimizer, tensor

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_hRMqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TUON, a tensor-aware optimizer for large language model pretraining that exploits correlations across Transformer layers. TUON uses a data-dependent unitary transform (DecorQ) to capture cross-layer correlations, followed by slice-wise polarization and an inverse transform to recover parameter updates. Experiments on different models show that TUON generally achieves lower validation loss than Muon and TEON with modest computational overhead.

Critical strengths
  • TUON exploits correlations across Transformer layers while preserving layer-specific characteristics, addressing limitations of existing matrix- and tensor-aware optimizers
Critical weaknesses
  • TUON achieves only modest improvements over existing optimizers. For example, on Qwen3-4B, its validation loss is only 0.0006 lower than TEON's. It also underperforms AdamW on GPT-2 124M. These results raise questions about the practical significance of the reported gains, particularly given the additional computational overhead.

  • There are several potentially incorrect theoretical claims and implementation inconsistencies. Appendix E claims that Frobenius orthogonality between transformed momentum matrices is preserved after polarization (Equation 95), which does not hold in general. For example, two Frobenius-orthogonal matrices can have nonorthogonal polar factors. Additionally, Algorithm 1 and Equation 101 use different normalization factors, potentially resulting in substantially different update magnitudes. The experimental configurations are also unclear: Appendix I.2 specifies a 1024-token context window for GPT-2 but later states that all runs use a sequence length of 2048. These issues raise concerns about the correctness of some theoretical claims and the reproducibility of the experiments.

Questions for authors

See the above weakness

Reviewer_6RnnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Authors propose new tensor-structure-aware optimizer TUON, that stacks compatible momentum matrices across transformer layers, rotates along the layer dimension, polarizes each transformed slice, and applies the inverse rotation. Its default transform, DecorQ, comes from the layer momentum Gram matrix. The paper interprets the exact update as a LMO under a transformed tensor norm and evaluates it on small LLM architectures. The experiments also address the design choices directly.

Critical strengths
  • The exact LMO construction and other optimizer's design choices rely on a good theoretical grounding.
  • Conducted ablations are helpful and address the proposed mechanism in different fashions: they vary stacking depth, module coverage, transforms, and polarization routines; DecorQ improves from two-layer to full stacking on all four reported models.
Critical weaknesses
  • The major concern is about the frobenius-orthogonal momentum. If I understood correctly, in Appendix E, where authors discuss the design of Q and their explanation for DecorQ choice, the key derivation relies heavilly on the fact, that polar factors should be an orthogonal matrices. However, if one chooses Mi=diag(2,1,1)M_i = \mathrm{diag} (2, 1, 1) and Mj=diag(1,1,−3)M_j = \mathrm{diag} (1, 1, -3), then these matrices are frobenius-orthogonal, but polar factors (that are Pi=diag(1,1,1)P_i = \mathrm{diag} (1, 1, 1) and Pj=diag(1,1,−1)P_j = \mathrm{diag} (1, 1, -1), respectively) are not. This counter-example destroys a central claim, that DecorQ creates independent optimization subproblems, and the intuition, that lies behind it.
  • The result of the Theorem 5 is misleading, because it's design relies on the full-gradient updates without stochastic momentum and approximate polarization. In Appendix H.7 authors show, that basis can change during the training process, but still retains an exact full-gradient oracle. A guarantee for the practical method needs an explicit error analysis along the lines illustrated by [1] or [2], with similar assumptions checked for TUON.
  • The smallest empirical improvements need uncertainty estimates and a fair time comparison. Repeated runs and time-to-fixed-loss curves would establish whether the small loss difference amounts to an efficiency gain. The transform ablation also needs a more restrained interpretation due to theoretical inaccuracy mentioned above.

[1] Kim G. Y., Oh M. Convergence of muon with newton-schulz //International Conference on Learning Representations. – 2026. – Т. 2026. – С. 45672-45718.

[2] Riabinin A. et al. Gluon: Making muon & scion great again! (bridging theory and practice of lmo-based optimizers for llms) //arXiv preprint arXiv:2505.13416

Questions for authors
  • Which geometric statement actually motivates the DecorQ choice?
  • Can you please provide different seeds and equal-time comparisons of TUON's gains over the strongest baseline?
Reviewer_6BEFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TUON, a tensor-aware optimizer for large language model pretraining. The method stacks shape-compatible momentum matrices across layers into a third-order tensor, applies a unitary transform along the layer dimension, extracts the polar factor of each transformed slice, and maps the update back through the inverse transform. The authors introduce a data-dependent transform, DecorQ, constructed from a layer-wise Gram matrix, and interpret the method through transform-induced tensor norms, linear minimization oracles, and the T-SVD polar factor. Experiments cover GPT-2 124M/350M and Qwen3 0.6B/4B, with comparisons across transforms, stacking depths, module coverage, and polar approximation methods.

Critical strengths
  1. The method addresses a meaningful problem and has a clear design. Exploiting cross-layer structure to improve independent matrix optimization is a worthwhile research direction. The transform–slice-wise-polar–inverse-transform pipeline offers an alternative to global matricization and can be implemented through batched matrix operations.
  2. Several mathematical characterizations are useful. For a fixed unitary transform, the paper defines the corresponding norm and dual norm and characterizes the update through a linear minimization oracle and a generalized T-SVD polar factor. The distinction between global unfolding and slice-wise polarization regarding transformation cancellation also helps explain the method.
  3. The experiments include several relevant ablations and strong TEON configurations. The authors compare fixed and adaptive transforms, two-layer and full stacking, QKV-only and broader module coverage, and different polar approximation methods. In Table 2, TUON outperforms Muon and TEON in all four settings and AdamW in three settings. These results are promising.
Critical weaknesses
  1. Improvements over the closest baseline are small, and the existing single-run evidence does not establish robustness. In Table 2, TUON improves validation loss over TEON by: 0.0085,0.0010,0.0091,0.0006. The differences at 350M and 4B are particularly small. The current evidence establishes improvements in the reported runs but is insufficient to determine whether these improvements are consistent. The interpretation of the transform mechanism also requires greater caution. In Table 7, DecorQ improves over DCT by 0.0502, but over the stronger fixed DFT transform by only 0.0033. In Table 11, replacing Newton–Schulz with PolarExpress improves full-stacking performance by 0.0029. These latter two differences are of comparable magnitude. Thus, the claim that the choice of polar approximation matters an order of magnitude less than the transform mechanism relies primarily on comparison with the weaker DCT baseline and is not yet sufficiently established. I recommend repeated runs for at least representative model scales and key ablations, focusing on DecorQ, strong fixed transforms, and polar approximation methods.
  2. Step-time measurements are provided, but a better quality–compute trade-off has not been established. Reporting absolute step times is a strength. However, on Qwen3-4B, the reported times are: TUON: 6150.7 ms; TEON: 6081.5 ms; Muon: 5794.7 ms; AdamW: 5466.9 ms. TUON is approximately 1.1% slower than TEON, 6.1% slower than Muon, and 12.5% slower than AdamW. In particular, its final loss improvement over TEON is only 0.0006. Validation loss versus wall-clock time, or time to a common target loss, is therefore needed to determine whether the additional overhead is worthwhile. Furthermore, an oracle advantage at the same operator-norm radius does not automatically imply an optimization advantage after matching update energy. The two-sided smoothness comparisons also permit either improvement or deterioration and cannot directly guarantee that DecorQ yields faster practical convergence.
Questions for authors
  1. How will the authors correct Equation (95) and the interference-free decoupling explanation? Can they report layer Gram matrices before polarization, after polarization, and after the inverse transform?
  2. Do the main experiments use the instantaneous raw Gram matrix, a moving average of normalized correlation matrices, or a moving average of unnormalized Gram matrices? What update scaling is actually used? Please provide the corresponding code revision, configurations, and update-RMS-matched comparisons.
  3. Are the gains over TEON and strong fixed transforms consistent across repeated runs? Is the effect of polar approximation demonstrably smaller than that of transform selection?
  4. Can the authors provide loss-versus-wall-clock curves or training time to a common quality target?
Reviewer_s7ibOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors present TUON, an optimizer in the family of tensor-aware optimizers. TUON stacks matrices of the same shape in a transformer along a 3rd axis, and then applies a unitary transform and polarizes each resulting slice. The authors prove that this corresponds to a steepest-descent step in a transformed spectral norm and provide convergence guarantees. The authors compare TUON in experiments against Muon and TEON.

The main contribution is the proposition of the novel TUON optimizer, theoretically validating it and performing accompanying pre-training experiments.

Critical strengths

The theoretical side of the TUON optimizer is well supported. The idea of mixing layers in a stack with a preliminary Q-transform is novel and interesting.

Critical weaknesses

Possibly flawed practical evaluation: In table 2, the authors report a validation loss of 2.92 on AdamW and 3.048 on Muon, making Muon achieve a worse validation loss than AdamW. This contradicts well supported findings by the optimization literature: Wen et al. (https://arxiv.org/pdf/2509.02046) show that Muon achieves a smaller validation loss than AdamW on small Chinchilla optimal models, and Muon (https://github.com/yxlllc/Muon) itself also reports that their optimizer achieves much better validation loss at any token budget compared to AdamW (https://x.com/kellerjordan0/status/1842300916864844014). The authors do not address this gap at all. Additionally, the gap to TEON is often extremely small (for example, 0.0091 validation loss improvement), but the authors do not include re-runs with different seeds at all. As the main contribution of this paper is a practical method that claims improvements over other optimizers, providing a sound evaluation is critical.

Concerns over practical impact: As mentioned before, the gap to TEON and Muon is very small in many cases. Additionally, the optimizer is slower than Muon (by ~6% in Qwen 4B). First of all, it is unclear how this gap to Muon scales as model size is increased. Additionally, as TUON is slower and only marginally better than Muon, it could be that TUON actually achieves a higher validation error than Muon when given the same wall-clock time.

Questions for authors
  • I would ask the authors to discuss the gap from Muon to AdamW, particularly in the 124M model case. Please describe in detail how the relevant hyperparameters for Muon have been tuned and selected.
  • Please report the achieved validation error in wall-clock time for all selected optimizers.
  • How would the run-time of the TUON optimizer scale as model size is further increased, relative to Muon? If it is hard to make a theoretical claim about this, the authors could run just a few steps on large models, which should not be computationally demanding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.