acceptodds
Under review as a conference paper at ICLR 2027

DAWN: Scalable Matrix-Whitening Optimization via Spectral Selectivity

Abstract

Matrix-whitening optimizers such as Muon improve the data efficiency of language-model pretraining, but their full-spectrum polar updates are expensive and treat all singular directions as equally reliable. We argue that this may be unnecessary: across training, gradient matrices have a stable low-rank head and a noisy residual tail. Polarizing the tail amplifies noise to unit magnitude and wastes the most expensive part of the per-step compute. We propose DAWN (Decoupled Adaptive Whitening of Noisy gradients), which polarizes only the empirical head and applies a row-wise RMS update on the residual—a block-Schatten partition with one coarse spectral block on the head and singleton row blocks on the tail. DAWN reduces the per-layer whitening cost from to , matches Muon on GPT-2XL-scale (1.5B) pretraining, and lowers optimizer wall-clock time.

Primary area:
optimization
Keywords:
matrix optimization, Muon, spectral whitening, low-rank optimization, stochastic optimization, adaptive optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bbtwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DAWN, a matrix-whitening optimizer that applies polar whitening to a low-rank gradient head and a row-wise RMS update to the residual tail. The head subspace is estimated with randomized SVD and refreshed periodically, with subspace tracking between refreshes. The motivation is that weak singular directions of stochastic gradients may be noise-dominated, so fully polarizing them can amplify noise while incurring substantial computation. The paper evaluates DAWN on synthetic matrices, GPT-2 124M, GPT-2XL 1.5B, and a custom wide GPT (4B). It reports that DAWN can approach Muon’s validation loss while reducing optimizer time, with smaller end-to-end step-time improvements. The paper also presents a BBP-inspired per-mode MSE argument and a block-Schatten interpretation of the head-tail split.

Critical strengths
  1. Relevant compute-quality tradeoff. DAWN addresses the practical cost of Muon-style matrix whitening, which can become significant for wide models. Across GPT-2XL and the wide 4B model, the paper reported successfule reduce of optimizer time. This supports the paper’s central claim that selective whitening can preserve much of Muon’s benefit at lower optimization cost.

  2. Meaningful empirical investigation of the design. The paper includes controlled synthetic experiments and ablations over head rank, SVD refresh frequency, head whitening, and tail processing. These experiments provide evidence that both the low-rank whitening head and the retained tail update are important, rather than the reported results arising solely from removing computation.

Critical weaknesses
  1. The theoretical justification and convergence analysis are incomplete for the implemented optimizer. The paper’s theory focuses on per-mode estimation MSE under an idealized spiked-noise model and on the tracking error of the randomized-SVD subspace. It does not establish convergence or stability of the actual DAWN training dynamics, which combine RowRMS tail preconditioning, momentum, periodic subspace refresh, stale subspaces, and a time-varying rank schedule. This gap is consequential because the practical update is not the same as the raw-residual update analyzed in the theory. The empirical loss curves provide evidence on the tested GPT-style models, but do not establish a general convergence guarantee.

  2. The empirical comparisons do not establish a sufficiently fair optimizer advantage. Recent Muon scaling work explicitly analyzes the shape-dependent RMS of Muon updates, rescales them to approximately match AdamW’s empirical update-RMS range, and then reuses AdamW-tuned learning rates and weight decay.(https://arxiv.org/pdf/2502.16982, https://arxiv.org/abs/2602.01105, https://arxiv.org/abs/2507.11005) In contrast, although DAWN includes a global target_rms calibration, the paper does not report its value, compare per-layer update RMS or update-to-weight ratios across DAWN, Muon, and AdamW, or explain whether the same effective-update alignment is applied to all optimizers. This is consequential because RowRMS and the head-tail normalization can change DAWN’s effective step size. Moreover, in the 4B experiment AdamW uses a learning rate ten times smaller than Muon and DAWN. Thus, the reported quality differences may partly reflect mismatched effective update scales and tuning rather than the optimizer algorithms alone.

Questions for authors

1.Can the total-MSE analysis in Section 3.3 be derived while retaining the off-diagonal overlaps between empirical and population singular vectors? If additional alignment assumptions are required, can they be stated and their relevance to training gradients justified?

  1. What target_rms is used in each pretraining experiment? Is the same update-RMS calibration applied to Muon, and can the authors report per-layer update RMS or update-to-weight ratios for DAWN, Muon, and AdamW?
  2. Can the 4B AdamW baseline be retuned, or can the authors provide a learning-rate sweep that makes the comparison more informative?
  3. Which GPT-2XL configuration corresponds to the main loss and timing results, and how were the reported uncertainty ranges computed?
  4. Can the authors clarify the distributed implementation behind the multi-GPU timing results, including how the matrix subspace computation interacts with parameter/gradient sharding and communication?
Reviewer_oAJSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a new matrix optimizer DAWN, based on the observation that the polar decomposition is expensive and can amplify batch-noise unnecessarily. Dawn attempts to split out the 'spike' signal from the tail noise, and apply Polar only to the head of the signal. Empirically they claim that their operation is cheaper and converges faster.

Critical strengths

The idea is very interesting - that there are directions in the batch-gradient that are uninformative, and so point in (Haar) random directions and should not be amplified seems like a strong bit of intuition and I welcome the authors' exploration of the idea.

Critical weaknesses
  1. The cost of the whitening operation as a significant problem with Muon etc is overstated, and can be solved with good engineering. For example in the Aro paper [1], they describe a round-robin ownership pattern of the whitening ("rotation" in Aro) matrix, resulting in a 0.5% overhead relative to Adam. Of course model-size, batch-size, etc will make a difference, but I'm unconvinced that the overhead cost is anything more than an implementation detail.

[1] https://arxiv.org/abs/2602.09006

  1. Readability is very poor. The notation can be hard to follow. For example, at the top of section 2, you introduce G* and Y. and then in 2.1 you write "given ~G...". You did not define ~G, but you did define G* that you're now not using. Very frustrating for a new reader trying to follow. Section 2.1 is critical to the paper, but is very hard to decipher - pseudocode would be welcome here. Many symbols are uysed without introduction: what is Delta_head? I feel like I should have read the appendix first.

  2. The main theoretical claim seems to be that unreliable, tail directions should be shrunk rather than polarized. I can see that you've centered the theory on the gradient estimation, but I don't see any theory that directly connects to faster convergence.

Questions for authors

Please can you write out the pseudo-code for your algorithm? Can you connect your theory to the actual convergence? Please define symbols before using them.

Reviewer_xRFBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DAWN, a Muon variant that splits each matrix gradient into a low-rank head and a residual tail. It applies Newton–Schulz orthogonalization to the head and row-wise RMS normalization to the tail, using periodically refreshed randomized SVD to track the head subspace.

Critical strengths
  1. The paper includes useful component ablations and experiments on 124M, 1.5B, and a wide 4B model, providing a large range of experiments.
Critical weaknesses
  1. The motivation of improving Muon efficiency seems to rise from hallucination and the efficiency comparison with Muon is not convincing. The motivation assumes that per-GPU optimizer cost stays constant as data parallelism increases, overlooking distributed Muon implementations (e.g., Khona et al., 2026). I also could not find support in the cited Kimi K2 report for the claimed ~20% optimizer overhead in line 32, which is also generally impossible. The paper does not clearly specify the Muon baseline implementation. If a proper distributed implementation is employed, I don't think Muon optimizer step is a bottleneck in training. Moreover, Table 3 reports median times over 200 steps, while DAWN refreshes its SVD every 40 steps, which can exclude expensive refresh steps and understate average cost. The authors should at least report the average number. It is therefore unclear how much of the reported speedup would remain against an optimized Muon implementation with a complete end-to-end timing comparison. Usually, from the infra side, an algorithm like DAWN is not very likely to be more efficient compared to Muon.
  2. The experimental reporting contains important inconsistencies. For GPT-2XL, Appendix F.2 reports a peak learning rate of 1.8e-4 with no warmup, whereas Appendix I.1 reports 3e-4 with a 100M-token warmup and a different decay schedule. Appendices D.2 and F.3 also give different rank-decay schedules. The main comparison stops at 90B tokens despite the stated 105B-token training run, and there is no clear learning-rate tuning across optimizers. These issues make the results difficult to reproduce and the final performance comparison hard to assess. The theory is mainly an idealized MSE analysis and does not establish an optimization advantage for the implemented RowRMS update.
Questions for authors
  1. What exact Muon implementation is used? Can the authors compare against an optimized or distributed Muon baseline and report mean end-to-end step time, including SVD refreshes and subspace tracking? Were the 32-GPU results in Table 3 directly measured?
  2. Can the authors reconcile the conflicting hyperparameters and rank schedules, report the final results at 105B tokens, and provide learning-rate sweeps for the main optimizers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.