- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DAWN, a matrix-whitening optimizer that applies polar whitening to a low-rank gradient head and a row-wise RMS update to the residual tail. The head subspace is estimated with randomized SVD and refreshed periodically, with subspace tracking between refreshes. The motivation is that weak singular directions of stochastic gradients may be noise-dominated, so fully polarizing them can amplify noise while incurring substantial computation. The paper evaluates DAWN on synthetic matrices, GPT-2 124M, GPT-2XL 1.5B, and a custom wide GPT (4B). It reports that DAWN can approach Muon’s validation loss while reducing optimizer time, with smaller end-to-end step-time improvements. The paper also presents a BBP-inspired per-mode MSE argument and a block-Schatten interpretation of the head-tail split.
Critical strengths
-
Relevant compute-quality tradeoff. DAWN addresses the practical cost of Muon-style matrix whitening, which can become significant for wide models. Across GPT-2XL and the wide 4B model, the paper reported successfule reduce of optimizer time. This supports the paper’s central claim that selective whitening can preserve much of Muon’s benefit at lower optimization cost.
-
Meaningful empirical investigation of the design. The paper includes controlled synthetic experiments and ablations over head rank, SVD refresh frequency, head whitening, and tail processing. These experiments provide evidence that both the low-rank whitening head and the retained tail update are important, rather than the reported results arising solely from removing computation.
Critical weaknesses
-
The theoretical justification and convergence analysis are incomplete for the implemented optimizer. The paper’s theory focuses on per-mode estimation MSE under an idealized spiked-noise model and on the tracking error of the randomized-SVD subspace. It does not establish convergence or stability of the actual DAWN training dynamics, which combine RowRMS tail preconditioning, momentum, periodic subspace refresh, stale subspaces, and a time-varying rank schedule. This gap is consequential because the practical update is not the same as the raw-residual update analyzed in the theory. The empirical loss curves provide evidence on the tested GPT-style models, but do not establish a general convergence guarantee.
-
The empirical comparisons do not establish a sufficiently fair optimizer advantage. Recent Muon scaling work explicitly analyzes the shape-dependent RMS of Muon updates, rescales them to approximately match AdamW’s empirical update-RMS range, and then reuses AdamW-tuned learning rates and weight decay.(https://arxiv.org/pdf/2502.16982, https://arxiv.org/abs/2602.01105, https://arxiv.org/abs/2507.11005) In contrast, although DAWN includes a global target_rms calibration, the paper does not report its value, compare per-layer update RMS or update-to-weight ratios across DAWN, Muon, and AdamW, or explain whether the same effective-update alignment is applied to all optimizers. This is consequential because RowRMS and the head-tail normalization can change DAWN’s effective step size. Moreover, in the 4B experiment AdamW uses a learning rate ten times smaller than Muon and DAWN. Thus, the reported quality differences may partly reflect mismatched effective update scales and tuning rather than the optimizer algorithms alone.
Questions for authors
1.Can the total-MSE analysis in Section 3.3 be derived while retaining the off-diagonal overlaps between empirical and population singular vectors? If additional alignment assumptions are required, can they be stated and their relevance to training gradients justified?
- What target_rms is used in each pretraining experiment? Is the same update-RMS calibration applied to Muon, and can the authors report per-layer update RMS or update-to-weight ratios for DAWN, Muon, and AdamW?
- Can the 4B AdamW baseline be retuned, or can the authors provide a learning-rate sweep that makes the comparison more informative?
- Which GPT-2XL configuration corresponds to the main loss and timing results, and how were the reported uncertainty ranges computed?
- Can the authors clarify the distributed implementation behind the multi-GPU timing results, including how the matrix subspace computation interacts with parameter/gradient sharding and communication?