- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes MSign as an optimizer for LLM pretraining aimed at preventing instabilities. It is designed based on the observations that, before collapse, there is a rapid decline in the weight matrix stable ranks as well as increased alignment between Jacobians from adjacent layers. To prevent this, MSign uses SVD every T iterations to set nonzero singular values of the weight matrices to 1, while also restoring the norm. The method is evaluated in pretraining 5M-3B models showing that it can prevent loss instabilities.
Critical strengths
The observations relating the loss divergence to matrix stable ranks and alignment between Jacobians are interesting, and the method appears to prevent instabilities in the settings tested.
Critical weaknesses
Experiment issues.
The paper's main results include only cases where pretraining with AdamW fails due to loss divergence. While instabilities can be an issue in LLM pretraining, properly chosen hyperparameters and modern architectures (e.g. with techniques known to prevent instabilities such as QK-norm or using the more recent Muon optimizer) can give stable training. Examples of recent papers reporting stable loss include [1,2,3,4,5].
Therefore, given that the proposed technique also requires SVD (and the paper reports a slowdown up to 7% even when it is applied periodically), I am very skeptical if it reasonable to "pay" for such a the slowdown just in case an instability might occur, despite examples of stable modern pretraining recipes.
Also, the paper currently does not compare the final loss achieved with MSign with the one achieved from a (successful) base optimizer run (i.e., one that achieved convergence). Whether applying MSign affects the best obtainable loss is not tested. Compatibility with the popular Muon optimizer is also not tested in the paper, which is important since the orthogonalization in Muon also can affect the matrix stable rank and is lately the optimizer of choice in large scale experiments [1,2].
[1] Muon is scalable for llm training. Liu et al., 2025.
[2] Kimi K2: Open Agentic Intelligence. Kimi Team, 2025.
[3] LLaMA: Open and Efficient Foundation Language Models. Touvron et al., 2023.
[4] Llama 2: Open Foundation and Fine-Tuned Chat Models. Touvron et al., 2023.
[5] Open Mixture-of-Experts Language Models. Muennighoff et al., 2025.
Presentation issues.
There are several presentation issues at the current version making reading the paper difficult.
For example: Table 1 is placed in page 8 right below "6.1 Experimental Setting", but belongs to section 6.3. Figure 1 left does not have a label in the legend for the grey curve, while Figure 1 right has no legend at all. Figure 2 also does not have a legend with labels identifying each curve, and its title is cut out (it reads "obian Lower (...) during Tr"). Throughput analysis in Table 1 and 3 does not list the environment settings (e.g., number/type of GPUs) and it appears there is a mismatch between Sigma throughputs (Table 1 reports 6,097 tokens/second, Table 3 all are above 18k).
Also the framing of MSign as an optimizer is in my opinion not very accurate, as it is an extra technique acting on top of the base optimizer (as shown in Algorithm 1). This is not clear for example from the abstract or contributions, which simply mention "we propose MSign, a new optimizer (...)".
Questions for authors
Have you tried approximate methods instead of SVD, or tested more efficient implementations from a systems perspective? Also what are the current system settings (e.g. number of GPUs, distributed training) used when measuring the slowdown, especially in the 1B and 3B model experiments?
Have you tried applying MSign to Muon and could you comment on the interaction of Muon's orthogonalization of the momentum to MSign's operations?