acceptodds
Withdrawn from ICLR 2027

MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training instability remains a critical challenge in large language model (LLM) pretraining, often manifesting as sudden gradient explosions that waste significant computational resources. We study training failures in a 5M-parameter NanoGPT model scaled via P, identifying two key phenomena preceding collapse: (1) rapid decline in weight matrix stable rank (ratio of squared Frobenius norm to squared spectral norm), and (2) increasing alignment between adjacent layer Jacobians. We prove theoretically that these two conditions jointly cause exponential gradient norm growth with network depth. To break this instability mechanism, we propose MSign, a new optimizer that periodically applies matrix sign operations to restore stable rank. Experiments on models from 5M to 3B parameters demonstrate that MSign effectively prevents training failures with less than 7.0% computational overhead.

Primary area:
optimization
Keywords:
Large Language Models, Training Stability, Stable Rank, Gradient Explosion, Jacobian Alignment, Optimizer Design, Matrix Sign, Transformer Training, Pretraining Failure, Spectral Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Rwr7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MSign as an optimizer for LLM pretraining aimed at preventing instabilities. It is designed based on the observations that, before collapse, there is a rapid decline in the weight matrix stable ranks as well as increased alignment between Jacobians from adjacent layers. To prevent this, MSign uses SVD every T iterations to set nonzero singular values of the weight matrices to 1, while also restoring the norm. The method is evaluated in pretraining 5M-3B models showing that it can prevent loss instabilities.

Critical strengths

The observations relating the loss divergence to matrix stable ranks and alignment between Jacobians are interesting, and the method appears to prevent instabilities in the settings tested.

Critical weaknesses

Experiment issues.

The paper's main results include only cases where pretraining with AdamW fails due to loss divergence. While instabilities can be an issue in LLM pretraining, properly chosen hyperparameters and modern architectures (e.g. with techniques known to prevent instabilities such as QK-norm or using the more recent Muon optimizer) can give stable training. Examples of recent papers reporting stable loss include [1,2,3,4,5].

Therefore, given that the proposed technique also requires SVD (and the paper reports a slowdown up to 7% even when it is applied periodically), I am very skeptical if it reasonable to "pay" for such a the slowdown just in case an instability might occur, despite examples of stable modern pretraining recipes.

Also, the paper currently does not compare the final loss achieved with MSign with the one achieved from a (successful) base optimizer run (i.e., one that achieved convergence). Whether applying MSign affects the best obtainable loss is not tested. Compatibility with the popular Muon optimizer is also not tested in the paper, which is important since the orthogonalization in Muon also can affect the matrix stable rank and is lately the optimizer of choice in large scale experiments [1,2].

[1] Muon is scalable for llm training. Liu et al., 2025.

[2] Kimi K2: Open Agentic Intelligence. Kimi Team, 2025.

[3] LLaMA: Open and Efficient Foundation Language Models. Touvron et al., 2023.

[4] Llama 2: Open Foundation and Fine-Tuned Chat Models. Touvron et al., 2023.

[5] Open Mixture-of-Experts Language Models. Muennighoff et al., 2025.

Presentation issues.

There are several presentation issues at the current version making reading the paper difficult.

For example: Table 1 is placed in page 8 right below "6.1 Experimental Setting", but belongs to section 6.3. Figure 1 left does not have a label in the legend for the grey curve, while Figure 1 right has no legend at all. Figure 2 also does not have a legend with labels identifying each curve, and its title is cut out (it reads "obian Lower (...) during Tr"). Throughput analysis in Table 1 and 3 does not list the environment settings (e.g., number/type of GPUs) and it appears there is a mismatch between Sigma throughputs (Table 1 reports 6,097 tokens/second, Table 3 all are above 18k).

Also the framing of MSign as an optimizer is in my opinion not very accurate, as it is an extra technique acting on top of the base optimizer (as shown in Algorithm 1). This is not clear for example from the abstract or contributions, which simply mention "we propose MSign, a new optimizer (...)".

Questions for authors

Have you tried approximate methods instead of SVD, or tested more efficient implementations from a systems perspective? Also what are the current system settings (e.g. number of GPUs, distributed training) used when measuring the slowdown, especially in the 1B and 3B model experiments?

Have you tried applying MSign to Muon and could you comment on the interaction of Muon's orthogonalization of the momentum to MSign's operations?

Reviewer_V7bmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates training instability through declining weight stable rank and increasing alignment between adjacent layer Jacobians. It develops conditional bounds connecting aligned, spectrally concentrated Jacobians to gradient amplification, and proposes MSign, which periodically equalizes nonzero weight singular values while preserving the Frobenius norm. Experiments across four configurations from 5M to 3B parameters show that MSign avoids the reported training collapses with measured overhead below 7%.

Critical strengths
  1. The paper addresses a practically important problem and provides useful observations connecting spectral concentration, Jacobian alignment, and training instability.

  2. MSign is straightforward to implement as an intervention on top of an existing optimizer. Its effect on weight stable rank is explicit and mathematically transparent.

  3. Experiments cover dense and MoE architectures, while layer-selection and application-period ablations provide useful information about where and how frequently to intervene.

Critical weaknesses
  1. The theoretical mechanism is insufficiently validated in the actual models. The amplification results require near-rank-one layer Jacobians, controlled spectral tails, and task-gradient coupling, whereas the main observations concern weight stable rank and average adjacent alignment. Section 4.1.2 provides upper bounds rather than establishing these required conditions, and Figure 2 does not explain how they are verified; consequently, the connection between the conditional theory and the observed failures remains incomplete.

  2. The intervention does not isolate the role of stable-rank restoration. At fixed Frobenius norm �0��0�, MSign also reduces the spectral norm from �1��1� to �2��2�. A control that matches this spectral-norm reduction through scalar rescaling, while leaving stable rank unchanged, would help determine whether spectral flattening provides benefits beyond simply reducing amplification.

  3. Comparisons with alternative stabilization methods are missing. The baseline already uses gradient clipping, but the paper does not evaluate adaptive alternatives such as ZClip or spectral approaches. Comparisons against these methods and learning-rate-tuned AdamW are needed to establish MSign’s practical advantage beyond rescuing the selected failing configurations.

  4. Reliability and model-quality evidence remain limited. The paper does not report failure rates across random seeds, and held-out perplexity is provided only for the 5M and 40M models; the larger-model results primarily show training trajectories. Repeated runs and held-out evaluation at larger scales would establish how reliably MSign prevents collapse and whether repeated spectral flattening introduces a generalization cost.

Questions for authors

Do the stable-rank decrease conditions in Appendix C hold for the actual AdamW update directions, after momentum and adaptive preconditioning, rather than only for raw gradient-descent updates?

Reviewer_AKCzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper associates training collapse with declining weight stable rank and increasing inter-layer jacobian alignment. It proposes msign, which periodically equalizes weight singular values while preserving the Frobenius norm, and reports stabilization across models from 5m to 3b parameters.

Critical strengths
  • msign is a simple, targeted intervention that stabilizes the reported failing runs across dense and moe architectures, with modest measured overhead.
  • The layer-selection and application-period ablations provide useful evidence about where and how often spectral intervention is effective.
Critical weaknesses
  • Experiments mainly compare msign against failing AdamW runs, without independent-seed results or comparisons to alternative stabilization methods. Its advantage over a tuned stable baseline, including final model quality, remains unclear.
  • The theory requires near-rank-one block Jacobians and additional alignment and signal-coupling conditions, whereas the main observations concern weight stable rank. The paper does not sufficiently verify these conditions in the reported runs.
Questions for authors
  • Can msign be compared with learning-rate tuning, stronger clipping, qk normalization, or spectral-norm control? Please report both failure rates and final validation quality.
  • Do the actual block jacobians satisfy the spectral-tail, alignment, and terminal-excitation conditions closely enough to make the theoretical lower bound positive? How is the Figure 2 diagnostic computed?
  • Can you report multi-seed results and validation or downstream quality for the 1b and 3b models, including settings where baseline training is already stable?
  • What hardware and measurement settings underlie the throughput results? Sigma throughput at p=100 differs substantially between Tables 1 and 3 (6,097 versus 24,559 tokens/s).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.