- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether Muon’s advantage over AdamW arises from different learned features and where that advantage acts within a transformer. At matched validation loss, the authors compare top-k SAE features by firing-pattern correlation, calibrated against within-optimizer model-seed baselines. Cross-optimizer agreement is close to the within-AdamW baseline, while Muon has higher cross-seed agreement and substantially different weight spectra and representation geometry.
Targeted optimizer reassignment shows that Muon on the attention value/output pathway alone recovers approximately 86% of the loss advantage. Muon on QK, MLP blocks, or all hidden matrices except OV recovers none. QK and OV rank controls show that endpoint stable rank is not sufficient for the benefit. A Newton–Schulz iteration sweep supports the importance of the update process, although orthogonalization fidelity and update norm are not fully separated.
Critical strengths
-
The targeted interventions provide stronger evidence than observational spectral analysis alone. Muon on OV recovers most of the loss advantage, while Muon on the remaining hidden matrices does not, providing complementary localization evidence. Raising QK rank without gains and promoting OV rank without improvement further show that endpoint stable rank is not a sufficient explanation. These experiments are the main contribution supporting acceptance: OV sufficiency extends prior work, while the complement arm and rank controls add useful evidence.
-
The feature comparison includes carefully designed baselines and measurement controls. The authors use matched-loss checkpoints, within-optimizer model-seed baselines, and different SAE seeds on the same model, and treat model seeds rather than numerous features as the inferential unit. Checks involving live-feature sets, depth, head count, and activation functions further expose the boundaries of the findings and strengthen the credibility of the comparison.
Critical weaknesses
-
The strength of the mechanistic localization remains conditional on the hybrid optimizer protocol, and the fidelity attribution is confounded by update norm. Hybrid arms share a warmup schedule and are not jointly retuned. Existing learning-rate sweeps reduce some concerns, but OV necessity should remain explicitly conditional on the tested protocol. Separately, changing the number of Newton–Schulz iterations changes both orthogonalization fidelity and update norm, so the monotone loss curve does not independently identify fidelity as the driver. This limits the specific mechanistic interpretation without negating the OV localization result.
-
The same-features and reproducibility claims depend on the instrument and architecture. The comparison uses one top-k SAE family, a small number of layers, and best firing-pattern correlation, which does not establish general feature-content identity. Some hybrid reproducibility rankings change with live-feature selection; the 350M OV reproducibility gain is cut-dependent, and the GeLU gap narrows to the seed spread. Loss localization, feature reproducibility, and geometry changes should therefore be stated separately rather than summarized as one universal mechanism.
Questions for authors
The following clarifications could affect my final assessment:
-
To what extent do the existing experiments rule out learning-rate and warmup interactions in the OV necessity claim? Are configuration checks available for the complement arm? If evidence is limited, please explicitly restrict necessity to the current protocol.
-
Is a norm-matched Newton–Schulz control available to distinguish orthogonalization fidelity from effective step size? If not, please narrow the causal interpretation of the fidelity sweep accordingly.
-
Please summarize which feature conclusions survive matching live-feature counts or frequency cuts and which depend on the original SAE live sets, especially the 350M OV results. Does cross-optimizer agreement near the AdamW seed baseline support the stated non-inferiority claim, and what allowable difference or margin is used?