acceptodds
Under review as a conference paper at ICLR 2027

Same Features, Different Geometry: Localizing Muon's Advantage to the OV Circuit

Abstract

Does the optimizer change what a language model learns, or mainly how it stores and reaches it? We ask this at the level of top-k sparse-autoencoder (SAE) features: we compare Muon and AdamW on a 124M GPT-2-class model, matching SAE features across independently trained networks by firing-pattern correlation and calibrating against within-optimizer seed baselines, so every feature claim here concerns the firing-pattern content of one SAE family at one or two layers, not features in general. Under this protocol, cross-optimizer feature agreement (0.599) meets the AdamW seed ceiling (0.594), while Muon is more seed-reproducible (0.618 vs. 0.594) and changes weight geometry, a gap that is fully formed by one tenth of training, holds at three, six, and twelve heads, and narrows to the seed spread under a GeLU MLP; the loss advantage itself replicates from 30M to 1.3B parameters. Reassigning Muon between sub-circuits of a fixed architecture gives a double dissociation: Muon on every hidden matrix except the value circuit WV WO recovers none of the loss gain, Muon on the MLP blocks recovers none, and Muon on the query/key scoring circuit WQW ⊤K raises stable rank to near-full-Muon levels yet recovers neither loss nor reproducibility, whereas Muon on the value circuit alone recovers most of both, a sufficiency result that replicates a concurrent block- level ablation under a tuned baseline; zero-shot HellaSwag accuracy follows the same split. A rank-promoting AdamW control rules out endpoint stable rank as a sufficient knob, and varying only the fidelity of the orthogonalization step shows loss falling monotonically with fidelity while reproducibility saturates much earlier; a second matrix-aware optimizer, SOAP, reproduces the same signature. The opti- mizer invariance holds at three, six, and twelve heads, and matching features across layers shows that the metric moves when the compared features differ. In this bench, Muon’s useful interaction with attention runs through the value/output pathway and through orthogonalization fidelity, not through the scoring-rank fingerprint that observational analyses highlight.

Primary area:
interpretability and explainable AI
Keywords:
optimizers, sparese-auto-encoders, transformers, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p5AvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether Muon’s advantage over AdamW arises from different learned features and where that advantage acts within a transformer. At matched validation loss, the authors compare top-k SAE features by firing-pattern correlation, calibrated against within-optimizer model-seed baselines. Cross-optimizer agreement is close to the within-AdamW baseline, while Muon has higher cross-seed agreement and substantially different weight spectra and representation geometry.

Targeted optimizer reassignment shows that Muon on the attention value/output pathway alone recovers approximately 86% of the loss advantage. Muon on QK, MLP blocks, or all hidden matrices except OV recovers none. QK and OV rank controls show that endpoint stable rank is not sufficient for the benefit. A Newton–Schulz iteration sweep supports the importance of the update process, although orthogonalization fidelity and update norm are not fully separated.

Critical strengths
  1. The targeted interventions provide stronger evidence than observational spectral analysis alone. Muon on OV recovers most of the loss advantage, while Muon on the remaining hidden matrices does not, providing complementary localization evidence. Raising QK rank without gains and promoting OV rank without improvement further show that endpoint stable rank is not a sufficient explanation. These experiments are the main contribution supporting acceptance: OV sufficiency extends prior work, while the complement arm and rank controls add useful evidence.

  2. The feature comparison includes carefully designed baselines and measurement controls. The authors use matched-loss checkpoints, within-optimizer model-seed baselines, and different SAE seeds on the same model, and treat model seeds rather than numerous features as the inferential unit. Checks involving live-feature sets, depth, head count, and activation functions further expose the boundaries of the findings and strengthen the credibility of the comparison.

Critical weaknesses
  1. The strength of the mechanistic localization remains conditional on the hybrid optimizer protocol, and the fidelity attribution is confounded by update norm. Hybrid arms share a warmup schedule and are not jointly retuned. Existing learning-rate sweeps reduce some concerns, but OV necessity should remain explicitly conditional on the tested protocol. Separately, changing the number of Newton–Schulz iterations changes both orthogonalization fidelity and update norm, so the monotone loss curve does not independently identify fidelity as the driver. This limits the specific mechanistic interpretation without negating the OV localization result.

  2. The same-features and reproducibility claims depend on the instrument and architecture. The comparison uses one top-k SAE family, a small number of layers, and best firing-pattern correlation, which does not establish general feature-content identity. Some hybrid reproducibility rankings change with live-feature selection; the 350M OV reproducibility gain is cut-dependent, and the GeLU gap narrows to the seed spread. Loss localization, feature reproducibility, and geometry changes should therefore be stated separately rather than summarized as one universal mechanism.

Questions for authors

The following clarifications could affect my final assessment:

  1. To what extent do the existing experiments rule out learning-rate and warmup interactions in the OV necessity claim? Are configuration checks available for the complement arm? If evidence is limited, please explicitly restrict necessity to the current protocol.

  2. Is a norm-matched Newton–Schulz control available to distinguish orthogonalization fidelity from effective step size? If not, please narrow the causal interpretation of the fidelity sweep accordingly.

  3. Please summarize which feature conclusions survive matching live-feature counts or frequency cuts and which depend on the original SAE live sets, especially the 350M OV results. Does cross-optimizer agreement near the AdamW seed baseline support the stated non-inferiority claim, and what allowable difference or margin is used?

Reviewer_KZBkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether Muon changes the features learned by language models and which Transformer components account for its optimization advantage. Using top-k sparse autoencoders, the authors compare feature agreement across optimizers and random seeds and examine how feature reproducibility is affected by training stage, model scale, architecture, activation geometry, weight decay, noise, and orthogonalization fidelity.

Through component-wise optimizer reassignment, the authors find that applying Muon only to the value/output pathway recovers most of the full-Muon loss advantage, whereas applying it to other components or to every hidden matrix except WVW_V and WOW_O recovers none. Additional experiments show that increasing endpoint stable rank alone does not reproduce the loss improvement.

Critical strengths
  1. The component-wise optimizer interventions provide strong evidence for the importance of the OV pathway. Applying Muon only to WV,WOW_V,W_O recovers most of the loss advantage, while applying it to every hidden matrix except WV,WOW_V,W_O recovers none.

  2. The authors use targeted controls to distinguish geometric changes from their effects on performance, showing that endpoint stable rank alone cannot explain Muon's loss advantage.

  3. The experiments cover different model architectures and scales. The SAE analysis includes careful controls, such as within-optimizer seed baselines and separate SAE-seed ceilings.

  4. The paper reports negative and inconclusive results and acknowledges the limitations of its SAE analyses.

Critical weaknesses
  1. The two main contributions remain conceptually disconnected. The SAE analysis suggests that Muon and AdamW learn similar firing-pattern features, with Muon producing greater cross-seed reproducibility, while the optimizer-reassignment experiments localize the loss advantage to WV,WOW_V,W_O. Neither analysis explains the other. More importantly, several experiments show that feature reproducibility does not track the loss improvement. Its functional significance remains unclear without evidence linking it to more stable causal features, better generalization, or more reliable interpretability. Currently, it appears to be a descriptive signature of Muon rather than an explanation for its performance advantage.

  2. The firing-pattern matching score has two limitations that may overstate agreement between feature dictionaries. First, it is sensitive to the number of candidate live features. For independent and identically distributed candidate correlations with cumulative distribution function FF, the maximum correlation follows Pr⁡(max⁡j≤nρj≤x)=F(x)n\Pr(\max_{j\leq n}\rho_j\leq x)=F(x)^n. Thus, larger candidate sets can produce higher matching scores even when correlations are dominated by noise. Symmetrizing the score does not correct this effect, and live-set matching controls candidate-set size without eliminating the underlying maximum-selection bias.

    Second, the metric allows many-to-one matching, so different features can select the same counterpart. This is particularly concerning because Muon has a lower same-model SAE-seed ceiling and higher decoder nearest-neighbor cosine similarity, suggesting greater feature crowding under the current SAE recipe. These issues make the score more suitable for measuring nearest-neighbor coverage than one-to-one feature correspondence. A maximum-weight perfect bipartite matching score on fixed-size top-NN live-feature sets, max⁡π1N∑i=1Nρ(ziA,zπ(i)B)\max_{\pi}\frac{1}{N}\sum_{i=1}^{N}\rho(z_i^A,z_{\pi(i)}^B), would provide a useful comparison. A jointly trained Crosscoder could offer an additional check.

Questions for authors
  1. The same-model SAE-seed ceiling is lower for Muon than AdamW, and Muon SAEs have higher decoder nearest-neighbor cosine similarity. This suggests greater feature crowding and ambiguity under the current SAE recipe. Moreover, the bidirectional nearest-neighbor matching score allows multiple features to share the same counterpart and is sensitive to the number of candidate features. Although live-set matching addresses the latter issue, it does not resolve many-to-one matching. Could the authors report a maximum-weight bipartite matching score on fixed-size top-N live-feature sets, or consider using a jointly trained Crosscoder to validate the feature agreement?

  2. Several results suggest that feature reproducibility does not explain Muon's loss advantage. What is the functional significance of higher feature reproducibility?

  3. How is the "fraction of clean firings preserved" defined in the activation-noise experiment?

  4. Why does the fixed SAE recipe produce 21–54% dead features on SOAP models? Is this caused by differences in activation distributions or unsuitable SAE hyperparameters?

My main concern is the second question, interpretation and significance of the feature reproducibility analysis. If the authors address this concern convincingly, I would consider raising my score. However, without a clearer explanation of its significance, I do not think the paper is complete.

Reviewer_zhKCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether models trained with different optimizers learn the same features, by comparing SAEs trained on GPT-2 like models trained with Muon and Adam. They vary which parts of the model are optimized with Muon and observe that Muon optimized models tend to share more features than Adam optimized models.

Critical strengths

The project idea is interesting, comparing LLMs trained with different optmizers and understanding if they end up learning the same concepts.

Critical weaknesses

The SAEs presented for analysis are severely undertrained at only 50M tokens.

'Cross-model matching and inference. Decoder directions are not comparable across independently trained models: even same-recipe seed pairs share no aligned decoder vectors (mean best cosine 0.14), as expected from permutation symmetry' fails to account for the fact that you could potentially match the decoder, for instance with the Hungarian algorithm, as done in the literature but the authors fail to interact with: 'Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs', Song et al. (2025), 'Sparse Autoencoders Trained on the Same Data Learn Different Features', Paulo and Belrose (2025) to name a few. This makes the first contribution much weaker, as it is not compared with other matching methods in the literature. In fact it does look like the authors cite at least one of these works (but have a corrupted citation), so this seems even worse. I agree that comparison across models, specially trained with different optimizers, have no reason to share the same decoder rows.

The paper is also confusing as written. It is shown as a comparison between features learned, but a large section of the paper is spent discussing how different interventions change the loss of the underlying model. I dont quite understand what the takeway should be. The authors talk about the value pathway, but it is unclear what the authors want us to learn about this.

Questions for authors

. The methods mention training only on layer 6 but table 1 shows multiple layers, are all of these retrained?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.