acceptodds
Under review as a conference paper at ICLR 2027

Norm Share Predicts Whether a Subspace Ablation Removes the Feature or the Model

Abstract

Feature ablation is used to isolate what an internal representation contributes to a language model. We show that two operations commonly called ablation produce widely different effects on the same selected features, ranging from language-specific control to broad model failure. For sparse-autoencoder features, latent zeroing subtracts the selected latents' contributions and preserves reconstruction error while directional projection removes every residual-stream component along their decoder directions at every token. Because a language-specific feature's decoder span can carry shared computation, projection can cause broad model failure. We predict this failure using norm share, ρ, the average fraction of a token's squared residual-stream norm in the ablated subspace. This requires one forward pass without labels, generation or intervention. All 52 measured subspaces with ρ above one half raise the other languages' cross-entropy by at least 6 nats. In language-identity tests across eleven languages of parallel text, Gemma 3's mid-network subspaces carry most of a typical token's norm. At layer 17, projection raises target cross-entropy by 7–31 nats (up to several times baseline), with similarly large increases on the other ten languages. Zeroing the same latents concentrates changes on the target and switches output language, with no switches under matched random-feature controls. The same subspace construction at the Llama 3.1 site has low ρ and selective projection, consistent with the decline in projection selectivity as ρ rises across measured subspaces in both families. Manipulating the dominant shared directions changes this outcome: removing them from Gemma's basis makes projection selective, and adding them to Llama's basis lowers selectivity. Adding a random direction of the same rank leaves projection selectivity unchanged. The diagnostic also separates destructive from harmless refusal interventions. We then use selective language removal to test whether language-specific representation interferes with multilingual reasoning. Across three model families and four removal constructions, no setting significantly raises pooled MGSM accuracy. The gains reported by Zhao et al. (2025) are per-language best points of a grid searched on the evaluated problems. Applying this rule to our reproduction, where no setting raises pooled accuracy, gives +2.1 points in-sample and 0.0 when selection and scoring use different halves of the problems. Subspace interventions should therefore report ρ and collateral effects before attributing behavioural changes to removal of the targeted property.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, feature ablation, mechanistic interpretability, activation steering, subspace projection, multilingual reasoning, language models, massive activations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZU7HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work compares two common ways of removing a feature inside an LLM (i.e., projection and SAE zeroing). Projection deletes the feature’s directions from every token’s hidden state, while SAE zeroing subtracts only the selected SAE features’ contribution where they fire. The authors propose norm share (i.e., the fraction of a token’s hidden-state norm that lies in the removed directions) to check when projection breaks the model. They evaluate it on language features in eleven languages of parallel text (i.e., FLORES-200 and MGSM) and on refusal directions. They use Gemma 3 (4B and 12B), Llama 3.1 8B, and Qwen-2.5 (3B and 7B) models. The authors report that projection breaks the model when norm share is high (e.g., Gemma 3’s middle layers), while SAE zeroing changes only the target language. They also report that removing language features does not improve multilingual reasoning on MGSM, contrary to previous work.

Critical strengths

This work provides a cheap check (i.e., one forward pass with no labels) that explains why projection and SAE zeroing can disagree on the same features. In Gemma, removing the shared directions from the subspace makes projection affect mostly the target language. In Llama, adding these directions makes projection affect the other languages more, while adding a random direction leaves it unchanged. Prior work used one method or the other and drew opposite conclusions (e.g., Deng et al., 2025; Zhao et al., 2025). The findings from this paper may guide ablation and steering studies to check for collateral damage before attributing an effect to the removed feature.

Critical weaknesses

W1: The key idea is not new. Norm share is the standard fraction of a token’s norm in a subspace. It is also known that zeroing a few very large shared activations (i.e., massive activations) breaks a model (e.g., Sun et al., 2024). The erasure methods that spare these directions already exist (e.g., LEACE). A related measure of steering damage, based on the same activation statistics, was also proposed by Nguyen et al. (2026), cited by the authors. The authors should clearly articulate their novelty and differences from existing works.

W2: The evidence that norm share predicts the outcome is limited. Most of it comes from two layers in two models (i.e., Gemma’s middle layers, where projection fails, and Llama’s layer 19, where it works). It seems that no single threshold works across properties (e.g., refusal directions do damage well below a norm share of one half). Does norm share predict the outcome for other properties (e.g., topic or sentiment) and other model families?

W3: The paper is hard to read and poorly presents the ideas. Many sections list numbers (e.g., nats, correlations, and p-values) without discussing what each result means for the main claims. Moreover, the paper mixes three topics (i.e., the norm-share check, refusal directions, and multilingual reasoning). Another presentation issue is that the rebuttal of Zhao et al. (2025) is listed as a main contribution, but it is a separate and limited study. It is only loosely connected to norm share.

Questions for authors

See weakness section.

Reviewer_xsGLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper compares two feature-ablation methods in language models: projecting out sparse autoencoder (SAE) decoder directions and zeroing the corresponding SAE latents. The authors propose norm share (ρ), the fraction of residual-stream energy contained in the removed subspace, as a simple diagnostic for collateral damage from projection. Experiments on Gemma 3 and Llama 3.1, with additional Qwen and refusal-intervention tests, show that high norm share is associated with less selective removal. Modifying dominant shared directions further supports this relationship, while SAE-native removal often produces more targeted language effects. Finally, the authors find no significant pooled multilingual-reasoning gains from language-feature removal and question the evaluation procedure behind previously reported improvements.

Critical strengths
  1. The paper includes a substantial experimental study across model families, layers, dictionaries, and intervention methods. In particular, the comparison of directional projection and SAE-native removal on the same selected features is informative.
  2. The experiments that remove or add dominant shared directions help explain why a projection may cause broad performance degradation rather than selectively affect the targeted language.
Critical weaknesses
  1. Limited methodological novelty and unclear scope of the diagnostic. The proposed norm share measures how much of the residual-stream squared norm lies in the subspace being removed. While this is a useful quantity, the underlying geometric intuition seems fairly straightforward. I would like the authors to explain more clearly what is new here beyond existing work on activation geometry and subspace interventions. The experiments show a strong association between norm share and collateral damage, but the paper also reports low-norm-share directions that still produce substantial damage. It is therefore not clear to me how reliable norm share is as a predictor of selectivity, rather than as a warning sign for especially destructive interventions. The correlation also appears weaker within individual site-and-rank settings, which raises questions about generalization.
  2. Presentation and motivation. a. I found the introduction harder to follow than necessary. The distinction between the two ablation operators is important, but the precise gap in the existing literature and the added value of the proposed diagnostic should be stated earlier and more directly. b. The abstract is overloaded with numerical results and separate findings, making the main contribution difficult to identify on a first reading. Some passages also use vague references (for example, “We establish this on language identity”), which interrupt the argument. c. Finally, broad claims such as “Removing a representation is a common way to test its contribution” would benefit from citations near the point where they are introduced.
Questions for authors
  1. What does norm share reveal that could not be inferred from existing geometric or activation-based measures? A more explicit comparison would help me assess the methodological novelty.
  2. How well does norm share predict selectivity within a fixed model, layer, and subspace rank, rather than mainly separating high- and low-norm-share regimes across different sites?
  3. Given the examples where low norm share still leads to large collateral effects, what are the practical limits of using it before an intervention? Would the authors recommend an additional diagnostic? Also see the “Critical Weaknesses” section.
Reviewer_W58LOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The core contribution is a one-pass, label-free diagnostic, norm share (rho), that predicts whether projecting a subspace out of the residual stream will remove a feature or wreck the model, together with a clear separation of two operations that the literature has been conflating under the word "ablation." Directional projection removes every component along the selected decoder directions at every token. Latent zeroing subtracts only what the selected latents contribute, where they fire.

The main claims are four.

  • First, projection selectivity falls with rho: across 102 projection subspaces, all 52 with rho above 0.5 raise cross-entropy on the other languages by at least 6 nats.
  • Second, the relation is causal within a model: holding model, site, and features fixed and changing only the basis moves selectivity together with rho, while a random direction of the same rank does nothing.
  • Third, the same quantity predicts collateral damage from the Arditi et al. refusal direction.
  • Fourth, clean removal of language features gives no reasoning gain on MGSM in any of three model families and four constructions; the gains reported by Zhao et al. (NeurIPS 2025) come from a per-language best-point rule applied to the evaluated problems, and vanish under a half-split.

The evidence is Gemma 3 4B and 12B, Llama 3.1 8B, and Qwen 2.5 3B and 7B; within-model basis manipulations with rank-matched controls; ten SAE dictionaries; a LEACE comparison; a full re-run of the Zhao et al. pipeline at two edit sites; and a supplement in which every reported number recomputes from the result files.

The conclusion is that subspace interventions should report rho and collateral effects before attributing a behavior change to the removed property, and that language features are real but removing them does not help reasoning.

Critical strengths

The within-model manipulation is what convinces me, and it is what makes rho usable rather than just correlated. A rho-selectivity correlation across 53 cells could be a site confound. But holding model, layer, and features fixed and changing only the basis shows selectivity tracking rho directly: on Gemma rho goes from 0.68 to 0.00 and the concentration index C goes from +0.01 to +0.79; adding top directions on Llama drops C from +0.82 to +0.40.

The controls are careful. Adding a random direction of the same rank leaves C at +0.82. Adding only the second to fourth directions, which barely changes the first massive-activation token's contribution, still lowers C. That rules out "rank" and "one massive direction" as alternative explanations. For anyone deploying steering or ablation as a model-editing tool, in safety, language control, or concept removal, this gives a cheap pre-check that distinguishes editing a feature from breaking the model.

Reproducibility is exceptional. The results directory contains the per-cell result files behind every table, and the verification script recomputes each reported number from them. I ran it: 625 checks, all passing. This covers Tables 1, 2, 9, 11, 13, and 15, the rho correlations, refusal, LEACE, the Zhao replication, and the selection-bias analysis. The MGSM baselines in the supplement match Table 2 exactly (en 0.600, es 0.604, sw 0.388). REPRODUCE.md gives the commands for every table, and the preregistration document is included.

The replication of Zhao et al. is also done carefully. The authors noticed that the public code edits the MLP output while the paper describes the residual stream, and ran both. They traced the reported gains to a per-language best-point selection rule in the released analysis script. Showing that this rule manufactures +2.1 pp from a null grid, and that it vanishes under a half-split, is a strong result even though it is a negative one. It saves others from building on the interference account.

Critical weaknesses

Rho is a useful warning signal but not a safety certificate. Low-rho subspaces can still be destructive: a mean-difference direction with rho below 0.03 costs 22 nats (Table 18). The regime boundary also differs between language (above 0.5) and refusal (at or above 0.17). The evidence is concentrated on language identity and refusal in two model families.

More importantly, the "repaired" projection is selective but its target effect collapses from 21 to 0.7 nats (Table 13). That reads less like fixing projection and more like showing that projection has very little it can do once the shared directions are removed. The abstract and conclusion say "repairs the intervention," and that should reflect the trade-off.

The refutation of the interference account should be read within its scope: these models, these settings, MGSM only. The replication baseline is 3.0 points below the original (Appendix B.10, concentrated in English, Spanish, and French), and the generation limits differ a lot (384 or 1,024 tokens versus 16,384).

The supplement records how many baseline generations were truncated at 1,024 tokens in the code-faithful run: Swahili 78 of 250, Bengali 30 of 250, Telugu 77 of 250, versus 0 or 1 for English, Spanish, and French. So truncation does not explain the gap on the high-resource languages. It does mean the Qwen results for Swahili, Bengali, and Telugu (baselines 6.4% and 8.4%) are largely truncation artifacts and should not be interpreted. The 384-token residual-stream runs (Table 15) have no truncation accounting at all. None of this is reported in the paper.

Questions for authors
  1. On the supplement, four items. (a) Please report the truncation counts from the code-faithful run in Appendix B.10, and add truncation rates for the 384-token residual-stream runs and for the Gemma and Llama accuracy runs in Tables 2 and 10. (b) The preregistration's ordering is supposed to be established by git history, which the anonymized archive does not contain. How will you provide hashes or timestamps at release? (c) The project file lists sae-lens, nnsight, and dictionary_learning as dependencies, while REPRODUCE.md says Gemma Scope 2 is loaded from safetensors without SAELens. Which reflects the actual environment? (d) The results directory contains a file named mechanism_BUGGY.json with implausible values (norm deltas around ten million). Please confirm no reported result depends on it.

  2. How should a practitioner handle the low-rho exceptions? Why does the regime boundary differ between language and refusal: property, site, or rank? Would a per-token distribution or a per-layer check separate the cases? What minimal checklist would you recommend given current results?

  3. Table 13 shows the repaired projection's target effect dropping to 0.7 nats, yet Table 22 shows it still switching output language for five of six languages. How does 0.7 nats produce a language switch, and how do the switch rates compare to latent zeroing?

  4. Of the gap to Zhao et al.'s numbers, how much is attributable to selection on the evaluated problems versus harness differences (edit site, prompt, token limit)? How did you rule out the 384-token limit as a cause of the 3.0-point baseline gap?

  5. Rho is measured on FLORES tokens but the interventions are applied to MGSM. How different is rho on the MGSM distribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.