acceptodds
Under review as a conference paper at ICLR 2027

Evidence consistent with feature-specific error correction in LLMs

Abstract

Understanding the features of large language models (LLMs) is a central goal of interpretability. LLMs are commonly assumed to use superposition to represent more features than they have dimensions. They may not only represent features in superposition but also perform computation in superposition. Theory predicts that computing in superposition requires error correction that privileges feature directions over generic ones, but this prediction has not been tested empirically. We propose an empirical test of error correction in LLMs based on activation perturbations. Perturbing residual-stream activations, we find that they are robust to perturbations, forming activation plateaus consistent with error correction. They are also less robust along candidate feature directions ("pure" directions, constructed from contrastive prompt pairs) than along mixtures of two such directions, indicating that the pure directions are privileged. We quantify this privilegedness by modeling the perturbation effect as a function of the -norm of its decomposition into feature components. For the response is a quadratic form with at most as many nonzero eigenvalues as the residual-stream dimension, which cannot privilege the many feature directions superposition requires. lifts this constraint and is consistent with feature-specific error correction. We find for contrastive, MELBO, and SAE-decoder directions, and for random and PCA directions (controls). These results replicate across Gemma-2-9B, Qwen3-1.7B, Llama-3.1-8B, Mistral-7B-v0.3, Aya-Expanse-8B, and Yi-1.5-9B. We further validate our method on a toy model of error correction with known ground-truth features, recovering for true feature directions, degrading toward as we rotate away from them.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, superposition, computation in superposition, error correction, activation plateaus, feature directions, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WF4JOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper utilizes the perturbation method, rotating toward the feature direction of the residual vector in an early layer, and measures the effect via distance metrics such as L2 and cosine similarity in a later layer. The results on feature-specific direction and random direction illustrate that contrastive, MELBO, and top-activating SAE directions are privileged, consistent with superposition and error correction across six model families.

Critical strengths
  1. A direct and simple way to perturb the residual vector to dig out the feature effect
  2. This paper empirically illustrates the role of superposition in connecting multiple features to increase the robustness of the activation plateaus.
  3. Propose a sensitivity measurement of p for the feature perturbation effect.
Critical weaknesses
  1. The main concern is that the L2 distance can indicate the connection between the feature and the residual vector. However, the feature's magnitude in the final response is insufficient. In other words, the chosen feature may not have enough effect on the results.
  2. The organization of the paper is chaos, the meaning and the motivation for the methodology are unclear and make it hard to understand and to follow.
  3. The AI-written content introduces a bunch of redundant descriptions. For example, Figure 1 includes too many details.
Questions for authors

NA

3 comments

Thank you for the review.

W1: L2 distance may not reflect the feature's effect on the final response. We are not sure we understand the point raised by the reviewer, but our best guess is: L2 distance at the second-to-last layer shows that a perturbation changes internal activations, but not that it changes the model's output. If this is the case, the submitted version of the paper addresses it in two ways. First, the response-metric ablation measures KL divergence of the output logits directly, and contrastive pairs still give p > 2 (Fig. 3, "Response metric": median 2.20 [2.10, 2.36], 81% of pairs above 2). Second, Appendix E (Fig. 6) shows that steering along these directions and their combinations visibly changes the model's completions. If the reviewer meant something else, we'd be glad to address it.

W2: Organization and motivation. We are sorry to hear the paper was hard to follow. If the reviewer can point to specific sections that were unclear, we'll be happy to revise them.

W3: Redundant descriptions; Figure 1 has too much detail. We simplified the Figure 1 caption in the revision.

Thanks for your response. The W2 covers the methodology section. If you provide a clear structure, such as subsections, readers will find it easier to follow. The other questions are resolved, and I will keep my score.

Thank you for following up. We are glad to hear that the other points you have raised are addressed. Following your suggestion regarding the methodology section, we've broken it up into five subsections: 3.1 Perturbation and downstream response, 3.2 Plateau-breaking angle, 3.3 Direction types (further split into candidate feature directions and non-feature baselines), 3.4 Perturbations along two directions, and 3.5 Response model and superellipse fit. We also removed some redundant sentences in the same section. We've uploaded the revised PDF with changes marked in blue. We thank you for this suggestion and we hope the revision makes the paper easier to follow. If this addresses your concern, we would be grateful if you would consider updating your score.

Reviewer_bcHFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether LLMs show the feature-specific error correction that computation in superposition is predicted to require. The authors rotate early-layer residual-stream activations along pairs of candidate directions and record the angle at which the downstream response crosses a threshold. They then fit the resulting boundary with a superellipse of exponent pp. A purely quadratic response gives p=2p = 2, while p>2p > 2 means pure directions are more sensitive than their mixtures. Contrastive, MELBO, and SAE-decoder directions yield p≈2.2p \approx 2.2–2.42.4. PCA, random, random-difference, and shuffled-weight controls stay at p≈2p \approx 2. The contrastive result replicates across six model families and many setup ablations.

Critical strengths
  • The superellipse exponent measures whether mixtures of two directions are less disruptive than the pure directions, which separates a privileged basis from anisotropic sensitivity. This matters because sensitivity alone cannot rule out a quadratic response, and prior plateau and MELBO work did not close that gap.
  • The random-difference baseline shares the covariance of contrastive directions but has no concept pairing, and it stays at p≈2p \approx 2 along with the PCA and random baselines. This makes it unlikely that p>2p > 2 comes from activation statistics rather than semantic structure.
Critical weaknesses
  • I think that p>2p > 2 does not specifically imply superposition. Any elementwise nonlinearity would give threshold-like, p>2p > 2 responses for directions that load on few neurons, even in a network with fewer features than dimensions. The superposition-specific prediction, more than dmodeld_{\text{model}} privileged directions, is not tested (only 33 directions in d=3584d = 3584), so the framing as a “test of computation in superposition” overstates what the evidence shows.
  • The perturbations probed are much larger than the interference noise that error correction is meant to suppress: plateaus break at roughly 40° rotations, which replace most of the activation, while CiS interference noise is small by construction. Since any smooth response is locally quadratic (p→2p \to 2 as the threshold shrinks), the observed p>2p > 2 may reflect how the model reacts to large steering-like interventions rather than an error-correction mechanism acting at the noise scale. The measured nonlinearity may therefore reflect large steering interventions rather than the error-correction mechanism the paper claims to probe.
Questions for authors
  1. Do the iso-response boundaries bulge symmetrically in all four quadrants? Sweeping φ∈[π/2,π]\varphi \in [\pi/2, \pi] (i.e., +d1,−d2+d_1, -d_2 combinations) would distinguish a superellipse from a tilted ellipse with δ<0\delta < 0 on a per-pair basis, which would strengthen the main claim if p>2p > 2 holds there too.
  2. How does the fitted pp change as the threshold fraction ff is lowered well below 0.5? At what perturbation size does pp first depart from 2, and how does that compare with interference magnitudes (e.g., SAE reconstruction residual norms)?
3 comments

Thank you for the detailed and technical review, and for the concrete four-quadrant suggestion, which we ran.

W1: p > 2 doesn't specifically imply superposition. We agree that p > 2 alone doesn't establish superposition, and that our experiments don't test whether more than d_model directions are privileged. We present the result as "consistent with" FSEC, but contribution 1's phrase "test of computation in superposition" overstates it. We have reworded contribution 1, and we now say explicitly in Section 6.2 that p > 2 does not prove computation in superposition. We also tested the alternative directly: if p > 2 came from directions that load on few neurons, MLP-neuron directions should show it most strongly. We added 33 randomly chosen neurons from the first MLP after the perturbation point, each taken as its gate-projection row in residual-stream coordinates. They give median p = 2.06 [1.95, 2.17], indistinguishable from the other baselines and far below contrastive directions. Hence, a perturbation being aligned with the neuron basis is not enough to produce the effect.

W2: The perturbations are much larger than interference noise. We disagree with three parts of this argument.

  1. "40° rotations replace most of the activation." A 40° norm-matched rotation leaves the activation with cosine similarity cos 40° ≈ 0.77 to the original, so most of it is kept.
  2. "CiS interference noise is small by construction." We don't think this is known for LLMs. Interference magnitudes depend on the number of active features and on how they're packed, and measuring them needs the true feature basis, which nobody has.
  3. "p > 2 may reflect large steering-like interventions rather than error correction." These aren't mutually exclusive. Error correction may be exactly the mechanism by which the model responds to large steering-like perturbations along feature directions.

On the scale question itself, see Q2: p > 2 persists down to plateau-breaking angles of about 19°.

Q1: Do the boundaries bulge symmetrically in all four quadrants? We swept all four quadrants for all 318 pairs (new appendix):

Quadrantmedian [95% CI]
(+, +) (paper)2.30 [2.22, 2.41]
(+, −)2.03 [1.94, 2.13]
(−, +)2.26 [2.13, 2.71]
(−, −)1.94 [1.89, 2.00]

The boundaries aren't symmetric, and we don't think FSEC predicts they should be. FSEC predicts p > 2 where both directions are features, and a negated feature direction usually isn't one. For example, "English → Arabic" is a feature direction for English anchors, but its negative is not. Any quadratic response, including a tilted ellipse, is symmetric under v → −v and so predicts equal p in (+, +) and (−, −). The data break this symmetry (2.30 vs 1.94). To reflect this, we made the response model (Eq. 8) one-sided in the revision.

Q2: How does p change as f is lowered? How does that compare with interference magnitudes (e.g., SAE reconstruction residual norms)? Contrastive p stays flat at about 2.3 from f = 0.5 down to f = 0.2 (plateau-breaking angles of 42° down to 23°), and its CI still excludes 2 at f = 0.15 (about 19°). The baselines stay at about 2 at every f. We include a new appendix with the full sweep, summarised here. Angles are the median single-axis plateau-breaking angle of contrastive pairs at each f:

fanglecontrastive median [95% CI]PCArandomrandom-diff
0.5 (paper)42.3°2.30 [2.21, 2.41]1.972.022.02
0.438.6°2.34 [2.23, 2.49]1.952.052.01
0.332.2°2.32 [2.20, 2.47]1.952.051.98
0.223.4°2.28 [2.17, 2.49]2.042.061.94
0.1518.7°2.15 [2.06, 2.33]2.012.091.94
0.112.8°2.15 [1.98, 2.32]2.002.041.95

Below f = 0.15, p declines toward 2. We expect this: any smooth response is locally quadratic, so p = 2 at small enough angles.

On the comparison with interference magnitudes: we don't think SAE reconstruction residual norms measure interference. They measure what the SAE fails to reconstruct, which mixes the SAE's own approximation error with anything else it misses. They don't isolate the noise the model's computation produces. Without the true feature basis we don't know how to estimate interference magnitudes, so we can't make the comparison.

Thank you again for your review. Given that the discussion period is coming to an end, we wanted to check whether our response addresses your concerns. If anything is unclear, we'd be happy to clarify before the deadline.

Thank you authors for your additional comments. I have read your arguments and the additional added experiments in the updated paper, and I think they addresses most of my concerns about the work. Therefore, I would like to increase my rating to weak acceptance.

Reviewer_oB9oOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper measures the downstream effect of perturbations induced by injecting different types of vectors into the residual stream of an LLM. The experiments show that models are more sensitive to individual meaningful feature directions than to combinations of two such directions with comparable norm. The authors support these findings with an extensive ablation study and examine a toy model of a related phenomenon.

Critical strengths
  • The geometric formulation is clear, and the method is well defined and easy to follow. The intuition behind using the exponent pp to characterize the model’s response to individual feature directions and their mixtures is well explained.

  • The empirical findings are supported by extensive ablations across models and experimental settings, providing convincing evidence for the robustness of the observed response geometry.

Critical weaknesses
  • The connection between the observed results and computation in superposition remains indirect. Although the authors motivate their interpretation through Hänni et al. (2024), Section 6.2 acknowledges alternative explanations that the experiments do not clearly distinguish. The paper would benefit from a more explicit argument connecting the measured phenomenon to feature-specific error correction, together with controls that distinguish this interpretation from generic nonlinear processing or learned robustness.

  • The baseline with random, PCA, and random-difference directions are interesting controls. However, they may differ not only in their alignment with features but also in their relationship to the structure of realistic activations. Random directions are globally valid unit vectors but may be off-manifold or violate the covariance and coordinate structure typical of LLM activations. The random-difference baseline is intended to mitigate this concern, but because it uses randomly paired activations, it may still be off-distribution and may not preserve the local activation structure of meaningful features. Thus, differences in response could partly reflect the structure of the perturbation rather than feature-specific error correction. A control preserving local activation structure while varying feature alignment would help establish specificity.

Questions for authors
  • The results are statistical on a large corpus of features and anchor texts. Why do individual feature pairs sometimes yield p<2 ?

  • Why do the other anchor texts give a smaller exponent p in the ablation study? For "code" p is very close to 2.

  • In section 4.1, could filtering feature pairs by cosine similarity introduce a bias towards p>2 ? How do the filtered-out feature combination behave under same analysis?

  • What additional evidence distinguishes FSEC from ordinary nonlinear sensitivity or learned robustness?

6 comments

Thank you for the careful review and for the positive comments on the method and ablations.

W1: The link to computation in superposition is indirect. We agree, and the paper says so: the title and main claims say "consistent with" FSEC, and Section 6.2 lays out the alternatives we can't exclude. A direct mechanistic link would need access to the true feature basis, which we don't have for any LLM. What we can do is rule out specific alternatives. In the revision we added a control for generic nonlinear processing: directions aligned with MLP neurons, where the elementwise nonlinearity acts. Random neurons give p ≈ 2 (median 2.06 [1.95, 2.17]), so a privileged basis from the nonlinearity alone doesn't produce what we see for contrastive directions (median 2.30 [2.22, 2.41]). On learned robustness, see Q4.

W2: The baselines may be off-manifold. The random-difference baseline is built exactly like the contrastive directions (Eq. 4): it is the mean difference of 30 pairs of real FineWeb-Edu activations. The only change is that the two prompts in each pair share no concept. So it shares the activation distribution and covariance of the contrastive directions. We don't see a way in which it would be more off-distribution than they are. Reviewer 2 reads this baseline the same way: it "shares the covariance of contrastive directions but has no concept pairing". The perturbation is also applied the same way for every family: a norm-matched rotation of a real activation, so the perturbed activation always keeps its norm. If the reviewer has a specific control in mind that preserves local activation structure further while varying feature alignment, we'd be glad to run it.

Q1: Why do some individual pairs give p < 2? Our claims are about populations of directions, not individual pairs. We don't have ground-truth features in LLMs, so contrastive, MELBO, and SAE directions are best guesses. We expect them to be better aligned with true features than random directions on average, not each one to be well aligned. Two results show that misalignment pulls p toward 2. In the toy model, where the features are known, ground-truth directions give median p = 3–3.4, and p falls monotonically to 2 as directions are rotated away from them (Section 5). The same holds in the LLM when we rotate contrastive directions toward random ones (Appendix F). Moderate misalignment (cosine ≈ 0.8 with the true feature) already brings the toy-model p down to 2.3–2.6, so single pairs below 2 are what we'd expect from imperfect candidate directions.

Q2: Why do other anchor sources give smaller p? For code p is close to 2. Per-source statistics (new appendix):

Anchor sourcemedian [95% CI]mean [95% CI]pairs with p > 2
FineWeb-Edu (default)2.30 [2.22, 2.41]2.42 [2.29, 2.62]88%
wikipedia-en2.15 [2.08, 2.26]2.25 [2.14, 2.38]73%
wikipedia-zh2.22 [2.08, 2.32]2.25 [2.13, 2.39]75%
the-stack-smol (code)2.11 [1.98, 2.22]2.24 [2.11, 2.38]64%

All four mean CIs exclude 2. The effect is weaker on code: there the median CI includes 2, and fewer pairs exceed 2. A possible, untested reason: most of our directions are natural-language concepts, which code contexts may not use.

Q3: Could the cosine filter bias results toward p > 2? What do the filtered-out pairs give? The filter drops the 210 of 528 pairs whose directions have |cos| ≥ 0.1. Orthogonalizing such a pair changes the second direction substantially, so the perturbation no longer points along the labelled feature. Fitting those pairs (new appendix):

  • Kept (318): median 2.30 [2.22, 2.41], mean 2.42 [2.29, 2.62], 88% above 2.
  • Dropped (210): median 2.07 [1.79, 3.10], mean 2.56 [2.27, 2.77], 51% above 2.

The two distributions aren't significantly different (two-sided Mann–Whitney p = 0.16). The dropped pairs fall into two groups. Pairs with |cos| between 0.2 and 0.5, mostly natural-language × programming-language pairs, give p ≈ 3.5. Pairs with |cos| > 0.5, almost all within one family (two languages, or two programming languages), give p ≈ 1.5. For |cos| > 0.5, orthogonalization makes the second direction very different from its feature direction, which could explain the lower p (see Q1).

Q4: What distinguishes FSEC from ordinary nonlinear sensitivity or learned robustness? We don't see FSEC and learned robustness as competing explanations. Error correction is one mechanism through which learned robustness could arise, so evidence for one isn't evidence against the other. Section 6.2 also notes that our main result, p > 2 for candidate features and ≈ 2 for baselines, doesn't depend on reading plateaus as error correction. For ordinary nonlinear sensitivity, the most concrete version is a privileged basis created by the elementwise MLP nonlinearity. The new MLP-neuron baseline tests this directly (see W1): random neurons give p ≈ 2.

Thank you again for your review. Given that the discussion period is coming to an end, we wanted to check whether our response addresses your concerns. If anything is unclear, we'd be happy to clarify before the deadline.

Thank you for your response to my concerns. There are some points still open to me:

  • The random diff takes points from the same distribution (the global distribution) so the difference is around 0. The diff makes sense if the two distribution are different, because the diff represent that separation. Using that could give a random feature vector that makes more sense, but the fact that we don't know what the feature describes doesn't mean it has no meaning.
  • For Q1, if I unterstand, you would justify the p<2 on individual samples by the misalignment of the feature directions ?
  • Intuition on Q2 sounds fair.
  • As I understand, the filtering makes the p>2 proportion higher, and you happen to filter out only the feature that are corresponding to human understandable concepts compositions, i.e, the concept that would validate your hypothesis if were p>2. Therefore I would consider the filtering process as a bias that needs to be ruled out with a baseline that can also gets a proper filtering and interpretable concepts. Also no sure if that this bias is problematic for the main takeaway.
  • Response to Q4: good point.

Comments on the paper

1 comment

We thank the reviewers for their time. We have uploaded a revised PDF, with all changes highlighted in blue. The changes are:

  1. MLP-neuron baseline (new column in Fig. 2). We added 33 randomly selected neurons from the first MLP after the perturbation point as a direction family. Each direction is the neuron's gate-projection row expressed in residual-stream coordinates, which is where the elementwise nonlinearity acts. These give median p = 2.06 [1.95, 2.17], mean 2.09 [2.00, 2.19], so they sit with the non-feature baselines and well below contrastive directions.
  2. Four-quadrant sweeps (new appendix). For all 318 contrastive pairs we swept the three remaining sign combinations (+d1, −d2), (−d1, +d2), and (−d1, −d2).
  3. Threshold sweep (new appendix). We refit p at threshold fractions f ∈ {0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5}.
  4. Filtered-out pairs and per-anchor-source statistics (new appendix).
  5. Framing. We reworded contribution 1 so it no longer presents FSEC as a test of computation in superposition, and Section 6.2 now states explicitly that p > 2 does not prove it.
  6. Response model. Eq. 8 now uses one-sided feature terms, max(0, f̂ᵢᵀv)ᵖ, since we assume features are one-sided. Nothing in the derivation or results changes, since our sweeps stay within the quadrant where both projections are nonnegative.
  7. Figure 1. We simplified the caption.
  8. Methodology section. Following a suggestion by one of the reviewers, we split the Methodology section into subsections and removed some redundant sentences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.