acceptodds
Under review as a conference paper at ICLR 2027

Language Model Activations Inhabit Privileged Error-Correcting Basins

Abstract

Language models exhibit remarkable robustness, continuing to produce coherent text even when their activations are perturbed by interventions like linear steering. We hypothesize that this robustness is a result of passive dynamics, i.e., constraining mechanisms in the forward pass that funnel activations toward “good” regions that produce coherent outputs, similar to how biomechanical constraints allow living organisms to produce natural movement with minimal effort. To understand such constraining mechanisms in language models, we probe the geometry of their activation space by observing the action of model layers on low-dimensional curves. In doing so, we discover that model activations occur within a cluster of distinct attracting basins, which differentiate natural activations geometrically from distributionally similar synthetic activations. Applying this lens to language model steering, we observe feature-specific basins along semantic steering directions, and find that steering moves activations between these basins. To demonstrate the active role of this geometry in neural computation, we show that adaptively modulating steering strength to transport activations across basins improves inter-language steering, significantly increasing the probability of sampling tokens from the target language compared to fixed-strength steering. Our findings establish analysis of activation space geometry as a promising approach to interpreting language models.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, mechanistic interpretability, language model, steering, language

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_eiXbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the geometry of language model activation spaces. The authors argue that activations arising from natural text lie in privileged attracting basins: regions in which transformer-layer dynamics tend to funnel perturbed or synthetic activations toward more “natural” configurations. They investigate this hypothesis using norm-growth and speed profiles along local spherical probes. They then apply this interpretation to activation steering: refusal/compliance and language-switching directions are argued to correspond to distinct basins, and selecting a steering strength that places an activation in the target basin improves model control.

Critical strengths
  • The paper puts forward a conceptually compelling hypothesis and connects it well to previous findings on steering and error correction.
  • It compares several model families and several types of synthetic controls.
  • The English-to-Chinese steering experiment includes both in-distribution and out-of-distribution evaluation.
  • The visualizations are particularly useful for connecting a steering direction, geometric regions in activation space, and output behavior.
  • The authors explicitly acknowledge the exploratory nature of the work and several open limitations, including the case of very nearby activations and the possibility of a low-dimensional manifold structure.
Critical weaknesses
  • The paper does not cite work on anisotropy in LLMs, such as [1-4]. Yet, at first glance, its geometric analysis and its notion of “attracting basins” strongly resemble anisotropy-related work. I had to examine the paper carefully to understand the distinction. A related-work discussion that explicitly distinguishes these notions would make the paper’s contributions clearer. I am not claiming that the two are identical; rather, such a discussion would help readers familiar with anisotropy understand more quickly what is genuinely different here.
  • The causal link between the observed geometry and the model’s overall robustness remains only partially established. The gains from adaptive steering support the hypothesis, but do not rule out alternative explanations.
  • The refusal result is essentially illustrative: it relies on one pair of prompts and a visualization, rather than on a broader benchmark measuring reliability, safety, and preservation of utility.

[1] Ethayarajh "How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embedding" EMNLP 2019

[2] Timkey et al. "All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality" EMNLP 2021

[3] Rudman et al. "Stable Anisotropic Regularization" ICLR 2024

[4] Bernas et al. "Revisiting Anisotropy in Language Transformers: The geometry of learning dynamics" ICML 2026

Note: I have included only 4 citations here to provide an overview of the field, but the field is quite rich; I recommend that the authors read the literature carefully.

Questions for authors
  • The paper should be more clearly situated within the existing literature, so that readers can identify its actual contributions earlier, without having to wonder whether the central observations have already been established or whether the claims are overstated.
  • The computational cost of finding, for each activation, the speed minimum along the steering direction is not discussed in much detail. I would be interested in knowing how expensive this procedure is in practice, and how it scales with the number of activations and candidate steering directions.
2 comments

Thank you for your thoughtful review. Your engagement and comments are very helpful.

Text in the uploaded revision that addresses reviewer concerns is highlighted in blue.

Anisotropy citations

Thank you for pointing out these citations. We will add them to our revision and relate our work to them (revised: L070, L143, L203).

Causal link between observed geometry and robustness.

Your observation is correct, but this is a necessity rather than an oversight. We did try to isolate a specific mechanism during our exploration, but found (in agreement with Shinkle et al) that the basins are implemented by a large and distributed set of MLP neurons. As a result, we do not have an isolated model component that we could realistically ablate to establish causality. We therefore must leave it to future work to isolate basin-implementing components (if it is indeed possible) and establish their causal role.

Instead of a single mechanism within the model, we think it more accurate to conceptualize our work as a set of observations about the model's activation space geometry that enable us to better understand the dynamics at play within the model and design better interventions.

We will be sure to address this in the revision (revised: L525).

Refusal result illustrative

We agree that the refusal result is illustrative, and we intended for it to be so. We presented the case study as a motivating example for our controlled language steering experiment.

Questions

Anisotropy

To situate our work within the anisotropy literature, our work analyzes the geometric sub-structure within the anisotropic region occupied by natural activations.
Our Gaussian and shuffled synthetic activations inherit the anisotropy of the natural activations, allowing us to control the generic structure of the activation space, i.e., norm growth and velocity basins, and identify sub-structure, i.e., local norm-growth maxima and velocity barriers between tokens. We will make this clear in our revision (revised: L070, L143, L203).

Computational cost

Our steering method is indeed more expensive than plain fixed-strength steering by O(n)O(n) where nn is the number of strengths tested per context. Our implementation evaluates a range of strengths all at once, so a granularity of 50 strength values means 50 additional parallel forward passes through the 4-layer block. In our experiments, we used a much finer grid of about 500 points per context, but, in retrospect, Figure 12 shows that this level of density is gross overkill, since the velocity profiles are quite smooth. We used finite differences to evaluate the velocity at each point, rather than rely on automatic differentiation, which saves compute. In total, the extra passes through a few middle layers is not prohibitively burdensome, though it is not free. The number of forward passes might be reduced using a zero-finding method like Newton's, at the expense of parallelizability. We did not prioritize optimizing the method, since the goal of the experiments was to demonstrate the role of basins in model behavior, not to recommend a more efficient steering method.

Ok thanks for your reply. That seems good for me, I'll keep my accept.

Reviewer_D46BOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The authors pose the existence of an error-correction mechanism in language models. They claim that this is an explanation for why steering interventions often lead to coherent generations. The mechanization consists of regions in the representation space they call "error-correction basins" where all representations (synthetic or natural) are eventually pushed.

To conduct the analysis in the paper they introduce a novel probing methodology they call spherical probes. A spherical probe is a path defined by an activation xx and some direction u^\hat{u}, it is a circle centered at the origin and passes through xx with the tangent u^\hat{u}. A lot of the analysis is based on how the layers of a model deform this circle.

With their findings, they claim that the effect of a block on the probe circle's geometry is different for natural activations when compared to the synthetic activations.

Finally, they describe the effect of steering interventions in their framework as moving from one basin to another. Using this phrasing, they design an adaptive steering method and show results steering from english to chinese tokens.

Critical strengths
  1. The paper introduces "circular probes", a novel way of studying representational geometry.
  2. The paper provides a reasonable setup to study error correction in language models i.e. the synthetic activations that preserve larger data statistic.
Critical weaknesses
  1. Crucial definitions have not been provided. It is impossible to truly understand and contextualize the findings.

    a. It is unclear what a "basin" refers to with respect to the paper. The authors should specifically define the term and state which variable the contraction mapping is valid for.

    b. The authors need to specify how the aggregate norm-growth metric for activations were calculated. Additionally, prima facie it would seem that the choice of u^\hat{u} can affect the conclusions because different circles in the high-dimensional space can potentially be deformed different, even for a fixed activation xx. Does the analysis average over several u^\hat{u}?

    c. The term "velocity barrier" is referred to as a specific concept but it is never explicitly defined, therefore it is hard to evaluate whether the interpretation is fair.

  2. Figure 4 seems to contradict the claim on line 195, "This radial norm growth profile is unique to natural activations". The figure suggests that Gaussian and Shuffled synthetic activations show a similar profile for norm-growth indicating that these patterns are a factor of just the mean and variance statistics. Later, the authors point a local maxima in the natural activations that isn't there in the gaussian and shuffled profiles. It is unclear whether this finding is robust to the choice of u^\hat{u} . The authors also state that the plots represent an average over "20 profiles", it is unclear what is being varied xx or (x,u^)(x, \hat{u}) -- regardless it doesn't seem nearly enough to make a geometric claim of extremely high dimensional spaces.

  3. Steering results are incomplete without a coherence check: Steering methods are always evaluated, in some way, for the acceptability of generation.

I would be willing to change my score if provided with the appropriate definitions and interpretations for W1

Questions for authors

How does the choice of u^\hat{u} usually affect the output of η\eta?

How are the aggregate metrics calculated?

4 comments

We thank you for the time and effort you put into your review. Your feedback is deeply appreciated.

Text in the uploaded revision that addresses reviewer concerns is highlighted in blue.

Precise definition of basin, variable and contraction mapping.

We use the term basin to refer to a region along a slice of the activation space where activations accumulate. Formally, if η\eta is the closed path of the probe at the readout layer parameterized by input angle θ\theta, a basin is the region around a local minimum of the probe's speed ∥η˙(θ)∥\lVert\dot\eta(\theta)\rVert, bounded on either side by local maxima. In terms of a mapping and variable, the mapping ff is the forward function through the set of layers, and the variable is θ\theta. Since there is no canonical scale that relates input and output spaces, a contracting map is not well defined in the absolute sense. Instead, we look at relative stretching and compression. Relative to the surrounding space, uniformly spaced activations concentrate within velocity minima. (Revised: L287.)

Synthetic norm growth profile similarity to natural profile, choice of uu, norm growth metric aggregation, robustness and sample size, synthetic norm growth profile, conclusions about high dimensional spaces.

The radial norm growth profile consists of two attributes: a wide basin of low norm growth, and a small norm growth maximum near the unperturbed activation. Norm matched synthetic activations exhibit none of these attributes, while Gaussian and shuffled synthetic activations exhibit only the norm growth basin, but no local norm growth maximum. Qwen 3 is a mild exception to the rule, since there appears to be a weak local maximum for Gaussian and shuffled activations, but this maximum is much weaker than that of natural activations, as confirmed by the separability experiment (L206).

In our experiments we aggregate jointly over randomly sampled uu and random activations, that is, to aggregate over 128 instances, we randomly and independently draw 128 tangents uu and 128 activations. We use random tangent directions sampled from a dd-dimensional standard normal Gaussian, then normalize so they are distributed uniformly on the unit sphere. It is true that the profile can vary based on choice of uu, but the consistency of the profile under the joint samples means that the profile is consistent when varying each independently.

We use figures 4 and 5 only to establish the norm growth pattern qualitatively with a set of 20 examples, then, to show that our observations hold generally, we use a larger set of 128 examples per layer per model
to demonstrate a high degree of separation between natural (higher norm growth within a low norm growth basin) and synthetic activations (low, or even negative norm growth) (see L206). We do the same for the velocity profiles (L301).

As a side note, it is not the case that every direction uu shows the same profile. As you say, this is a high dimensional space, and there may be a small space of weird directions that do not have the profile, but are unlikely to be sampled randomly. We do not claim that these weird directions do not exist, only that a vast majority of directions in a random sample are not weird.

Velocity barrier definition

We use velocity barrier to refer to a local velocity maximum. We will make sure that this term gets an explicit definition in the final version (revised: L297).

Coherence

Thank you for pointing this out. Anecdotally, the continuations that we saw while running the experiments were generally coherent and acceptable. To make this observation concrete, we used an LLM judge to review the continuations from both tuned and adaptive steering methods and found them both about 97% acceptable (97.0% for tuned, 97.2% for adaptive), with no significant difference between the two. In particular, for each continuation, we used GPT-6-Luna to judge whether the predicted token could be continued coherently given the context, asking the model to produce the continuation. We then had Claude Opus 5.5 and a human author read a balanced subset of the judgements and flag any disagreements as a quality check. We will add this detail to the paper (revised: L504, Appendix C L946).

Thanks for including a measurement about steering coherence.

However, certain details about the probe and its interpretations are still unclear:

  1. Please provide the exact formulation for the aggregation. What is it an average over? What does it average?

  2. Please provide at least an empirical convergence plot (for the aggregation numbers) i.e. the estimate on the y-axis and number of sample for u on the x-axis.

  3. Please provide some quantification of the variance induced by the choice of u in your experiments.

Here are the more specific clarifications:

Exact formulation for aggregation

For the norm growth curves in Figures 3 and 4, the exact formula would be, for a distribution of activations XX and uniform distribution UU over the dd-dimensional hypersphere, sample n=20n=20 activation-tangent pairs (x1,u1⊥x1),…,(xn,un⊥xn)(x_1,u_1\perp x_1),\ldots,(x_n,u_n\perp x_n), and average their norm growth as 1n∑i=1n∥f(γ(θ;xi,ui))∥/∥γ(θ;xi,ui)∥\frac1n\sum_{i=1}^n \lVert f(\gamma(\theta;x_i,u_i))\rVert/\lVert \gamma(\theta;x_i,u_i)\rVert.

Convergence plot

We add this as appendix Figure 13 for angles 0, 10, 30, 90, 120, and 180. For most blocks, the mean norm growth at each angle is already stable at just a single tangent per activation.

Quantifying variance from uu.

Variance from uu mainly impacts our qualitative observations in section 3. We measure the standard deviation of the plotted value due to choice of uu as having a median (across panels) of 1.2% at θ=90∘\theta=90^\circ in Figure 3, based on a draw of 32 random tangents per activation.

Since the other quantitative experiments use only metrics at the unperturbed activation xx, or use non-random uu, variance from random choice of uu does not affect the result, e.g., we use the difference between two activations or activation centroids.

Thanks for your response and clarifications. Please improve the general coherence of the draft, it reads like a set of findings.

I have improved the score to reflect my better understanding of the findings.

Reviewer_tLDjOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes a way to probe the activation space by constructing a circle in high-dimensional space that passes through an activation of interest, then studying how its norm growth and velocity magnitude (or tangent area expansion) change after passing through several transformer layers.

Looking at the norm-growth profiles, they find that natural activations tend to increase in norm and often lie at a local maximum within a broader low-growth basin, while synthetic activations generally lack this property. Interestingly, after passing through more layers, synthetic activations gradually develop similar norm-growth profiles to natural ones. This hints that the model might have some kind of attracting mechanism that brings synthetic activations back toward more natural regions of activation space.

They also study the velocity magnitude by probing circles through pairs of natural activations. They find low-velocity regions around each activation, where nearby points are compressed, separated by high-velocity barriers. Together, these observations suggest that activation space may contain distinct attracting basins around natural activations.

The authors then apply these findings to understand activation steering. In the refusal task, they show that steering along the refusal direction can move an activation from a compliance basin into a refusal basin, causing the model to refuse a request. Similarly, in the language steering task, they identify separate basins for English, Chinese, and Greek, with English appearing to occupy a larger region. Finally, they demonstrate that this geometric insight can be used to adaptively choose the steering strength for each activation, improving performance compared to using a fixed steering strength.

Critical strengths

The paper provides clear experimental evidence across different language models, and the findings are also consistent with previous work on activation space geometry and error-correction mechanisms. The experiments with natural and synthetic activations make the observations more convincing, while the steering experiments further demonstrate that these geometric structures are relevant to model behavior.

The paper also provides deeper insights into how activation space is organized, particularly the existence of distinct attracting basins and barriers between them. More importantly, it offers a new perspective on interpreting language models through the geometry and dynamics of their activation space, rather than only focusing on individual features or circuits.

Critical weaknesses

The experiment in Section 4.1 lacks some quantitative details. The authors mention that they apply steering at a "relatively higher strength" to only a single token position, while achieving the same behavioral effect as previous methods that steer across all token positions. However, it is unclear how much higher this strength is compared to the original method, and there is no direct quantitative comparison of their steering effectiveness. Such a comparison would assess the validity of the claims about basins in this experiment.

Another limitation is the practicality of the adaptive steering method in Section 4.2. Unlike fixed-strength steering, the adaptive method requires probing the activation space over different angles and finding the velocity minimum for each activation, which likely introduces much higher computational overhead. It would be useful to evaluate whether the performance improvement is worth the extra computation, especially for practical applications.

Questions for authors
  1. In Section 4.2, how do you find the optimal steering strength for each activation? Do you use gradient-based optimization, grid search, or another method to locate the velocity minimum?

  2. How do you determine the boundaries of an attracting basin? Is there a specific criterion for deciding whether two activations belong to the same basin or different basins? Can multiple natural activations share the same basin?

  3. In Figure 12, why do the English and Chinese prediction regions sometimes alternate along the steering direction instead of being clearly separated? Does this make it more difficult to select the appropriate steering strength?

2 comments

Thank you for your review. We appreciate your effort, curiosity, and feedback.

Text in the uploaded revision that addresses reviewer concerns is highlighted in blue.

Quantitative details

For our Section 4.1 steering experiment, we compare to the steering strength used by Arditi et al., which applies a raw difference-in-means steering vector at every token position. Our method applies the same steering vector, but scaled to about 3.37x its original magnitude, and only at a single token position. We will add this detail to the paper (revised: L374–375).

We agree that the refusal experiment is illustrative, not quantitative, and we intended for it to be so. Our intent was to present the case study as a motivating example for our controlled language steering experiment.

Practicality of steering, how to find optimal steer strength

Our steering method is indeed more expensive than plain fixed-strength steering by O(n)O(n) where nn is the number of strengths tested per context, i.e., we use grid search to find the optimal steer strength. Our implementation evaluates a range of strengths all at once, so a granularity of 50 strength values means 50 additional parallel forward passes through the 4-layer block. In our experiments, we used a much finer grid of about 500 points per context, but, in retrospect, Figure 12 shows that this level of density is gross overkill, since the velocity profiles are quite smooth. We used finite differences to evaluate the velocity at each point, rather than rely on automatic differentiation, which saves compute. In total, the extra passes through a few middle layers is not prohibitively burdensome, though it is not free. The number of forward passes might be reduced using a zero-finding method like Newton's, at the expense of parallelizability. We did not prioritize optimizing the method, since the goal of the experiments was to demonstrate the role of basins in model behavior, not to recommend a more efficient steering method.

Other questions

Determining boundaries

To determine boundaries between attracting basins, for 1-dimensional probes each local minimum is a basin, whose region is bounded by the adjacent local maximums. For our 2-dimensional probes, the basin for each local minimum can be the region where gradient descent on the stretch magnitude ends up at the minimum, though we do not prescribe this definition, since it is not required for our quantitative experiments. Multiple activations can belong to the same basin under this definition. For instance, pairs of synthetic activations occupy the same basin in Figure 7.

Alternating regions

We do not know why English and Chinese tokens sometimes alternate along the steering direction. One hypothesis we have is that the space "folds back on itself", so that steering along that direction at an earlier layer passes multiple times through the two regions at a later layer. Since we use the geometry rather than the output behavior as our steering criterion, and there is usually only a single dominant second basin, the alternating behavior does not cause us issues.

Thank you for your response.

I think some of my concerns, namely determining boundaries and finding optimal steer strength, are resolved. Meanwhile, the quantitative evidence request for the refusal experiment is still open.

Though I particularly interest to see whether last token transferring is enough in refusal experiment, I believe the contribution of the analysis and other experiments, together with the additional information in the rebuttal, are substantial for a conference paper. I therefore increase my score to reflect this decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.