- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the geometry of language model activation spaces. The authors argue that activations arising from natural text lie in privileged attracting basins: regions in which transformer-layer dynamics tend to funnel perturbed or synthetic activations toward more “natural” configurations. They investigate this hypothesis using norm-growth and speed profiles along local spherical probes. They then apply this interpretation to activation steering: refusal/compliance and language-switching directions are argued to correspond to distinct basins, and selecting a steering strength that places an activation in the target basin improves model control.
Critical strengths
- The paper puts forward a conceptually compelling hypothesis and connects it well to previous findings on steering and error correction.
- It compares several model families and several types of synthetic controls.
- The English-to-Chinese steering experiment includes both in-distribution and out-of-distribution evaluation.
- The visualizations are particularly useful for connecting a steering direction, geometric regions in activation space, and output behavior.
- The authors explicitly acknowledge the exploratory nature of the work and several open limitations, including the case of very nearby activations and the possibility of a low-dimensional manifold structure.
Critical weaknesses
- The paper does not cite work on anisotropy in LLMs, such as [1-4]. Yet, at first glance, its geometric analysis and its notion of “attracting basins” strongly resemble anisotropy-related work. I had to examine the paper carefully to understand the distinction. A related-work discussion that explicitly distinguishes these notions would make the paper’s contributions clearer. I am not claiming that the two are identical; rather, such a discussion would help readers familiar with anisotropy understand more quickly what is genuinely different here.
- The causal link between the observed geometry and the model’s overall robustness remains only partially established. The gains from adaptive steering support the hypothesis, but do not rule out alternative explanations.
- The refusal result is essentially illustrative: it relies on one pair of prompts and a visualization, rather than on a broader benchmark measuring reliability, safety, and preservation of utility.
[1] Ethayarajh "How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embedding" EMNLP 2019
[2] Timkey et al. "All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality" EMNLP 2021
[3] Rudman et al. "Stable Anisotropic Regularization" ICLR 2024
[4] Bernas et al. "Revisiting Anisotropy in Language Transformers: The geometry of learning dynamics" ICML 2026
Note: I have included only 4 citations here to provide an overview of the field, but the field is quite rich; I recommend that the authors read the literature carefully.
Questions for authors
- The paper should be more clearly situated within the existing literature, so that readers can identify its actual contributions earlier, without having to wonder whether the central observations have already been established or whether the claims are overstated.
- The computational cost of finding, for each activation, the speed minimum along the steering direction is not discussed in much detail. I would be interested in knowing how expensive this procedure is in practice, and how it scales with the number of activations and candidate steering directions.
2 comments
Ok thanks for your reply. That seems good for me, I'll keep my accept.
Thank you for your thoughtful review. Your engagement and comments are very helpful.
Text in the uploaded revision that addresses reviewer concerns is highlighted in blue.
Thank you for pointing out these citations. We will add them to our revision and relate our work to them (revised: L070, L143, L203).
Your observation is correct, but this is a necessity rather than an oversight. We did try to isolate a specific mechanism during our exploration, but found (in agreement with Shinkle et al) that the basins are implemented by a large and distributed set of MLP neurons. As a result, we do not have an isolated model component that we could realistically ablate to establish causality. We therefore must leave it to future work to isolate basin-implementing components (if it is indeed possible) and establish their causal role.
Instead of a single mechanism within the model, we think it more accurate to conceptualize our work as a set of observations about the model's activation space geometry that enable us to better understand the dynamics at play within the model and design better interventions.
We will be sure to address this in the revision (revised: L525).
We agree that the refusal result is illustrative, and we intended for it to be so. We presented the case study as a motivating example for our controlled language steering experiment.
Questions
To situate our work within the anisotropy literature, our work analyzes the geometric sub-structure within the anisotropic region occupied by natural activations.
Our Gaussian and shuffled synthetic activations inherit the anisotropy of the natural activations, allowing us to control the generic structure of the activation space, i.e., norm growth and velocity basins, and identify sub-structure, i.e., local norm-growth maxima and velocity barriers between tokens. We will make this clear in our revision (revised: L070, L143, L203).
Our steering method is indeed more expensive than plain fixed-strength steering by O(n) where n is the number of strengths tested per context. Our implementation evaluates a range of strengths all at once, so a granularity of 50 strength values means 50 additional parallel forward passes through the 4-layer block. In our experiments, we used a much finer grid of about 500 points per context, but, in retrospect, Figure 12 shows that this level of density is gross overkill, since the velocity profiles are quite smooth. We used finite differences to evaluate the velocity at each point, rather than rely on automatic differentiation, which saves compute. In total, the extra passes through a few middle layers is not prohibitively burdensome, though it is not free. The number of forward passes might be reduced using a zero-finding method like Newton's, at the expense of parallelizability. We did not prioritize optimizing the method, since the goal of the experiments was to demonstrate the role of basins in model behavior, not to recommend a more efficient steering method.