- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper challenges a common assumption: that concepts can be represented as global linear directions in neural network activations. The paper theoretically argues that if a concept is actively involved in computations by the attention mechanism, it typically cannot remain linear at all times within the Transformer module. The empirical section quantifies the degree of alignment and effective dimension of concept directions, measured by the mean difference, under various contextual conditions across 15 models, 9 datasets, and three modalities: images, text, and audio. Based on the observed directional changes, the paper proposes CACTUS: a method that models concepts by considering the union of all contextual concepts representing the target attribute, or using it as context. On language and sentiment concepts across three large language models, CACTUS achieved lower confusion scores compared to global linear steering and several baselines.
Critical strengths
1.This work tests the hypothesis from mechanistic interpretability and activation steering that concepts can be represented by a single global linear direction, and suggests that concept representations may exhibit significant context dependence.
2.Within the framework of the global linear representation hypothesis, this paper derives the constraints that must be satisfied for a concept vector to maintain a globally fixed linear output after passing through a Transformer block. This work establishes a mechanistic link between the empirical question of whether a concept possesses a linear representation and the attention-driven mechanisms of information routing and representation transformation. Although the scope of applicability of these theoretical conclusions remains to be further clarified, the formalization of the problem and the analytical perspective presented in this paper are highly insightful.
3.The empirical section spans 15 models, nine datasets, and three modalities (image, text, audio), encompassing both Transformer and non-Transformer architectures. Its conclusions therefore enjoy stronger evidence for generalizability compared with observations obtained solely from a single model or task.
Critical weaknesses
1.The theoretical section discusses the conceptual directions in which attention mechanisms actually participate in computations; however, the empirical section analyzes the conditional mean difference directions constructed based on externally supervised labels. The paper does not verify whether the directions derived from these labels indeed correspond to the concepts actually used by the attention mechanisms within the model. Therefore, the observed cross-context changes in direction may also stem in part from label definitions, conditional data distributions, covariance structures, or the direction estimation process, and do not necessarily represent the nonlinearity of the model’s intrinsic conceptual representations.
2.The effective dimension metric d_eff suffers from interpretive ambiguity. Computed over the Gram matrix of context wise concept discriminant vectors, it measures dispersion among context specific directions rather than the intrinsic dimension of the concept manifold in raw activation space. The main text does not draw a clear distinction, risking misinterpretation by readers.
3.The ablation and practical usability analysis of CACTUS is insufficient. Although the paper compares the Single-Context baseline and discusses the performance of certain directions when used individually, it lacks a systematic ablation analysis of each component, making it difficult to distinguish whether performance improvements stem from the two types of context directions, the max aggregation mechanism, or the larger set of candidate directions. Furthermore, the context-oracle relies on true context labels; the paper does not explain how this information is obtained in unlabeled real-world scenarios or how errors in this information affect performance.
4.The formalized proposition in the paper is derived exclusively for Transformer architectures. Nevertheless, empirical results show that context-dependent concept directions also exist in convolutional models. The paper does not discuss the potential mechanisms within convolutional layers that give rise to this phenomenon. In the absence of such discussion, readers cannot determine whether this contextual variability is an artefact specific to Transformers or a general property of discriminative representation learning.
Questions for authors
1.All concept directions are constructed from external supervision labels. How do the authors validate or support that these label-derived directions genuinely correspond to intrinsic concept representations that are actually utilized by the model’s attention mechanism, rather than primarily reflecting statistical correlations induced by dataset labels, conditional data distributions, or the probe estimation procedure?
2.Experiments have observed that CNN models also exhibit context-dependent conceptual directions, but your theory applies only to Transformers. Please supplement your manuscript with a discussion of which candidate mechanisms within convolutional networks could cause conceptual directions to vary with context; even if you do not provide a new formal proof, an analysis should be included.
3.The component ablation of CACTUS remains incomplete. Could the authors separately quantify the performance contributions of the two types of context directions, the max aggregation mechanism, and the size of the candidate direction set? In addition, please clarify whether CACTUS’s performance gains persist under baselines matched in parameter count or the number of candidate directions.
4.In the CACTUS implementation, gradient directions are computed only once from the original activations and are not recalculated during steering. Please discuss the potential biases introduced by this approximation and identify the conditions under which this approximation degrades steering performance.
Summary of Paper Revisions & New Experiments
We thank the reviewers for their detailed feedback and challenging questions. They allowed us to clarify our arguments, and widen our analysis to more baselines, models, tasks, datasets and metrics. From those, our article is now more robust empirically and theoretically. To summarize:
After revisions, our analysis is now substantially larger-scale, more varied, and theoretically stronger. Therefore, our claims are now more robust, and our analysis is richer. We remain open to further discussion if any concerns remain.
[1] Parekh, Jayneel, et al. "Learning to steer: Input-dependent steering for multimodal llms." Advances in Neural Information Processing Systems 38 (2026): 159799-159834.
[2] Sun, Jiuding, et al. "Hypersteer: Activation steering at scale with hypernetworks." arXiv preprint arXiv:2506.03292 (2025).