acceptodds
Under review as a conference paper at ICLR 2027

The Saussurean Representation Hypothesis: A Causal Account of Representational Structures in LLMs

Abstract

Mechanistic Interpretability represents a dominant set of approaches and often seeks to localize model responses to particular input features in specific attention heads, neurons, or circuits, typically validating such localizations by ablating the identified model components and measuring degradation of the targeted capability. This target-centric approach frequently fails to engage with a core tenet of semiotics: that meaning arises from the relations among concepts rather than the concepts themselves. Here we examine collateral damage, investigating how ablating one concept within a model affects performance on other similar concepts. We introduce a non-parametric, information-theoretic method for performing ablations. By estimating attention-head-level mutual information, we reveal which components of a model encode information related to specific features in a data distribution. Examining languages and WordNet-derived knowledge domains, we show that model representations recover independently defined similarity structure in both, placing languages of the same subfamily, and topically related domains, closer together in their representation space. Ablating heads with the highest pointwise mutual information for a given language causes significant damage to the model's performance on that language, and also affects members of the same language subfamily. When we apply this non-parametric method to 51 WordNet domains derived from Wikipedia across Qwen, Llama, OLMo, and Gemma models, we find that collateral damage across concepts consistently correlates with the semantic distance between them. We term this the Saussurean Representation Hypothesis: in LLMs, the functional representation of a language or conceptual domain is constituted in substantial part by its relations to neighboring representations. Consequently, causal localization cannot be evaluated only by what an intervention removes; what it removes in parallel with the target reveals the structure of the representation itself.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, large language models, multilingual language models, causal intervention, attention heads, ablation, representational geometry, cross-lingual transfer, knowledge localization, information theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4pv3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Saussurean Representation Hypothesis (SRH), suggesting that LLM representations depend partly on their relationships with neighboring representations. The authors introduce a training-free, PMI-based approach to identify and intervene on attention heads. Experiments across multiple model families, languages, and 51 knowledge domains show that collateral damage decreases with semantic distance, providing empirical evidence of structured functional dependencies among representations.

Critical strengths
  1. An interesting and meaningful research perspective. The paper investigates collateral damage rather than focusing exclusively on target-specific performance degradation. This provides a valuable perspective for understanding the relational organization of representations in LLMs and extends conventional target-centric interpretability analysis.

  2. Broad empirical validation across models and domains. The experiments cover multiple model families, languages, and knowledge domains. The consistent relationship between semantic proximity and collateral damage, supported by statistical analyses, provides useful empirical evidence of structured functional dependencies.

Critical weaknesses
  1. Insufficient evidence to distinguish the core hypothesis from alternative explanations. Although the paper reports significant correlations between collateral damage and semantic distance, these observations may also arise from shared features, representational overlap, or compression mechanisms. The current experiments do not adequately distinguish these competing explanations. This limitation directly affects the strength of the causal and theoretical claims underlying the proposed SRH.

  2. Insufficient validation of the intervention mechanism and its functional implications. The proposed category-conditioned mean substitution differs from conventional attention-head ablation, and the current experiments primarily use random-head controls without systematic comparisons against alternative head-selection or intervention methods. Furthermore, relying mainly on NLL does not fully establish whether the observed damage reflects changes in conceptual understanding, factual knowledge, or relational reasoning. These limitations weaken the methodological justification and functional interpretation of the results.

Questions for authors
  1. How can the proposed SRH be distinguished from alternative explanations based on shared features, representational overlap, or compression? Could the authors provide controlled analyses to determine whether semantic similarity remains predictive of collateral damage after accounting for these factors?

  2. Since Equation (4) applies a category-conditioned mean shift rather than conventional head removal, how sensitive are the reported findings to this specific intervention? Would zero ablation or standard mean ablation produce similar collateral damage patterns?

  3. How does the proposed PMI-based head-selection strategy compare with representative gradient-based, activation-based, or alternative information-theoretic methods? Could the authors clarify whether the observed advantages are specific to PMI-based selection?

  4. To what extent do the NLL-based collateral damage patterns correspond to changes in downstream capabilities, such as knowledge question answering, relation classification, or multilingual understanding? Are any task-level results available to support this interpretation?

Reviewer_7TcGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Saussurean Representation Hypothesis (SRH): representations are determined substantially by their relationships to neighboring representations. It introduces a non-parametric, information-based method for selecting attention heads and reports that category-conditioned interventions lead to greater collateral damage to related languages and knowledge domains.

Critical strengths
  1. The paper quantifies how collateral damage varies with linguistic and topic distance. This is useful for understanding which categories are vulnerable to the same interventions.
  2. Similar patterns appear across multiple model families and in both language and knowledge-domain experiments, strengthening the conclusion.
  3. The analysis of writing systems adds nuance to the results. In particular, the Serbian comparison helps study how surface form contributes to the observed effects.
Critical weaknesses

Overall, I think the experiments do not sufficiently support and refute hypotheses as the authors claim.

  1. The paper overstates the assumptions of prior work. The statement (L112) that "localization predicts negligible effects on concepts other than the target" seems too strong. Finding a feature doesn't mean it uses separate mechanism or can be removed without affecting other features. Superposition explicitly allows overlapping representations and interference (Elhage et al., 2022), and Nam et al. (2025) show that individual heads’ roles depend on their interactions with other heads. The idea that SAEs recover a canonical set of atomic features is also debated (Leask et al., 2025). The authors acknowledge these papers, but attribute independent removability to them. Thus, I think this assumption premise needs more justification.

  2. Shared features could explain the same results. French and Spanish share certain lexicon and syntax. Similarly, biology and medicine share terminology and facts. Intervening on computations that both categories use could produce the reported pattern without necessarily confirming SRH. The authors should identify a prediction that separates out SRH or narrow the conclusions.

  3. The controls do not match the strength of the intervention. The random controls match head counts and layers, but not the magnitude of the resulting change after the output projection. The targeted intervention could therefore introduce a larger perturbation than the controls.

References

  • Elhage et al. (2022). Toy Models of Superposition.
  • Leask et al. (2025). Sparse Autoencoders Do Not Find Canonical Units of Analysis. ICLR.
  • Nam et al. (2025). Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers. NeurIPS.
Questions for authors

What in these results favors SRH over the explanation that related concepts share features?

Reviewer_HdJEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a hypothesis that the representation of a language or a knowledge domain is constituted by its relations to neighboring representations. To test this, the authors score each attention head by its mutual information with a property of the input, such as the language or topic of a text (Indo-European family tree or WordNet-derived Wikipedia as test domains). They then ablate the top-scoring heads for one target and measure the loss increase on all other categories. The authors conclude that collateral damage is structured and informative, and that causal localization cannot be judged just based the target.

Critical strengths
  • Investigating the representations with the proposed testbed makes sense and can yield interesting results.
  • Case studies are well chosen with sensible controls.
Critical weaknesses
  • (W1) The framing misrepresents both the prior literature and the outcomes of the study. The introduction attributes an "atomist" assumption to mechanistic interpretability, but the cited works tend to document the opposite. Neurons are polysemantic, and dictionary learning, feature splitting, and absorption never really claim atomicity, since they all document relational structure among recovered features. The abstract and discussion then claim that the relations themselves encode meaning, and that the structure reflects no shared reality and no compression byproduct. No experiment here separates these accounts. I don't find the recovered topic similarity and language entanglement particularly surprising, given the works you cited and the ones I added below. The paper should temper the claim or add an observation that digs deeper.
  • (W2) Key alternative explanations remain untested. The intervention ablates a large share of all heads at once. Under such a broad perturbation, damage could come from some surface features like script, token statistics, or topic vocabulary. The paper's own result on Serbian shows that script contributes. A surface-sharing account is missing from the three alternatives in §1.
    • Circularity: The domain reference comes from Wikipedia links, and the evaluation text is Wikipedia too. The reference is not independent of the text statistics the model may encode. WordNet path distance or human similarity judgments would fix this.
    • The pre-intervention structure is measured in hidden states, whereas the intervention acts on attention-head outputs.
  • (W3) The novelty relative to the closest prior work is unclear.
    • Rakshit et al. (2026) already showed that LLM representations recover the Indo-European tree, and that structural fidelity predicts multilingual performance. The submission seems to extend that work but still lists the language structure recovery as its own result. If Rakshit et al. is the authors' own prior work, as the overlap suggests, the incremental character of the language part should be stated openly. The method also follows an established recipe.
    • Amnesic probing (Elazar et al., 2021) [1] removes a property from representations and measures the behavioral consequence. Held and Yang (2023) [2] score attention heads by their effect on a target language and prune them. Neither is cited. What is new is the hypothesis framing, the damage profile as the object of study, and the knowledge-domain testbed. Claiming exactly that would strengthen the paper.

[1] Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (Elazar et al., 2021) [2] Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers (Held & Yang, 2023)

Questions for authors
  • (Q1) Which claim do you take the experiments to establish: the constitutive reading, that relations encode meaning, or the methodological reading, that damage profiles reveal representational structure?
  • (Q2) What is the relationship between this submission and Rakshit et al. (2026)? Which parts of the language-side results are new here, and which replicate that work?
  • (Q3) How do your PMI-selected heads overlap with the heads Held and Yang (2023) identify and prune? Can both pictures be right, and under what conditions does removing language-associated heads help a language (their finding) versus hurt its relatives (yours)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.