acceptodds
Under review as a conference paper at ICLR 2027

Receptors of Language Models: From Singular Directions to Token Predictions

13 views

Abstract

While mechanistic interpretability has largely investigated the information/context flowing through the transformer architecture, the structural blueprint statically/inherently encoded in its parameters remains comparatively underexplored, leaving open a more fundamental question, what predictive structure is already encoded in the weights (inherent to model parameters)? We show that the static weight matrices of transformer components contain sparse, token-aligned directions, which we term , that show a strong causal link with model predictions. This structure is consistent across a wide family of architectures. Across 21 open-weight models spanning 5 architecture families (, , , , and ), we demonstrate the presence of receptors providing an interpretability lens that is applicable to wider range of contexts. These receptors emerge and stabilize over training. Additionally, we find that a kurtosis-based analysis of individual receptors reveals additional structure in their weight-space geometry. Our work provides a detailed examination of the model's inherent structural properties, which could serve as a complementary foundation for studying, measuring, and intervening in learned behavior.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, Explainable AI, Mechanistic Interpretability, Transformer Models, Large Language Models, Singular Value Decomposition, Representation Analysis, Weight-Space Analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZcZDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses Singular Value Decomposition to analyse the OV map of each attention head and the output matrix of each MLP block. It then measures how the left singular vectors (“receptors”) affect token probabilities. The paper claims that the receptors have sparse effects on outputs, partially predicted by weight alignment. It also studies the emergence over training, and uses kurtosis-based analysis to reveal further vocabulary structure.

Critical strengths

The SVD analysis is weight-based and independent of any particular prompt (although the ranking depends on context).

The randomly initialised GPT-2 and random receptor controls strengthen the results.

The paper applies the method to a large set of 21 open-weight models.

Critical weaknesses

Novelty: How does this work improve upon previous SVD weight analyses such as the cited works, or Millidge & Black (2022)’s “The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable”?

In the main analyses, the receptor effect on tokens considers only the direct effect via the residual stream, not the indirect effect mediated via other components. This substantially reduces the amount of the model that can be understood with this method.

Receptors are selected by which ones raise the target probability (Appendix G) and receptor isolation (Algorithm 1) removes all other directions. Given that the target is used for this selection I worry that this removes receptors contributing towards wrong answers (e.g. other multiple choice answers), making the “perfect (1.0) target probability” artificial.

Questions for authors

To test whether the receptor isolation inflates the results (“perfect (1.0) target probability”): Can you run the algorithm but select the receptors corresponding to a wrong answer, and check whether this then leads to a perfect probability for that wrong answer?

Tables 1, 7, and 8 have identical captions. Is this a typo?

4 comments

We thank the reviewer for the thoughtful/constructive and detailed feedback. We really liked the positive remarks about the presentation and the recognition of the prompt-independent nature of the weight-based analysis. We address the queries below:

[w1]

Regarding the contributions, we would like to clarify that applying SVD to Transformer weight matrices is an active area of interest and has been studied by multiple prior arts. Our contribution is more towards investigating/establishing the functional properties of the resulting singular directions (which turn out to naturally exist), which we refer to as receptors (also see the equation on line 147), and characterizing how they relate to vocabulary/prediction-level model behavior.

  1. We establish a connection between the left singular vectors of attention OV maps and MLP output matrices and their effects on vocabulary probabilities, allowing us to characterize the functional selectivity of individual receptors rather than interpreting weight matrices solely through their algebraic structure.
  2. We investigate the sparsity of these effects and examine the extent to which they can be predicted by weight alignment, which provides a quantitative connection between the geometry of the learned parameters and their functional consequences.
  3. we investigate how these properties emerge during training and examine additional vocabulary structure through kurtosis-based analyses. We believe the kurtosis-based analysis devised in Algorithm-2, adds another interesting finding about the receptors, that provides a second lens for looking at the interpretable directions that are natively existing (not context-dependent like other interp methods) in the (trained) transformer architectures.

Overall, the range of experiments we perform is completely complementary to the prior art, both in design and in findings. We believe our work enriches our understanding of this underlying structure, opening up new avenues for future investigations.

[w2]

Regarding the direct effect, we would like to clarify that our primary objective is to characterize the immediate functional contribution associated with individual weight-derived directions (which can only be observed via direct effect). This also provides a controlled setting in which the relationship between the weight geometry and vocabulary-level effects can be studied without conflating it with subsequent transformations by other components (which also adds the additional advantage of faster interp, where a Llama-8 B model only takes 3min to find all receptors, also see Table 4 for other models). However, we do understand that direct effects do not capture the full causal influence of these directions on the final model prediction. We will clarify this distinction in the camera-ready version and avoid interpreting direct receptor effects as complete explanations of model computation (if stated anywhere). We thank you for this comment on clarification.

[w3]

We agree that (when observing only Fig. 13) it may look like, selecting receptors according to their effect on a predefined target introduces a selection bias that must be considered when interpreting the isolation results. We specifically rule out this possibility in initial experiments. We would like to point to the observations in Fig. 5 of the main paper, where we observe that even random context (random-init-context, Figure 18), leads to the prob. reaching one for a few of the receptors (notice the first subfigure Fig. 5, top green dots, prob. increasing by 10,000x). This establishes that the receptors we amplify directly affect the target probability even with random tokens in the context, making them independent. Additionally, we run the experiments, as the reviewer suggested.

[Q1]

We ran the experiment corresponding to Figure 13, where we took only the top1% receptors corresponding to token “_A” and ablated the rest. The following table shows the probability of tokens “_A” and “_B” across the Llama3-8B model layers, where MMLU questions with the correct answer as “_B” are chosen.

LayerP(“_A”)P(“_B”)
L0-L90.0000.000
L100.0010.000
L110.0020.000
L120.0050.000
L130.0400.000
L140.0730.000
L150.2720.000
L160.3730.000
L170.6060.000
L180.6910.000
L190.9450.000
L200.9850.000
L210.9840.000
L220.9960.000
L230.9990.000
L241.0000.000
L250.9790.000
L260.9840.000
L270.9910.000
L280.9990.000
L291.0000.000
L301.0000.000
L311.0000.000

[Q2]

We thank you for this detailed review, yes, this is a typo, all the three are for different models, which can be seen in the index of the appendix, we forgot to update the same in the actual captions. We will fix this in the camera-ready version of the paper.

I thank the authors for their response.

I acknowledge the response to W1 and appreciate the clarification of the paper's contributions.

I appreciate the clarifications regarding W2, W3, and Q1. I believe those confirm my initial judgement of the weaknesses.

I maintain my initial recommendation for a weak rejection.

We thank you for engaging with our responses and acknowledging the clarifications. We are happy that our response could address the initial concerns. We also liked the suggestion regarding an additional experiment performed during the rebuttal, which help clarify other concerns regarding the receptor isolation, we will be adding it to the updated version of the paper.

We believe that the clarified contributions, 1) the training-time emergence and stabilization of receptors, 2) their internal vocabulary geometry through kurtosis-based analysis, and 3) the extensive evaluation across 21 models, establish a broader contribution than the initial assessment may suggest.

Given these clarifications and the additional experimental evidence, we would sincerely appreciate it if you could reconsider your assessment and whether these contributions lead to an increase in the current score. We would be happy to answer/clarify/address any remaining specific concerns that might help improve this work further.

We thank you again for your constructive feedback.

Dear Reviewer ZcZD,

Since we are near the end of the discussion phase, we would also like to point you to an additional experiment done during the rebuttal, on the Reviewer 6RFk's suggestion/question, that help provide an important alternative explanation. i.e., the similarities between token unembedding vectors could contribute to correlated output effects. For instance, since the unembedding vectors of "Paris'' and "France'' are relatively similar, activating receptors that promote "Paris'' could potentially also increase the probability of "France''.

To investigate this possibility, as suggested by the reviewer, we first computed the cosine similarity between the two unembedding vectors and confirmed that it is relatively high. However, selectively activating/scaling "Paris''-associated receptors increases P(Paris)P(Paris), without correspondingly increasing P(France)P(France), suggesting that unembedding similarity alone cannot explain receptor-level selectivity.

To understand this behavior more closely, we ran another experiment to find the receptors corresponding to France'', and independently ranked all 263,168 receptors according to their effects on "Paris'' and "France'' in Llama-3-8B. We then examined the cross-token rankings of the top-10 receptors for each token. (see table in different comment below Reviewer 6RFk's response ) https://openreview.net/forum?id=PflKalCfkh&noteId=3bQMJX6EG3

Importantly, these bottom-ranked receptors are not simply inactive for "France'', they have negative effects, such that activating them suppresses 𝑃(France)𝑃(France). Hence, the receptor population contains not only receptors that promote a particular token but also receptors that actively oppose other tokens, even when those tokens have similar unembedding vectors.

This becomes particularly striking in our receptor-isolation experiment. Isolating and activating the top 0.01% of "Paris'' receptors yields 𝑃(Paris)=1.0𝑃(Paris)=1.0 and 𝑃(France)=0.0𝑃(France)=0.0. In other words, despite the high similarity between their unembedding vectors, the receptor-level intervention produces an entirely token-specific outcome.

Overall, we believe these results suggest that the vocabulary-selective structure of receptors is richer than what can be explained by unembedding-vector similarity alone, and kurtosis analysis adds another interesting layer for inspecting this structure. Receptors can contain shared selectivity across related tokens, but they can also exert opposing effects, actively promoting one token while suppressing another, leading to this interesting control mechanism that is inbuilt in the model. We believe this also provides direct evidence that the co-activity patterns captured by our kurtosis analysis reflect structured receptor-level selectivity, than simply inherited similarities in the unembedding space.

We hope the above added analysis strengthens the paper more, which will help in reconsidering our work. We are pleased to see your involvement in the discussion, we will be happy to answer any other questions/clarification you have regarding the novelty and the supporting evidence.

Reviewer_4pX7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether predictive structure can be identified directly from the weights of pretrained language models. The authors introduce receptors, defined as singular directions of attention OV and MLP output matrices, whose contributions can be amplified or suppressed to influence target-token predictions. Across 21 models from five architecture families, they find that target-token effects are concentrated in a small subset of directions, and that retaining selected receptors can substantially increase target-token probabilities. They further investigate how these directions emerge and stabilize during training, and apply a kurtosis-based optimization method to identify vocabulary-selective channels within individual receptors. Free-generation experiments show that targeted interventions can increase the frequency of specific tokens while moderate interventions often preserve overall language-model perplexity.

The paper presents extensive empirical evidence for the controllability of token predictions through weight-defined directions. However, the novelty relative to existing SVD-based interpretability methods and the extent to which these interventions reveal the model's original predictive mechanisms remain unclear.

Critical strengths
  • The evaluation spans 21 models across five architecture families, providing broad empirical evidence for the observed receptor effects.

  • Targeted interventions substantially affect token generation, while moderate amplification often causes limited perplexity degradation.

  • The analysis of receptor emergence and stability during training provides useful empirical observations.

Critical weaknesses
  • Limited novelty over prior work. Ahmad et al. (2025) already study Logit Receptors and causal interventions on SVD-derived directions. Xue and Andrzejak (2026) also investigate SVD-based functional units, while Algorithm 2 builds on ROTATE. The paper does not sufficiently establish what new conceptual or methodological insight distinguishes its framework from these approaches.

  • Insufficient justification of causal sufficiency. Algorithm 1 modifies cached residual representations without recomputing downstream layers. Thus, selective retention demonstrates direct effects on target-token predictions but does not necessarily establish sufficiency for the model's original computation. The pseudocode also appears to preserve contributions from components without selected receptors, which may affect the reported sparsity results.

  • Unclear mechanistic specificity. The experiments demonstrate effective token-level steering, but do not fully distinguish this effect from context-dependent predictive mechanisms. Comparisons with alternative orthogonal bases would also help establish whether the observed sparsity is specific to SVD.

Questions for authors
  1. What is the main contribution beyond the Logit Receptors proposed by Ahmad et al. (2025)?

  2. Does Algorithm 1 remove contributions from components without selected receptors?

  3. Can the authors distinguish token-level steering from task-specific causal mechanisms?

  4. Is the observed sparsity specific to the SVD basis?

4 comments

We thank the reviewer for the thoughtful comments and for recognizing the breadth of our empirical evaluation. We address the concerns below:

[w1]/[Q1]

We would like to emphasize that applying SVD to Transformer weights or intervening in singular directions is one of the active areas of interest in some of the recent works (we cite/discuss all of these works in detail in the paper). We list some of the prominent differences below:

  1. In terms of architectural scope and objective, our analysis focuses on both attention OV projections and MLP output matrices when compared to Xue and Andrzejak (2026), who only focus on MLP. These transformations write directly into the residual stream, allowing us to investigate how individual weight-defined directions contribute to vocabulary-level predictions. In contrast, the Xue and Andrzejak (2026) work primarily examines MLP weight matrices to understand intermediate feature processing.
  2. We introduce receptors as residual-stream directions associated with individual rank-one components of these transformations. These directions provide a more granular unit for studying contributions to vocabulary logits. The Xue and Andrzejak (2026) work instead introduces Detector-Effector Units, which decompose MLP computation into detector, effector, and gain components to characterize semantic feature processing, and although both approaches employ SVD, they study different interpretability questions and perform intervention differently (direct effect vs total effect).
  3. Primarily the mechanistic objective is also completely different, our work investigates whether predictive structure is naturally encoded in weight-defined residual-stream directions. In addition to characterizing their vocabulary-level selectivity, we introduce a kurtosis-based optimization procedure to uncover vocabulary-selective structure within individual receptors. We further investigate weight alignment, emergence during training, and the stability of these properties across 21 models spanning five architecture families.

We therefore view these approaches as complementary to the prior arts.

We would also like to highlight that the motivation/objective, as well as the design of experiments that we perform, are completely different, with different findings observed when compared to any existing prior art. Overall, our work quantify how individual receptors affect vocabulary probabilities and characterize the concentration and sparsity of these effects. help investigate whether the functional selectivity of receptors can be partially predicted from their alignment with vocabulary-related directions. study how receptor selectivity emerges and stabilizes during training. introduce a kurtosis-based optimization procedure to identify vocabulary-selective channels within individual receptors. evaluate these properties across 21 open-weight models spanning five architecture families. examine whether receptor interventions influence generated token frequencies while preserving reasonable language-model perplexity under moderate interventions.

[w2]/[Q2]

We thank the reviewer for this comment; we believe the color-coding in Algorithm 1 (though appreciated by Reviewer 6RFk) may be causing this confusion. We would like to clarify that Algorithm 1 contains two distinct procedures, represented by the orange and green sections of the pseudocode.

  • The orange section performs receptor amplification. Given a selected receptor set, it amplifies the contributions along those directions by a factor of , while preserving contributions from the remaining receptors. This procedure is intended to investigate how amplification of selected directions influences token probabilities and generation.
  • The green section performs receptor isolation. Unlike amplification, this procedure retains only the contributions associated with the selected receptor set and masks the contributions from all remaining receptors, setting them to zero. Consequently, the resulting residual representation is constructed exclusively from the selected receptors, without contributions from unselected receptors within the analyzed transformation (as asked by the reviewer)

Importantly, the sparsity analyses reported in Figures 3 and 13, among others, use the green receptor-isolation procedure rather than the orange amplification procedure. Therefore, these analyses measure the predictive effects of the selected receptor contributions in isolation, rather than effects arising from their amplification in the presence of other receptor contributions.

We acknowledge that the distinction between these two procedures was not sufficiently explicit in the original presentation. We will explain Algorithm 1 and its accompanying description to clearly distinguish amplification from isolation and specify the treatment of unselected receptor contributions. We thank the reviewer for this clarity comment. We hope it addresses your concern.

[w3]/[Q3]

We agree that the ability to steer token probabilities does not, by itself, establish that an individual receptor implements a complete task-specific mechanism. We would like to clarify that the objective of this work is more specific, i.e., to identify whether learned weight matrices contain directions that exert measurable functional effects on target-token predictions. These directions provide a more granular basis for interpretability than treating an entire attention head or MLP module as an indivisible computational unit (as done in mech interp/circuit discovery literature throughout). An important distinction in our framework is between the fixed nature of receptors (existing natively) and their context-dependent functional effects. Receptor directions are determined by the learned weight matrices and therefore remain fixed across inputs. However, their activation coefficients and resulting contributions to predictions depend on the input context. Thus, receptors should not be interpreted as standalone semantic or task-specific mechanisms. Rather, they represent weight-space directions that can contribute directly to token-level predictions across different contexts, and understanding how these directions interact with other components to implement higher-level, context-dependent computations is an important direction for future work. We will clarify this distinction in the updated version of the paper and avoid equating token-level functional selectivity with complete mechanistic explanations of task-specific behavior, if present anywhere in the paper.

Response to questions:

[q1] please refer response to w1 above.

[q2] please refer response to w2 above.

[q3] please refer response to w3 above.

[q4] As highlighted before, we make use of SVD because it provides a principled orthogonal decomposition of the OV and MLP output matrices into rank-one maps, separating fixed residual-stream directions from their input-dependent scalar coefficients (also see equation in line 147). This provides a natural coordinate system for identifying and intervening on individual weight-defined directions. As explained before, regarding the orthogonal basis, our experiments demonstrate that predictive effects are strongly concentrated within a small subset of these SVD directions across 21 models spanning five architecture families.

Reviewer_4pX7OpenReview ↗

Official Comment by Reviewer 4pX7

I thank the authors for their thoughtful responses.

I appreciate the clarifications regarding W1. However, I remain unconvinced that extending the analysis to additional weight matrices and introducing receptors as a finer-grained unit of analysis sufficiently distinguish this work from existing SVD-based approaches, particularly Ahmad et al. (2025). The authors have highlighted differences in scope and experiments, but the fundamental methodological and conceptual novelty remains unclear.

I also appreciate the clarifications regarding W2 and W3, which address most of my concerns. Regarding Q4, I acknowledge the explanation of why SVD provides a natural basis for this analysis. However, demonstrating sparsity within the SVD basis does not establish that this property is specific to SVD. Without comparisons to alternative orthogonal bases, this question remains unresolved.

Overall, while the responses have clarified several aspects of the paper, my main concerns regarding novelty and the supporting evidence remain. I therefore maintain my initial recommendation of weak rejection.

I thank the authors again for their efforts and constructive engagement.

We thank you for engaging in the discussion and for understanding the distinction between our work and that of Xue and Andrzejak (2026). We respectfully believe that the distinction between our work and Ahmad et al. (2025) deserves further clarification. While both works employ SVD and investigate the functional significance of singular directions, shared mathematical tools do not imply shared scientific objectives/contributions/findings.

We would like to clarify that Ahmad et al. primarily investigate the functional decomposition of transformer components and whether singular directions capture distinct computational sub-functions. Whereas our work asks a fundamentally different question, i.e., Is predictive structure inherently organized within the native weight space of language models, and how is this structure distributed, developed, and internally organized?

We would also like to mention that our contribution goes substantially beyond identifying or intervening on singular directions, specifically, we investigate:

  • Predictive selectivity and concentration, how individual directions causally influence target-token probabilities, and whether a small subset can recover substantial predictive influence through selective retention and removal (not studied in the past).
  • Weight-to-vocabulary alignment, whether functional selectivity can be partially predicted directly from the alignment of weight-space directions with vocabulary embeddings.
  • Training-time emergence and stability, how target-sensitive directions emerge across layers and become increasingly stable during training (tracing the emergence of receptors).
  • Internal vocabulary geometry, we perform a kurtosis-based optimization procedure to uncover vocabulary-selective channels within individual receptors, revealing finer-grained vocabulary structure beyond their overall target-token alignment.
  • Broad empirical characterization, we investigate these properties across 21 models (5 architecture families), using complementary settings including 1) controlled retrieval, 2) model-generated contexts, 3) MMLU, 4) targeted factual prompts, 5) free generation, and 6) perplexity evaluation (Ahmad et al. only studied GPT-2 on synthetic datasets).

All these findings/experimental designs are completely new, and we strongly believe all these investigations establish several distinct empirical properties of weight-space predictive structure that are not the primary subject of Ahmad et al. In particular, our work connects target-conditioned causal effects with their concentration, 2) emergence during training, and 3) internal vocabulary organization, and, the contribution is not just an extension to additional weight matrices or architectures, but a different prediction-centric investigation with distinct research questions, experimental designs, and findings.

We are also happy that the clarifications regarding W2 and W3 were able to address most of your concerns, and we would be happy to implement/perform any specific suggestions regarding an alternate orthogonal basis that would strengthen our work. We believe that SVD, which provides a natural basis for this analysis, has attracted recent research interest in the interpretability community; we thank you for acknowledging this.

We respectfully maintain that the novelty of this work should be assessed in terms of the new empirical knowledge it establishes (as per the reviewer guidelines), rather than solely through the mathematical tools it shares with prior work.

We hope the above clarifications will help in reconsidering our work. We are pleased to see your involvement in the discussion, we will be happy to answer any other questions/clarification you have regarding the novelty and the supporting evidence.

Reviewer_6RFkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper approaches interpretability through unpacking the model's parameters, as opposed to the model's intermediate activations on inputs. It introduces a specific parameter-space interpretability method based on singular value decomposition (SVD). This method helps the authors identify linear directions in weights, that they call logit receptors, which upweight logits of specific tokens. The authors show that logit receptors only emerge after training, ruling out the alternative hypothesis that logit receptors are a feature of the transformer architecture. They also show that logit receptors have a causal effect on token output: increasing the magnitude of the linear direction increases the target token's logit. Finally, the authors also show, using kurtosis analysis, that these logit receptors activate for multiple related concepts, such as "Paris" and "France". This work suggests that trained transformers have structure that can be read directly from their parameters, without considering their activations on inputs.

Critical strengths
  • The authors introduce a novel technique for an under-explored area of interpretability, focusing on decomposing the static parameters of the model to understand its internal structure.
  • The paper includes thorough analyses, including a detailed and well-outlined appendix.
  • The finding of logit receptors opens avenues for future research into the structure in transformer parameters.
Critical weaknesses
  • The paper doesn't include an AI Use Statement, which is required.
  • The figures are hard to read (small fonts, unclearly labeled axes, small gridlines/points in scatter plots). This is likely a minor concern, since the authors will have more space in the final submission.
Questions for authors
  • The kurtosis analysis is interesting, in that it shows that logit receptors capture related concepts. One alternative is that the co-activated concepts are due to similar tokens in the model's unembedding matrix. For instance, since logit receptors increase the probability of outputting "Paris", they might also increase the probability of other tokens that share a high similarity to "Paris" in the model's unembedding matrix, such as "France". How does the co-activity of logit receptors on different concepts such as "Paris" and "France" relate to those tokens' similarities in the embedding matrix?
  • Since logit receptors write a direction to the residual stream that increases a token's output probability, they might explain the success of logit lens, which applies the unembedding matrix at earlier layers to decode the "intermediate output" formed at those layers. Could we explain the emergence of such an "intermediate output" in logit lens through the activity/placement of logit receptors in the model's parameters?
  • Upon a closer look, it seems that logit lens is plotted in Figure 4, but never mentioned anywhere in the paper (not even the figure's caption). It's a little hard to infer from the plot, but it seems like logit lens potentially matches the predictions when keeping 0.05% of receptors. Could you please provide more detail on this experiment?
  • Small comment: I like the color-coding in the algorithms, it adds a lot of clarity!
3 comments

We thank the reviewer for the detailed review of our work. We are pleased that the reviewer recognizes the potential of parameter-space interpretability as a complementary direction for understanding the internal structure of Transformer models. There are some clarifications/questions we address them below:

Response to weaknesses:

[w1]

We mention the synthetic datasets used and their procedure in Fig. 6, 8,18. We thank you for pointing this out, we will include another AI use statement separately in the paper.

[w2]

We thank you for understanding the space constraints, we will make the fonts more readable wherever necessary.

Response to questions:

[Q1]

We thank the reviewer for asking this question and providing an important alternative explanation. We agree that similarities between token unembedding vectors could contribute to correlated output effects. For instance, since the unembedding vectors of "Paris'' and "France'' are relatively similar, activating receptors that promote "Paris'' could potentially also increase the probability of "France''.

To investigate this possibility, as suggested by the reviewer, we first computed the cosine similarity between the two unembedding vectors and confirmed that it is relatively high. However, selectively activating/scaling "Paris''-associated receptors increases P(Paris)P(Paris), without correspondingly increasing P(France)P(France), suggesting that unembedding similarity alone cannot explain receptor-level selectivity.

To understand this behavior more closely, we ran another experiment to find the receptors corresponding to France'', and independently ranked all 263,168 receptors according to their effects on "Paris'' and "France'' in Llama-3-8B. We then examined the cross-token rankings of the top-10 receptors for each token. (see table in different comment below) https://openreview.net/forum?id=PflKalCfkh&noteId=3bQMJX6EG3

Importantly, these bottom-ranked receptors are not simply inactive for "France'', they have negative effects, such that activating them suppresses 𝑃(France)𝑃(France). Hence, the receptor population contains not only receptors that promote a particular token but also receptors that actively oppose other tokens, even when those tokens have similar unembedding vectors.

This becomes particularly striking in our receptor-isolation experiment. Isolating and activating the top 0.01% of "Paris'' receptors yields 𝑃(Paris)=1.0𝑃(Paris)=1.0 and 𝑃(France)=0.0𝑃(France)=0.0. In other words, despite the high similarity between their unembedding vectors, the receptor-level intervention produces an entirely token-specific outcome.

Overall, we believe these results suggest that the vocabulary-selective structure of receptors is richer than what can be explained by unembedding-vector similarity alone, and kurtosis analysis adds another interesting layer for inspecting this structure. Receptors can contain shared selectivity across related tokens, but they can also exert opposing effects, actively promoting one token while suppressing another, leading to this interesting control mechanism that is inbuilt in the model. We believe this also provides direct evidence that the co-activity patterns captured by our kurtosis analysis reflect structured receptor-level selectivity, rather than simply inherited similarities in the unembedding space.

[Q2]

We thank the reviewer for this insightful observation, we believe that the connection between receptors and the Logit Lens provides an interesting perspective on the relationship between parameter-space structure and intermediate vocabulary predictions. Also, we agree that the vocabulary-level contribution of an individual receptor can be understood through its projection onto the unembedding space. This establishes a natural connection between the two perspectives, the Logit Lens reveals vocabulary-level information present in intermediate residual representations, whereas our framework identifies weight-defined directions that contribute to constructing those representations.

[Q3]

We thank the reviewer for pointing out this omission. The Logit Lens is included in the legend of the right panel of Figure 4 (visible when zooming in). This experiment compares vocabulary-level predictions obtained through receptor selection and isolation against those obtained from the residual stream using the Logit Lens. As the reviewer correctly observes, the two begin to show correspondence when approximately 0.05% of receptors are retained.

For a more detailed visualization, we refer the reviewer to Figure 17 in the appendix. We also observe similar behavior in LLaMA-3-8B (Figure 13), while Figure 16 examines how these vocabulary-selective directions emerge during training.

[Q4]

We thank the reviewer for this constructive suggestion and are happy for the positive feedback on the color-coding used in our algorithms.

Additional experiment to find the receptors corresponding to France'', and independently ranked all 263,168 receptors according to their effects on Paris'' and France'' in Llama-3-8B. We then examined the cross-token rankings of the top-10 receptors for each token.

RankParis top-10Its rank for FranceFrance top-10Its rank for Paris
1L31.H7.SV09L31.H7.SV1262,649
2L31.H7.SV3881L24.H3.SV89
3L24.H3.SV123L24.H3.SV123
4L24.H3.SV34L24.H3.SV34
5L31.H7.SV14262,053L24.H3.SV2522
6L31.MLP.SV1293261,288L24.H3.SV616
7L31.MLP.SV998262,981L31.H7.SV10512
8L31.MLP.SV743415L31.H7.SV1224
9L24.H3.SV82L31.H7.SV01
10L31.H7.SV562,399L31.H7.SV9810

The results reveal an interesting combination of shared and opposing selectivity. On the one hand, four receptors overlap between the two top-10 lists, suggesting that some receptors indeed respond similarly to both tokens. On the other hand, several of the top Paris'' receptors are ranked near the very bottom for France''. For example, L31.H7.SV14 ranks 5th for Paris'' but 262,053rd for France'', while L31.MLP.SV998 ranks 7th for Paris'' but 262,981st for France''.

I appreciate the additional analysis and comparison to Logit Lens. These additions help me better understand the contribution of this paper. I maintain my decision to accept this submission.

Comments on the paper

1 comment

We thank all the reviewers for their detailed-reviews/constructive-feedback, and valuable suggestions on clarity. We are pleased that the reviewers recognized the novelty [Reviewer 6RFk] and potential of exploring parameter-space interpretability as a complementary direction for understanding the internal structure of Transformer models/architectures [Reviewer 6RFk, Reviewer 4pX7, Reviewer ZcZD]. We are also happy that the reviewers appreciated the broad empirical evaluation (across 21 open-weight models) [Reviewer 4pX7, Reviewer ZcZD], the prompt-independent nature of the weight-based analysis [Reviewer ZcZD], and the investigation of the emergence/stability/predictive selectivity of receptors [Reviewer 6RFk, Reviewer 4pX7]. Moreover, the reviewers found the proposed analysis informative, with interesting findings on the relationship between learned weight-space structure and vocabulary-level predictions [Reviewer 6RFk, Reviewer ZcZD].

There were some concerns regarding the distinction from prior SVD-based interpretability approaches [Reviewer 4pX7, Reviewer ZcZD], the interpretation of direct causal effects and receptor isolation [Reviewer 4pX7, Reviewer ZcZD], and the presentation clarity [Reviewer 6RFk]. We thank the reviewers for raising these important points, in the rebuttal we have provided additional clarifications and experiments, including further analysis (adding two additional experiments) of 1) token-specific receptor selectivity, 2) comparisons with Logit Lens, and 3) experiments addressing potential selection bias in receptor isolation. We are happy that these clarifications helped address several of the initial concerns as acknowledged by all the reviewers, with Reviewer 6RFk explicitly acknowledging the additional analysis and leaning towards an acceptance. We hope incorporating the suggestions made by the reviewers (with additional analysis), will help enhance the presentation quality of our work, making the presentation and clarity on the contributions of our work, better.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.