- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses Singular Value Decomposition to analyse the OV map of each attention head and the output matrix of each MLP block. It then measures how the left singular vectors (“receptors”) affect token probabilities. The paper claims that the receptors have sparse effects on outputs, partially predicted by weight alignment. It also studies the emergence over training, and uses kurtosis-based analysis to reveal further vocabulary structure.
Critical strengths
The SVD analysis is weight-based and independent of any particular prompt (although the ranking depends on context).
The randomly initialised GPT-2 and random receptor controls strengthen the results.
The paper applies the method to a large set of 21 open-weight models.
Critical weaknesses
Novelty: How does this work improve upon previous SVD weight analyses such as the cited works, or Millidge & Black (2022)’s “The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable”?
In the main analyses, the receptor effect on tokens considers only the direct effect via the residual stream, not the indirect effect mediated via other components. This substantially reduces the amount of the model that can be understood with this method.
Receptors are selected by which ones raise the target probability (Appendix G) and receptor isolation (Algorithm 1) removes all other directions. Given that the target is used for this selection I worry that this removes receptors contributing towards wrong answers (e.g. other multiple choice answers), making the “perfect (1.0) target probability” artificial.
Questions for authors
To test whether the receptor isolation inflates the results (“perfect (1.0) target probability”): Can you run the algorithm but select the receptors corresponding to a wrong answer, and check whether this then leads to a perfect probability for that wrong answer?
Tables 1, 7, and 8 have identical captions. Is this a typo?
4 comments
I thank the authors for their response.
I acknowledge the response to W1 and appreciate the clarification of the paper's contributions.
I appreciate the clarifications regarding W2, W3, and Q1. I believe those confirm my initial judgement of the weaknesses.
I maintain my initial recommendation for a weak rejection.
We thank you for engaging with our responses and acknowledging the clarifications. We are happy that our response could address the initial concerns. We also liked the suggestion regarding an additional experiment performed during the rebuttal, which help clarify other concerns regarding the receptor isolation, we will be adding it to the updated version of the paper.
We believe that the clarified contributions, 1) the training-time emergence and stabilization of receptors, 2) their internal vocabulary geometry through kurtosis-based analysis, and 3) the extensive evaluation across 21 models, establish a broader contribution than the initial assessment may suggest.
Given these clarifications and the additional experimental evidence, we would sincerely appreciate it if you could reconsider your assessment and whether these contributions lead to an increase in the current score. We would be happy to answer/clarify/address any remaining specific concerns that might help improve this work further.
We thank you again for your constructive feedback.
Dear Reviewer ZcZD,
Since we are near the end of the discussion phase, we would also like to point you to an additional experiment done during the rebuttal, on the Reviewer 6RFk's suggestion/question, that help provide an important alternative explanation. i.e., the similarities between token unembedding vectors could contribute to correlated output effects. For instance, since the unembedding vectors of "Paris'' and "France'' are relatively similar, activating receptors that promote "Paris'' could potentially also increase the probability of "France''.
To investigate this possibility, as suggested by the reviewer, we first computed the cosine similarity between the two unembedding vectors and confirmed that it is relatively high. However, selectively activating/scaling "Paris''-associated receptors increases , without correspondingly increasing , suggesting that unembedding similarity alone cannot explain receptor-level selectivity.
To understand this behavior more closely, we ran another experiment to find the receptors corresponding to France'', and independently ranked all 263,168 receptors according to their effects on "Paris'' and "France'' in Llama-3-8B. We then examined the cross-token rankings of the top-10 receptors for each token. (see table in different comment below Reviewer 6RFk's response ) https://openreview.net/forum?id=PflKalCfkh¬eId=3bQMJX6EG3
Importantly, these bottom-ranked receptors are not simply inactive for "France'', they have negative effects, such that activating them suppresses . Hence, the receptor population contains not only receptors that promote a particular token but also receptors that actively oppose other tokens, even when those tokens have similar unembedding vectors.
This becomes particularly striking in our receptor-isolation experiment. Isolating and activating the top 0.01% of "Paris'' receptors yields and . In other words, despite the high similarity between their unembedding vectors, the receptor-level intervention produces an entirely token-specific outcome.
Overall, we believe these results suggest that the vocabulary-selective structure of receptors is richer than what can be explained by unembedding-vector similarity alone, and kurtosis analysis adds another interesting layer for inspecting this structure. Receptors can contain shared selectivity across related tokens, but they can also exert opposing effects, actively promoting one token while suppressing another, leading to this interesting control mechanism that is inbuilt in the model. We believe this also provides direct evidence that the co-activity patterns captured by our kurtosis analysis reflect structured receptor-level selectivity, than simply inherited similarities in the unembedding space.
We hope the above added analysis strengthens the paper more, which will help in reconsidering our work. We are pleased to see your involvement in the discussion, we will be happy to answer any other questions/clarification you have regarding the novelty and the supporting evidence.
We thank the reviewer for the thoughtful/constructive and detailed feedback. We really liked the positive remarks about the presentation and the recognition of the prompt-independent nature of the weight-based analysis. We address the queries below:
[w1]
Regarding the contributions, we would like to clarify that applying SVD to Transformer weight matrices is an active area of interest and has been studied by multiple prior arts. Our contribution is more towards investigating/establishing the functional properties of the resulting singular directions (which turn out to naturally exist), which we refer to as receptors (also see the equation on line 147), and characterizing how they relate to vocabulary/prediction-level model behavior.
Overall, the range of experiments we perform is completely complementary to the prior art, both in design and in findings. We believe our work enriches our understanding of this underlying structure, opening up new avenues for future investigations.
[w2]
Regarding the direct effect, we would like to clarify that our primary objective is to characterize the immediate functional contribution associated with individual weight-derived directions (which can only be observed via direct effect). This also provides a controlled setting in which the relationship between the weight geometry and vocabulary-level effects can be studied without conflating it with subsequent transformations by other components (which also adds the additional advantage of faster interp, where a Llama-8 B model only takes 3min to find all receptors, also see Table 4 for other models). However, we do understand that direct effects do not capture the full causal influence of these directions on the final model prediction. We will clarify this distinction in the camera-ready version and avoid interpreting direct receptor effects as complete explanations of model computation (if stated anywhere). We thank you for this comment on clarification.
[w3]
We agree that (when observing only Fig. 13) it may look like, selecting receptors according to their effect on a predefined target introduces a selection bias that must be considered when interpreting the isolation results. We specifically rule out this possibility in initial experiments. We would like to point to the observations in Fig. 5 of the main paper, where we observe that even random context (random-init-context, Figure 18), leads to the prob. reaching one for a few of the receptors (notice the first subfigure Fig. 5, top green dots, prob. increasing by 10,000x). This establishes that the receptors we amplify directly affect the target probability even with random tokens in the context, making them independent. Additionally, we run the experiments, as the reviewer suggested.
[Q1]
We ran the experiment corresponding to Figure 13, where we took only the top1% receptors corresponding to token “_A” and ablated the rest. The following table shows the probability of tokens “_A” and “_B” across the Llama3-8B model layers, where MMLU questions with the correct answer as “_B” are chosen.
[Q2]
We thank you for this detailed review, yes, this is a typo, all the three are for different models, which can be seen in the index of the appendix, we forgot to update the same in the actual captions. We will fix this in the camera-ready version of the paper.