- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper hypothesizes that LMs perform number comparison via computation among linear representations and propose a representation-based interpretation of the computation algorithm. The paper demonstrates that their interpretation is highly consistent with model's actual behaviors through causal intervention experiments.
Critical strengths
The paper identified vector directions for each number in an LM's number comparison task and discovered that such a subspace formed by these vectors is linearly separable according to the LM's comparison results. They use multiple causal intervention experiments to back up their hypothesis. The result is non-trivial and interesting.
Critical weaknesses
- The paper only use a predefined language template to conduct all experiments. It is unclear if the interpretation still works under other natural language templates.
- The procedure of how to discover v1 and v2 is unclear, which is important to the core soundness of the paper.
- Page 6., Line 316-318: "This observation is nontrivial because v2 is obtained in layer 13’s residual stream and v1 is from head H14’s output (which itself uses the layer 13 residual stream in computation), making nonlinear interactions possible." is confusing.
- Does the low IIA of v1 for y1 in Figure 2(f) suggest the weak dependency between comparison computation and v1?
I will increase the score if the authors address my questions.
Questions for authors
See the weakness above.
2 comments
The procedure of how to discover v1 and v2 is unclear, which is important to the core soundness of the paper.
Thank you for pointing this out. We clarify the procedure used to obtain v1 and v2 below. Further details are included in Appendix B.3. We will also include these descriptions in the main paper for greater clarity.
We obtain v1 using one-dimensional (1-d) distributed alignment search from Geiger et al 2024 (DAS) at the outputs of the copy head H14 in layer L14. 1-d DAS optimizes for a direction v1 in H14’s output space, such that the projection onto this direction has maximum causal effect on model logits. This means that performing activation patching along this direction, while keeping the remaining components of the activation fixed, changes the model’s logits toward the patched answer position.
We obtain v2 as the first principal component (PC1) of the residual stream activations at layer L13 in Qwen. We observe that this direction encodes the second number y2 monotonically, and patching along this direction has a significant causal effect on the model’s logits.
For models where the first principal component (PC1) does not show causal effect, as seen in Llama-3.1-8B and Gemma-2-9B-it (Appendix C.9), we obtain v2 using a modified version of DAS which incorporates number encoding (proportional to log of the number) along with causal effect on model outputs (details in Appendix C.9).
Page 6., Line 316-318: "This observation is nontrivial because v2 is obtained in layer 13’s residual stream and v1 is from head H14’s output (which itself uses the layer 13 residual stream in computation), making nonlinear interactions possible." is confusing.
Thank you for raising this issue. We have added a Figure 2 to the main paper to illustrate the key components used in the comparison algorithm. We rephrase the lines 316-318 below: We observe that patching the rank-two shared representation space (span(v1,v2)) has a similar causal effect as patching the directions v1 and v2 individually using rank-one patches. Since v2 is obtained in layer 13’s residual stream, which feeds into head H14 of layer L14, v2 is an input to H14. Therefore, v2 can affect the outputs of H14, including v1, but patching the shared space shows that v1 and v2 behave independently in the two-dimensional space span(v1, v2). We have included this rephrasing in our updated manuscript.
Does the low IIA of v1 for y1 in Figure 2(f) suggest the weak dependency between comparison computation and v1?
Thank you for the question. While the IIA for v1 (0.46) is individually lower than IIA for u (0.94), v1 is significantly involved in subsequent steps of the comparison algorithm. The shared (v1, v2) plane has a high IIA (0.81) when y1 is perturbed. Thus, v1 is used by the model in computation, but seems to have a low IIA individually. Potential reasons for low IIA of v1 itself include the small non-monotonicity of y1 along v1 (for y1 less than 20) and higher dimensional structure at this location. It must also be noted that the baseline IIA for a random direction in the representation space is ~0, since nonzero IIA requires the model’s logits to shift such that the largest logit moves to the answer expected after patching. Therefore v1’s IIA of 0.46 is significantly higher than a random baseline. We observe high IIA scores for v1 and v2 in additional results on other models (see Appendix C.9): for Gemma-2-9B-it, v1 has IIA 0.92 and v2 has IIA of 1, while Llama-3.1-8B has v1 IIA 0.62 and v2 IIA 0.96.
Summary: We thank the reviewer for their constructive comments and feedback on writing clarity. We hope our responses and additional experiments have addressed their questions satisfactorily, and we would greatly appreciate it if the reviewer considered raising their score. We would be happy to answer any further questions during the rebuttal period!
We thank the reviewer for their feedback! We are glad that the reviewer found our paper “clear and easy to follow”, and thought our results were “non-trivial and interesting”. We appreciate the reviewer for highlighting our main contributions: 1) number comparison is performed using linear representations, 2) we identify the computation algorithm using the representations, 3) we demonstrate that the algorithm captures the model’s behaviors through causal intervention experiments.
We address specific questions in our responses below.
Thank you for raising this important point. We have now performed additional experiments on multiple language templates (prompt formats). Across different templates, we find that the same algorithm is used by the model for comparison, with many of the same model components–attention head H14L14 copies the first number (App. C.8 Figure 44), the same two-dimensional (v1,v2) plane enables comparison (App. C.8 Figure 45, 42), MLP neurons in L14, L15 perform comparison (App. C.8 Fig 46). While the directions u, v1, v2 also work as-is for some prompts, we note that they require refitting in other cases–the refit directions are rotated versions of the original directions. Even when we change the prompt from natural language to code (P4), the same algorithm (with refitted components) works! All the supporting figures and tables for this analysis are included in Appendix C.8.
The prompt templates used in our new results are stated below:
Table. The five prompt templates, shown with y₁ = 37 and y₂ = 58. P0 is the prompt used in the paper.
Answer in the following format with a single answer. The maximum of 12 and 4 is 12. The maximum of 37 and 58 isReply with one number only. The maximum of 12 and 4 is 12. The maximum of 37 and 58 isGive only the answer. The larger of 12 and 4 is 12. The larger of 37 and 58 isWhich number is bigger, 12 or 4? Answer: 12. Which number is bigger, 37 or 58? Answer:max(12, 4) = 12<br>max(37, 58) =The interchange intervention accuracy (IIA) for different model components across the five prompt templates is stated below. We first describe the transfer accuracy (Table 1): when the same components found from our original prompt are used for other prompts, and then refit the components (Table 2) for prompts where transfer is weak (mainly P3).
Table 1. Components fitted on the original prompt P0, applied without refitting to four reworded prompts with the same operands. IIA is measured on 400 counterfactual pairs (SE ≤ 0.025). † Below 0.8 × the P0 value.
Table 2. Components refitted on P3 and P4, the prompts where frozen transfer is weakest (Table 1). † IIA below 0.8 × the P0 value. * Below the P0 fitting noise.
In summary, across different prompt templates, the model uses the same algorithm by reusing many components found in our analysis. Major changes in prompt format require refitting the causal directions u, v1, v2, and the answer direction from L15 DAS.
(continued ...)