- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Prior work indicates that many concepts can be represented using linear or low-dimensional subspaces. This work argues that LLMs explicitly use these subspaces in ICL, ie that ICL is identifying a direction.
To make this argument, they consider an ICL task where query and answer are single words (single tokens? not 100% sure). They claim that (the latent representation of) the answer is a linear function of (the latent representations of) the query and separator tokens.
The authors do find that, for the pairs they look at, the data live on a low-dimensional subspace, and that there is a roughly linear relationship between input and output. This is consistent with the literature on steering/linear subspaces.
However... this doesn't necessarily mean that ICL is specifically learning to operate along these subspaces. As far as I can tell, this is all consistent with there being linear dimensions for concepts, and the LLM not exploiting that at all in ICL. Some things (a non-exhaustive list) the authors might have looked at include:
- Is the linear relationship stronger in the ICL setting vs words embedded in isolation? Ie, is the ICL setting explicitly giving contextual embeddings that are more linearly aligned than non-contextual embeddings?
- Is the model more likely to "fail" to predict examples that are less aligned with the linear relationship? (whether this is examples within a task that do not lie on the line, or a class of examples that have a more non-linear relationship)
- Does having in-context examples of the linear relationship help more than having a textual description of the problem?
- Can we find adversarial examples that are correct but not linearly aligned, and does including them in context lead to a drop in accuracy?
I'm a little confused about what is going on in 3.5. The actual s token is always ":", so its non contextual embedding doesn't change -- all that changes is its context. So, I assume that the perturbation is actually changing the part caused by the query (and/or the other ICL examples). So (please forgive wild oversimplification...) if we have "big : small, short : tall, sharp : blunt, hot :" and we perturb the representation of the last ":", we can't be meaningfully changing from ":" to say "~", instead we are maybe changing from ": that is after hot" to say ": that is after cold", or ": that is taking the role of separator" to ": that is a taking the role of query", or ": that is part of a list of opposites" to ": that is part of a list of synonyms". I don't see how any of these changes support (or otherwise) the hypothesis. Changing the ICL pattern changes the output... but wouldn't we expect that?
Critical strengths
- Understanding how in-context learning works is an important problem.
Critical weaknesses
- We already know that many concepts can be represented using linear or low-dimensional subspaces of the residual stream
- I do not see any evidence that shows that these subspaces are specifically used in ICL
Questions for authors
I found the paper hard to read, which means maybe I missed something. Can you explain to me what you feel this work shows about the link between ICL and linear subspaces?
5 comments
Response to Reviewer bvS9 (part2)
Response to “Is the model more likely to "fail" to predict examples that are less aligned with the linear relationship? (whether this is examples within a task that do not lie on the line, or a class of examples that have a more non-linear relationship)”
We tested this possibility using the perturbation experiment in our study, in which the separator tokens are perturbed in the subspace. By perturbing the separator tokens, we effectively misalign the the query and separator tokens, as suggested.
Response to “Does having in-context examples of the linear relationship help more than having a textual description of the problem?”
The reviewer’s question raises an interesting possibility, but it is almost impossible to quantify how much information is encoded or precisely identify what type of information is encoded in textual descriptions due to the ambiguity of human languages. ICL setting is considered an effective way to convey contextual information with minimal ambiguity, as shown in the literature. This is why we use ICL setting in this study.
Response to “Can we find adversarial examples that are correct but not linearly aligned, and does including them in context lead to a drop in accuracy?”
We believe it is possible to craft adversarial examples that can change the relationships between tokens in subspaces. Based on the fact that our perturbation experiment replicate the effects of adversarial examples targeting linear relationships between the tokens, we think the adversarial examples that reduce the linear relationships likely lower LLMs’ accuracy.
It is still not clear whether it is possible to craft adversarial examples that can perturb linear relationships in subspaces and also keep LLMs’ answers intact. That would be an interesting adversarial attack problem on its own, but at this time, we believe such attack may be beyond the scope of our study because this type of adversarial examples may utilize other subspaces to keep LLMs’ answers intact. More specifically, even if we craft the adversarial examples targeting subspace while keeping LLMs’ answers intact, the adversarial examples may use a different subspace to keep LLMs’ answers intact, which would limit the utility of adversarial examples as a way to probe functions of subspaces.
As the discussion period is ending, we would like to ask the reviewer if there is any additional concern or question that should be addressed. Thank you again for your efforts.
Response to Reviewer bvS9 (part1)
We would like to thank the reviewer for reading our manuscript and providing valuable feedbacks and questions.
Response to “We already know that many concepts can be represented using linear or low-dimensional subspaces of the residual stream”
Our study has been inspired by the linear representation hypothesis (LRH) suggesting that LLMs encode high level concepts with linear directions, but we are not testing if the “concepts” are encoded with linear directions. Instead, we are seeking potential mechanisms by which LLMs use subspaces to support ICL. Inspired by LRH, we studied how transformer blocks generate subspaces and use them for decision-making. Our analysis supports that subspaces can be the computing units for LLMs. More specifically, our findings are two folds. First, each ICL token is encoded with multiple directions. That is, the tokens are encoded in subspaces spanned by these directions. Second, in the subspaces, the tokens are clustered according to the token types, and vector algebra can explain the relationships between the tokens. This means that ICL tasks can be solved using simple operations in some ``functional'' subspaces, where tokens’ directions are directly associated with the task information (i.e., query, separator and answer in our study), not just their semantic meanings alone. To the best of our knowledge, this type of subspaces has not been reported yet.
Response to the comment on section 3.5.
In Section 3.5, we perturb only the last separator in 5 intermediate layers to disrupt the linear relationships between the last separator and query tokens, qlast and alast while keeping all the relevant contexts and task information intact. If LLMs use αqlast+βslast+γ in some subspaces to predict the final answer, which is our hypothesis, perturbing slast would disrupt LLMs’ answers. This has been supported by our perturbation experiment. When we perturb the top-3 components with the highest R2, LLMs change their answers.
Further, when we perturb the bottom-3 components with the lowest R2, LLMs answers largely remain intact. This result indicates that perturbation alone does not change any important task format (i.e. ICL pattern) or prevent LLMs from solving ICL.
Response to “I do not see any evidence that shows that these subspaces are specifically used in ICL”
We performed the perturbation experiment to ask if the subspaces identified by linear regression analyses have any impact on LLMs’ decision-making process. Our results in section 3.5 clearly show that LLMs’ decision-making processes are strongly disrupted when the subspaces are perturbed. This perturbation experiment does provide strong evidence for the subspaces’ critical roles in supporting ICL.
Response to “Is the linear relationship stronger in the ICL setting vs words embedded in isolation? Ie, is the ICL setting explicitly giving contextual embeddings that are more linearly aligned than non-contextual embeddings?”
Earlier studies of ICL have found that LLMs accuracy improves, as more examples are given in the input prompt. Thus, we expect the ICL setting to be ideal enough to provide contextual information to LLMs.