acceptodds
Under review as a conference paper at ICLR 2027

Functional Subspace, where language models can use vector algebra to solve problems

Abstract

Large language models (LLMs) were invented for natural language tasks such as translation, but they have proved that they can perform highly complex functions across domains. Additionally, they have been thought to develop new skills without being trained on them. These learning capabilities lead to LLMs adoption in a wide range of domains. Thus, it is imperative that we understand their operating mechanisms and limitations for proper diagnostics and repair. The earlier studies proposed that high level concepts are encoded as linear directions in LLMs activation space and that the geometry of embeddings have semantic meanings. Inspired by these studies, we hypothesize that LLMs may use subspaces and vector algebra in subspaces to perform tasks. To address this hypothesis, we analyze LLMs' functional modules and residual streams collected from LLMs engaging in in-context learning (ICL), one of the emergent abilities. Our analyses suggest that 1) LLMs can create subspaces, where evidence can be accumulated and 2) ICL tasks can be solved via simple algebraic operations in subspaces.

Primary area:
learning theory
Keywords:
language models, subspace, in-context learning, transformers, hidden layer activation space

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bvS9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Prior work indicates that many concepts can be represented using linear or low-dimensional subspaces. This work argues that LLMs explicitly use these subspaces in ICL, ie that ICL is identifying a direction.

To make this argument, they consider an ICL task where query and answer are single words (single tokens? not 100% sure). They claim that (the latent representation of) the answer is a linear function of (the latent representations of) the query and separator tokens.

The authors do find that, for the pairs they look at, the data live on a low-dimensional subspace, and that there is a roughly linear relationship between input and output. This is consistent with the literature on steering/linear subspaces.

However... this doesn't necessarily mean that ICL is specifically learning to operate along these subspaces. As far as I can tell, this is all consistent with there being linear dimensions for concepts, and the LLM not exploiting that at all in ICL. Some things (a non-exhaustive list) the authors might have looked at include:

  • Is the linear relationship stronger in the ICL setting vs words embedded in isolation? Ie, is the ICL setting explicitly giving contextual embeddings that are more linearly aligned than non-contextual embeddings?
  • Is the model more likely to "fail" to predict examples that are less aligned with the linear relationship? (whether this is examples within a task that do not lie on the line, or a class of examples that have a more non-linear relationship)
  • Does having in-context examples of the linear relationship help more than having a textual description of the problem?
  • Can we find adversarial examples that are correct but not linearly aligned, and does including them in context lead to a drop in accuracy?

I'm a little confused about what is going on in 3.5. The actual s token is always ":", so its non contextual embedding doesn't change -- all that changes is its context. So, I assume that the perturbation is actually changing the part caused by the query (and/or the other ICL examples). So (please forgive wild oversimplification...) if we have "big : small, short : tall, sharp : blunt, hot :" and we perturb the representation of the last ":", we can't be meaningfully changing from ":" to say "~", instead we are maybe changing from ": that is after hot" to say ": that is after cold", or ": that is taking the role of separator" to ": that is a taking the role of query", or ": that is part of a list of opposites" to ": that is part of a list of synonyms". I don't see how any of these changes support (or otherwise) the hypothesis. Changing the ICL pattern changes the output... but wouldn't we expect that?

Critical strengths
  • Understanding how in-context learning works is an important problem.
Critical weaknesses
  • We already know that many concepts can be represented using linear or low-dimensional subspaces of the residual stream
  • I do not see any evidence that shows that these subspaces are specifically used in ICL
Questions for authors

I found the paper hard to read, which means maybe I missed something. Can you explain to me what you feel this work shows about the link between ICL and linear subspaces?

5 comments
AuthorsOpenReview ↗

Response to Reviewer bvS9 (part1)

We would like to thank the reviewer for reading our manuscript and providing valuable feedbacks and questions.

Response to “We already know that many concepts can be represented using linear or low-dimensional subspaces of the residual stream”

Our study has been inspired by the linear representation hypothesis (LRH) suggesting that LLMs encode high level concepts with linear directions, but we are not testing if the “concepts” are encoded with linear directions. Instead, we are seeking potential mechanisms by which LLMs use subspaces to support ICL. Inspired by LRH, we studied how transformer blocks generate subspaces and use them for decision-making. Our analysis supports that subspaces can be the computing units for LLMs. More specifically, our findings are two folds. First, each ICL token is encoded with multiple directions. That is, the tokens are encoded in subspaces spanned by these directions. Second, in the subspaces, the tokens are clustered according to the token types, and vector algebra can explain the relationships between the tokens. This means that ICL tasks can be solved using simple operations in some ``functional'' subspaces, where tokens’ directions are directly associated with the task information (i.e., query, separator and answer in our study), not just their semantic meanings alone. To the best of our knowledge, this type of subspaces has not been reported yet.

Response to the comment on section 3.5.

In Section 3.5, we perturb only the last separator in 5 intermediate layers to disrupt the linear relationships between the last separator and query tokens, q⃗last\vec{q}^{last} and a⃗last\vec{a}^{last} while keeping all the relevant contexts and task information intact. If LLMs use αq⃗last+βs⃗last+γ\alpha\vec{q}^{last}+\beta\vec{s}^{last}+\gamma in some subspaces to predict the final answer, which is our hypothesis, perturbing s⃗last\vec{s}^{last} would disrupt LLMs’ answers. This has been supported by our perturbation experiment. When we perturb the top-3 components with the highest R2R^2, LLMs change their answers.

Further, when we perturb the bottom-3 components with the lowest R2R^2, LLMs answers largely remain intact. This result indicates that perturbation alone does not change any important task format (i.e. ICL pattern) or prevent LLMs from solving ICL.

Response to “I do not see any evidence that shows that these subspaces are specifically used in ICL”

We performed the perturbation experiment to ask if the subspaces identified by linear regression analyses have any impact on LLMs’ decision-making process. Our results in section 3.5 clearly show that LLMs’ decision-making processes are strongly disrupted when the subspaces are perturbed. This perturbation experiment does provide strong evidence for the subspaces’ critical roles in supporting ICL.

Response to “Is the linear relationship stronger in the ICL setting vs words embedded in isolation? Ie, is the ICL setting explicitly giving contextual embeddings that are more linearly aligned than non-contextual embeddings?”

Earlier studies of ICL have found that LLMs accuracy improves, as more examples are given in the input prompt. Thus, we expect the ICL setting to be ideal enough to provide contextual information to LLMs.

AuthorsOpenReview ↗

Response to Reviewer bvS9 (part2)

Response to “Is the model more likely to "fail" to predict examples that are less aligned with the linear relationship? (whether this is examples within a task that do not lie on the line, or a class of examples that have a more non-linear relationship)”

We tested this possibility using the perturbation experiment in our study, in which the separator tokens are perturbed in the subspace. By perturbing the separator tokens, we effectively misalign the the query and separator tokens, as suggested.

Response to “Does having in-context examples of the linear relationship help more than having a textual description of the problem?”

The reviewer’s question raises an interesting possibility, but it is almost impossible to quantify how much information is encoded or precisely identify what type of information is encoded in textual descriptions due to the ambiguity of human languages. ICL setting is considered an effective way to convey contextual information with minimal ambiguity, as shown in the literature. This is why we use ICL setting in this study.

Response to “Can we find adversarial examples that are correct but not linearly aligned, and does including them in context lead to a drop in accuracy?”

We believe it is possible to craft adversarial examples that can change the relationships between tokens in subspaces. Based on the fact that our perturbation experiment replicate the effects of adversarial examples targeting linear relationships between the tokens, we think the adversarial examples that reduce the linear relationships likely lower LLMs’ accuracy.

It is still not clear whether it is possible to craft adversarial examples that can perturb linear relationships in subspaces and also keep LLMs’ answers intact. That would be an interesting adversarial attack problem on its own, but at this time, we believe such attack may be beyond the scope of our study because this type of adversarial examples may utilize other subspaces to keep LLMs’ answers intact. More specifically, even if we craft the adversarial examples targeting subspace Ωk\Omega_k while keeping LLMs’ answers intact, the adversarial examples may use a different subspace Ωj≠k\Omega_{j \neq k} to keep LLMs’ answers intact, which would limit the utility of adversarial examples as a way to probe functions of subspaces.

As the discussion period is ending, we would like to ask the reviewer if there is any additional concern or question that should be addressed. Thank you again for your efforts.

Reviewer_r4PUOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors hypothesized that LLMs use vector arithmetic to solve ICL tasks. They find that LLMs create subspaces and use the prompt to accumulate evidence in the residual stream and use simple vector arithmetic over relevant/activated subspaces to solve problems.

Critical strengths

Strength 1: The paper addresses an important gap in the field, where we can definitely use a solid systematic investigation of how language models use vector arithmetic to solve ICL problems.

Strength 2: The paper is very well written and clearly explains every aspect of the problem. The line of argumentation is very clear.

Critical weaknesses

I do not see any critical weaknesses, but I do think the findings in section 3.2 need to be quantified.

You describe two findings in section 3.2: Finding 1: R-squared is lower in early layers and higher in later layers. Finding 2: The principal components with high R-squared appear largely consistent across layers. → Can you quantify these claims with statistical tests? I do not see this pattern in the figures.

Minor: The abstract mostly discusses the motivation and could be improved to better reflect the findings and implications.

Questions for authors

Clarification request: First off, I consider an ICL task (e.g., an antonym task like Examples: black -> white; top -> bottom; happy -> sad. Q: day ->) in essence being an analogy (A:B::C:?) task, where there are multiple A:B pairs presented and the query provides C and the task is to provide the solution D (i.e., night in the example above). I think that these two papers that trace the word2vec-like B-A+C algorithm (e.g., king-man+woman) in transformer-based models could be interesting for your work: → Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. (2023) Language models implement simple word2vec-style vector arithmetic. In North American Chapter of the Association for Computational Linguistics. URL https://api.semanticscholar.org/CorpusID:258887799 → Hellwig, P., Zuidema, W., Stevenson, C. E., & Lewis, M. (2026). Transformer See, Transformer Do: Copying as an Intermediate Step in Learning Analogical Reasoning. Third Conference on Language Modeling (COLM 2026). URL https://arxiv.org/pdf/2604.06501

Future work: My question is whether the regression you performed that demonstrates the linear combination of the prompt and separator essentially summing to the answer is in conflict with the B-A+C algorithm findings of the cited papers, especially the first as the architecture is similar to the models you investigated.

3 comments

We would like to thank the reviewer for reading our manuscript and greatly appreciate the detailed comments and meaningful suggestions, which we incorporated in our revision. We believe the reviewer’s comments helped us improve our study.

As the reviewer made suggestion on the quantification in Section 3.2, we have added the quantification of our observations in Fig. 3 accordingly. First, to quantify how R2R^2 evolves across layers, we identify 10 components using average R2R^2 and compute the mean values and standard deviations from them. Fig. 3A displays them as a function of layer depth. Second, we estimate the variability of 10 highest R2R^2 for all components and all models. Specifically, we compute the maximum values and standard divisions from 10 highest R2R^2 in each component. In this analysis, we display the maximum value on xx-axis and the standard deviation on yy-axis of Fig. 3B. That is, each component is mapped onto a data point in (B), and the data are dense in the left bottom, where the components have high maximum R2R^2 and low standard deviations.

As the reviewer pointed out, our analysis suggests that “A:B::C:?” tasks can be solved by linear operations in some subspaces. This finding may not be directly related to “B-A+C” tasks, but it can still be possible for the linear operations in our study to support “B-A+C”, and we have added a new Appendix A.3 to discuss this possibility. In this new Appendix A.3, we present a potential mechanism by which the self-attention input to the separator token s⃗\vec{s} may carry the information associated with “B-A+C”.

We would also like to note that LLMs can use different subspaces to perform different ICL tasks. As LLMs’ residual streams contain activations from all layers, we expect LLMs’ activation space to be decomposed into a massive number of subspaces, each of which can implement a distinct function. So, when LLMs need to perform a task (e.g., “B-A+C” or “A:B:C?”), they would need to find the right subspaces that implement correct functions. Currently, it remains uncertain how LLMs choose the correct subspaces for the given tasks, and for our next project, we plan to study the potential mechanisms underlying subspace selection.

As the discussion period is ending, we would like to ask the reviewer if there is any additional concern or question that should be addressed. Thank you again for your efforts.

Reviewer_r4PUOpenReview ↗

Excellent work, scores revised

I appreciate you adding Appendix A.3. This is very interesting work that I will certainly reference in the future. I've adjusted my scores accordingly. Good luck!

Reviewer_6cazOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I can't figure out what this paper is about.

Critical strengths

None

Critical weaknesses

The main hypothesis (and other statements) seem vague and potentially vacuous, e.g.: “LLMs may use subspaces and vector algebra in subspaces to perform tasks.” What would it mean for this to be true/false?

I would’ve expected that Section 2 would make it clear what all the discussion of “subspaces” is about, but it did not.

The language is generally too informal and imprecise for me to undertand, e.g.:

  • “LLMs can create subspaces, where evidence can be accumulated”
  • “LLMs can be mapped onto low-dimensional subspace.” I don't know what it would mean for an LLM to "create subspaces" or "be mapped onto" them.
Questions for authors

Can you please clarify what hypotheses you mean to test and how the experiments test them?

Why are q and s added together in eqn 2? This doesn't seem to accurately reflect how transformers work (subsequent tokens / their embeddings are not simply added together).

2 comments

We would like to thank the reviewer for reading our manuscript and providing feedbacks.

Response to “I would’ve expected that Section 2 would make it clear what all the discussion of “subspaces” is about, but it did not.“

Respectfully, we would like to point out that “subspace” is a widely used term in machine learning and AI communities. As it is challenging to analyze high dimensional data, multiple component analyses have been developed to find “effective” components that can capture the gist of the data in low-dimensional space. For instance, principal components analysis can project high dimensional data into a low dimensional space. In fact, top-3 components are often chosen for visualization. A line of studies suggests that the activation space of large language models (LLMs) can also be described by low-dimensional spaces, which is often referred to as “subspace”.

Response to “Can you please clarify what hypotheses you mean to test and how the experiments test them?”

We hypothesize that LLMs’ functions/computations occur in the subspaces of LLMs’ activation space (see Section 2). To address this hypothesis, we studied the representations of ICL tokens in subspaces and their relationships. The linear regression analysis and control experiments in Sections 3.2-3.4 suggest that 1) ICL tokens in subspaces are projected to functional clusters depending on the token types, and 2) they are linearly related with one another. Additionally, the perturbation experiment in Section 3.5 indicates a crucial role of subspaces in LLMs’ decision-making. These results support our hypothesis.

Response to Question “Why are q and s added together in eqn 2? This doesn't seem to accurately reflect how transformers work (subsequent tokens / their embeddings are not simply added together).”

To explain our motivation more clearly, we revised Section 2, and Eq. 2 has been removed accordingly.

Response to Question “LLMs can be mapped onto low-dimensional subspace.” I don't know what it would mean for an LLM to "create subspaces" or "be mapped onto" them.

We are referring to the following facts. LLMs can operate in a few steps. First, they tokenize words and encode them to create the embeddings of words/tokens. Second, the embeddings are high dimensional vectors, and the transformer blocks sequentially transform these embeddings. And earlier studies found that these embedding spaces can be projected onto low-dimensional “subspaces” or “manifolds”. We hope that the revised Section 2 would provide more clarification.

Response to Question ““LLMs can create subspaces, where evidence can be accumulated”

We have added a new Section 2.3 to explicitly explain what we meant by this statement and would like to provide a summary of this new section below.

By definition, self-attention layers mix multiple tokens (embeddings in residual streams). When it comes to ICL, the triplets of query, separator and answer provide self-attention inputs to the following tokens. We note that the query, separator and answer tokens can be decomposed into multiple components NkN_k, which are associated with potential next tokens. To predict the next tokens accurately, there must be overlapped components among NkN_k, which are {Nic}\{N^c_i\} in newly revised Section 2 (lines 148-201). Then, the attention inputs (AqA^q, AsA^s, AaA^a) of query, separator and answer tokens can be effectively accumulated along {NicN^c_i}.

As the discussion period is ending, we would like to ask the reviewer if there is any additional concern or question that should be addressed. Thank you again for your efforts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

First of all, we would like to thank the reviewers and the areal chair for reviewing our manuscript and providing valuable feedbacks. To address their concerns, we have revised our manuscript as follows:

  • The abstract has been edited, as suggested by Reviewer r4PU
  • We revised Section 2 to clarify our motivation (see lines 144-198). In response to Reviewer 6caz’s question, we added Section 2.2.2.
  • We added Fig. 3 to summarize our finding in Section 3.2.
  • To address Reviewer r4PU’s question, we added a new section (A.3) in the appendix.

Our point-to-point responses are provided below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.