acceptodds
Under review as a conference paper at ICLR 2027

In-Context Learning Operates as Concept Subspace Learning

Abstract

Regression and Bayesian accounts of in-context learning (ICL) explain how demonstrations can induce predictors, while mechanistic analyses often identify compact activation directions that steer prompted behavior. However, it remains unclear whether structured demonstrations induce low-dimensional concept inference. We study this question through a concept-subspace view of ICL, in which tasks vary only along intrinsic concept coordinates, although inputs are observed in a high-dimensional ambient space. For ridge and least-squares ICL proxies, prediction decomposes exactly into concept-coordinate regression and off-subspace leakage. We show that an idealized single-layer linear attention head optimal for this task family instead stores the concept subspace in its weights and reads the prompt only through it, so patching its written subspace carries the head's entire pooled clean–corrupted effect. Testing this prediction in Llama-3-8B on CounterFact-derived multi-relation prompts, we find that a -dimensional subspace of the -dimensional residual stream restores of the clean–corrupted accuracy gap, whereas patching the complementary subspace restores . Concept swaps redirect predictions toward injected relations, while random and cross-task matched-rank controls are largely ineffective. Additional experiments on Qwen2.5-7B, a controlled cross-lingual rule task, context-defined nonce mappings and two-step compositional rules show the same qualitative pattern. These results support concept subspaces as compact, task-aligned mediators of recoverable ICL behavior in structured task families, without implying full-circuit recovery. Our code is available at the following anonymous repository https://anonymous.4open.science/r/ICL-Concept-Subspace.

Primary area:
interpretability and explainable AI
Keywords:
In-Context Learning, Concept Subspace Learning, Mechanistic Interpretability, Activation Patching, Intrinsic Dimensionality

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rX3dOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies in-context learning (ICL) from the perspective of low-dimensional concept-subspace estimation. It assumes that a family of linear regression tasks shares an intrinsic parameter subspace and develops theoretical results characterizing concept-coordinate estimation, off-subspace leakage, and subspace identifiability. The analysis also establishes a lower bound for predictors without knowledge of the shared subspace. The paper then considers an idealized single-layer linear attention model and derives properties of its population-optimal solution, including a structured relationship between the attention head's write matrix and an external readout, as well as predictions about low-dimensional causal mediation and selective readout. Empirically, the authors extract task-related subspaces from pretrained Transformer residual streams using SVD of hidden-state/target-output cross-moments. Through activation patching, concept-subspace swapping, complementary-space interventions, and matched-rank controls, they show that relatively low-dimensional subspaces can recover substantial portions of ICL performance and redirect predictions. The experiments cover factual relations, cross-lingual translation, context-defined mappings, and compositional tasks, using Llama-3-8B and Qwen2.5-7B.

Critical strengths

1. A systematic theoretical framework connecting low-dimensional task estimation to in-context learning.

The paper provides a relatively comprehensive theoretical treatment of ICL under a shared concept-subspace assumption. Lemma 1 and Theorem 1 decompose ridge-based ICL into concept-coordinate estimation and off-subspace leakage, clarifying how the intrinsic task dimension and nuisance covariance affect prediction. Proposition 2 further establishes a lower bound for subspace-agnostic predictors, illustrating the statistical advantage of knowing the shared concept subspace. Theorem 2 extends this analysis to an idealized attention model and derives properties of its optimal write–read mechanism and the resulting hidden-state subspace.

The theoretical results are generally well motivated and supported by detailed derivations. In particular, the use of Gaussian moment calculations and symmetry arguments to characterize the population-optimal attention mechanism is mathematically interesting. Although the assumptions are restrictive, the analysis provides a coherent framework connecting statistical estimation, attention computation, and causal subspace interventions.

2. Evidence for low-dimensional subspace recovery across different task settings and intervention conditions.

The experimental evaluation examines low-dimensional subspace recovery from several complementary perspectives rather than relying solely on linear probing. On CounterFact relation tasks, the authors use activation patching, concept-subspace swapping, complementary-space interventions, and matched-rank controls to assess the causal relevance of the extracted directions. The results show that a relatively small residual-stream subspace can recover a substantial portion of the clean–corrupted performance gap and redirect predictions toward alternative task relations.

The appendix further explores this phenomenon in different task settings, including cross-lingual translation, context-defined symbol mappings, and two-step compositional rules, as well as experiments on Qwen2.5-7B. I find the compositional rule experiments particularly interesting because they investigate whether subspace-based interventions can influence behavior requiring the combination of multiple mappings, rather than merely recovering a single factual relation. Together, these results provide useful evidence that low-dimensional causal mediation can arise under different task structures and intervention conditions, although they do not by themselves establish a common underlying computational mechanism.

Critical weaknesses

1. The theoretical model relies on highly restrictive architectural assumptions, and its write–read mechanism is not connected to actual Transformer components.

Theorem 2 characterizes the population-optimal solution of an idealized single-layer linear attention model. Its central result is that the attention head's effective write matrix WW and an additional learnable readout matrix Ω\Omega satisfy ΩW=UΓcU⊤.\Omega W=U\Gamma_cU^\top. This describes a specific computational mechanism: the attention head aggregates demonstration statistics and writes them into the hidden state, while the readout extracts concept-aligned information to make predictions. Importantly, the readout is explicitly defined as f(P)=xq⊤Ωh(P)f(P)=x_q^\top\Omega h(P), rather than derived from a standard Transformer architecture.

However, Assumption 2 introduces several strong architectural simplifications. Attention scores are computed using a fixed, query-independent vector κ\kappa, whereas standard Transformer queries are dynamically computed from contextualized hidden states. Each demonstration is represented as a single vector ei=(xi,yi)e_i=(x_i,y_i), abstracting away the multi-token structure of natural-language demonstrations and the computations required to encode their inputs and outputs. The model also omits softmax normalization, assumes that the attention output lies in a subspace orthogonal to the baseline query representation, and employs the specially designed bilinear readout mentioned above. These assumptions directly facilitate the separation between demonstration-dependent task statistics and query-dependent prediction.

Furthermore, Section 4 does not investigate the corresponding write or read operations in pretrained Transformers. The experiments identify a low-dimensional subspace in the query-token residual stream and demonstrate its causal influence through activation patching. Since the residual stream aggregates contributions from many attention heads and MLPs, these observations do not establish that the model implements the computational mechanism characterized by ΩW=UΓcU⊤\Omega W=U\Gamma_cU^\top. The paper does not identify which components write to and read from the extracted subspace or whether the relevant attention heads exhibit the computational properties derived in Theorem 2.

Consequently, although the theoretical result is valid within its specified model, it remains unclear how this specific mechanism would arise in a standard multi-layer softmax Transformer. The experiments establish low-dimensional causal mediation without directly validating the theoretical write–read computation.

2. The empirical tasks are not connected to the theoretical task family, and the experiments do not directly verify several of the theorem's distinctive predictions.

The theoretical analysis assumes a Gaussian linear regression task family:

w_\tau=U\beta_\tau,\qquad y=x^\top U\beta_\tau,$$ where all tasks share a fixed low-dimensional concept subspace $U$, and task variation is entirely captured by $\beta_\tau$. In contrast, the experiments involve discrete factual relations from CounterFact, cross-lingual translation, context-defined symbol mappings, and compositional rules. It is unclear how these experimental tasks correspond to the assumed generative model. For example, for a country-to-capital relation, the paper does not specify what constitutes the input vector $x$, how a discrete output token corresponds to the scalar label $y$, or what empirical evidence supports a shared linear parameter subspace across different relations. The reported transfer from 24 seen to 10 unseen relations suggests some generalization of task-relevant directions, but does not establish that these relations share the particular low-dimensional structure assumed in the theory. More importantly, several distinctive predictions of Theorem 2 are not directly evaluated. Theorem 2(v) distinguishes the presence of decodable information from its selective use in prediction. The corresponding prediction requires the concept subspace to exhibit recovery substantially exceeding its share of activation-difference energy: $\operatorname{Rec}(\hat{\mathcal{W}})\gg E(\hat{\mathcal{W}})$. However, the experiments do not report $E(\hat{\mathcal{W}})$. As the authors acknowledge in Appendix C.3 and the Limitations section, outperforming matched-rank random subspaces does not establish selective readout. A subspace may produce a large intervention effect simply because it contains a disproportionate fraction of the clean–corrupted activation difference, even under an unstructured readout mechanism. Furthermore, the theoretical recovery metric is defined through pooled prediction differences: $\operatorname{Rec}(V)=\frac{\mathbb{E}[(f_V-\tilde{f})(f-\tilde{f})]}{\mathbb{E}[(f-\tilde{f})^2]},$whereas the empirical evaluation primarily reports recovery of the top-1 accuracy gap. These quantities are not equivalent, and the theoretical additivity property does not generally hold for accuracy-based recovery. Indeed, the empirical concept and complementary recovery rates do not consistently sum to one. Other theoretical predictions, including the approximately collinear task centroids in the complementary subspace and the structure of centered task-level representations, are also not directly evaluated.
Questions for authors

Clarification Questions

1. How does the proposed subspace identification and intervention approach differ from existing work on causal subspace patching and task representations?

The extracted concept subspace is obtained through SVD of H⊤YH^\top Y, where YY consists of correct-answer unembedding vectors. This procedure explicitly favors directions associated with target outputs. What is the novel methodological contribution or mechanistic insight beyond identifying output-predictive directions and demonstrating their causal influence through activation patching? How do the authors distinguish the extracted task-level concept representation from more generic answer-related representations?

Additionally, Section 4.2 states that the leading left singular directions of H⊤YH^\top Y are equivalently obtained from the weight matrix of a fitted linear probe. However, ordinary least-squares regression yields B=(H⊤H)†H⊤YB=(H^\top H)^\dagger H^\top Y, whose leading left singular subspace is generally different. Under what assumptions or probe-fitting procedure does the claimed equivalence hold?

2. What shared low-dimensional structure underlies the 24 seen and 10 unseen CounterFact relations?

Could the authors specify which relations belong to each group and how the split was determined? More importantly, what common low-dimensional structure is shared by these relations, and how does it correspond to the concept subspace assumed in the theoretical model? Without characterizing this structure, how should the reported transfer performance on unseen relations be interpreted?

3. Could the authors clarify the experimental baselines, subspace estimation procedures, and inconsistent dimensionality reports?

Section 4.1 describes corrupted contexts as demonstrations with shuffled labels, whereas Figure 2 labels the baseline as zero-shot (40.0%), and Figure 1 reports a corrupted baseline of 41.0%. Are the intervention recipients genuinely zero-shot prompts or prompts containing corrupted demonstrations, and how do the different baselines relate? Furthermore, Appendix B.2 states that the selected subspace rank decreases from 69 to approximately 61–62, whereas Figure 3 shows values decreasing from approximately 78 to 71–72. Other experiments report dimensionalities such as 37 and 62. What accounts for these differences? Also, Appendix B.1 describes subspace estimation using either hidden-state/target-unembedding cross-covariance or counterfactual activation differences. Which estimator is used in each experiment, and are the reported subspaces directly comparable across these settings?

Questions Regarding the Mechanistic Conclusions

4. What evidence supports the selective readout mechanism predicted by Theorem 2(v)?

Without reporting the activation-difference energy E(V)E(V) or demonstrating a substantial difference between Rec⁡(V)\operatorname{Rec}(V) and E(V)E(V), how can the current results distinguish structured concept-subspace readout from an unstructured mechanism in which the identified directions simply contain a large fraction of the relevant activation-difference energy?

5. Do the actual attention heads associated with the extracted concept subspace exhibit the computational properties characterized in Theorem 2?

In particular, is there evidence that the relevant heads perform label-dependent aggregation of demonstration information, construct a query-independent task representation, and support a downstream readout analogous to ΩW=UΓcU⊤\Omega W=U\Gamma_cU^\top? If these properties have not been established, what justifies interpreting the theoretical single-head computation as an explanation of the observed residual-stream interventions?

Reviewer_BBUoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether task information in in-context learning is represented in a low-dimensional concept subspace, connecting statistical accounts of ICL with causal analyses of internal representations in language models.

Theoretically, the authors assume that task parameters share a low-dimensional subspace. They decompose ridge-regression and least-squares predictions into concept-coordinate estimation and off-subspace leakage, and analyze the statistical cost of not knowing the subspace. For an idealized single-layer linear attention head, they show that shared task structure can be stored in the model weights, while in-context demonstrations provide information for estimating task coordinates. Patching the corresponding internal subspace captures the entire pooled clean–corrupted effect.

Empirically, the authors study Llama-3-8B and Qwen2.5-7B. They extract low-dimensional residual subspaces using activation–target associations and SVD, then evaluate them through clean-to-corrupted patching, cross-relation swapping, and rank-matched controls. In the main factual-relation experiments, the extracted subspaces recover most of the clean–corrupted accuracy gap in both models, while complementary subspaces and random controls have substantially weaker effects. Cross-lingual rules, context-defined nonce mappings, and two-step compositional tasks exhibit similar patterns, although recovery rates vary.

These findings support the view that compact, task-aligned subspaces carry substantial recoverable ICL signal in the structured tasks studied. However, the experiments do not clearly distinguish query-independent task states from query-specific answer states, or establish how the joint subspace divides into shared and task-specific directions. The correspondence between the theoretical mechanism and the behavior of real models therefore requires further verification.

Critical strengths
  1. The theory connects low-dimensional task structure with testable predictions about internal interventions. Beyond proposing a low-dimensional representation hypothesis, the paper analyzes concept estimation, off-subspace leakage, and the statistical cost of an unknown subspace. Within an idealized attention model, it derives properties concerning subspace mediation, cross-query transfer, and the distinction between decodability and causal use. This provides explicit mechanistic predictions that go beyond post-hoc dimensionality reduction of activations

  2. Controlled experiments provide substantial evidence for low-dimensional intervention effects. In Llama-3-8B and Qwen2.5-7B, patching a small set of task-aligned directions recovers 78.8% and 83.7%, respectively, of the clean–corrupted accuracy gap in the main factual-relation experiments. Complementary subspaces, rank-matched random subspaces, and unrelated-task subspaces exhibit much weaker effects. Experiments on cross-lingual rules, nonce mappings, and compositional tasks extend the scope of the findings and support the interpretation of these subspaces as compact mediators of recoverable ICL behavior.

Critical weaknesses
  1. The experiments do not clearly distinguish the query-independent task state predicted by the theory from a query-specific answer state. In the core recovery experiments, the query is held fixed while late-layer activations at the final input-token position are patched from a clean run into a corrupted run. Recovery could therefore reflect either the transfer of a task rule or the transfer of an already-computed answer representation. Answer-supervised subspace extraction, random-subspace controls, and the reported relation-swapping results do not resolve this ambiguity. Although the authors acknowledge the issue in Appendix C.3, evidence for cross-query task transfer remains insufficient. Consequently, the experiments establish a low-dimensional mediator of output behavior more directly than they establish the task state characterized by the theory.

  2. The shared structure across tasks within the joint subspace remains insufficiently characterized. Pooling multiple relations to extract a single low-dimensional space could recover shared directions, task-specific directions, or a combination of both. Aggregate recovery rates cannot distinguish whether tasks use the same directions or different portions of the joint space. Transfer to unseen relations provides some evidence of sharing, but does not establish the relative contributions of shared and task-specific components. This does not invalidate the existence of a low-dimensional joint space, but limits the interpretation of its cross-task structure.

Questions for authors
  1. How are donor and recipient queries paired in the swapping experiments? Do any experiments explicitly use different queries for which the donor answer differs from the recipient’s target answer under the donor task, e.g., France → capital injected into Japan → language? This would help distinguish whether the subspace encodes task information or the answer itself.

  2. To what extent is the joint subspace shared across tasks, rather than composed of shared and task-specific directions? Have you separately estimated candidate subspaces for individual relations and compared their geometric overlap and cross-task patching effects?

Reviewer_qDUuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether in-context learning (ICL) in structured tasks operates through low-dimensional concept inference. The authors formalize this setting through linear regression proxies, showing that task prediction splits into concept estimation and nuisance leakage, and prove that an idealized single-layer attention head reads the prompt solely through this concept subspace. Empirically, the authors extract low-dimensional task directions from residual streams at late layers via cross-covariance with target supervision. Across Llama-3-8B and Qwen2.5-7B on multi-relation prompts, patching a compact subspace (e.g., 73 dimensions out of 4096) restores roughly 79% of the clean–corrupted accuracy gap, while the complementary space restores none. Swapping experiments and matched-rank controls further show that causal influence is concentrated in these task-aligned directions.

Critical strengths
  1. The paper formalizes in-context learning (ICL) as estimation within an intrinsic concept subspace rather than as unconstrained regression in the ambient space, which offers a novel theoretical perspective. This framing yields clear insights into how task estimation scales with the concept dimension, while orthogonal perturbations are shown to act as nuisances rather than task-relevant signal.

  2. The theoretical analysis rigorously separates the subspace written by the attention head from the subspace read by the output projection. This distinction provides a clear mathematical account of why knowledge of the concept subspace is necessary to avoid paying for ambient dimensions, and it is one of the stronger conceptual contributions of the work.

  3. Rather than relying solely on linear probing, the authors validate their claims through causal activation patching and representation swapping. The sharp contrast between the 73-dimensional subspace (78.8% recovery) and the 4023-dimensional complement (0% recovery) is particularly striking and makes the causal case considerably more convincing than a decodability-only analysis would.

  4. The paper includes crucial negative controls, showing that random orthogonal subspaces (2.3%) and cross-task subspaces (3.3%) fail to override model predictions, whereas the learned concept subspace succeeds (58.8%). These matched-rank and cross-task controls substantially strengthen the specificity claim.

  5. The core empirical phenomenon reproduces on Qwen2.5-7B, implicit cross-lingual translation, context-defined nonce mappings, and compositional rules, which suggests that the observation is not a narrow artifact of a single architecture or prompt set. This breadth of validation is reassuring, although the authors are appropriately careful not to overclaim that full ICL circuits are low-dimensional.

Critical weaknesses
  1. One concern is the ambiguity between concept inference and late-layer output steering. The subspace is extracted at layer 30 using target-token unembedding supervision, immediately before vocabulary projection, which raises the possibility that the intervention operates as an output-token shortcut (a logit-lens artifact) rather than as genuine concept-coordinate inference. Crucial theoretical checks, such as reporting difference energy and demonstrating that swaps succeed across different query entities, are missing and would help rule out this alternative interpretation.

  2. The generalization to unseen task relations is not fully verified. When evaluating transfer to unseen relations (Appendix C.5), recovery drops from 56.1% to 48.8%, yet matched-rank controls and variance across splits are not reported. Because the primary subspace is estimated using all 34 evaluation relations, it remains unproven whether the model relies on a shared concept geometry or on an overfitted task-specific projector.

  3. There are inconsistencies in reporting and baseline figures. In Appendix B.2, Figure 3 shows estimated ranks decreasing from 78 to roughly 71–73, whereas the text reports a decline from 69 to 61–62. Similarly, baseline accuracies differ between Figure 1 (63%/41%) and Figure 2 (66%/40%) without explanation. These discrepancies undermine the precision of the empirical claims and should be reconciled.

Questions for authors
  1. Can you report the logit-level recovery and difference energy baselines defined in Equation 12, and verify whether concept swaps still succeed when donor and recipient prompts use different query entities?
  2. In the 24-seen to 10-unseen relation transfer experiment (Table 4), does the transferred subspace outperform a matched-rank random baseline on the unseen relations? Please provide standard deviations across multiple random splits.
  3. Could you clarify the numerical contradictions between the text and plots in Appendix B.2 regarding rank counts, and explain the baseline shift between Figures 1 and 2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.