acceptodds
Under review as a conference paper at ICLR 2027

The Unembedding Bridge: From Hidden-State Movement to In-Context Learning Predictions

Abstract

Mechanistic accounts of in-context learning (ICL) characterize internal representations and causal components, whereas output-level accounts measure demonstration-induced changes in the predictive distribution. We argue that the unembedding bridges these levels: ICL reflects the interaction of a context-induced hidden-state movement with training-shaped output geometry. Across seven open-weight language models and seven classification tasks, ICL final states generally have higher mean unembedding alignment than paired zero-shot states, against a frequency-organized, spectrally concentrated output geometry. Synthetic-model training, synthetic–LLM comparison, and theory show how cross-entropy attraction and repulsion can create this geometry when the optimizer rescales rows individually. Splitting the movement at the label-contrast space of each task, we find that its label part lowers mean alignment, because the demonstrated labels point against the mean unembedding direction, whereas its orthogonal part raises it and carries the vocabulary-wide shift; remapping the labels to non-outlier symbols removes the label part but leaves the orthogonal shift. Finally, targeted ablation, attribution, probability shifts, and steering identify induction heads as a principal causal source of the movement. Thus training supplies the geometry, demonstrations the movement, and their interaction the behavior.

Primary area:
interpretability and explainable AI
Keywords:
in-context learning, unembedding geometry, cross-entropy training, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2scqOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studied the geometry involved in ICL regarding representation and unembedding matrix. The authors have several findings.

Finding 1 (4.1): After applying in-context learning, the final-layer representation of the model is shifted to be more aligned with unembedding direction munitm_{unit}, giving us A(xICL)>A(xZSL)A(x_{ICL})>A(x_{ZSL})

Finding 2 (4.2): This part gives a theoretical explaination about how cross-entropy training shapes unembedding geometry. They decompose the gradient of loss to an attraction and a repulsion part, and find that attraction strength explicitly depends on token frequency, while the repulsion acts on every vocabulary token. This eventually leads to the frequency-organized unembedding geometry.

Finding 3 (4.3-4.4): This is the most interesting finding. The ICL shift mentioned in 4.1 occurs outside the task-label directions. In fact, label-related directions contribute negatively to AA (global alignment), so movement along the task-label contrasts actually reduces global alignment, whereas movement outside those contrasts increases it. If we replace natural labels with uncommon symbols, then the reduction effect is gone but the positive orthogonal shift remains. The authors further find that induction head plays an important role in the global alignment, as removing them reversed the alignment shifts created by ICL.

Critical strengths
  1. The paper contributes towards an underexplored phenomenon in ICL geometry, and they use a combination of theoretical analysis and extensive empirical validation.

  2. The finding 3 (4.3-4.4) is very interesting and novel. It's counter-intuitive that the ICL alignment shift occurs outside the task-label directions.

Critical weaknesses

The biggest weakness of this paper is about how the authors position the paper.

The paper establishes an interesting geometric phenomenon without fully explaining its functional significance. The research target is the global unembedding alignment (function AA), however the paper does not fully explain why this is important. To my understanding, the paper does not answer whether this global movement actually contribute to improved ICL performance, or it is merely correlated with the computations responsible for ICL. Section 4.4 shows that removal of induction head reversed the unembedding alignment shifts created by ICL, but it does not establish if alignment itself improves the performance. There can be two hypothesis:

  1. induction head -> unembedding alignment -> better performance comparing to zero-shot
  2. induction head -> unembedding alignment, and seperately induction head -> better performance comparing to zero-shot

The paper will be much stronger if 1) can be proven (it shows the reason why ICL/few shot is effective), but if it's 2), then it shows that unembedding alignment is simply a byproduct introduced by ICL. Nevertheless, this is an important piece of the research story that is missing.

Questions for authors

Please address the weakness meantioned above. I do not have further questions.

6 comments

Dear Reviewer 2scq,

We sincerely thank you for your careful reading of our paper and your constructive comments. Our responses follow; the changes to the paper are in Revision Notes 1, 2, 6 and 7.


W1 and Q1 (does the alignment shift contribute to ICL performance, or is it a by-product?)

Thank you for the suggestion to test whether the alignment shift contributes to ICL performance or is only correlated with the computation behind it. All three reviews raise this question, so the ablation that tests it is in the Common Response to all reviewers above. In all seven models, ablating both parts of the alignment shift, the label part and the orthogonal part, removes most of the ICL gain (accuracy falls by 0.42–0.82 where the gain is 0.54–0.85), and the same movement along random directions removes nothing. The alignment shift therefore contributes causally to ICL accuracy and is not a by-product of it. Below we show (a) why and on which tasks each part affects accuracy, (b) how the two parts relate to the induction heads, and that the results hold (c) in three further model families and (d) on an open-ended generation task.

(a) Why and when each part is necessary. The two parts act on different tokens. Every demonstration label points against munitm_{\rm unit} (Fig. 5, right; App. H.6), so the labels lie on the −w-w side of the other tokens, and the negative ΔAlabel\Delta\mathcal A_{\rm label} is a movement along −w-w that raises the logits of the labels relative to all other tokens. Along m⊥m_\perp, which is orthogonal to SL\mathcal S_L, every label has the same coordinate as the mean non-label token, and the frequent tokens, which point against munitm_{\rm unit} (Fig. 3, right), lie below this mean. The positive ΔA⊥\Delta\mathcal A_\perp is therefore a movement along +m⊥+m_\perp that lowers the logits of the frequent tokens relative to the labels.

The effect of an ablation on a query follows from the ICL and ZSL states alone. Let MM be the logit margin of the correct label over the strongest non-label token in the ICL state; the model can answer correctly only when M>0M > 0. The logits are linear in the final state xx, so removing the movement along ww lowers the margin of the correct label ℓ\ell over a token vv by ∥x∥(Δx^⋅w)(uℓ−uv)⋅w\Vert x\Vert \\ (\Delta\widehat x\cdot w)\\ (u_\ell-u_v)\cdot w (up to renormalization), with uu the unembedding vectors, and likewise for m⊥m_\perp. We write ClabelC_{\rm label}, C⊥C_\perp and CbothC_{\rm both} for the resulting decrease of MM when the movement along ww, along m⊥m_\perp, or along both is removed. For each query and each ablation we thus predict the outcome: the correct label falls below a non-label token, and the answer becomes wrong, if and only if the contribution of the ablated part exceeds the margin (Clabel>MC_{\rm label} > M, C⊥>MC_\perp > M or Cboth>MC_{\rm both} > M for the respective ablation). Checked against the exact ablation, this prediction is right on 95–100% of the queries in each of the ten models (seven of the paper, three of (c); task means, at least 83% in every task). The table averages these per-query quantities over each task of Llama3.2-3B, whose tasks cover three cases (both contributions exceed the margin, only one does, or neither does); the label-token rate is the share of queries whose top-1 token is one of the task's label tokens.

TaskAcc. ICLAcc. w/o label partAcc. w/o orthogonal partAcc. w/o bothMMClabelC_{\rm label}C⊥C_\perpCbothC_{\rm both}Label-token rate w/o label partLabel-token rate w/o orthogonal partLabel-token rate w/o both
SST-20.700.000.050.001.87.85.612.30.000.060.00
TREC0.580.140.020.031.64.46.77.10.510.020.06
MR0.900.000.480.002.88.53.010.80.000.510.00
Subj0.870.410.870.165.49.11.910.80.501.000.16
SNLI0.500.420.500.385.12.42.14.41.001.000.93
RTE0.690.550.680.307.13.54.27.41.000.990.54
MNLI0.440.370.380.155.02.83.85.91.000.900.38

Accuracy follows the comparison of CC with MM. On SST-2, TREC and MR both contributions reach or exceed the margin, so ablating either part lowers accuracy. On Subj only ClabelC_{\rm label} exceeds MM, and only ablating the label part lowers accuracy. On the three NLI tasks the margin is larger and both contributions are smaller, so neither ablation alone changes accuracy much; accuracy falls when both are ablated, where the joint contribution CbothC_{\rm both} is comparable to MM. The label-token rate falls together with accuracy: when CC exceeds MM, the prediction moves from the label tokens to a non-label token.

App. H.10 of the revised paper gives the accuracies, MM and CC for every task of all models.

W1 and Q1, continued (induction heads, further model families, and a generation task)

(b) Induction heads act through the two parts of the alignment shift. Having tested the two parts of the alignment shift on ICL accuracy, we ask how they relate to the induction heads, which Section 4.4 identifies as the causal source of the ICL movement. We take the IH-steered states of Fig. 8 (ZSL prompts with the mean IH output added), ablate the same two directions from them, and measure accuracy. As a further condition we repeat the ablations on ICL states whose labels are remapped to non-outlier symbols (Section 4.3), where the label part is near zero by construction. For each condition the table gives the accuracy gain over ZSL and the change in accuracy under the three ablations (App. I.7).

ModelIH-steered: gain over ZSLAblate ΔA⊥\Delta\mathcal A_\perpAblate ΔAlabel\Delta\mathcal A_{\rm label}Ablate bothSymbol labels: gain over ZSLAblate ΔA⊥\Delta\mathcal A_\perpAblate ΔAlabel\Delta\mathcal A_{\rm label}Ablate both
Llama3.2-1B+0.49−0.31−0.32−0.40+0.46−0.11−0.03−0.14
Llama3.2-3B+0.55−0.33−0.40−0.44+0.48−0.09−0.07−0.13
Llama3-8B+0.63−0.56−0.44−0.60+0.59−0.57−0.15−0.52
Llama3.1-8B+0.63−0.56−0.42−0.58+0.58−0.56−0.12−0.54
Llama3-70B+0.66−0.65−0.45−0.68+0.63−0.63−0.14−0.63
GLM4-9B+0.69−0.02−0.54−0.52+0.680.00−0.52−0.53
GLM4-32B+0.45−0.01−0.20−0.24+0.780.00−0.64−0.64

Random-direction controls change accuracy by at most 0.02 in every cell. On the IH-steered states the pattern of the ICL states repeats: ablating the orthogonal part that the IH output induces removes more than half of the accuracy that steering produces in every Llama model, and ablating the label part does so in GLM4-9B and removes about half in GLM4-32B. With symbol labels the orthogonal part still carries the effect in the three larger Llama models, so its role does not depend on the labels being outliers; in the two small Llama models, ablating the orthogonal part also lowers accuracy more than ablating the label part, but by less (0.09–0.11 against 0.56–0.63). Together with Fig. 8 and App. I.5, where IH steering reproduces 0.40–0.85 of the alignment shift in all seven models, this shows that the induction heads act on ICL accuracy through the two parts of the alignment shift.

(c) Three further model families. To test whether the ablation results are robust, we repeated them on three further model families, one base model each. The table has the columns of the table in the Common Response (see also App. J of the revised paper).

ModelICL − ZSL gainAblate ΔA⊥\Delta\mathcal A_\perpAblate ΔAlabel\Delta\mathcal A_{\rm label}Ablate bothRandom directions (the same three edits)
Pythia-6.9B+0.53−0.14−0.31−0.480.00 / 0.00 / 0.00
Qwen3-8B-Base+0.65−0.15−0.22−0.350.00 / 0.00 / 0.00
Yi-1.5-9B+0.39−0.25−0.27−0.350.00 / −0.01 / −0.01

The result holds in all three: each part alone lowers accuracy, ablating both removes between half and nearly all of the ICL gain, and random directions remove nothing. The condition C>MC > M predicts the outcome of the ablations in these models as it does in the others. The causal role of the two parts of the alignment shift is therefore robust across five model families, which points to a general mechanism of ICL rather than a property of the Llama and GLM4 models.

(d) An open-ended generation task. On explanation generation with e-SNLI, where the answer is a free-form sentence (our reply to Reviewer hXkQ, W1; App. K), we observe the same two results at the positions where the demonstrations offer tokens to copy. Ablating both parts of the alignment shift from the ICL state lowers the log-probability of the gold token by 0.38 where ICL raises it by 0.47, and random directions change it by less than 0.01. Ablating the induction heads shrinks both parts together with the quality of the generated explanations, and steering with their output restores part of both.


Thank you again for the time and effort you have devoted to our paper. We hope that the new experiments resolve your concern, and we would be grateful if you could take them into account in your final assessment.

Dear Reviewer 2scq,

We sincerely appreciate your insightful comments and the opportunity to further clarify our work. As the discussion period draws to a close, we wanted to gently follow up on our previous responses. We hope they have addressed your concerns, and we would welcome any further questions or feedback. Thank you very much for your time and consideration. Wishing you a wonderful day!

Reviewer_hXkQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes the unembedding matrix as a bridge connecting internal representations and output predictions in in-context learning, showing that ICL behavior emerges from the interaction between a context-induced hidden-state movement and a training-shaped, frequency-organized output geometry. A decomposition of this movement within the task’s label-contrast space indicates that the label-aligned component reduces the mean alignment, whereas the orthogonal component accounts for the shift observed across the full vocabulary; this interpretation is corroborated by label-remapping experiments. Induction heads are characterized as a primary causal driver of the observed behavior, as evidenced by ablation and steering analyses, and induction-head (IH) steering is presented as a competitive, training-free alternative to few-shot prompting.

Critical strengths
  1. The paper makes a clear, well-motivated contribution by showing that the unembedding matrix provides a principled coordinate system linking internal and output-level accounts of ICL, bridging a longstanding gap in mechanistic interpretability. This unifying view reframes ICL as the interaction of hidden-state and output-distribution changes via a structured geometric readout.

  2. The paper presents a rigorous, well-supported theory linking frequency-organized, spectrally concentrated unembedding geometry to asymmetric cross-entropy attraction and softmax repulsion, validated through synthetic Transformer training and comparisons with pretrained LLMs. This validation approach—encompassing theoretical analysis, synthetic modeling, and evaluation using large-scale large language models (LLMs)— enhances the evidential support for the proposed geometric account.

  3. The authors show that induction heads are the main driver of the hidden-state movement that ICL induces. This is well supported by complementary experiments: attribution, ablation, probability-distribution recovery, and steering, which have been carefully designed.

Critical weaknesses
  1. Although the paper’s primary contribution is theoretically ambitious, the empirical assessment is largely restricted to relatively simple tasks. These tasks employ a highly constrained demonstration protocol and a small, fixed label space, which may be insufficiently challenging or diverse to rigorously evaluate the robustness and generalizability of the proposed framework.

  2. The heavy reliance on the Llama3 model family raises concerns about how well the findings generalize to models trained under different architectures or pretraining objectives.

  3. The paper thoroughly characterizes ICL-driven hidden-state movement geometrically but does not clearly quantify how the proposed unembedding bridge relates to ICL task performance.

Questions for authors

Could the authors suggest experimental directions that would address Weaknesses 1 and 3?

7 comments

Dear Reviewer hXkQ,

We are grateful for your thoughtful review and for your suggestions on the generality of our findings, which prompted the new experiments on a further task and further model families. Our responses follow; the related changes to the paper are in Revision Notes 1, 2, 6 and 7.


W3 and Q1 (how the unembedding bridge relates to ICL task performance)

Thank you for the suggestion to investigate how the alignment shift that ICL produces through the unembedding bridge affects ICL accuracy. We now measure this by ablating the shift from the ICL state. All three reviews raise this point, so the experiment and its table are in the Common Response to all reviewers above.

(a) What the movement contributes to accuracy. The alignment shift has two parts, the label part and the orthogonal part, each carried by one direction of the final state (Eq. 7), and we set the coordinate of the ICL state along either direction back to its zero-shot value. In all seven models, ablating both parts removes most of the ICL gain (accuracy falls by 0.42–0.82 where the gain is 0.54–0.85), and the same movement along random directions removes nothing. The movement that the unembedding makes measurable therefore contributes causally to ICL accuracy and is not a by-product of it. Which part a task depends on can be predicted before any ablation, by comparing what that part contributes to the logit margin of the correct label with the margin itself (Common Response, (b)).

(b) The link to the induction heads. Section 4.4 identifies the induction heads as the causal source of the ICL movement. We repeat the ablation on zero-shot states steered with the mean induction-head output (App. I.7 and Table 28 of the revised paper): ablating the orthogonal part that the head output induces removes more than half of the accuracy that steering produces in every Llama model, and ablating the label part does so in GLM4-9B. The induction heads therefore act on ICL accuracy through the two parts of the alignment shift.

Your question asks for experiments that would address Weaknesses 1 and 3. We have run one for each: the ablation above measures what the alignment shift contributes to accuracy (Weakness 3), and the explanation-generation task in our reply to W1 below tests the mechanism where the answer is not a label copied from the demonstrations (Weakness 1).

W1 and Q1 (tasks beyond a small, fixed label space)

Thank you for the suggestion to evaluate the framework on a task that is less constrained than classification. We therefore tested the mechanism on an open-ended generation task whose answer is a sentence that does not appear in the demonstrations (App. K of the revised paper).

(a) Task. We use e-SNLI [1], the SNLI dataset of the paper with a human-written explanation for each pair. Each demonstration is the paper's SNLI record followed by its explanation: "{premise} The question is: {hypothesis}? True or maybe or false? The answer is: {label}. Explanation: {explanation}". The query gives the gold label and stops after "Explanation:"; the model's answer is the explanation (about 15 tokens). ICL prompts have ten demonstrations, and ZSL prompts contain the query alone. We use 500 test queries and Llama3.1-8B. We measure the quality of the explanation in two ways. Under teacher forcing, we append the gold explanation to the prompt and measure the mean log-probability per token that the model assigns to it. Under free decoding, the model generates the explanation (greedy, up to 48 tokens), and we score it with ROUGE-L against the three reference explanations of the test set.

(b) Induction heads for a multi-token answer. We select the heads as in the classification tasks, on separate prompts to which we append the gold explanation, and apply the selection rule of the paper at every explanation position, that is, every position that predicts a token of the explanation (details in App. K). For an explanation position tt with token xtx_t, let KtK_t be the positions inside the demonstration explanations whose preceding token is xtx_t; an induction head at tt attends to KtK_t. Positions with non-empty KtK_t (48% of the explanation positions) are the valid positions, and the score of a head is its mean attention from the valid positions to KtK_t. We take the top 10% of heads (102 heads); 68% of them are also in the SNLI induction-head set of the paper. We ablate these heads in the ICL prompt, or steer the ZSL prompt with their mean ICL − ZSL output at every explanation position, and use ten size-matched random head sets as baselines for both interventions.

StateGold explanation, mean log-probability per tokenGenerated explanation, ROUGE-L
ZSL−2.130.325
ICL−1.640.465
ICL, induction heads ablated−2.040.365
ICL, random heads ablated−1.85±0.120.435±0.037
ZSL + induction-head steering−1.930.396
ZSL + random-head steering−2.07±0.020.350±0.022

The first two rows show what the demonstrations do here: the model assigns the gold explanation a higher probability and writes explanations closer to the references. Ablating the induction heads takes most of this away: the gold explanation loses most of the probability it had gained, and the generated explanations fall back about two thirds of the way toward ZSL quality. Steering shows the converse: adding only the induction heads' mean output to the ZSL prompt restores about half of the gain on both measures.

[1] Camburu et al., e-SNLI: Natural Language Inference with Natural Language Explanations.

W1 and Q1, continued (the alignment shift on a multi-token answer)

(c) The alignment shift on a multi-token answer. In the classification tasks the label set LL is the set of tokens that the induction heads copy. The corresponding set at a valid position tt is LtL_t, the tokens at KtK_t: the tokens that followed the current token in the demonstration explanations (on average four tokens, 84% of them raw-mean outliers, and the gold next token is among them 19% of the time). At each valid position we compute ΔA\Delta\mathcal A against the paired ZSL state at the same position, and its label part and orthogonal part with the label subspace SLt\mathcal S_{L_t} built from LtL_t. The table gives their means over the valid positions (Table 11 of the revised paper).

StateΔA\Delta\mathcal ALabel part ΔAlabel\Delta\mathcal A_{\rm label}Orthogonal part ΔA⊥\Delta\mathcal A_\perp
ICL+0.0035−0.0028+0.0064
ICL, induction heads ablated+0.0002−0.0010+0.0011
ICL, random heads ablated+0.0007±0.0015−0.0022±0.0009+0.0029±0.0019
ZSL + induction-head steering+0.0010−0.0003+0.0013
ZSL + random-head steering−0.0000±0.0005−0.0002±0.0001+0.0002±0.0004

The shift has the composition of the classification tasks: under ICL the label part is negative, the orthogonal part is positive and larger, and the total shift is positive. Ablating the induction heads shrinks each part more than most sets of random heads do. Steering with the induction-head output brings back a positive shift (about a quarter of the ICL shift), carried by the orthogonal part; both exceed the values of all ten random sets. The shift is smaller than in classification for a simple reason: at most positions the gold token is not among the copied tokens, so what the heads copy is one input to the prediction among others rather than the answer itself.

(d) What the two parts contribute to the prediction. ICL raises the mean log-probability of the gold token at the valid positions by 0.47 over ZSL. Ablating both parts of the alignment shift from the ICL state at these positions, with the edit of the Common Response, lowers it by 0.38 (0.26 for the orthogonal part alone, 0.16 for the label part alone), and the same movements along random directions change it by less than 0.01 (last paragraph of App. K of the revised paper).

Together with the explanation quality above, these results indicate that our findings on the induction heads and the alignment shift also hold on an open-ended generation task.

Reviewer_JtWaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how demonstrations change language-model predictions through the unembedding matrix. The authors analyze training-induced output geometry, decompose hidden-state movement into label-related and orthogonal components, and use ablation and steering to investigate selected attention heads. They argue that the interaction between learned geometry and demonstration-induced movement connects internal computation to ICL predictions.

Critical strengths

The paper’s idea of using the unembedding to connect internal changes during ICL with observable prediction changes is interesting. The framing brings together two factors: geometry learned during training and hidden-state movement induced by demonstrations; and provides a common framework for studying how they interact

Critical weaknesses

My main concern is that the paper presents many analyses without making their relative importance clear. Across section 4, I often found myself reading an experiment’s definition followed by a description of its figure, while still wondering why that experiment was necessary and what we should learn from it. A clearer explanation on what insight is useful would be appreciated.

Second, the connection between the alignment signature and successful ICL remains insufficiently established. GLM4-9B achieves strong accuracy despite a negative alignment shift, while synthetic controls substantially change the geometry without comparably changing accuracy (Table 2 and Table 8). These results leave me unsure what the signature explains about successful prediction beyond describing a recurring geometric effect.

Third, the interventions establish the importance of selected label-attending heads more directly than an induction mechanism. The selection score measures attention to demonstration-label positions; it does not directly test whether a head matches the query to an earlier example and retrieves its associated label. Ablation and held-out steering provide meaningful causal evidence, but other ways of using label information could also produce those effects.

Questions for authors

What is the central claim connecting the analyses in section 4? Is the alignment signature intended primarily as a description of learned geometry, an explanation of ICL predictions, or a basis for steering? Which results are essential to establishing that claim?

8 comments

Dear Reviewer JtWa,

Thank you for your detailed and critical review, which helped us to state the central claim of the paper more clearly and to test it more directly. Our point-by-point responses follow; the corresponding changes to the paper are listed in Revision Notes 1, 3, 4 and 5.


W2 (what the signature explains about successful prediction: GLM4-9B and the synthetic controls)

The two observations have one answer: the total alignment shift ΔA\Delta\mathcal A is not the quantity that carries the function, and Section 4.2 does not claim that the geometry is needed for ICL.

(a) The functional quantities are the two parts of the alignment shift, not their sum. The ICL − ZSL movement projects onto the unit-mean axis munitm_{\rm unit} (Eq. 4) as

ΔA=ΔAlabel+ΔA⊥,\Delta\mathcal A=\Delta\mathcal A_{\rm label}+\Delta\mathcal A_\perp,

the label part of the alignment shift inside the label-contrast space SL\mathcal S_L plus the orthogonal part outside it (Eq. 7). Each part has its own function: the label part raises the demonstration labels above the other tokens, and the orthogonal part lowers the frequent tokens against the labels (Section 4.3 and App. H.10 of the revised paper). The two parts have opposite signs in every model, because the demonstration labels point against munitm_{\rm unit}, so they partly cancel, and the sign of ΔA\Delta\mathcal A records only which of the two is larger. We tested this directly (App. H.10): resetting the single coordinate of the ICL state along munitm_{\rm unit} sets ΔA\Delta\mathcal A to zero but leaves both parts in place, where they cancel. In the two GLM4 models this edit changes accuracy by 0.00, whereas ablating the two parts, which also sets ΔA\Delta\mathcal A to zero, lowers it by 0.48 and 0.42 (Common Response to all reviewers above). In the four smaller Llama models the edit costs 0.02–0.27 and in Llama3-70B 0.76, in every model less than ablating the two parts (0.46–0.82). It costs most in Llama3-70B because the label part is small there (−0.002 against +0.017 for the orthogonal part), so ΔA\Delta\mathcal A is almost the orthogonal part itself, and setting it to zero removes four fifths of that part. Accuracy thus depends on how much of the two movements remains, not on the value of their sum.

(b) GLM4-9B: high accuracy with a negative alignment shift. GLM4-9B is the extreme case of (a): its orthogonal part is nearly zero (ΔA⊥=+0.0001\Delta\mathcal A_\perp=+0.0001), so its negative ΔA\Delta\mathcal A is its label part alone. Please refer to the Common Response for the ablation experiment (its part (a)) and for the analysis of the margin MM and the contributions CC (its part (b)). The ablation shows that the accuracy of GLM4-9B depends on exactly this part: ablating the label part lowers accuracy by 0.48 on average and by at least 0.27 on every task, whereas ablating the orthogonal part changes the average by less than 0.01. The condition C>MC > M accounts for both numbers (details in App. H.10): the contribution of the label part to the margin of the correct label, ClabelC_{\rm label}, is about equal to the margin MM or above it on every task, and that of the orthogonal part stays far below it (C⊥≤0.7C_\perp\le0.7 against MM of 2.9–7.0). GLM4-9B needs no orthogonal part, consistent with what competes with its labels: the strongest non-label competitor of the correct label is a frequent token on only 3% of its queries, against 75–100% in the Llama models once their orthogonal part is ablated (App. H.10, Table 23). GLM4-9B thus reaches high accuracy with a negative alignment shift because this shift is the label part, the part its prediction depends on.

(c) Synthetic controls (Table 2) and the cross-model geometry (Table 8). Points (a) and (b) concern how the alignment shift relates to the ICL prediction, which is the question of Sections 4.3 and 4.4. Section 4.2 answers an earlier question: where the geometry in which this shift is read comes from, that is, why the unembedding has one axis that separates frequent from non-frequent tokens (Fig. 3). The synthetic controls vary the training distribution to test the predictions of the theorem about this geometry; their accuracy column only confirms that each variant still performs ICL, so the geometric differences are not confounded by a loss of the ICL ability. Geometry changing while accuracy does not is therefore the expected outcome of that section, not evidence against the paper, because the geometry is the coordinate system in which the movement is measured, not its cause. Likewise, Table 8 of the submission (Table 14 of the revised paper) shows that the geometric regularities of Table 1 hold across the LLMs; it does not predict accuracy from them. The relation between the alignment shift and accuracy is established on the LLMs in Sections 4.3–4.4 (the ablations of the Common Response, IH ablation and steering).

W1 and Q1 (the central claim of Section 4 and the results it rests on)

Thank you for the suggestion to make clear what each analysis of Section 4 is for and which results the central claim needs.

(a) The central claim. Section 4 makes one claim: ICL changes the prediction through a movement of the final hidden state that the unembedding makes measurable. Read along the unit-mean axis munitm_{\rm unit}, this movement has two parts (Eq. 7):

ΔA=ΔAlabel+ΔA⊥.\Delta\mathcal A=\Delta\mathcal A_{\rm label}+\Delta\mathcal A_\perp.

The label part ΔAlabel\Delta\mathcal A_{\rm label} is the movement inside the label-contrast space, which raises the demonstration labels above the other tokens. The orthogonal part ΔA⊥\Delta\mathcal A_\perp is the movement outside it, which lowers the frequent tokens against the labels. Induction heads produce both parts (Section 4.4), and ICL accuracy depends on both: ablating the two parts from the ICL state removes most of the accuracy that ICL adds, whereas the same movements along random directions change nothing (Section 4.3).

(b) What the signature is intended to be (the first part of your question). Of the three readings you list, the signature is intended as an explanation of ICL predictions, and its two parts are causally important for them: ablating the two parts removes most of the ICL accuracy gain in every model (Common Response). They explain how ICL makes the prediction a label token at all, which is the task-recognition part of ICL in the terms of [1]. The learned geometry is not the claim itself. It is the coordinate system in which the movement is read, and Section 4.2 explains why this coordinate system has one axis that separates frequent from non-frequent tokens. Steering is an application and a test of sufficiency, not the basis of the claim.

(c) Which results are essential to the claim (the second part of your question). Each subsection answers one question, and the claim needs one result from each. The table lists these results together with the further analyses of each subsection, which test that the result is not an artifact.

SectionQuestion it answersResult the claim rests onFurther analyses
4.1Does ICL leave a trace over the whole vocabulary?ICL shifts the mean alignment A\mathcal A of the final state with all token unembeddings, and this shift equals the movement of the state along one axis, ΔA=Δx^⋅munit\Delta\mathcal A=\Delta\widehat x\cdot m_{\rm unit} (Fig. 2, Eq. 4).Frequency organization of the unembedding (Fig. 3).
4.2Why does this axis exist, and what does it encode?Cross-entropy training with an adaptive optimizer separates a few frequent unembedding rows from a coherent non-frequent majority, which makes munitm_{\rm unit} a nonzero axis pointing away from the frequent tokens (Theorem 1, Corollary 1, Table 1).Training dynamics (Fig. 4), synthetic controls (Table 2), optimizer analysis (App. G.6).
4.3Which part of the movement produces the shift, and does accuracy depend on it?The shift is the sum of a negative label part and a positive orthogonal part (Eq. 7, Fig. 5), and ablating the two removes most of the ICL accuracy gain (Table 3).Symbol labels (Fig. 6), prompt controls (App. H.7).
4.4Which components of the model produce the movement?Ablating the induction heads removes the shift and the accuracy (Fig. 7). Adding their output to zero-shot prompts reproduces about half of the shift and moves the output distribution about halfway to ICL (Fig. 8), and ablating the two parts from these steered states removes most of the accuracy that steering produces (App. I.7).Direct attribution of the head outputs (App. I.4), comparison with other steering methods (Fig. 9).

The third row is the link that the submission left implicit: it located the alignment shift but did not show that accuracy depends on it. The Common Response and our reply to W2 above supply it.

(d) How the paper is revised. You note that experiments were described without saying why each was needed. Section 4 of the revised paper now makes this explicit in three ways. First, before its first subsection it states the claim of (a) and the role of each subsection in it. Second, each subsection opens with the question it answers, the questions of the table above. Third, the step from the signature to accuracy is now in the main text (the ablation paragraph and Table 3 in Section 4.3), and Section 4.2 states that the accuracy of the synthetic controls only confirms that each control still performs ICL. Thank you for this suggestion, which has made the focus of the paper much clearer.

[1] Pan et al., What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning.

W3 (label-attending heads vs. induction heads)

Thank you for the suggestion to test directly whether the selected heads match the query to an earlier example and retrieve its label. We agree that the score by which we define the induction heads in the paper does not by itself test this. Below we explain why this score targets the induction pattern, check the selected heads with the standard test on random tokens, and add a direct retrieval test.

(a) What the score measures, and why it targets induction. The score by which we define the induction heads (Eq. 23, App. E) is, for each head, the share of its attention from the final position of the prompt that lands on the demonstration-label positions. Every demonstration has the form [input][delimiter][label] and the query ends with the same delimiter, so the final position is a repeated token, and the demonstration labels are the tokens that followed its earlier occurrences. The foundational work [1], which introduced the concept of induction heads, defines them by the attention pattern [A][B] … [A] → [B]: at a later occurrence of a token [A], the head attends to the token [B] that followed the earlier occurrence of [A], and thereby predicts [B]. In an ICL prompt, [A] is the delimiter and [B] is a demonstration label, so a head that attends from the final delimiter to the labels after the earlier delimiters shows exactly this pattern. [2], [3] and [4] identify induction heads in LLM in-context learning with this score, and [5], [6] and [7] use the same attention-to-labels quantity, in some cases contrasting same-class and different-class labels.

(b) The selected heads pass the standard test. We follow the standard test for induction heads [1], which uses no dataset and no prompt: we build 50 sequences, each a string of 100 distinct random tokens repeated once, and run the model on them. The prefix-matching score of a head is its attention from a token in the second copy to the token that followed the same token in the first copy, averaged over the positions and the sequences. We rank all heads of a model by this score and compare the ranking with our selected heads, the top 10% by our score on each task. The table gives the share of our selected heads that are also among the top 10% by the prefix-matching score, their mean percentile in this ranking, and the mean scores; values are means over the seven tasks (App. E.5 of the revised paper).

ModelSelected heads within the top 10% by prefix-matching score (chance 0.10)Mean percentile of the selected headsMean prefix-matching score, selected headsMean prefix-matching score, all heads
Llama3.2-1B0.570.890.2250.037
Llama3.2-3B0.500.870.1920.033
Llama3-8B0.470.860.1760.032
Llama3.1-8B0.460.860.1740.032
Llama3-70B0.600.890.0680.010
GLM4-9B0.440.830.2560.055
GLM4-32B0.450.800.1490.030

In every model about half of the selected heads are among the top 10% of all heads by the prefix-matching score, four to six times the chance rate, and their mean score is about five to seven times the mean over all heads. The heads selected on ICL prompts are therefore, for the most part, the heads that copy in the classic setting.

[1] Olsson et al., In-context Learning and Induction Heads.

[2] Cho et al., Revisiting In-context Learning Inference Circuit in Large Language Models.

[3] Yang et al., Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning.

[4] Yang et al., Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis.

[5] Halawi et al., Overthinking the Truth: Understanding How Language Models Process False Demonstrations.

[6] Reddy, The Mechanistic Basis of Data Dependence and Abrupt Learning in an In-Context Classification Task.

[7] Singh et al., What Needs to Go Right for an Induction Head? A Mechanistic Study of In-Context Learning Circuits and Their Formation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision Note

Revision Note

Dear PCs, SACs, ACs, and Reviewers,

We sincerely thank all reviewers for their valuable suggestions, which have improved the paper. Our revisions (highlighted in orange) are as follows.

1. Reviewer 2scq, JtWa, hXkQ: Whether the alignment shift contributes to ICL accuracy (Section 5, Table 3; Appendix H.10) We agree that the paper should establish whether the alignment shift contributes to ICL accuracy or only accompanies it. We therefore ablate the shift from the ICL state by setting the coordinate that carries its label part, or its orthogonal part (Eq. 7), back to its zero-shot value (Common Response). In all seven models, ablating both parts removes most of the ICL accuracy gain, while the same movement along random directions removes nothing. This shows that the alignment shift contributes causally to ICL accuracy and is not a by-product of it.

2. Reviewer 2scq: Whether induction heads improve accuracy through the alignment shift (Section 6, Appendix I.7) We concur that the submission showed that induction heads produce the alignment shift, but not that their effect on accuracy passes through it. On zero-shot states steered with the induction-head output, ablating the two parts that this output induces removes most of the accuracy gained by steering. This shows that the induction heads act on ICL accuracy through these two parts.

3. Reviewer JtWa: One claim in three sections, with takeaway and implications (Abstract; Sections 1 and 4–7) We agree that the former Section 4 was overloaded and the paper lacked a clear takeaway. Following the reviewer's suggestion, it is now three sections, one per step of a single claim: the unembedding geometry in which the ICL movement is read (Section 4), the two parts of this movement on which ICL accuracy depends (Section 5, former 4.3), and induction heads as a principal source of both (Section 6, former 4.4). Each section opens with its question. The abstract, introduction and conclusion state the three steps and the takeaway: ICL improves task accuracy through a two-part final-state movement, driven largely by induction heads, in a coordinate system shaped by cross-entropy training. Sections 1, 5 and 7 state the implications for the mechanistic interpretability of ICL and for model safety.

4. Reviewer JtWa: What the signature explains about accuracy (Sections 4 and 5) We understand the concern that GLM4-9B and the synthetic controls seem to separate the signature from accuracy. Section 5 now states that the total shift sums two parts of opposite sign, so its sign records only which part is larger, and point 1 shows that GLM4-9B depends on the label part. Section 4 now states that Section 4.2 gives the origin of the geometry on which the shift rests. The signature thus explains accuracy through its two parts, read in this geometry.

5. Reviewer JtWa: Whether the selected heads are induction heads (Section 6, Appendix E.5 and E.6) We appreciate the suggestion to test whether the selected heads are the induction heads defined by the prefix-matching score, and whether they retrieve the label of a matching demonstration. About half of them are among the top 10% of all heads by that score, and when one demonstration is the query itself with a wrong label, ablating them removes most of the rise in the probability of that label. This shows that the selected heads overlap strongly with the standard induction heads and perform their retrieval function.

6. Reviewer hXkQ: A task beyond a small, fixed label space (Appendix K) We agree that the framework should be tested beyond single-token answers from a small label set, and add explanation generation on e-SNLI, whose answer is a free-form sentence absent from the demonstrations. Ablating the induction heads removes most of the gain in explanation quality together with the alignment shift, and steering with their output restores part of both. This shows that the mechanism extends beyond single-token classification.

7. Reviewer hXkQ: Model families beyond Llama (Appendix J) We agree that the findings should be tested beyond the Llama family, and add three families (Pythia-6.9B, Qwen3-8B-Base and Yi-1.5-9B). The geometry of Table 1, the two parts of the alignment shift and their contribution to accuracy, and the role of the induction heads replicate in all three (steering is weakest in Yi-1.5-9B). The findings are thus not specific to the Llama family.

8. Additional controls: the source of the orthogonal part (Section 5, Appendix H.7–H.9) A length-matched WikiText prefix and demonstrations of another task each reproduce only a small fraction of the orthogonal part, and in the Llama models it persists when part of the labels are replaced by the most frequent token. The orthogonal part thus comes from demonstrations of the queried task, not from a longer context.

We are grateful for the time and effort you have devoted to our submission.

Sincerely, The Authors

AuthorsOpenReview ↗

Common Response: Alignment Shift and ICL Accuracy

All three reviews ask whether the alignment shift contributes to ICL accuracy or only accompanies it (Reviewer 2scq, W1 and Q1; Reviewer JtWa, W2; Reviewer hXkQ, W3 and Q1). We tested this directly: we ablate the alignment shift from the ICL state and measure accuracy. The individual replies build on this experiment.

(a) Ablating the two parts of the alignment shift. By Eq. (7) the alignment shift has two parts, the label part and the orthogonal part, each carried by one unit direction:

ΔA=ΔAlabel+ΔA⊥,ΔAlabel=∥PLmunit∥(Δx^⋅w),ΔA⊥=∥(I−PL)munit∥(Δx^⋅m⊥),\Delta\mathcal A=\Delta\mathcal A_{\rm label}+\Delta\mathcal A_\perp,\qquad \Delta\mathcal A_{\rm label}=\Vert P_Lm_{\rm unit}\Vert \\ (\Delta\widehat x\cdot w),\qquad \Delta\mathcal A_\perp=\Vert (I-P_L)m_{\rm unit}\Vert \\ (\Delta\widehat x\cdot m_\perp),

with $w=P_Lm_{\rm unit}/\Vert P_Lm_{\rm unit}\Vert $ and $m_\perp=(I-P_L)m_{\rm unit}/\Vert (I-P_L)m_{\rm unit}\Vert $. Here munitm_{\rm unit} is the unit-mean axis (Eq. 4), $\widehat x=x/\Vert x\Vert $ the unit-normalized final state, Δx^\Delta\widehat x its ICL − ZSL difference, and PLP_L the projection onto the label-contrast space SL\mathcal S_L. We ablate a part by setting the coordinate of the ICL x^\widehat x along ww, or along m⊥m_\perp, back to its value in the paired ZSL state, which sets ΔAlabel\Delta\mathcal A_{\rm label}, or ΔA⊥\Delta\mathcal A_\perp, to zero and leaves the other unchanged; we also ablate both. As a control, we move the ICL state by the same distances along random directions in place of ww and m⊥m_\perp. Each ablation column gives the accuracy after the edit minus the ICL accuracy, averaged over the seven tasks (Section 4.3 and Table 3 of the revised paper).

ModelICL − ZSL gainAblate ΔA⊥\Delta\mathcal A_\perpAblate ΔAlabel\Delta\mathcal A_{\rm label}Ablate bothRandom directions (the same three edits)
Llama3.2-1B+0.54−0.28−0.32−0.46−0.02 / −0.01 / −0.01
Llama3.2-3B+0.63−0.24−0.40−0.520.00 / 0.00 / 0.00
Llama3-8B+0.77−0.75−0.53−0.770.00 / 0.00 / 0.00
Llama3.1-8B+0.75−0.69−0.49−0.760.00 / 0.00 / 0.00
Llama3-70B+0.80−0.81−0.49−0.820.00 / 0.00 / 0.00
GLM4-9B+0.850.00−0.48−0.480.00 / 0.00 / 0.00
GLM4-32B+0.71−0.01−0.26−0.420.00 / 0.00 / 0.00

In every model, ablating both parts removes most of the ICL gain, and random directions remove nothing. The alignment shift therefore contributes causally to ICL accuracy and is not a by-product of it. Which part matters more differs: in Llama3.2-1B and 3.2-3B ablating either part costs accuracy, in the Llama models from 8B upward the orthogonal part costs more, and in the two GLM4 models only the label part costs accuracy when ablated alone.

(b) When each part is necessary. Which part a task depends on can be predicted from the ICL and ZSL states, before any ablation. For a query, let MM be the logit margin of the correct label over the strongest non-label token in the ICL state. The model can predict the correct label only if M>0M > 0. Because the logits are linear in the final state, the two states directly give how much each movement adds to this margin: ClabelC_{\rm label} is the amount by which MM falls when the movement along ww is removed, and C⊥C_\perp the amount by which it falls when the movement along m⊥m_\perp is removed. Ablating a part therefore pushes the correct label below a non-label token, and makes the answer wrong, when the contribution of that part exceeds the margin, C>MC > M. Predicting this outcome for every query and every ablation agrees with the exact ablation on 95–100% of the queries in every model (task means, at least 83% in every task; details in App. H.10 of the revised paper). In the Llama models from 8B upward, C⊥C_\perp exceeds MM on all seven tasks (5.6–30.0 against 2.5–7.2), which is why the orthogonal part costs more there. In Llama3.2-1B and 3.2-3B, C⊥C_\perp exceeds MM on three tasks and ClabelC_{\rm label} on four, so each ablation costs accuracy on its own tasks. GLM4-9B makes almost no movement along m⊥m_\perp (C⊥≤0.7C_\perp\le0.7), and GLM4-32B makes one whose contribution stays below the margin on every task (C⊥≤3.5C_\perp\le3.5 against MM of 4.2–7.5), so in both models ablating the orthogonal part alone costs nothing. A positive MM means that the correct label beats every non-label token, so the model answers within the label set. This is the task-recognition function of ICL in the terms of [1], and both parts serve it by adding to MM: the label part raises the labels above the other tokens, and the orthogonal part holds the frequent tokens below them.

[1] Pan et al., What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.