- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studied the geometry involved in ICL regarding representation and unembedding matrix. The authors have several findings.
Finding 1 (4.1): After applying in-context learning, the final-layer representation of the model is shifted to be more aligned with unembedding direction , giving us
Finding 2 (4.2): This part gives a theoretical explaination about how cross-entropy training shapes unembedding geometry. They decompose the gradient of loss to an attraction and a repulsion part, and find that attraction strength explicitly depends on token frequency, while the repulsion acts on every vocabulary token. This eventually leads to the frequency-organized unembedding geometry.
Finding 3 (4.3-4.4): This is the most interesting finding. The ICL shift mentioned in 4.1 occurs outside the task-label directions. In fact, label-related directions contribute negatively to (global alignment), so movement along the task-label contrasts actually reduces global alignment, whereas movement outside those contrasts increases it. If we replace natural labels with uncommon symbols, then the reduction effect is gone but the positive orthogonal shift remains. The authors further find that induction head plays an important role in the global alignment, as removing them reversed the alignment shifts created by ICL.
Critical strengths
-
The paper contributes towards an underexplored phenomenon in ICL geometry, and they use a combination of theoretical analysis and extensive empirical validation.
-
The finding 3 (4.3-4.4) is very interesting and novel. It's counter-intuitive that the ICL alignment shift occurs outside the task-label directions.
Critical weaknesses
The biggest weakness of this paper is about how the authors position the paper.
The paper establishes an interesting geometric phenomenon without fully explaining its functional significance. The research target is the global unembedding alignment (function ), however the paper does not fully explain why this is important. To my understanding, the paper does not answer whether this global movement actually contribute to improved ICL performance, or it is merely correlated with the computations responsible for ICL. Section 4.4 shows that removal of induction head reversed the unembedding alignment shifts created by ICL, but it does not establish if alignment itself improves the performance. There can be two hypothesis:
- induction head -> unembedding alignment -> better performance comparing to zero-shot
- induction head -> unembedding alignment, and seperately induction head -> better performance comparing to zero-shot
The paper will be much stronger if 1) can be proven (it shows the reason why ICL/few shot is effective), but if it's 2), then it shows that unembedding alignment is simply a byproduct introduced by ICL. Nevertheless, this is an important piece of the research story that is missing.
Questions for authors
Please address the weakness meantioned above. I do not have further questions.
6 comments
W1 and Q1, continued (induction heads, further model families, and a generation task)
(b) Induction heads act through the two parts of the alignment shift. Having tested the two parts of the alignment shift on ICL accuracy, we ask how they relate to the induction heads, which Section 4.4 identifies as the causal source of the ICL movement. We take the IH-steered states of Fig. 8 (ZSL prompts with the mean IH output added), ablate the same two directions from them, and measure accuracy. As a further condition we repeat the ablations on ICL states whose labels are remapped to non-outlier symbols (Section 4.3), where the label part is near zero by construction. For each condition the table gives the accuracy gain over ZSL and the change in accuracy under the three ablations (App. I.7).
| Model | IH-steered: gain over ZSL | Ablate | Ablate | Ablate both | Symbol labels: gain over ZSL | Ablate | Ablate | Ablate both |
|---|---|---|---|---|---|---|---|---|
| Llama3.2-1B | +0.49 | −0.31 | −0.32 | −0.40 | +0.46 | −0.11 | −0.03 | −0.14 |
| Llama3.2-3B | +0.55 | −0.33 | −0.40 | −0.44 | +0.48 | −0.09 | −0.07 | −0.13 |
| Llama3-8B | +0.63 | −0.56 | −0.44 | −0.60 | +0.59 | −0.57 | −0.15 | −0.52 |
| Llama3.1-8B | +0.63 | −0.56 | −0.42 | −0.58 | +0.58 | −0.56 | −0.12 | −0.54 |
| Llama3-70B | +0.66 | −0.65 | −0.45 | −0.68 | +0.63 | −0.63 | −0.14 | −0.63 |
| GLM4-9B | +0.69 | −0.02 | −0.54 | −0.52 | +0.68 | 0.00 | −0.52 | −0.53 |
| GLM4-32B | +0.45 | −0.01 | −0.20 | −0.24 | +0.78 | 0.00 | −0.64 | −0.64 |
Random-direction controls change accuracy by at most 0.02 in every cell. On the IH-steered states the pattern of the ICL states repeats: ablating the orthogonal part that the IH output induces removes more than half of the accuracy that steering produces in every Llama model, and ablating the label part does so in GLM4-9B and removes about half in GLM4-32B. With symbol labels the orthogonal part still carries the effect in the three larger Llama models, so its role does not depend on the labels being outliers; in the two small Llama models, ablating the orthogonal part also lowers accuracy more than ablating the label part, but by less (0.09–0.11 against 0.56–0.63). Together with Fig. 8 and App. I.5, where IH steering reproduces 0.40–0.85 of the alignment shift in all seven models, this shows that the induction heads act on ICL accuracy through the two parts of the alignment shift.
(c) Three further model families. To test whether the ablation results are robust, we repeated them on three further model families, one base model each. The table has the columns of the table in the Common Response (see also App. J of the revised paper).
| Model | ICL − ZSL gain | Ablate | Ablate | Ablate both | Random directions (the same three edits) |
|---|---|---|---|---|---|
| Pythia-6.9B | +0.53 | −0.14 | −0.31 | −0.48 | 0.00 / 0.00 / 0.00 |
| Qwen3-8B-Base | +0.65 | −0.15 | −0.22 | −0.35 | 0.00 / 0.00 / 0.00 |
| Yi-1.5-9B | +0.39 | −0.25 | −0.27 | −0.35 | 0.00 / −0.01 / −0.01 |
The result holds in all three: each part alone lowers accuracy, ablating both removes between half and nearly all of the ICL gain, and random directions remove nothing. The condition predicts the outcome of the ablations in these models as it does in the others. The causal role of the two parts of the alignment shift is therefore robust across five model families, which points to a general mechanism of ICL rather than a property of the Llama and GLM4 models.
(d) An open-ended generation task. On explanation generation with e-SNLI, where the answer is a free-form sentence (our reply to Reviewer hXkQ, W1; App. K), we observe the same two results at the positions where the demonstrations offer tokens to copy. Ablating both parts of the alignment shift from the ICL state lowers the log-probability of the gold token by 0.38 where ICL raises it by 0.47, and random directions change it by less than 0.01. Ablating the induction heads shrinks both parts together with the quality of the generated explanations, and steering with their output restores part of both.
Thank you again for the time and effort you have devoted to our paper. We hope that the new experiments resolve your concern, and we would be grateful if you could take them into account in your final assessment.
Dear Reviewer 2scq,
We sincerely appreciate your insightful comments and the opportunity to further clarify our work. As the discussion period draws to a close, we wanted to gently follow up on our previous responses. We hope they have addressed your concerns, and we would welcome any further questions or feedback. Thank you very much for your time and consideration. Wishing you a wonderful day!
Dear Reviewer 2scq,
We sincerely thank you for your careful reading of our paper and your constructive comments. Our responses follow; the changes to the paper are in Revision Notes 1, 2, 6 and 7.
W1 and Q1 (does the alignment shift contribute to ICL performance, or is it a by-product?)
Thank you for the suggestion to test whether the alignment shift contributes to ICL performance or is only correlated with the computation behind it. All three reviews raise this question, so the ablation that tests it is in the Common Response to all reviewers above. In all seven models, ablating both parts of the alignment shift, the label part and the orthogonal part, removes most of the ICL gain (accuracy falls by 0.42–0.82 where the gain is 0.54–0.85), and the same movement along random directions removes nothing. The alignment shift therefore contributes causally to ICL accuracy and is not a by-product of it. Below we show (a) why and on which tasks each part affects accuracy, (b) how the two parts relate to the induction heads, and that the results hold (c) in three further model families and (d) on an open-ended generation task.
(a) Why and when each part is necessary. The two parts act on different tokens. Every demonstration label points against munit (Fig. 5, right; App. H.6), so the labels lie on the −w side of the other tokens, and the negative ΔAlabel is a movement along −w that raises the logits of the labels relative to all other tokens. Along m⊥, which is orthogonal to SL, every label has the same coordinate as the mean non-label token, and the frequent tokens, which point against munit (Fig. 3, right), lie below this mean. The positive ΔA⊥ is therefore a movement along +m⊥ that lowers the logits of the frequent tokens relative to the labels.
The effect of an ablation on a query follows from the ICL and ZSL states alone. Let M be the logit margin of the correct label over the strongest non-label token in the ICL state; the model can answer correctly only when M>0. The logits are linear in the final state x, so removing the movement along w lowers the margin of the correct label ℓ over a token v by ∥x∥(Δx⋅w)(uℓ−uv)⋅w (up to renormalization), with u the unembedding vectors, and likewise for m⊥. We write Clabel, C⊥ and Cboth for the resulting decrease of M when the movement along w, along m⊥, or along both is removed. For each query and each ablation we thus predict the outcome: the correct label falls below a non-label token, and the answer becomes wrong, if and only if the contribution of the ablated part exceeds the margin (Clabel>M, C⊥>M or Cboth>M for the respective ablation). Checked against the exact ablation, this prediction is right on 95–100% of the queries in each of the ten models (seven of the paper, three of (c); task means, at least 83% in every task). The table averages these per-query quantities over each task of Llama3.2-3B, whose tasks cover three cases (both contributions exceed the margin, only one does, or neither does); the label-token rate is the share of queries whose top-1 token is one of the task's label tokens.
Accuracy follows the comparison of C with M. On SST-2, TREC and MR both contributions reach or exceed the margin, so ablating either part lowers accuracy. On Subj only Clabel exceeds M, and only ablating the label part lowers accuracy. On the three NLI tasks the margin is larger and both contributions are smaller, so neither ablation alone changes accuracy much; accuracy falls when both are ablated, where the joint contribution Cboth is comparable to M. The label-token rate falls together with accuracy: when C exceeds M, the prediction moves from the label tokens to a non-label token.
App. H.10 of the revised paper gives the accuracies, M and C for every task of all models.