- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper aims to investigate how the internal separability of input text representations during ICL affects predictions. The authors conduct the following experiments: 1. Verify the existence of such separability. 2. Actively intervene on or reverse the directions associated with this separability and observe that the output is also reversed. 3. Locate and analyze the circuit responsible for this output effect, and perform a QKV decomposition of its functionality.
Critical strengths
-
The experiment in Section 4.2, which shifts the centroids of demonstration encodings, is a new experiment that can provide additional causal evidence for existing frameworks.
-
The experiment in Section 5.1, which decomposes the functionality of induction heads (? I am not entirely sure whether the heads you intervene on qualify as induction heads by definition, but they perform demonstration retrieval) into Q, K, and V functions, is consistent with our intuition about induction heads. Moreover, attempting to decompose the functionality of attention heads is an interesting direction.
Critical weaknesses
-
Existing work on hidden-state geometry and induction heads has already covered most of the contributions of this paper. How the cluster separability of sentence encodings, particularly linear separability, participates in downstream ICL computation has already been extensively investigated.
-
Experiments such as those in Section 4.1, which examine how separability and the accuracy of linear classifiers increase with model depth, have already been thoroughly investigated. Although the authors claim that their work differs by using the representations of x appearing in earlier demonstrations to predict the labels of x appearing in later demonstrations, (1) What is the purpose of this? What question are you trying to answer? Are you trying to demonstrate that hidden states do not rotate as demonstrations accumulate? Why not simply use established methods to reaffirm previous findings about separability as the foundation of your study, rather than presenting this as a new experimental finding with an unclear purpose? (2) Note that prior work has already performed essentially the same experiment. [1] considered this approach a way to improve the efficiency of training such linear classifiers, and therefore conducted this experiment, obtaining results similar to yours.
-
Regarding the experiment in Section 4.2, although I acknowledge that it is a new experiment that can provide additional causal evidence for existing frameworks, its results can be predicted from the existing induction-head framework. Prior research suggests that induction heads use joint features formed from x–y pairs as classification anchors. Consequently, reversing the direction of x would also reverse the direction of these joint features. Previous studies have even demonstrated more fine-grained error-correction mechanisms involved in this process.
That being said, I do believe, in fairness, that this experiment has some value, because it provides relatively solid experimental evidence for a phenomenon that we intuitively expected to occur, thereby offering additional support for existing frameworks.
- The experiment in Section 5.1 can be evaluated in much the same way.
-
-
Technical concerns.
-
The attempt in Section 3 to select words with "weaker separability" in order to reduce potential interference from direct shortcut decoding deserves some credit. However, this does not constitute complete control of the relevant variables, since the authors merely select a subset of words with "weaker separability." Consequently, the resulting conclusions still do not advance beyond prior research.
-
These experiments are restricted to classification tasks with a limited set of single-token labels (even the experiment in Section 4.2, which I find relatively interesting, is restricted to binary classification). This is a rather outdated experimental setting in 2027, given that classification-based ICL has already been investigated very extensively.
-
-
(Regrettably, I could copy this weakness verbatim into every paper in my review batch.) The manuscript appears to be AI-generated without sufficient manual refinement, with little consideration for human readability. It retains numerous characteristics that make AI-generated papers difficult to read, requiring extensive revisions throughout the manuscript. For example:
-
Extremely poor presentation of experimental results. Specific numerical results are interspersed throughout the English prose, greatly distracting the reader. Reading the entire paper feels like reading an appendix. Moreover, this way of presenting numerical results is quite outdated. Normally, figures and tables should be used to separate such details from the main logical narrative of the paper.
-
Overuse of mathematical formulas: Some theoretical details and mathematical representations could be moved to the appendix, while the content essential to the argument and the more intuitive, easily understandable explanations could remain in the main text. I do not think the mathematical intuition underlying this paper is particularly complicated, but the authors have made it unnecessarily complex and extensively used various subscripts and superscripts, making the paper considerably more difficult to read.
-
Extensive use of nonstandard, invented, or reworded terminology, which gradually proliferates throughout the paper. One example is the aforementioned "emergent geometry."
-
Note that I acknowledge having used an AI agent to assist me in reading this paper, as it was exceptionally difficult to read. However, the review comments were written entirely by myself and subsequently translated from my native language into English by AI.
-
[1] Token-based Decision Criteria Are Suboptimal in In-context Learning
Questions for authors
See Weaknesses.
2 comments
Thanks for your reply. I think some of my concerns are addressed. Therefore, I return 1 score for you.
Moreover, would you like to reduce the number of mathematical formulas in your paper? I think they are not so necessary but difficult to read.
Response to Reviewer KRBf
We thank the reviewer for their careful assessment and detailed feedback. We address novelty, significance, and generalizability in the General Response above, which we invite the reviewer to read for out full argument. There, we clarify how our contribution goes beyond established findings on class separability and retrieval heads by causally characterizing the native prototype computation, and present new experiments demonstrating generalization to sentence classification across four models. Below, we address the reviewer's specific concerns.
Purpose of Section 4.1
Section 4.1 establishes the prototype geometry whose direct causal effect is established in Section 4.2. As noted in line 150, our approach is conceptually similar to the paper cited in the review, but it goes further. While the cited paper shows that representations from different prompts cluster, we show that representations cluster within the same prompt. This distinction is important because the LLM makes its decision based on a single prompt. We highlight this point in lines 199–201 of the original manuscript. We revised Section 4.1 to further clarify its role in the paper.
Sec. 4.2 and induction-head mechanisms
Note that this section shows that the classification prediction is causally affected by the prototypes of the two classes.
We do not understand how our findings can be anticipated from existing induction-head mechanisms, as the review claims. Identifying induction heads does not establish how they combine class information to produce predictions. Further, the induction head mechanism does not per se suggest that predictions are built on the basis of the class prototypes of the word representations.
Purpose of Section 5.1
The results of sec. 5.1 are not claimed as novel - indeed, we cite the prior works that do such a task in this section. This section computes a head selection, and validates basic properties of these heads (the Q,K,V experiments). This set of heads, and the basic prediction circuit introduced through them is the key tool to establish the effect of label key prototypes and margins in the rest of section 5. We emphasized this positioning of 5.1 in our revision.
Technical concern on selecting weaker separability
We are not sure we understand the review's concern and would appreciate further clarification. The goal of our controlled selection of words is to remove obvious prototype information from the input and test whether the model fails without it. Our results show that the model can still recover prototype information on its own, even when it is not clearly present in the input.
Presentation
We agree that the presentation could better highlight the paper's central contributions. We updated the introduction and reorganized parts of the manuscript in our revision to make the mechanistic argument and its supporting evidence easier to follow.
Regarding the review's specific concerns:
"Emergent" geometry: We use this term only twice; once when discussing related work and once in the limitations. The term also appears in the titles of three papers we cite, so its use has precedent in the literature.
Numerical results in prose: We consistently present numerical results in figures and tables, mentioning only particularly relevant or illustrative results in the text.
Notation: All notation is defined throughout the paper, and we use formulas to state our conclusions precisely. We will further improve clarity where needed.