acceptodds
Under review as a conference paper at ICLR 2027

A Sparse Prototype Readout for LLM In-Context Classification

Abstract

What class-level information in a language model’s contextual representations drives in-context classification? How is that information transformed into a prediction? Across four open-weight LLMs, we trace this computation from contextual representations to the final readout. Contextual processing progressively constructs a low-dimensional class geometry whose class-mean directions are causally linked to the model’s decision: continuously exchanging these directions reverses the output. We then follow this decision variable into a sparse set of attention heads. Contextual class means are reflected in class-mean keys, yielding headwise query–prototype margins in the native attention computation. Under the same geometric intervention, these margins vary approximately linearly and cross near zero where the contextual class-mean directions meet. On held-out instances, a fixed sparse combination of the native prototype margins predicts the model’s output with to and % to % prompt-level agreement. Thus, we identify a class-level geometric decision variable for in-context classification and a sparse prototype-based circuit that reads it out.

Primary area:
interpretability and explainable AI
Keywords:
in-context learning, LLMs, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KRBfOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to investigate how the internal separability of input text representations during ICL affects predictions. The authors conduct the following experiments: 1. Verify the existence of such separability. 2. Actively intervene on or reverse the directions associated with this separability and observe that the output is also reversed. 3. Locate and analyze the circuit responsible for this output effect, and perform a QKV decomposition of its functionality.

Critical strengths
  1. The experiment in Section 4.2, which shifts the centroids of demonstration encodings, is a new experiment that can provide additional causal evidence for existing frameworks.

  2. The experiment in Section 5.1, which decomposes the functionality of induction heads (? I am not entirely sure whether the heads you intervene on qualify as induction heads by definition, but they perform demonstration retrieval) into Q, K, and V functions, is consistent with our intuition about induction heads. Moreover, attempting to decompose the functionality of attention heads is an interesting direction.

Critical weaknesses
  1. Existing work on hidden-state geometry and induction heads has already covered most of the contributions of this paper. How the cluster separability of sentence encodings, particularly linear separability, participates in downstream ICL computation has already been extensively investigated.

    1. Experiments such as those in Section 4.1, which examine how separability and the accuracy of linear classifiers increase with model depth, have already been thoroughly investigated. Although the authors claim that their work differs by using the representations of x appearing in earlier demonstrations to predict the labels of x appearing in later demonstrations, (1) What is the purpose of this? What question are you trying to answer? Are you trying to demonstrate that hidden states do not rotate as demonstrations accumulate? Why not simply use established methods to reaffirm previous findings about separability as the foundation of your study, rather than presenting this as a new experimental finding with an unclear purpose? (2) Note that prior work has already performed essentially the same experiment. [1] considered this approach a way to improve the efficiency of training such linear classifiers, and therefore conducted this experiment, obtaining results similar to yours.

    2. Regarding the experiment in Section 4.2, although I acknowledge that it is a new experiment that can provide additional causal evidence for existing frameworks, its results can be predicted from the existing induction-head framework. Prior research suggests that induction heads use joint features formed from x–y pairs as classification anchors. Consequently, reversing the direction of x would also reverse the direction of these joint features. Previous studies have even demonstrated more fine-grained error-correction mechanisms involved in this process.

    That being said, I do believe, in fairness, that this experiment has some value, because it provides relatively solid experimental evidence for a phenomenon that we intuitively expected to occur, thereby offering additional support for existing frameworks.

    1. The experiment in Section 5.1 can be evaluated in much the same way.
  2. Technical concerns.

    • The attempt in Section 3 to select words with "weaker separability" in order to reduce potential interference from direct shortcut decoding deserves some credit. However, this does not constitute complete control of the relevant variables, since the authors merely select a subset of words with "weaker separability." Consequently, the resulting conclusions still do not advance beyond prior research.

    • These experiments are restricted to classification tasks with a limited set of single-token labels (even the experiment in Section 4.2, which I find relatively interesting, is restricted to binary classification). This is a rather outdated experimental setting in 2027, given that classification-based ICL has already been investigated very extensively.

  3. (Regrettably, I could copy this weakness verbatim into every paper in my review batch.) The manuscript appears to be AI-generated without sufficient manual refinement, with little consideration for human readability. It retains numerous characteristics that make AI-generated papers difficult to read, requiring extensive revisions throughout the manuscript. For example:

    • Extremely poor presentation of experimental results. Specific numerical results are interspersed throughout the English prose, greatly distracting the reader. Reading the entire paper feels like reading an appendix. Moreover, this way of presenting numerical results is quite outdated. Normally, figures and tables should be used to separate such details from the main logical narrative of the paper.

    • Overuse of mathematical formulas: Some theoretical details and mathematical representations could be moved to the appendix, while the content essential to the argument and the more intuitive, easily understandable explanations could remain in the main text. I do not think the mathematical intuition underlying this paper is particularly complicated, but the authors have made it unnecessarily complex and extensively used various subscripts and superscripts, making the paper considerably more difficult to read.

    • Extensive use of nonstandard, invented, or reworded terminology, which gradually proliferates throughout the paper. One example is the aforementioned "emergent geometry."

    • Note that I acknowledge having used an AI agent to assist me in reading this paper, as it was exceptionally difficult to read. However, the review comments were written entirely by myself and subsequently translated from my native language into English by AI.

[1] Token-based Decision Criteria Are Suboptimal in In-context Learning

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KRBf

We thank the reviewer for their careful assessment and detailed feedback. We address novelty, significance, and generalizability in the General Response above, which we invite the reviewer to read for out full argument. There, we clarify how our contribution goes beyond established findings on class separability and retrieval heads by causally characterizing the native prototype computation, and present new experiments demonstrating generalization to sentence classification across four models. Below, we address the reviewer's specific concerns.

Purpose of Section 4.1

Section 4.1 establishes the prototype geometry whose direct causal effect is established in Section 4.2. As noted in line 150, our approach is conceptually similar to the paper cited in the review, but it goes further. While the cited paper shows that representations from different prompts cluster, we show that representations cluster within the same prompt. This distinction is important because the LLM makes its decision based on a single prompt. We highlight this point in lines 199–201 of the original manuscript. We revised Section 4.1 to further clarify its role in the paper.

Sec. 4.2 and induction-head mechanisms

Note that this section shows that the classification prediction is causally affected by the prototypes of the two classes.

We do not understand how our findings can be anticipated from existing induction-head mechanisms, as the review claims. Identifying induction heads does not establish how they combine class information to produce predictions. Further, the induction head mechanism does not per se suggest that predictions are built on the basis of the class prototypes of the word representations.

Purpose of Section 5.1

The results of sec. 5.1 are not claimed as novel - indeed, we cite the prior works that do such a task in this section. This section computes a head selection, and validates basic properties of these heads (the Q,K,VQ,K,V experiments). This set of heads, and the basic prediction circuit introduced through them is the key tool to establish the effect of label key prototypes and margins in the rest of section 5. We emphasized this positioning of 5.1 in our revision.

Technical concern on selecting weaker separability

We are not sure we understand the review's concern and would appreciate further clarification. The goal of our controlled selection of words is to remove obvious prototype information from the input and test whether the model fails without it. Our results show that the model can still recover prototype information on its own, even when it is not clearly present in the input.

Presentation

We agree that the presentation could better highlight the paper's central contributions. We updated the introduction and reorganized parts of the manuscript in our revision to make the mechanistic argument and its supporting evidence easier to follow.

Regarding the review's specific concerns:

  • "Emergent" geometry: We use this term only twice; once when discussing related work and once in the limitations. The term also appears in the titles of three papers we cite, so its use has precedent in the literature.

  • Numerical results in prose: We consistently present numerical results in figures and tables, mentioning only particularly relevant or illustrative results in the text.

  • Notation: All notation is defined throughout the paper, and we use formulas to state our conclusions precisely. We will further improve clarity where needed.

Thanks for your reply. I think some of my concerns are addressed. Therefore, I return 1 score for you.

Moreover, would you like to reduce the number of mathematical formulas in your paper? I think they are not so necessary but difficult to read.

Reviewer_iJs4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies in-context binary classification using WordNet-derived class pairs, single-word demonstrations, and arbitrary output labels. It characterizes the contextual class geometry that develops across layers and demonstrates that sustained rotations exchanging support-class mean directions can reverse predictions. The authors then localize sparse attention heads using a counterfactual-gradient score related to attribution patching and validate them through bidirectional activation patching. They connect the contextual word geometry to class-mean keys at support-label positions and show that a fitted combination of the selected heads' query–prototype margins predicts native output margins with high fidelity. The study provides useful evidence about the readout mechanism, but I find its contribution too limited within this restricted task setting.

Critical strengths
  1. Complementary interventions support the readout account. Exchanging contextual class-mean directions or selected heads' key means changes predictions, while replacing individual support-label keys with their class means largely preserves native behavior. These tests support the connection between contextual geometry and attention readout beyond correlation alone.
Critical weaknesses
  1. Limited incremental contribution in a restricted setting. Previous literature has explored related ideas (Xiong et al.; Yu and Ananiadou; Cho et al.; Sun et al.), as the paper acknowledges. The controlled connection between contextual class means and selected heads' key means is useful, but I find its significance insufficient within balanced, single-word binary classification. Given growing LLM capabilities, this familiar setting needs a consequential new insight. Fidelity to native predictions, including errors, supports a descriptive account rather than practical performance improvement.

  2. Limited method novelty and comparative validation. The counterfactual-gradient score is a two-endpoint variant of attribution patching, but an empirical comparison with standard attribution patching is not reported. Also, I do not find random-head controls for the interventions in the paper. These gaps leave the added value and specificity of the methodological choices insufficiently established. Class-geometry construction remains unexplained (§6). The existing model-size comparisons and semantic-group-held-out fitting provide useful within-setup evidence, but all experiments use the fixed word -> label template; robustness to separators, wording or arrangements remains unestablished.

Questions for authors
  1. Prompt-format dependence. Do the intervention results and prototype-readout fidelity persist with alternative separators or instruction-based templates? On the already evaluated Llama sizes, do they remain stable when semantic examples and label bindings are held fixed?

  2. Presentation suggestion. The abstract and introduction need a clearer claim linking contextual geometry to attention readout, followed by the experiments testing each link. Task-by-task results and technical detail currently obscure that sequence, including in Figure 1.

  3. Method comparison and intervention specificity. How do head rankings and patching effects compare with standard attribution patching at matched head budgets? What do matched random-rotation or random-head controls show beyond the random-partition input baseline?

3 comments
AuthorsOpenReview ↗

Response to Reviewer iJs4 1/2

We thank the reviewer for their careful assessment and constructive suggestions. We address novelty, significance, and generalizability of our findings in the General Response at the beginning of our rebuttal, which we invite the reviewer to read for the full argument. There, we clarify our contribution of causally identifying a native prototype-level computation, motivate our controlled setting, and present new experiments demonstrating generalization of our core claims to sentence classification across four models, including intervention results and prototype-readout fidelity. Below, we address the reviewer's specific methodological concerns.

Relation to attribution patching

As mentioned in the review, our head-localization score is a symmetric variant of standard attribution patching, rather than a fundamentally different method.

We average the gradients computed on the original and label-reversed prompts because both label assignments are equally valid; neither is intrinsically the clean or corrupted condition. This makes the score symmetric with respect to the arbitrary a/b label choice. We do not claim a methodological improvement over standard attribution patching: the score only ranks candidate heads, whose causal importance is subsequently established through exact activation patching (Section 5.1). We already discuss this connection in the manuscript (lines 780-793) and have made it more explicit in the main text in our revision (lines 254-258 in revision).

To further evaluate the impact of using the averaged gradient over a single gradient, we ran the head ranking with standard attribution patching for the Llamma-3.1-8B model and found that the set of top-16 ranked heads are exactly identical.

Intervention specificity

We have added a new Section 5.5 on intervention specificity to our revision that shows the relevant results.

We agree that the specificity of our interventions was not sufficiently established in the manuscript. We therefore evaluate matched random controls across all four models.

First, we compare rescue and corruption using the identified heads against random head sets of the same size. The following table reports classification accuracy (%) when patching the top K=64K=64 heads in either direction (new Table 3). Rescue refers to restoring the original head activations in the label-swapped run, while corruption refers to inserting label-swapped head activations into the original run.

ModelOriginalRescue SelectedRescue RandomCorruption SelectedCorruption Random
Llama-3.1-8B82.082.818.617.180.9
Qwen2.5-7B83.283.317.316.382.4
Mistral NeMo 12B84.585.015.914.983.9
Llama-3.1-70B82.782.315.016.782.3

The selected heads almost completely rescue or reverse predictions, whereas random heads have negligible effects.

We also compare class-directed prototype exchanges with magnitude-matched random perturbations. The table reports accuracy (%) at full intervention strength (β=1\beta=1; new Table 2 and Figure 7). Swap perturbs representations to exchange class prototypes, while Random dir. perturbs them in random directions. The magnitude of the perturbation (angle for word-space rotation and displacement for key-space translation) are matched to the semantic intervention.

ModelCleanAttention-key SwapAttention-key Random dir.Word-prototype SwapWord-prototype Random dir.
Llama-3.1-8B82.021.479.116.863.9
Qwen2.5-7B83.428.480.722.058.7
Mistral NeMo 12B84.522.682.315.869.3
Llama-3.1-70B82.727.784.317.766.7

Exchanging the semantic class prototypes strongly reverses predictions, whereas random-direction changes produce substantially smaller effects. Together, these controls demonstrate that the observed causal effects are specific to the identified heads and semantic class directions, rather than merely resulting from perturbations to internal activations.

AuthorsOpenReview ↗

Response to Reviewer iJs4 2/2

Class-geometry construction remains unexplained

We agree that identifying the upstream computations responsible for constructing class geometry remains an important open question in the field, as acknowledged in Section 6. However, this is beyond the scope of our present work. Our focus is the complementary question of how this geometry causally governs native predictions and what computation implements its readout. Our experiments establish this connection without requiring a complete account of the upstream construction process.

Thank you for your thoughtful responses and revised manuscript. However, I still find the presentation not easy to follow. Also, I'm not convinced if prototypes sufficiently explain models' internal mechanism, or are just approximations behind the real mechanism. Although the revised manuscript is better now, the presentation (e.g., Figure 1) and the explanations are still not fulfilling. As a result, I would retain my original score.

Reviewer_QTrHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies binary in-context classification in four open-weight LLMs and shows that within-prompt class-mean directions in contextual representations causally control the decision. By manipulating these directions, the output can be reversed. The paper traces this to a sparse set of attention heads, where query–key-prototype margins track the intervention linearly, and replacing support-label keys with their class means preserves most predictions.

Critical strengths
  1. It connects the full path from internal representation to answer. Instead of only showing that category information can be read from the model, the paper tests each step: the category averages affect the answer, those averages show up in specific heads, and those heads predict the output
  2. Clear answer to an open question. It gives direct evidence that, in this simplified setting, the model compares against category averages rather than individual examples
Critical weaknesses
  1. This paper is not easy to follow. Structurally, in the introduction, the authors already introduce the notation, the task, and the method. This kind of density of information makes it difficult to understand what the main idea and contributions of this paper are.
  2. Much of it builds closely on earlier work. The paper itself says that finding a small set of heads, and the split of roles between queries, keys, and values, was shown before. It also names Xiong et al. (2026) as the closest earlier work linking category averages to the model’s output. The new part is mainly linking these findings together
  3. The setting is narrow, and there is no practical use shown. All experiments use two balanced categories in a very simpe setup. The authors acknowledge that more realistic tasks remain untested, and the paper does not show how the findings could be used, for example to improve or debug in-context learning.

Reference:

  1. Hua-Dong Xiong, Li Ji-An, Robert C. Wilson, Kwonjoon Lee, and Xue-Xin Wei. Large language models reorganize representational geometry during in-context learning. In Conference on Language Modeling, 2026. URL https://arxiv.org/abs/2605.28854.
Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer QTrH

We thank the reviewer for their careful assessment and for recognizing the value of our causal account connecting contextual representations to native predictions. We address the reviewer's main concerns about novelty, significance, and generalizability in detail in the General Response at the beginning of our rebuttal, which we invite the reviewer to read for the full argument.

In particular, we clarify how our contribution goes beyond prior work: we identify and causally validate a prototype-level computation within the model's native readout, whereas prior work either identifies implicated attention heads without establishing their computation or provides descriptive evidence of prototype geometry without establishing its causal role. We also explain the scientific motivation for our controlled setting and present new experiments showing that the same mechanism extends to natural-language sentence classification, with more varied demonstrations and prompt formats. We also added a new Section 6 to the manuscript to present new evidence supporting the generality of our claims across broader and more diverse settings.

We agree that the presentation could better highlight these contributions. We have streamlined the introduction and reorganized the manuscript to make the central mechanistic argument easier to follow.

Thanks for the response. On W1: The introduction still walks through everything section by section for multiple dense paragraphs before the contribution. It is still not easy to follow. On W3: The experiment in the new section 6 addresses part of my concern. However, no use or practical implications are demonstrated, so that part of the weakness stands. Overall, the rebuttal partially addresses W3 and clarifies W2. However, W1 and the practical use remain. I will maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to the Reviewers and Area Chair 1/2

We thank all reviewers for their careful evaluations and helpful feedback. We would like to address three central points raised by reviews.

Novelty and significance

Revised Sections 1 and 2, including the new comparison in Table 1, make these distinctions explicit.

Our core novelty lies in three high-level results:

  • We establish, in a causal way, that both word-level prototypes and key-level prototypes in the output heads, have a large effect on the model outputs. Prior work either ignores the effect of these prototypes, or gives weak associative evidence.
  • We identify the precise computation of the output heads to a high fidelity. While prior work has also identified output heads, our work leads to a much sharper causal characterisation of exactly what these heads are computing.
  • Finally, we establish the causal effect of word representation prototypes on output head label-key prototypes, thus directly showing how information in the features is transferred to the output heads.

Below we will give a more detailed account of prior work, and our novelty.

  • Word representations and "prototypes"
    • Cho et al. ("Token-based Decision...") take hidden state representations at the last layer of the query points averaged across many different prompts. Then they argue that these vectors are separated according the the class, and that a centroid-based classifier can separate them well. Note that this is at best an associative study: since information is averaged across prompts, the LLM can in no way access such separation.
    • Xiong et al. look at sentence repsresentations within a prompt at five different layers in the LLMs. Their conclusion, again, is that these are separated according to the class. Finally, they take last layer representations, and argue that a prototype classifier on these reps gets 60-70% fidelity to the LLM decision. Again this is associative: just because the features are separated at the last layer, does not mean that the LLM uses this last layer information.
    • Our approach is very different. We expilcitly intervene on the word-prototypes (class means) to produce a change in the LLM output. There are two salient points:
      • We obtain causal evidence that the word prototypes are used by the LLMs to produce their decisions by intervening on the prototypes directly.
      • This evidence is high-fidelity, across prompts: applying β=1\beta= 1 nearly reverses the decisions, applying β=1/2\beta = 1/2 sets the decisions to chance. Further, we capture the variation of the margin directly in terms of the intervention strength.
    • Thus, both the result and the evidence are new and much more pertinent to describing LLM behaviour.
  • Output heads
    • There are three main papers which identify output heads. In summary, these papers are identifying output heads, and together isolating that the information passes through only the label channels for these heads:
      • Yu and Ananiandou find output heads by setting their writeouts to zero. This only establishes that the output flows through these heads.
      • Sun et al: exchange attention across classes in these output heads. This shows that the classification is encoded through class-level differences in these heads
      • Cho et al: block attention on labels in these heads to produce failure to classify. This shows that the information passes through the label states in these heads.
    • What we do:
      • We identify that it is specifically label-key prototypes that matter, in a prompt by prompt way.
        • This is shown causally: manipulating these prototypes directly affects the output margin (sec 5.3)
        • Further, this shows that within class variation does not matter to the main degree (sec 5.3).
        • As a result, we can write a very clean formula for what these output heads are contributing (equation 13).
      • We further identify that the effect of these output heads on the final output of the LLM is additive (equation): m=∑hαhmhm = \sum_h \alpha_h m_h. (sec 5.4)
        • Thus, not only are these heads all affecting the final output, we can exactly figure out just how they are doing it: it is linearly.
      • We connect the word-prototype information to these label-keys, thus identifying the flow of information in the network (sec 5.2).
        • Again, note that this is by direct intervention: rotating the word prototypes leads to a directly measured change in the output head margins, which in turn bears an approximately linear relationship to the degree of mixing β\beta across prompts.
      • Let us note again that the fidelity of all of these results is very high (>90>90% agreeement over individual prompts).
    • Thus, we go much further than this prior work by establishing, in a high-fidelity sense,
      • the computation performed by these output heads, including their dependence only on label-key prototype separation
      • their joint contribution to the final decision
      • their direct dependence on the word-representation prototypes.
AuthorsOpenReview ↗

General Response to the Reviewers and Area Chair 2/2

Setting and Practical use

New Section 6 includes these results in the revision and establishes how our conclusions generalize beyond the single-token word setup.

Our goal is not to improve semantic-word classification, but to understand how LLMs learn from in-context examples and produce predictions. We consider identifying this computation a scientific contribution in its own right. We use a controlled setting to enable precise causal analysis, but agree that testing whether our conclusions generalize to more complex classification tasks is important. We therefore provide additional experiments below.

To test generalization beyond the original setting, we conduct additional experiments on natural-language sentence classification using CLINC150, evaluating 25 disjoint intent pairs with 10 held-out queries per pair. Compared with our word-classification experiments, this setting introduces:

  • Sentence-level classification: Full natural-language sentences instead of individual words.

  • Unbalanced demonstrations: 4--10 examples per class.

  • Randomized labels: Two randomly chosen distinct letters per prompt.

  • Varied prompt formatting: Arrow separators sampled from ->, :, and =; demonstration separators sampled from newline, ;, |, and /.

Crucially, we reuse the heads ranked in the original word-classification experiments, without reselecting them. Across all four models, these heads remain causally important: patching their clean outputs into label-swapped prompts restores predictions, whereas patching their label-swapped outputs into clean prompts corrupts predictions. The following table reports classification accuracy (%) when patching the 16/32 highest-ranked heads in either direction (new Table 4).

ModelCleanCorruptionLabel-swappedRescue
Llama-3.1-8B99.60.00.099.6
Qwen2.5-7B98.02.01.698.4
Mistral NeMo 12B100.00.00.0100.0
Llama-3.1-70B100.00.00.0100.0

Across all four models, the same prototype-based computation remains causally important: replacing individual label keys with their class means largely preserves predictions, whereas swapping class means reverses them. Similarly, swapping class-mean directions in upstream sentence representations reverses predictions. The following table reports classification accuracy (%) under these interventions (new Table 5).

ModelCleanAttention key collapseAttention key swapUpstream sentence swap
Llama-3.1-8B99.699.63.611.6
Qwen2.5-7B98.497.212.820.8
Mistral NeMo 12B100.0100.06.411.6
Llama-3.1-70B100.0100.07.212.8

Furthermore, the selected heads' query--key prototype margins accurately predict the LLM's classification margin on held-out prompts (R2=.94R^2=.94 to .96.96). Together, these results show that the same representation-to-readout mechanism generalizes to sentence-level classification under more varied demonstration and prompt conditions.

Presentation

We agree that the presentation could better highlight the paper's central contributions. We revised the abstract and introduction (Section 1) to highlight the core novelty more clearly and streamlined the presentation throughout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.