acceptodds
Under review as a conference paper at ICLR 2027

In-Context Task Learning: Reading Out a Kernel Classifier

Abstract

Task learning in context requires using demonstrated mappings when pretrained knowledge and other task cues do not determine them. We investigate whether small sets of attention heads can support direct prediction and reuse of demonstration evidence. Starting from Llama-3.1-8B on TREC-fine with 180 demonstrations, we identify mapping-sensitive carrier heads through paired intact/randomized-label contrasts. Their writes support a kernel classifier based on query-dependent retrieval of contextual class votes, characterized through query-weight controls and value interventions. This account motivates Selective TL Memory, which caches demonstration-side states while preserving live retrieval and direct readout. Across models, natural-language tasks, and demonstration budgets, thirty-head full-access readouts match or exceed final-model accuracy in 12 of 18 settings. Selective access to additional demonstrations improves a fixed carrier readout in all 18 settings at both tested head budgets; thirty-head selective readout exceeds every adapted vector baseline in 14. Contextual label states often support compact reuse, while access and receiver controls show that the selected readouts depend on states constructed by the surrounding network. In a prefix-cached benchmark, shared memory supports repeated queries with latency close to the smaller prompt and reduced resident storage relative to full access.

Primary area:
interpretability and explainable AI
Keywords:
In-context learning, Task learning, Mechanistic interpretability, Kernel classifiers, Efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_CR2KOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates the mechanisms underlying in-context task learning (TL), particularly how pretrained language models infer arbitrary input–label mappings from demonstrations. The authors introduce a mapping-sensitive attention-head selection criterion based on paired contrasts between intact and randomized demonstration labels. They identify a small subset of "carrier heads" whose outputs support direct classification through query-dependent retrieval of contextual class votes.

Critical strengths
  1. A systematic and informative investigation of mapping-sensitive attention heads.

The proposed paired label-randomization criterion provides a meaningful extension of prior attention-head localization methods by targeting changes in class-directed outputs under different demonstrated mappings.The authors support their interpretation through several complementary analyses, including query-weight replacement, demonstration reordering. In particular, the substantial degradation under query-independent retrieval weights suggests that the selected readouts exploit query-specific information.

  1. A meaningful connection between mechanistic analysis and memory reuse.

The paper translates its mechanistic observations into Selective TL Memory, a concrete inference-time mechanism that preserves live retrieval over cached demonstrations.The evaluation covers multiple model families, tasks, and demonstration budgets. The matched no-access controls show consistent improvements when additional memory becomes visible to the selected heads. Comparisons against vector-based methods further illustrate the potential limitations of compressing demonstration evidence into fixed representations.

Critical weaknesses
  1. The methodological novelty is relatively incremental compared with existing work.

Several central ingredients have already been explored in prior literature. TSLA identifies attention heads associated with task learning, kernel-based accounts describe in-context learning as weighted retrieval over demonstrations, and label-anchor studies establish the importance of contextualized label positions. Furthermore, the kernel-vote decomposition in Equation (3) follows directly from the linearity of attention's value aggregation and output projection. Therefore, the decomposition itself does not establish a new learning algorithm.

The main methodological novelty appears to lie in the paired mapping-sensitive head-selection criterion and its integration with selective memory access. While useful, it remains unclear whether this combination yields a substantially new understanding beyond existing accounts of semantic retrieval and label information flow, or whether much of the practical benefit could be reproduced by simpler selection and caching strategies.This limits the significance of the algorithmic contribution, although the accompanying empirical analysis remains valuable.

  1. Limited generalization evidence and an incomplete practical comparison for selective memory.

Most evaluations focus on arbitrary label reassignment over familiar semantic classification tasks. The class-to-token mapping is fixed within each model–task configuration, and the head-selection procedure uses labeled validation queries. It is therefore unclear how well the selected carrier heads generalize to unseen label assignments or genuinely new decision rules without task-specific reselection.

In addition, the efficiency evaluation excludes one-time memory preparation and carrier discovery, and the memory states are initially constructed from full-prefix encoding. The paper does not provide a sufficiently direct comparison with competitive equal-budget KV-cache selection or retrieval-based attention methods.

Questions for authors

Q1. The class-to-token assignments are fixed across seeds within each model–task configuration. How well do the identified carrier heads transfer to previously unseen label permutations without head reselection? For example, could the authors select heads using one arbitrary label assignment and evaluate them on independent assignments of the same classification problem? This experiment would help establish whether the selection criterion identifies a reusable task-learning mechanism rather than a readout specialized to a particular mapping.

Q2.Could the authors compare Selective TL Memory against simpler KV-cache selection strategies under matched resident-memory and online-computation budgets, including retaining only contextualized label-position states without mapping-sensitive head selection? Additionally, how many repeated queries are required to amortize the cost of full-prefix memory preparation and head discovery?

Q3. The value-repair interventions modify value rows across all query heads in carrier-containing layers, rather than isolating the selected carriers. Can the authors provide more targeted interventions to distinguish the causal contribution of selected carrier heads from that of other heads in the same layers? This would strengthen the interpretation that the identified carriers implement a distinct task-learning pathway, rather than primarily serving as useful readout locations for information constructed elsewhere in the model.

2 comments

We thank the reviewer for the careful review and address each point below.

1. (W1) Novelty relative to TSLA, kernel accounts and label-anchor studies.

We agree that the kernel-vote decomposition itself follows from linearity and does not establish a learning algorithm. Our work builds on TSLA and on the kernel and label-anchor accounts, and differs in three ways. First, their evidence uses 4–16 demonstrations, and we work with 70–180, where task learning is actually expressed (Figure 1). Second, our selection uses a paired contrast between intact and randomized labels, which cancels any contribution unchanged by relabeling (Eq. 8), and the heads it selects read out 14.67–25.87 points more accurately than TSLA-TL heads at the same budget on TREC-fine (Table 1). Third, we test the kernel account with query-weight, order and value-intervention controls, and we use the readout as a classifier and as a memory interface, which also turns the label-anchor observation into a compact memory.

2. (Q1, W2) Transfer to unseen label assignments without reselection.

Following the suggestion, we kept the heads selected on the original class-to-token assignment and evaluated them on a dfferent label set by random selection (Llama-3.1-8B/TREC-fine 2→5, 3 seeds, test accuracy %):

ReadoutConditionNew label setOriginal
ModelNatural35.6036.93
ModelOurs, 8 heads38.0038.40
ModelOurs, 30 heads44.8046.67
ModelAll heads48.4051.60
DirectNo access, 8 / 30 heads27.60 / 30.9335.07 / 36.93
DirectOurs, 8 / 30 heads32.93 / 39.4744.40 / 47.47
DirectFull access, 8 / 30 heads38.93 / 43.7350.27 / 51.60

On the new label set, the ordering No access < selective access < full access holds for both budgets and both readouts, and the model readout with 30 heads keeps most of the gain over Natural (35.60 → 44.80, against 48.40 for all heads). Selective access still improves the fixed direct readout by 5.3 points with 8 heads and 8.5 with 30, against 9.3 and 10.5 on the original assignment. The direct readout is lower in absolute terms on the new label set, including under full access (43.73 against 48.40 for all heads). The new assignment is also slightly harder in itself (all heads 48.40 against 51.60). We will add this experiment to the paper.

3. (Q2, W2) Simpler KV-cache selection at matched budgets, and amortization.

At matched resident memory we compared with H2O [1], a widely used dynamic KV-cache selection method that retains positions for all heads without any head selection (Llama-3.1-8B/TREC-fine 5→10, eight heads):

Matched budgetOursH2OH2O + Ours (30 heads)
Ours (anchor, 8 heads) [0.001 GiB]51.7349.4752.53
Ours (direct, 8 heads) [0.009 GiB]51.6052.9354.67

At the anchor budget the resident memory is so small that H2O cannot retain enough to help. At the direct budget the two perform similarly. Our method is not in competition with dynamic KV-cache selection: the two are complementary, and combining H2O with our 30-head memory at the same budget gives a further gain.

The table gives the number of queries after which the total cost of our method, one-time cost plus per-query cost, is within 10% of the total cost of the standard prompt, for Llama-3.1-8B/TREC-fine 5→10 with the timings of Table 6:

Compared withAttentionSettingMemory preparationMemory preparation + head discovery
Standard K=5SDPAFull prompt17837
Standard K=5SDPAPrefix cached929,500
Standard K=5EagerFull prompt40474
Standard K=5EagerPrefix cached1,05017,500
Standard K=10SDPAFull prompt278
Standard K=10SDPAPrefix cached01,340
Standard K=10EagerFull prompt114
Standard K=10EagerPrefix cached06,800

Head discovery is the main one-time cost. Compared with standard K=10, memory preparation pays for itself almost immediately, and head discovery within tens of queries in the full-prompt setting and within a few thousand with prefix caching. Compared with standard K=5, which never encodes the extra demonstrations, the counts are larger. We will add these numbers to the cost section.

4. (Q3) Isolating the selected heads from other heads in the same layers.

We repeated the repair experiment with the intervention restricted to the eight selected heads, leaving the other query heads in the same layers with the corrupted values (experiment in Figure 3C):

Repair scopeSelfSame-classCross-class
Full rows (all 32 heads in 6 layers, as in the paper)17.617.212.0
Eight selected heads only15.214.010.8

The pattern is the same as with full rows. Repairing only the eight selected heads raises accuracy from 5.6% to 15.2% with self donors, against 52.4% clean, and same-class and cross-class donors give 14.0% and 10.8%. This recovers 20.5% of the clean–corrupt gap, compared with 25.6% for full rows, so the eight heads account for more than half of the full-row effect. Self and same-class donors again outperform cross-class donors, which supports a role for class-consistent contextual values and shows that this role is carried mainly by the selected heads. We will use this to replace the existing full rows result.

[1] Zhang, Zhenyu, et al. "H2O: Heavy-hitter oracle for efficient generative inference of large language models." Advances in Neural Information Processing Systems 36 (2023): 34661–34710.

Reviewer_r9ZwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies attention heads that respond to demonstrated class mappings and uses their class scores for direct prediction. It tests whether these readouts retrieve information according to the query. Selective TL Memory reuses cached demonstration states through the selected heads while keeping query computation active. Experiments examine arbitrary label classification across language models and include synthetic rule learning tasks. Access controls show that the selected readout also relies on states constructed by the surrounding network.

Critical strengths
  • The paper provides an in-depth investigation of how LLMs learn arbitrary class-label mappings from many ICL demonstrations, along with detailed explanations.

  • I think the authors provide useful evidence beyond the algebraic decomposition of attention. eg in Table 7, replacing each query's retrieval weights with mean or permuted weights substantially reduces direct-readout accuracy. The order controls and comparisons with class-averaged intervention predictors make the retrieval account more convincing than a visualization of attention alone.

  • I appreciate the fixed readout comparison in Table 2. Selective access improves over no access in all 18 natural language settings at both head budgets. Holding the readout fixed makes the benefit of additional visibility easier to interpret.

Critical weaknesses
  • The mechanism account remains more complete for reading evidence than for explaining how the mapping is constructed. Equation 3 follows from attention's linear aggregation for any head set. The query controls make the selected readouts informative, but the repair interventions affect whole value rows across carrier-containing layers and therefore also affect unselected query heads. Table 14 shows small carrier-only ablation losses but much larger joint losses with MLPs in the Llama settings, so the individual losses do not establish that carriers are unimportant. Still, I think the unresolved construction of their states limits the explanatory advance over the label-vote account of Yu and Ananiadou and the head-localization work of Yang et al. The main new evidence concerns a useful readout and memory interface, rather than a localized computation of task learning.
  • The storage comparison in section 3.3 seems incomplete because it lacks a cache retention baseline at matched storage. Comparing accuracy with attention based retention at identical KV bytes would clarify whether carrier selection offers a better use of the memory budget.
  • I noticed that Table 2 includes a Qwen setting where final prediction quality declines despite improved direct readout. Section 3.3 acknowledges this divergence. I think tracing the retrieved evidence through later layers would help explain when selective memory harms final decisions.
Questions for authors
  • Can the authors clarify how performance changes when the base receiver is encoded independently of the increment demonstrations, rather than using contextualized base states from the full prefix?
  • How would the 14/18 comparison change if corrected increment-vector TSLA were evaluated across all settings? Can the authors separate the benefit of selective memory from that of changing the prediction readout?
  • How many repeated queries are needed to offset head discovery and memory preparation in section 3.3? Reporting those costs would clarify when selective reuse becomes worthwhile.
4 comments

We thank the reviewer for the careful review and address each point below.

1. (W1) Relation to Yu and Ananiadou and to Yang et al.

Thanks for pointing out Yu and Ananiadou and Yang et al.. Our work builds on both. Yu and Ananiadou describe few-shot in-context heads as a metric-learning kernel whose value–output pathway votes for labels, and TSLA locates task-recognition and task-learning heads through label-subspace geometry and a TL score normalized within one forward pass. We differ in three ways. First, their evidence uses 4–16 demonstrations, and we work with 70–180, where task learning is actually expressed (Figure 1). Second, our selection uses a paired contrast between intact and randomized labels, which cancels any contribution unchanged by relabeling (Eq. 8), and the heads it selects read out 14.more accurately than TSLA-TL heads at the same budget on TREC-fine (Table 1). Third, we test the kernel account with query-weight, order and value-intervention controls, and we use the readout as a classifier and as a memory interface, which goes beyond both accounts. How the states that these heads read are constructed is left open, as the reviewer says, and we state this as future work in the conclusion.

2. (W2) Cache-retention baseline at matched storage.

At matched resident memory we compared with H2O [1], a widely used dynamic KV-cache selection method (Llama-3.1-8B/TREC-fine 5→10, eight heads):

Matched budgetOursH2OH2O + Ours (30 heads)
Ours (anchor, 8 heads) [0.001 GiB]51.7349.4752.53
Ours (direct, 8 heads) [0.009 GiB]51.6052.9354.67

At the anchor budget the resident memory is so small that H2O cannot retain enough to help. At the direct budget the two perform similarly. Our method is not in competition with dynamic KV-cache selection: the two are complementary, and combining H2O with our 30-head memory at the same budget gives a further gain.

3. (W3) Tracing the Qwen divergence through later layers.

Thanks for the suggestion. On the current evidence, the late MLPs narrow the gap between the top-1 and top-2 tokens and sometimes flip the prediction (Appendix B.6), in line with the late-MLP counterbalancing reported in [2]. As you note, the effect can even be negative, and that is why we chose the direct variant as the main variant instead of the model variant.

Following your suggestion, we traced the per-layer readout on Qwen2.5-7B/TREC-fine 5→10 with 30 heads. Natural and Ours (model) read the residual stream at the answer position after each sublayer, and the two direct columns read the accumulated writes of the selected heads (◆ marks layers that contain selected heads):

LayerNaturalOurs (model)No access (direct)Ours (direct)
attn:22 ◆9.208.6714.9317.07
mlp:229.078.4014.9317.07
attn:23 ◆14.678.6726.4020.13
mlp:2314.277.6026.4020.13
attn:24 ◆24.1312.0028.6725.60
mlp:2423.6014.0028.6725.60
attn:25 ◆21.3313.3334.6738.53
mlp:2523.2014.4034.6738.53
attn:26 ◆26.8015.8740.8042.40
mlp:2633.2020.0040.8042.40
attn:27 ◆36.5329.4741.2052.93
mlp:2739.3330.0041.2052.93

The recomputed selected heads at attn:23 lower the direct readout below its no-access counterpart (20.13 vs 26.40) and hurt the model readout from there on. Ours (model) falls below Natural at attn:23 and never recovers (30.00 against 39.33 at mlp:27), while the direct readout recovers from attn:25 on and ends at 52.93 against 41.20. We will add this trace to the paper.

4. (Q1) Base receiver encoded independently of the increment.

We replaced the receiver with the standard ICL K=5 prompt (query at its own positions) on Llama-3.1-8B/TREC-fine 5→10, keeping the cached increment states. Test accuracy (%), three seeds × 250 queries:

ReceiverBaseOurs (model) 8Ours (model) 30Ours (direct) 8Ours (direct) 30
Standard ICL K=550.9352.4053.0752.9350.27
Original (Natural)49.3350.0050.4051.6053.07

With the standard K=5 receiver the method still improves over the base prompt in 3 of the 4 variants. The cached increment states keep the positions of the full prefix, so the receiver attends to them with mismatched RoPE offsets. This appears to lower the direct readout of the larger head set (50.27 with 30 heads vs 52.93 with 8), and the downstream computation corrects it (Ours (model), 30 heads: 53.07). We will add this comparison to the paper.

5. (Q2) Corrected TSLA across all settings, and memory versus readout.

The table below adds increment-vector TSLA (TL and TR) to all 18 settings.

ModelTaskLevelNaturalTSLA (original) TLTSLA (original) TRTSLA (increment) TLTSLA (increment) TROurs (direct) 8Ours (direct) 30
Llama-3.1-8BTREC-fine2→536.9332.6712.4038.5336.6744.4047.47
Llama-3.1-8BTREC-fine5→1049.3339.2019.0749.4748.2751.6053.07
Llama-2-7B + SelfExtendTREC-fine2→514.9315.6013.8713.7313.8714.9321.07
Llama-2-7B + SelfExtendTREC-fine5→1034.1325.8728.6731.3332.8024.6731.87
Qwen3-14BTREC-fine2→540.2717.7310.5341.4740.0045.0749.07
Qwen2.5-7BTREC-fine2→533.6027.600.6729.0726.8041.3343.47
Qwen2.5-7BTREC-fine5→1038.5333.330.8035.7322.6745.8752.67
Qwen3-8BTREC-fine2→537.335.205.0731.4733.3340.8043.47
Qwen3-8BTREC-fine5→1033.605.604.4028.9330.0037.6042.27
Qwen3-4BTREC-fine2→533.604.5312.1332.2729.4741.7346.67
Qwen3-4BTREC-fine5→1034.9316.2723.8729.2034.4040.4041.33
Llama-3.1-8Bbanking771→254.4010.4016.4054.1355.0748.0061.87
Llama-3.1-8Bclinc1501→269.0735.0733.2069.8769.4758.9376.40
Llama-3.1-8Bdbpedia142→571.2063.8749.6079.8770.0083.2088.93
Llama-3.1-8Bdbpedia145→1090.9384.1371.2092.8089.2091.4793.47
Llama-3.1-8Byahoo2→538.2722.2716.1333.3335.3335.4739.60
Llama-3.1-8Byahoo5→1057.7331.8716.0058.0057.2054.6755.33
Llama-3.1-8Byelp2→521.3322.8024.8024.6722.8022.0024.93

Test accuracy (%), three seeds × 250 queries. Bold marks the highest score in each row among the adapted methods (Natural excluded).

The increment vector helps in some settings, e.g., Qwen3-14B/TREC-fine 2→5 and Llama-3.1-8B/yelp 2→5, and gives the top score in two rows (Llama-2 5→10 via TR, yahoo 5→10 via TL), though it is still lower than Natural on Llama-2 5→10 via TR. Overall, our thirty-head direct readout has the highest score in 16 of the 18 rows.

On separating the two benefits, Table 2 already contains both conditions. Changing only the prediction rule (No access (direct) minus Natural, 30 heads) changes accuracy by −0.6 points on average across the 18 settings, with mixed sign. Opening selective access with the readout fixed (Ours (direct) minus No access (direct)) adds 7.4 points on average and is positive in all 18 settings, from +0.40 on Llama-3.1-8B/TREC-fine 5→10 to +18.53 on dbpedia14 2→5.

6. (Q3) Repeated queries needed to offset discovery and memory preparation.

The table gives the number of queries after which the total cost of our method, one-time cost plus per-query cost, is within 10% of the total cost of the standard prompt, for Llama-3.1-8B/TREC-fine 5→10 with the timings of Table 6:

Compared withAttentionSettingMemory preparationMemory preparation + head discovery
Standard K=5SDPAFull prompt17837
Standard K=5SDPAPrefix cached929,500
Standard K=5EagerFull prompt40474
Standard K=5EagerPrefix cached1,05017,500
Standard K=10SDPAFull prompt278
Standard K=10SDPAPrefix cached01,340
Standard K=10EagerFull prompt114
Standard K=10EagerPrefix cached06,800

Head discovery is the main one-time cost. Compared with standard K=10, memory preparation pays for itself almost immediately, and head discovery within tens of queries in the full-prompt setting and within a few thousand with prefix caching. Compared with standard K=5, which never encodes the extra demonstrations, the counts are larger. We will add these numbers to the cost section.

[1] Zhang, Zhenyu, et al. "H2O: Heavy-hitter oracle for efficient generative inference of large language models." Advances in Neural Information Processing Systems 36 (2023): 34661–34710.
[2] McGrath, Thomas, et al. "The hydra effect: Emergent self-repair in language model computations." arXiv preprint arXiv:2307.15771 (2023).

Thank you for the detailed response and the additional experiments. The receiver comparison, updated TSLA results, and targeted repair help address most of my concerns. However, the receiver and H2O comparisons remain limited to one setting, with H2O achieving higher accuracy at the larger budget. Head discovery also requires many queries to amortize with prefix caching. The layer trace is useful, but how the contextual states are formed and why selective access can hurt final predictions remain only partly explained for me.

Thanks for your kind reply.

H2O comparison

To prove our adavantage on Ours(anchor) and the combination of H2O and Ours method. We do the comparison with H2O with multiple LLMs.

(1) Matched budget with Ours (anchor, 8 heads):

ModelTaskLevelNaturalH2OOurs(anchor)
Llama-3.1-8BTREC-fine2→536.9337.4744.40
Llama-3.1-8BTREC-fine5→1049.3349.4750.67
Qwen2.5-7BTREC-fine2→533.6036.0041.07
Qwen2.5-7BTREC-fine5→1038.5349.7344.53
Qwen3-8BTREC-fine2→537.3344.2743.33
Qwen3-8BTREC-fine5→1033.6051.4737.07
Qwen3-4BTREC-fine2→533.6035.6040.80
Qwen3-4BTREC-fine5→1034.9337.4738.93

Ours (anchor) outperforms H2O in 5 out of 8 settings, and has a similar performance on Qwen3-8B/TREC-fine/2→5. This suggests that Ours (anchor) is a useful method to reduce the resident memory usage.

(2) Matched budget with Ours (direct, 30 heads):

ModelTaskLevelNaturalH2OH2O+Ours(30 heads)
Llama-3.1-8BTREC-fine2→536.9347.3350.13
Llama-3.1-8BTREC-fine5→1049.3355.2057.33
Qwen2.5-7BTREC-fine2→533.6044.5352.53
Qwen2.5-7BTREC-fine5→1038.5357.4764.93
Qwen3-8BTREC-fine2→537.3351.8752.00
Qwen3-8BTREC-fine5→1033.6060.6762.80
Qwen3-4BTREC-fine2→533.6036.9339.87
Qwen3-4BTREC-fine5→1034.9349.8753.60

H2O+Ours(30 heads) consistently deliver a better performance than H2O, especially over 5 percentage points on Qwen2.5-7B, indicating that our method can be plugged into existing dynamic KV cache methods to further improve them.

Head discovery

We want to clarify that the cost of head discovery involves running the validation set, currently just 144 data samples for TREC-fine, by six times, so it is not prohibitively expensive. The number reported on table is large in the prefix-cached setting mainly due to the low cost of prefix-cached inference.

How the contextual states are formed and why selective access can hurt final predictions

We agree that these questions are not fully answered, and thanks for the suggestion of layer trace. However, given the fact that MLPs are complex non-linear functions, that is the reason why this paper focuses on attention layers and leaves the behavivors of MLPs as an exploratory part. We believe that this requires a series of experiments to explore it, and we hope that we can answer it in the future work.

Reviewer_JUArOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how in-context learning works and identifies specific heads responsible for transferring information from demonstrations. They use retrieval of pre-encoded demonstrations at only these heads and show that this also causes ICL performance to improve with more demonstrations at these heads.

Critical strengths

The topic is timely.

Critical weaknesses

W1. In general I think it's challenging to identify what the main contribution of this paper is. Is it a better understanding of how ICL works (through these carrier heads)? A method that is comparable to short-context ICL latency with higher accuracy? A method that maintains short-context ICL accuracy but at lower latency? It's hard to effectively evaluate the contribution.

W2. I'm a bit concerned with the latency numbers (which also should appear in the main text, not an appendix, as they are part of one of the main claims of the abstract). The latencies in the prefix caching setting are often lower for longer inputs in the table, which suggests to me that either the measurement does not use proper warm-starts + clearing between benchmarked inferences or that the variance on these numbers may be quite high. The method does not appear to substantially reduce latency below the baseline.

W3. The paper is extremely hard to follow. It uses dense, non-standard jargon and in parts reads like an experiment log rather than a conference publication. I strongly suspect that the text was largely or wholly generated by an AI model. While I generally would not consider presentation issues a critical weakness of a paper, in this case I think the presentation significantly impacts the potential value of the work.

Questions for authors

Q1. Can you provide a comparison of your method, regular long-context ICL, and efficient long-context ICL methods (like Cartridges or DBSA) in terms of accuracy, memory cost, and latency, averaged over a set of runs?

Q2. Can you summarize the primary benefits of using selective TL memory over standard ICL?

Q3. What is the relationship between your mapping-sensitive carrier heads and the notion of induction heads?

Q4. Can you define "live retrieval"?

1 comment

We thank the reviewer and address each point below.

1. (W1) Main contribution.

Our main contribution is the mechanism analysis of many-shot task learning, and the method comes out of that analysis. Building on existing head-level work, especially TSLA, we extend the analysis to the many-shot regime and, following recent kernel-based accounts, add a TL-specific kernel readout to the TSLA framework. Its usefulness is supported by the query-weight controls and by the held-out method evaluation, both with 70–180 demonstrations. Selective TL Memory follows from this analysis and keeps cached demonstration keys and values for the selected readers while recomputing how each new query retrieves them.

2. (W2) Latency.

We agree that the latency results should be in the main text and will move them there. The difference between K=5 (35.0 ms) and K=10 (34.6 ms) in the prefix-cached setting of Table 6 is within measurement noise, because the online pass only processes the query suffix and at batch size 1 the fixed per-step overhead matters more than the cache length, so the two latencies are essentially the same. We do not aim to reduce latency below K=5, which is generally the lower bound for steering-vector methods as well. Compared with standard K=10, our method and the vector baselines keep K=5-level latency and throughput (SDPA, prefix-cached, B=32: about 220 q/s versus 140 q/s) with about half the resident memory (0.37 versus 0.74 GiB), and they still use the K=10 demonstrations.

3. (W3) Presentation. The paper reports many settings in support of its claims, and we regret that so many settings made the text hard to follow. We will revise the presentation to make it clearer, including a single summary of the named conditions.

4. (Q1) Comparison with efficient long-context methods at matched memory.

At matched resident memory we compared with H2O, a widely used dynamic KV-cache selection method [1]:

Matched budgetOursH2OH2O + Ours (30 heads)
Ours (anchor, 8 heads) [0.001 GiB]51.7349.4752.53
Ours (direct, 8 heads) [0.009 GiB]51.6052.9354.67

At the anchor budget the resident memory is so small that H2O cannot retain enough to help. At the direct budget the two perform similarly. Our method is not in competition with dynamic KV-cache selection: the two are complementary, and combining H2O with our 30-head memory at the same budget gives a further gain.

5. (Q2) Benefits over standard ICL.

We think there are two main advantages: (i) Less resident memory than caching the full demonstration set (Table 6). (ii) A controllable efficiency budget, through the number of selected heads and the positions retained (8 vs 30 heads; direct vs anchor).

6. (Q3) Relation to induction heads.

As observed in TSLA, induction heads are closer to the heads involved in task recognition than to task-learning heads, and the overlap between TL heads and induction heads is small. Consistent with this, the selected heads weight demonstrations of the query's class by identity, , whereas induction heads match repeated tokens.

7. (Q4) Definition of live retrieval.

Live retrieval means that the attention weights over the cached demonstration states are recomputed for every query from that query's own current residual state. They are not fixed or averaged in advance. The alternative, a query-independent summary of the demonstrations, is what task-vector and function-vector methods inject. The mean-weight control tests the difference: replacing each query's weights by the across-query mean drops the eight-head readout from 50.4% to 11.6% (Figure 3A).

[1] Zhang, Zhenyu, et al. "H2o: Heavy-hitter oracle for efficient generative inference of large language models." Advances in neural information processing systems 36 (2023): 34661-34710.

Reviewer_mvwgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how attention heads contribute to in-context task learning. The authors identify a small set of "carrier" heads by comparing their outputs under correct and randomized demonstration labels. They show that these heads can act as a kernel classifier, retrieving task-specific information from demonstrations.

Based on this observation, the authors propose Selective TL Memory, which allows only selected heads to access additional cached demonstrations. They evaluate the approach across several Llama and Qwen models, classification tasks, and demonstration budgets, comparing against full-context inference and task-vector baselines.

Critical strengths
  1. Interesting mechanism analysis. The proposed head-selection criterion is well motivated, and the retrieval and intervention experiments provide useful evidence that certain attention heads carry task-specific information. The analysis goes beyond simply identifying heads with high attention to demonstrations.

  2. Good experimental coverage. The authors evaluate multiple models and tasks, include several useful controls, and report negative results. In particular, the comparison with kNN in Table 18 helps establish that the selected heads implement a useful retrieval mechanism rather than simply performing nearest-neighbor classification over hidden representations.

Critical weaknesses
  1. The main baseline comparison is not convincing. The "Natural" baseline is not standard prompting with fewer demonstrations, but a masking intervention on the full prompt (Table 2). This makes the reported improvements difficult to interpret. For example, on Qwen3-8B/TREC, standard K=5 prompting achieves 57.60%, whereas the proposed method at K=5 to K=10 achieves only 42.27%. A comparison against standard K_base prompting is necessary to establish the practical benefit of the method. I also have concerns about the vector baselines. Their adaptation to a receiver that already sees base demonstrations is not sufficiently explained. The authors' own base-subtraction experiment improves TSLA substantially (Table 5), but this variant is not included in the main comparison.

  2. The role of the selected heads may be overstated. Although the selected heads provide useful readouts, the evidence that they are the main components responsible for task learning is less convincing. Ablating the eight carriers produces relatively modest accuracy drops (Table 14). More importantly, Table 20 shows that on Qwen3-8B, allowing other heads to access the additional demonstrations is substantially more beneficial than allowing only the selected carriers to do so. This suggests that much of the relevant computation happens elsewhere in the network. The value-repair experiments also intervene on all heads in the selected layers, making it difficult to attribute the observed effects specifically to the carriers.

Questions for authors
  1. Can the authors compare against standard K_base prompting at matched inference cost? In particular, why does the Natural baseline perform substantially worse than standard K=5 prompting on Qwen3-8B?

  2. Can the authors include results with base-subtracted TSLA vectors and clarify how FV, TV, ICV, and I2CL were adapted? This could materially change the conclusions from Table 3.

  3. Given the results in Table 20, how much of the task-learning computation can actually be attributed to the selected carriers? Is selecting these heads still the best approach when other-head access is more useful?

  4. Can the authors report seed-level variance or paired confidence intervals for the main accuracy comparisons? Some of the reported improvements are quite small.

3 comments

We thank the reviewer for the careful review and address each point below.

1. (W1, Q1) The Natural reference and standard K_base prompting.

We agree that a comparison with Natural does not by itself establish practical value, and that the gap between Natural and standard K_base prompting can be large on some models, notably the Qwen series. Our primary goal is the mechanism. The method is then presented as a trade-off, controlled by the head budget, between No access, which is usually close to Natural, and Full access, which is usually close to All heads (Table 2). Its performance is accordingly limited in two cases: (i) the selected head set is not effective enough, e.g., 5→10 with eight heads on Llama-2-7B+SelfExtend/TREC-fine; (ii) the gap between Natural and standard ICL is large.

To answer the matched-cost question directly, we replaced the receiver with the standard ICL K=5 prompt (query at its own positions) on Llama-3.1-8B/TREC-fine 5→10, keeping the cached increment states, so the online cost is that of K=5 prompting. Test accuracy (%), three seeds × 250 queries:

ReceiverBaseOurs (model) 8Ours (model) 30Ours (direct) 8Ours (direct) 30
Standard ICL K=550.9352.4053.0752.9350.27
Original (Natural)49.3350.0050.4051.6053.07

With the standard K=5 receiver the method still improves over the base prompt in 3 of the 4 variants. The cached increment states keep the positions of the full prefix, so the receiver attends to them with mismatched RoPE offsets. This appears to degrade the direct readout of the larger head set (50.27 with 30 heads vs 52.93 with 8), while the downstream computation corrects it (Ours (model), 30 heads: 53.07). We will add this comparison to the paper.

Why Natural falls well below standard K=5 on Qwen3-8B: Natural is not a standard way of using an LLM, and the size of its gap to standard prompting is a model-internal, model-dependent phenomenon; in our observations Qwen3 shows a much larger gap than Llama-3.1. This is a limitation of the current method.

2. (W1, Q2) Adaptation of the vector baselines.

We follow each method's own implementation with two changes. (i) The zero-shot receiver is replaced by a few-shot (base-access) receiver: Table 15 shows that at zero base the TL mechanism is not activated by steering vectors or by recompute-based interventions. (ii) Demonstrations are drawn only from the training split, and any hyperparameters are chosen on a separate validation set. We will add these details to Appendix A.2.

3. (W1, Q2) The TSLA variants.

We were surprised that the standard TSLA intervention (adding the full vector) performed much worse than the un-intervened receiver, most strikingly for its TL heads on TL tasks. This motivated us to test other forms of intervention; the increment vector (Eq. 7) recovers the loss, and Ours (model) is designed around this observation rather than around direct addition. The other vector baselines were not designed for TL, so their weaker results are expected; that they behave differently on TL than on the ICL tasks they were developed for is one reason TL needs its own mechanism analysis. The table below adds the increment-vector TSLA (TL and TR) to all 18 settings.

ModelTaskLevelNaturalTSLA (original) TLTSLA (original) TRTSLA (increment) TLTSLA (increment) TROurs (direct) 8Ours (direct) 30
Llama-3.1-8BTREC-fine2→536.9332.6712.4038.5336.6744.4047.47
Llama-3.1-8BTREC-fine5→1049.3339.2019.0749.4748.2751.6053.07
Llama-2-7B + SelfExtendTREC-fine2→514.9315.6013.8713.7313.8714.9321.07
Llama-2-7B + SelfExtendTREC-fine5→1034.1325.8728.6731.3332.8024.6731.87
Qwen3-14BTREC-fine2→540.2717.7310.5341.4740.0045.0749.07
Qwen2.5-7BTREC-fine2→533.6027.600.6729.0726.8041.3343.47
Qwen2.5-7BTREC-fine5→1038.5333.330.8035.7322.6745.8752.67
Qwen3-8BTREC-fine2→537.335.205.0731.4733.3340.8043.47
Qwen3-8BTREC-fine5→1033.605.604.4028.9330.0037.6042.27
Qwen3-4BTREC-fine2→533.604.5312.1332.2729.4741.7346.67
Qwen3-4BTREC-fine5→1034.9316.2723.8729.2034.4040.4041.33
Llama-3.1-8Bbanking771→254.4010.4016.4054.1355.0748.0061.87
Llama-3.1-8Bclinc1501→269.0735.0733.2069.8769.4758.9376.40
Llama-3.1-8Bdbpedia142→571.2063.8749.6079.8770.0083.2088.93
Llama-3.1-8Bdbpedia145→1090.9384.1371.2092.8089.2091.4793.47
Llama-3.1-8Byahoo2→538.2722.2716.1333.3335.3335.4739.60
Llama-3.1-8Byahoo5→1057.7331.8716.0058.0057.2054.6755.33
Llama-3.1-8Byelp2→521.3322.8024.8024.6722.8022.0024.93

Test accuracy (%), three seeds × 250 queries. Bold marks the highest score in each row among the adapted methods (Natural excluded).

The increment vector helps in some settings, e.g., Qwen3-14B/TREC-fine 2→5 and Llama-3.1-8B/yelp 2→5, and gives the top score in two rows (Llama-2 5→10 via TR, yahoo 5→10 via TL), though it is still lower than Natural on Llama-2 5→10 via TR. Overall, our thirty-head direct readout has the highest score in 16 of the 18 rows.

4. (W2, Q3) Importance of the selected heads.

Table 14 shows the pattern that self-repair studies predict: on Llama-3.1, ablating the eight heads alone or the carrier-layer MLPs alone has a limited effect, whereas ablating both together is dramatic (e.g., a 27.2-point drop against 3.6 for the two individual drops combined at seed 42), because MLPs compensate for a removed component. Prior work [1] therefore recommends joint ablations to expose the real effect, and single-component drops understate the heads' role. Since MLPs play several roles in the final prediction, we leave their analysis to future work and focus on attention heads here.

On Table 20: X is the complement of the selected set, i.e., 1,016 of 1,024 heads in Llama-3.1-8B and 1,144 of 1,152 in Qwen3-8B for the eight-head rows, over 30 times the size of the thirty-head set. Measured per head, the selected 8 or 30 heads are far more effective than the rest.

Is selecting these heads still the best approach? Granting access to more heads helps further, at the cost of memory and computation; the 8- versus 30-head comparison, and the 64/128-head budgets in Table 19, show this trade-off.

5. (Q4) Size of the improvements.

We agree that the yahoo 2→5 gain over Natural is not significant. The seed-level standard deviations on yahoo 2→5 are 3.35 for Natural (38.27) and 2.60 for Ours (direct, 30) (39.60), so the 1.33-point gap is well within the variation across the three seeds. By contrast, on yahoo 5→10, Natural 57.73 ± 1.29 vs Ours (model, 30) 61.33 ± 1.29: the 3.60-point gap is well beyond the seed-level variation.

[1] McGrath, Thomas, et al. "The hydra effect: Emergent self-repair in language model computations." arXiv preprint arXiv:2307.15771 (2023).

Comments on the paper

1 comment

Thanks for your kind reply. To prove our adavantage on Ours(anchor) and the combination of H2O and Ours method. We do the comparison with H2O with multiple LLMs.

(1) Matched budget with Ours (anchor, 8 heads):

ModelTaskLevelNaturalH2OOurs(anchor)
Llama-3.1-8BTREC-fine2→536.9337.4744.40
Llama-3.1-8BTREC-fine5→1049.3349.4750.67
Qwen2.5-7BTREC-fine2→533.6036.0041.07
Qwen2.5-7BTREC-fine5→1038.5349.7344.53
Qwen3-8BTREC-fine2→537.3344.2743.33
Qwen3-8BTREC-fine5→1033.6051.4737.07
Qwen3-4BTREC-fine2→533.6035.6040.80
Qwen3-4BTREC-fine5→1034.9337.4738.93

Ours (anchor) outperforms H2O in 5 out of 8 settings, and has a similar performance on Qwen3-8B/TREC-fine/2→5. This suggests that Ours (anchor) is a useful method to reduce the resident memory usage.

(2) Matched budget with Ours (direct, 30 heads):

ModelTaskLevelNaturalH2OH2O+Ours(30 heads)
Llama-3.1-8BTREC-fine2→536.9347.3350.13
Llama-3.1-8BTREC-fine5→1049.3355.2057.33
Qwen2.5-7BTREC-fine2→533.6044.5352.53
Qwen2.5-7BTREC-fine5→1038.5357.4764.93
Qwen3-8BTREC-fine2→537.3351.8752.00
Qwen3-8BTREC-fine5→1033.6060.6762.80
Qwen3-4BTREC-fine2→533.6036.9339.87
Qwen3-4BTREC-fine5→1034.9349.8753.60

H2O+Ours(30 heads) consistently deliver a better performance than H2O, especially over 5 percentage points on Qwen2.5-7B, indicating that our method can be plugged into existing dynamic KV cache methods to further improve them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.