acceptodds
Under review as a conference paper at ICLR 2027

Emergent Structure in the Attention Space of Language Models

Abstract

While representation similarity across independently trained language models is well-documented, how internal mechanics such as attention behave across models remains far less characterized. Inspired by this gap, we examine the structure of post-softmax attention weights by marginalizing over query positions, mapping them into a joint token-head "marginal attention space". Evaluating across 60+ diverse LLMs, we find that different properties emerge when reducing this space along its token and head axes. When reduced token-wise, marginal attention yields a text-intrinsic signal robustly conserved across models. To explain this property, we empirically connect marginal attention to the input-output Jacobian of the network, and prove theoretically that under a smoothness assumption, models with similar next-token distributions are guaranteed to have similar input-output Jacobian statistics. When reduced head-wise, it forms a model-private signature conserved across documents. Practically, this provides a natural way to estimate a per-head budget for key-value (KV) cache eviction, effectively decoupling model-specific budget allocation from text-intrinsic token scoring. On standard eviction benchmarks, a per-head budget precomputed offline on pretraining text, combined with a training-free token score, shows competitive performance with methods that recompute the budget on every document or train it per target.

Primary area:
interpretability and explainable AI
Keywords:
attention, attention space, emergent properties, identifiability, KV cache eviction, LLM, transformer

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Bw2cOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

As i understand it, the fundamental question being looked at in this paper is: when different neural networks lean to solve the same problem, how much of their internal weights is determined by the problem rather than architecture/initialisation etc. I think it has been reasonably well shown that attention contains common structure across LLMs, and this papers focus is on a simple aggregate statistic. What the authors introduce is a notion of functional identifiability via a jacobian to see if any two independent LLMs approximate the same function, then the jacobians of their next token logits could be similar. I do not know this field well, but this seems novel and, to be, interesting and clever.

Critical strengths

This paper is really excellently presented and clear. The per marginal attention is sensible, and i like how simple it is, and the definitions are clear. I found figure 2 and 3 really useful as a visualisation. The experimental setting is on an extensive number of open models of varying size. As i said above I like the jacobian idea, and the fact their simpler aggregate is an approximation of this is a very useful result.

Critical weaknesses

Honestly, It would be nit picking to for me to write anything here. I enjoyed reading this paper, and I could not find anything obviously wrong as a "critical weakness".

Questions for authors

I suppose the one thing is the correlation of 0.6 between the two is decent but to me still a little weak. However, this is all relative. So i guess this correlation established agreement in token ranks, but it does not show fully that marginal attention approximates the jacobian influence numerically, Given how much the authors have done I do not expect them to fix this, just wondering what are their thoughts on this. I dont know if its possible to determine whether the component of attention that agrees across models is specifically the component predicted by Jacobian influence.

2 comments

We want to thank the reviewer for appreciating our work and strongly supporting its acceptance.

On whether the shared component of attention is the one predicted by the Jacobian

We also find this question interesting. Even though our main experiments do not isolate this component directly, we believe one result supports the “shared” interpretation strongly: the Jacobian-based influence of Mamba and RWKV models (models which have no attention mechanism) agrees with the attention consensus of the census about as well as that of a transformer (noted in Section 4.1). Our reading is that the shared structure we observe does not strictly depend on attention being the underlying mechanism, which is what our Jacobian account predicts.


We hope our replies clarify the points raised, and we are happy to discuss them further.

Thanks! happy to maintain my positive assessment! great work

Reviewer_kjSKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose marginal attention as a relevant quantity to study in LLMs. The per-token marginal attention is a model-agnostic property of the text, and the per-head marginal attention is a document-agnostic property of the LLM. They then connect attention statistics in LLMs to the input-output Jacobian and show that model smoothness and next-token distribution similarity alone are theoretically sufficient for input-output Jacobian near-identifiability across models. They finally show that cache optimization based on per-head marginal attention is effective on multiple LLM families based on RULER and LongBench evaluation.

Critical strengths
  1. The observations on per-token and per-head marginal attention are interesting, and the authors analyze them with substantial mathematical analysis.
  2. The authors evaluate the method across many model families and series, including Qwen, Llama, Phi, and Mistral, and compare it with multiple baselines such as AdaKV and KVzip.
Critical weaknesses
  1. The phenomenon is interesting and supported by extensive analysis, but the models used to evaluate the cache optimization method are outdated. Most were released more than a year ago, which weakens the timeliness of the paper.
  2. The method comparison is confusing. EA, KVzap, and Guide appear to be different scoring strategies, while AdaKV, LU-KV, and HM are different head-wise budget strategies. This design admits many combinations, so it is unclear why the table includes only the reported ones, and why KVzip appears as an inconsistent entry. On Qwen3-32B and Mistral-7B, the proposed method's advantage over KVzip is also not significant. Average scores would make the comparison clearer. Finally, although the method is presented as a cache optimization method, the paper does not compare prefill, decoding, or end-to-end efficiency.
Questions for authors
  1. Beyond cache optimization, have the authors considered applying the emergent structure to other settings, such as monitoring LLM pretraining or mechanistic interpretability? Moving more of the analyses currently in the appendix into the main text would also improve the presentation.
  2. Section 3 contains only Subsection 3.1. I recommend organizing this section more clearly.
5 comments

We thank the reviewer for their feedback and address their points below.

W1. Models for the KV cache eviction application were released more than 1 year ago.

Following the reviewer’s advice, we ran our eviction experiments in the setting compatible with Table 1 on two more recent target models. They are IBM Granite 4.2 8B (August 2026) and Ministral-3-14B-Instruct (December 2025).

rowGranite 4.2 8BMinistral-3-14Bmean ± sd
uncompressed92.794.993.8
KVzip94.5 / 93.5 / 87.994.6 / 92.7 / 79.690.5 ± 5.9
AdaKV + EA73.5 / 27.3 / 6.380.6 / 53.1 / 28.244.8 ± 29.1
LU-KV + EA92.6 / 90.7 / 75.794.2 / 89.4 / 58.583.5 ± 13.9
HM + EA93.6 / 92.2 / 83.394.2 / 88.1 / 66.086.2 ± 10.7

The results show that our head allocation is still relevant for newer models, since they follow the same trend of the others in Table 1. More generally, we built the cache eviction evaluation on the kvpress library for reproducibility and ease of comparison across other published methods. However, this choice limits the target models for eviction to the ones supported by the library. We now explicitly state this in the revised Section 5.2.

If the reviewer has a specific recent model in mind, we are happy to add it in the remaining time, provided it uses standard attention layers so that kvpress can run it.

W2. Table 1 formatting

We thank the reviewer for the feedback on the main evaluation table. We now show the budget and the scorer of each row in separate columns and also a final average column with standard deviation to improve readability.

On the combinations of budget and scorer: while we agree that a full grid of budgets and scorers would be informative, given computational constraints we chose combinations that change one component at a time to isolate its contribution from a shared baseline. AdaKV + EA, LU-KV + EA and HM + EA share the scorer and differ in the budget. AdaKV + EA and AdaKV + KVzap share the budget and differ in the scorer.

On KVzip: it sets the budget and the score together from a second pass over each document, so it does not belong to the new two column format and we now show it as an unranked reference row at the top of the table. The new KVzip + EA row separates the two parts: it keeps KVzip's budget and scores with EA. Section 5.2 now states that it stays ahead of HM + EA on Qwen3-32B and Mistral-7B, at the cost of that second pass.

If the reviewer finds a specific further combination informative, we are happy to run it in the remaining time.

Q1 + Q2. Possible new applications

We moved one experiment, previously in the appendix due to the length limit, into Section 3.2 in the main text. It analyzes the convergence of marginal attention during training, and finds that marginal attention converges gradually, and is very closely associated with the drop in language modelling loss. We have noted in the conclusion that this may suggest the quantity has utility in monitoring during training, although we leave details for future work.

If the reviewer has another specific application in mind, we would be glad to hear it and to discuss in the remeaining time whether/how marginal attention applies there.


We hope these replies and the revised manuscript address the reviewer's concerns, and we are happy to discuss them further.

Thank you for your response. I have noticed the revisions to Table 1 and the additional experimental results on Granite 4.2 8B and Ministral-3-14B. However, I apologize for not expressing my concern clearly enough. My main concern is that, given the shift in LLM architectures from softmax-attention-only models toward hybrid models over the past year, softmax-attention-only models may become much less common in the future. Do the observations and optimizations proposed for softmax attention still hold for the standard attention layers in current hybrid models?

If the authors can provide experimental results on models such as Qwen3.5, I will raise my ranting to 3. Without results on such models, however, the practical meaning of this work would be limited.

We thank the reviewer for the clarification and their prompt response. To validate the observations of Section 3 on marginal attention for the requested architecture, we studied five recent hybrid models, including the requested Qwen 3.5.

modelsoftmax-attention layerstoken-wise agreement with the 67-model consensushead-wise agreement between two halves of the documents
Qwen3.5-9B8 of 320.810.99
Qwen3.8-27B16 of 640.800.99
LFM2.5-8B6 of 240.680.98
LFM2.5-24B10 of 400.820.98
Olmo-Hybrid-7B8 of 320.750.99

The results show that both observations hold on all five hybrid models. Their token-wise agreement is within the range of the census models (0.63 to 0.94), and their head-wise profile is stable across documents.

Regarding the application, your response raises a broader question: how relevant is cache eviction for models that keep a KV cache in only a quarter of their layers and were designed in part to reduce it [a, b, c]? We consider this to be out of scope: despite the increase in popularity of hybrid models, the vast majority of modern language models under study contain some measure of softmax attention models, for which KV cache eviction remains a very active research area [d].

However, for completeness and transparency, we have extended kvpress (which is the de facto standard eviction library and does not support hybrid models out of the box). We ran the setting of Table 1 on Qwen3.5-9B and Qwen3.8-27B, with the compression level computed on the attention layers.

Qwen3.5-9B (uncompressed 96.7)4×8×16×
AdaKV + EA68.150.926.9
KVzip86.148.020.0
KVzip (layer-wise)83.534.121.1
HM + EA (layer-wise)72.750.132.5
Qwen3.8-27B (uncompressed 97.2)4×8×16×
AdaKV + EA81.349.325.9
KVzip93.260.025.1
HM + EA91.572.747.1

HM + EA is never below the AdaKV + EA baseline on either model and is ahead of KVzip at strong compression levels. On the smaller Qwen3.5-9B we apply the Head Mass budget layer by layer, since a single cut across all layers leaves some attention layers almost empty.

Though we share the reviewer’s interest in hybrid architectures, we would like to stress again that the aim of our work is to study marginal attention, with KV cache eviction as an application to an active area of research. Where a KV cache is evicted, our results show that marginal attention offers a good signal to guide the process.


[a] Lieber, Lenz, Bata, et al.; Jamba: A Hybrid Transformer-Mamba Language Model; arXiv, 2024.

[b] Glorioso, Anthony, Tokpanov, Golubeva, Shyam, Whittington, Pilault, Millidge; The Zamba2 Suite: Technical Report; arXiv, 2024.

[c] Liquid AI; LFM2 Technical Report; arXiv, 2025.

[d] Xie, Yang, Hu, Zhang, Wu, Hong, Xu; HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models; arXiv, 2026.

Reviewer_Scs6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies value-weighted attention after marginalizing over query positions. Across 67 language models, token-level profiles agree across models, while head-level profiles remain stable across documents. It connects these observations to a conditional Jacobian-stability result and proposes Head Mass (HM), an offline per-head KV-budget estimator evaluated with Expected Attention and a small-model guide.

Critical strengths
  1. The 67-model census is substantial, with character-level alignment and useful controls for position, punctuation, random seeds, and different-document comparisons.
  2. With the EA scorer held fixed, HM consistently improves over AdaKV across the reported RULER settings. Comparisons with LU-KV also address a relevant static-budget baseline.
  3. Table 12 provides an informative negative result: a stable reading-pass budget can perform poorly, showing that stability alone does not imply usefulness for eviction.
Critical weaknesses
  1. Novelty needs clearer separation. Liang et al. (2025) already quantify cross-model attention similarity; KVzip and LU-KV already support offline head allocation. Value-norm weighting also has a precedent in Kobayashi et al. (2020). The distinctive contributions are the controlled census and HM’s averaging of reconstruction-induced keep rates, whose advantages over these precedents need sharper isolation.
  2. The theoretical explanation is not yet validated. Theorem 1 requires uniform logit agreement throughout a shared embedding neighbourhood, whereas experiments measure the gap only at natural inputs. The 16-model study also substitutes a log-probability-gradient proxy for Jacobian block norms; direct within-model attention/block-norm validation covers only Qwen2.5-0.5B.
  3. The head-level interpretation needs stronger controls. Without value weighting, p_h = 1/H identically. Appendix C’s layer-level norm approximation and cross-model depth binning leave unresolved whether the signature captures head-specific routing or mainly depth-dependent magnitude.
  4. The guide’s transfer benefit is inconsistent. It requires a separate fitted map for each target and reduces LongBench scores for all four targets at 16× compression (Table 14), including 40.5 to 37.1 on Llama-3.1-8B. Although presented as a probe, these results warrant narrower claims about a transferable token-scoring signal.
Questions for authors
  1. What does HM add over KVzip’s existing static head-scoring mode? Can document-specific KVzip budgets and their offline average be compared using the same EA scorer?
  2. Can the theoretical conditions be tested under embedding perturbations in a specified common input space, using actual Jacobian block norms across multiple models?
  3. Does head-profile stability survive exact per-head weighting and normalization within each layer?
  4. Why does the guide degrade every 16× LongBench setting, and can a shared token score work without target-specific fitting?
2 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank the reviewer for their careful review and attention to detail. We respond point-by-point below.

W1. Novelty clarifications

We agree with the reviewer, and have sharpened the related work section. Specifically, Liang et al. (2025) note that raw attention statistics can visually tell documents apart across models. We expand this by carefully analyzing this agreement across models quantitatively in a large census of modern LLMs (per-token marginal attention), along with which part stays specific to each model (per-head marginal attention), and why the agreement arises at all (by analyzing the Jacobian).

Section 3 now attributes value norm-reweighting to Kobayashi et al. (2020). Table 1 compares Head Mass with LU-KV, and Q1 below compares it with KVzip.

W2 + Q2. Theoretical conditions and validation

We have completed the requested validation experiments. We have established that agreement is not substantially deflated in the neighbourhood of real inputs (via perturbation; Appendix E.2). Additionally, we have replaced the log-probability-gradient proxy with a better estimator based on Hutchinson’s stochastic trace trick (Section 4.1 and Figure 6; implementation details in Appendix E.1). Finally, we have expanded the validation from Qwen2.5-0.5B to 16 models (Section 4.1 and Figure 6).

W3 + Q3 Isolate head-specific vs depth-dependent routing

We agree with the reviewer that this control is needed given our implementation. We now normalise the per-head marginal attention per-layer, and compare the profiles head-by-head. This removes the term quantifying how much each layer writes and keeps only how its magnitude is divided among the heads. The following table gives the Spearman correlation between profiles on 170 documents, for models which share the same number of layers and heads.

comparisonnwithin-layershuffleddepth only
document, split halves67 models0.970.020.99
leave-one-document-out67 models0.860.000.98
base vs derived model16 model pairs0.980.001.00
shared ancestor5 model pairs0.950.020.99
same architecture, trained separately9 model pairs0.000.000.62

The results show that the profile is indeed head-specific. It stays stable after the normalisation (within-layer), and not when the heads are permuted inside each layer (shuffled). Depth also carries a stable signal, as the reviewer suggested (depth only, each head replaced by the mean of its layer), but our claim on the head profile holds independently of it (within-layer).

We plan to update the right part of Figure 3 (the head-level comparison) with this more restrained analysis, as it is indeed much more robust, and rules out the “purely layerwise” hypothesis. We will move the relative-depth binning to the Appendix.

W4 + Q4. The role of the guide

We used the extra page of the revision to qualify HM + Guide more sharply as a probe. The Introduction, Section 5 and the Conclusions now state only that a smaller model can carry part of the eviction signal, and Table 1 shows HM + Guide in the unranked block of expensive reference methods. We also state where the guide reaches its limits, which is on LongBench at 16× and mostly on passage retrieval (Table 15).

On a shared token score with no target-specific fitting: it is a compelling idea, and indeed it was the first application we tried. We used the shared per-token profile as the score of every head, and it scored below Expected Attention alone. Models agree on attention summed over all heads, while eviction decides what each single head keeps, and a single head follows the shared profile only loosely. On six census models, a single head agrees with the consensus of the other models at a median Spearman of 0.43, where the sum over the heads of the same model reaches 0.77 to 0.93. This is why the guide necessarily passes through a per-target map.

AuthorsOpenReview ↗

Rebuttal 2/2

Q1. What Head Mass adds over KVzip's static head mode.

Following the reviewer's question, we added KVzip's static mode (KVzip-H) to Table 1 on all five models, in two forms. KVzip-H is the mode as released, which keeps the best heads whole and empties the rest. KVzip-H + EA uses its head scores as a per-head budget and scores with EA, so it differs from HM + EA only in where the budget comes from. We compute its head scores in float32, since the released 16-bit scores tie at the top. The table gives the RULER score at 4× / 8× / 16×.

rowQwen3-8BLlama-3.1-8Bphi-4Qwen3-32BMistral-7Bmean ± sd
KVzip-H79.7 / 23.8 / 5.380.8 / 12.0 / 5.183.1 / 53.9 / 32.990.2 / 54.0 / 9.242.8 / 12.6 / 3.839.3 ± 32.2
KVzip-H + EA77.3 / 64.2 / 55.682.5 / 63.2 / 41.780.8 / 64.9 / 48.168.7 / 52.5 / 36.762.0 / 39.7 / 17.157.0 ± 18.1
HM + EA95.0 / 91.8 / 73.395.1 / 92.3 / 66.391.5 / 87.8 / 68.595.2 / 93.2 / 79.190.5 / 84.3 / 65.384.6 ± 11.1

The results show that the Head Mass budget is the stronger one in every setting. KVzip-H + EA and HM + EA share the scorer, so the gap comes from the budget.

On document-specific budgets with the same EA scorer: we agree with the reviewer that this comparison is informative, and we added it to Table 1 as KVzip + EA*. In Table 1, HM + EA and KVzip differ in both the budget and the scorer. The new row has KVzip's per-document budget and the EA scorer, so against HM + EA it changes only the budget, and against KVzip it changes only the scorer. It needs KVzip's pass on every document, so it is no longer an offline method and we show it as an unranked reference row.

rowQwen3-8BLlama-3.1-8Bphi-4Qwen3-32BMistral-7B
KVzip95.0 / 92.3 / 59.495.3 / 92.8 / 74.291.5 / 89.6 / 68.895.6 / 95.6 / 91.891.2 / 89.2 / 72.2
KVzip + EA95.1 / 92.9 / 78.795.2 / 92.3 / 71.191.1 / 87.9 / 71.795.5 / 94.8 / 86.090.5 / 84.9 / 63.3
HM + EA95.0 / 91.8 / 73.395.1 / 92.3 / 66.391.5 / 87.8 / 68.595.2 / 93.2 / 79.190.5 / 84.3 / 65.3

The results show that the offline Head Mass budget recovers most of what an expensive document-specific budget gives, in line with the agreement of single-document budgets with their average (Table 9). The two differ only at the highest compression, where the budgets of single documents depart most from the Head Mass table and the expensive document-specific budget is ahead on average. With the budget fixed, neither scorer is consistently ahead of the other.


We hope that our replies satisfy all the main concerns raised by the reviewer. If it is not the case, we will be happy to further discuss in the remaining time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision summary

We thank all reviewers for their careful reading and their constructive feedback. The revised manuscript makes the changes below and highlights them in red:

  1. Improved readability of the eviction methods comparison, and added KVzip's static mode. Table 1 now shows budget and scorer of each row in separate columns, and a last column gives each row's mean ± standard deviation. KVzip is highlighted as reference row and is not ranked. Two new KVzip-H rows give KVzip's context-independent head mode on all 5 target models. A new KVzip + EA reference row gives KVzip's per-document budget with the EA scorer. Tables 14 and 15 follow the same layout (Scs6, kjSK).
  2. Enriched Section 3. Now it holds the analysis of emergence during pretraining, moved unchanged from Appendix E.3 to improve readability and to highlight some insights on pretraining (kjSK).
  3. Expanded the discussion of related work. Related work now states what Liang et al. (2025) leaves open, and how we address it. Section 3 cites Kobayashi et al. (2020) where the value-norm weight is defined (Scs6).
  4. Clarified the scope of the evaluation and the trade-off with KVzip. Section 5 bounds the targets to models supported by kvpress and states the trade-off with KVzip. It now also points to the budget stability table (Table 9) and it introduces HM + Guide as a probe, reporting that the guide lowers HM + EA on LongBench at 16× (Scs6, kjSK).
  5. Narrowed the claims on the guide and reinforced its cost presentation. The Introduction, the takeaway of Section 5 and the Conclusions now state the role of the smaller model more narrowly. The cost table (Table 11) and the Limitations say that the guide reads each document twice (Scs6).
  6. Improved theoretical validation. We now use a more robust Hutchinson’s estimate for the Jacobian norm measurements, and assess the logit agreement within a neighbourhood and across 16 models. (Scs6).

The replies to Scs6 and kjSK also report three results that are currently not in the revision. They are a depth control for the per-head profile, the agreement of single heads with the shared profile, and eviction results on two more recent models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.