acceptodds
Under review as a conference paper at ICLR 2027

Binding Breaks First: Diagnosing Contextual Hallucinations under KV-Cache Compression

Abstract

Key–value (KV) cache compression reduces the memory cost of long-context inference, but can also degrade context-dependent reasoning and increase hallucinations that contradict the provided context. Existing evaluations mainly measure how much performance drops, leaving unclear what information is actually lost or corrupted by compression. We propose BindProbe, a diagnostic framework that separates two possible failure modes: item-access failure, where an entity or attribute value can no longer be recovered, and binding failure, where both remain accessible but the model can no longer determine which value belongs to which entity. BindProbe places controlled entity–value associations at different positions in long contexts and evaluates them with separate entity, value, and binding probes. Across several evaluated compression settings, we observe regimes in which relational binding degrades substantially while access to the individual items remains comparatively stable. This produces item-preserving misbinding: the model retains the relevant entities and values but associates them incorrectly. The effect is strongly position dependent, with associations in the middle of the context becoming vulnerable at lower compression rates. Mechanistic analyses further show that KV states important for recovering these associations are systematically assigned lower importance by compression criteria, making them more likely to be removed or distorted as the cache budget shrinks. Our results show that compression-induced contextual hallucinations can arise not only because information is forgotten, but also because the relationships among still-accessible information are disrupted.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
KV-cache compression, hallucination, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jBhbOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BINDPROBE, a diagnostic framework designed to distinguish two information-level failure modes under KV-cache compression: item-access failure (relevant entities or attribute values become inaccessible) and binding failure (constituent items remain accessible but their association is no longer correctly recovered). By constructing controlled contexts with unique entity-attribute-value triples and evaluating paired item-access and binding probes, the authors reveal an "item-preserving misbinding" phenomenon: binding performance degrades substantially faster than item accessibility, with middle-context bindings being particularly vulnerable. Mechanistic analyses on Qwen3-8B and KVZip further show that compression importance scores (like KVZip) systematically underestimate KV states that are causally important for relation recovery (e.g., structural positions like intro[5], binding[0,2]), leading to their premature eviction.

Critical strengths
  1. Conceptual Novelty: Decomposing compression-induced hallucinations into item-access failure and binding failure is a critical and previously underexplored distinction. This reframes how we evaluate KV cache compression: it's not just about "forgetting" facts, but about "corrupting" the relationships between them.
  2. Mechanistic Insight: The discovery that current compression scores (e.g., KVZip) systematically undervalue structural/bridging tokens (intro[5], binding[0,2]) provides a concrete, mechanistic explanation for the "binding breaks first" phenomenon. This is highly actionable for the community to design better importance metrics.
Critical weaknesses
  1. Limited Generalizability (Synthetic Contexts): The BINDPROBE framework, as implemented, can only evaluate structured factual triples. It remains unclear whether these findings transfer to natural language contexts with complex coreference, multi-hop reasoning, or noisy RAG scenarios. The paper claims this is a "diagnostic framework," but its current scope is quite narrow.
  2. Insufficient Model/Method Coverage for Mechanistic Claims: The mechanistic case study is limited to Qwen3-8B with KVZip. Since the behavioral results already show that Llama-3.1-8B behaves differently, the mechanistic conclusions (e.g., underestimation of structural states) may not be universal. More diverse model architectures and compression methods (e.g., SnapKV, quantization) are needed to solidify the claims.
Questions for authors
  1. Could the BINDPROBE methodology be extended to natural language contexts? For example, how would you handle coreference resolution or multi-hop reasoning where the "binding" spans multiple sentences or is implicit?

  2. You observe that KVZip underestimates structural positions. Have you experimented with a simple fix, such as adding a positional prior or a "binding-score" bonus to the compression metric? If so, does it mitigate the binding failure?

  3. Since Llama-3.1-8B showed a less distinct ordering of degradation, do you have a hypothesis about what architectural or training differences cause this variation?

3 comments

We sincerely thank the reviewer for the constructive feedback, which has helped improve the quality and clarity of our manuscript. We will upload the revised manuscript shortly.

For W1 and Q1:

Natural language contexts.

We tested alternative context formats (Markdown tables, possessive sentences, value-first sentences, and reference paragraphs), question templates (direct, lookup-style, and formal), and paraphrases. All settings use a 1:1 ratio of task-relevant content to irrelevant padding.

Most settings retain clear binding–access separation under at least one probe format (Yes/No or recall), although the magnitude varies.

Context formatRateEntityValueForward bindingBackward binding
Markdown table0.004.92e-10 [1.66e-10, 9.74e-10]0.013 [0.012, 0.014]3.03e-09 [1.02e-09, 6.02e-09]0.003 [0.001, 0.006]
Markdown table0.922.54e-08 [5.61e-09, 4.69e-08]2.96e-06 [6.84e-07, 5.31e-06]0.255 [0.235, 0.273]0.538 [0.485, 0.581]
Possessive sentences0.002.82e-16 [2.34e-17, 5.47e-16]0.006 [0.006, 0.006]0.002 [4.05e-08, 0.004]0.009 [0.004, 0.014]
Possessive sentences0.921.07e-08 [3.12e-09, 1.94e-08]5.88e-05 [3.08e-06, 1.15e-04]0.149 [0.117, 0.185]0.710 [0.631, 0.780]
Reference paragraph0.005.67e-13 [1.99e-13, 1.09e-12]0.003 [0.001, 0.005]0.001 [1.93e-04, 0.001]0.007 [0.005, 0.009]
Reference paragraph0.922.18e-08 [7.37e-09, 4.07e-08]2.02e-05 [9.68e-06, 3.01e-05]0.262 [0.244, 0.279]0.727 [0.657, 0.789]

Qwen3-8B with KVZip, 30 entities × 10 attributes, three contexts and three paraphrase repeats. All three formats use direct questions. Cells report mean loss [95% CI]. Entity/Value use positive cases; binding probes are pooled. Full results will be included in the Appendix.

Complex scenarios.

The framework can extend to implicit and multi-hop associations. For example:

Implicit binding:

Alice owns a red car.
Bob owns a blue car.
Charlie owns a green car.

The person who owns the red car lives in Paris.
The person who owns the blue car lives in London.
The person who owns the green car lives in Tokyo.

Probe: Where does Alice live?
Answer: Paris

Multi-hop binding:

Record 001: name Solomon; food pie; animal elephant; spouse Ben; contact Piet;
Record 011: name Piet; spouse Megan; contact Brenda; animal elk; food sandwich;
Record 012: name Brenda; food cookie; contact Carlton; spouse Clyde; animal panda;

Probe: What is the animal of the contact of the contact of Solomon?
Answer: panda

We this 3-hop setting on Qwen3-8B with KVZip. However, binding loss was already near its peak at 2-hop inference without compression, limiting conclusions about compression effects. Full results will be included in the Appendix.

For W2: We extended the mechanistic analysis to Llama-3.1-8B and Expected Attention. In Llama, removing binding-sensitive heads increases forward/backward losses from 0.0035/0.0091 to 0.1670/0.7588 (layer-matched random: 0.0505/0.0382), while recognition remains comparatively stable. Removing the post-name separator increases forward loss by 0.1116, while removing value positions increases forward/backward losses by 0.7740/0.4180.

At 95% KVZip compression, structural versus name/value retention rates are 12.08%/66.62% in Qwen and 0.63%/45.86% in Llama, respectively. Expected Attention shows a similar pattern: at 95% compression, structural versus name/value retention is 0.89% vs. 22.37% in Qwen and 0.74% vs. 26.46% in Llama. This suggests that our explanation may partially generalize beyond KVZip.

To Q2: We tested restoring separator and attribute KV states on selected binding heads against matched-capacity random-head restoration. The targeted intervention improves binding loss at both compression settings:

Probe0.97 Random0.97 Target0.98 Random0.98 Target
Forward0.729 [0.602, 0.897]0.574 [0.516, 0.678]1.338 [1.255, 1.422]0.915 [0.884, 0.953]
Backward0.893 [0.720, 1.070]0.804 [0.641, 0.946]1.539 [1.383, 1.718]1.296 [1.148, 1.420]
Verification0.393 [0.244, 0.587]0.310 [0.282, 0.335]0.479 [0.340, 0.638]0.380 [0.349, 0.400]
Entity0.056 [0.032, 0.091]0.076 [0.047, 0.121]0.027 [0.019, 0.040]0.029 [0.022, 0.040]
Value0.151 [0.099, 0.223]0.193 [0.168, 0.218]0.188 [0.141, 0.246]0.196 [0.163, 0.236]

Mean loss [95% CI], three contexts. Target restores separator/attribute states on 110 selected KV heads; Random uses three matched-capacity head draws. Nominal rates are before restoration; both arms have equal final KV-entry counts, higher than the original compressed budget. Binary probes pool both labels.

Final bit compression after addback: 0.876952–0.877471 (nominal 0.97) and 0.883984–0.884531 (nominal 0.98).

To Q3: We hypothesize that architectural differences, particularly QK-Norm (used in Qwen3 but not Llama-3.1) and layer depth (36 vs. 32), may affect attention distributions and the retention of binding-critical KV states. Differences in training data and post-training objectives may also contribute to the variation. However, their specific effects remain to be verified.

Dear Reviewer jBhb,

As the discussion period is nearing its end, we wanted to follow up on our response. We would greatly appreciate knowing whether these address your concerns, and we are glad to clarify anything further.

Thank you for your detailed and careful review.

Reviewer_Why3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper presents a study of the influence of KV cache compression on item binding, that is, entity-attribute pair retrieval from long context. Authors compare nine different compression techniques including scored based approaches like KNorm and predictive ones like KVZip. The authors generate records by sampling entities and attributes and create unique sets of pairs to populate the context. They compare Qwen3-8B and Llama3.1-8B. They identify a particular failure mode of context degradation when the entities and attributes are still present in the context, but binding still fails, item-preserving misbinding. Overall this is an interesting study that identifies a gap in our understanding of binding mechanisms in LLMs. Whereas the results of attention head specialization will be familiar and are well documented already, the focus on item preserving errors is novel.

Critical strengths
  • Broad comparison of main KV compression techniques and shows the failure mode is present across compression techniques, which strengthens the findings.
  • Clearly identified gap in the literature supported by a review of the main methods.
  • Novel documented failure mode supported by empirical results.
Critical weaknesses
  • Missing controls specifically related to textual structure including natural language paraphrasing weakens the strength of the conclusions. Further, no varíance or confidence intervals further limits interpretability of the results.
  • The results are mainly correlational. There is a missed opportunity to causally examine the role of the intervening attention heads by using patching methods and ambiguous binding conditions, e.g., to explore whether the binding failure is positional, lexical, or reflexive (Gur-Arieh et al 2026)
  • Most important are my concerns about the construct validity of the experiment, in particular, the generated data. Formally, binding is a semantic phenomenon with evident lexical and syntactic properties. As Gur-Arieh, Geva, and Geiger (2026) demonstrate, this appears to be the case for LLMs too. As the intervention condition requires the LLM to retrieve entity-attribute pairs from records, and not from text, I’m not sure that the experiment can causally attribute the failure to context corruption alone. As it appears to me that the item binding failures could also be imputable to the lack of textual structure.

Minor nitpick: In Figure 3 the graph is misleading because the ticks are not evenly spaced.

Questions for authors
  • The structured data does not have natural language syntax, how does the data format affect the results? I would expect binding qua semantics to be highly influenced by syntactic structure.

  • What do the head ablation results add beyond contextual retrieval head specialization (Wu et al., 2024)?

  • The experimental condition is based on tabular data, and better candidate models exist for exactly these structured data formats. How would tabular foundation models fare on this task?

4 comments

We sincerely thank the reviewer for the constructive feedback, which has helped improve the quality and clarity of our manuscript. We will upload the revised manuscript shortly.

For W1, W3 and Q1: We tested alternative context formats, question formats, and paraphrases. Most settings show clear binding–access separation under at least one probe format (Yes/No or recall), although its magnitude varies. Context and question formats have a greater impact than paraphrasing.

Context formatRateEntityValueForward bindingBackward binding
Markdown table0.004.92e-10 [1.66e-10, 9.74e-10]0.013 [0.012, 0.014]3.03e-09 [1.02e-09, 6.02e-09]0.003 [0.001, 0.006]
Markdown table0.922.54e-08 [5.61e-09, 4.69e-08]2.96e-06 [6.84e-07, 5.31e-06]0.255 [0.235, 0.273]0.538 [0.485, 0.581]
Possessive sentences0.002.82e-16 [2.34e-17, 5.47e-16]0.006 [0.006, 0.006]0.002 [4.05e-08, 0.004]0.009 [0.004, 0.014]
Possessive sentences0.921.07e-08 [3.12e-09, 1.94e-08]5.88e-05 [3.08e-06, 1.15e-04]0.149 [0.117, 0.185]0.710 [0.631, 0.780]
Reference paragraph0.005.67e-13 [1.99e-13, 1.09e-12]0.003 [0.001, 0.005]0.001 [1.93e-04, 0.001]0.007 [0.005, 0.009]
Reference paragraph0.922.18e-08 [7.37e-09, 4.07e-08]2.02e-05 [9.68e-06, 3.01e-05]0.262 [0.244, 0.279]0.727 [0.657, 0.789]

Qwen3-8B with KVZip, 30 entities × 10 attributes, three contexts and three paraphrase repeats. All three formats use direct questions. Cells report mean loss [95% CI]. Entity/Value use positive cases; binding probes are pooled. Full results will be included in the Appendix.

For W2: We added head-ablation controls and targeted KV retention. At 10% pruning, binding-head removal increases forward/backward loss by 1.041/1.387, versus 0.230/0.129 for random heads. Matched-capacity retention of separator/attribute states reduces forward-binding loss compared with random-head retention at both tested rates.

Head-ablation controls.

StrategyHead fractionΔ ForwardΔ BackwardΔ Value
Access0.10+0.193 [+0.160, +0.223]+0.455 [+0.419, +0.499]+0.131 [+0.119, +0.141]
Binding0.10+1.041 [+0.963, +1.117]+1.387 [+1.361, +1.415]+0.021 [+0.011, +0.033]
Random0.10+0.230 [+0.033, +0.406]+0.129 [+0.020, +0.298]+0.276 [+0.010, +0.764]

Paired mean loss increases [95% CI], three contexts; Random averages three head draws per context. Nominal 10% removes 28/288 KV heads, selected after GQA score pooling.

Targeted KV retention.

Probe0.97 Random0.97 Target0.98 Random0.98 Target
Forward0.729 [0.602, 0.897]0.574 [0.516, 0.678]1.338 [1.255, 1.422]0.915 [0.884, 0.953]
Backward0.893 [0.720, 1.070]0.804 [0.641, 0.946]1.539 [1.383, 1.718]1.296 [1.148, 1.420]
Verification0.393 [0.244, 0.587]0.310 [0.282, 0.335]0.479 [0.340, 0.638]0.380 [0.349, 0.400]
Entity0.056 [0.032, 0.091]0.076 [0.047, 0.121]0.027 [0.019, 0.040]0.029 [0.022, 0.040]
Value0.151 [0.099, 0.223]0.193 [0.168, 0.218]0.188 [0.141, 0.246]0.196 [0.163, 0.236]

Mean loss [95% CI], three contexts. Target restores separator/attribute states on 110 selected KV heads; Random uses three matched-capacity head draws. Nominal rates are before restoration; both arms have equal final KV-entry counts, higher than the original compressed budget. Binary probes pool both labels.

Final bit compression after addback: 0.876952–0.877471 (nominal 0.97) and 0.883984–0.884531 (nominal 0.98).

To Q2: At the same 10% ablation rate, removing binding-sensitive heads increased forward and reverse binding loss more than removing access or random heads, while affecting value recognition less. This adds to Wu et al. (2024) by showing that head ablation can selectively impair entity–value binding.

To Q3: We evaluated TabPFN using adapted binding probes (30 × 10). TabPFN achieves 100% accuracy on both forward and backward binding (900 queries each), with losses of 0.597/0.581, respectively. Compared with Qwen3-8B with KVZip, TabPFN achieves similarly high accuracy but slightly higher losses.

Dear Reviewer Why3,

As the discussion period is nearing its end, we wanted to follow up on our response. We would greatly appreciate knowing whether these address your concerns, and we are glad to clarify anything further.

Thank you for your detailed and careful review.

I thank the authors for their substantive rebuttal. The extra controls do appear to confirm that item-preserving mis-binding is a real phenomenon. However, my initial assessment holds: The study is a good start but it still needs some refinement. The new textual contexts address my concerns about the validity of the intervention. Although the sample size is too small for reliable CI estimations.

While the extra ablations are indicative, the results are still correlational.

As the authors write in response to Reviewer AjLj: "Most [but not all] compression methods show a clear separation between binding and item access". The divergence in entity preservation across compression methods (PyramidKV, ChunkKV) needs explanation as it is the opposite behavior of the item-preserving mis-binding and puts into question the generality of the phenomenon.

I maintain that some more causal interventions like activation patching would help distinguish the influence of the lexical, positional, and reflexive binding conditions.

The authors pose an important question and make important head-way in characterizing the phenomenon, however it falls short of the proving its claims that this is true of KV cache compression generally or due to the underscoring of structural tokens. I am raising my soundness score but maintaining my overall rating. However, I encourage the authors to resubmit their paper when the full controls are available.

We sincerely thank the reviewer for the constructive feedback, which has been valuable in improving our paper. We agree with your suggestions regarding mechanistic analysis and acknowledge the importance of more comprehensive causal controls. We have also included additional control results in the revised manuscript. We greatly appreciate your thoughtful comments and careful assessment.

Reviewer_AjLjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that benchmark-level evaluations of KV-cache compression show how much performance drops but not what is lost. It proposes BindProbe, which splits compression-induced contextual errors into two kinds. Item-access failure means an entity or value can no longer be recognized. Binding failure means both are still recognized but their association is lost. Contexts are synthetic records ("Record 001: name Alice; occupation doctor; pet cat; …") with unique entities, and values unique within each attribute type, in randomized order (§3.1). Each fact (e, k, v) gets binary presence probes for e and v, plus forward ("The pet of Alice is:") and reverse binding probes.

All are scored with Brier-style losses on first-token logits, normalized over a candidate pool (Eqs. 4–8). The cache is compressed after prefill and before any probe, and each probe starts from the same compressed cache. Wrong forward answers are split into contextual misbinding (another in-context value) and out-of-context predictions (Eqs. 9–12).

Nine methods are listed on Qwen3-8B, with KVZip also run on Llama-3.1-8B (Table 1). The paper reports a "binding breaks first" regime for Qwen3-8B + KVZip (Fig. 1a) and partial versions for Merging Knorm, Knorm and SnapKV (Fig. 2). It also reports middle-position vulnerability (Fig. 3) and a two-stage pattern: contextual misbinding comes first, then out-of-context failure (Fig. 4). Section 5 (Qwen3-8B + KVZip, ~2K tokens, 30×10) proceeds in four steps:

    1. It scores heads by rank-weighted attention to answer-relevant positions and finds binding scores concentrated in a few heads (Fig. 5).
    1. Ablating those heads raises binding loss while recognition stays low (Fig. 6).
    1. Ablating template slots shows that the separator (";") and attribute-name slots also affect binding (Fig. 7).
    1. KVZip scores those slots lower than name/value slots (Fig. 8). The paper offers this as a plausible mechanism for binding-first degradation.
Critical strengths

The question is well-posed and useful. Separating "the fact is gone" from "the facts are there but wired wrong" is a real gap in compression evaluation, and the protocol is careful in several ways:

  • every probe starts from the same compressed cache;
  • the same contexts are reused across compression levels;
  • record and attribute order are randomized;
  • absent-item negatives guard against a yes-bias;
  • binding candidates come from the full pool, so out-of-context errors can be measured.

The misbinding vs. out-of-context decomposition (Fig. 4) is the most convincing and most practically relevant result. Compressed models first fail plausibly, by recombining real context items, which is harder to detect than outright confabulation. The score-vs-function analysis (Figs. 7–8) points to a sensible hypothesis: a reconstruction-based score like KVZip's undervalues predictable structural tokens that nonetheless carry relational structure. The authors also report the counter-evidence from Llama and SnapKV rather than hiding it.

Critical weaknesses

Validity and generality of the central claim:

  • Format confound. Item access is tested with binary yes/no recognition (chance loss 0.5, Eq. 6). Binding is tested with many-way forced recall over the whole pool (Eqs. 7-8). Recall is harder than recognition, so "binding breaks first" may partly be "the harder probe breaks first". There is no format-matched control, such as binary binding verification ("Is Alice's pet a cat?") or cue-free value recall ("Name a pet in the records"). The disclaimer against comparing families (l. 208–211) does not resolve this, because the headline claim is an ordering across families.
  • The paired design is not used. It is advertised (l. 120–124), but no per-fact conditional is reported, e.g. P(binding wrong | e and v both recognized for that same fact). Fig. 4's in-context rate is aggregate and never conditions on the gold value being recognized.
  • Eq. 7 and Eq. 8 are inconsistent. Eq. 7 sums squared wrong-candidate probabilities; Eq. 8 squares their sum. As printed, Eq. 8 = 2(1−P(e))², so reverse chance ≈ 2 versus ≈ 1 for forward. This likely explains why backward binding pins at 2.0 while forward plateaus near 1.25 (Fig. 1a). The two binding curves are not comparable.
  • Generality is overstated. The asymmetry is clean only for Qwen3-8B + KVZip. Five of the nine Table 1 methods, and the K=5/11 settings, are never shown. The intro claims "a consistent divergence" across methods (l. 054–056), which the paper's own l. 322–329 contradicts.
  • Missing experimental details. The number of contexts T, the pool sizes |U_E| and |U_V,k|, the positive/negative ratio, seeds and confidence intervals are all unspecified (l. 127, 138–140, 166). "Compression rate" is never defined, which matters most for KIVI (quantization), ThinK (channel pruning) and merging. Fig. 1 and Fig. 2 also use different x-axes.

Mechanism, scope and presentation:

  • Section 5 is correlational.

    • p^ans is undefined (l. 404–405).
    • Head ablation (Fig. 6) has no random-head or retrieval-head control and no held-out split between selecting and evaluating heads.
    • The decisive intervention is missing: force-retain the separator and attribute tokens at a fixed budget and check whether binding recovers.
    • The "structural" ablation effects are about 10× smaller than for name/value slots (−0.1 to −0.2 vs −1.1 to −1.9, Fig. 7). Also, binding[0] is the attribute word itself (content, not structure), and binding[2] also hurts value retrieval.
    • All of this is KVZip-only, while the abstract generalizes to "compression criteria".
  • Item information is stored in a distributed way, which complicates the position story. Recognition is ≈0 at rate 0.92, when only about 8% of a ~2K-token cache remains. That is fewer entries than the 330 name/value tokens if retention were even. Item information must therefore survive in other positions' contextualized states or in a subset of heads, which makes the position-ablation story harder to read.

  • Scope. Contexts are ~2K tokens despite the "long-context" framing (l. 363–365), with a single synthetic template. Scoring is single-token forced choice on logits with no free-form generation, so "hallucination" is an indirect proxy.

  • Missing related work. Lost-in-the-Middle (for the positional claim), retrieval-head-aware compression (DuoAttention, RazorAttention, HeadKV), RULER multi-key and variable-tracking tasks, and binding-mechanism follow-ups (Dai et al. 2024; Prakash et al.) are all uncited.

  • Presentation issues.

    • Paragraphs are duplicated (l. 040–045; l. 266–270 vs 294–296).
    • PyramidKV is cited as "Li et al., 2024" (l. 262); the reference list has Cai et al.
    • KIVI is cited for prefill/decode staging (l. 083–084).
    • Table 1 labels KVZip "question-aware", but KVZip is query-agnostic.
    • Table 1 labels KVZip "question-aware", but KVZip is query-agnostic.
    • Fig. 7's y-axis is inverted symlog, and the 3D surfaces in Fig. 3 are hard to read.
    • There is no appendix.
Questions for authors
  1. For facts where both e and v are recognized, what fraction of forward/reverse binding probes fail at rates 0.89 and 0.92?
  2. Does the gap survive format-matched controls (binary binding verification; cue-free value recall)?
  3. Please report T, |U_E|, |U_V,k| (and hence the chance in-context rate 30/|U_V,k| for Fig. 4), the positive/negative ratio, and confidence intervals.
  4. Where are the results for ChunkKV, PyramidKV, Expected Attention, ThinK and KIVI, and for K=5/11? How is "compression rate" defined for quantization, channel pruning and merging?
  5. Was Qwen3-8B run with its chat template, and with thinking mode on or off?
6 comments

We sincerely thank the reviewer for the constructive feedback, which has helped improve the quality and clarity of our manuscript. We will upload the revised manuscript shortly.

For Validity W1 and W3: We added two probes to all evaluations: binary binding verification (Is {entity}'s {attribute} a {value}?, Yes/No) and cue-free value recall (Recall a type of {attribute} mentioned in the context.).

We standardized the probe losses:

\bigl(P_q(\mathrm{yes})-y\bigr)^2+ \bigl(P_q(\mathrm{no})-(1-y)\bigr)^2.}$$ $$\boxed{\ell_{nocue}(q)= \left(1-\sum_{x\in\mathcal A_q}P_q(x)\right)^2+ \sum_{x\in\mathcal C_q\setminus\mathcal A_q}P_q(x)^2.}$$ Here, $\mathcal A_q$ denotes the set of correct answers, and $P_q(x)$ sums the probabilities of accepted realizations in the synonym group $S(x)$; for example, $P(dog) = p(dog) + p(puppy) + \cdots$. $$\boxed{P_q(x)=\sum_{w\in\mathcal S(x)}p_q(w)}.$$ Binary probes share the Yes/No candidate set; cue-free recall uses the unified softmax pool $U_{EV}$. **Format-matched control results (KVZip).** Qwen3-8B retains a binding–access gap under both binary verification and recall controls; Llama-3.1-8B shows a clearer gap for binary probes than for recall. | Model | Rate | Entity | Value | Binding verification | Forward binding | Cue-free recall | | ------------ | ---- | ----------------------------- | ----------------------------- | ----------------------- | -------------------- | ----------------------------- | | Qwen3-8B | 0.00 | 4.36e-19 [7.51e-20, 9.10e-19] | 0.077 [0.070, 0.084] | 0.003 [9.60e-10, 0.008] | 0.028 [0.015, 0.041] | 7.10e-05 [1.49e-08, 1.53e-04] | | Qwen3-8B | 0.89 | 1.14e-15 [1.88e-16, 2.51e-15] | 5.88e-06 [7.59e-08, 1.73e-05] | 0.009 [0.003, 0.016] | 0.090 [0.051, 0.131] | 0.002 [9.03e-06, 0.006] | | Qwen3-8B | 0.92 | 5.47e-12 [4.04e-14, 1.61e-11] | 1.92e-04 [4.61e-08, 0.001] | 0.012 [0.003, 0.026] | 0.192 [0.144, 0.235] | 0.002 [8.65e-06, 0.005] | | Qwen3-8B | 0.97 | 7.89e-09 [5.48e-09, 1.02e-08] | 0.005 [0.002, 0.009] | 0.219 [0.152, 0.281] | 1.289 [1.238, 1.326] | 0.440 [0.302, 0.554] | | Llama-3.1-8B | 0.00 | 5.35e-05 [2.27e-05, 8.60e-05] | 0.008 [0.003, 0.014] | 0.026 [0.018, 0.036] | 0.004 [0.002, 0.006] | 0.019 [0.014, 0.023] | | Llama-3.1-8B | 0.89 | 0.001 [4.46e-04, 0.001] | 0.024 [0.018, 0.030] | 0.274 [0.227, 0.322] | 0.022 [0.019, 0.026] | 0.082 [0.049, 0.120] | | Llama-3.1-8B | 0.92 | 0.002 [0.001, 0.004] | 0.057 [0.037, 0.081] | 0.491 [0.401, 0.586] | 0.040 [0.031, 0.050] | 0.118 [0.048, 0.214] | | Llama-3.1-8B | 0.97 | 0.080 [0.065, 0.095] | 0.499 [0.465, 0.535] | 0.717 [0.653, 0.775] | 0.555 [0.525, 0.586] | 0.320 [0.273, 0.353] | KVZip, 30 entities × 10 attributes, five contexts per model. Cells are loss mean [95% CI]. Entity, Value and Binding verification use positive (Yes) cases, matching the original rebuttal; Forward binding and Cue-free recall use all their unlabelled cases. Backward binding and the full positive-case sweep are retained in the accompanying source tables; negative-case results are reported separately. **For Validity W2 and Q1:** Conditional forward-binding failures persist when both items are recognized (Qwen3-8B, KVZip): | Rate | Total failure (%) | Both recognized (%) | Conditional failure (%) | Conditional in-context error (%) | Conditional out-of-context error (%) | | ---- | ----------------------- | -------------------------- | ----------------------- | -------------------------------- | ------------------------------------ | | 0.00 | 1.468 [0.733, 2.267] | 96.064 [95.730, 96.400] | 1.389 [0.556, 2.230] | 1.389 [0.556, 2.230] | 0.000 [0.000, 0.000] | | 0.89 | 4.870 [2.870, 7.067] | 100.000 [100.000, 100.000] | 4.870 [2.870, 7.067] | 4.803 [2.870, 7.000] | 0.067 [0.000, 0.200] | | 0.92 | 10.674 [7.872, 13.218] | 100.000 [100.000, 100.000] | 10.674 [7.872, 13.218] | 10.474 [7.738, 12.942] | 0.200 [0.000, 0.467] | | 0.95 | 30.020 [28.200, 32.355] | 99.933 [99.800, 100.000] | 29.973 [28.200, 32.310] | 28.571 [26.502, 30.933] | 1.402 [0.870, 1.800] | | 0.98 | 97.465 [96.798, 98.132] | 97.865 [96.994, 98.667] | 97.410 [96.735, 98.082] | 65.031 [58.860, 69.589] | 32.379 [27.747, 39.340] | Qwen3-8B with KVZip, 30 entities × 10 attributes, five independent contexts. Cells are mean [95% CI]. Total failure uses all evaluated facts; conditional errors use only facts for which both entity and value are recognized (threshold 0.5). 95% CIs are computed by bootstrapping independent contexts. These statistics concern forward binding only.

To Q2: The gap persists under format-matched controls for Qwen3-8B, with a less distinct recall-format gap for Llama-3.1-8B (see Validity W1/W3 and its table).

For Validity W4 and Q4: We consistently define compression rates across all experiments as r=1−Bcompressed/Boriginalr = 1 - B_{\text{compressed}}/B_{\text{original}}, where BB denotes theoretical KV-cache storage in bits, with fp16 as the baseline. For eviction and merging, the rate equals the fraction of removed entries; for ThinK, it is half the fraction of removed key channels. For KIVI, we define the nominal compression rate from quantization bitwidths alone, excluding metadata, full-precision residuals, and key-alignment overhead. Selected results are shown below, with full sweeps to be included in the Appendix.

Most compression methods show a clear separation between binding and item access under recall-based probes, while their losses remain comparable under Yes/No probes.

MethodCompression rateΔ EntityΔ ValueΔ ForwardΔ Backward
ChunkKV[0.498745, 0.504062]†+1.8667 [+1.8667, +1.8667]+1.8301 [+1.8132, +1.8428]+1.6939 [+1.6018, +1.8245]+1.5671 [+1.4937, +1.6458]
PyramidKV0.500000+1.6366 [+1.5983, +1.6668]+1.5156 [+1.4600, +1.5904]+1.5006 [+1.4864, +1.5290]+1.6510 [+1.4502, +1.8133]
Expected Attention0.500140+0.0444 [+3.27e-15, +0.0667]+0.0727 [+0.0622, +0.0857]+0.0454 [+0.0066, +0.1026]+0.0533 [+0.0319, +0.0676]
ThinK0.492188+0.0030 [+0.0002, +0.0085]+0.1597 [-0.0247, +0.4170]+1.4636 [+1.4276, +1.4831]+1.1382 [+1.1013, +1.1845]
KIVI0.450533-2.72e-19 [-7.84e-19, +2.24e-21]-0.0028 [-0.0080, +0.0010]+0.0016 [-0.0006, +0.0055]-0.0033 [-0.0064, +0.0019]

Qwen3-8B, 30 entities × 10 attributes, three independent contexts. Each method is evaluated at its available compression rate closest to 0.5. Δ denotes the loss increase relative to the uncompressed baseline. Cells report mean [95% CI] from 2,000 paired context-bootstrap resamples. Entity/Value use positive cases; binding probes are pooled.

† ChunkKV's compression rate is reported as a range because the final partial chunk's selection was not recorded. No interpolation is used. Full sweeps will be included in the Appendix.

For Validity W5 and Q3: We use T=5T=5 independent contexts for KVZip and T=3T=3 for other compression methods, with ∣UE∣=1,511|U_E|=1{,}511. For each attribute kk, the value pool size ∣UV,k∣|U_{V,k}| ranges from 30 to 1,511 across 11 attribute types, of which 10 are sampled per context. The unified softmax candidate pool UEVU_{EV} contains 2,377 groups, giving a chance in-context rate of 30/2377=1.26%30/2377=1.26\% for forward binding under uniform sampling.

Root seeds: 4242 (contexts), 4343 (wording), 4545 (random heads), and 142142 (head-score calibration); each seeds a sampling stream, not an individual trial.

Positive and negative cases are evaluated separately; current figures show positives, and negatives will be included in the Appendix. The positive/negative sampling ratio is not specified in the current results.

For Mechanism W1:

We distinguish retrieval from binding with contrastive attention scoring.

For each layer ℓ\ell and head hh, we define:

=\mathrm{KeyProj}_{\ell,h}(E), \qquad K_{\mathrm{res}}^{\ell,h} =K_{\mathrm{ctx}}^{\ell,h}-K_{\mathrm{raw}}^{\ell,h},$$ where $\mathrm{KeyProj}$ includes the model-specific normalization and positional encoding operations. The corresponding attention distributions are: $$A_R^{\ell,h} =\operatorname{softmax}\left( \frac{Q_{\mathrm{ctx}}^{\ell,h}(K_{\mathrm{raw}}^{\ell,h})^\top}{\sqrt{d_h}} \right),$$ $$A_B^{\ell,h} =\operatorname{softmax}\left( \frac{Q_{\mathrm{ctx}}^{\ell,h}(K_{\mathrm{res}}^{\ell,h})^\top}{\sqrt{d_h}} \right).$$ For $K$ queries and their corresponding targets, let $M_{ij}^{\ell,h}$ denote the attention mass assigned to target $j$ under query $i$. The head score is: $$\boxed{ S^{\ell,h} =\frac{1}{K}\sum_{i=1}^{K}M_{ii}^{\ell,h} -\frac{1}{K(K-1)} \sum_{i=1}^{K}\sum_{j\neq i}M_{ij}^{\ell,h} }$$ We compute **retrieval scores** using $A_R$ with content-access queries, and **binding scores** using $A_B$ with entity-conditioned attribute queries. Head scores were evaluated on three random contexts generated with root seed $142$. Ablations used three contexts generated with seed $42$. Random heads were sampled three times per context using root seed $45$.

Head ablation.

StrategyHead fractionΔ ForwardΔ BackwardΔ Value
Access0.10+0.193 [+0.160, +0.223]+0.455 [+0.419, +0.499]+0.131 [+0.119, +0.141]
Binding0.10+1.041 [+0.963, +1.117]+1.387 [+1.361, +1.415]+0.021 [+0.011, +0.033]
Random0.10+0.230 [+0.033, +0.406]+0.129 [+0.020, +0.298]+0.276 [+0.010, +0.764]

Paired mean loss increases [95% CI], three contexts; Random averages three head draws per context. Nominal 10% removes 28/288 KV heads, selected after GQA score pooling.

Force-retention add back.

Probe0.97 Random0.97 Target0.98 Random0.98 Target
Forward0.729 [0.602, 0.897]0.574 [0.516, 0.678]1.338 [1.255, 1.422]0.915 [0.884, 0.953]
Backward0.893 [0.720, 1.070]0.804 [0.641, 0.946]1.539 [1.383, 1.718]1.296 [1.148, 1.420]
Verification0.393 [0.244, 0.587]0.310 [0.282, 0.335]0.479 [0.340, 0.638]0.380 [0.349, 0.400]
Entity0.056 [0.032, 0.091]0.076 [0.047, 0.121]0.027 [0.019, 0.040]0.029 [0.022, 0.040]
Value0.151 [0.099, 0.223]0.193 [0.168, 0.218]0.188 [0.141, 0.246]0.196 [0.163, 0.236]

Mean loss [95% CI], three contexts. Target restores separator/attribute states on 110 selected KV heads; Random uses three matched-capacity head draws. Nominal rates are before restoration; both arms have equal final KV-entry counts. Binary probes pool both labels.

After addback, actual compression is 0.876952–0.877471 (nominal 0.97) and 0.883984–0.884531 (nominal 0.98).

To "structural" ablation effects

Our hypothesis is that preferential removal of attribute and separator tokens contributes to the early rise in binding loss, even though their overall ablation effects are smaller. The retention results support this contribution.

Mechanistic Analysis.

We extended the mechanistic analysis to Llama and Expected Attention. In Llama, ablating CBS-selected heads increases forward/backward binding losses by 0.164/0.750, compared with 0.047/0.029 under random ablation, while largely preserving item recognition. Position-level ablation further confirms the importance of structural positions, with separator removal increasing forward-binding loss by 0.112.

Across both models, KVZip assigns lower importance to structural positions than to name/value positions. At 95% compression, their retention rates are 12.08% vs. 66.62% in Qwen and 0.63% vs. 45.86% in Llama, respectively. Expected Attention shows a similar pattern: at 95% compression, structural versus name/value retention is 0.89% vs. 22.37% in Qwen and 0.74% vs. 26.46% in Llama. This suggests that our explanation may partially generalize beyond KVZip.

For Mechanism W2: With grouped-query attention (GQA), each token has KV states across KV heads. Both Qwen3-8B and Llama-3.1-8B have 32 attention heads and 8 KV heads. We prune over the pooled KV heads, giving token length×KV heads\text{token length} \times \text{KV heads} KV pairs before compression. At 92% compression, approximately 2k×8×0.08≈12802k \times 8 \times 0.08 \approx 1280 KV pairs remain, exceeding the 300 entity–attribute values.

For Scope W1: We acknowledge the limited context length (~2K tokens). Candidate normalization preserves the relative probabilities and ranking of first-token answers, so it does not introduce the observed preference for incorrect candidates. However, this is not equivalent to free-form generation; we will clarify this limitation.

For Presentation W: We will address the presentation issues in the revised manuscript.

To Q5: Both Qwen3-8B and Llama-3.1-8B use their respective chat templates, with thinking mode disabled.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Introduction to the Revised PDF

We thank all reviewers for their time and constructive feedback.

We have revised the manuscript to address the reviewers' comments and incorporated additional analyses and experimental results. The main updates are summarized below:

  1. Probe Formulation: We refined the definitions and formulations of the evaluation probes to improve clarity and consistency.

  2. Robustness Analysis: We added a discussion of robustness across different models, compression methods, linguistic structures, and complex reasoning scenarios.

  3. Additional Studies: We incorporated additional experiments, including causal interventions and targeted KV retention, to further validate our findings.

  4. Appendix Updates: We expanded the appendix with detailed experimental results, supplementary analyses, and supporting evidence for the reviewers' questions.

These revisions strengthen the empirical validation and mechanistic understanding of our findings while preserving the paper's core contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.