acceptodds
Withdrawn from ICLR 2027

How Active Readout Paths Shape KV-Cache Sensitivity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Across 64 shared QASPER document caches on Qwen2.5-1.5B, changing the question produces a median absolute same-entry log-sensitivity change of 0.573. We ask what a score must observe to predict the effect of a fixed KV perturbation. Controlled query substitution isolates reader dependence; an audited tail case gives a -fold change with storage held fixed. Under radial perturbations, static vector magnitude gives a mean within-reader position-rank correlation of , squared current-reader routing reaches 0.747, and the first-order last-query response (S4) reaches 0.778. In an eight-document matched perturbation-family control, S4 falls from 0.780 under radial perturbations to 0.403 under unit-Gaussian directions. In a separate seven-document length control, it falls from 0.807 at 512 tokens to 0.580 at 8,192 tokens; routing remains close. Tracking the same entry across questions requires more path information: on a matched path sample, complete-suffix differentiation raises layer-3 tracking from 0.100 for the last-query downstream-Fisher diagnostic to 0.698, with little gain at layer 23. Actual joint deletion reveals a parallel depth pattern: measured-sensitivity selection reduces output divergence beyond routing at layer 3, while no tested score gives a resolved improvement over routing at layer 23. Current routing supports position ranking; broader path coverage improves early-layer tracking, and joint deletion locates a corresponding early-layer gap in output preservation.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, large language models, KV cache, causal interventions, attention, sensitivity analysis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_VZ8SOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper essentially tries to make an argument toward answering the following question: "Does the importance of a cached token change depending on the current question reading it?" This question seems useful enough since it could motivate further kv cache compression algorithms if the cached token change stays largely unchanged.

The paper gives the following answer which I agree but does't seem to very interesting nor useful. The answer is essentially that a cached token that matters for one question may matter much less, or more, for another.

Critical strengths

I think the authors have made reasonable attempt to compare ways to predict which cache positions matter.

Below are the ones I have noted.

  • Vector size alone performs poorly.
  • Previous attention history is somewhat useful.
  • The current question’s attention weights—called routing—perform well.
  • More complicated local calculations improve predictions only slightly.
  • Following the complete computational path helps in early layers, but is expensive and not offered as a practical selection method.
Critical weaknesses
  • i don't find the claims and findings of the paper to be sufficiently insightful and interesting. Cache importance depends on the current query; this is already intuitive and related query-aware cache methods exist.

  • It proposes no usable new cache-compression algorithm (again i understand it's more of a analysis paper, but if the analysis is not good enough a method would be useful to have)

  • The empirical results are also rather small scale and toy. This is minor if the results are consistently good; however, results weaken substantially for longer contexts and different perturbation directions.

  • The paper is badly written. I do not find the abstract to be a good way of presenting the paper. As an example, the starting sensente "Across 64 shared QASPER document caches on Qwen2.5-1.5B, changing the question produces a median absolute same-entry log-sensitivity change of 0.573." Please do not assume reader would know what QASPER document is and what does "same-entry log-sensitivity change of 0.573" enetail specifically? i am already quite lost from reading the abstract..

  • Terminology is quite poorly defined. Much of this language is author-created framing. For example reader: the current question/query consuming the cache (at least give a definition in the introduction please) . Active readout path, which i assume is the the computation carrying information from a cached entry to the model output. Sensitivity, which i think is how much the output distribution changes when one cached entry is slightly perturbed.

Questions for authors
  1. What practical cache-selection algorithm results may from these findings? Some discussion would be useful to motivate the analysis.

  2. Does the conclusion survive actual deletion rather than small radial perturbations?

  3. Can the early-layer full-path signal be approximated cheaply?

Reviewer_yP7wOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how active readout paths influence KV-cache sensitivity in language models. Through controlled KV perturbations and query substitutions, the authors analyze the roles of attention routing, query alignment, value contrast, and downstream computation. The results suggest that current-query attention captures most of the signal needed for position ranking, while complete suffix-path information is important for tracking sensitivity changes in early layers.

Critical strengths
  1. Systematic empirical analysis. The paper provides a structured investigation of KV-cache sensitivity through controlled interventions and path decomposition. The distinction between position ranking and cross-reader sensitivity tracking offers potentially useful insights into KV-cache importance estimation.
Critical weaknesses
  1. Limited novelty and practical contribution. The central observation that KV-cache importance depends on the active query is already reflected in existing query-aware cache management methods. The downstream Jacobian analysis largely follows established sensitivity principles. While the paper provides extensive empirical characterization, it introduces neither a fundamentally new theoretical framework nor a practical method that demonstrates improvements over existing KV-cache compression approaches.

  2. Questionable numerical reliability and limited validation. The paper reports substantial discrepancies between finite-difference sensitivity measurements and their analytical first-order counterparts. Sensitivity estimates also vary significantly with perturbation scale and execution layout, raising concerns about the reliability of the primary measurement target. Moreover, the main experiments focus on relatively small models and short contexts, while the joint-deletion experiments do not establish benefits in realistic multi-layer KV-cache compression or downstream task accuracy.

Questions for authors
  1. How can the authors establish that the reported sensitivity rankings and path-coverage conclusions are robust to numerical errors, given the substantial discrepancies between finite-difference measurements and analytical derivatives, as well as their dependence on execution layout?

  2. What is the fundamental new insight beyond established query-dependent KV-cache importance and downstream Jacobian-based sensitivity analysis?

  3. Can the authors demonstrate that the identified early-layer path-coverage effects translate into meaningful improvements in realistic multi-layer KV-cache compression, particularly in generation quality or downstream task accuracy?

Reviewer_y9PSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how KV-cache sensitivity depends on the queries accessing cached entries. In the main experiments, the authors show that the same KV perturbation can have different effects under different queries. They find that current-query attention provides a strong signal for sensitivity ranking, while accounting for earlier positions in the question suffix is important for tracking sensitivity changes across queries in shallow layers.

Critical strengths

S1: Interesting insights. The paper shows that the same KV perturbation can have different effects under different queries. It also finds that current-query attention provides a strong signal for sensitivity ranking, while accounting for earlier positions in the question suffix improves sensitivity tracking in shallow layers.

S2: Controlled experimental design. The authors isolate reader-dependent effects by keeping the stored KV entries and perturbations fixed while substituting the query activations.

Critical weaknesses

W1:Poor Presentation. The paper lacks a clear narrative structure, making it difficult to follow. The figures, particularly Figure 3, are unclear and accompanied by very long captions, which further reduces readability.

W2:Limited Experiments. While the appendix provides some cross-model validation, the main experiments are primarily conducted on the Qwen2.5 family, with limited replication on other architectures. The long-context and joint deletion evaluations are also small-scale, and downstream task performance is not evaluated. These limitations leave the generalizability and practical relevance of the findings insufficiently established.

W3: Limited practical implications. While the paper provides insights into KV-cache sensitivity, it remains unclear how these findings could translate into practical improvements in KV-cache compression. The authors do not propose a concrete method to leverage these findings.

Questions for authors

Q1: Cross-model generalization. Could the authors provide additional experiments on other model architectures, such as Llama, to further validate the generalizability of the main findings?

Q2: Downstream evaluation. Could the authors evaluate the impact of sensitivity-based KV-cache selection on downstream task performance?

Q3: Practical applicability. How could the findings about earlier question-suffix positions be incorporated into a practical KV-cache compression method without substantial computational overhead?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.