acceptodds
Under review as a conference paper at ICLR 2027

Not All Queries Are Equal: Identifying Informative Queries for KV Cache Compression

Abstract

Long prompts make the KV cache the dominant memory cost, so a common remedy is to keep only part of it once prefill ends. The standard methods share one form: they sum the attention each key receives from prefill queries and keep the keys with the largest sums. They differ only in which queries they include. We show that queries differ sharply in how useful they are for this sum. A few informative queries point to the keys that generation will read, while most are noisy and do not, and the choice between them largely determines accuracy. We propose Q-Trust, which identifies informative queries from the attention of the user's request and selects the keys they point to through 2-hop attention. Without any training, Q-Trust improves accuracy over SnapKV by 1.72–2.56 points on Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B-v0.3, and by 0.78–1.58 points over SnapKV with pooling. A Triton kernel that reuses the log-sum-exp from FlashAttention limits the prefill overhead to 3.5% for 16K-token prompts.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache compression, long-context inference, large language models, attention, token eviction, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_AeQWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the limitations of SnapKV’s fixed observation window by proposing a 2-hop attention analysis technique. The authors define a query informativeness score and empirically show that queries receiving more attention from the request query tend to have higher informativeness scores. Based on this observation, they propose Q-Trust, a training-free KV cache compression method. Q-Trust improves average task performance over SnapKV, both with and without pooling, at small KV cache budgets, with modest additional prefill overhead.

Critical strengths
  • The analysis provides a clear motivation for the proposed method. The paper starts with a query informativeness metric that requires future attention, then proposes request attention as a proxy available during prefill. Table 1 provides empirical evidence supporting this connection.

  • The method addresses computational cost as well as accuracy. The authors address the prefill overhead introduced by 2-hop scoring through selective computation and kernel optimization, limiting the additional overhead to 3.5% for 16K-token prompts.

Critical weaknesses
  • The performance gap on QA and retrieval tasks is insufficiently analyzed. SnapKV with pooling outperforms Q-Trust on single-document and multi-document QA across all three evaluated models, as well as on Needle-in-a-Haystack retrieval. However, it remains unclear whether this gap reflects the benefits of preserving neighboring context or limitations of Q-Trust’s scoring and selective computation. Since pooling appears orthogonal to the proposed query-weighting mechanism, evaluating Q-Trust with pooling under the same KV cache budget would help clarify the source of the gap and whether it can be mitigated.
Questions for authors
  • Could you evaluate Q-Trust with pooling under the same KV cache budget and compare it with both the original Q-Trust and SnapKV with pooling?

  • Could you repeat the compression experiment in Figure 1(b), and the informativeness-based query-weighting experiment in §2.4, using informativeness scores without common-component removal? This would clarify whether the downstream benefits attributed to informative queries are robust to this analytical choice.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AeQW: Q-Trust + Pooling and Shared-Component Ablation

We thank the reviewer for the concrete suggestion to combine Q-Trust with pooling. G1 to G4 are in the General Response.

Q1 / W1. "Could you evaluate Q-Trust with pooling under the same KV cache budget and compare it with both the original Q-Trust and SnapKV with pooling?"

Q1 / W1. With pooling, Q-Trust closes most of the QA gap, keeps most of the few-shot gain, and has higher Needle-in-a-Haystack accuracy than SnapKV + pooling. We trace part of the QA difference to the direction of the second hop.

As the task-group table in G1 shows, Q-Trust + pooling is above SnapKV + pooling on Multi-doc QA for all three models, and the Single-doc QA gap is now 0.2 to 0.9 points. On Few-shot it stays 3.8 to 5.0 points ahead. On Needle-in-a-Haystack it scores 97.5%, against 97.1% for SnapKV + pooling, both at the level of the full cache (96.9%).

Attention is causal, so a trusted query adds score only to keys before it, and the tokens right after a span the request refers to, such as the tail of an answer, receive less score. In the needle task, most wrong answers of Q-Trust were numbers with only their leading digits correct. Our pooling gives each key the highest score among itself and the keys just before it, which covers these tokens. Pooling can lower the LongBench average of Q-Trust, but Q-Trust + pooling stays above both SnapKV and SnapKV + pooling on all three models, so we report both Q-Trust rows (Section 4.3, Tables 2 and A.1).

Q2. "Could you repeat the compression experiment in Figure 1(b), and the informativeness-based query-weighting experiment in §2.4, using informativeness scores without common-component removal?"

Q2. Without removing the shared component, the conclusion is the same, with a slightly larger gap.

As G3 shows, the quartile intervention gives top − random +4.00 and top − bottom +12.45, against +3.50 and +12.39 for our definition. In the query-weighting experiment, weighting in proportion to informativeness gives 41.02 without removal and 41.08 with it (s weighting: 36.43). The share of s on the top quartile rises from 58.0% to 69.2%. We have added both experiments to Appendix G.

Thank you for additional experiments and clarifications. All my questions have been answered.

Reviewer_EhUiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed Q-Trust which is a training-free method that uses the final request tokens' attention to identify supporting tokens for retention. It shows that only a small subset of prefill queries reliably the keys needed during generation, while combining attention from other queries can weaken the signal. Across LongBench and three 7B–8B models, Q-Trust improves over SnapKV, particularly under small cache budgets and on few-shot tasks; a custom Triton implementation limits the reported 16K-prefill overhead to 3.5%.

Critical strengths
  1. It unifies attention-based KV Cache compression methods as different ways of weighting prefill queries.

  2. It demonstrates that query informativeness is highly uneven.

  3. It requires no learned predictor or model specific modification.

  4. The largest benefits appear at the small 128-token KV budget, where careful token selection matters most.

  5. The custom Triton kernel reduces the reported prefill overhead to approximately 3.5% at 16k context.

Critical weaknesses
  1. Q-Trust assumes the final 16 tokens represent the user's request. This may fail for long questions or prompts where the task appears elsewhere.

  2. The paper reports additional prefill time, but not end-to-end latency, throughput, batch size, peak memory in realistic serving workloads.

  3. NACL also uses question-token attention and is closely related, but is discussed without an experimental comparison.

  4. The main benchmark is LongBench, with an additional synthetic needle task. Some reasoning would strengthen the claims.

Questions for authors
  1. How does Q-Trust perform when the user request is longer than 16 tokens or is not located at the end of the prompt?

  2. Can the authors provide a direct experimental comparison with NACL, which also uses attention from question tokens?

  3. Can the authors report realistic end-to-end serving measurements, including total latency, throughput, peak memory, and achievable batch size?

  4. Can the authors clarify how robust the “informative query” conclusions are to alternative definitions of future importance, such as output perturbation or value-aware metrics rather than future attention alone?

  5. Could Q-Trust be combined with pooling? Would this improve QA and needle-retrieval performance while preserving its few-shot gains?

2 comments
AuthorsOpenReview ↗

Response to Reviewer EhUi: NACL, Serving Cost, Q-Trust + Pooling, HELMET, and GSM8K

We thank the reviewer for the questions on deployment and robustness. G1 to G4 are in the General Response.

Q1 / W1. "How does Q-Trust perform when the user request is longer than 16 tokens or is not located at the end of the prompt?"

Q1 / W1. 71% of LongBench requests are already longer than 16 tokens, and the fixed window works better than the full request. With the question before the document, Q-Trust keeps its gain.

On questions longer than 16 tokens, Q-Trust is 1.08 to 3.22 points above SnapKV, and, as G2 shows, it also scores above SnapKV with the question before the document and with an instruction in the last 16 tokens (Appendix D).

Q2 / W3. "Can the authors provide a direct experimental comparison with NACL, which also uses attention from question tokens?"

Q2 / W3. Even in its most favorable setting, NACL stays below SnapKV, and Q-Trust is 2.6 to 3.3 points above NACL on all three models.

The official code cannot run below 25.6K input tokens, so we implemented NACL's algorithm in our codebase: proxy-token attention sum plus a random part of the budget sampled from the softmax of the scores. On Llama we searched the proxy span (last 16 tokens, last 1%) and the top-k:random ratio (2:7, 12:7) and used the best (last 16 tokens, 12:7) for all models. Three seeds vary little (34.67 to 34.85). As G4 shows, NACL stays 0.6 to 1.3 points below SnapKV with this setting (Appendix C).

Q3 / W2. "Can the authors report realistic end-to-end serving measurements, including total latency, throughput, peak memory, and achievable batch size?"

Q3 / W2. Q-Trust matches SnapKV in latency, throughput, memory, and batch size. Both answer a 16K-token request in about 40% less time than the full cache and fit twice as many concurrent requests.

Llama-3.1-8B, one RTX 3090 (24 GB), KV budget 128, 256 output tokens. A batch is prefilled in one forward pass and decoded together, and the batch grows until the first out-of-memory error.

Full cacheSnapKVQ-Trust
Latency of one request, 16K (s)19.9112.2812.12
Largest batch, 16K / 8K2 / 44 / 84 / 8
Peak memory at batch 1, 16K, with weights (GiB)18.7816.8016.80
End-to-end throughput at the largest batch, 16K (tokens/s)15.039.339.3

Prefill activations set the batch limit. For one 16K-token request, time to first token is 4.68 s for Q-Trust and 4.62 s for SnapKV (Appendix C). These measurements are now in Appendix E.

Q4. "Can the authors clarify how robust the 'informative query' conclusions are to alternative definitions of future importance, such as output perturbation or value-aware metrics ...?"

Q4. The conclusion holds under both value-aware importance and output perturbation.

As G3 shows, under output perturbation the top quartile is 4.26 points above a random subset and 11.71 above the bottom quartile, and under value-aware importance 4.17 and 11.90 (Appendix G). Query weighting gives 41.11 and 40.87 under the two definitions, close to ours (41.08).

Q5. "Could Q-Trust be combined with pooling? ..."

Q5. Yes. QA reaches the level of SnapKV + pooling, most of the few-shot gain remains, and Needle-in-a-Haystack accuracy improves.

As G1 shows, on Llama, Q-Trust + pooling is within 0.2 points of SnapKV + pooling on Single-doc QA, above it on Multi-doc QA, and 4.1 points ahead on Few-shot. On Needle-in-a-Haystack it scores 97.5%, against 97.1% for SnapKV + pooling (Sections 4.3 and 4.4).

W4. "The main benchmark is LongBench, with an additional synthetic needle task. Some reasoning would strengthen the claims."

W4. Beyond LongBench, Q-Trust is above SnapKV on HELMET in all six model and length settings. On chain-of-thought reasoning, it is about 1 point above SnapKV and within 3 points of the full cache while keeping 2% of the prompt.

HELMET's seven categories cover more task types than LongBench, including RAG, re-ranking, citation, and in-context learning. The table reports the average over the seven categories (budget 1024).

HELMET avgLlamaQwenMistral
8K, SnapKV41.1931.8930.64
8K, Q-Trust42.3933.2631.72
8K, full cache59.0557.2357.06
16K, SnapKV38.6130.3328.48
16K, Q-Trust38.7834.0430.08
16K, full cache56.8456.4153.67

For chain-of-thought reasoning, we adopt the many-shot GSM8K setup and reasoning model of ChunkKV, which uses it as a long-context reasoning test (DeepSeek-R1-Distill-Llama-8B, 1,319 questions, 50 CoT examples in a prompt of about 7.2K tokens, prefill-only compression, greedy decoding up to 2,048 tokens):

Retained share of the promptSnapKVQ-TrustQ-Trust − SnapKV
2%84.5385.52+0.99
5%85.6786.73+1.06
Full cache88.32

We have added these benchmarks to Appendix F.

Thanks for the answers. I will maintain my positive score.

Reviewer_NSxhOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

Q-Trust compresses the KV cache by using the request's attention to weight other prefill queries, then following their attention to select keys. The paper studies query informativeness, evaluates three models on LongBench, adds a needle-retrieval test, and implements sparse scoring with Triton kernels.

Critical strengths
  1. The query-informativeness analysis goes beyond an attention correlation. Selecting different query quartiles changes answer quality substantially, with controls for position and random selection. This gives a useful explanation of why accumulating more queries can hurt compression.

  2. The scoring rule is simple and needs no training. Gains over pooled SnapKV hold across three models at budget 128, and the kernel reduces the extra prefill time to 3.5% at 16K tokens. The threshold analysis makes the accuracy-cost tradeoff easy to assess.

Critical weaknesses
  1. The claim that earlier query weighting is position-only seems too broad. Nexus Sampling already uses content-dependent weights, and QuoKA selects informative queries for prefill acceleration. The request-guided two-hop rule is different, but its benefit over closer alternatives such as Lookahead Q-Cache remains unclear.

  2. SnapKV uses a 16-token observation window here, rather than 32 in its original LongBench experiments. Table 4 varies only Q-Trust's window. Given the 0.78-1.58-point gain over pooled SnapKV, a comparison with the original or separately tuned baseline settings would make the advantage more convincing.

Questions for authors
  1. How would Q-Trust handle practical prompts where the question precedes the context or formatting instructions come last? Would locating the actual request require a separate extraction model? Any extraction or extra prefill cost could reduce the practical gains. This does not affect my assessment or score; I’m just curious about how it is applied.

  2. At the same KV budget, how does Q-Trust compare with Lookahead Q-Cache in answer quality and time to first token?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NSxh: Lookahead Q-Cache, Nexus Sampling, and Tuned SnapKV

We thank the reviewer for the precise pointers to related work and baseline settings. G1 to G4 are in the General Response.

Q1. "How would Q-Trust handle practical prompts where the question precedes the context or formatting instructions come last? Would locating the actual request require a separate extraction model?"

Q1. No extraction model or extra prefill is needed. With the question before the document, Q-Trust keeps its gain.

As G2 shows, Q-Trust scores above SnapKV both with the question before the document and with an instruction after the question (Appendix D).

Q2. "At the same KV budget, how does Q-Trust compare with Lookahead Q-Cache in answer quality and time to first token?"

Q2. LAQ++ is more accurate, but it holds the full cache until its lookahead tokens are generated. On a 24 GB GPU, LAQ++ runs out of memory at 64K tokens, while Q-Trust still runs.

We ran LAQ++ (EMNLP 2025, 8 lookahead tokens) in our codebase.

LongBench avgBudgetQ-TrustLAQ++
Llama-3.1-8B12837.4040.29
Qwen2.5-7B12841.8645.21
Mistral-7B-v0.312841.6243.79
Llama-3.1-8B25638.8741.36
Llama-3.1-8B51240.0842.10

At 16K tokens, the time to first token (TTFT) is 4.62 s for SnapKV, 4.68 s for Q-Trust, and 5.00 s for LAQ++ (best of three runs, Appendix C). Excluding the model weights, LAQ++ needs about twice the peak memory of Q-Trust (3.8 vs. 1.9 GiB at 16K, G4). The accuracy difference comes mostly from QA tasks and TREC. We have added this comparison to Appendix C.

W1. "The claim that earlier query weighting is position-only seems too broad ..."

W1. We have revised the statement so that the distinction from Nexus Sampling and QuoKA is precise (Abstract, Introduction, Related Work), and Nexus Sampling, the closest eviction method, stays at SnapKV's level.

Nexus SamplingQuoKAQ-Trust
GoalKV evictionChunked-prefill acceleration (all KV kept)KV eviction
Queries used for scoringLast 1616 per chunk, farthest in cosine from the chunk meanAll prefill queries
Query weightingOverlap with the accumulated attentionNone (selection only)Attention from the request
Attention of queries outside the windowNot usedNot applicable (cosine to keys)Used
  • Original: "The methods differ only in the weight w given to each query, and so far w has been a function of position, blind to the query's content."
  • Revised: "The methods differ in the weight w they give to each query. They choose by position which queries score the keys (the last window, the question span, or all queries) and weight the chosen queries equally (SnapKV, NACL, H2O) or by their overlap with the accumulated attention (Nexus Sampling)." Later in the Introduction: "Earlier methods set this weight by position or by overlap with the accumulated attention, whereas Q-Trust sets it from the request and extends it to queries outside the observation window."

We now cite QuoKA, which is complementary to eviction. The comparison with Lookahead Q-Cache is under Q2.

Nexus Sampling has no public code, so we implemented it from its equations. Its block-level selection falls below SnapKV at budget 128 (Llama −2.63), and its best variant (per-KV-head score, token-level top-k) is at SnapKV's level, with Q-Trust 1.66 to 2.47 points higher (G4, Appendix C).

W2. "SnapKV uses a 16-token observation window here, rather than 32 ... a comparison with the original or separately tuned baseline settings would make the advantage more convincing."

W2. Q-Trust stays ahead under SnapKV's original setting (+1.7 to +2.3 points) and under separately tuned windows.

In the tables, P denotes pooling.

LongBench avgSnapKV + P, window 32 (original)Q-TrustGainQ-Trust + PGain
Llama-3.1-8B35.1137.40+2.2936.52+1.41
Qwen2.5-7B40.1241.86+1.7442.47+2.35
Mistral-7B-v0.339.5441.62+2.0841.52+1.98

The original setting is below SnapKV + P with window 16 (G1) on all three models (−0.71, −0.96, −0.94). We then tuned the window over 8, 16, and 32 for SnapKV, SnapKV + P, and Q-Trust + P, with Q-Trust fixed at 16 (chosen window in parentheses):

LongBench avgSnapKVQ-TrustGainSnapKV + PQ-Trust + PGain
Llama-3.1-8B35.88 (8)37.40 (16)+1.5236.66 (8)37.06 (8)+0.40
Qwen2.5-7B40.14 (16)41.86 (16)+1.7241.72 (8)42.47 (16)+0.75
Mistral-7B-v0.339.10 (8)41.62 (16)+2.5241.12 (8)42.26 (8)+1.14

Q-Trust stays ahead in all six comparisons. We have added the original and tuned settings to Appendix C.

Thanks for the added comparisons. These address W1 and W2, and I will raise my rating to 4. It would also be good to mention TokenSelect (Wu et al., EMNLP 2025) in related work, since it also uses query information for KV selection. A brief clarification of the connection and differences would be helpful.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Q-Trust + Pooling, Request Position, Informativeness, and Baselines

We thank all reviewers for the careful reading and the concrete questions. Each reply refers to the experiments below as G1 to G4. All runs use the paper's codebase and settings (LongBench, KV budget 128 unless stated otherwise, Llama-3.1-8B, Qwen2.5-7B, Mistral-7B-v0.3). "Significant" refers to the paired bootstrap within tasks of Section 4.2. Changes in the revised PDF are colored by reviewer (NSxh blue, EhUi green, AeQW orange).

G1. Q-Trust + pooling closes most of the QA gap and keeps most of the few-shot gain (Section 4.3, Tables 2 and A.1)

Pooling acts along the key axis and Q-Trust weights the query axis, so the two combine directly. Q-Trust + pooling (P in the tables) gives each key the highest Q-Trust score among itself and the keys just before it, at the same budget.

ModelMethodSingle-doc QAMulti-doc QAFew-shotLongBench avg
Llama-3.1-8BSnapKV + P22.3120.9860.5535.82
Q-Trust21.7620.7065.7237.40
Q-Trust + P22.1121.0764.6436.52
Qwen2.5-7BSnapKV + P34.9840.2355.4241.08
Q-Trust34.1840.0659.0241.86
Q-Trust + P34.8041.2360.4542.47
Mistral-7B-v0.3SnapKV + P32.7034.1561.5540.48
Q-Trust31.2333.2566.1141.62
Q-Trust + P31.7734.3865.3741.52

On Needle-in-a-Haystack (Llama, 4K to 32K, Figure 4), Q-Trust + pooling scores 97.5%, against 97.1% for SnapKV + pooling and 96.9% for the full cache.

G2. No extraction model is needed, and the request need not fit in the last 16 tokens (Appendix D)

Q-Trust always uses the last 16 tokens as the observation window, with no extra model or prefill. In LongBench, 71% of requests, including their instructions, are longer than 16 tokens (median 43). The fixed window is better than using the last sentence or the full request (37.40, 36.87, 35.22 in Table 4), and within each task the gain does not depend on question length.

As a stress test, we rebuilt the six LongBench QA tasks in two other layouts (Llama-3.1-8B, average QA score):

LayoutSnapKVQ-TrustFull cache
Original20.0821.2328.16
Instruction after the question16.4116.6423.74
Question before the document12.9514.2820.17

In both new layouts, the prompt ends with an instruction, so the last 16 tokens contain no question tokens. A token's importance also depends on its position. The last tokens can attend to every key, so their attention is still a meaningful first hop. Q-Trust scores above SnapKV in both layouts. Both layouts lower the score of every method, including the full cache.

G3. The informative-query conclusion holds under other definitions of informativeness (Appendix G)

In the quartile intervention of Figure 1(b) (sampled LongBench, 315 prompts, Llama-3.1-8B), we changed the definition of informativeness in three ways:

DefinitionTop − randomTop − bottomShare of s on the top quartile
Ours (shared component removed)+3.50+12.3958.0%
Shared component not removed+4.00+12.4569.2%
Value-aware (attention × ‖v‖)+4.17+11.9057.6%
Output perturbation (change in each layer's output when the key is evicted)+4.26+11.7157.8%

Used as query weights, the four definitions give 41.08, 41.02, 41.11, and 40.87 (s weighting: 36.43).

G4. Additional baselines: NACL, Nexus Sampling, and LAQ++ (Appendix C)

NACL and Nexus Sampling fall below SnapKV in their original settings at this budget, so we searched their settings on Llama and used the best one on all three models.

LongBench avgSnapKVNACL (best)Nexus Sampling (best)Q-TrustLAQ++
Llama-3.1-8B35.4934.8535.7437.4040.29
Qwen2.5-7B40.1438.8939.8941.8645.21
Mistral-7B-v0.339.0638.2839.1541.6243.79

Q-Trust is 2.55, 2.97, and 3.34 points above NACL and 1.66, 1.97, and 2.47 points above Nexus Sampling (all significant). LAQ++ is more accurate. It keeps the full cache until it has generated its lookahead tokens and rescored the prompt, so it needs about twice the memory of Q-Trust beyond the model weights. On a 24 GB GPU, LAQ++ runs out of memory at 64K tokens, while Q-Trust still runs.

Peak memory excl. weights (Llama-3.1-8B, RTX 3090)16K32K64K
Q-Trust1.9 GiB3.7 GiB7.3 GiB
LAQ++3.8 GiB7.6 GiBOOM

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.