acceptodds
Under review as a conference paper at ICLR 2027

QueEn: Query Enrichment for KV Cache Eviction in Long Reasoning

Abstract

Key-value (KV) cache eviction alleviates the memory bottleneck of long-reasoning inference by selectively retaining important KV pairs. Most eviction methods score each cached KV pair by the attention it receives from the queries of the most recent tokens. However, these scores can undervalue KV pairs that future decoding still needs, since the queries cover only recent context and, under causal masking, each cannot see later tokens in the window. To address these limitations, we propose Query Enrichment (QUEEN), a training-free method that refines the queries of the most recent tokens so that each reflects all of them (intra-window refinement), and reuses refined queries from earlier reasoning and the input prompt (cross-context reuse), with additional batch-scalable reuse scheduling, which reduces the peak GPU memory of the reused queries for higher-throughput serving. On three long- reasoning benchmarks, the proposed QUEEN consistently outperforms existing eviction methods. Especially on LiveCodeBench, it matches full-KV accuracy with 5.2× less KV memory and 4.1× higher throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Eviction, Long-Reasoning LLMs, Query Enrichment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zQXmOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Prior work which views KV cache eviction as a retrieval problem has primarily focused on two aspects:(1) the signals from the cached entries (i.e., keys and values) used to assess relevance, and (2) the ranking strategy that determines which KV pairs are retained. Unlike prior methods, this paper proposed Query Enrichment (QUEEN), a training-free method looking at the query used for scoring instead. QUEEN contains two major components: (1) Intra-window refinement re-feeds the window tokens immediately after the window in a single extra forward pass, so each refined query attends to the complete window context. (2) Cross-context reuse maintains a pool of past refined window queries, pruned per-layer by diversity selection, plus a retained question query anchoring retrieval to the global objective. Experimental results show QUEEN consistently outperforms existing eviction methods: evaluated on AIME 2025, MATH500, and LiveCodeBench across Qwen3-1.7B/8B/14B models against SnapKV, R-KV, BeaconKV, and TriAttention, QUEEN is the best method.

Critical strengths
  • The paper is well written and easy to follow.
  • The proposed method is novel and experimental results show it is effective. Prior work has optimized key/value signals and ranking strategies while treating the scoring query as fixed. Identifying the query as the under-explored lever — and diagnosing its two failure modes crisply — is a clear contribution.
  • The proposed method is training-free, calibration-free, architecture-agnostic.
  • The experimental result is strong and shows the effectiveness of the proposed method: QUEEN is the best at every budget × benchmark × model (Qwen3 plus R1-distilled Llama/Qwen).
  • The paper is well ablated: oracle-KV recall under prefill compression, critical-KV retention across all decoding positions/layers/heads, FIFO-vs-diversity comparison, the without-scheduling ablation, latency breakdown, test hyperparameter sensitivity, and so on.
Critical weaknesses
  • Every method compresses every 256 tokens except BeaconKV, which uses its own ratio-based schedule.
  • Cross-context reuse is only evaluated on reasoning traces, where its core assumption is safest since reasoning naturally revisits earlier steps. The only non-reasoning evidence covers refinement alone (LongBench, prefill compression).
Questions for authors
  • Can you report BeaconKV under the same 256-token compression interval as the other baselines?
  • Does full QUEEN (with reuse) improve over observation-window scoring on non-reasoning long-context tasks?
  • Minor: several "Sec. ??" cross-references need fixing.
3 comments
AuthorsOpenReview ↗

Author Response #1 to Reviewer zQXm

We sincerely thank the reviewer for the positive and constructive evaluation of our work. We greatly appreciate the recognition of our clear writing, the diagnosis of the two failure modes of the scoring query, and our training-free method with strong results and extensive ablations. We address the reviewer's questions in detail below. For clarity, we mark responses to weakness comments with W# and responses to explicit questions with Q#.


W1, Q1

BeaconKV compression interval. Every method compresses every 256 tokens except BeaconKV, which uses its own ratio-based schedule. Can you report BeaconKV under the same 256-token compression interval?

→\to Good suggestion! As requested, we re-evaluated BeaconKV [1] under the same 256-token compression interval as the other methods, as shown in Table A. QueEn still leads in all 8 conditions after matching the interval, as under the original schedule.

We have added these results to App. G of the revised manuscript.

Same 256-token compression interval.

Table A: Accuracy of BeaconKV at the same 256-token interval and of QueEn (Qwen3-8B)

ConditionBeaconKV originalBeaconKV interval 256DifferenceQueEn
MATH500 B=6410.2%22.2%+12.062.6%
MATH500 B=12832.8%39.4%+6.671.6%
MATH500 B=25649.6%52.8%+3.277.4%
MATH500 B=51263.8%69.6%+5.883.2%
LCB B=51222.9%24.0%+1.138.0%
LCB B=102432.6%31.5%-1.144.4%
LCB B=204841.2%43.7%+2.547.7%
LCB B=307248.0%47.7%-0.449.8%

(Updated to match the MATH500 values of Table 26 in the revision.)

In Table A, matching the interval helps BeaconKV most at the smallest MATH500 budgets (+12.0 and +6.6pt at B=64 and 128), while LiveCodeBench changes little (-1.1 to +2.5pt). This is because, at these two budgets, BeaconKV's original schedule compresses about every 9 and 18 tokens. A just-generated token can thus be evicted within a few steps, whereas a 256-token interval keeps recent tokens until the next compression. Even so, QueEn still leads by 40.4 and 32.2pt.

[1] Kim, Janghyeon, et al. "BeaconKV: Key-value cache compression guided by beacon queries for efficient large reasoning model inference." ICML. 2026.


W2, Q2

Cross-context reuse on non-reasoning tasks. Cross-context reuse is only evaluated on reasoning traces. Does full QueEn (with reuse) improve over observation-window scoring on non-reasoning long-context tasks?

→\to Thank you for asking whether reuse helps beyond reasoning tasks. Not for long-context inputs; more precisely, QueEn's reuse does not apply when a long input is compressed once at prefill, as in LongBench. This is because reuse carries window queries from one compression to the next and thus takes effect only when compression repeats during long decoding. As the reviewer noted, only refinement applies in this setting. We briefly note this scope in App. H of the submission (App. J of the revision), where we leave reuse at prefill to future work, and we have made it clearer in Sec. 4.4 of the revision.


Minor

Broken cross-references. Several "Sec. ??" cross-references need fixing.

→\to Thank you! We have fixed all broken cross-references in the revised manuscript.

Thank you again for your positive assessment and helpful suggestions. We would be glad to answer any further questions.

Thank the authors for the new experiments and detailed response. I would keep my rating at 4.

AuthorsOpenReview ↗

Acknowledgment to Reviewer zQXm

We sincerely thank the reviewer for the positive assessment and constructive feedback. The reviewer's questions on the compression interval of BeaconKV and on applying cross-context reuse beyond long reasoning made our comparison fairer and the scope of QueEn clearer.

Best regards,
Authors of QueEn

Reviewer_Amj4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QueEn to improve the queries used for KV cache eviction during long reasoning. It re-encodes the recent window to obtain refined queries and maintains diverse historical-window queries together with queries from the original question. CPU offloading and tiled scoring reduce the additional GPU memory overhead.

Critical strengths
  1. The paper addresses a practically important problem. Eviction based only on recent queries can discard historical information and original task constraints needed later. Analyses of critical-KV retention and function-name reproduction provide concrete evidence supporting this motivation.

  2. The reported accuracy gains are substantial. QueEn performs well across multiple models and mathematical and coding benchmarks, particularly under tight KV budgets. Ablations also demonstrate meaningful benefits from combining query refinement with historical reuse.

Critical weaknesses
  1. The novelty is limited by substantial overlap with prior work. Repeated encoding for improved representations has precedents such as Echo Embeddings; DapQ already appends tokens and extracts pseudo queries to guide KV eviction; and BeaconKV already scores cached entries using representative historical queries. CPU offloading and tiled maximum aggregation are also relatively straightforward execution optimizations. The contribution therefore appears primarily to be a combination of existing mechanisms and specific retention heuristics, requiring clearer justification of its substantive advance.

  2. The comparisons do not sufficiently isolate the contributions of the proposed designs. QueEn uses substantially more scoring queries than BeaconKV and employs different storage and scoring implementations. Its gains may therefore reflect both broader historical coverage and execution optimizations. Although the ablations support combining refinement and reuse, comparisons with matched query counts and identical offloading and tiling implementations, alongside a direct comparison with DapQ-style query construction, are needed to establish the sources of its accuracy and efficiency advantages.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Author Response #1/3 to Reviewer Amj4

We sincerely thank the reviewer for the constructive evaluation of our work. We greatly appreciate the recognition of the practical importance of our motivation, the analyses supporting it, and our accuracy gains under tight KV budgets. We address the reviewer's questions and concerns in detail below. For clarity, we mark responses to weakness comments with W#.


W1

The novelty is limited by substantial overlap with prior work. The contribution appears primarily to be a combination, requiring clearer justification of its substantive advance.

→\to While the individual components of QueEn have precedents, we argue that QueEn's novelty lies in selectively reusing refined scoring queries across compression cycles to preserve globally relevant reasoning context throughout long-horizon reasoning, rather than in the individual components themselves.

Importantly, query refinement and cross-context reuse are not merely combined; they reinforce each other to address limitations that neither can fully overcome alone. Refinement incorporates the complete window context into query representations, but alone cannot address recency-biased scoring across compression cycles. Conversely, reuse extends query coverage beyond the recent window, but without refinement, it repeatedly relies on queries that lack the complete window context due to causal masking. Thus, we argue that by reusing refined queries across compression cycles, QueEn exploits the complementary strengths of both mechanisms. As shown in Table 1 of the paper, combining them achieves 71.6%/44.4% accuracy, compared with 11.0%/22.9% using refinement alone and 62.4%/34.8% using reuse alone (MATH500/LCB). Notably, adding refinement to reuse yields +9.2/+9.6 points, exceeding its standalone gains (+0.6/+1.8 points), resulting in superadditive gains of +8.6/+7.8 points. These results provide empirical evidence that refinement and reuse interact synergistically rather than merely contributing independently.

Beyond these superadditive gains, QueEn's reuse of refined queries from past contexts introduces substantial peak GPU memory overhead, limiting batch scalability. To address this challenge, which has not been investigated in prior KV cache eviction work, we propose batch-scalable reuse scheduling, increasing the maximum serving batch size from 10 to 46 (Table 4). Thus, QueEn's substantive advance lies in enabling both effective and scalable query reuse for long-horizon reasoning, rather than in individual retention heuristics, whose contributions are separately validated through ablations (Fig. 8, Table 2, App. D).

AuthorsOpenReview ↗

Author Response #2/3 to Reviewer Amj4

W1 (continued)

(Limited novelty) Repeated encoding for improved representations has precedents such as Echo Embeddings.

→\to While repeated encoding itself has precedents, acknowledged in Sec. 2 of the manuscript, we note that QueEn differs from existing Echo Embeddings [1] in three ways. (1) Echo Embeddings pools the last-layer hidden states of the repeated input into a single text embedding, whereas QueEn refines per-layer, per-head query vectors that score each layer's KV pairs. (2) Echo Embeddings encodes a given input once, whereas QueEn refines the generated window at each compression during long decoding. (3) Echo Embeddings uses only the current input, whereas QueEn reuses the refined queries of earlier windows at later compressions. Most of the effect arises from this reuse: refinement raises accuracy by only +0.6/+1.8pt alone but by +9.2/+9.6pt on top of reuse (MATH500/LCB, Table 1 of the paper).


(Limited novelty) DapQ already appends tokens and extracts pseudo queries to guide KV eviction.

→\to DapQ [2] also scores KV pairs with queries obtained by appending tokens (pseudo queries), but QueEn differs from DapQ in two ways. (1) DapQ finds that position matters more than content for its pseudo queries, whereas for QueEn content is decisive: replacing the tokens repeated for refinement with random tokens at the same positions, with everything else unchanged, lowers accuracy from 46.6% to 4.8% (Qwen3-1.7B, MATH500, B=64). (2) DapQ compresses once at prefill, whereas QueEn compresses repeatedly during long reasoning. We have cited DapQ in Sec. 2 and discussed these differences in App. H.5 of the revision.


(Limited novelty) BeaconKV already scores cached entries using representative historical queries.

→\to BeaconKV [3] also scores KV pairs with representative historical queries (beacon queries), a relationship App. G of the paper discusses, but QueEn differs from BeaconKV in two ways. (1) BeaconKV keeps the queries of individual tokens, whereas QueEn keeps refined window queries for the local contexts of earlier reasoning steps and question queries for the global reasoning objective. Because this difference lies in how the queries are composed rather than in how many there are, simply adding beacon queries does not close the accuracy gap: even with 512 beacon queries, more than QueEn's median of 317, BeaconKV reaches 32.0%, half of QueEn's 64.4% (Table A). (2) BeaconKV keeps its beacon queries on the GPU, whereas QueEn keeps its query pool in host memory and scores it in tiles (batch-scalable reuse scheduling). With BeaconKV's original implementation, raising the beacon queries from 16 to 512 thus drops its maximum batch from 52 to 25 and its throughput from 354.8 to 157.7 tok/s, while QueEn serves a batch of 46 at 523.7 tok/s with a median of 317 queries (Table B).

Table A: Increasing BeaconKV's beacon queries (Qwen3-8B, interval 256)

BeaconKV beacon queries16 (official)256512QueEn
Accuracy (MATH500, B=64)22.2%29.6%32.0%64.4%

Table B: Serving performance of BeaconKV with more beacon queries (Qwen3-8B, budget 3072, original implementation, same query-selection schedule as QueEn)

BeaconKV beacon queries16256512QueEn
Max batch52342546
Throughput (tok/s)354.8250.2157.7523.7

(Limited novelty) CPU offloading and tiled maximum aggregation are also relatively straightforward execution optimizations.

→\to While CPU offloading and tiling are not new execution optimizations in themselves, they resolve a bottleneck that query reuse creates. Reuse raises the scoring queries to hundreds, so the query pool and the score matrix grow with the batch size, and the peak GPU memory limits the maximum batch to 10, smaller even than uncompressed Full-KV (13). Batch-scalable reuse scheduling keeps the pool on the CPU and scores over query tiles, raising the maximum batch to 46 and throughput to 4.3× that of Full-KV (Table 4 of the paper). QueEn's contribution here is thus not the optimizations themselves but making query reuse scalable in serving.

[1] Springer, Jacob Mitchell, et al. "Repetition improves language model embeddings." ICLR. 2025.
[2] Tian, Zhenxu, et al. "Where matters more than what: Decoding-aligned KV cache compression via position-aware pseudo queries." arXiv preprint arXiv:2603.11564 (2026).
[3] Kim, Janghyeon, et al. "BeaconKV: Key-value cache compression guided by beacon queries for efficient large reasoning model inference." ICML. 2026.

AuthorsOpenReview ↗

Author Response #3/3 to Reviewer Amj4

W2

Isolating the sources of the gain. QueEn's gains may reflect both broader historical coverage and execution optimizations. Comparisons with matched query counts and identical offloading and tiling implementations, alongside a direct comparison with DapQ-style query construction, are needed.

→\to Thank you for your suggestion. We conducted all three requested comparisons below.

(1) Matched query counts. With the beacon queries of BeaconKV [1] (16 by default) increased to 256 and 512, around QueEn's median of 317, accuracy rises from 22.2% to 29.6% and 32.0%, while QueEn reaches 64.4% (Table C). More beacon queries alone therefore do not reproduce QueEn's gain.

(2) Identical offloading and tiling implementations. We applied QueEn's implementation optimizations to BeaconKV with 256 and 512 beacon queries and measured the maximum batch and throughput.

Table C: Accuracy and serving performance (Qwen3-8B; accuracy on MATH500 at B=64, serving performance at budget 3072 with 16,384 generated tokens)

MethodAccuracyMax batchThroughput (max batch)
QueEn64.4%46523.7
BeaconKV default (16 queries)22.2%56367.8
BeaconKV 256 queries + optimization29.6%65438.1
BeaconKV 512 queries + optimization32.0%61296.6

With all implementations matched, BeaconKV fits larger batches (61-65 vs. 46). This is because its temporary memory during compression is about 0.04 GB per sequence, smaller than QueEn's (about 0.17 GB). At matched query counts, BeaconKV has lower throughput because, by its algorithm design, it reselects its beacon queries every 16 steps, and this reselection adds an overhead that grows with the number of beacon queries. Overall, QueEn leads in both accuracy and throughput, even when BeaconKV uses more beacon queries and the same implementation optimizations.

(3) DapQ-style query construction. DapQ [2] treats the content of its pseudo queries as secondary to their positions, showing that even random tokens work. Following this, we replaced the tokens used for refinement with random tokens at the same positions, with everything else unchanged, and accuracy fell from 46.6% to 4.8% (Qwen3-1.7B, MATH500, B=64; see our response on DapQ above), so DapQ-style query construction does not reproduce QueEn's gain either.

We have added these results (Tables A, B, and C) to App. G and H.5 of the revision.

[1] Kim, Janghyeon, et al. "BeaconKV: Key-value cache compression guided by beacon queries for efficient large reasoning model inference." ICML. 2026.
[2] Tian, Zhenxu, et al. "Where matters more than what: Decoding-aligned KV cache compression via position-aware pseudo queries." arXiv preprint arXiv:2603.11564 (2026).

We hope our responses clarify how QueEn differs from prior work and where its gains come from. We would be happy to address any remaining concerns.

AuthorsOpenReview ↗

Gentle Reminder Before the Discussion Ends

Dear Reviewer Amj4,

We thank the reviewer again for the time and care devoted to reviewing our paper. As the discussion period ends soon, we would like to briefly summarize how our responses address the two concerns raised.

  • Novelty beyond a combination (W1): QueEn's novelty lies in identifying two limitations of the observation-window query, restricted query encoding under causal masking and a retrieval biased toward recent context, and in reusing refined scoring queries across compression cycles to address these limitations. Refinement and reuse reinforce each other, with superadditive gains of +8.6pt on MATH500 and +7.8pt on LiveCodeBench (Table 1), and batch-scalable reuse scheduling makes this reuse scalable by raising the maximum batch from 10 to 46 (Table 4).
  • Isolating the sources of QueEn's accuracy and efficiency gains (W2): With matched query counts, BeaconKV reaches 32.0% vs. 64.4% for QueEn, and with the same implementation optimizations, QueEn still leads in throughput (App. G). With DapQ-style query construction, accuracy falls from 46.6% to 4.8% (App. H.5).

We would be sincerely grateful if you could let us know whether our responses have addressed your concerns, and we would be glad to answer any remaining questions before the discussion period ends.

Best regards,
Authors of QueEn

Reviewer_8DqqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes QUEEN, a KV-cache eviction during long reasoning. Every 256 generated tokens, it repeats the last 32 tokens to obtain refined queries, stores selected queries from previous windows together with queries from the original question, and uses this pool to score the currently cached keys before eviction. It also keeps the historical-query pool in CPU memory and scores in tiles to reduce peak GPU memory.

I find the empirical results interesting, especially at very small KV budgets. My main concern, however, is with the explanation of why the method works. I am not convinced that the central design choice—refining and reusing old queries to score the current cache—follows from the problem the authors identify. The experiments show that QUEEN preserves more useful information than recent-window scoring, but they do not sufficiently rule out simpler explanations or establish that historical-query reuse is the right mechanism.

Critical strengths
  • The reported accuracy improvements are substantial across AIME 2025, MATH500, and LiveCodeBench, several KV budgets, and multiple model sizes.

  • The paper includes meaningful component ablations, a full-KV-based analysis of retained keys, a concrete function-name example, and a breakdown of memory and compression costs. These are useful; my concerns are about what the evidence establishes, not about an absence of experiments.

  • Moving the query pool to host memory and tiling the scoring computation addresses a real implementation bottleneck. This may be a useful engineering contribution, subject to fair comparisons with similarly optimized baselines.

Critical weaknesses

My main concerns:

C1. The motivation does not clearly justify reusing historical queries

The authors' starting observation makes sense: a KV pair that mattered in an earlier reasoning step may matter again later. If eviction decisions use only a recent observation window, an older, still-useful KV pair can be permanently removed.

But why does this imply that we should store old queries and repeatedly rescore the current cache with them? If a key was important to the model during an earlier step, a more direct response is to remember that importance. At each compression cycle, we could keep historical importance scores for each surviving KV pair and combine them with scores from the current window. An older key would then be protected because it was genuinely useful before, rather than because an old query gives it a high score in a new, artificial attention computation. The policy could use a maximum, an accumulated score, or a decayed score so that old importance does not dominate forever. It would still have to obey the same KV budget.

This is not a new or unreasonable line of thinking: H2O and Scissorhands study historical attention and persistence of importance. Both appear in QUEEN's bibliography, but the paper does not seriously discuss this alternative or compare against a matched historical-score-accumulation baseline. Comparisons against a recent-window-only policy do not answer this question.

I understand one difference: saved historical scores exist only for keys available when they were recorded, while a saved query can also score keys generated later. But that brings me to the next, more fundamental concern.

C2. Why should an old query evaluate a key generated after it?

Under standard causal decoding, a query at position i attends only to keys at positions up to i. If key k_j is generated at a later position j > i, the original query q_i never attends to it. QUEEN nevertheless keeps a refined version of q_i, repositions it using RoPE at a later compression step (Appendix A), and computes an attention score against k_j.

This is not historical attention that the LLM actually used. It is a hypothetical attention interaction. The model is trained to use the query generated from the current hidden state to attend to preceding keys; it is not specifically trained to take a query representing an earlier reasoning state, move it to a later position, and use it to judge newly generated keys. Changing its RoPE position does not update the content of that old query to reflect the newer reasoning state.

There may be a useful heuristic here: perhaps future queries are often similar to certain past queries, so the old ones act as proxies. I am not claiming that this cannot work. But why should these hypothetical scores predict the attention of actual future causal queries? That is the key assumption I want the authors to test. The argument that old keys might be useful again is not enough to justify using old queries against future-generated keys.

One simple experiment would be to allow each retained historical query to score only keys that already existed when that query was collected. If QUEEN loses little accuracy, then the supposed benefit of evaluating newer keys with older queries may not be important at all. If accuracy drops substantially, the authors should show that those new-query–old-state interactions really predict future causal attention, rather than only correlating with a different retention bias.

I believe the scope of this paper is narrower than what is explained in the abstract and introduction. See C3 and C4:

C3. Query refinement is not the same thing as predicting future information needs

The abstract and introduction motivate the method by the need to retain keys that will be relevant in future decoding. But the proposed method does not explicitly estimate future query vectors or future reasoning states. It repeats an already observed window, refines those queries using the now-completed window context, and reuses them later. That can make the query representations more informative about past and present context; it is not, by itself, a mechanism for predicting future causal attention.

I do not think predicting future queries is the only possible way to estimate future key importance. A proxy could work. But the paper needs to make that connection explicit and test it, rather than treating richer or more diverse historical queries as sufficient evidence that future relevance is better predicted. Refinement also produces queries different from those used by the model in ordinary causal decoding. This is fine as an auxiliary heuristic, but it makes the missing justification more important, not less.

The paper itself distinguishes QUEEN from BeaconKV's future-query motivation in Appendix G, and Appendix H calls predicting future queries a future direction. So I would not say the authors explicitly claim to generate future queries. My concern is more precise: the broad motivation about future relevance is not adequately connected to the actual past/current-context scoring mechanism.

C4. The scope is narrower than the general KV-retrieval motivation suggests

The failure mode motivating QUEEN is specific to irreversible KV eviction. Once an older KV pair has been discarded, even a future query that needs it cannot recover it. This is a real issue for eviction methods, including SnapKV, R-KV, H2O, Scissorhands, and BeaconKV.

Retrieval-based sparse attention has a different problem. Logically, at decoding step t, the model uses its current causal query to search over the retained full history of keys and select a subset for attention. The key/value history remains available, potentially in CPU memory or an index. If something from a much earlier reasoning step becomes important again, the current query can retrieve it then; there is no need to predict its importance several steps in advance or permanently keep it in a smaller evicted cache. Examples include Quest, SparQ Attention, RetrievalAttention, and Louver.

Of course retrieval can miss relevant keys when its pruning or approximate search is imperfect, and even exact top-k selection changes attention by dropping the remaining keys. That is a selection-quality issue, not the permanent-history-loss issue that QUEEN is designed around.

To be fair, Section 2 already acknowledges that retrieval-based methods avoid irreversible eviction, and the paper explicitly presents QUEEN as an eviction method. I am not claiming the authors say they solve retrieval-based sparse attention. My concern is that the broader introduction and query-as-retrieval framing should emphasize that the proposed solution is specifically for permanent eviction and does not address a general deficiency of query-based retrieval. Their characterization of retrieval mainly as CPU offloading also does not cover the full range of retrieval methods.

C5. The experiments show what helps, but not convincingly why it helps

I appreciate that the authors try to analyze the method, particularly in Section 4.4 and Appendices D and E. The results should be acknowledged accurately:

  • Table 1: Query reuse is responsible for most of the gain. On MATH500, removing reuse drops accuracy from 71.6 to 11.0 with refinement kept, whereas removing refinement but keeping reuse gives 62.4. This shows that reusing information is important; it does not prove that saving query vectors is better than saving historical scores.

  • Figure 5 and Appendix E.1: Refinement improves oracle-KV recall in prefill compression against ordinary window queries. This is a relevant positive result, but not a test of old queries predicting later decoding-time attention.

  • Figure 6 and Appendix E.2: QUEEN retains important keys across earlier decoding positions, rather than mainly recent positions. Importantly, the authors use full-KV attention and replay eviction with matched candidate keys; this is a useful controlled analysis. Still, it compares historical-query reuse with recent-window scoring, not with historical-score accumulation or simple time-stratified retention. The reported critical-key measure also aggregates importance over the full trace; it does not directly measure how well a scoring decision at time t predicts attention after t.

  • Figure 7: Reusing question queries retains function-name information in the prompt and increases correct function-name reproduction from 37% to 68%. This is persuasive evidence that protecting the original problem statement matters. It is not yet evidence that refined question queries are needed: reserving KV slots for prompt tokens might achieve the same result.

  • Figure 8 and Appendix D: Diversity selection beats a FIFO query pool (+7.1 points on AIME, +1.2 on MATH500, +1.7 on LiveCodeBench). This suggests benefit from covering earlier parts of the trace, but does not establish whether the specific historical query vectors are the important ingredient.

  • Table 2 and Appendix D: Restricting each question query to at most one selected KV pair improves accuracy. This could also reflect a useful cache-budget allocation rule, independent of the broader query-reuse hypothesis.

  • Figure 9: QUEEN reduces decoding-length inflation. The authors conjecture that it avoids having to reconstruct lost intermediate information. That is plausible, but not directly established by the figure.

Taken together, I can believe the method improves accuracy while remaining unconvinced by the stated explanation. Several other mechanisms could explain the same observations: keeping KV pairs from different stages of reasoning, preventing excessive recency bias, protecting the original prompt, preserving historical heavy hitters, or simply spreading the limited budget more evenly across positions and heads.

I would like to see at least three simpler controls: (i) accumulated historical attention scores (H2O/Scissorhands-style) with the same budget and update interval; (ii) explicit prompt protection plus ordinary eviction elsewhere; and (iii) a position-stratified or age-balanced retention policy. A fourth useful control is the historical-query restriction described above, which would isolate the value of scoring future-generated keys with old queries. All of these should use identical compression schedules and, as far as possible, the same scoring and implementation optimizations.

Most importantly, use a full-KV reference trace and ask, at each compression step, which policy retains the keys that the actual subsequent causal queries will attend to? Measure future-attention recall over several horizons (for example, the next 256, 512, and 1,024 tokens) and relate it to downstream accuracy. If QUEEN wins that comparison, it would directly support the proposed mechanism. If another simple policy does as well, that would be an equally interesting scientific finding.

C6. Code availability and the fairness of the efficiency results

The reproducibility statement (p. 10) says the authors will release code upon publication. I could not inspect an implementation from the material supplied with this review. ICLR does not require code at submission, so I do not treat its absence as proof of an incorrect result. But it limits my ability to verify fairly complicated parts of the method: repeated-window forwarding without modifying the original cache, RoPE reapplication for stored queries, grouped-query attention and per-head cache layouts, the single-KV restriction, variable-length cache rebuilding, and tiled scoring.

It also matters for the efficiency claims. Appendix F.3 attributes much of QUEEN's larger supported batch size relative to TriAttention to temporary compression memory: 0.165 GB per sequence for QUEEN versus 1.021 GB for TriAttention beyond the common KV-cache cost. QUEEN avoids materializing the full query-by-key score matrix by tiling. Could the same general memory-efficient optimization be used for TriAttention? If so, how much of the reported batch-size and throughput advantage is from QUEEN's query-selection idea, and how much is from a better-optimized scoring implementation? The authors do give a same-batch comparison in Appendix F.2 (360.9 versus 234.9 tokens/s at batch 21), which is useful, but it does not settle the matched-implementation question.

Appendix G also finds substantial continual query-selection overhead in BeaconKV's released implementation. This makes me want to know how performance changes under more comparable query-maintenance schedules or optimized implementations. I am not asking the authors to rewrite every baseline, but the paper should separate algorithmic improvements from implementation-level advantages as clearly as possible.

Finally, the historical-query pool is not free. Appendix F.1 reports 151 MB of pinned CPU memory per sequence for a 512-query pool, compared with 453 MB of KV cache per sequence at the stated budget. This may be reasonable, but the memory and throughput discussion should account for total CPU-plus-GPU memory, host transfers, and longer prompts, rather than concentrating almost entirely on peak GPU memory. More realistic input lengths would be helpful.

Minor issue

  • Sec ?? in line 500.

References

  • H2O — Zhang et al., H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023. arXiv

  • Scissorhands — Liu et al., Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time, NeurIPS 2023. arXiv

  • StreamingLLM — Xiao et al., Efficient Streaming Language Models with Attention Sinks, ICLR 2024. arXiv

  • SnapKV — Li et al., SnapKV: LLM Knows What You are Looking for Before Generation, NeurIPS 2024. arXiv

  • R-KV — Cai et al., R-KV: Redundancy-aware KV Cache Compression for Training-Free Reasoning Models Acceleration, NeurIPS 2025. arXiv

  • BeaconKV — Kim et al., BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference, ICML 2026. arXiv

  • Quest — Tang et al., Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML 2024. arXiv

  • SparQ Attention — Ribar et al., SparQ Attention: Bandwidth-Efficient LLM Inference, ICML 2024. arXiv

  • RetrievalAttention — Liu et al., RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval, NeurIPS 2025. arXiv

  • Louver — Dehghankar and Asudeh, Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache, NeurIPS 2026. arXiv

Questions for authors

Q1. Why should reusing a query from an earlier reasoning step help predict which KV pairs the model will actually attend to in future steps? Can the authors directly test whether this assumption holds?

Q2. Why save and rescore historical queries instead of accumulating the importance scores of surviving keys observed during earlier causal decoding? Please compare against a matched historical-score baseline.

Q3. How much of the gain remains if historical queries are prevented from scoring keys generated after the queries were first collected?

Q4. How much of the gain is explained by simpler policies that protect prompt tokens or allocate a fixed share of the cache across different decoding periods?

Q5. Can the authors provide anonymized code or enough executable implementation detail to verify query refinement, RoPE positions, scoring, and cache rebuilding? Can they report comparisons where relevant baselines have comparable memory-efficient scoring implementations?

Q6. Can they report future-attention recall against full-KV decoding at each compression cycle, plus end-to-end CPU/GPU memory and throughput with longer prompts?

12 comments
AuthorsOpenReview ↗

Author Response #1/7 to Reviewer 8Dqq

We sincerely thank the reviewer for the thoughtful and constructive evaluation of our work. We greatly appreciate the recognition of our accuracy gains across benchmarks, budgets, and model sizes, our ablations and analyses, and the engineering contribution of offloading and tiled scoring. We address the reviewer's questions and concerns in detail below. For clarity, we mark responses to concerns with C# and responses to explicit questions with Q#, following the numbering in the review.


C1, Q2

Why store and rescore historical queries instead of remembering historical scores? If a key was important during an earlier step, a more direct response is to remember that importance. Please compare against a matched historical-score baseline.

→\to Thank you for suggesting this alternative. We store queries because, as the reviewer notes, a score can be recorded only for keys that exist at that time, so keys generated later never receive scores from earlier steps, whereas stored queries can also score those keys. Within irreversible eviction, QueEn views choosing which KV pairs to keep as retrieval over the KV cache as a corpus: it selects the most important KV pairs in the current cache to keep, assuming that they may be needed again in later decoding, and uses the scoring queries like the queries of a web search engine. A search query can search the entire current corpus no matter when it was formed, so QueEn stores the queries of earlier reasoning steps and of the question and rescores the entire current cache with them at every compression to identify the important KV pairs. This scoring is importance estimation for choosing which KV pairs to keep, not decoding that produces the next token, so a query need not be limited to the keys preceding it, as in causal decoding. Whether this design actually keeps keys that are used in the future is tested in our response to C2.

Matched historical-score baseline. As the reviewer suggested, we compared accuracy against score-memory policies (QueEn-ScoreMem, H2O [1], Scissorhands [2]) under the same budget and the same compression interval (256 tokens) (Table A; Qwen3-1.7B, MATH500, per-head KV budget B). In particular, QueEn-ScoreMem is a variant of QueEn that remembers scores instead of storing queries.

Table A: Accuracy of QueEn and the score-memory policies

PolicyB=64B=256
QueEn46.6%68.6%
QueEn-ScoreMem21.8%42.2%
H2O18.8%46.8%
Scissorhands2.4%17.8%

The accuracy of QueEn-ScoreMem is lower than that of QueEn by 24.8pt at B=64 and by 26.4pt at B=256, and H2O and Scissorhands also fall far short of QueEn. The gain therefore comes from storing queries and rescoring the cache with them at every compression rather than from remembering scores.

[1] Zhang, Zhenyu, et al. "H2O: Heavy-hitter oracle for efficient generative inference of large language models." NeurIPS. 2023.
[2] Liu, Zichang, et al. "Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time." NeurIPS. 2023.

AuthorsOpenReview ↗

Author Response #2/7 to Reviewer 8Dqq

C2, Q1

Why should an old query evaluate a key generated after it? Under causal decoding, an old query never attends to keys generated after it, so its scores for them are hypothetical. Why should they predict future causal attention, and can the authors directly test this assumption?

→\to Thank you for raising this key question. The causal mask is a constraint on generating the next token, whereas QueEn's scoring is a search for which KV pairs to keep (response to C1). Just as a search engine measures relevance regardless of when a document was created, QueEn's design is motivated by the idea that old queries can also evaluate the importance of keys generated after them. QueEn does not predict future attention directly but aims to retain the most important KV pairs among the given ones using multiple queries. Whether this design actually keeps keys that are used in the future needs to be tested, so we applied the reviewer's historical-query restriction to QueEn (QueEn-Restricted; the next two responses).


C2, Q3

Historical-query restriction. How much of the gain remains if each retained historical query may score only keys that already existed when that query was collected?

→\to Thank you for suggesting this useful experiment. As the reviewer suggested, we restricted each historical query to scoring only earlier keys (QueEn-Restricted).

Table B: QueEn with the historical-query restriction (Qwen3-1.7B, MATH500, B=64)

Future-attention recallAccuracy
QueEn66.6%46.6%
QueEn-Restricted55.4%25.0%

Future-attention recall: the fraction of future critical keys that a compression retains, where future critical keys are the 16 keys per head receiving the largest attention over the next 1,024 tokens of the uncompressed generation.

Under the restriction, accuracy drops by nearly half, from 46.6% to 25.0%, and future-attention recall drops from 66.6% to 55.4%. This indicates that letting old queries score later keys helps retain future critical keys and improves accuracy.


C2

Old-query scoring of new keys: future attention or a retention bias? If accuracy drops under the restriction, the authors should show that those new-query–old-state interactions really predict future causal attention, rather than only correlating with a different retention bias.

→\to Thank you for pointing out this possibility. As noted in our response to C2, Q1, QueEn does not predict future attention but aims to retain the most important keys among the given keys using multiple queries, so here we check whether the benefit of letting old queries score new keys merely reflects a retention bias.

Does QueEn merely retain more recent keys? The most direct candidate is a recency bias, that is, the tendency of attention to be high on positionally recent keys regardless of their content [1]. In the original QueEn, old queries can also score new keys, so more new keys can be retained than in QueEn-Restricted. Our analysis shows that, among the recent keys generated in the 256 tokens before compression, QueEn retains 12.4% and QueEn-Restricted retains 7.1%. Given the recency bias, QueEn's accuracy advantage over QueEn-Restricted could simply result from retaining more recent keys, rather than from old queries identifying which recent keys are important.

Recent keys kept only by QueEn versus random selection. The number of retained recent keys shows only how many are retained, not whether the important ones are selected. We therefore collected the recent keys that QueEn keeps but QueEn-Restricted discards, and checked whether they were actually used later. We restricted all compared keys to the recent keys from the 256 tokens before compression, so that recency is controlled for. If old queries favored recent keys only because of their position, they would not identify which recent keys are important, and the share of future critical keys among the keys kept only by QueEn would be similar to that of random selection.

Table C: Share of future critical keys among recent keys (Qwen3-1.7B, MATH500, B=64)

SelectionNext 256 tokensNext 512 tokensNext 1,024 tokens
QueEn only16.1%15.5%14.9%
QueEn-Restricted only10.9%9.9%9.0%
Random selection4.3%4.0%3.7%

Random selection: the expected share of future critical keys when keys are selected at random from the recent keys.

Recency alone does not explain the gain. At the 1,024-token horizon, 14.9% of the recent keys kept only by QueEn were future critical keys, about 4× random selection (3.7%) and higher than those kept only by QueEn-Restricted (9.0%). The same trend holds at all three horizons. Old queries thus identify which recent keys are important, and QueEn's advantage is not explained by a recency bias alone.

[1] Hsieh, Cheng-Yu, et al. "Found in the middle: Calibrating positional attention bias improves long context utilization." Findings of ACL. 2024.

AuthorsOpenReview ↗

Author Response #3/7 to Reviewer 8Dqq

C3

Connecting the future-relevance motivation to QueEn's scoring. QueEn does not estimate future queries; it refines and reuses past and current queries. The paper needs to make that connection explicit and test it, rather than treating richer or more diverse queries as sufficient evidence.

→\to Thank you for this precise point. As the reviewer notes, QueEn does not estimate future queries. Instead, it uses richer and more diverse queries to retain the KV pairs important to the reasoning so far, namely to the question, earlier steps, and the recent context. This connects to future relevance if many of these KV pairs are needed again in later decoding. As the reviewer suggested, we tested this connection with future-attention recall below.

Refinement retains more future critical keys. To isolate the effect of refinement without reuse, we fixed the retention rule (remember each key's highest score and keep keys in that order) and changed only the scoring queries, from the original window queries to refined queries, comparing future-attention recall.

Table D: Effect of refinement (Qwen3-1.7B, MATH500, B=64)

HorizonOriginal window queriesRefined queriesDifference
256 tokens31.8%44.4%+12.6
512 tokens33.7%46.4%+12.7
1,024 tokens38.8%50.9%+12.1

Replacing only the queries with refined queries raises recall by about 12pt at all three horizons. Original window queries are formed without seeing the later part of the window because of the causal mask, whereas refined queries are formed after seeing the entire window, which likely explains the higher recall. As the reviewer notes, refined queries differ from those the model uses in causal decoding, but scoring with them helps retain more future critical keys than scoring with the original queries.

With reuse, QueEn retains far more long-range critical keys. Since reuse is designed to protect important old keys, we also measured long-range recall, restricted to keys generated more than 1,024 tokens before the compression.

Table E: Effect of combining refinement and reuse (Qwen3-1.7B, MATH500, B=64)

MetricPolicy256 tokens512 tokens1,024 tokens
Future-attention recallObservation window63.3%60.2%60.0%
QueEn64.4%64.7%66.6%
Long-range recallObservation window3.0%3.0%2.9%
QueEn22.4%25.0%26.5%

The observation window retains only about 3% of long-range critical keys, while QueEn retains 22-27%. The gap in future-attention recall also grows with the horizon (+1.1 → +6.6pt). QueEn's recall advantage is thus largest on long-range critical keys. We have stated the connection in the Introduction and added this test to App. H.3 of the revision.


C4

Scope specific to permanent eviction. The paper should emphasize that QueEn does not address a general deficiency of query-based retrieval, and its description of retrieval should cover more than CPU offloading.

→\to Thank you for clearly pointing out the scope. As the reviewer notes, Section 2 already states that retrieval-based methods avoid irreversible eviction, and the paper presents QueEn as an eviction method. We agree that the failure mode QueEn targets arises only in irreversible eviction, not as a general deficiency of query-based retrieval. We have made this distinction clearer in two places.

  • Introduction: We now state that an evicted KV pair cannot be recovered even if later decoding needs it.
  • Related Work: We have broadened the description of retrieval beyond CPU offloading: some methods keep the full KV cache and reduce only the amount read at each step (Quest [1], SparQ [2]) or index keys for range search (Louver [3]), while others offload the history to host memory (e.g., RetrievalAttention [4]), at the cost of KV memory and per-step host access, respectively. We also now state explicitly that QueEn targets permanent eviction, not retrieval-based methods, which can retrieve a missed KV pair later.

[1] Tang, Jiaming, et al. "Quest: Query-aware sparsity for efficient long-context LLM inference." ICML. 2024.
[2] Ribar, Luka, et al. "SparQ attention: Bandwidth-efficient LLM inference." ICML. 2024.
[3] Dehghankar, Mohsen, and Abolfazl Asudeh. "Sparse attention as a range searching problem: Towards an inference-efficient index for KV cache." NeurIPS. 2026.
[4] Liu, Di, et al. "RetrievalAttention: Accelerating long-context LLM inference via vector retrieval." NeurIPS. 2025.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers (Part 2 of 2)

Summarization of Rebuttal Comments and Revision

Reviewer 8Dqq

  • Matched historical-score baseline (C1, Q2)
    • Remembering scores instead of storing queries lowers accuracy from 46.6/68.6% to 21.8/42.2%; H2O and Scissorhands also fall far short.
    • Added App. H.1.
  • Historical-query restriction (C2, Q1, Q3)
    • Restricting old queries to earlier keys lowers accuracy from 46.6% to 25.0% and future-attention recall from 66.6% to 55.4%.
    • Added App. H.2.
  • Old-query scoring of new keys vs. a retention bias (C2)
    • The recent keys kept only by QueEn are future critical keys at about 4× the rate of random selection (14.9% vs. 3.7%).
    • Added App. H.2.
  • Connecting future relevance to the scoring (C3)
    • Refinement raises future-attention recall by about 12pt, and QueEn retains 22-27% of long-range critical keys vs. about 3% for the observation window.
    • Stated the connection in the Introduction and added App. H.3.
  • Scope specific to permanent eviction (C4)
    • Clarified the scope in the Introduction and Sec. 2, and broadened the description of retrieval (Quest, SparQ, Louver, RetrievalAttention).
  • Simpler controls and future-attention recall (C5, Q4, Q6)
    • All simpler policies are at least 15.8pt below QueEn, and QueEn has the highest future-attention recall at B=64.
    • Added App. H.4.
  • Single-KV restriction (C5)
    • Introduced Table 2 as an ablation of the budget-allocation rule (Sec. 4.3).
  • Code availability (C6, Q5)
    • Released anonymized code.
  • TriAttention and BeaconKV with the same optimizations (C6, Q5)
    • QueEn's optimizations can also improve the efficiency of other methods, while QueEn's accuracy advantage remains (64.4% vs. 22.2% for BeaconKV).
    • Added App. F.2, F.3, and G.
  • Total memory, host transfers, and longer prompts (C6, Q6)
    • GPU memory, not CPU memory or host transfers, limits serving; 2K-token prompts reduce the maximum batch from 46 to 19.
    • Added App. I and stated this as a limitation in App. J.
  • Broken cross-reference (Minor)
    • Fixed all broken cross-references.

Reviewer Amj4

  • Novelty and overlap with prior work (W1)
    • QueEn's novelty lies in reusing refined scoring queries across compression cycles: refinement and reuse are superadditive (+8.6/+7.8pt on MATH500/LCB), and batch-scalable reuse scheduling makes this reuse scalable (maximum batch from 10 to 46).
    • Clarified how QueEn differs from Echo Embeddings, DapQ, and BeaconKV.
    • Cited DapQ in Sec. 2 and added App. H.5.
  • Isolating the sources of the gain (W2)
    • With matched query counts, BeaconKV reaches 32.0% vs. 64.4% for QueEn; with matched query counts and identical implementations, QueEn still leads in throughput; DapQ-style queries reach 4.8% vs. 46.6%.
    • Added App. G and H.5.

Reviewer zQXm

  • BeaconKV compression interval (W1, Q1)
    • Under the same 256-token interval, QueEn still leads in all 8 conditions.
    • Added App. G.
  • Cross-context reuse on non-reasoning tasks (W2, Q2)
    • Reuse applies only when compression repeats during decoding, so it does not apply to one-time compression of long inputs.
    • Clarified in Sec. 4.4 and noted in App. J.
  • Broken cross-references (Minor)
    • Fixed all broken cross-references.

 

Additional Experiments

  • Mechanism (Reviewer 8Dqq): a matched historical-score baseline (QueEn-ScoreMem, H2O, Scissorhands), the historical-query restriction, future-attention recall against full-KV decoding, and simpler control policies (prompt protection, age-balanced retention).
  • Matched comparisons (Reviewers 8Dqq, Amj4, zQXm): TriAttention and BeaconKV with the same implementation optimizations, BeaconKV with matched query counts, DapQ-style query construction, BeaconKV under the same 256-token compression interval, and BeaconKV running FPS-based beacon selection once per compression instead of every 16 steps, with and without QueEn's implementation optimizations.
  • Efficiency (Reviewer 8Dqq): total CPU-plus-GPU memory, host transfers, and longer prompts up to 2K tokens.
  • Code (Reviewer 8Dqq): anonymized code.

We have incorporated all additional results and clarifications into the revision, with changes highlighted in blue.

 

We thank the reviewers again for their careful assessments and constructive feedback, which have strengthened both the analysis and the clarity of this work.

AuthorsOpenReview ↗

General Response to All Reviewers (Part 1 of 2)

Dear Reviewers, ACs, SACs and PCs,

We sincerely appreciate your time and effort in the review process, and we value the thoughtful and constructive feedback from all reviewers. Below, we summarize the strengths identified by each reviewer and the additional experiments we conducted in response to their concerns.


Strengths Recognized by Reviewers

Problem and Contributions

  • Reviewer zQXm: Noted that the proposed method is novel, and that identifying the query as the under-explored lever and crisply diagnosing its two failure modes is a clear contribution.
  • Reviewer Amj4: Noted the practical importance of the problem that eviction based only on recent queries can discard historical information and original task constraints needed later.

Method Design

  • Reviewer zQXm: Noted that the method is training-free, calibration-free, and architecture-agnostic.
  • Reviewer 8Dqq: Noted that moving the query pool to host memory and tiling the scoring computation addresses a real implementation bottleneck and may be a useful engineering contribution.

Empirical Performance

  • Reviewer 8Dqq: Noted substantial accuracy improvements across AIME 2025, MATH500, and LiveCodeBench, several KV budgets, and multiple model sizes.
  • Reviewer Amj4: Noted substantial accuracy gains across multiple models and mathematical and coding benchmarks, particularly under tight KV budgets.
  • Reviewer zQXm: Noted that QueEn is the best at every budget, benchmark, and model.

Analyses and Presentation

  • Reviewer 8Dqq: Noted the meaningful component ablations, full-KV-based analysis of retained keys, function-name example, and breakdown of memory and compression costs.
  • Reviewer Amj4: Noted that the critical-KV retention and function-name reproduction analyses provide concrete evidence for the motivation, and that the ablations demonstrate meaningful benefits from combining query refinement with historical reuse.
  • Reviewer zQXm: Noted that the paper is well ablated, well written, and easy to follow.
AuthorsOpenReview ↗

Gentle Reminder of Discussion Phase Nearing Completion

Dear Reviewers,

As the discussion phase comes to a close, we thank all reviewers for the time and care devoted to our paper. Your questions led to new analyses and clarifications, which are now part of the revision.

We would be glad to answer any remaining questions before the discussion period ends. For reviewers who have not yet had a chance to participate in the discussion, we would be sincerely grateful if you could let us know whether our responses have addressed your concerns.

To briefly reiterate, our submission QueEn:

  • Identifies two limitations of the observation-window query that guides KV cache eviction: restricted query encoding under causal masking and a retrieval biased toward recent context.
  • Introduces query enrichment, a training-free approach that refines window queries with the completed window context and reuses refined past-window and question queries across compression cycles.
  • Proposes batch-scalable reuse scheduling, which keeps the reused queries in host memory and scores them in tiles, enabling high-throughput serving with a large query pool.

We believe these contributions will help guide future research on KV cache eviction for long reasoning, while delivering practical gains in both accuracy and serving efficiency.

Thank you again for your time and effort in reviewing our paper.

Best regards,
Authors of QueEn

AuthorsOpenReview ↗

Summary of the Discussion

Dear ACs, SACs and PCs,

We thank you and all reviewers for the time and effort devoted to our paper. As the discussion period ends, we briefly summarize the discussion with each reviewer. All changes are highlighted in blue in the revised manuscript.

Reviewer 8Dqq (raised the score from weak rejection to weak acceptance after rebuttal)
→\to The reviewer acknowledged that our responses resolved key concerns and that the additional analyses improved the quality of the paper.

  • Why query reuse (C1-C3, C5): We explained the motivation for query reuse and demonstrated why it works through the additional experiments the reviewer proposed (App. H).
  • Scope (C4): We clarified the scope of QueEn throughout the paper (Introduction, Sec. 2, and Conclusion).

Reviewer zQXm (reaffirmed clear acceptance after rebuttal)
→\to The reviewer noted that "the proposed method is novel and experimental results show it is effective," and we addressed the reviewer's two concerns.

  • Matched comparison (W1): We reported that QueEn outperforms BeaconKV [1] in all 8 conditions under the requested setting (up to 2.8×; App. G).
  • Scope (W2): We clarified the scope of query reuse (Sec. 4.4, App. J).

Reviewer Amj4 (awaiting response)
→\to During the rebuttal, we addressed the reviewer's concerns by (1) clarifying QueEn's novelty over prior work and (2) showing through the requested controlled comparisons that this novelty yields gains unattainable by incremental changes to existing methods.

  • Novelty (W1):
    • Synergy: QueEn's components not only overcome their targeted limitations of the query but also reinforce each other superadditively (up to 15× synergy; Table 1), as Reviewer Amj4 also noted their "meaningful benefits."
    • Source of the synergy: It comes from enriching the scoring query itself through QueEn's components (query refinement and reuse). Prior work treated this query "as fixed," and Reviewer zQXm considered QueEn "novel" for this reason.
    • Serving: Batch-scalable reuse scheduling overcomes the serving bottleneck of query reuse, making QueEn practical (4.6× larger batch; Table 4), which Reviewer 8Dqq noted as a "useful engineering contribution."
  • Controlled comparisons (W2):
    • Substantial margins over matched baselines: Under the settings the reviewer requested, QueEn achieves 2.0× accuracy and 1.2-1.8× throughput over BeaconKV [1] (App. G), and 9.7× accuracy over DapQ-style query construction [2] (App. H.5).
    • Implication: QueEn's gains come from its distinctive query design and cannot be obtained through simple incremental changes to existing methods.

We believe the discussion has strengthened both the analysis and the clarity of the paper. We sincerely appreciate your time and consideration.

Best regards,
Authors of QueEn

[1] Kim, Janghyeon, et al. "BeaconKV: Key-value cache compression guided by beacon queries for efficient large reasoning model inference." ICML. 2026.
[2] Tian, Zhenxu, et al. "Where matters more than what: Decoding-aligned KV cache compression via position-aware pseudo queries." arXiv preprint arXiv:2603.11564 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.