My main concerns:
C1. The motivation does not clearly justify reusing historical queries
The authors' starting observation makes sense: a KV pair that mattered in an earlier reasoning step may matter again later. If eviction decisions use only a recent observation window, an older, still-useful KV pair can be permanently removed.
But why does this imply that we should store old queries and repeatedly rescore the current cache with them? If a key was important to the model during an earlier step, a more direct response is to remember that importance. At each compression cycle, we could keep historical importance scores for each surviving KV pair and combine them with scores from the current window. An older key would then be protected because it was genuinely useful before, rather than because an old query gives it a high score in a new, artificial attention computation. The policy could use a maximum, an accumulated score, or a decayed score so that old importance does not dominate forever. It would still have to obey the same KV budget.
This is not a new or unreasonable line of thinking: H2O and Scissorhands study historical attention and persistence of importance. Both appear in QUEEN's bibliography, but the paper does not seriously discuss this alternative or compare against a matched historical-score-accumulation baseline. Comparisons against a recent-window-only policy do not answer this question.
I understand one difference: saved historical scores exist only for keys available when they were recorded, while a saved query can also score keys generated later. But that brings me to the next, more fundamental concern.
C2. Why should an old query evaluate a key generated after it?
Under standard causal decoding, a query at position i attends only to keys at positions up to i. If key k_j is generated at a later position j > i, the original query q_i never attends to it. QUEEN nevertheless keeps a refined version of q_i, repositions it using RoPE at a later compression step (Appendix A), and computes an attention score against k_j.
This is not historical attention that the LLM actually used. It is a hypothetical attention interaction. The model is trained to use the query generated from the current hidden state to attend to preceding keys; it is not specifically trained to take a query representing an earlier reasoning state, move it to a later position, and use it to judge newly generated keys. Changing its RoPE position does not update the content of that old query to reflect the newer reasoning state.
There may be a useful heuristic here: perhaps future queries are often similar to certain past queries, so the old ones act as proxies. I am not claiming that this cannot work. But why should these hypothetical scores predict the attention of actual future causal queries? That is the key assumption I want the authors to test. The argument that old keys might be useful again is not enough to justify using old queries against future-generated keys.
One simple experiment would be to allow each retained historical query to score only keys that already existed when that query was collected. If QUEEN loses little accuracy, then the supposed benefit of evaluating newer keys with older queries may not be important at all. If accuracy drops substantially, the authors should show that those new-query–old-state interactions really predict future causal attention, rather than only correlating with a different retention bias.
I believe the scope of this paper is narrower than what is explained in the abstract and introduction. See C3 and C4:
C3. Query refinement is not the same thing as predicting future information needs
The abstract and introduction motivate the method by the need to retain keys that will be relevant in future decoding. But the proposed method does not explicitly estimate future query vectors or future reasoning states. It repeats an already observed window, refines those queries using the now-completed window context, and reuses them later. That can make the query representations more informative about past and present context; it is not, by itself, a mechanism for predicting future causal attention.
I do not think predicting future queries is the only possible way to estimate future key importance. A proxy could work. But the paper needs to make that connection explicit and test it, rather than treating richer or more diverse historical queries as sufficient evidence that future relevance is better predicted. Refinement also produces queries different from those used by the model in ordinary causal decoding. This is fine as an auxiliary heuristic, but it makes the missing justification more important, not less.
The paper itself distinguishes QUEEN from BeaconKV's future-query motivation in Appendix G, and Appendix H calls predicting future queries a future direction. So I would not say the authors explicitly claim to generate future queries. My concern is more precise: the broad motivation about future relevance is not adequately connected to the actual past/current-context scoring mechanism.
C4. The scope is narrower than the general KV-retrieval motivation suggests
The failure mode motivating QUEEN is specific to irreversible KV eviction. Once an older KV pair has been discarded, even a future query that needs it cannot recover it. This is a real issue for eviction methods, including SnapKV, R-KV, H2O, Scissorhands, and BeaconKV.
Retrieval-based sparse attention has a different problem. Logically, at decoding step t, the model uses its current causal query to search over the retained full history of keys and select a subset for attention. The key/value history remains available, potentially in CPU memory or an index. If something from a much earlier reasoning step becomes important again, the current query can retrieve it then; there is no need to predict its importance several steps in advance or permanently keep it in a smaller evicted cache. Examples include Quest, SparQ Attention, RetrievalAttention, and Louver.
Of course retrieval can miss relevant keys when its pruning or approximate search is imperfect, and even exact top-k selection changes attention by dropping the remaining keys. That is a selection-quality issue, not the permanent-history-loss issue that QUEEN is designed around.
To be fair, Section 2 already acknowledges that retrieval-based methods avoid irreversible eviction, and the paper explicitly presents QUEEN as an eviction method. I am not claiming the authors say they solve retrieval-based sparse attention. My concern is that the broader introduction and query-as-retrieval framing should emphasize that the proposed solution is specifically for permanent eviction and does not address a general deficiency of query-based retrieval. Their characterization of retrieval mainly as CPU offloading also does not cover the full range of retrieval methods.
C5. The experiments show what helps, but not convincingly why it helps
I appreciate that the authors try to analyze the method, particularly in Section 4.4 and Appendices D and E. The results should be acknowledged accurately:
-
Table 1: Query reuse is responsible for most of the gain. On MATH500, removing reuse drops accuracy from 71.6 to 11.0 with refinement kept, whereas removing refinement but keeping reuse gives 62.4. This shows that reusing information is important; it does not prove that saving query vectors is better than saving historical scores.
-
Figure 5 and Appendix E.1: Refinement improves oracle-KV recall in prefill compression against ordinary window queries. This is a relevant positive result, but not a test of old queries predicting later decoding-time attention.
-
Figure 6 and Appendix E.2: QUEEN retains important keys across earlier decoding positions, rather than mainly recent positions. Importantly, the authors use full-KV attention and replay eviction with matched candidate keys; this is a useful controlled analysis. Still, it compares historical-query reuse with recent-window scoring, not with historical-score accumulation or simple time-stratified retention. The reported critical-key measure also aggregates importance over the full trace; it does not directly measure how well a scoring decision at time t predicts attention after t.
-
Figure 7: Reusing question queries retains function-name information in the prompt and increases correct function-name reproduction from 37% to 68%. This is persuasive evidence that protecting the original problem statement matters. It is not yet evidence that refined question queries are needed: reserving KV slots for prompt tokens might achieve the same result.
-
Figure 8 and Appendix D: Diversity selection beats a FIFO query pool (+7.1 points on AIME, +1.2 on MATH500, +1.7 on LiveCodeBench). This suggests benefit from covering earlier parts of the trace, but does not establish whether the specific historical query vectors are the important ingredient.
-
Table 2 and Appendix D: Restricting each question query to at most one selected KV pair improves accuracy. This could also reflect a useful cache-budget allocation rule, independent of the broader query-reuse hypothesis.
-
Figure 9: QUEEN reduces decoding-length inflation. The authors conjecture that it avoids having to reconstruct lost intermediate information. That is plausible, but not directly established by the figure.
Taken together, I can believe the method improves accuracy while remaining unconvinced by the stated explanation. Several other mechanisms could explain the same observations: keeping KV pairs from different stages of reasoning, preventing excessive recency bias, protecting the original prompt, preserving historical heavy hitters, or simply spreading the limited budget more evenly across positions and heads.
I would like to see at least three simpler controls: (i) accumulated historical attention scores (H2O/Scissorhands-style) with the same budget and update interval; (ii) explicit prompt protection plus ordinary eviction elsewhere; and (iii) a position-stratified or age-balanced retention policy. A fourth useful control is the historical-query restriction described above, which would isolate the value of scoring future-generated keys with old queries. All of these should use identical compression schedules and, as far as possible, the same scoring and implementation optimizations.
Most importantly, use a full-KV reference trace and ask, at each compression step, which policy retains the keys that the actual subsequent causal queries will attend to? Measure future-attention recall over several horizons (for example, the next 256, 512, and 1,024 tokens) and relate it to downstream accuracy. If QUEEN wins that comparison, it would directly support the proposed mechanism. If another simple policy does as well, that would be an equally interesting scientific finding.
C6. Code availability and the fairness of the efficiency results
The reproducibility statement (p. 10) says the authors will release code upon publication. I could not inspect an implementation from the material supplied with this review. ICLR does not require code at submission, so I do not treat its absence as proof of an incorrect result. But it limits my ability to verify fairly complicated parts of the method: repeated-window forwarding without modifying the original cache, RoPE reapplication for stored queries, grouped-query attention and per-head cache layouts, the single-KV restriction, variable-length cache rebuilding, and tiled scoring.
It also matters for the efficiency claims. Appendix F.3 attributes much of QUEEN's larger supported batch size relative to TriAttention to temporary compression memory: 0.165 GB per sequence for QUEEN versus 1.021 GB for TriAttention beyond the common KV-cache cost. QUEEN avoids materializing the full query-by-key score matrix by tiling. Could the same general memory-efficient optimization be used for TriAttention? If so, how much of the reported batch-size and throughput advantage is from QUEEN's query-selection idea, and how much is from a better-optimized scoring implementation? The authors do give a same-batch comparison in Appendix F.2 (360.9 versus 234.9 tokens/s at batch 21), which is useful, but it does not settle the matched-implementation question.
Appendix G also finds substantial continual query-selection overhead in BeaconKV's released implementation. This makes me want to know how performance changes under more comparable query-maintenance schedules or optimized implementations. I am not asking the authors to rewrite every baseline, but the paper should separate algorithmic improvements from implementation-level advantages as clearly as possible.
Finally, the historical-query pool is not free. Appendix F.1 reports 151 MB of pinned CPU memory per sequence for a 512-query pool, compared with 453 MB of KV cache per sequence at the stated budget. This may be reasonable, but the memory and throughput discussion should account for total CPU-plus-GPU memory, host transfers, and longer prompts, rather than concentrating almost entirely on peak GPU memory. More realistic input lengths would be helpful.
Minor issue
References
-
H2O — Zhang et al., H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023. arXiv
-
Scissorhands — Liu et al., Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time, NeurIPS 2023. arXiv
-
StreamingLLM — Xiao et al., Efficient Streaming Language Models with Attention Sinks, ICLR 2024. arXiv
-
SnapKV — Li et al., SnapKV: LLM Knows What You are Looking for Before Generation, NeurIPS 2024. arXiv
-
R-KV — Cai et al., R-KV: Redundancy-aware KV Cache Compression for Training-Free Reasoning Models Acceleration, NeurIPS 2025. arXiv
-
BeaconKV — Kim et al., BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference, ICML 2026. arXiv
-
Quest — Tang et al., Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML 2024. arXiv
-
SparQ Attention — Ribar et al., SparQ Attention: Bandwidth-Efficient LLM Inference, ICML 2024. arXiv
-
RetrievalAttention — Liu et al., RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval, NeurIPS 2025. arXiv
-
Louver — Dehghankar and Asudeh, Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache, NeurIPS 2026. arXiv
Author Response #1 to Reviewer zQXm
We sincerely thank the reviewer for the positive and constructive evaluation of our work. We greatly appreciate the recognition of our clear writing, the diagnosis of the two failure modes of the scoring query, and our training-free method with strong results and extensive ablations. We address the reviewer's questions in detail below. For clarity, we mark responses to weakness comments with W# and responses to explicit questions with Q#.
W1, Q1
→ Good suggestion! As requested, we re-evaluated BeaconKV [1] under the same 256-token compression interval as the other methods, as shown in Table A. QueEn still leads in all 8 conditions after matching the interval, as under the original schedule.
We have added these results to App. G of the revised manuscript.
Same 256-token compression interval.
Table A: Accuracy of BeaconKV at the same 256-token interval and of QueEn (Qwen3-8B)
(Updated to match the MATH500 values of Table 26 in the revision.)
In Table A, matching the interval helps BeaconKV most at the smallest MATH500 budgets (+12.0 and +6.6pt at B=64 and 128), while LiveCodeBench changes little (-1.1 to +2.5pt). This is because, at these two budgets, BeaconKV's original schedule compresses about every 9 and 18 tokens. A just-generated token can thus be evicted within a few steps, whereas a 256-token interval keeps recent tokens until the next compression. Even so, QueEn still leads by 40.4 and 32.2pt.
[1] Kim, Janghyeon, et al. "BeaconKV: Key-value cache compression guided by beacon queries for efficient large reasoning model inference." ICML. 2026.
W2, Q2
→ Thank you for asking whether reuse helps beyond reasoning tasks. Not for long-context inputs; more precisely, QueEn's reuse does not apply when a long input is compressed once at prefill, as in LongBench. This is because reuse carries window queries from one compression to the next and thus takes effect only when compression repeats during long decoding. As the reviewer noted, only refinement applies in this setting. We briefly note this scope in App. H of the submission (App. J of the revision), where we leave reuse at prefill to future work, and we have made it clearer in Sec. 4.4 of the revision.
Minor
→ Thank you! We have fixed all broken cross-references in the revised manuscript.
Thank you again for your positive assessment and helpful suggestions. We would be glad to answer any further questions.