- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies query–cache mismatch when independently compiled document KV caches are reused under a new request, and proposes QueryLens to improve how the model reads the reused memory while keeping document keys and values fixed. It introduces QueryLens-T, which prepends learned soft tokens to the query, and QueryLens-H, which applies a gated low-rank residual transformation to attention queries. Both are trained by self-distillation from the frozen backbone with full context, and the head-level transformation can be folded into the query projection for deployment. The evaluation covers Llama-3.1-8B-Instruct and Qwen3-4B-Instruct-2507, four tasks, and combinations with CacheBlend, A3, EPIC, and KVPacket. With A3 at 10% recomputation, the reported three-task aggregate score increases from 26.5 to 69.1 on Llama and from 18.39 to 54.85 on Qwen; Llama Biography F1 increases from 6.5 to 98.8. Fixed-cache diagnostics further indicate that query changes can recover substantial distortion, while the statically folded implementation changes measured median time to first token by less than 0.3% in magnitude across the reported settings.
Critical strengths
- A practical and lightweight intervention. QueryLens-H operates on the query path while preserving the realized document cache, and the paper implements combinations with several cache-reuse methods. Static folding removes the need for an additional online correction module, with measured median TTFT changes below 0.3% in magnitude. The large improvements under aggressive reuse, particularly the Llama A3–10 aggregate increase from 26.5 to 69.1, make this a potentially useful addition to existing cache-reuse systems.
- Evidence at several levels supports the proposed mechanism. The paper connects a fixed-key reachability analysis with attention-space diagnostics, output-distribution interventions, and downstream task results. In particular, fixed-cache logit projections recover 51.7%–67.2% of the distortion in the four highlighted settings, while the separate output-space intervention removes 85.0% of next-token KL with learned QueryLens-H and 99.6% with unconstrained per-instance query edits. These complementary observations support the existence of a useful query-side correction opportunity in the tested settings, although the oracle studies are small and do not establish how widespread the problem is.
Critical weaknesses
- The incremental contribution is not sufficiently isolated from closely related approaches. The central contribution is the particular QueryLens-T/H design, rather than identifying cache-reuse errors as a new problem. APE (ICLR 2025), which calibrates attention for independently encoded contexts, is highly relevant but is not discussed or compared. Moreover, the clean-context Q-LoRA control changes the student training condition and therefore does not isolate the benefit of the proposed head-wise parameterization and gate from the benefit of training on reused caches. A standard query-projection LoRA baseline trained with the same reused-cache student, teacher, data, and loss is needed to establish what the proposed design adds. Without these comparisons, the substantial gains demonstrate effectiveness but leave the methodological novelty uncertain.
- The evidence does not yet establish the breadth and reliability of the practical benefit. The key oracle diagnostics use only 16 instances per highlighted fixed-cache setting and 12 requests for the output-space intervention. The downstream evaluation is broader, but uses two relatively small backbones and defaults to 100 examples with one seed; it lacks a consistent full-context task reference and uncertainty estimates for the main gains. Consequently, it is difficult to judge how much of the original task-performance loss is recovered or how reliably the correction helps beyond the tested settings. The paper specifies query-level splits without clearly establishing generalization to unseen documents, and some settings show regressions. In addition, ostensibly corresponding results differ across tables, such as Llama/CacheBlend–10 NIAH in Table 1(a) versus Table 1(c), and Qwen/A3–10 NIAH in Table 7 versus Table 5. These issues limit confidence in the scope and reproducibility of the main claims.
Questions for authors
- Closest comparisons and attribution. How does QueryLens differ from APE (ICLR 2025), and are comparable results available? Can you provide a standard query-projection LoRA control trained on the same reused-cache student with the same teacher, training split, objective, and comparable parameter budget? This comparison would help determine whether the proposed parameterization contributes beyond the training setup.
- Reference quality and reliability. What are the full-context task scores for the main evaluation settings, and what fraction of the reuse-induced performance loss does QueryLens recover? Can you provide uncertainty estimates for the main gains and clarify how frequently the method degrades performance? These results would materially affect the assessment of the practical benefit.
- Reconciliation of reported scores. Why does Llama/CacheBlend–10 NIAH score 66.43 in Table 1(a) but 68.02 in Table 1(c), and why does Qwen/A3–10 NIAH score 56.53 in Table 7 but 63.91 in Table 5? Please identify any differences in correction scale, checkpoint, data, or evaluation configuration, and correct any reporting errors.
10 comments
Response 1.2: LoRA Baselines under Identical Reused-Cache Training Conditions
Can you provide a standard query-projection LoRA control trained on the same reused-cache student with the same teacher, training split, objective, and comparable parameter budget? This comparison would help determine whether the proposed parameterization contributes beyond the training setup.
We thank the reviewer for this important suggestion. We have added module-placement baselines with comparable parameter budgets under the same reused-cache training conditions. Specifically, we evaluated standard Q-LoRA, O-LoRA, FFN-LoRA, and Q+O-LoRA using the same reused-cache student, full-prefill teacher, training split, and KL-distillation objective. The trainable-parameter budget was controlled at approximately 1.05M.
All methods keep document KV states fixed, varying only the placement and parameterization of the adapters. Results are shown in Table 3.
Table 3. Parameter-matched LoRA placement baselines versus QueryLens-H on fixed reused KV caches (task scores, %).
| Method | Trainable parameters | NIAH ↑ | Biography ↑ | HotpotQA ↑ | Macro ↑ |
|---|---|---|---|---|---|
| Cache-only | 0 | 38.70 | 6.33 | 23.80 | 22.94 |
| Standard Q-LoRA | 1.05M | 54.76 | 96.91 | 27.85 | 59.84 |
| O-LoRA | 1.05M | 59.86 | 97.67 | 28.86 | 62.13 |
| FFN-LoRA | 1.05M | 0.00 | 2.49 | 0.00 | 0.83 |
| Q+O-LoRA | 1.05M | 54.38 | 96.30 | 30.89 | 60.52 |
| QueryLens-H | 1.05M | 61.25 | 98.38 | 37.67 | 65.77 |
First, although all LoRA methods are trained with distillation under the same cache-mismatch conditions, QueryLens-H achieves the highest score on each of the three tasks. This suggests that the gains cannot be explained solely by reused-cache distillation or by the number of trainable parameters.
Second, these findings are consistent with the structured parameterization of QueryLens-H. Standard Q-LoRA applies a shared, globally low-rank update to the entire query projection, with update rank at most . In contrast, QueryLens-H allocates a separate rank- correction subspace to each attention head, raising the upper bound on the rank of the global update to , while maintaining a comparable parameter budget:
This head-wise low-rank parameterization allows distinct heads to learn different correction directions for cache mismatch, while per-head learnable gates further regulate the correction magnitude. Together, these mechanisms enable finer-grained calibration of heterogeneous attention-routing errors.
Response 2.1: Reliability of the Oracle Diagnostics
We thank the reviewer for raising concerns about the statistical reliability of the oracle diagnostics. To test whether the recoverability of query-side correction is not merely an artifact of a small number of examples, we increased the number of requests for fixed-cache reachability from 16 to 100 per configuration, expanded the output-space oracle from 12 to 50 requests, and evaluated the attention oracle on 50 requests. We also report request-level bootstrap 95% confidence intervals.
Theoretical analysis. For a fixed reused key matrix, the fraction of squared attention-logit error that can be eliminated through query correction can be characterized by an orthogonal decomposition: the component of the target-logit error lying in the column space of the actual key matrix is query-recoverable, whereas the component outside that space cannot be recovered when the keys are fixed. We conducted larger-scale oracle experiments to examine both this theoretical recovery space and its effects on the output distribution, as reported in Table 4.
Table 4. Expanded fixed-cache, output-space, and attention Oracle diagnostics with request-level 95% bootstrap intervals.
| Oracle experiment | Setting | N | Main result (95% CI) |
|---|---|---|---|
| Fixed-cache | NIAH / CB–30 | 100 | 52.25% [49.39%, 55.24%] |
| Fixed-cache | NIAH / CB–50 | 100 | 52.23% [49.36%, 55.24%] |
| Fixed-cache | Biography / A3–50 | 100 | 59.31% [58.99%, 59.63%] |
| Fixed-cache | Biography / CB–50 | 100 | 66.82% [66.59%, 67.06%] |
| Fixed-cache | Biography / KVPacket | 100 | 40.46% [40.20%, 40.72%] |
| Output-space | QueryLens-H / NIAH | 50 | 85.52% [83.81%, 87.03%] |
| Output-space | Free Query Oracle / NIAH | 50 | 99.56% [99.51%, 99.61%] |
| Attention | NIAH / A3–10 | 50 | JS ↓ 0.239 [0.228, 0.250]; Top-8 ↑ 0.446 [0.427, 0.464] |
Across the five expanded fixed-cache configurations, 40.46%–66.82% of the squared attention-logit distortion is query-recoverable. This indicates that the phenomenon persists across the evaluated tasks and cache-reuse methods. The median recoverability on NIAH is approximately 43.5%, also revealing heterogeneity across requests.
In the output-space experiment, QueryLens-H eliminates an average of 85.52% of the teacher-KL distortion and improves the result on all 50 of the 50 evaluated requests. The unconstrained free-query oracle eliminates 99.56% of this distortion. The attention oracle further demonstrates that query modification alone, without changing the cached KV states, can substantially improve agreement with full-prefill attention distributions.
Response 1.1: Comparison with APE and Attribution of Contributions
We thank the reviewer for bringing APE (ICLR 2025), a highly relevant work, to our attention. We have examined its approach and conducted additional methodological analyses and comparisons of task accuracy and inference efficiency to clarify the incremental contribution of QueryLens.
(1) Shared Observations and Key Differences from APE
APE identifies attention-distribution mismatch caused by independently encoded KV caches, including repeated attention sinks and diffused attention. It addresses these issues through training-free calibration using a shared prefix, attention temperature, and a scaling factor.
Consistent with APE, we observe that independently reused KV states can shift attention routing away from full-prefill behavior even when task-relevant information has not necessarily been lost. Our further observation is that, with the existing KV cache fixed, adjusting only the query may still recover part of the original model behavior. Building on this shared observation, QueryLens analyzes the recoverability of attention errors under fixed KV and introduces a learnable, fine-grained correction mechanism.
We will discuss APE in the revised manuscript and revise our earlier claim of being the “first to identify query–cache mismatch.” Instead, we will center our contribution on the theoretical boundary of query-side correction and the proposed learnable calibration mechanism.
(2) Additional Task-Accuracy Comparison
We have added a comparison of APE and QueryLens-only on downstream tasks, as shown in Table 1.
Table 1. Downstream task scores of APE and QueryLens-only under the reported comparison setup (%).
QueryLens obtains higher scores on all four tasks in this comparison. QueryLens can also complement cache-side recomputation. For example, under A3–10%, QueryLens-H increases the three-task macro-F1 on Llama-3.1-8B from 26.5 to 69.1 (+42.6 percentage points) and on Qwen3-4B from 18.4 to 54.9 (+36.5 points), without increasing the cache-recomputation budget.
(3) Additional Inference-Efficiency Comparison
We also measured inference efficiency on Llama-3.1-8B-Instruct using a single A100 80GB GPU, BF16 precision, and batch size 1. Each task includes 100 test requests; measurements were repeated three times, with greedy decoding and FlashAttention. Table 2 reports the detailed results.
Table 2. TTFT and decoding throughput of APE and QueryLens-only on Llama-3.1-8B.
In these settings, QueryLens-only reduces TTFT by approximately 16.8%–27.6% relative to APE and achieves approximately 1.94 times its decoding throughput. Furthermore, the independent deployment experiments in Appendix F show that, when QueryLens-H is statically merged, the TTFT difference relative to the corresponding unmodified cache-reuse baseline remains within approximately ±0.3% for context lengths from 4K to 32K. This further supports its low additional online overhead.