acceptodds
Under review as a conference paper at ICLR 2027

QueryLens: Lightweight Query Recalibration for Reused KV Caches

Abstract

Long-context language model serving increasingly relies on key–value (KV) cache reuse to avoid repeated prefill computation over recurring documents. Yet caches compiled independently and assembled under a new request cannot fully reproduce the contextual interactions of joint prefill and require correction. Existing approaches typically operate on the cache side through selective token recomputation, aiming to make reused caches more consistent with their full-context counterparts. However, cache-side repair can increase time to first token (TTFT) as the recomputation budget grows, while the native query may remain mismatched with the realized cache. To address this issue, we propose **QueryLens**, a query-correction method that introduces a new query-side perspective on KV cache reuse. It learns a lightweight, shared transformation of queries to improve how reused KV states are read. Moreover, QueryLens supports token-level and attention-head-level calibration and can be combined with existing KV cache methods to further improve performance; its statically folded head-level variant incurs negligible measured TTFT overhead. Experiments show that adding QueryLens to A3 at recomputation raises the three-task macro-F1 on Llama-3.1-8B-Instruct from to .

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache reuse, long-context language models, query calibration, attention alignment, cache recomputation, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DJA3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies query–cache mismatch when independently compiled document KV caches are reused under a new request, and proposes QueryLens to improve how the model reads the reused memory while keeping document keys and values fixed. It introduces QueryLens-T, which prepends learned soft tokens to the query, and QueryLens-H, which applies a gated low-rank residual transformation to attention queries. Both are trained by self-distillation from the frozen backbone with full context, and the head-level transformation can be folded into the query projection for deployment. The evaluation covers Llama-3.1-8B-Instruct and Qwen3-4B-Instruct-2507, four tasks, and combinations with CacheBlend, A3, EPIC, and KVPacket. With A3 at 10% recomputation, the reported three-task aggregate score increases from 26.5 to 69.1 on Llama and from 18.39 to 54.85 on Qwen; Llama Biography F1 increases from 6.5 to 98.8. Fixed-cache diagnostics further indicate that query changes can recover substantial distortion, while the statically folded implementation changes measured median time to first token by less than 0.3% in magnitude across the reported settings.

Critical strengths
  1. A practical and lightweight intervention. QueryLens-H operates on the query path while preserving the realized document cache, and the paper implements combinations with several cache-reuse methods. Static folding removes the need for an additional online correction module, with measured median TTFT changes below 0.3% in magnitude. The large improvements under aggressive reuse, particularly the Llama A3–10 aggregate increase from 26.5 to 69.1, make this a potentially useful addition to existing cache-reuse systems.
  2. Evidence at several levels supports the proposed mechanism. The paper connects a fixed-key reachability analysis with attention-space diagnostics, output-distribution interventions, and downstream task results. In particular, fixed-cache logit projections recover 51.7%–67.2% of the distortion in the four highlighted settings, while the separate output-space intervention removes 85.0% of next-token KL with learned QueryLens-H and 99.6% with unconstrained per-instance query edits. These complementary observations support the existence of a useful query-side correction opportunity in the tested settings, although the oracle studies are small and do not establish how widespread the problem is.
Critical weaknesses
  1. The incremental contribution is not sufficiently isolated from closely related approaches. The central contribution is the particular QueryLens-T/H design, rather than identifying cache-reuse errors as a new problem. APE (ICLR 2025), which calibrates attention for independently encoded contexts, is highly relevant but is not discussed or compared. Moreover, the clean-context Q-LoRA control changes the student training condition and therefore does not isolate the benefit of the proposed head-wise parameterization and gate from the benefit of training on reused caches. A standard query-projection LoRA baseline trained with the same reused-cache student, teacher, data, and loss is needed to establish what the proposed design adds. Without these comparisons, the substantial gains demonstrate effectiveness but leave the methodological novelty uncertain.
  2. The evidence does not yet establish the breadth and reliability of the practical benefit. The key oracle diagnostics use only 16 instances per highlighted fixed-cache setting and 12 requests for the output-space intervention. The downstream evaluation is broader, but uses two relatively small backbones and defaults to 100 examples with one seed; it lacks a consistent full-context task reference and uncertainty estimates for the main gains. Consequently, it is difficult to judge how much of the original task-performance loss is recovered or how reliably the correction helps beyond the tested settings. The paper specifies query-level splits without clearly establishing generalization to unseen documents, and some settings show regressions. In addition, ostensibly corresponding results differ across tables, such as Llama/CacheBlend–10 NIAH in Table 1(a) versus Table 1(c), and Qwen/A3–10 NIAH in Table 7 versus Table 5. These issues limit confidence in the scope and reproducibility of the main claims.
Questions for authors
  1. Closest comparisons and attribution. How does QueryLens differ from APE (ICLR 2025), and are comparable results available? Can you provide a standard query-projection LoRA control trained on the same reused-cache student with the same teacher, training split, objective, and comparable parameter budget? This comparison would help determine whether the proposed parameterization contributes beyond the training setup.
  2. Reference quality and reliability. What are the full-context task scores for the main evaluation settings, and what fraction of the reuse-induced performance loss does QueryLens recover? Can you provide uncertainty estimates for the main gains and clarify how frequently the method degrades performance? These results would materially affect the assessment of the practical benefit.
  3. Reconciliation of reported scores. Why does Llama/CacheBlend–10 NIAH score 66.43 in Table 1(a) but 68.02 in Table 1(c), and why does Qwen/A3–10 NIAH score 56.53 in Table 7 but 63.91 in Table 5? Please identify any differences in correction scale, checkpoint, data, or evaluation configuration, and correct any reporting errors.
10 comments

Response 1.1: Comparison with APE and Attribution of Contributions

We thank the reviewer for bringing APE (ICLR 2025), a highly relevant work, to our attention. We have examined its approach and conducted additional methodological analyses and comparisons of task accuracy and inference efficiency to clarify the incremental contribution of QueryLens.

(1) Shared Observations and Key Differences from APE

APE identifies attention-distribution mismatch caused by independently encoded KV caches, including repeated attention sinks and diffused attention. It addresses these issues through training-free calibration using a shared prefix, attention temperature, and a scaling factor.

Consistent with APE, we observe that independently reused KV states can shift attention routing away from full-prefill behavior even when task-relevant information has not necessarily been lost. Our further observation is that, with the existing KV cache fixed, adjusting only the query may still recover part of the original model behavior. Building on this shared observation, QueryLens analyzes the recoverability of attention errors under fixed KV and introduces a learnable, fine-grained correction mechanism.

We will discuss APE in the revised manuscript and revise our earlier claim of being the “first to identify query–cache mismatch.” Instead, we will center our contribution on the theoretical boundary of query-side correction and the proposed learnable calibration mechanism.

(2) Additional Task-Accuracy Comparison

We have added a comparison of APE and QueryLens-only on downstream tasks, as shown in Table 1.

Table 1. Downstream task scores of APE and QueryLens-only under the reported comparison setup (%).

DatasetAPE ↑QueryLens-only ↑Gain (pp)
Biography23.5198.38+74.87
NIAH48.7155.61+6.90
HotpotQA31.3437.67+6.33
MultiFieldQA-en47.8348.05+0.22

QueryLens obtains higher scores on all four tasks in this comparison. QueryLens can also complement cache-side recomputation. For example, under A3–10%, QueryLens-H increases the three-task macro-F1 on Llama-3.1-8B from 26.5 to 69.1 (+42.6 percentage points) and on Qwen3-4B from 18.4 to 54.9 (+36.5 points), without increasing the cache-recomputation budget.

(3) Additional Inference-Efficiency Comparison

We also measured inference efficiency on Llama-3.1-8B-Instruct using a single A100 80GB GPU, BF16 precision, and batch size 1. Each task includes 100 test requests; measurements were repeated three times, with greedy decoding and FlashAttention. Table 2 reports the detailed results.

Table 2. TTFT and decoding throughput of APE and QueryLens-only on Llama-3.1-8B.

DatasetAPE TTFT ↓QueryLens TTFT ↓APE decoding ↑QueryLens decoding ↑
NIAH 8K50.02 ms41.61 ms23.06 tok/s44.68 tok/s
Biography46.39 ms33.59 ms23.62 tok/s45.70 tok/s
HotpotQA46.10 ms34.03 ms23.70 tok/s46.39 tok/s

In these settings, QueryLens-only reduces TTFT by approximately 16.8%–27.6% relative to APE and achieves approximately 1.94 times its decoding throughput. Furthermore, the independent deployment experiments in Appendix F show that, when QueryLens-H is statically merged, the TTFT difference relative to the corresponding unmodified cache-reuse baseline remains within approximately ±0.3% for context lengths from 4K to 32K. This further supports its low additional online overhead.

Response 1.2: LoRA Baselines under Identical Reused-Cache Training Conditions

Can you provide a standard query-projection LoRA control trained on the same reused-cache student with the same teacher, training split, objective, and comparable parameter budget? This comparison would help determine whether the proposed parameterization contributes beyond the training setup.

We thank the reviewer for this important suggestion. We have added module-placement baselines with comparable parameter budgets under the same reused-cache training conditions. Specifically, we evaluated standard Q-LoRA, O-LoRA, FFN-LoRA, and Q+O-LoRA using the same reused-cache student, full-prefill teacher, training split, and KL-distillation objective. The trainable-parameter budget was controlled at approximately 1.05M.

All methods keep document KV states fixed, varying only the placement and parameterization of the adapters. Results are shown in Table 3.

Table 3. Parameter-matched LoRA placement baselines versus QueryLens-H on fixed reused KV caches (task scores, %).

MethodTrainable parametersNIAH ↑Biography ↑HotpotQA ↑Macro ↑
Cache-only038.706.3323.8022.94
Standard Q-LoRA1.05M54.7696.9127.8559.84
O-LoRA1.05M59.8697.6728.8662.13
FFN-LoRA1.05M0.002.490.000.83
Q+O-LoRA1.05M54.3896.3030.8960.52
QueryLens-H1.05M61.2598.3837.6765.77

First, although all LoRA methods are trained with distillation under the same cache-mismatch conditions, QueryLens-H achieves the highest score on each of the three tasks. This suggests that the gains cannot be explained solely by reused-cache distillation or by the number of trainable parameters.

Second, these findings are consistent with the structured parameterization of QueryLens-H. Standard Q-LoRA applies a shared, globally low-rank update to the entire query projection, with update rank at most rr. In contrast, QueryLens-H allocates a separate rank-rr correction subspace to each attention head, raising the upper bound on the rank of the global update to HrHr, while maintaining a comparable parameter budget:

​rank⁡(ΔWQLoRA)≤r,rank⁡(ΔWQQL)≤Hr.\operatorname{rank}(\Delta W_Q^{\mathrm{LoRA}})\le r, \qquad \operatorname{rank}(\Delta W_Q^{\mathrm{QL}})\le Hr.​

This head-wise low-rank parameterization allows distinct heads to learn different correction directions for cache mismatch, while per-head learnable gates further regulate the correction magnitude. Together, these mechanisms enable finer-grained calibration of heterogeneous attention-routing errors.

Response 2.1: Reliability of the Oracle Diagnostics

We thank the reviewer for raising concerns about the statistical reliability of the oracle diagnostics. To test whether the recoverability of query-side correction is not merely an artifact of a small number of examples, we increased the number of requests for fixed-cache reachability from 16 to 100 per configuration, expanded the output-space oracle from 12 to 50 requests, and evaluated the attention oracle on 50 requests. We also report request-level bootstrap 95% confidence intervals.

Theoretical analysis. For a fixed reused key matrix, the fraction of squared attention-logit error that can be eliminated through query correction can be characterized by an orthogonal decomposition: the component of the target-logit error lying in the column space of the actual key matrix is query-recoverable, whereas the component outside that space cannot be recovered when the keys are fixed. We conducted larger-scale oracle experiments to examine both this theoretical recovery space and its effects on the output distribution, as reported in Table 4.

Table 4. Expanded fixed-cache, output-space, and attention Oracle diagnostics with request-level 95% bootstrap intervals.

Oracle experimentSettingNMain result (95% CI)
Fixed-cacheNIAH / CB–3010052.25% [49.39%, 55.24%]
Fixed-cacheNIAH / CB–5010052.23% [49.36%, 55.24%]
Fixed-cacheBiography / A3–5010059.31% [58.99%, 59.63%]
Fixed-cacheBiography / CB–5010066.82% [66.59%, 67.06%]
Fixed-cacheBiography / KVPacket10040.46% [40.20%, 40.72%]
Output-spaceQueryLens-H / NIAH5085.52% [83.81%, 87.03%]
Output-spaceFree Query Oracle / NIAH5099.56% [99.51%, 99.61%]
AttentionNIAH / A3–1050JS ↓ 0.239 [0.228, 0.250]; Top-8 ↑ 0.446 [0.427, 0.464]

Across the five expanded fixed-cache configurations, 40.46%–66.82% of the squared attention-logit distortion is query-recoverable. This indicates that the phenomenon persists across the evaluated tasks and cache-reuse methods. The median recoverability on NIAH is approximately 43.5%, also revealing heterogeneity across requests.

In the output-space experiment, QueryLens-H eliminates an average of 85.52% of the teacher-KL distortion and improves the result on all 50 of the 50 evaluated requests. The unconstrained free-query oracle eliminates 99.56% of this distortion. The attention oracle further demonstrates that query modification alone, without changing the cached KV states, can substantially improve agreement with full-prefill attention distributions.

Reviewer_6t2cOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes QueryLens, a query-correction method that introduces a new query-side perspective on KV cache reuse. Building on existing block-wise KV caching, it proposes transforming the query side to mitigate the attention loss caused by block-wise KV caching.

Critical strengths

Sufficient experiments: This paper demonstrates the effectiveness of the proposed method through extensive experiments, covering hyperparameter selection, cache budgets, and different block-wise KV cache methods, which is convincing.

Simple yet effective method: The method in this paper is easy to understand and simple yet effective.

Innovative method: Prior work has extensively studied block-wise KV caching and targeted recomputation, but query-level correction of this kind is, to my knowledge, still relatively rare.

Easy to integrate: It can be integrated into existing block-wise KV cache methods.

Critical weaknesses

Lack of experiments on larger-scale models: The properties and conclusions of block-wise KV methods may differ as model size increases, and it is currently unclear whether query-level correction exhibits similar properties. I hope experiments on models of 32B or above can be added.

The method's gains show diminishing marginal returns, and even reversal, as the KV budget increases: I hope this phenomenon can be further explained.

Questions for authors

see above

4 comments

Weakness 1 — Generalization to a 32B Model

Lack of experiments on larger-scale models: The properties and conclusions of block-wise KV methods may differ as model size increases, and it is currently unclear whether query-level correction exhibits similar properties. I hope experiments on models of 32B or above can be added.

We thank the reviewer for this important suggestion regarding model-scale generalization. We extended our experiments to Qwen3-32B, systematically evaluating QueryLens with three KV-cache repair methods—CacheBlend, A3, and EPIC—across five budget settings for each method.

(1) Theoretical Motivation

The central mechanism of QueryLens does not intrinsically depend on model size. Rather, it exploits the degrees of freedom for query correction that remain when the KV cache is fixed. According to Proposition 1, whenever the original attention-logit error of a reused cache contains a recoverable component in the column space of its key matrix, modifying the query can potentially improve model behavior without additional cache recomputation. The new experiments therefore test whether query-side correction continues to provide benefits to different cache-repair methods when the backbone is scaled to 32B parameters. Results are reported in Table 1.

(2) Additional Experiments on the 32B Model

Model: Qwen/Qwen3-32B.

Tasks and metrics: Biography, NIAH/RULER-8192, and HotpotQA. For each task, F1 is calculated using aggregated token-level TP/FP/FN counts; the three task scores are then averaged with equal weights.

Evaluation size: 100 examples per task. All methods and budget settings use the same saved sample IDs, questions, and reference answers. Both the test sampling and evaluation run use seed 42.

Table 1. Qwen-32B results under A3, CacheBlend, and EPIC across their respective recomputation budgets (task scores, %).

MethodBudgetBaselineQueryLensGain (pp)
A310%28.4743.61+15.13
A330%36.5644.72+8.16
A350%42.7048.08+5.38
A370%53.9055.31+1.41
A390%60.6163.63+3.03
CacheBlend10%35.8745.17+9.30
CacheBlend30%43.2551.80+8.55
CacheBlend50%53.0258.54+5.52
CacheBlend70%60.0261.70+1.68
CacheBlend90%63.9166.11+2.20
EPIC10 tokens/chunk34.3444.71+10.36
EPIC30 tokens/chunk41.2049.49+8.29
EPIC50 tokens/chunk47.3452.06+4.71
EPIC70 tokens/chunk53.9056.88+2.99
EPIC90 tokens/chunk57.5557.43−0.12

First, QueryLens yields positive gains in 14 of the 15 evaluated configurations. This indicates that query-side correction remains effective on a 32B backbone and is not restricted to the smaller models evaluated in the original manuscript.

Second, the benefits are observed across distinct cache-repair strategies. At the lowest repair budgets, QueryLens improves A3, CacheBlend, and EPIC by 15.13, 9.30, and 10.36 percentage points, respectively. Because these methods rely on different cache-repair mechanisms, the results provide additional evidence that QueryLens can function as a complementary query-side calibration mechanism across methods.

Weakness 2 — Diminishing or Reversed Gains at Higher KV Budgets

The method's gains show diminishing marginal returns, and even reversal, as the KV budget increases: I hope this phenomenon can be further explained.

We thank the reviewer for the careful observation. We also find that the incremental benefit of QueryLens generally decreases as the KV recomputation budget grows. We believe that two factors help explain this behavior.

First, larger recomputation budgets reduce the residual error available for query correction. QueryLens aims to compensate for attention-routing mismatches between reused caches and full prefill while keeping the reused cache fixed. At low recomputation ratios, the mismatch can be substantial, creating considerable room for query correction. As more tokens are recomputed, the cache and subsequent query representations become closer to those obtained through full prefill. The underlying cache-reuse method has already recovered much of the lost performance, naturally reducing the headroom for further gains from QueryLens. Thus, diminishing marginal returns are consistent with the complementary nature of our mechanism.

Second, high recomputation budgets may introduce mild calibration mismatch. QueryLens learns shared query-correction parameters through self-distillation from a full-prefill teacher and applies the same correction across different recomputation budgets. As the residual cache error becomes smaller, the learned correction directions may no longer align perfectly with the remaining error, potentially causing overcorrection. This is one motivation for introducing bounded learnable gates for individual layers and attention heads. The correction-strength ablation in Table 1(b) of the original paper further shows that controlling the correction magnitude can improve performance.

Importantly, reversals are neither universal nor large in the aggregate results. In Tables 4–5 of Appendix D, we compare macro-F1 across 20 configurations: two backbones, two cache-reuse methods, and five recomputation budgets. QueryLens achieves a nonnegative gain in 19 of these 20 configurations. The sole negative macro-F1 gain occurs for Qwen3-4B with A3 at the 90% budget, where the score decreases by 1.86 percentage points (55.00 to 53.14). Thus, QueryLens retains positive overall benefits in most evaluated settings, although some high-budget configurations and individual tasks can exhibit negative corrections.

Moreover, recomputing 90% of document tokens leaves substantially less opportunity for computational savings from cache reuse. We include such high-budget settings primarily to examine how the method behaves near extensive recomputation, rather than as the intended operating regime for efficient cache reuse. The limited additional gains and occasional reversals at high budgets should therefore be interpreted in light of the remaining recoverable error and computational costs, rather than as evidence against the benefits at low recomputation budgets.

We will clarify this mechanism and the corresponding operating limits in the revised manuscript.

We sincerely thank the reviewer for the constructive feedback. The suggestions have helped us improve both our experimental analysis and the clarity of our presentation. We hope that the new results and explanations address these concerns and better demonstrate the contribution and practical value of QueryLens. We welcome further discussion and respectfully invite the reviewer to reconsider the assessment in light of this additional evidence.

Thank you for taking the time to address my concerns. As most of my questions have been resolved, I will maintain my rating.

Dear Reviewer 6t2c,

Thank you so much for taking the time to read our rebuttal and follow up with us. We're truly glad that it was able to address most of your questions. Are there any remaining concerns or points that you'd like us to clarify? We would sincerely appreciate the opportunity to address them.

As our current average score is still below the acceptance threshold, if you believe that our response has properly addressed your concerns and that our work makes valuable contributions, we would be deeply grateful if you could consider further raising your score and supporting our work during the upcoming AC-Reviewer Discussion Phase.

We'll be available over the next eight hours before the discussion closes.


Thank you again for your time, consideration, and support.

Best regards,

The Authors of Submission 5508

Reviewer_TReiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Summary

  • The authors identify that correcting queries for KV cache stitching at the test time is critical, and thus propose a method that fine-tunes a query side low rank adapters, training on self-distillation objective, while keeping the model backbone fozen. The teacher proces context jointly together while the student sees the context with separately prefilled KV cache stitched afterwards. The authors implement on top of a recomputation baseline and a soft prompt tuning baseline, and show relative gains compared to those.
Critical strengths

Strength

  • They propose a new method that specifically fine-tunes the query projection layer side to correct the behaviour of KV cache stitching for the model, while keeping the backbone of the model frozen.
Critical weaknesses

Weakness

  • The authors argue that query-side correction is important for repairing errors from test-time KV-cache stitching, and thus propose training parameter-efficient low-rank components on the query projections with a distillation loss. However, the current comparisons do not fully isolate the benefit of adapting the query projections specifically compared to reasonable parameter-efficient fine-tuning baselines, and comparisons with training-free recomputation baselines like A3 or EPIC do not isolate this benefit. To establish the specific value of query-side adaptation, the authors should compare against other reasonable parameter-efficient fine-tuning baselines under the same training objective. For instance, the main baselines should include LoRA on different modules or across multiple modules, with matched total parameter budgets and the same fixed document caches, with the same training objectives. If their contribution is identifying query-side correction as critical, they should demonstrate its practical efficiency gains compared to these alternatives. It is already known that fine-tuning can improve performance under KV-cache stitching, as shown by KVLink. The authors seem to be applying this adaptation idea specifically to the query projections through parameter-efficient fine-tuning. If they claim that the practical gains come from where to fine-tune, rather than fine-tuning itself, they should compare with such baselines in addition to the current recomputation baselines. Comparison with KVPacket also adds trainable parameters, which is not a fully fair comparison.
  • The authors train a separate adapter for each task, but they don’t test how these generalize across different tasks. The authors should show how their fine-tuned adapters can generalize beyond the tasks that are trained on, and report how well these adapters generalize on different domains.
  • The authors should report the fine-tuning steps and practical cost (e.g. GPU hours, and peak training memory) for fine-tuning the models with the proposed method, compared to alternative baselines.
Questions for authors

See the weakness please.

10 comments

Response 1.1: PEFT Baselines and the Distinct Contribution of Query-Side Correction

We thank the reviewer for this important suggestion. We introduce Q-LoRA, O-LoRA, FFN-LoRA, and Q+O-LoRA baselines under identical fixed KV caches, training data, KL-distillation objectives, and comparable parameter budgets.

Our design is motivated by the structure of attention-routing errors under KV-cache reuse. Even when cached values retain relevant information, attention misalignment may prevent the model from accessing it. For fixed reused keys $\tilde K$, the attention logits are $z=\tilde Kq/\sqrt{d}$. Any target logit correction $\Delta z$ can be decomposed as

​Δz=PK~Δz+(I−PK~)Δz,\Delta z=P_{\tilde K}\Delta z+(I-P_{\tilde K})\Delta z,​

where the first component is recoverable through query correction, while the second is irreducible with fixed keys. Thus, query adaptation directly addresses the recoverable routing error without modifying cached KV states. In contrast, O-LoRA and FFN-LoRA primarily provide indirect compensation through output or feature transformations.

Table 1. Parameter-matched PEFT baselines trained with fixed reused KV caches (task scores, %).

MethodTrainable parametersNIAH ↑Biography ↑HotpotQA ↑Macro ↑
Cache-only038.706.3323.8022.94
Standard Q-LoRA1.05M54.7696.9127.8559.84
O-LoRA1.05M59.8697.6728.8662.13
FFN-LoRA1.05M0.002.490.000.83
Q+O-LoRA1.05M54.3896.3030.8960.52
QueryLens-H1.05M61.2598.3837.6765.77

Parameter-efficient adaptation itself helps mitigate the performance degradation caused by KV-cache reuse. Nevertheless, QueryLens-H exhibits more consistent advantages on tasks requiring cross-document information integration and multi-hop reasoning. This pattern is consistent with our mechanism-based interpretation: direct, head-specific correction of attention routing over a fixed KV cache can make more effective use of the available cached information, rather than relying solely on downstream representation transformations to compensate for incorrectly aggregated evidence.

CacheBlend, EPIC, and A3 selectively recompute document tokens to restore representations affected by missing contextual interactions. Although they reduce the cost of full prefill, they still require online processing of a subset of document tokens, whereas the additional overhead of QueryLens is negligible. The inference-efficiency comparison is provided in Table 2.

Table 2. TTFT comparison between native cache reuse and folded QueryLens-H across models and context lengths.

ModelContextMethodBaseline TTFT (ms)+QueryLens-H (ms)ΔTTFT (%)
Qwen-4B8KA3203.5203.50.00%
Qwen-4B8KCacheBlend226.3225.9−0.18%
Qwen-4B8KEPIC113.5113.7+0.18%
Llama-8B8KA3220.3220.30.00%
Llama-8B8KCacheBlend248.6248.60.00%
Llama-8B8KEPIC109.5109.2−0.27%
Qwen-4B32KA31500.71500.2−0.03%
Qwen-4B32KCacheBlend1739.91739.90.00%
Qwen-4B32KEPIC383.8383.80.00%
Llama-8B32KA31459.41459.8+0.03%
Llama-8B32KCacheBlend1734.31736.0+0.10%
Llama-8B32KEPIC350.9350.3−0.17%

We will incorporate these ablation results, parameter-budget controls, and inference-efficiency comparisons into the revised manuscript to clarify the distinct contribution of QueryLens-H.


Response 1.2: Comparison with KVLink

It is already known that fine-tuning can improve performance under KV-cache stitching, as shown by KVLink. The authors seem to be applying this adaptation idea specifically to the query projections through parameter-efficient fine-tuning. If they claim that the practical gains come from where to fine-tune, rather than fine-tuning itself, they should compare with such baselines in addition to the current recomputation baselines.

We thank the reviewer for this important comment. Unlike KVLink, which fine-tunes the entire backbone, QueryLens learns only low-rank query corrections for individual layers and attention heads, with learnable gates controlling their magnitudes. This design directly affects attention routing without updating the backbone parameters.

To evaluate its practical benefits, we adapted QueryLens to Llama-3.2-3B and compared it against the official KVLink5 implementation. For each dataset, both methods were evaluated on the same 100 test examples sampled with seed 42, using identical prompts, greedy decoding, BF16 precision, and a single A100 80GB GPU.

(1) Comparison of Trainable Parameter Counts

Table 3 compares the number of trainable parameters.

Table 3. Trainable parameter counts and adaptation strategies of KVLink5 and QueryLens-H.

MethodTrainable parametersRelative to KVLinkTraining strategy
KVLink53,606,752,256100%Full fine-tuning
QueryLens-H688,8000.02%Frozen backbone; head-wise query adaptation over 28 layers (rank 4)

QueryLens uses approximately 1/5,236 as many trainable parameters as KVLink, a reduction of about 99.981%.

(2) Comparison of Downstream Task Performance

Task scores and full-context performance retention are reported in Table 4.

Table 4. Downstream task scores and full-context performance retention of QueryLens-H versus KVLink5.

DatasetQueryLens F1 ↑QueryLens full-context F1Retention ↑KVLink F1 ↑KVLink full-context F1Retention ↑
Biography0.89550.8696102.98%0.88110.974490.42%
NIAH0.73160.793092.26%0.65890.762886.38%
HotpotQA0.38720.620162.44%0.13200.297744.34%
MuSiQue0.15870.278357.02%0.07020.279425.13%
Mean retention——78.68%——61.57%

First, QueryLens achieves higher reused-cache F1 than KVLink5 on all four datasets, with particularly pronounced advantages on the multi-hop reasoning tasks HotpotQA and MuSiQue. Second, its mean full-context performance retention reaches 78.68%, exceeding KVLink's 61.57% by 17.11 percentage points. Importantly, these gains are achieved using only approximately 0.019% as many trainable parameters, without full-backbone fine-tuning.

In summary, KVLink establishes the effectiveness of training-based cache adaptation, while our results further show that constrained query-side correction can achieve competitive, and in these settings better, reused-cache performance with a substantially smaller trainable-parameter budget. This provides additional evidence for lightweight attention-routing correction under fixed KV caches.


Response 1.3: Parameter-Budget Fairness in the Comparison with KVPacket

Comparison with KVPacket also adds trainable parameters, which is not a fully fair comparison.

We thank the reviewer for this comment. We agree that the original combination experiment introduced additional trainable parameters, and have therefore conducted a comparison with matched trainable-parameter budgets. We also clarify that the central contribution of QueryLens is not to propose another cache-repair strategy, but to correct query-side readout errors that existing cache-reuse methods leave unaddressed, providing a complementary mechanism for improving cache utilization.

Existing KV-cache repair methods (CacheBlend, A3, EPIC, KVPacket) improve contextual consistency through selective token recomputation, boundary repair, or enriched cached representations. However, even after cache-side repair, the model may still read the resulting cache with a query projection trained for full-prefill representations, leaving residual attention-routing mismatch. QueryLens exploits this overlooked degree of freedom on the readout side, applying lightweight query correction without modifying the existing KV cache or increasing the recomputation budget.

As formalized by Proposition 1 in Section 4, for a fixed key matrix, the attention-logit distortion can be decomposed into a query-recoverable component and an irreducible component. Even after a cache has been constructed or repaired, a residual query-recoverable component may remain that can be eliminated by modifying the query. Cache-side repair and query-side correction are thus complementary—acting on cache contents and information retrieval, respectively—allowing QueryLens to serve as a separate calibration module alongside several cache-reuse methods under the same cache budget.

To eliminate the effect of additional trainable parameters, we designed a strictly parameter-matched comparison with KVPacket: we reduced its head/tail soft tokens from 8 to 4 and allocated the released budget to QueryLens-T, so that the combined method and the original KVPacket have identical trainable-parameter counts, using the same models, training data, and evaluation protocol. Table 5 presents the results.

Table 5. Parameter-matched KVPacket and KVPacket + QueryLens-T results on two backbone models (task scores, %).

ModelMethodTrainable parametersBiography ↑NIAH ↑HotpotQA ↑MultiFieldQA-en ↑
Llama-3.1-8BKVPacket65,53696.1380.2141.9046.97
Llama-3.1-8BKVPacket + QueryLens-T65,53698.8481.2142.9348.86
Qwen3-4BKVPacket40,96081.9446.7028.1433.12
Qwen3-4BKVPacket + QueryLens-T40,96088.2045.1428.7233.86

Under exactly matched parameter counts, the combined method improves performance in 7 of 8 task settings (all four on Llama-3.1-8B; on Qwen3-4B, Biography improves by 6.26 points, with NIAH the sole modest decline of −1.56 points). These gains are therefore not simply due to adding more parameters, but are consistent with addressing readout-side errors that cache construction alone does not correct.

Together with our results on CacheBlend, A3, and EPIC, these findings support the applicability of QueryLens across different cache-reuse strategies: through the shared query–KV interface, it can calibrate caches produced by different methods and improve the accuracy–efficiency trade-off without additional cache-side recomputation. The core contribution of QueryLens is thus to extend cache-reuse error correction from cache-side reconstruction alone to two complementary dimensions—cache-side repair and query-side calibration—and our parameter-matched experiments show that this complementarity yields practical gains without increasing the trainable-parameter budget.

Comments on the paper

3 comments
Area_Chair_JgByOpenReview ↗

Reviewers: please respond to the author rebuttal

Dear reviewers,

The authors have posted responses to your reviews. The author-reviewer discussion closes on November 18 (AoE).

Please read the responses and reply here before then: let the authors know whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Engaging with the other reviews is also welcome.

Thank you, Area Chair

Area_Chair_JgByOpenReview ↗

Final reminder: discussion closes today (Nov 18 AoE)

The author-reviewer discussion closes today, November 18 (AoE).

Dear reviewers,

Thank you to those who have already responded. If you have not yet replied: Please read the authors' response and post a brief reply here before the deadline: whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Your response is important for the final recommendation.

Thank you, Area Chair

AuthorsOpenReview ↗

Summary of the Discussion Phase

Dear Area Chair,

Thank you for handling our submission. We are writing to summarize our paper and the discussion phase, in the hope of reducing your workload as much as possible.

Review and Discussion Status

The initial ratings for our paper were 1, 3, and 2, with an average score of 2.00. During the discussion phase, we sincerely appreciate that Reviewer DJA3 increased their score from 2 to 3 and explicitly expressed support for acceptance. Reviewer 6t2c also confirmed that most concerns had been addressed and maintained their positive rating of 3. Consequently, our current ratings are 1, 3, and 3, with an average score of 2.33.

We have devoted substantial time and effort to this work and conducted extensive additional experiments during the rebuttal period. We sincerely appreciate the reviewers' constructive feedback and would like to further clarify the remaining disagreement regarding our central contribution.

Significant Contributions of QueryLens

QueryLens addresses a fundamental yet largely overlooked problem in KV-cache reuse: even when the cached information remains available, the model may fail to retrieve it correctly due to mismatches between queries and reused KV states.

We believe QueryLens represents an important milestone in KV-cache reuse research. Unlike existing approaches that primarily repair KV caches through recomputation, QueryLens establishes a lightweight, theoretically grounded query-side correction paradigm, formally characterizing the recoverable attention-routing errors and their theoretical limits under fixed KV caches.

Our contribution extends beyond proposing an effective algorithm. More importantly, QueryLens introduces a new correction dimension complementary to existing cache-side repair methods. We believe this perspective provides an important foundation for future research on query-side correction and may inspire a broader family of efficient KV-cache reuse methods.

Positive Feedback from Reviewers

We sincerely appreciate the reviewers' recognition of the novelty and practical significance of QueryLens:

  • Reviewer 6t2c: Recognized the method as innovative, simple yet effective, with convincing experimental results and straightforward integration into existing KV-cache reuse methods. During the discussion, the reviewer confirmed that most concerns had been resolved.
  • Reviewer DJA3: Acknowledged the practical performance gains, compatibility with existing methods, and negligible inference overhead. The reviewer explicitly stated that our additional APE comparisons, parameter-matched PEFT baselines, and reliability analyses addressed most concerns, and consequently increased the rating from 2 to 3 in support of acceptance.

Regarding Reviewer TRei's remaining concerns, we fully respect the reviewer's perspective. However, we respectfully believe that the remaining disagreement stems primarily from interpreting QueryLens as another PEFT method, whereas our central contribution is to establish a new correction paradigm for KV-cache reuse. Following the reviewer's suggestions, we independently designed and implemented parameter-matched PEFT baselines under identical training conditions, demonstrating that QueryLens outperforms the evaluated conventional PEFT alternatives. The reviewer subsequently requested further comparisons combining different PEFT methods with multiple KV-cache recomputation strategies. While we acknowledge the value of these extensions, we respectfully believe that they constitute a broader investigation of PEFT and cache-repair combinations, extending beyond our primary research objective of establishing the theoretical foundation and practical effectiveness of query-side correction.

We would be deeply grateful if you could consider the reviewers' positive assessments, the distinction between these research objectives, and the substantial efforts we have devoted to theoretical analysis, algorithm development, and additional experiments. We sincerely hope that QueryLens can be evaluated in light of its fundamental problem formulation, paradigm-level contribution, theoretical grounding, and practical significance.

Thank you very much for your help and consideration.

Best regards,

The Authors of Submission 5508

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.