acceptodds
Withdrawn from ICLR 2027

LatchQ: Latching onto Hot-Cache Query Interactions for KV Cache Quantization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

KV cache quantization reduces the memory footprint of large language model (LLM) inference, but minimizing reconstruction error alone may fail to preserve attention behavior. Although future queries are unavailable at quantization time, keys can interact with subsequent queries while residing in the full-precision hot cache. Motivated by this observation, we introduce ***LatchQ***, which uses these observed interactions to guide key clipping while preserving the baseline bit allocation. *QTrace* accumulates attention-weighted query energy for each key and channel, preserving differences in observed key usage and channel sensitivity. *QClip* leverages this evidence to select shared clipping ranges by minimizing interaction-weighted reconstruction error. Our method preserves the baseline storage format and cache schedule without extending hot-cache residency. Our analysis reveals 63% agreement between clipping range selections based on hot-cache observations and those based on both observed and future queries. Furthermore, ***LatchQ*** achieves the highest average LongBench scores among evaluated KV quantizers at 3.0 and 2.25 bits per dimension (bpd). It also outperforms the baseline quantizer on reasoning and code generation. Code is included in the supplementary material, and the project page is available at https://anonymous-latchq.github.io/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, KV Cache, Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RxhdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LatchQ, a query-interaction-aware KV cache quantization method designed to improve quantization quality without changing the baseline bit allocation or compressed KV storage format. The central motivation is that minimizing key reconstruction error does not necessarily preserve attention behavior, as different reconstruction errors can have different impacts on subsequent queries. The authors argue that query–key interactions observed during full-precision hot-cache residency can provide useful evidence for selecting quantization parameters.

LatchQ consists of two components. QTrace accumulates attention-weighted query energy for each key and channel, constructing a query-sensitive reconstruction-error metric. QClip subsequently uses these statistics to select clipping ranges from 25 candidates by minimizing interaction-weighted reconstruction error. The method preserves the baseline 2-bit key quantization scheme, value quantization, and nominal cache schedule.

The authors evaluate LatchQ on LongBench, WikiText, GSM8K, MATH-500, GPQA Diamond, and LiveCodeBench using several 7B–8B language models. The experiments report improved accuracy over existing KV cache quantization baselines at 3.0 and 2.25 bits per dimension, together with approximately 63% agreement between clipping decisions based on observed interactions and those obtained using an oracle with access to both observed and future queries.

However, the reported improvements introduce substantial computational overhead, particularly during prefill. Moreover, the evaluated implementation relies on full-prompt query observations during prefill and reports runtime measurements only at batch size one. These limitations raise significant questions about whether the proposed method can provide comparable benefits under practical incremental and high-throughput inference workloads.

Critical strengths
  1. An interesting perspective on query-aware KV cache quantization.

    The paper identifies a meaningful limitation of conventional reconstruction-error-based quantization: reconstruction error alone does not necessarily reflect its impact on attention behavior. The proposed QTrace and QClip components provide a reasonably motivated approach to incorporating observed query–key interactions into clipping decisions. Unlike methods that modify precision allocation, LatchQ improves the quantization objective while retaining the baseline bit allocation and storage format. This offers an interesting perspective on utilizing runtime query information for KV cache compression.

  2. Empirical improvements under the evaluated compression settings.

    The experiments demonstrate improvements over KIVI-2 and other evaluated baselines across long-context and reasoning benchmarks, particularly at the more aggressive 2.25-bpd setting. The ablation studies further support the effectiveness of combining attention weights and query-channel energy rather than relying solely on unweighted reconstruction error. These results suggest that interaction-aware clipping can improve quantization quality under the evaluated conditions.

Critical weaknesses
  1. Dependence on full-prompt observations limits online applicability.

    Appendix A.1.2 (Eq. 10) allows QTrace to observe all subsequent prompt queries before quantizing a key. However, in incremental or chunked prefill, earlier KV blocks may need to be compressed before these queries are available. The paper does not explain how LatchQ operates under such constraints without delaying quantization or retaining additional full-precision KV states. This raises concerns about the claimed preservation of the baseline hot-cache residency and quantization schedule.

  2. Insufficient validation of decoding-time interaction evidence.

    During decoding, QTrace only observes subsequent queries within the scheduled 128-token block (Eq. 11). Consequently, keys near the end of a block receive very limited evidence, and the final key receives none. The paper does not establish whether these short observation windows reliably predict interactions with future queries after quantization. It is therefore unclear whether the reported improvements primarily depend on the more extensive observations available during prefill.

  3. Lack of evaluation under realistic multi-request batching.

    The runtime experiments are restricted to batch size one. In high-throughput LLM serving, QTrace statistics and QClip decisions would need to be maintained independently across concurrent requests, introducing additional computation and memory overhead. The paper provides no throughput or scalability evaluation under multi-request or continuous batching, leaving its practicality for large-scale deployment unverified.

  4. Substantial computational overhead undermines practical efficiency.

    Table 8 reports a 3.37× prefill slowdown at 32,768 tokens (4.532 s to 15.258 s), alongside a 6.9% increase in decoding latency. Appendix B.4 further reports approximately 2.0 GB of additional peak memory during prefill. These costs are substantial relative to the reported accuracy improvements and challenge the practical value of the method for efficient inference.

  5. Missing end-to-end evaluation with compressed KV storage.

    The runtime experiments use a quantize–dequantize framework that retains KV entries in FP16 rather than an integrated inference system with packed quantized KV storage. Consequently, the reported memory savings are calculated from the storage format, while the actual end-to-end performance and memory trade-offs remain unverified. This further limits the practical significance of the evaluation.

Questions for authors
  1. Compatibility with online and chunked prefill.

    Appendix A.1.2 (Eq. 10) collects interactions from all subsequent prompt queries before quantization. How can LatchQ operate when earlier KV blocks must be compressed before these queries are computed? Can the authors evaluate the method under strictly online quantization without extending full-precision KV residency?

  2. Effectiveness of decoding-time observations.

    During decoding, each key observes at most 127 subsequent queries within its 128-token block, with some keys receiving little or no evidence. How reliably do these observations predict future interactions? Can the authors separately ablate prefill-time and decoding-time QTrace to determine their respective contributions?

  3. Scalability under multi-request batching.

    How are QTrace statistics and QClip decisions managed across concurrent requests with different sequence lengths and quantization schedules? Can the authors provide throughput, latency, and memory measurements at larger batch sizes (e.g., 4, 8, and 16)?

  4. Justification of computational overhead.

    Given the reported 3.37× prefill slowdown and additional memory consumption, under what inference workloads do the accuracy improvements justify these costs? Can the authors provide a more complete accuracy–latency–memory trade-off analysis?

  5. End-to-end inference performance.

    Can the authors evaluate LatchQ in an actual inference system using packed quantized KV storage rather than the current quantize–dequantize framework? Such results are necessary to verify whether the claimed storage benefits translate into practical inference efficiency.

Reviewer_Tp9TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LatchQ improves KV-cache quantization by using query–key interactions observed while keys reside in the hot cache. QTrace accumulates attention- and query-energy-based statistics that weight key reconstruction errors. QClip then uses these statistics to select clipping ranges under the existing quantization format and cache schedule. The evaluation reports improvements on long-context, reasoning, and code-generation benchmarks while retaining the base quantizer's storage allocation.

Critical strengths
  1. A clearly motivated attention-aware surrogate.

The paper connects a local KL expansion for single-key perturbations to an implementable reconstruction surrogate. It explicitly describes the diagonal approximation and the replacement of the curvature factor a(1−a) with attention weight a, making the approximations inspectable.

  1. Evidence that observed queries inform later quantization choices.

The reported 63% agreement with an all-query oracle and the component ablations support the usefulness of hot-cache query evidence. The combined QTrace/QClip results improve over the underlying quantization schedule on the tested tasks.

Critical weaknesses
  1. The Table 4 full variant and the weighting substitution need separate clarification.

Page 4 and Appendix A.1.1 explicitly define the standard diagonal method using a instead of a(1−a). My reproducibility question concerns the cross-channel “LatchQ (full)” variant in Table 4. Its complete clipping objective and the coefficient used when retaining cross-channel terms should be stated explicitly. The substitution from curvature to attention weighting can change candidate rankings, so an isolated comparison would help assess its contribution. The supplementary code also uses “full” to name the all-component diagonal statistic; this should be distinguished from the table’s cross-channel variant.

  1. The severity of QClip-only degradation deserves a more direct diagnosis.

Table 3 shows GSM8K decreasing from 72.86 to 68.17 and MATH-500 from 39.00 to 23.06 when clipping uses uniform weights without QTrace. The paper explains that tensor reconstruction error ignores query–key interactions. This is plausible, and lower tensor MSE need not improve downstream accuracy. Measurements of reconstruction and attention/output errors would help diagnose why the selected clipping ranges hurt relative to the original quantizer.

Overall assessment: I lean toward weak acceptance. The use of observed query interactions is well motivated and supported by useful ablations. Clarifying the full variant and diagnosing the uniform-weight failure would strengthen reproducibility and the interpretation of the surrogate.

Questions for authors
  1. Please give the exact objective and reproduction entry point for the cross-channel LatchQ (full) row in Table 4. Does it use a or a(1−a)? Can the two weighting choices be compared while holding the cross-channel treatment fixed?

  2. For QClip-only versus the baseline, what happens to actual key MSE, attention-distribution or output error, and saturation/outlier behavior? Which measurements explain the task degradation?

Reviewer_KPbpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LatchQ, which uses query–key interactions observed before compression to guide KV cache quantization. QTrace collects attention-weighted query energy, and QClip uses these statistics to select clipping ranges while preserving the baseline bit allocation and storage format. Experiments demonstrate accuracy improvements on long-context understanding, reasoning, and code generation.

Critical strengths

S1: KV quantization is an important research problem.

S2: Query–key interactions observed before compression provide a sensible signal for quantization sensitivity. LatchQ uses this information to adapt clipping without changing the baseline precision allocation or cache schedule. This focused design could complement other KV compression techniques.

Critical weaknesses

W1: Incomplete validation of practical efficiency. Although latency is measured, the evaluation retains FP16 KV caches in a quantize–dequantize framework. The reported compressed storage reductions are therefore calculated rather than demonstrated in deployment. Moreover, prefill takes 2.46–3.37× as long as KIVI-2, and decoding latency increases by 4.7%–6.9%. More realistic evaluation is needed to establish actual memory savings and end-to-end performance.

W2: The reported 63.4% agreement compares observed-query selections with an oracle using both observed and future queries. Because these selections share evidence, their agreement does not isolate prediction of future-only preferences. Although the NLL comparison against a future-query oracle provides supporting evidence, direct future-only agreement and candidate-frequency baselines would strengthen the claim.

W3: reasoning experiments use only Llama-3.1-8B-Instruct, without reported additional reasoning-trace fine-tuning or generation-length statistics. The benchmark scores therefore do not establish effectiveness on reasoning models producing long outputs.

Questions for authors

Q1: What peak memory and end-to-end latency does LatchQ achieve with really implemented quantized KV caches?

Q2: What are the reasoning-output lengths, and do the gains extend to specialized long-form reasoning models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.