acceptodds
Under review as a conference paper at ICLR 2027

Think Global, Act Local: Efficient KV Cache Quantization via Global Prefill Calibration of Early Decoding

Abstract

Efficient mixed-precision KV cache quantization requires identifying which cached tokens will remain important during future decoding, before aggressively compressing them. To address this challenge, we propose ForeKV, an efficient KV cache quantization framework that estimates future token importance from early decoding queries and calibrates this local estimate with global query information from prefill. Our key insight is that the first few decoding queries provide the earliest direct evidence of the model's emerging attention behavior, but such evidence is limited by the short observation window. Therefore, we delay compression for a few decoding steps to collect real decoding queries, estimate token importance from these local observations, and calibrate the estimate using the mean query over the global prefill sequence. The resulting query directly scores cached keys to guide mixed-precision allocation without materializing the full attention matrix. We further design a paged attention kernel for K caches with token-wise K4/K2 allocation and channel-wise quantization, enabling efficient decoding directly over the compressed cache. Experiments across multiple LLM backbones on LongBench show that ForeKV remains close to BF16 at an effective compressed-cache storage ratio of approximately , with the average score dropping by no more than approximately 1%. On Qwen3-8B, our end-to-end ForeKV implementation achieves up to the generation throughput of the BF16 baseline under the same 80 GB GPU memory budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache quantization, long-context inference, mixed-precision quantization, efficient LLM inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WVNAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ForeKV, a mixed-precision KV-cache quantization method. It combines early decoding queries with the mean prefill query to estimate token importance, assigning INT4 to selected keys and INT2 to the remaining keys and all values. A paged attention kernel operates directly on the compressed cache.

Critical strengths
  1. Using real decoding queries before assigning precision is a simple and sensible design.
  2. The paper addresses both cache compression and the cost of using the compressed cache during decoding.
  3. Experiments cover several model families and both LongBench and RULER.
Critical weaknesses
  1. The gain from global prefill calibration is small in Table 2: only 0.19 points over decoding-only estimation. No error bars or confidence intervals are reported for this comparison.
  2. The query estimate stays fixed, and previously compressed keys are not reassigned precision. Performance may suffer if attention shifts later in generation.
  3. The throughput results depend on different kernels and batch sizes, making it difficult to tell how much the precision allocation itself contributes.
  4. The weighting rule assumes that agreement among early query directions makes them a better predictor of future attention, but this assumption needs more evidence.
Questions for authors
  1. Does adding the prefill mean consistently improve over decoding-only estimation across models and tasks? Are the gains statistically significant?
  2. How well does ForeKV handle attention shifts during long generations? Would updating the query estimate help?
  3. Could the authors report latency at the same batch size and compare allocation strategies with the same attention kernel?
3 comments

Does adding the prefill mean consistently improve over decoding-only estimation across models and tasks? Are the gains statistically significant?

W1-Q1. Effectiveness of Global Prefill Calibration

Thank you for your careful review and for pinpointing the issue with the original ablation.

We extended the key-selection ablation in Table 2 to Qwen2.5-3B-Instruct and Qwen3-4B on LongBench-11 and RULER-4K. Both methods use the same precision budget, attention kernel, and compression delay (Td=8T_d=8). We used 2,000 paired bootstrap resamples to estimate 95% confidence intervals for the differences in macro-average scores.

BackboneBenchmarkDecode MeanForeKVForeKV − Decode Mean (points) [95% CI]
Qwen2.5-3BLongBench-1143.7943.99+0.200 [+0.084, +0.314]
Qwen2.5-3BRULER-4K83.1384.44+1.310 [+0.006, +2.679]
Qwen3-4BLongBench-1152.0152.13+0.120 [+0.020, +0.219]
Qwen3-4BRULER-4K93.1493.20+0.060 [−0.166, +0.314]

Adding the prefill mean improved the average score in all four settings. The 95% confidence intervals exclude zero on both benchmarks for Qwen2.5-3B and on LongBench-11 for Qwen3-4B. The smaller gain on RULER-4K for Qwen3-4B has an interval that spans zero.

How well does ForeKV handle attention shifts during long generations? Would updating the query estimate help?

W2-Q2. Query Updates during Long Generation

Thank you for pointing out that a fixed query estimate may miss shifts in attention during long generations. We agree and plan to adopt periodic query updates in a future version of ForeKV. When packing a 128-token buffer, ForeKV-Update recomputes the estimate from the eight most recent decoding queries in that buffer and uses it to assign precision to the new keys.

Oracle Gap. We compared fixed and periodically updated query estimates on the same BF16 generation traces from Qwen3-8B with thinking enabled, using a top-40% selection budget.

MethodSteps 129–256Steps 257–512Steps 513–1024
ForeKV (fixed query)39.14%37.03%34.46%
ForeKV-Update26.43%25.93%26.42%

Periodic updates reduced the Oracle Gap across the three intervals, bringing the selection of new keys closer to the oracle.

GSM8K accuracy. We evaluated the same model on all 1,319 test questions. Both quantized variants used 40% K4, 60% K2, and V2.

MethodAccuracyAvg. Output TokensOutputs ≥512 TokensOutputs ≥1,024 TokensOutputs ≥2,048 Tokens
BF1687.64%2,070.521,3191,173501
ForeKV (fixed query)88.86%1,969.251,3171,120468
ForeKV-Update88.63%1,952.591,3161,120446

ForeKV-Update's overall accuracy was closer to the BF16 baseline. It answered three fewer questions correctly than the fixed variant, a drop of 0.23 percentage points. One possibility is that the update makes attention distributions more similar to those of BF16, producing more similar behavior on some questions.

Could the authors report latency at the same batch size and compare allocation strategies with the same attention kernel?

W3-Q3. Efficiency at Matched Batch Sizes and Kernels

Matched batch sizes. We use the Fig. 6 workload with Qwen3-8B, a 1,183-token prompt, and 4,096 generated tokens per request.

Throughput (tokens/s)

MethodBS 8BS 16BS 32BS 64BS 96BS 128BS 160BS 192BS 216BS 224
BF16294.96434.97531.90———————
ForeKV207.30399.62706.551,137.291,315.301,476.311,568.481,557.431,655.89—
KIVI INT2145.75271.28374.87441.00463.49—————
ZipCache48.7053.6256.2757.8157.7157.6056.83———
ChanMix102.34144.12175.92201.32207.94212.15214.45215.08—216.54

Decode time (s per batch)

MethodBS 8BS 16BS 32BS 64BS 96BS 128BS 160BS 192BS 216BS 224
BF16111.07150.63246.36———————
ForeKV158.03163.96185.46230.44298.88355.05417.73504.83534.17—
KIVI INT2224.76241.52349.56594.29848.17—————
ZipCache672.691,221.912,328.654,533.636,811.649,100.1611,529.12———
ChanMix320.12454.61744.901,301.811,890.552,470.743,055.303,655.60—4,236.10

ForeKV has lower decode times than each quantized baseline at every shared batch size. Fig. 6 plots throughput against batch size for these methods.

Matched attention kernel. We compared four precision allocation strategies using the same ForeKV fused attention kernel, cache format, and precision budget at BS 16, 32, and 64. Each result comes from one timed run after one warm-up. End-to-end timing includes host-to-device transfer, prefill, and generation.

MethodBS 16 Time (s)BS 16 Throughput (tokens/s)BS 32 Time (s)BS 32 Throughput (tokens/s)BS 64 Time (s)BS 64 Throughput (tokens/s)
Random 40%158.70412.97189.03693.39240.221,091.29
Prefill Mean158.66413.06189.19692.82239.151,096.14
Decode Mean158.39413.75191.39684.85240.361,090.61
ForeKV159.25411.52190.09689.54240.411,090.40

With the kernel held fixed, the fastest and slowest selectors differed in throughput by at most 1.25% at each batch size. This suggests that the speedup over the quantized baselines comes primarily from the fused attention kernel designed to operate directly on ForeKV’s packed KV cache.

Thank you for your response. Efficiency and query experiments address my concern. I will keep my weak acceptance rating.

Reviewer_Qdo9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes ForeKV, a mixed-precision KV cache quantization framework that aims to predict which cached tokens will be important during future decoding. Instead of compressing the KV cache immediately after prefill, ForeKV delays compression for the first few decoding steps and uses the observed queries to estimate future token importance. The mean of these early decoding queries is calibrated using the global prefill query mean, with a directional-consistency measure controlling their relative contributions. Based on the estimated token importance, ForeKV assigns INT4 precision to selected important keys and INT2 precision to the remaining keys, while uniformly quantizing values to INT2. The paper also introduces a customized paged attention kernel to efficiently process the mixed-precision KV cache. The authors evaluate ForeKV on LongBench and RULER across several Llama and Qwen models.

Critical strengths

S1. Intuitive motivation and reasonable importance estimation strategy. Using real early decoding queries to estimate future token importance is intuitive and potentially useful. Compared with relying exclusively on prefill statistics, early decoding queries provide direct information about the attention behavior emerging during generation.

S2. Practical mixed-precision KV cache implementation. The paper combines token-wise precision allocation with channel-wise key quantization and introduces a customized paged attention kernel. The ability to directly perform attention over compressed mixed-precision KV caches without reconstructing the full-precision cache is a useful systems contribution.

S3. Reasonably broad evaluation across model families. The authors evaluate the approach on multiple Llama, Qwen2.5, Qwen3, and Qwen3.5 models, including models up to 27B parameters. The reported results suggest that aggressive KV cache compression can preserve much of the original model performance while improving memory efficiency and generation throughput.

Critical weaknesses

W1. Limited Algorithmic Novelty and Marginal Improvements My primary concern is the limited algorithmic novelty of the proposed method. The underlying quantization approach is largely based on existing techniques, including asymmetric K/V quantization, channel-wise key quantization, and importance-aware mixed-precision allocation. ForeKV does not introduce a fundamentally new quantization algorithm. Instead, its main contribution is an importance estimation strategy that determines which cached tokens should receive higher precision. However, the proposed importance estimator is relatively simple. It combines the mean of early decoding queries with the global prefill query mean through a directional-consistency-based interpolation. Similar ideas of predicting future query relevance and allocating KV cache resources according to token importance have already been explored in prior work. More importantly, the empirical improvement attributable to the proposed global calibration mechanism is very limited. The improvements in future-importance estimation shown in Figure 1(b) are also relatively small compared with using early decoding queries alone. Furthermore, on Qwen3-8B, ForeKV achieves a LongBench average score of 55.82, compared with 55.71 for KIVI under similar KV cache compression ratios. These results raise an important question: Does the proposed global calibration mechanism provide sufficient additional value over a simple early-decoding-based importance estimator? I do not find the current experimental evidence convincing enough to establish the significance of the main algorithmic contribution.

W2. Missing Comparisons with Recent Strong KV Cache Quantization Baselines The experimental comparison does not adequately cover recent strong methods in KV cache quantization. The primary baselines are KIVI, ZipCache, PolarQuant, and ChanMix. While these are relevant methods, several more recent and closely related approaches are missing, including:

• Kitty (MLSys 2026): An efficient low-bit KV cache quantization method with dynamic precision adjustment and a customized inference implementation.

• MixKVQ (ACL 2026): A query-aware mixed-precision KV cache quantization method, particularly relevant to the importance estimation strategy proposed in ForeKV.

• PM-KVQ (ICLR 2026): A progressive mixed-precision KV cache quantization approach targeting long-chain-of-thought generation.

• KVmix (AAAI 2026): A mixed-precision KV cache quantization approach based on layer importance.

Although these methods differ in their allocation granularity or implementation requirements, they are highly relevant to the claimed algorithmic or system-level contributions of ForeKV. The omission of Kitty is particularly concerning because the paper's efficiency evaluation adopts the GSM8K prompt used in Kitty's latency benchmark, yet no direct comparison with Kitty is reported. Without comparisons against these methods under appropriately matched memory budgets, model configurations, and evaluation settings, it is difficult to assess whether ForeKV provides meaningful advantages over recent state-of-the-art approaches.

W3. Insufficient Validation of Future Token Importance Prediction, Especially for Long Generation Another major concern is the insufficient validation of the central assumption behind ForeKV. The method uses only the first eight decoding queries to estimate future token importance. After the initial calibration, the query representation remains fixed and is reused to determine the precision allocation of subsequently generated keys. However, attention patterns may evolve substantially during generation. Tokens that appear unimportant in the early decoding stage may become important much later, particularly when generation involves multiple stages of reasoning or shifts between different topics. Once a cached key has been assigned low precision, ForeKV does not revisit its precision allocation. Therefore, errors in the initial importance estimation may persist throughout the generation process. Although Figure 1(b) evaluates future-importance estimation, the analysis only considers decoding windows up to step 192. This is insufficient to establish whether the initial query representation remains reliable over thousands of generated tokens. The paper includes a 4096-token generation benchmark for throughput evaluation, but this does not directly demonstrate that the method preserves generation quality or accurately predicts future key importance over long outputs. I believe this is a major limitation because the reliability of future token importance prediction is the central motivation of the proposed method. A more convincing evaluation should examine how prediction quality changes across substantially longer decoding windows, such as 512, 2048, 4096, and 8192 tokens. It should also evaluate the downstream accuracy impact of using a fixed initial importance estimator.

W4. Incomplete Analysis of Additional Computational Overhead and Attribution of Speedup The proposed framework introduces additional operations beyond standard KV cache quantization, including collecting early decoding statistics, scoring cached keys using calibrated queries, selecting important keys, reorganizing tokens, and packing the mixed-precision cache. Although the key-scoring computation avoids materializing the full attention matrix, it still introduces additional computation across KV heads and Transformer layers. The cost of scoring, selection, and cache reorganization may become more significant as the input context length increases. Furthermore, delaying compression requires temporarily retaining full-precision KV states during the initial decoding steps, potentially affecting peak memory usage for long prompts and large batches. The paper reports end-to-end throughput and provides limited measurements of the effect of the early-decoding observation window. However, it does not provide a sufficiently detailed latency breakdown of the additional operations introduced by ForeKV. There is also a concern about the attribution of the reported throughput gains. According to Appendix F, some baselines reconstruct dense KV tensors before attention, whereas ForeKV uses a customized fused attention kernel that directly processes the compressed cache. Therefore, the reported 3.1× throughput improvement may reflect differences in kernel implementation and memory access patterns, rather than advantages of the proposed importance estimation algorithm itself. A more detailed computational overhead analysis and appropriate kernel-level comparisons are necessary to distinguish the algorithmic contribution from the systems contribution.

W5. Limited Evaluation on Modern Reasoning Models and Insufficient Separation of Long-Context and Long-Generation Settings Although the paper includes several recent Qwen3 and Qwen3.5 backbones, the evaluation lacks dedicated experiments on modern reasoning-oriented models, such as Qwen3-4B-Thinking-2507, or larger reasoning-oriented models. More importantly, I believe the evaluation should clearly distinguish between long-input and long-output settings, as they introduce different challenges. For long input contexts, the main concerns are whether importance estimation remains reliable over large KV caches and whether the additional selection and compression operations introduce significant time or peak-memory overhead. For long generation, the main concern is whether the fixed early-decoding query representation remains predictive as attention patterns evolve throughout generation. Thinking models provide another important evaluation dimension and should not be treated as equivalent to long generation. Their attention behavior should be evaluated across different input lengths and generation workloads. I encourage the authors to include more systematic experiments covering different input context lengths and output lengths, ideally on modern reasoning-oriented models.

W6. Limited Clarity and Informativeness of the Figures Several figures do not effectively communicate the main technical ideas. Figure 1(a) visualizes changes in token importance but does not provide a sufficiently intuitive explanation of the evolving attention patterns. Furthermore, Appendix A.1 indicates that this visualization is based on a single constructed prompt and one selected layer/query head, which limits its value as general evidence for the motivation. Figure 3 attempts to illustrate the entire pipeline, including query calibration, importance scoring, token selection, cache reordering, and quantization kernels. However, the diagram is overly dense, making it difficult to understand the exact relationships between these operations. The paper would benefit from clearer and more technically precise figures, particularly a step-by-step visualization of the importance estimation, precision allocation, and quantization procedures.

Questions for authors

Q1. What is the actual benefit of global prefill calibration over using early decoding queries alone? Table 2 shows only a 0.19-point improvement from the proposed calibration. Could the authors provide per-model and per-task ablations, ideally with variability estimates, to demonstrate whether this improvement is consistent and meaningful? This is important for assessing the novelty and effectiveness of the main algorithmic contribution.

Q2. How does ForeKV compare with recent strong KV cache quantization methods? Could the authors provide comparisons with Kitty, MixKVQ, and PM-KVQ, especially under matched effective KV cache memory budgets and model settings? In particular, how does the throughput of ForeKV compare with Kitty, given that the efficiency evaluation uses a workload adopted from Kitty's latency benchmark?

Q3. How reliable is the fixed early-decoding importance estimator during long generation? The current future-importance analysis only covers the first 192 decoding steps. How does the agreement between ForeKV-selected keys and future-attention-based oracle selections evolve over much longer generation windows? More importantly, does the quality of generated outputs deteriorate as the generation length increases? Results on modern reasoning models, including Qwen3-4B-Thinking-2507, would be particularly informative. The experiments should distinguish long-input settings from long-output settings.

Q4. What is the end-to-end computational overhead introduced by ForeKV, and how much of the reported speedup is attributable to its importance estimation strategy? Could the authors report separate latency measurements for early-query collection, importance scoring, TopK selection, KV reordering, quantization/packing, and compressed-cache attention? Additionally, could the authors clarify how much of the reported throughput advantage comes from the customized fused attention kernel rather than the proposed token selection strategy? A comparison using the same ForeKV attention backend with different importance estimators would help isolate the algorithmic contribution.

8 comments

Q1. What is the actual benefit of global prefill calibration over using early decoding queries alone?

W1-Q1. Effectiveness of Global Prefill Calibration

Thank you for your careful review and for pinpointing the issue with the original ablation.

We extended the key selection ablation to Qwen2.5-3B-Instruct and Qwen3-4B on LongBench-11 and RULER-4K. All five selectors share the same precision budget, attention kernel, and compression delay (T_d=8). Prompt-end Mean averages the last eight prefill queries.

MethodQwen2.5-3B LongBench-11Qwen2.5-3B RULER-4KQwen3-4B LongBench-11Qwen3-4B RULER-4K
Random 40%43.1480.7551.9392.52
Prefill Mean43.5282.4651.9792.98
Prompt-end Mean43.6282.6052.0293.09
Decode Mean43.7983.1352.0193.14
ForeKV43.9984.4452.1393.20

We used 2,000 paired bootstrap resamples to estimate 95% confidence intervals for the differences in macro-average scores.

Backbone / BenchmarkForeKV − Decode Mean [95% CI]ForeKV − Prompt-end Mean [95% CI]
Qwen2.5-3B / LongBench-11+0.200 [+0.084, +0.314]+0.370 [+0.211, +0.522]
Qwen2.5-3B / RULER-4K+1.310 [+0.006, +2.679]+1.840 [+0.468, +3.193]
Qwen3-4B / LongBench-11+0.120 [+0.020, +0.219]+0.110 [−0.019, +0.234]
Qwen3-4B / RULER-4K+0.060 [−0.166, +0.314]+0.110 [−0.155, +0.427]

On Qwen2.5-3B, the intervals exclude zero for both comparisons on both benchmarks. The gains are smaller on Qwen3-4B, where only the LongBench-11 comparison with Decode Mean excludes zero.

Q2. How does ForeKV compare with recent strong KV cache quantization methods?

W2-Q2. Comparison with Recent Baselines

We evaluated Kitty, MixKVQ, and PM-KVQ on Llama-3.2-3B-Instruct (Llama) and Qwen2.5-3B-Instruct (Qwen). The tables report the individual task scores and their macro averages.

LongBench-11

BackboneMethodpassage_retrieval_enlccrepobench-photpotqamusique2wikimqasamsumtrecqaspermultifieldqa_enmulti_newsMacro average
LlamaKitty96.0045.4849.8953.2925.6741.1942.5371.5040.1251.3325.8649.35
LlamaMixKVQ95.5045.4049.8853.2725.6540.9543.4972.0039.9449.7226.2149.27
LlamaPM-KVQ96.0044.8049.4053.3126.2940.7942.0572.0040.8950.7126.0249.30
LlamaForeKV97.5051.9154.0853.2825.0239.8442.1771.0040.1150.5125.8250.11
QwenKitty46.0049.6552.0846.6319.4135.7143.9969.5037.3247.2922.7242.75
QwenMixKVQ46.0049.7052.1846.4820.7735.9544.3069.5036.6348.2622.9342.97
QwenPM-KVQ45.5048.8049.2143.7219.2132.2343.3868.5036.0146.8224.0541.58
QwenForeKV50.0056.4956.5646.6720.8335.8245.0568.5036.1046.1021.8243.99

RULER-4K

BackboneMethodcwefweniah_multikey_1niah_multikey_2niah_multikey_3niah_multiqueryniah_multivalueniah_single_1niah_single_2niah_single_3qa_hotpotqaqa_squadvtMacro average
LlamaKitty76.6093.0099.00100.0099.0099.0098.7599.00100.00100.0051.0090.0097.0092.49
LlamaMixKVQ77.8094.67100.00100.0099.0097.50100.0099.0099.00100.0053.0089.0098.2092.86
LlamaPM-KVQ78.3095.33100.00100.0097.0098.7599.7599.0099.0097.0052.0091.0097.2092.64
LlamaForeKV83.9091.33100.00100.0099.0098.7599.5099.00100.0099.0051.0088.0099.2092.98
QwenKitty59.9084.6797.0098.0082.0099.2593.25100.00100.0088.0051.0084.0030.2082.10
QwenMixKVQ67.5083.0097.0088.0059.0097.7588.5099.00100.0081.0052.0086.0026.6078.87
QwenPM-KVQ36.7068.00100.0095.0058.0099.2592.50100.00100.0024.0051.0084.0030.6072.23
QwenForeKV66.6083.67100.0099.0089.0099.7594.75100.00100.0099.0051.0088.0027.0084.44

These experiments extend the evaluation to the channel-wise, query-aware, and progressive mixed-precision methods raised in W2. ForeKV achieves the highest macro-average score among the compared methods on both LongBench-11 and RULER-4K for each backbone.

Q2. How does ForeKV compare with recent strong KV cache quantization methods?

Kitty efficiency comparison

Thank you for suggesting this comparison. Decode throughput is batch size × 4,095 / decode time.

Batch size816326496128160192216248280
Kitty (tokens/s)224.70459.68825.321284.531468.591631.461702.281720.351808.781791.741843.15
ForeKV (tokens/s)207.30399.62706.551137.291315.301476.311568.481557.431655.89——
Kitty decode (s)145.797142.533158.774204.027267.685321.283384.896457.023489.016566.800622.088
ForeKV decode (s)158.034163.955185.464230.442298.882355.048417.730504.830534.170——
Kitty peak nvidia-smi (GiB)10.22112.16616.16623.94331.70739.48447.15654.80960.62568.46176.166
ForeKV peak nvidia-smi (GiB)11.3813.9919.2729.6240.0150.3160.6571.0678.87——

ForeKV's main contribution is a query-based algorithm for assigning precision to individual key tokens. Kitty focuses on accelerating inference with channel-wise mixed precision and a specialized backend. Because the methods use different allocation granularities and execution paths, the original paper did not include a direct efficiency comparison with Kitty. In the comparison above, Kitty achieves 8.4–16.8% higher decode throughput and uses less peak GPU memory across all nine shared batch sizes.

ZipCache also allocates precision by token. For this comparison, ForeKV achieves substantially higher end-to-end throughput (Sec. 4.5), aided by a fused attention kernel that reads packed KV directly.

Reviewer_MJHyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ForeKV, a token-wise mixed-precision KV cache quantization method that decides which cached keys to keep at higher precision based on an estimate of their future attention. Compression is deferred for the first T_d = 8 decoding steps. For each query head, the mean of these early decoding queries (mu_d) is blended with the mean of all prefill queries (mu_p), weighted by the directional concentration c_d of the early queries (mu_hat = c_d * mu_d + (1 - c_d) * mu_p, Eqs. 3-5). Cached keys are scored by softmax(mu_hat^T k / sqrt(d)), taking the max over the query heads that share a KV head (Eqs. 6-7). The top 40% of keys per KV head are stored in INT4 and the rest in INT2 (channel-wise, 32-token pages), values are uniformly INT2, and newly generated tokens are buffered in FP16 and packed every 128 tokens using the frozen mu_hat. A fused paged decode-attention kernel processes the INT4 pool, the INT2 pool and the FP16 buffer with a shared online-softmax state, without reconstructing a dense cache. Accuracy is evaluated on LongBench for 11 Llama/Qwen backbones (3B-27B) and on RULER (up to 32K, four backbones) against KIVI, ZipCache, PolarQuant and ChanMix under a common storage accounting (about 5.0x for ForeKV, App. E), with ablations on Llama-3.2-3B. Efficiency is measured on Qwen3-8B with a 1,183-token prompt and 4,096 generated tokens per request, where ForeKV reaches up to 3.1x the BF16 throughput under an 80 GB budget.

Critical strengths
  1. The compression actually translates into throughput. Tokens are reordered into precision-specific page pools, and both pools plus the FP16 buffer are handled in one decode kernel (Sec. 3.3, App. H.2), so attention reads the packed K4/K2 + V2 cache directly. ForeKV's throughput keeps growing with batch size well after the quantized baselines saturate below BF16, peaking at about 1,656 tok/s (Fig. 1(c), Fig. 6(b)). This matters because token-wise heterogeneous precision is awkward to execute efficiently, and many mixed-precision KV methods report storage savings without any real speedup.

  2. The evaluation is broad and storage-matched. It covers eleven backbones across Llama 2/3 and Qwen2.5/3/3.5 (3B-27B) on LongBench and RULER, and App. E costs every method with the same accounting of payload, quantization parameters and FP16 buffers, so the four ~5x methods are compared at closely matched storage (4.70-4.96x). ForeKV stays within about 1% of BF16 on every backbone (Tables 1 and 4). This makes the robustness claim credible on the storage side, subject to Weakness 1 below.

  3. The importance signal is cheap and informative. Fig. 1(b) suggests that the mean of the first eight decoding queries recovers far more of the future high-attention keys than the global prefill mean (Oracle Gap 20.4 vs. 43.8 for steps 9-64), at O(Nd) scoring cost per head, with memory and runtime essentially flat across T_d (Table 6). The observation is simple and could be reused by other compression schemes.

Critical weaknesses
  1. The deferred-compression window confounds the accuracy comparison (Sec. 3.2, App. G-H, Listing 1; Tables 1, 3, 4, 6). As far as I can tell, the whole cache stays in full precision for the first T_d = 8 decoding steps: Listing 1 packs the cache only after T_d queries, and App. G mentions a "transient full-precision phase". So ForeKV attends over an uncompressed cache while generating its first 8 tokens, whereas KIVI, ZipCache and ChanMix quantize the prompt right after prefill. Many LongBench/RULER answers are only a few tokens long, so a large part of the scored output may be produced before any compression happens, i.e., at an effective ratio of 1x.

The results look like what this would predict. ForeKV is within 1 point of BF16 on PassageRetrieval-en on all 11 backbones, while the baselines drop sharply on several (Qwen2.5-3B: 50.00 vs. BF16 49.00, KIVI 26.00, ChanMix 28.50). On Qwen2.5-3B, ForeKV's largest margin (+2.80 over ChanMix) comes entirely from PR-en (about 70% of the gap) and LCC/RB-P, and on the other eight tasks it is slightly below ChanMix on average. On the long-output MultiNews it shows no advantage and is the lowest ~5x method on Llama-2-7B-Chat and Qwen2.5-3B. In Table 6, accuracy rises with T_d, and T_d = 16 (85.36) even beats storing every key in INT4 at the default T_d = 8 (85.04, Table 3), which better importance estimates alone cannot explain.

The delay is orthogonal to the allocation criterion and could be applied to any baseline. Until that comparison is made, the main tables do not isolate the contribution of future-aware allocation.

  1. The distinctive parts of the estimator are not shown to matter beyond noise (Table 2, Fig. 1(b), Eqs. 3-5, App. A.2). The global prefill calibration that gives the paper its title adds 0.19 points over the plain early-decoding mean (84.56 to 84.75), in a single setting (Llama-3.2-3B, RULER-4K) with no repeated runs or intervals. The paper's own ablations already contain larger differences that importance estimation cannot explain (the inversion noted in Weakness 1). In Fig. 1(b), calibration lowers the Oracle Gap by only 0.5-1.2 points, and App. A.2 does not say which prompts, how many samples, or how layers and heads are aggregated.

The distribution of c_d is not reported either. If a head's consecutive decoding queries are usually well aligned (c_d close to 1), Eq. 5 is close to a no-op.

  1. Accuracy and efficiency are measured in regimes that do not overlap (Sec. 4.1, 4.5; App. F, H). Generated tokens stay in the FP16 buffer until 128 have accumulated. LongBench's standard generation limits are 128 tokens or fewer for every evaluated task except MultiNews (assuming default settings), and RULER answers are short, so generated tokens are essentially never quantized in the accuracy experiments. Assigning precision to generated tokens via the frozen mu_hat is therefore untested.

The throughput benchmark is the opposite case (1,183-token prompt, 4,096 generated tokens). There, most of the cache consists of exactly those tokens, but no accuracy is reported for it. Fig. 1(b) already shows the Oracle Gap growing from 19.9 to 33.1 within 192 steps. The abstract pairs "close to BF16" with "3.1x throughput", but the two numbers come from different settings.

  1. The 3.1x speedup cannot be attributed to ForeKV's design as reported (Sec. 4.5, App. F, Figs. 1(c) and 6). The BF16 baseline's attention kernel and cache allocator are not specified. The quantized baselines run research code with dense reconstruction, and all of them peak below BF16's 532 tok/s (Fig. 1(c)), so the comparison mostly reflects implementation maturity.
Questions for authors

Clarification requests:

  1. Do decoding steps 1..T_d attend to the uncompressed cache, and does any baseline get a similar full-precision window? Please report KIVI, ZipCache and ChanMix with the same T_d = 8 delay (or ForeKV with a quantized cache during the observation window) on at least two backbones. Please also give the per-task generation limits, whether Qwen3/Qwen3.5 thinking was disabled, and the share of scored output tokens generated after compression. If ForeKV keeps its margins under an equal delay, Weakness 1 is resolved and I would raise my rating.

  2. Please extend Table 2 to more backbones and to LongBench, with variance estimates (seeds or bootstrap CIs over examples). Please also add a prompt-end observation-window criterion, and report the distribution of c_d and the fraction of keys whose precision changes because of calibration. A consistent and significant gain from calibration would establish the paper's main methodological contribution.

  3. Listing 1 averages post-RoPE prefill queries, which carry different rotary phases, so their high-frequency components largely cancel. Did you compare against pre-RoPE averaging with rotation to the expected decoding position? The answer determines whether mu_p is a meaningful global reference or mostly a low-frequency prior, and therefore whether Eq. 5 is well motivated.

  4. What is the accuracy of BF16, KIVI and ForeKV in the long-generation regime of Sec. 4.5 (for example the same GSM8K workload, or a math benchmark with Qwen3 thinking enabled)? Without this, the accuracy and throughput claims do not apply to the same setting.

Suggestions for future work:

  1. Periodically re-estimate mu_hat and re-classify packed tokens during long generations, given the growing Oracle Gap in Fig. 1(b). This would make long generations more robust and does not affect my rating.

  2. Use progressive or chunked packing so the full prompt cache does not have to sit in BF16 during the observation window, and evaluate it under continuous batching with long prompts. This would extend the memory benefit to long-prompt serving and does not affect my rating.

6 comments

Do decoding steps 1..T_d attend to the uncompressed cache, and does any baseline get a similar full-precision window? If ForeKV keeps its margins under an equal delay, Weakness 1 is resolved and I would raise my rating.

W1-Q1. Fairness of Delayed Compression

Thank you for pointing out the fairness issue in the original comparison. We reran KIVI, ZipCache, and ChanMix on three backbones with the same eight-step compression delay as ForeKV (Td=8T_d=8). The tables report LongBench-11 scores and the macro-average over all 13 RULER-4K tasks. We reuse the LongBench-11 results for ForeKV from the paper.

Llama-3.2-3B-Instruct

Methodpassage_retrieval_enlccrepobench-photpotqamusique2wikimqasamsumtrecqaspermultifieldqa_enmulti_newsLongBench-11 Macro Avg.RULER-4K Macro Avg.
KIVI97.5044.9248.4953.3425.1740.8041.3271.5038.2749.6825.3648.7678.61
ZipCache97.5045.5949.4753.0225.3440.8842.2871.5038.6049.2925.9449.0484.56
ChanMix97.5045.3749.8453.1025.1640.8642.1571.5040.4050.9925.8449.3491.67
ForeKV97.5051.9154.0853.2825.0239.8442.1771.0040.1150.5125.8250.1192.98

Qwen2.5-3B-Instruct

Methodpassage_retrieval_enlccrepobench-photpotqamusique2wikimqasamsumtrecqaspermultifieldqa_enmulti_newsLongBench-11 Macro Avg.RULER-4K Macro Avg.
KIVI50.0049.5951.1246.7920.9235.6443.4669.0035.3344.4721.9042.5750.44
ZipCache50.0049.6450.9346.6320.6035.3743.9569.0034.1341.8621.5842.1552.68
ChanMix50.0050.0351.9346.7220.7536.0344.4769.0036.9346.4322.3743.1573.82
ForeKV50.0056.4956.5646.6720.8335.8245.0568.5036.1046.1021.8243.9984.44

Qwen3-4B

Methodpassage_retrieval_enlccrepobench-photpotqamusique2wikimqasamsumtrecqaspermultifieldqa_enmulti_newsLongBench-11 Macro Avg.RULER-4K Macro Avg.
KIVI96.5052.7152.2655.5931.6243.3142.9573.0042.0852.4824.6751.5684.36
ZipCache96.5052.7952.6955.5431.5843.4943.3873.0041.5451.1725.0951.5284.06
ChanMix†96.5032.8533.3554.8931.5142.4729.2673.0033.0644.983.5143.2223.00
ForeKV96.5052.6053.5055.9731.3943.8043.3373.0044.1154.0125.1852.1393.20

† ChanMix produced repeated tokens on Qwen3-4B. We report the observed scores and exclude this run when identifying the strongest baseline.

We agree that the original comparison favored ForeKV on passage_retrieval_en because short answers can finish before KV cache compression begins. With the delay matched, all four methods obtain the same score on this task within each backbone. ForeKV retains an overall accuracy advantage under this matched setting, leading the strongest valid baseline by 0.57–0.84 points on LongBench-11 and 1.31–10.62 points on RULER-4K across the three backbones.

Do decoding steps 1..T_d attend to the uncompressed cache, and does any baseline get a similar full-precision window? If ForeKV keeps its margins under an equal delay, Weakness 1 is resolved and I would raise my rating.

Generation limits and thinking mode. We disabled thinking for Qwen3 and Qwen3.5 in the main accuracy evaluations and for Qwen3-4B in the matched-delay runs.

TaskMax. new tokens
passage_retrieval_en32
lcc64
repobench-p64
hotpotqa32
musique32
2wikimqa32
samsum128
trec64
qasper128
multifieldqa_en64
multi_news512

Scored output tokens after compression. We count the original output tokens retained for scoring, excluding special tokens such as EOS. The table reports the count generated using compressed KV and its share of all scored tokens.

MeasureLlama-3.2-3B-InstructQwen2.5-3B-InstructQwen3-4B
Scored tokens after compression / total scored tokens92,646 / 108,675100,793 / 116,76997,099 / 113,409
Overall share (token-weighted)85.25%86.32%85.62%

Please extend Table 2 to more backbones and to LongBench, with variance estimates (seeds or bootstrap CIs over examples).

W2-Q2. Effectiveness of Global Prefill Calibration

We extended the key-selection ablation in Table 2 to Qwen2.5-3B-Instruct and Qwen3-4B on LongBench-11 and RULER-4K. All five selectors share the same precision budget, attention kernel, and compression delay (Td=8T_d=8). Prompt-end Mean averages the last eight prefill queries.

MethodQwen2.5-3B LongBench-11Qwen2.5-3B RULER-4KQwen3-4B LongBench-11Qwen3-4B RULER-4K
Random 40%43.1480.7551.9392.52
Prefill Mean43.5282.4651.9792.98
Prompt-end Mean43.6282.6052.0293.09
Decode Mean43.7983.1352.0193.14
ForeKV43.9984.4452.1393.20

We used 2,000 paired bootstrap resamples to estimate 95% confidence intervals for the macro-average score differences (in points).

Backbone / BenchmarkForeKV − Decode Mean [95% CI]ForeKV − Prompt-end Mean [95% CI]
Qwen2.5-3B / LongBench-11+0.200 [+0.084, +0.314]+0.370 [+0.211, +0.522]
Qwen2.5-3B / RULER-4K+1.310 [+0.006, +2.679]+1.840 [+0.468, +3.193]
Qwen3-4B / LongBench-11+0.120 [+0.020, +0.219]+0.110 [−0.019, +0.234]
Qwen3-4B / RULER-4K+0.060 [−0.166, +0.314]+0.110 [−0.155, +0.427]

On Qwen2.5-3B, the 95% intervals exclude zero for both comparisons on both benchmarks. The gains on Qwen3-4B are smaller, and only the LongBench-11 comparison with Decode Mean has an interval that excludes zero.

Calibration at first compression. We computed the mechanism statistics on samples that reached their first KV cache compression. We measure the reassignment rate as the total number of candidate key slots whose K4/K2 precision differs from Decode Mean divided by the total number of candidate key slots across these samples.

BackboneSamples reaching first compressionMean cdc_dMedianP10P90
Qwen2.5-3B3,010/4,0500.81240.81630.74200.8777
Qwen3-4B2,978/4,0500.76720.77230.67390.8538
BackboneCompressed samplesCandidate key slotsReassigned key slotsReassigned (%)
Qwen2.5-3B3,0101,262,493,43251,598,1064.087%
Qwen3-4B2,9785,038,884,000274,568,7895.449%

The prefill mean receives an average weight of 18.76% on Qwen2.5-3B and 23.28% on Qwen3-4B because its weight is 1−cd1-c_d. Compared with Decode Mean, calibration changes the assigned precision of 4.087% and 5.449% of candidate key slots, respectively, at first compression.

Comments on the paper

2 comments
Area_Chair_RfiKOpenReview ↗

Please respond to the rebuttal

Dear Reviewers, the authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions, if you haven't done so. Thank you!

AuthorsOpenReview ↗

Summary of Additional Evidence

Dear Chairs and Reviewers,

Thank you for your time, careful reviews, and constructive suggestions. We have added experiments and clarifications to address the main concerns.

  • Fair comparisons. We reran KIVI, ZipCache, and ChanMix with the same compression delay across multiple backbones. ForeKV retains its accuracy advantage under this matched setting. We also clarified generation limits, thinking settings, and the share of scored tokens generated after compression.

  • Calibration benefit. We extended the ablations across models and benchmarks, added a prompt-end query baseline, and reported confidence intervals and calibration statistics. These experiments assess when calibration helps and how it changes precision allocation. We also compared RoPE averaging variants to clarify the role of the prefill mean.

  • Recent baselines. We added accuracy comparisons with Kitty, MixKVQ, and PM-KVQ, together with a direct throughput comparison with Kitty. These experiments assess ForeKV against the recent methods suggested by the reviewers.

  • Long generation and modern thinking models. We evaluated GSM8K with Qwen3-8B thinking enabled and the requested Qwen3-4B-Thinking-2507. These experiments test accuracy and throughput during long reasoning outputs. We also compared fixed and periodically updated query estimates to examine the effect of attention shifts.

  • Efficiency and overhead. We added comparisons at matched batch sizes, tested selection strategies with the same attention kernel, and measured the costs of the main processing stages. These experiments separate the contribution of precision selection from the fused kernel and quantify the added overhead.

We hope these additions clarify ForeKV’s contribution and practical value. Thank you for your time and consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.