acceptodds
Under review as a conference paper at ICLR 2027

Blaze: Block-Level Adaptive KV Quantization Guided by Attention

Abstract

The KV cache is the memory bottleneck of long-context inference and grows with sequence length. Attention concentrates on a few positions, yet compression methods treat every token alike or evict tokens irreversibly. We present Blaze, a block-level adaptive KV cache quantization scheme that derives the precision of each block from the attention its positions receive. During prefill, Blaze marks a position closed when its attention stays below a threshold that decays with layer depth, since deeper layers attend to fewer positions, and it requires agreement across consecutive layers, so single-layer fluctuations do not drive the decision. Blocks are stored in INT8 or INT4 according to how many of their positions closed, blocks that carry long-distance positions keep the higher precision, and every decision is fixed during prefill, so decoding reads a lookup table and adds work per step. Across five models, Blaze compresses the KV cache by 70–71% at 4K and 8K prefixes at a perplexity within 1.6% of FP16, and it stays within 2 points of FP16 on eight downstream tasks and hard four-needle retrieval, where uniform INT4 falls 6 to 14 points behind. Blaze lowers the memory barrier for long-context inference by assigning precision to blocks instead of discarding tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache quantization, adaptive quantization, cross-layer attention voting, long-context inference, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_kZ1nOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1(−2)
Read review
Paper summary

This paper introduces Blaze, which collects attention statistics across layers during prefill and uses voting across consecutive layers to identify important positions, then assigns INT4 or INT8 precision to each block. The method also incorporates long-distance token protection to safeguard important tokens in long contexts. The resulting precision allocation remains fixed throughout decoding. The paper reports perplexity results across five models, along with evaluations on commonsense, reasoning, and long-context retrieval tasks.

Critical strengths
  1. The paper is clearly written, and the introduction to the method is clear and easy to follow.

  2. The paper evaluates a wide range of tasks. In addition to PPL, it also includes other tasks such as long-context retrieval.

Critical weaknesses

Main

  1. The benefits of the proposed components are unclear. At line 368, the paper states: “At 16K, K ∈ {1,3,5} and α ∈ {0.90,0.99} give the same average PPL and compression ratio, so the assignment is insensitive to K and α in this range.” These results do not demonstrate the benefits of cross-layer voting or layer-wise threshold decay, weakening the justification for introducing these components. Furthermore, Table 3 shows that removing PA protection changes PPL only from 10.05 to 10.07 while increasing the compression ratio from 62.1% to 74.6%. Table 4 even reports better performance without PA protection at the 32K. These findings call into question the utility of PA protection.

  2. The paper discusses existing importance-aware precision allocation methods, such as ZipCache and MiKV. Its main distinguishing features are block-level allocation and cross-layer voting. However, as discussed in W1, the experiments do not demonstrate the benefits of cross-layer voting. The paper also lacks experiments demonstrating the advantages of block-level precision allocation over token-level allocation, as well as direct comparisons with these closely related methods. Moreover, all the evaluated baseline systems date back to 2024, with no more recent methods included. Therefore, the current evidence is insufficient to establish a meaningful technical advance over prior work.

  3. The efficiency evaluation does not establish practical deployment benefits. The evaluation relies on simulated quantization and therefore does not demonstrate the speedups or GPU memory savings achievable in a real deployment.

Minor

  1. The proposed method determines precision assignments solely during prefill and keeps them fixed throughout decoding. However, prior research has shown that token importance can change during generation for a given input. The paper should evaluate whether such changes affect the effectiveness of its fixed precision assignments.
Questions for authors
  1. Could the authors provide additional ablation studies to quantify the benefits of each proposed component?

  2. Could the authors include comparisons with importance-aware precision allocation methods such as ZipCache and MiKV as well as some newer methods in this field?

  3. Could the authors report the actual speedup and GPU memory usage of the proposed method under the same implementation setup as the baselines? If implementing optimized kernels is challenging, using straightforward implementations for all methods would still provide a useful comparison.

  4. For long-context inputs, could the authors compare the initial precision assignments with those recomputed using query tokens from later decoding steps, and quantify how much they differ?

2 comments

Reply to Reviewer kZ1n

W1. Benefits of the components are unclear.

We ran the matched-compression ablations (Section 3.11; Table 14 in Appendix G), with θ re-tuned so every block-level variant compresses 62.1% at 16K (Qwen3-4B, n=50):

Variantθ_baseCompressAvg PPLΔPPL (%)
Blaze (K=3, α=0.95)0.000262.1%10.05−0.82±0.24
single layer (K=1)0.000262.1%10.05−0.82±0.24
constant threshold (α=1)0.000262.1%10.05−0.82±0.24
token-level assignment0.000272.2%10.01−1.12±0.34

We report this honestly: at this operating point, K=1 and α=1 produce per-sample identical assignments to K=3 and α=0.95, so the voting and decay add no measurable gain. What they provide is a robust operating region — the knobs do steer the assignment outside it (at K=100 no block closes; at α=0.50 the closed-block count moves 126→110) — and the consecutive-layer scale is measured, not asserted (adjacent-layer fingerprints correlate at 0.819, Figure 8). We have rewritten the component claims to say exactly this. The token-level row also answers W2's granularity question: removing the 64-token constraint reaches 72.2% compression with slightly lower PPL, so the block constraint costs about ten points of compression and buys the contiguous layout with one precision record per block and O(1) decode-time bookkeeping.

On PA protection: its effect is visible in exact retrieval, not PPL (PA tokens are ~10% of positions, so PPL moves by ≤0.05). Removing protection drops all-needle accuracy by 6 points on Qwen3-4B at 8K (98→92) and by 2 points in three more cells (Tables 16 and 13); averaged over 4K–16K, accuracy is 98.7 with vs. 96.7 without. At 32K the no-PA variant is indeed better (96 vs. 92) while saving 17 points of compression; we do not dispute this cell, and the budget-constant rule in our reply to myBP W4 is the fix we report (90–92% at 68.8–68.9% compression, Section 4; Table 15).

W2. No block-level vs. token-level comparison; no ZipCache/MiKV or newer baselines.

Both are covered: the token-level row in the ablation table above, and ZipCache/MiKV/SpectrumKV at matched compression in G1.

W4 / Q4. Precision assignments are fixed at prefill, but token importance can change during generation.

We ran the suggested measurement (Section 3.12, "Assignment drift during decoding"): recomputing the closure assignment at decode step 256 with decode-time queries included changes the closure status of only 0.143%/0.150% of positions and the precision of 1.35%/1.52% of blocks (Qwen3-4B/Llama-3.1-8B, prefix 16K, n=20), with PA sets agreeing at Jaccard 0.985. Together with the existing evidence — ΔPPL does not widen from 256 to 1,024 generated tokens (Table 4, Appendix A.1), and per-step loss drift is smaller for Blaze than for FP16 (0.0074 vs. 0.0105, Table 8, Appendix A.5) — fixed prefill assignments are stable in this regime.

Q3. See G2: real-storage peak memory and same-implementation latency are both reported in the revision; fused kernels remain future work and are stated as such.

Thank you for the response.

Regarding W1, the response does not resolve my concern about the usefulness of PA protection. The cited accuracy change from 98% to 92% is accompanied by a change in the INT8 budget. A comparison at the same INT8 budget is needed to establish the value of the selection strategy.

Regarding Q3, the reported results show that saving roughly 2 GB of GPU memory comes at the cost of approximately 7–8 times the decoding latency per step. I consider this overhead unacceptable for practical deployment. Without a fused implementation demonstrating that the latency can be reduced to an acceptable level, the practical feasibility of the approach remains uncertain. Fused kernels are therefore essential to validating the proposed approach and should not be deferred to future work.

I recognize that this assessment may be disappointing, but I do not believe the paper has reached the level of completeness required for acceptance. I will therefore lower my score.

Reviewer_myBPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Blaze, a block-level adaptive KV cache quantization scheme that derives the precision of each block from the attention its positions receive. During prefill, Blaze marks a position closed when its attention stays below a threshold that decays with layer depth, since deeper layers attend to fewer positions, and it requires agreement across consecutive layers, so single-layer fluctuations do not drive the decision. Blocks are stored in INT8 or INT4 according to how many of their positions closed, blocks that carry long-distance positions keep the higher precision, and every decision is fixed during prefill, so decoding reads a lookup table and adds O(1) work per step. Across five models, Blaze compresses the KV cache by 70–71% at 4K and 8K prefixes at a perplexity within 1.6% of FP16, and it stays within 2 points of FP16 on eight downstream tasks and hard four-needle retrieval, where uniform INT4 falls 6 to 14 points behind. Blaze lowers the memory barrier for long-context inference by assigning precision to blocks instead of discarding tokens.

Critical strengths
  1. The proposed method is clean and easy to use. For example, Blaze divides the sequence into blocks of B = 64 tokens, each receiving a single precision assignment; Blaze performs O(1) work per step during decode.

  2. The experiments are comprehensive. Blaze is evaluated on various models, downstream tasks and deployment configurations.

  3. The evaluation results are sound. Blaze outperforms uniform quantization on retrieval tasks.

Critical weaknesses
  1. The Random-Block Control study admits that "a random partition of the same size is not distinguishable from the attention-based selection of protected blocks". This weakens the claim as the number of blocks rather than which blocks to select are more important.

  2. Although many methods are listed in the related works, the paper only uses KIVI and KVQuant for comparison.

  3. Real deployment concern: Blaze's measured throughput is lower than FP16 because fake quantization is used. This leads to less utilization of the proposed method.

  4. Degradation is obvious for very-long context, "Compression is 70.2–71.0% from 4K to 8K and falls to 56–57% at 32K, because beyond 8K every block that holds a protected long-distance token stays in INT8". The method relies on carefully selected hyperparameters, and threshold sensitivity is only studied on Qwen3-4B.

Questions for authors

See Weaknesses.

1 comment

Reply to Reviewer myBP

W1. Random-Block Control: "the number of blocks rather than which blocks matters".

The control varies only which blocks the long-distance (PA) protection holds in INT8; it does not touch the closure-based INT4/INT8 assignment, which is the core of Blaze and remains attention-derived. Two results show block identity matters for the core mechanism: at matched compression (62%), the Fixed Zone partition — a fixed INT4/INT8 alternation with no attention feedback — is 0.26 PPL worse than Blaze (10.31 vs. 10.05, Table 3), and uniform INT8 at 12 points lower compression (50%) is still worse (10.16). On the four-needle task, needles sit at uniformly random depths, so a randomly chosen INT8 block covers a needle with the same probability as an attention-selected one: the task measures the protection budget, not the selection. The revision states this scope explicitly (Appendix D): closure-based assignment is attention-guided; long-distance protection is a budget mechanism whose selection the task does not validate.

W4. Compression falls to 56–57% at 32K; threshold sensitivity studied only on Qwen3-4B.

(i) The revision adds a budget-constant protection variant capping the protected share at 10% of all blocks, keeping the highest-attention candidates (Section 4; Table 15 in Appendix H; PPL on WikiText-2, generation 256; NIAH = all-needle accuracy on the four-needle task at 32K):

ModelConfigurationPPL (32K)Compress (%)NIAH (%)
Qwen3-4Bfull protection9.6957.392
Qwen3-4Bbudget-constant9.7568.890
Qwen3-4Bno protection9.8174.896
Llama-3.1-8Bfull protection6.9457.392
Llama-3.1-8Bbudget-constant6.9868.992
Llama-3.1-8Bno protection6.9874.896

(n=20 per cell; n=19 on Llama-3.1-8B, where one sample is 44 tokens short of the window on its tokenizer.) At 16K the same variant compresses 68.6% vs. 62.1% (full) and 74.6% (none), at PPL 10.02 vs. 10.05/10.07. The variant recovers most of the long-prefix cost with PPL between the fully protected and unprotected runs; at 32K protection still does not pay on this task, and the budget rule keeps most of the compression when protection is retained.

(ii) The θ_base sweep now includes Llama-3.1-8B (Appendix A.4; prefix 2,048, n=20): compression is 63.6/67.8/69.2/69.3/69.3/69.3% at θ=0.00015/0.0002/0.0003/0.0004/0.0006/0.0008 with PPL flat within 0.02 — the same saturation shape as Qwen3-4B, so the default transfers across models. It also transfers across datasets without retuning (71.0% compression within 0.8% PPL of FP16 on C4, Appendix A.2).

Comments on the paper

1 comment

General comments (shared by both reviews)

We thank both reviewers for the careful reading. We have run the requested comparisons, ablations, and measurements; they are included here and in the revised manuscript.

G1. Baselines stop at 2024; no direct comparison with importance-aware methods (ZipCache, MiKV, newer work).

Added in the revision (Section 3.8; Table 17 in Appendix I). ZipCache, MiKV, and SpectrumKV (2026) re-implemented in our harness under the identical protocol — same WikiText-2 sample pool, paired per-sample ΔPPL vs. FP16, same quantizer, compression by the same byte accounting (Eq. 2), each method's INT8 share tuned to within 1.5 points of Blaze's compression (n=50 per cell, generation 256):

MethodCompress 4K/16K (%)4B @4KLlama @4K4B @16KLlama @16K
Blaze70.3 / 62.1−1.42±0.25+0.59±0.10−0.74±0.15+0.28±0.07
MiKV68.8 / 61.8−1.35±0.21+0.64±0.08−0.86±0.35+0.38±0.06
ZipCache68.8 / 61.8−1.12±0.21+0.45±0.07−0.43±0.10+0.09±0.03
SpectrumKV68.7 / 61.8−1.15±0.22+0.61±0.06−0.91±0.34+0.34±0.06

(paired per-sample ΔPPL % vs. FP16, mean ± SE; negative = below FP16)

At matched compression, all three token-level methods land in the same ±1% band as Blaze on both models and lengths, and none is consistently closest to FP16: the benefit comes from the block-level assignment itself (one precision per contiguous 64-token block, no per-token search or calibration), not from a more elaborate importance signal.

G2. Efficiency evidence uses fake quantization; no real speedup or GPU-memory measurement.

Both are now in the revision (Section 3.13; Table 10 in Appendix B). With true integer storage (INT8 with per-token fp16 scales; packed INT4 with group-32 fp16 scales), dequantized before attention (Qwen3-4B, batch 1, 64 decode steps):

CachePrefixStored (GiB)Peak decode (GB)Median step (ms)
FP1616,3842.4210.2642.9
INT816,3841.149.1854.8
INT4 (packed)16,3840.638.6665.6
Blaze16,3840.898.92304.9
FP1632,7684.8312.8272.3
INT832,7682.2910.6892.8
INT4 (packed)32,7681.279.63115.4
Blaze32,7682.0110.39573.1

(4K rows omitted for space; full table in Appendix B.) The memory saving is real and matches Eq. 2 exactly (2.01 vs. 4.83 GiB at 32K), and quality under true integer storage matches fake quantization (Table 6, Appendix A.3). The latency column is the honest boundary of the prototype: a straightforward unfused implementation dequantizes before attention and is slower than FP16, so wall-clock speed requires fused kernels (future work); the measured per-step KV traffic (50.8%/28.1% of FP16 for INT8/INT4) bounds what such kernels could realize.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.