- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1(−2)
Read review
Paper summary
This paper introduces Blaze, which collects attention statistics across layers during prefill and uses voting across consecutive layers to identify important positions, then assigns INT4 or INT8 precision to each block. The method also incorporates long-distance token protection to safeguard important tokens in long contexts. The resulting precision allocation remains fixed throughout decoding. The paper reports perplexity results across five models, along with evaluations on commonsense, reasoning, and long-context retrieval tasks.
Critical strengths
-
The paper is clearly written, and the introduction to the method is clear and easy to follow.
-
The paper evaluates a wide range of tasks. In addition to PPL, it also includes other tasks such as long-context retrieval.
Critical weaknesses
Main
-
The benefits of the proposed components are unclear. At line 368, the paper states: “At 16K, K ∈ {1,3,5} and α ∈ {0.90,0.99} give the same average PPL and compression ratio, so the assignment is insensitive to K and α in this range.” These results do not demonstrate the benefits of cross-layer voting or layer-wise threshold decay, weakening the justification for introducing these components. Furthermore, Table 3 shows that removing PA protection changes PPL only from 10.05 to 10.07 while increasing the compression ratio from 62.1% to 74.6%. Table 4 even reports better performance without PA protection at the 32K. These findings call into question the utility of PA protection.
-
The paper discusses existing importance-aware precision allocation methods, such as ZipCache and MiKV. Its main distinguishing features are block-level allocation and cross-layer voting. However, as discussed in W1, the experiments do not demonstrate the benefits of cross-layer voting. The paper also lacks experiments demonstrating the advantages of block-level precision allocation over token-level allocation, as well as direct comparisons with these closely related methods. Moreover, all the evaluated baseline systems date back to 2024, with no more recent methods included. Therefore, the current evidence is insufficient to establish a meaningful technical advance over prior work.
-
The efficiency evaluation does not establish practical deployment benefits. The evaluation relies on simulated quantization and therefore does not demonstrate the speedups or GPU memory savings achievable in a real deployment.
Minor
- The proposed method determines precision assignments solely during prefill and keeps them fixed throughout decoding. However, prior research has shown that token importance can change during generation for a given input. The paper should evaluate whether such changes affect the effectiveness of its fixed precision assignments.
Questions for authors
-
Could the authors provide additional ablation studies to quantify the benefits of each proposed component?
-
Could the authors include comparisons with importance-aware precision allocation methods such as ZipCache and MiKV as well as some newer methods in this field?
-
Could the authors report the actual speedup and GPU memory usage of the proposed method under the same implementation setup as the baselines? If implementing optimized kernels is challenging, using straightforward implementations for all methods would still provide a useful comparison.
-
For long-context inputs, could the authors compare the initial precision assignments with those recomputed using query tokens from later decoding steps, and quantify how much they differ?
2 comments
Thank you for the response.
Regarding W1, the response does not resolve my concern about the usefulness of PA protection. The cited accuracy change from 98% to 92% is accompanied by a change in the INT8 budget. A comparison at the same INT8 budget is needed to establish the value of the selection strategy.
Regarding Q3, the reported results show that saving roughly 2 GB of GPU memory comes at the cost of approximately 7–8 times the decoding latency per step. I consider this overhead unacceptable for practical deployment. Without a fused implementation demonstrating that the latency can be reduced to an acceptable level, the practical feasibility of the approach remains uncertain. Fused kernels are therefore essential to validating the proposed approach and should not be deferred to future work.
I recognize that this assessment may be disappointing, but I do not believe the paper has reached the level of completeness required for acceptance. I will therefore lower my score.
Reply to Reviewer kZ1n
W1. Benefits of the components are unclear.
We ran the matched-compression ablations (Section 3.11; Table 14 in Appendix G), with θ re-tuned so every block-level variant compresses 62.1% at 16K (Qwen3-4B, n=50):
We report this honestly: at this operating point, K=1 and α=1 produce per-sample identical assignments to K=3 and α=0.95, so the voting and decay add no measurable gain. What they provide is a robust operating region — the knobs do steer the assignment outside it (at K=100 no block closes; at α=0.50 the closed-block count moves 126→110) — and the consecutive-layer scale is measured, not asserted (adjacent-layer fingerprints correlate at 0.819, Figure 8). We have rewritten the component claims to say exactly this. The token-level row also answers W2's granularity question: removing the 64-token constraint reaches 72.2% compression with slightly lower PPL, so the block constraint costs about ten points of compression and buys the contiguous layout with one precision record per block and O(1) decode-time bookkeeping.
On PA protection: its effect is visible in exact retrieval, not PPL (PA tokens are ~10% of positions, so PPL moves by ≤0.05). Removing protection drops all-needle accuracy by 6 points on Qwen3-4B at 8K (98→92) and by 2 points in three more cells (Tables 16 and 13); averaged over 4K–16K, accuracy is 98.7 with vs. 96.7 without. At 32K the no-PA variant is indeed better (96 vs. 92) while saving 17 points of compression; we do not dispute this cell, and the budget-constant rule in our reply to myBP W4 is the fix we report (90–92% at 68.8–68.9% compression, Section 4; Table 15).
W2. No block-level vs. token-level comparison; no ZipCache/MiKV or newer baselines.
Both are covered: the token-level row in the ablation table above, and ZipCache/MiKV/SpectrumKV at matched compression in G1.
W4 / Q4. Precision assignments are fixed at prefill, but token importance can change during generation.
We ran the suggested measurement (Section 3.12, "Assignment drift during decoding"): recomputing the closure assignment at decode step 256 with decode-time queries included changes the closure status of only 0.143%/0.150% of positions and the precision of 1.35%/1.52% of blocks (Qwen3-4B/Llama-3.1-8B, prefix 16K, n=20), with PA sets agreeing at Jaccard 0.985. Together with the existing evidence — ΔPPL does not widen from 256 to 1,024 generated tokens (Table 4, Appendix A.1), and per-step loss drift is smaller for Blaze than for FP16 (0.0074 vs. 0.0105, Table 8, Appendix A.5) — fixed prefill assignments are stable in this regime.
Q3. See G2: real-storage peak memory and same-implementation latency are both reported in the revision; fused kernels remain future work and are stated as such.