- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a efficient mixed-precision method for KV-cache quantization on block diffusion language models. Motivated by the attention sink, they propose query-weighted basis and query-aware bit allocation to address the bottleneck of the current methods. Across different tasks and backbones, they show that under 1.99 bits their methods still keep very high accuracy and achieve good efficiency.
Critical strengths
- The motivation of this paper is from attention sink, which is useful and informative for the query-weighted basis and query-aware bit allocation design. And their results on very low bits (1.99 bits) are very good cause their method still keeps high accuracy across multiple tasks and backbones.
Critical weaknesses
-
The paper shows the best accuracy comes from 1.99 bits but they report their best efficiency results based on the 1.55 bits. And the low bit setting is more efficient but sacrifice a lot of performance of the accuracy so they should report the result at a balanced level (the trade-offs of accuracy and efficiency are very essential to discuss), not separately claim the advantages of two different setting.
-
The ablation study is not sufficient to distinguish the effectiveness of different components they proposed. They do not prove the different effects of query-weighted basis and query-aware bit allocation. More ablation study for this paper is needed to show how each component perform and work.
Questions for authors
-
Could the authors provide a more detailed ablation comparison on query-weighted basis and query-aware bit allocation separately? This could help to distinguish the real effects of different method components.
-
Could the authors report the throughput, memory, and latency under 1.99 bits setting? It can show how efficient the method is when keeping a very good accuracy and performance.
3 comments
My questions are resolved and I will raise my score.
Thank you for reading our response and for reconsidering your assessment. Your two requests made the paper clearer: the revision will report accuracy and efficiency together at 1.99 bits and include the ablation that separates the basis from the allocation. We appreciate your review.
Response to Reviewer U8G6
Thank you for the two concrete requests. Both are now measured, and both support the paper's claims: the efficiency gains hold at the setting that preserves accuracy, and the gain comes from query-guided allocation.
1. Accuracy and efficiency at the same operating point (W1, Q2).
At 1.99 bits, QuMix keeps 98% of the 16-bit LongBench score with an 8× smaller KV cache, an 8× larger maximum batch, and a 1.8× faster denoising step. The serving numbers are at 32K input, where QuMix retains 93% of the 16-bit RULER score (Table R3 in our response to 6NqN).
Table R5. Quality and serving at each operating point.
Notes. SDAR-8B, 32K input, RTX PRO 6000; LongBench scores are from the submission. The kernel is not retuned for the 1.99- and 1.69-bit settings, so these numbers are conservative.
Throughput at 1.99 bits is 1.8× that of BF16 at the same batch and 2.8× when each method runs at its maximum batch. We will report accuracy and efficiency together at 1.99 bits in the abstract and Section 5.3, and present 1.55 bits as the aggressive end of the range, which shrinks the cache by a further 22% at the cost of accuracy on code tasks.
2. Separate effects of the basis and the allocation (W2, Q1).
Table G1 in the general response reports the two effects separately, as you asked. The allocation is the component that works: with the basis fixed, query-aware allocation adds 1.6 and 2.4 points on SDAR at 1.69 and 1.55 bits. The query-weighted basis adds nothing on top and is worse than a plain key-PCA basis at 1.55 bits (39.1 versus 43.0). We will therefore report both bases and present the allocation as the core of the method. On Nemotron, which is near the floor on the code tasks where this gain arises, the variants are within about one point (general response).