acceptodds
Under review as a conference paper at ICLR 2027

QuMix: Query-Guided Mixed-Precision KV-Cache Quantization for Block Diffusion Language Models

Abstract

Block diffusion lets diffusion language models (dLLMs) reuse completed blocks through a key–value (KV) cache, but storing and repeatedly reading this growing cache limits context length, batch size, and decoding speed. KV-cache quantization reduces these costs, yet has been studied almost entirely in autoregressive (AR) models. Applying an existing AR quantizer to a block-diffusion model, we find that two-bit quantization preserves nearly all of its original TriviaQA score, whereas reducing the precision to one bit causes a large drop. This motivates mixed precision, which lowers average storage by retaining more bits for sensitive components and using fewer bits elsewhere. Such allocation must preserve information that attention relies on, including the key of the attention sink—a token that many heads repeatedly attend to. AR models typically place this sink at the first token, making it possible to protect by position, but sink positions can vary in dLLMs. We therefore introduce QuMix, which uses queries to guide precision allocation without assuming a sink position. From calibration queries and keys, QuMix constructs a query-weighted basis and assigns more bits to directions whose errors have a larger effect on attention scores, without model training. To our knowledge, this is the first study of KV-cache quantization for block diffusion. QuMix retains 98–99% of the 16-bit LongBench score across three models at an average of 1.99 bits per cache element. A custom CUDA kernel that attends directly to the compressed cache supports a larger maximum batch and faster denoising steps at the same batch size (batch 15) than BF16 FlashAttention4 on an RTX PRO 6000 GPU.

Primary area:
generative models
Keywords:
diffusion language models, block diffusion, KV-cache quantization, mixed precision, efficient AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U8G6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a efficient mixed-precision method for KV-cache quantization on block diffusion language models. Motivated by the attention sink, they propose query-weighted basis and query-aware bit allocation to address the bottleneck of the current methods. Across different tasks and backbones, they show that under 1.99 bits their methods still keep very high accuracy and achieve good efficiency.

Critical strengths
  1. The motivation of this paper is from attention sink, which is useful and informative for the query-weighted basis and query-aware bit allocation design. And their results on very low bits (1.99 bits) are very good cause their method still keeps high accuracy across multiple tasks and backbones.
Critical weaknesses
  1. The paper shows the best accuracy comes from 1.99 bits but they report their best efficiency results based on the 1.55 bits. And the low bit setting is more efficient but sacrifice a lot of performance of the accuracy so they should report the result at a balanced level (the trade-offs of accuracy and efficiency are very essential to discuss), not separately claim the advantages of two different setting.

  2. The ablation study is not sufficient to distinguish the effectiveness of different components they proposed. They do not prove the different effects of query-weighted basis and query-aware bit allocation. More ablation study for this paper is needed to show how each component perform and work.

Questions for authors
  1. Could the authors provide a more detailed ablation comparison on query-weighted basis and query-aware bit allocation separately? This could help to distinguish the real effects of different method components.

  2. Could the authors report the throughput, memory, and latency under 1.99 bits setting? It can show how efficient the method is when keeping a very good accuracy and performance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer U8G6

Thank you for the two concrete requests. Both are now measured, and both support the paper's claims: the efficiency gains hold at the setting that preserves accuracy, and the gain comes from query-guided allocation.

1. Accuracy and efficiency at the same operating point (W1, Q2).

At 1.99 bits, QuMix keeps 98% of the 16-bit LongBench score with an 8× smaller KV cache, an 8× larger maximum batch, and a 1.8× faster denoising step. The serving numbers are at 32K input, where QuMix retains 93% of the 16-bit RULER score (Table R3 in our response to 6NqN).

Table R5. Quality and serving at each operating point.

FA4 BF16QuMix 1.99QuMix 1.69QuMix 1.55
LongBench avg. (10 tasks)47.246.244.839.1
LongBench vs. 16-bit100%98%95%83%
KV cache per request (GiB)4.500.5650.4810.441
Batch 15: cache / prefill peak (GiB)67.5 / 86.18.5 / 27.27.2 / 25.96.6 / 25.3
Batch 15: denoising step (ms)61.733.830.626.9
Batch 15: decode positions/s194347382430
Batch 1 / 8: step (ms)51.2 / 50.715.8 / 25.115.5 / 24.014.9 / 21.1
Maximum batch at 90 GiB15123not measured157
Maximum batch: step (ms)61.7169.5—149.4
Maximum batch: decode positions/s194545—754

Notes. SDAR-8B, 32K input, RTX PRO 6000; LongBench scores are from the submission. The kernel is not retuned for the 1.99- and 1.69-bit settings, so these numbers are conservative.

Throughput at 1.99 bits is 1.8× that of BF16 at the same batch and 2.8× when each method runs at its maximum batch. We will report accuracy and efficiency together at 1.99 bits in the abstract and Section 5.3, and present 1.55 bits as the aggressive end of the range, which shrinks the cache by a further 22% at the cost of accuracy on code tasks.

2. Separate effects of the basis and the allocation (W2, Q1).

Table G1 in the general response reports the two effects separately, as you asked. The allocation is the component that works: with the basis fixed, query-aware allocation adds 1.6 and 2.4 points on SDAR at 1.69 and 1.55 bits. The query-weighted basis adds nothing on top and is worse than a plain key-PCA basis at 1.55 bits (39.1 versus 43.0). We will therefore report both bases and present the allocation as the core of the method. On Nemotron, which is near the floor on the code tasks where this gain arises, the variants are within about one point (general response).

My questions are resolved and I will raise my score.

Thank you for reading our response and for reconsidering your assessment. Your two requests made the paper clearer: the revision will report accuracy and efficiency together at 1.99 bits and include the ablation that separates the basis from the allocation. We appreciate your review.

Reviewer_6NqNOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QuMix, a training-free mixed-precision KV-cache quantization method for block diffusion language models. It constructs a query-weighted key basis using calibration statistics and allocates different bit widths according to attention sensitivity. A custom CUDA kernel enables efficient attention computation over the compressed cache. Experiments across multiple models and benchmarks demonstrate competitive accuracy under aggressive compression, together with improvements in memory consumption and decoding efficiency.

Critical strengths

The proposed query-weighted basis provides a principled way to identify sensitive key directions and allocate precision accordingly.

The custom CUDA kernel translates the proposed compression method into meaningful memory and decoding efficiency improvements.

Critical weaknesses

The query-weighted basis relies on pre-RoPE query statistics without explicitly accounting for relative positional rotations. Its impact on quantization quality, particularly at long context lengths, remains unclear.

QuMix uses only 64 sequences of 512 tokens for calibration, but does not sufficiently evaluate sensitivity to calibration data size or distribution shifts.

Query-aware KV-cache transformations and mixed-precision allocation have been explored in prior work. The paper needs to more clearly establish the methodological differences and independent contributions of its proposed basis construction and bit allocation strategy.

Questions for authors

How reliable is the pre-RoPE query-weighted approximation across different relative positions and longer contexts?

How sensitive is QuMix to calibration data size and distribution? Do the authors have results using different calibration datasets?

What are the fundamental differences between QuMix and existing query-aware methods such as MixKVQ, OSCAR, and NOVA-KV? Which specific components account for the improvements?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6NqN

Thank you for the review and the clear questions. In short: the pre-RoPE statistics hold up against a post-RoPE control and at longer inputs, calibration at the larger budget is robust to corpus and size, and a ten-task ablation shows that on SDAR the gain comes from query-guided bit allocation, the paper's central idea.

1. Reliability of the pre-RoPE approximation (W1, Q1).

A post-RoPE control is worse. Collecting the statistics after RoPE, with everything else fixed, leaves HotpotQA within noise and lowers LCC by 17 points on SDAR. Keys are stored before RoPE and rotated when read (Fig. 3c), so pre-RoPE statistics let one basis and one bit assignment serve every position.

Longer contexts. With calibration and bit assignment fixed at the 4K settings, QuMix at 1.99 bits still retains 93% of the 16-bit RULER score at 32K on SDAR.

Table R3. Share of the 16-bit RULER score retained (%).

MethodKV bits4K8K16K32K
KIVI 2-bit, SDAR-8B2.9498.196.590.889.1
QuMix, SDAR-8B1.9998.898.294.892.6
QuMix, Nemotron-8B1.9997.892.488.9—

The loss relative to 16-bit grows with length for both methods. QuMix loses less than KIVI 2-bit, which uses no query statistics, at every length and with 0.95 fewer bits, so we see no sign that the pre-RoPE approximation is what limits it at long inputs. Table G1 further shows that the basis need not be built from these statistics at all; they are needed only to rank directions for allocation.

2. Calibration size and distribution (W2, Q2).

At the larger budget, QuMix is not sensitive to the calibration corpus or size. We recalibrated with WikiText and C4 at 8, 16, 32, and 64 segments of 512 tokens.

Table R4. SDAR scores under eight calibration sets.

CorpusSegments (×512 tokens)HotpotQA, larger budgetHotpotQA, smaller budgetLCC, larger budgetLCC, smaller budget
WikiText844.343.562.154.1
WikiText1645.045.463.844.4
WikiText3246.647.764.745.0
WikiText6444.942.564.844.4
C4848.641.662.746.3
C41643.143.865.849.5
C43247.945.364.446.2
C46448.244.662.744.0

Notes. SDAR-8B, 150 examples per task. 16-bit reference: HotpotQA 47.7, LCC 65.2. The larger and smaller budgets use the allocations of the 1.99- and 1.55-bit settings.

At the larger budget, scores show no trend with corpus or size: HotpotQA spans 43.1–48.6 and LCC 62.1–65.8. Eight segments, captured in 57 s, do as well as 64. At the smaller budget, LCC is 11–21 points below the reference for every calibration set and varies more across sets. No calibration set removes that loss; Table G1 traces it to the basis.

3. Differences from MixKVQ, OSCAR, and NOVA-KV, and which components matter (W3, Q3).

Differences. MixKVQ allocates bits over the original key channels at 3.28 bits; at 1.54 bits, that loses 24 points on TriviaQA (Table 3), so a transformed basis is needed at the low end. OSCAR and NOVA-KV, two concurrent arXiv preprints, transform the keys using attention statistics but give every coefficient the same width at two bits per element. QuMix assigns integer widths, including zero, to directions under a budget, which takes the cache to 1.55–1.99 bits. To our knowledge it is also the first to target block diffusion, where the sink's position differs across models and prompt formats, and it comes with a kernel that turns the smaller cache into measured capacity and speed.

Which components matter. You asked for the independent contributions of the basis construction and the bit allocation (Table G1, general response). The allocation accounts for the improvement: it adds about two points on SDAR over an allocation without query information. The basis construction does not: a plain key-PCA basis does as well, and better at 1.55 bits. This is consistent with the differences above: like OSCAR and NOVA-KV, QuMix transforms the keys, and what is new is the budgeted integer-width allocation that takes the cache below two bits. Unlike SVDq, which also allocates precision over a key basis, QuMix fixes that basis by calibration instead of computing an SVD per request.

We sincerely thank you for your thoughtful review and for the clarity of your questions.

Your questions gave us the chance to put direct evidence behind two design choices of QuMix. The post-RoPE control and the long-context results you prompted show that pre-RoPE statistics outperform the alternative and hold up at longer inputs, and the calibration study shows that QuMix at its main setting is robust to the calibration corpus and size, even with eight segments captured in under a minute.

Your request to state the independent contributions of the basis and the allocation was especially valuable. It led to an ablation that makes the contribution of QuMix precise, the query-aware allocation, and gives a clearer account of how the work relates to prior methods.

We genuinely appreciate the care behind your review. It has made the paper clearer and better supported, and we would be glad to discuss anything that remains unclear.

Reviewer_wkUbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies KV-cache quantization for block-diffusion language models, in which completed blocks are cached exactly and read at every denoising step. The authors describe this as the first study of KV-cache quantization in this setting.

Motivating experiments on SDAR-8B indicate that sub-2-bit quality depends on protecting a small set of early keys, which includes an attention sink that is not located at the first token.

The proposed method, QuMix, works as follows:

  1. A one-time calibration of query and key second moments.
  2. A query-weighted, non-orthogonal key basis in which the expected attention-score error decomposes over directions.
  3. Per-direction bit widths from {0, 1, 2, 4, 8, 16}, assigned by a log-variance rule.
  4. Values kept at 2 bits.
  5. A fused CUDA kernel that reconstructs keys, applies RoPE and computes attention over the packed cache.
Critical strengths

S1. A new problem setting with an insightful analysis of block diffusion models.

  • The paper investigates KV-cache quantization for block-diffusion language models, which the authors state is the first study of this setting (L97–98).
  • Its analysis of attention-sink behavior contrasts SDAR with the autoregressive Qwen3 (§3; Fig. 2):
    • Averaged over all layers and heads, the first token receives 35.8% of attention in Qwen3 but only 0.025% in SDAR (L744–745).
    • In the examined SDAR head, the sink lies at the third token (L200–201).
    • Restoring the first key recovers most of Qwen3's quantization loss but little of SDAR's.
  • Why it matters: exploring KV-cache quantization in block-diffusion models opens a potentially valuable research direction. More importantly, the comparison between SDAR and Qwen3 provides empirical evidence of the differences between block-diffusion and autoregressive architectures. This strengthens the motivation for a quantization method tailored to block diffusion.

S2. Practical systems optimization and scalability.

  • The paper presents a fused kernel that reconstructs keys, applies RoPE and computes attention directly over the packed mixed-precision cache (§4.3).
  • At 1.55 bits on SDAR-8B with 32K-token prompts, compared with BF16 FlashAttention-4 (Tables 13–14):
    • the maximum batch increases from 15 to 157 (10.5×);
    • a denoising step at batch 15 is 2.3× faster (61.7 vs. 26.9 ms).
  • Why it matters: the optimized kernel is a valuable systems-level contribution that improves the practical applicability and scalability of the method. By reducing memory requirements and improving inference efficiency, it makes KV-cache quantization more suitable for real-world deployment, particularly in memory-constrained and high-throughput serving.
Critical weaknesses

W1. Limited generalizability of the attention-sink analysis, and insufficient baseline comparisons.

  • Sink behavior varies across models. The generalizability of the sink analysis remains unclear:
    • In SDAR, the sink lies within the first block, and restoring the first four keys (the first block) raises TriviaQA from 73% to 95% of the BF16 score (Fig. 2a).
    • In contrast, Nemotron's native decoder exhibits a first-token sink (L748–749).
    • Sink patterns therefore vary across models, which raises questions about the robustness and general applicability of the motivation.
  • Sink behavior after quantization is not measured. The paper does not evaluate how quantization affects sink behavior, for example the attention mass on sink tokens after quantization.
  • Missing comparisons.
    • There is no comparison against sink-aware quantization baselines, such as KIVI or SVDq with the first block kept at full precision.
    • In the long-context RULER evaluation (Table 2), KIVI 2-bit is the only quantization baseline.

Why it matters: attention-sink behavior is a central motivation for the method, so analyses across more architectures are needed to establish that the findings generalize. Without sink-aware baselines and broader comparisons, it is difficult to tell whether the improvements come from the proposed innovations or could be achieved with existing techniques. This weakens the empirical case for the method's novelty and effectiveness.

W2. Insufficient ablations and limited experimental depth.

  • Basis ablation is narrow.
    • It is run only on TriviaQA with SDAR (150 examples).
  • Query weighting is not isolated from the rotation. For example, the paper does not compare against an unweighted key-PCA basis with the same allocation rule.
  • Other design choices are not systematically investigated: value bit width, group size and calibration strategy.
  • Limited statistical depth.
    • Each LongBench task uses 150 examples, and each RULER task–length pair uses 50.
    • QuMix uses a single calibration draw.
    • No confidence intervals are reported.
  • Single latency baseline. BF16 FlashAttention-4 is the only latency baseline.

Why it matters: the evaluation and ablations lack the depth expected of an ICLR paper. More rigorous experiments are needed to isolate the contribution of each component, justify key design choices, and show that the method is robust across settings. Without them, it is difficult to assess the scientific significance of the proposed techniques or distinguish them from existing approaches.

Presentation and methodological clarity (reflected in my Presentation score).

  • The figures lack the scientific detail needed to communicate the key ideas and findings.
  • The method is presented through equations (Eqs. 1–5) and a schematic (Fig. 3), but no step-by-step algorithm summarizes the full pipeline of calibration, bit allocation, packing and attention.
  • These shortcomings make the approach harder to understand and reproduce. Clearer figures and a more systematic description of the method would substantially strengthen the paper.
Questions for authors
  1. Relation to concurrent query-aware transforms (clarification). How does the query-weighted basis (Eqs. 2–3) differ, in theory and in practice, from NOVA-KV's non-orthogonal transform and OSCAR's covariance-aware rotation?
  2. Sink preservation and sink-aware baselines (clarification / limited experiment).
    • Is attention-sink behavior preserved after quantization, as measured by the attention mass assigned to sink tokens or by changes in attention maps?
    • How does QuMix compare against sink-aware baselines under matched memory and compute budgets, such as KIVI or SVDq with the first block kept at full precision?
  3. Calibration sensitivity (clarification). How sensitive is the method to the calibration data (domain, size, seed), especially for code tasks, where degradation is largest? This matters because QuMix uses a single WikiText calibration draw, whereas KVQuant on SDAR is reported as the best of three calibration draws.
5 comments
AuthorsOpenReview ↗

Response to Reviewer wkUb (1/2): sink-aware baselines and sink behavior

Thank you for the detailed review. We ran both experiments you asked for. They show that QuMix at 1.99 bits needs no positional rule (W1), and that its gain comes from query-guided allocation (W2).

1. Sink-aware baselines (W1, Q2).

KIVI and SVDq have no sink rule as published; as you suggested, we added one to each, and to QuMix. The first block (four tokens on SDAR) is stored exactly, at a cost of 0.014 bits per element. With a chat template it contains the sink, so this is the best case for a positional rule.

Table R1. First-block protection on SDAR-8B.

MethodKV bitsAvg. (10 tasks)Avg. with first blockGain [95% CI]
16-bit1647.5——
KIVI 2-bit2.9446.847.5+0.7 [+0.1, +1.4]
KIVI 1-bit2.1511.812.6+0.8 [+0.1, +1.7]
QuMix1.9946.446.4−0.0 [−0.6, +0.6]
SVDq-1.01.8843.245.7+2.6 [+1.6, +3.5]
QuMix1.8845.246.1+0.9 [+0.2, +1.7]
QuMix1.6944.645.4+0.8 [+0.0, +1.6]
QuMix1.5539.144.5+5.4 [+4.4, +6.5]

Notes. Ten LongBench tasks, 150 examples each, exact prefill for all methods.

At 1.99 bits, QuMix already preserves what the rule protects. Protection does not change its score, whereas SVDq-1.0 gains 2.6 points. Without the rule, QuMix at 1.99 bits is within noise of KIVI 2-bit while using 0.95 fewer bits. KIVI 1-bit stays near 12 with or without the rule.

At the same storage, QuMix is 2.0 points above SVDq without a positional rule, and the two are on par once the rule is added to both. QuMix at 1.99 bits with no rule is also on par with SVDq-1.0 with the rule. The compute is not matched: SVDq computes an SVD of each request's keys at prefill, whereas QuMix's basis is fixed after one calibration.

The rule works only because the first block happens to contain the sink. On the two code tasks, protecting the sink token alone recovers as much as the whole first block, for both SVDq-1.0 and QuMix at 1.55 bits; the other three tokens recover nothing, including the first token, the usual rule for autoregressive models. The sink's position differs across models and prompt formats: token 2 on SDAR, token 0 on Nemotron, and past the first block in 45% of head–layer slots without a chat template (SDAR, 32-token blocks). A positional rule has to be set per model and prompt format; QuMix at 1.99 bits needs one calibration and no such rule.

RULER. We used KIVI 2-bit in Table 2 because it is the strongest baseline below three bits; QuMix exceeds it at every length with 0.95 fewer bits.

2. Sink behavior after quantization and across models (W1, Q2).

Yes, at 1.99 bits the sink is preserved. We measured the attention mass on the sink key during denoising.

Table R2. Sink preservation after quantization.

ModelMethodKV bitsSink massvs. 16-bitTV vs. 16-bit
SDAR-8B16-bit160.3011.00×—
SDAR-8BKIVI 2-bit2.940.2870.95×0.202
SDAR-8BKIVI 1-bit2.150.1870.62×0.636
SDAR-8BQuMix1.990.2930.97×0.212
SDAR-8BQuMix1.690.2540.85×0.273
SDAR-8BQuMix1.550.2180.73×0.320
Nemotron-8B16-bit160.4431.00×—
Nemotron-8BQuMix1.990.4400.99×0.259
Nemotron-8BQuMix1.690.4180.94×0.295
Nemotron-8BQuMix1.550.4240.96×0.313

Notes. The sink is key index 2 on SDAR and index 0 on Nemotron. Sink mass is averaged over all layers and heads on 150 TriviaQA examples. TV is the total-variation distance from the 16-bit attention.

At 1.99 bits the sink keeps 97% of its mass on SDAR and 99% on Nemotron, and the attention map on SDAR changes about as much as under KIVI 2-bit, which uses 0.95 more bits. The two models place the sink at different positions, and the same procedure preserves both. At 1.55 bits the mass on SDAR falls to 73%, which is where QuMix loses accuracy on code; a key-PCA basis avoids most of that loss (Table G1, general response).

3. Isolating query weighting from the basis (W2).

We ran the comparison you suggested, and it separates the two cleanly. With an unweighted key-PCA basis and the same allocation rule, on ten tasks and two models instead of TriviaQA alone (Table G1, general response), query-aware allocation adds about two points on SDAR, making the basis query-weighted as well adds nothing, and at 1.55 bits key-PCA is better (43.0 versus 39.1). Your control therefore leads to a simpler QuMix, and we will report both bases.

4. Other points.

Statistical depth (W2). The ten-task comparisons (Tables R1 and G1) use 1,500 paired examples with 95% bootstrap intervals, whose half-widths are about one point. We will add intervals to Table 1.

Calibration (Q3). Across eight calibration sets (WikiText and C4, 8 to 64 segments), scores show no trend with corpus or size at the larger budget: the code task stays within about three points of the 16-bit reference and HotpotQA within five (Table R4, response to 6NqN). We did not vary the random seed separately. Best-of-three selection applies only to the KVQuant baseline and favors it.

Value bits, group size, and the latency baseline (W2). KIVI 2-bit, SVDq, and QuMix share the same 2-bit value format, so the comparisons isolate the key quantizer; we did not sweep the value width or the group size. To our knowledge, the quantized baselines reconstruct full-precision keys and values before dense attention, so BF16 FlashAttention-4 is a lower bound on their step time.

Relation to NOVA-KV and OSCAR (Q1). Both are concurrent arXiv preprints (May and August 2026) discussed in Section 2. Our basis is in the same family as NOVA-KV's transform; the difference is what happens after it. They give every coefficient the same width at two bits per element and target autoregressive decoding. QuMix assigns each direction an integer width, including zero, under a budget, which takes the cache below two bits, and it targets block diffusion.

Presentation. We will add a step-by-step algorithm covering calibration, bit allocation, packing, and attention, and revise the figures as you suggest.

We sincerely thank you for your careful and constructive review.

Your suggestions led to the experiments that most strengthened this paper. The sink-aware baselines you proposed gave us a direct way to show what sets QuMix apart: at 1.99 bits it keeps the sink without any positional rule, and protecting the first block leaves its score unchanged. Your question about sink behavior after quantization let us confirm this at the level of attention itself, on two models whose sinks sit at different positions.

The control you proposed, an unweighted key-PCA basis with the same allocation rule, was especially valuable. It isolates where the gain comes from, the query-aware allocation, and it gave us a simpler variant that improves our lowest setting from 39.1 to 43.0.

We are genuinely grateful for the care and insight behind your review. It has made the paper's evidence considerably stronger, and we would be glad to discuss anything that remains unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.