- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Mix-Quant, a phase-aware quantization scheme for agentic LLM inference. Prefill is compute-bound, so the paper uses NVFP4 to speed up prefill compute without dequantizing (Blackwell executes W4A4 natively). Decode is memory-bound but more sensitive to quantization, so it is kept in BF16. This can be directly plugin existing prefill-decode disaggregation framework. Together this achieves good speedup while largely preserving task performance.
Critical strengths
-
Practical. The method is simple and deployable on existing infrastructure, and it targets a real bottleneck in agentic serving.
-
Broad evaluation. It evaluated on many agentic & long-context benchmarks.
Critical weaknesses
- The error-propagation analysis is weak. This is quite unacceptable because this is the only central justification of the paper.
- Within-prefill propagation is unanalyzed. Causal prefill is mathematically equivalent to teacher-forced decoding. Quantization error in token i's hidden state propagates across layers and to all later positions through attention. Appendix A treats the KV perturbation as a given bounded quantity (δ_v) and analyzes only how decode-time attention aggregates it. It does not analyze how δ_v arises or whether it grows with depth and context length. The claim that prefill error does not grow with context length is therefore assumed, not shown.
- The attention-concentration bound does not support the argument. Since attention weights sum to one, the bound ‖Σᵢ aᵢΔvᵢ‖ ≤ δ_v holds for any attention distribution, concentrated or not. The same argument also applies to KV produced by a quantized decoder, so it cannot explain why prefill tolerates quantization better than decode. Moreover, the high-attention tokens, whose errors are not attenuated, are the ones most associated with activation outliers. A high-attention token with weight 0.5 contributes 50% of its error, so its error passes through almost fully.
- The impact of the perturbed KV cache on decoding is not quantified. If I understand correctly, the prefill quantization scheme W4A4 still yields KV cache in BF16. The KV values are computed from FP4 operands. It is read at every decode step and makes up most of the context in multi-turn settings. The paper does not measure how much it shifts the decode-time output distribution, for example via teacher-forced KL of BF16 decoding on FP4-prefilled vs. BF16-prefilled KV. A comparison with KV-cache quantization methods (KIVI, KVQuant, Turbo Quant), which show KV caches tolerate 2–4 bit storage reasonably well, would help separate two sources of error.
- The ablation shows a weak phase asymmetry. In Table 3, quantizing only prefill costs 2.10 points on Qwen3-8B and 3.54 on Gemma-4-26B. Quantizing only decode costs 3.68 and 3.85 respectively. On Gemma the two phases degrade almost equally, and P16D4 outperforms Mix-Quant on several individual benchmarks. This does not support the claim that prefill exhibits "substantially higher quantization redundancy." No variances are reported across the three runs, and several differences look like noise. For example, Mix-Quant exceeds BF16 on AIME24, and FP8 exceeds BF16 on Gemma.
- The efficiency evaluation does not match the target regime. The motivation is 128K+ agentic contexts, but latency is measured only up to 32K, and speedup drops with length (3.51× at 2K to 1.96× at 32K for Qwen3-8B). Is this because attention remains in BF16? The 3.74× headline is at 2K tokens with batch size 32. I am also curious about the end-to-end agentic latency, KV transfer cost, it would be nice if you can provide.
Questions for authors
Questions for the authors
- How does relative error in hidden states and K/V evolve across layers and positions during FP4 prefill, and does it grow from 2K to 128K context?
- Under teacher forcing the decode token, what is the KL divergence and top-1 agreement of BF16 decoding on FP4-prefilled KV versus BF16-prefilled KV?
- How does Mix-Quant compare to QSpec and to BF16 prefill with a quantized KV cache?
I would be willing to reconsider my score if the author addresses my concern.