acceptodds
Withdrawn from ICLR 2027

Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use, memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that, despite being the dominant source of computation, prefilling exhibits substantially higher quantization redundancy than autoregressive decoding. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM agents. Extensive experiments across long-context and agentic benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant improving efficiency, achieving up to a 3.74 speedup during prefilling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XnG4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Mix-Quant, a phase-aware quantization scheme for agentic LLM inference. Prefill is compute-bound, so the paper uses NVFP4 to speed up prefill compute without dequantizing (Blackwell executes W4A4 natively). Decode is memory-bound but more sensitive to quantization, so it is kept in BF16. This can be directly plugin existing prefill-decode disaggregation framework. Together this achieves good speedup while largely preserving task performance.

Critical strengths
  • Practical. The method is simple and deployable on existing infrastructure, and it targets a real bottleneck in agentic serving.

  • Broad evaluation. It evaluated on many agentic & long-context benchmarks.

Critical weaknesses
  1. The error-propagation analysis is weak. This is quite unacceptable because this is the only central justification of the paper.
    • Within-prefill propagation is unanalyzed. Causal prefill is mathematically equivalent to teacher-forced decoding. Quantization error in token i's hidden state propagates across layers and to all later positions through attention. Appendix A treats the KV perturbation as a given bounded quantity (δ_v) and analyzes only how decode-time attention aggregates it. It does not analyze how δ_v arises or whether it grows with depth and context length. The claim that prefill error does not grow with context length is therefore assumed, not shown.
    • The attention-concentration bound does not support the argument. Since attention weights sum to one, the bound ‖Σᵢ aᵢΔvᵢ‖ ≤ δ_v holds for any attention distribution, concentrated or not. The same argument also applies to KV produced by a quantized decoder, so it cannot explain why prefill tolerates quantization better than decode. Moreover, the high-attention tokens, whose errors are not attenuated, are the ones most associated with activation outliers. A high-attention token with weight 0.5 contributes 50% of its error, so its error passes through almost fully.
  2. The impact of the perturbed KV cache on decoding is not quantified. If I understand correctly, the prefill quantization scheme W4A4 still yields KV cache in BF16. The KV values are computed from FP4 operands. It is read at every decode step and makes up most of the context in multi-turn settings. The paper does not measure how much it shifts the decode-time output distribution, for example via teacher-forced KL of BF16 decoding on FP4-prefilled vs. BF16-prefilled KV. A comparison with KV-cache quantization methods (KIVI, KVQuant, Turbo Quant), which show KV caches tolerate 2–4 bit storage reasonably well, would help separate two sources of error.
  3. The ablation shows a weak phase asymmetry. In Table 3, quantizing only prefill costs 2.10 points on Qwen3-8B and 3.54 on Gemma-4-26B. Quantizing only decode costs 3.68 and 3.85 respectively. On Gemma the two phases degrade almost equally, and P16D4 outperforms Mix-Quant on several individual benchmarks. This does not support the claim that prefill exhibits "substantially higher quantization redundancy." No variances are reported across the three runs, and several differences look like noise. For example, Mix-Quant exceeds BF16 on AIME24, and FP8 exceeds BF16 on Gemma.
  4. The efficiency evaluation does not match the target regime. The motivation is 128K+ agentic contexts, but latency is measured only up to 32K, and speedup drops with length (3.51× at 2K to 1.96× at 32K for Qwen3-8B). Is this because attention remains in BF16? The 3.74× headline is at 2K tokens with batch size 32. I am also curious about the end-to-end agentic latency, KV transfer cost, it would be nice if you can provide.
Questions for authors

Questions for the authors

  • How does relative error in hidden states and K/V evolve across layers and positions during FP4 prefill, and does it grow from 2K to 128K context?
  • Under teacher forcing the decode token, what is the KL divergence and top-1 agreement of BF16 decoding on FP4-prefilled KV versus BF16-prefilled KV?
  • How does Mix-Quant compare to QSpec and to BF16 prefill with a quantized KV cache?

I would be willing to reconsider my score if the author addresses my concern.

Reviewer_QZDhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Mix-Quant, a phase-aware quantization framework that uses NVFP4 for prefilling and BF16 for decoding to balance inference efficiency and accuracy. It employs prefill–decode disaggregation with KV-cache transfer between workers. Experiments on long-context and agentic benchmarks show largely preserved accuracy and up to 3.74× prefill speedup.

Critical strengths
  1. The paper identifies the substantial prefilling overhead in input-heavy agentic workflows and targets an important efficiency bottleneck in long-context LLM inference.
  2. Mix-Quant adopts a simple phase-aware quantization strategy that combines NVFP4 prefilling with BF16 decoding, making it relatively straightforward to integrate with existing prefill–decode disaggregated serving systems.
Critical weaknesses
  1. The core insight that prefilling is more tolerant to aggressive quantization than decoding is not sufficiently justified. Although Table 3 provides a phase-wise quantization ablation, the results alone do not sufficiently establish the claimed differences in error propagation between prefilling and decoding. In particular, the paper lacks a systematic analysis of how quantization errors affect generation quality as decoding length increases, or how prefill-induced errors in the KV cache influence subsequent decoding. Moreover, prior work such as Progressive Mixed-Precision Decoding (PMPD) explores a different precision-allocation perspective, favoring higher precision in earlier inference stages. Since errors introduced during prefilling can propagate through the KV cache and influence subsequent token generation, the authors should clarify the relationship between these findings and provide stronger empirical evidence for the proposed phase-wise precision allocation.
  2. The framework lacks several critical implementation details and appears to primarily combine existing NVFP4 quantization and prefill–decode disaggregation techniques, with limited technical novelty. For example, it is unclear how FP4 prefilling constructs KV caches compatible with BF16 decoding, including when the KV caches are stored in BF16 and how they are transferred between workers. Furthermore, the paper does not explain how prefix caching is managed across multi-turn interactions, including where KV caches are retained, how repeated prefixes are reused, and how cache consistency is ensured when identical token prefixes may originate from different precision paths. These details are important for understanding both the correctness and practical efficiency of the proposed framework.
Questions for authors
  1. The efficiency evaluation is limited to prefill latency, without reporting end-to-end latency or throughput for agentic workflows. It remains unclear how the reported prefill speedups translate into practical performance gains, particularly when accounting for decoding and KV-cache transfer overhead.
  2. The paper reports prefill speedups only on RTX 5090 GPUs, despite also listing B200 GPUs in the evaluation. The authors should clarify the role of B200, the GPU allocation for prefill–decode disaggregation, whether the baseline and Mix-Quant use the same hardware resources, and whether the reported speedups include KV-cache transfer overhead.
  3. The baseline comparisons are insufficient. The paper primarily compares Mix-Quant against BF16 and uniformly quantized models, without comprehensive comparisons with existing efficient quantization methods or alternative mixed-precision configurations in terms of both accuracy and latency. This makes it difficult to assess whether Mix-Quant achieves a superior accuracy–efficiency trade-off over existing approaches.
Reviewer_JJqWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper demonstrates the benefits of combining prefill–decode disaggregation with model checkpoints at different precision levels for agentic workloads, using NVFP4 for prefill and BF16 for decoding.

Critical strengths

The paper empirically demonstrates that low-precision prefill can accelerate context processing in agentic workloads while largely preserving task performance when decoding remains in BF16.

Critical weaknesses

The general idea of assigning different precisions to prefill and decoding in disaggregated serving has already been explored in prior work, including SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving (Woo et al., 2026) and ERNIE 4.5 (Baidu-ERNIE-Team, 2025), as well as in existing implementations. Therefore, phase-specific precision allocation alone offers limited novelty.

Questions for authors
  1. Prior work, including QSUN in SUN (Woo et al., 2026) and §6.3 (“Deployment”) of the ERNIE 4.5 Technical Report (Baidu-ERNIE-Team, 2025), as well as implementation reports such as SGLang issue #22716, has already explored using different precisions for prefill and decoding in disaggregated serving. Could the authors expand the related work section to discuss these efforts combining mixed-precision inference with prefill–decode disaggregation? What is the main contribution that distinguishes Mix-Quant from these prior efforts, particularly regarding its precision allocation and evaluation on agentic workloads?

  2. Could the authors provide a detailed description of the KV cache policy used in the multi-turn experiments? Specifically, are KV entries generated during decoding retained and reused by the prefill worker in subsequent turns, or does the prefill worker recompute the conversation history, including previously generated tokens? Please clarify whether prefix caching is enabled and how KV entries are managed across turns. Have the authors evaluated a configuration in which the prefill worker reuses KV entries produced by the decode worker? If so, how does this reuse affect task accuracy and prefill latency compared with recomputing those entries using the quantized prefill model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.