acceptodds
Under review as a conference paper at ICLR 2027

Disaggregated Quantization: Specializing LLM Prefill and Decode

Abstract

The prefill and decode phases of LLM inference reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce memory traffic during generation. We propose "disaggregated quantization" (DQ), which specializes computational formats, weights and storage placement to both of these phases. Starting with format disaggregation, removing activation quantization specifically on decode improves accuracy on decode-heavy tasks without increasing inference cost. Moving to full disaggregation, training separate compute-native prefill weights accelerates prompt processing relative to weight-only inference while exceeding weight-only accuracy on both decode-heavy and prefill-heavy tasks for 2-3-bit quantization. To accommodate the extra prefill checkpoint on a single device, offloaded disaggregated prefill (ODP) streams its weights from SSD, amortizing loading over prompt length. Using released Qwen3.8-27B GGUF decoders, training an auxiliary NVFP4 prefill model in a fully-disaggregated fashion improves 1-bit accuracy by 32.5 points on MMLU-Pro and 35.3 on MMMU-Pro, without modifying the decode checkpoint. Combined with ODP, it delivers a 1.78x prefill speedup over the weight-only baseline at 8K prompt length in llama.cpp at no device memory overhead. Additional PTQ validation includes models with up to 2.8T parameters.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
llm, compression, disaggregated, quantization, inference, serving, gpu, cuda, blackwell, prefill, decode, vllm, distillation, offloading

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_NhdJOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The bottlenecks of prefill and decode are different. Prefill is usually compute bound and suitable for hardware low precision; Decode is memory bound when batch size is 1, more suitable for compressing weights and reducing weight transfer.

The paper proposes three methods: -Format: Sharing model weights, but prefill is calculated using NVFP4, decode skips activation quantization, and adopts a weight first path. -Full: prefill and decode use different quantization weights. -Offloaded Disaggregated Prefill (ODP): Place additional Prefill checkpoints in the SSD and load them in a transformer block stream to reduce device side memory usage.

Critical strengths

The paper does not simply view quantization as a single "compression model" problem, but rather links it to the different hardware bottlenecks of prefill / decode. This entry point is in line with the performance structure of actual LLM serving and also distinguishes the paper from general unified quantization methods. From format disaggregation with shared weights, to full disaggregation with independent weights, and then to ODP with SSD streaming loading, the paper gradually increases the degree of decoupling. Figure 1, Figure 4, and Table 1 can basically illustrate what problems each level of design solves.

Critical weaknesses

1-bit quality improvement may be confused by generation length and truncation rate The prefill of Qwen3.8-27B not only changes accuracy, but also significantly alters generation behavior. For example, on MMMU Pro, the average generation length of IQ1_S decreased from 14579 to 6582, and the truncation rate decreased from 24.57% to 4.16%. Therefore, the increase of 32.5/35.3 percentage points may partially come from: -Less abnormally long reasoning; -Lower truncation rate; -More stable answer termination behavior; It does not necessarily indicate an improvement in quality itself.

Questions for authors

Suggested supplement: -Comparison between fixed maximum generation length and fixed output budget; -Conduct separate analysis on exam match, stop behavior, and answer parsing failure rate; -Repeat MMLU Pro/MMMU Pro results using multiple random seeds.

Reviewer_mRWJOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Disaggregated Quantization (DQ), a phase-aware approach that treats the prefill and decode phases of LLM inference separately to optimize performance. While low-precision compute accelerates prompt processing during prefill, compact weights minimize memory traffic during generation. To achieve this, the authors propose Quantization-Aware Distillation with Disaggregation (QADD), which utilizes label masks to train phase-specific pathways toward a shared response objective in a single forward-backward pass. Building on this framework, DQ encompasses three compounding techniques: (i) format disaggregation, (ii) full disaggregation, and (iii) offloaded disaggregated prefill.

Critical strengths
  • The idea of quantizing separately for prefill and decode stages is a natural and effective quantization approach for LLMs, which fits with different compute/memory characteristics of two stages.

  • The proposed method combines the benefits of hardware-native, low-precision compute for fast prefill with compact, low-bitwidth weights for memory-bound decode, thereby improving quality on both prefill-heavy and decode-heavy workloads.

  • Offloaded disaggregated prefill (ODP) reduces the device memory occupation during the prefill stage.

Critical weaknesses
  • The proposed phase-level mixed-precision quantization has been presented in other papers which limits the main novelty ("Mix-Quant" with NVFP4 on prefill and full-precision on decode phase or "DynaCore" with W8A8 on prefill and W4A16 for decode).

  • The idea of applying knowledge distillation to train quantized models is common. Using the PTQ method on each pathway without going through the KD iteration may provide more efficient quantization method.

  • Noticeable accuracy drop compared to the full-precision (BF16) baseline in both Table 1 and 2.

  • The paper lacks performance evaluation at batched LLM decode scenarios, which are more common LLM serving scenarios.

Questions for authors

(1) Comparative study on other phase-aware LLM quantization methods would be great to see during the rebuttal period (in terms of both accuracy and performance). For instance, "A Shape-Adaptive Architecture with Disaggregated Quantization for Efficient LLM Serving" (https://arxiv.org/pdf/2610.07443).

(2) During the prefill stage for large LLMs with MoE, fetching prefill weights from SSD may incur significant amount of data reads since tokens during prefill may invoke most of experts. Any thoughts?

(3) Any latency/throughput issues due to limited SSD bandwidth in fetching prefill weights directly from the storage? Fig. 5(a) shows that the load latency gets hidden when the context length exceeds 8K. Then, what if multiple user requests are less than 8K (10x slowdown at 2K)? Do you suggest to load prefill weights to the device memory doubling up the memory requirement?

Reviewer_uktwOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes disaggregated quantization (DQ), which treats prefill and decode as separate quantization targets. Three schemes build on each other: 1) Format disaggregation. One set of weights, with NVFP4 W4A4 used for prefill and weight-only computation (no activation quantization) used for decode. 2) Full disaggregation. A separate NVFP4 prefill checkpoint is trained jointly with a low-bit weight-only LUT decode checkpoint. 3) Offloaded disaggregated prefill (ODP). Prefill weights are streamed from SSD block by block into two buffers carved out of decode weight memory. This hides the extra checkpoint’s device residency at long prompts.

Critical strengths
  • Simple, well-motivated idea: It follows directly from the different roofline regimes of prefill and decode. Format disaggregation is close to free and could be adopted immediately in serving stacks.
  • Elegant training mechanism: Using the SFT mask to select the pathway lets both pathways train toward the response objective in one pass.
  • Real systems work: The authors write custom LUT decode kernels (the LUT2 kernel reaches 96% of achievable bandwidth), integrate with vLLM and llama.cpp, and break down per-component latency (Table 11) so readers can see where speedup is diluted.
  • Strong low-bit result: At 2-bit and below, the gains are large and transfer to MMMU-Pro even though the distillation corpus is text-only.
Critical weaknesses
  • ODP’s claim of no additional device weight memory is well supported by the carve-out design. On a unified-memory platform such as GB10, please report peak total memory, including pinned staging buffers, to substantiate the abstract’s broader “no device memory overhead” wording.
  • Minor typos: ~397 “fully-disaggregated LUT2 substantially outperform LUT2A16” should be “outperforms”, ~300–301 “As the result” should be “As a result”
Questions for authors
  • What is the accuracy with BF16 prefill plus each frozen 27B GGUF decoder, without training? And a decode-side LoRA or QAD baseline matched to the 95M training tokens?
  • What precision is the KV cache stored in, and does QADD interact with KV quantization? How do the device-memory numbers change once the KV cache and prefill activation workspace are included?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.