- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The bottlenecks of prefill and decode are different. Prefill is usually compute bound and suitable for hardware low precision; Decode is memory bound when batch size is 1, more suitable for compressing weights and reducing weight transfer.
The paper proposes three methods: -Format: Sharing model weights, but prefill is calculated using NVFP4, decode skips activation quantization, and adopts a weight first path. -Full: prefill and decode use different quantization weights. -Offloaded Disaggregated Prefill (ODP): Place additional Prefill checkpoints in the SSD and load them in a transformer block stream to reduce device side memory usage.
Critical strengths
The paper does not simply view quantization as a single "compression model" problem, but rather links it to the different hardware bottlenecks of prefill / decode. This entry point is in line with the performance structure of actual LLM serving and also distinguishes the paper from general unified quantization methods. From format disaggregation with shared weights, to full disaggregation with independent weights, and then to ODP with SSD streaming loading, the paper gradually increases the degree of decoupling. Figure 1, Figure 4, and Table 1 can basically illustrate what problems each level of design solves.
Critical weaknesses
1-bit quality improvement may be confused by generation length and truncation rate The prefill of Qwen3.8-27B not only changes accuracy, but also significantly alters generation behavior. For example, on MMMU Pro, the average generation length of IQ1_S decreased from 14579 to 6582, and the truncation rate decreased from 24.57% to 4.16%. Therefore, the increase of 32.5/35.3 percentage points may partially come from: -Less abnormally long reasoning; -Lower truncation rate; -More stable answer termination behavior; It does not necessarily indicate an improvement in quality itself.
Questions for authors
Suggested supplement: -Comparison between fixed maximum generation length and fixed output budget; -Conduct separate analysis on exam match, stop behavior, and answer parsing failure rate; -Repeat MMLU Pro/MMMU Pro results using multiple random seeds.