- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MosaicQuant, which decomposes weights into a dense 4-bit base and sparse 4-bit residual blocks, avoiding mixed-precision computation for outlier handling. A Hessian-guided strategy is used to select important residual blocks, and ZipperEngine is designed to efficiently fuse the dense and sparse computation. Experiments on several LLMs show competitive quantization accuracy and practical inference speedup.
Critical strengths
-
The paper proposes a unified 4-bit quantization framework that avoids mixed-precision computation for outlier handling, which is meaningful for efficient low-bit deployment.
-
The block-wise residual compensation is well aligned with hardware-friendly tiled computation, providing a reasonable co-design between the quantization method and the inference engine.
-
The paper includes both accuracy evaluation and real kernel-level implementation, demonstrating that the proposed method can achieve competitive quantization performance with practical inference speedup.
Critical weaknesses
-
The authors claim that outliers exhibit block-wise concentration, but this claim is not sufficiently supported by the experimental evidence. Although Figure 2(a) provides some qualitative visualization, more quantitative evidence is necessary given that this observation serves as an important premise for the method design.
-
The authors observe that “residual blocks naturally align with the tiled structure of low-bit GEMM,” which is an important premise for the fused operator design. However, this observation is not sufficiently discussed or supported by experimental evidence.
-
The efficiency evaluation emphasizes kernel fusion more than the actual cost–accuracy trade-off introduced by the sparse residual branch. While Fig. 5 reports the memory and latency overhead at the default setting, the paper does not systematically show how these overheads scale with the residual block budget, nor how they trade off against accuracy recovery.
Questions for authors
-
How consistent is the claimed block-wise concentration across different layers, modules, and models?
-
What evidence supports the claimed alignment between residual blocks and low-bit GEMM tiles?
-
How does the choice of block budget affect the accuracy–memory–latency trade-off?
3 comments
Author Response to Reviewer cGkF
Q3&W3: Residual Budget and the Accuracy–Memory–Latency Trade-off
Increasing the residual block budget raises memory use and latency and lowers perplexity, but does not consistently improve average task accuracy. We evaluated budgets of 5%, 10%, and 20% on Qwen3-14B and Qwen3-32B:
| Model | Block budget | Memory (GiB) ↓ | Effective BPW ↓ | Decode latency ratio ↓ | Prefill latency ratio ↓ | Avg. Acc. (%) ↑ | PPL ↓ |
|---|---|---|---|---|---|---|---|
| 14B | 5% | 10.985 | 4.3314 | 0.966 | 0.962 | 71.20 | 9.36 |
| 14B | 10% | 11.277 | 4.5377 | 1.000 | 1.000 | 71.86 | 9.29 |
| 14B | 20% | 11.862 | 4.9504 | 1.064 | 1.068 | 71.78 | 9.16 |
| 32B | 5% | 22.281 | 4.3314 | 0.954 | 0.962 | 72.75 | 8.55 |
| 32B | 10% | 22.966 | 4.5377 | 1.000 | 1.000 | 72.27 | 8.51 |
| 32B | 20% | 24.336 | 4.9504 | 1.057 | 1.071 | 71.99 | 8.38 |
Effective BPW is the stored bits per quantized weight, including scales and sparse indices. Decode (token generation) and prefill (prompt processing) latency ratios are normalized to the 10% configuration for each model. PPL (perplexity) measures next-token prediction quality; lower is better.
1. Larger budgets increase memory use. Moving from 10% to 20% increased memory use by 5.2% for 14B and 6.0% for 32B.
2. Larger budgets also increase latency. The same budget increase raised decode and prefill latency by approximately 6–7% across the two models.
3. Accuracy depends on the model and metric. For 14B, moving from 5% to 10% improved average accuracy by 0.66 percentage points, reaching the highest tested accuracy of 71.86%; 20% provided no further improvement. For 32B, 5% gave the highest average accuracy (72.75%) and the lowest resource cost. The 20% budget achieved the lowest PPL for both models (9.16/8.38), with the highest memory use and latency.
Among the tested budgets, the preferred choice therefore depends on the model and whether the priority is task accuracy, perplexity, or resource use.
Response to Authors’ Rebuttal
Thank you for the detailed responses and additional experiments. The broader block-selection analysis, clarification of tile compatibility, and residual-budget evaluation substantially address my original concerns. Although the individual components build on established ideas, their integration into a unified low-bit compensation and execution framework provides a meaningful contribution, supported by the additional ablations. I will therefore maintain my Weak acceptance rating.
Author Response to Reviewer cGkF
Q1&W1: Block-wise Concentration across Models, Layers, and Projections
Block-wise gains vary across projections. We compared the estimated Δ-Hessian compensation benefit captured by block-wise and channel-wise selection across six attention/MLP projection per layer in Qwen3-14B (L0–L39; 240 matrices) and Qwen3-32B (L0–L63; 384 matrices). Under the same 10% residual budget limit.
Each entry is averaged over 40 layers for Qwen3-14B or 64 layers for Qwen3-32B. Δ = Block − Channel, in percentage points. Both selections use a 10% residual budget, except 14B
down_proj(9.926471% for both).On average, block-wise selection captures more estimated compensation benefit than channel-wise selection in 10 of the 12 tested model–projection settings. The largest gains occur in
v_proj(+6.67/+5.13 percentage points for 14B/32B), whilegate_projandup_projare nearly tied.Q2&W2: Residual Block Alignment with GEMM Tiles
Alignment means that the dense weights and residual blocks cover matching input and output channels. Section 4.2 and Fig. 3(b) explain how this enables shared computation, and Fig. 3 reports the benefit of the resulting fused, overlapped execution.
1. Residual blocks match regions of the dense weights. GEMM (matrix multiplication) processes matrices in small pieces called tiles. Each residual block spans 128 input channels and 128 output channels and can be split into smaller execution tiles. It need not match a single hardware matrix multiply–accumulate (MMA) instruction. At a fixed number of compensated weights, smaller blocks allow finer selection, while larger blocks can reduce indexing and scheduling overhead.
2. Matching regions allow input reuse and direct accumulation. Both the dense weights and residuals use 4-bit values. For each selected region, ZipperEngine reuses the same quantized input (activation) tile and combines the two contributions, with their respective quantization scales, in the same output tile. This avoids loading the inputs again, writing a separate intermediate result, and launching another kernel to merge the outputs (Sec. 4.2).
3. Fusion and overlap reduce measured latency. On the Qwen3-8B
v_projlayer, separate residual computation and output merging took 22.3 μs. ZipperEngine fuses these operations with the dense computation and overlaps data movement with computation, achieving a 1.9× speedup over the naive pipeline (Fig. 3). This measures the combined benefit of fusion and overlap.We will document the exact kernel tile sizes, how each 128×128 residual block is partitioned, and the rationale for this block size. We will also replace “naturally align” with “are organized to be compatible with GEMM tiling.”