acceptodds
Under review as a conference paper at ICLR 2027

MosaicQuant: Inlier–Outlier Disaggregation for Unified 4-Bit LLM Quantization

Abstract

4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs). However, its limited bit-width representation struggles to faithfully capture both dense common values (*inliers*) and rare large-magnitude values (*outliers*), causing substantial accuracy degradation. Existing mixed-precision methods mitigate this by retaining outliers in high precision, but at the cost of breaking the uniformity of low-bit execution, introducing precision conversion and extra data movement that undermine practical speedup. We propose **MosaicQuant**, a unified 4-bit LLM quantization paradigm built on a novel principle of *inlier–outlier disaggregation*. Rather than elevating outlier precision, MosaicQuant quantizes the full weight matrix into a dense 4-bit base component, where inliers are captured faithfully while outlier are inevitably quantized. A sparse 4-bit residual component is then introduced to compensate for these quantization errors, selectively targeting the most error-critical weight blocks where output distortion is shown to be concentrated. However, a unified representation alone is insufficient, as naïvely executing the sparse residual as a separate kernel still breaks the unified low-bit inference pipeline. To bridge this gap, we introduce **ZipperEngine**, which fuses sparse block computation into the dense 4-bit GEMM kernel via an overlapped pipeline, unifying not only the representation but also the execution into a single coherent low-bit inference pipeline. Extensive experiments on LLaMA3 and Qwen3 demonstrate that MosaicQuant preserves near-FP16 accuracy while achieving up to speedup over the W16A16 baseline.

Primary area:
generative models
Keywords:
LLM quantization, kernel optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cGkFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MosaicQuant, which decomposes weights into a dense 4-bit base and sparse 4-bit residual blocks, avoiding mixed-precision computation for outlier handling. A Hessian-guided strategy is used to select important residual blocks, and ZipperEngine is designed to efficiently fuse the dense and sparse computation. Experiments on several LLMs show competitive quantization accuracy and practical inference speedup.

Critical strengths
  1. The paper proposes a unified 4-bit quantization framework that avoids mixed-precision computation for outlier handling, which is meaningful for efficient low-bit deployment.

  2. The block-wise residual compensation is well aligned with hardware-friendly tiled computation, providing a reasonable co-design between the quantization method and the inference engine.

  3. The paper includes both accuracy evaluation and real kernel-level implementation, demonstrating that the proposed method can achieve competitive quantization performance with practical inference speedup.

Critical weaknesses
  1. The authors claim that outliers exhibit block-wise concentration, but this claim is not sufficiently supported by the experimental evidence. Although Figure 2(a) provides some qualitative visualization, more quantitative evidence is necessary given that this observation serves as an important premise for the method design.

  2. The authors observe that “residual blocks naturally align with the tiled structure of low-bit GEMM,” which is an important premise for the fused operator design. However, this observation is not sufficiently discussed or supported by experimental evidence.

  3. The efficiency evaluation emphasizes kernel fusion more than the actual cost–accuracy trade-off introduced by the sparse residual branch. While Fig. 5 reports the memory and latency overhead at the default setting, the paper does not systematically show how these overheads scale with the residual block budget, nor how they trade off against accuracy recovery.

Questions for authors
  1. How consistent is the claimed block-wise concentration across different layers, modules, and models?

  2. What evidence supports the claimed alignment between residual blocks and low-bit GEMM tiles?

  3. How does the choice of block budget affect the accuracy–memory–latency trade-off?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer cGkF

Q1&W1: Block-wise Concentration across Models, Layers, and Projections

Block-wise gains vary across projections. We compared the estimated Δ-Hessian compensation benefit captured by block-wise and channel-wise selection across six attention/MLP projection per layer in Qwen3-14B (L0–L39; 240 matrices) and Qwen3-32B (L0–L63; 384 matrices). Under the same 10% residual budget limit.

Projection14B Block (%)14B Channel (%)Δ (pp)32B Block (%)32B Channel (%)Δ (pp)
q_proj67.94065.767+2.17261.86259.545+2.316
k_proj68.93666.729+2.20763.82261.625+2.197
v_proj55.94249.269+6.67346.43041.300+5.131
gate_proj68.09468.140−0.04628.51328.282+0.231
up_proj25.68725.707−0.02016.42916.175+0.254
down_proj54.26751.253+3.01451.46249.441+2.022

Each entry is averaged over 40 layers for Qwen3-14B or 64 layers for Qwen3-32B. Δ = Block − Channel, in percentage points. Both selections use a 10% residual budget, except 14B down_proj (9.926471% for both).

On average, block-wise selection captures more estimated compensation benefit than channel-wise selection in 10 of the 12 tested model–projection settings. The largest gains occur in v_proj (+6.67/+5.13 percentage points for 14B/32B), while gate_proj and up_proj are nearly tied.

Q2&W2: Residual Block Alignment with GEMM Tiles

Alignment means that the dense weights and residual blocks cover matching input and output channels. Section 4.2 and Fig. 3(b) explain how this enables shared computation, and Fig. 3 reports the benefit of the resulting fused, overlapped execution.

1. Residual blocks match regions of the dense weights. GEMM (matrix multiplication) processes matrices in small pieces called tiles. Each residual block spans 128 input channels and 128 output channels and can be split into smaller execution tiles. It need not match a single hardware matrix multiply–accumulate (MMA) instruction. At a fixed number of compensated weights, smaller blocks allow finer selection, while larger blocks can reduce indexing and scheduling overhead.

2. Matching regions allow input reuse and direct accumulation. Both the dense weights and residuals use 4-bit values. For each selected region, ZipperEngine reuses the same quantized input (activation) tile and combines the two contributions, with their respective quantization scales, in the same output tile. This avoids loading the inputs again, writing a separate intermediate result, and launching another kernel to merge the outputs (Sec. 4.2).

3. Fusion and overlap reduce measured latency. On the Qwen3-8B v_proj layer, separate residual computation and output merging took 22.3 μs. ZipperEngine fuses these operations with the dense computation and overlaps data movement with computation, achieving a 1.9× speedup over the naive pipeline (Fig. 3). This measures the combined benefit of fusion and overlap.

We will document the exact kernel tile sizes, how each 128×128 residual block is partitioned, and the rationale for this block size. We will also replace “naturally align” with “are organized to be compatible with GEMM tiling.”

AuthorsOpenReview ↗

Author Response to Reviewer cGkF

Q3&W3: Residual Budget and the Accuracy–Memory–Latency Trade-off

Increasing the residual block budget raises memory use and latency and lowers perplexity, but does not consistently improve average task accuracy. We evaluated budgets of 5%, 10%, and 20% on Qwen3-14B and Qwen3-32B:

ModelBlock budgetMemory (GiB) ↓Effective BPW ↓Decode latency ratio ↓Prefill latency ratio ↓Avg. Acc. (%) ↑PPL ↓
14B5%10.9854.33140.9660.96271.209.36
14B10%11.2774.53771.0001.00071.869.29
14B20%11.8624.95041.0641.06871.789.16
32B5%22.2814.33140.9540.96272.758.55
32B10%22.9664.53771.0001.00072.278.51
32B20%24.3364.95041.0571.07171.998.38

Effective BPW is the stored bits per quantized weight, including scales and sparse indices. Decode (token generation) and prefill (prompt processing) latency ratios are normalized to the 10% configuration for each model. PPL (perplexity) measures next-token prediction quality; lower is better.

1. Larger budgets increase memory use. Moving from 10% to 20% increased memory use by 5.2% for 14B and 6.0% for 32B.

2. Larger budgets also increase latency. The same budget increase raised decode and prefill latency by approximately 6–7% across the two models.

3. Accuracy depends on the model and metric. For 14B, moving from 5% to 10% improved average accuracy by 0.66 percentage points, reaching the highest tested accuracy of 71.86%; 20% provided no further improvement. For 32B, 5% gave the highest average accuracy (72.75%) and the lowest resource cost. The 20% budget achieved the lowest PPL for both models (9.16/8.38), with the highest memory use and latency.

Among the tested budgets, the preferred choice therefore depends on the model and whether the priority is task accuracy, perplexity, or resource use.

Reviewer_cGkFOpenReview ↗

Response to Authors’ Rebuttal

Thank you for the detailed responses and additional experiments. The broader block-selection analysis, clarification of tile compatibility, and residual-budget evaluation substantially address my original concerns. Although the individual components build on established ideas, their integration into a unified low-bit compensation and execution framework provides a meaningful contribution, supported by the additional ablations. I will therefore maintain my Weak acceptance rating.

Reviewer_EVkMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MosaicQuant, a unified 4-bit post-training quantization framework for large language models. The authors first adopt an equivalent transformation in the SmoothQuant style to shift the quantization difficulty of activation outliers to the weight side. Then weights are represented as a dense 4-bit base component and a sparse 4-bit residual component to compensate for quantization errors of some critical weight blocks. To select the blocks requiring compensation, the authors put forward the Δ-Hessian score based on diagonal Hessian and prove that Top-K selection achieves optimality under the corresponding simplified reconstruction objective. In addition, the authors design ZipperEngine to reduce extra memory access and computation overhead by fusing dense/sparse INT4 computation and applying overlapped scheduling. The paper evaluates W4A8/W4A4 quantization accuracy on six LLaMA3 and Qwen3 models, and reports an end-to-end throughput speedup of up to 1.24× on the RTX 4090.

Critical strengths

Strength 1: The paper attempts to combine residual compensation with unified low-bit GPU execution. By quantizing both the dense base and sparse residual to 4-bit, MosaicQuant avoids part of the execution overhead of traditional high-precision outlier fallback. ZipperEngine further fuses the two computation paths and provides evaluations of practical GPU kernels and end-to-end inference efficiency, demonstrating certain value in the co-design of algorithms and systems.

Strength 2: The paper conducts W4A8/W4A4 experiments on six LLaMA3 and Qwen3 models of different scales, and compares INT4, FP4, multiple representative PTQ baselines, as well as the storage and latency overhead of residual compensation. Particularly on Qwen3-14B and Qwen3-32B, the FP4 version achieves accuracy close to FP16, indicating that low-bit sparse residual compensation has certain practical potential.

Critical weaknesses

Weakness 1: The core method has limited novelty and is highly related to existing Dense-and-Sparse Quantization work. SqueezeLLM [1] has proposed identifying and isolating outliers and sensitive weights via second-order sensitivity information, decomposing weights into a low-bit dense component and a sparse component, and also investigated GPU execution for the two computational paths. SpQR [2] has also studied separating quantization-sensitive outlier weights from the low-bit main matrix and designed corresponding efficient sparse quantization representations. Therefore, the Inlier–Outlier Disaggregation proposed in this paper is not an entirely new quantization principle. Compared with these works, the main differences of MosaicQuant lie in changing the sparse component to a 4-bit residual, adopting block-wise selection, and designing fused execution kernels. Although these improvements carry certain engineering value, the paper does not sufficiently elaborate on the new properties they bring that cannot be directly obtained by existing Dense-and-Sparse methods, nor does it directly discuss its relationship with SqueezeLLM and SpQR. The current contributions are closer to a combination and extension of existing quantization and system techniques, with insufficient independent methodological innovation.

Weakness 2: The Δ-Hessian score exhibits weak algorithmic innovation and theoretical contributions. The authors first adopt existing SmoothQuant [3]-style equivalent transformation, then sort weight blocks through diagonal Hessian-weighted residual reconstruction error reduction. This essentially applies existing activation-aware second-order sensitivity estimation [1, 4] to residual block selection. The Top-K optimality proven in Theorem 4.1 mainly stems from the additivity of the objective function after diagonal Hessian approximation: when each block has identical cost, selecting the K blocks with the largest gains is a straightforward result. Hence, this theorem does not establish non-trivial optimality guarantees for the full Hessian or the actual W4A4 inference error.

Weakness 3: The theoretical objective of Δ-Hessian is not fully consistent with the practical joint weight–activation quantization objective. The actual reconstruction error defined in Eq. (6) includes both weight quantization and activation quantization, yet the Δ-Hessian derivation in Appendix A mainly optimizes residual reconstruction error based on unquantized activations, without explicitly accounting for activation quantization error and its interaction term with residual compensation. Accordingly, the optimality of Theorem 4.1 only applies to the simplified diagonal-Hessian surrogate and cannot be directly generalized to the real W4A4/W4A8 objectives. Given that activation quantization may introduce substantial errors under W4A4, this discrepancy may affect the practical effectiveness of the selected residual blocks.

Weakness 4: The actual storage budget of the unified 4-bit representation does not fully match the baseline comparison. MosaicQuant stores a dense INT4 base for all weights and additionally saves INT4 residuals for approximately 10% of selected blocks. For equal-sized blocks, the theoretical storage cost of quantized values alone is roughly 4.4 BPW, excluding residual indices, quantization scales, and other metadata. Although Figure 5 reports that the sparse branch incurs only 8.5%–9.5% memory overhead, the paper does not provide complete effective BPW accounting nor conduct comparisons with vanilla W4A4 or other quantization methods under the same practical storage budget. As such, the current accuracy improvements may partially arise from extra residual representation capacity and cannot be fully attributed to the proposed block selection or quantization design.

Weakness 5: The experiments insufficiently validate the proposed motivation and the applicability of the method. The core observation regarding block-wise error concentration in the paper is primarily derived from two v_proj layers in Qwen3-4B, and the stability of this phenomenon across other attention/MLP projections, different layers and models has not been adequately demonstrated. Meanwhile, the authors frame their method as Inlier–Outlier Disaggregation, but the actual block selection relies on Hessian-weighted residual reduction rather than explicit identification of outlier weights, leaving the strength of the connection between selected blocks and the true outlier distribution unclear. Furthermore, the accuracy evaluation mainly focuses on WikiText-2 perplexity and six zero-shot commonsense tasks, lacking more challenging benchmarks such as AIME and MBPP.

Questions for authors

Q1: SqueezeLLM [1] and SpQR [2] have investigated separating outliers or sensitive weights from low-bit dense components and adopting sparse representations to improve quantization accuracy and inference efficiency. Could the authors further clarify the independent technical innovations of MosaicQuant compared with these works, especially whether the unified 4-bit residual representation and block-wise compensation yield new properties that cannot be directly obtained by existing methods?

Q2: The Δ-Hessian score is mainly built upon existing Hessian-aware sensitivity estimation [1, 4], and the Top-K optimality in Theorem 4.1 also directly stems from the additivity of the simplified objective. Could the authors further illustrate the substantial innovations of this scoring mechanism and its theoretical analysis relative to existing Hessian-aware quantization methods?

Q3: The practical reconstruction objective defined by Eq. (6) considers both weight quantization and activation quantization, yet the derivation of Δ-Hessian in Appendix A is mainly based on residual reconstruction error under unquantized activations. Could the authors explain the rationality of ignoring activation quantization error and its interaction term with residual compensation? In particular, can the authors compare block selection based on FP16 activations, quantized activations and full-Hessian information, and verify the correlation between Δ-Hessian scores and actual W4A4/W4A8 output-error reduction to demonstrate whether this theoretical approximation remains reliable under low-bit activation conditions?

Q4: MosaicQuant uses a dense INT4 base plus approximately 10% sparse INT4 residuals, so the theoretical storage cost of quantized values alone is around 4.4 BPW instead of a strict 4.0 BPW. Could the authors provide a complete effective BPW accounting, including dense weights, residual values, quantization scales, sparse indices and other metadata? In addition, can the authors re-compare the accuracy and actual GPU memory footprint of each method under matched effective BPW, and report the accuracy–compression trade-off under different residual sparsity budgets to distinguish the respective benefits brought by extra storage overhead and the proposed quantization mechanism?

Q5: The primary evidence for block-wise error concentration in the paper comes from two v_proj layers of Qwen3-4B, while actual Δ-Hessian selection does not explicitly identify outlier weights. Could the authors verify the universality of block-wise error concentration across different models, Transformer layers, and attention/MLP projections, and analyze the relationship between selected residual blocks and the actual outlier distribution? Moreover, given that current evaluations mainly focus on WikiText-2 PPL and six zero-shot commonsense tasks, could the authors further add more challenging mathematical reasoning and code generation benchmarks such as GSM8K, MATH-500, AIME or MBPP to verify the preservation of complex model capabilities by this method?

[1] Kim, S., Hooper, C. R. C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. SqueezeLLM: Dense-and-Sparse Quantization. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 23901–23923, 2024. [2] Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression. In International Conference on Learning Representations (ICLR), 2024. [3] Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning (ICML), pp. 38087–38099, 2023. [4] Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (GPTQ). In International Conference on Learning Representations (ICLR), 2023.

8 comments
AuthorsOpenReview ↗

Author Response to Reviewer EVkM

Q1&W1: Technical Distinctions from SqueezeLLM and SpQR

Low-bit compensation shares the dense computation. MosaicQuant enables 4-bit residual compensation under W4A4/W4A8 without a high-precision weight branch and integrates that compensation into the dense low-bit matrix multiplication (GEMM).

Prior methods retain high-precision components. Dense-and-sparse decomposition is established by SqueezeLLM [1] and SpQR [2]. These methods target weight-only quantization, retaining 16-bit activations and selected outlier or sensitive weights at high precision. MosaicQuant instead quantizes both weights and activations and represents the residual correction in 4 bits (Sec. 3, Eq. (4)).

Low-bit corrections change the allocation problem. Activation smoothing can make weights harder to quantize, and a 4-bit residual correction introduces its own quantization error. Regions that benefit most from high-precision protection therefore need not offer the greatest benefit from low-bit correction. Δ-Hessian ranks blocks by their estimated error reduction after residual quantization, using a diagonal-Hessian approximation. This accounts for the error left by the correction itself (Sec. 4.1, Algorithm 1).

Block allocation reduces perplexity. Under W4A4 with the same 10% INT4 residual budget, replacing channel-wise compensation with our block selection reduces WikiText-2 perplexity from 14.5 to 13.2 on Qwen3-4B and from 13.1 to 10.8 on LLaMA3-8B (Table 3). These results support the block-allocation design.

Fusion reduces execution overhead. ZipperEngine aligns residual blocks with GEMM tiles, reuses the quantized activation tiles loaded for dense computation, and accumulates residual corrections into the same output tiles (Sec. 4.2, Fig. 3(b)). On the Qwen3-8B V projection, separate residual computation and output merging cost 22.3 μs. Fusion and overlapped execution together achieve a 1.9× speedup over the naive pipeline (Fig. 3).

The contribution combines selection and execution. Simply reducing the precision of sparse values does not by itself provide this benefit-based block allocation or shared execution. Our contribution combines the quantized residual representation, its selection rule, and fused execution. The ablations above evaluate these choices within MosaicQuant.

AuthorsOpenReview ↗

Author Response to Reviewer EVkM

Q2&W2: Δ-Hessian Scoring and Theoretical Guarantees

The score measures achievable correction gains. Δ-Hessian uses established Hessian-based sensitivity information to allocate a limited budget of quantized residual corrections. It scores the error reduction achievable after the correction itself has been quantized.

The allocation problem differs from prior methods. GPTQ [4] uses second-order information to guide weight quantization and error compensation, while SqueezeLLM [1] uses sensitivity information for quantization and retains selected sensitive weights at FP16. Here, the dense base and candidate 4-bit corrections are fixed; the decision is which residual blocks to add.

The score compares error before and after correction. For residual RR, quantized correction Q(R)Q(R), and input-sensitivity weights hih_i,

\left\lbrack \sum_{(i,j)\in B}h_iR_{ij}^{2}- \sum_{(i,j)\in B}h_i\bigl(R_{ij}-Q(R)_{ij}\bigr)^{2} \right\rbrack_{+}.$$ **The remaining error matters.** A block with a larger initial error need not yield a larger gain. For illustration, if block A's weighted error falls from 10 to 8 and block B's from 7 to 1, a one-block budget should select B: its gain is 6 rather than 2. **Top-K selection is optimal for the diagonal approximation.** With fixed candidates and disjoint equal-cost blocks, the diagonal-Hessian objective is additive, so selecting up to $K$ blocks with the largest positive gains gives its exact optimum. **The full-Hessian approximation has a conditional bound.** Let $\Delta$ and $\widetilde\Delta$ denote the full- and diagonal-Hessian compensation gains, and let $S^\star$ and $\widehat S$ maximize them under the same candidates and budget. Appendix A gives $$\Delta(S^\star)-\Delta(\widehat S)\le 2\epsilon_K, \qquad \epsilon_K=\max_{|S|\le K} \left|\Delta(S)-\widetilde\Delta(S)\right|.$$ **Omitted interactions control the approximation error.** Writing $H=D+O$, where $D=\operatorname{diag}(H)$, and $E_S=R-M_S\odot Q(R)$, the appendix bounds $$\left|\Delta(S)-\widetilde\Delta(S)\right| \le \|O\|_2\left(\|R\|_F^2+\|E_S\|_F^2\right).$$ The bound depends on off-diagonal Hessian terms and residual energy, and vanishes when $H$ is diagonal. These guarantees use unquantized calibration activations and do not imply optimality for the full W4A4 objective; Q3 addresses activation quantization. **Existing experiments support the overall design.** Fig. 2(b) compares direct output-error-based selection with weight magnitude, residual norm, and random selection at the same budget, supporting selection by compensation benefit. Table 3 supports the combined block-allocation design, as summarized in Q1. These experiments do not separately measure the contribution of the post-compensation error term to actual output MSE or model perplexity.
AuthorsOpenReview ↗

Author Response to Reviewer EVkM

Q3&W3: Block Selection under Activation Quantization

Activation quantization can change compensation gains. Our score approximates the joint objective: activation-only error cancels between selections, but its interaction with residual compensation remains. We account for this interaction and evaluate selections under the joint objective.

The joint error includes an interaction term. For fixed inputs, quantizers, and candidate corrections, let X,WX,W be the smoothed activations and weights, Z=Q(X)Z=Q(X), R=W−Q(W)R=W-Q(W), CS=MS⊙Q(R)C_S=M_S\odot Q(R), and A=(X−Z)WA=(X-Z)W. Then Eq. (6) becomes

E(S)=∣A+Z(R−CS)∣F2.\mathcal E(S)=\\|A+Z(R-C_S)\\|_F^2.

The term ∣A∣F2\\|A\\|_F^2 is independent of SS, but the interaction between AA and Z(R−CS)Z(R-C_S) depends on SS.

The gain difference identifies the omitted terms. Using unnormalized squared errors throughout, define D=diag⁡(X⊤X)D=\operatorname{diag}(X^\top X) and T=Z⊤Z−DT=Z^\top Z-D. Let Δ(S)=E(∅)−E(S)\Delta(S)=\mathcal E(\varnothing)-\mathcal E(S) be the actual gain and Δ~(S)\widetilde\Delta(S) the diagonal-surrogate gain, equal to the sum of selected blocks' unclipped gains. Then

Δ(S)−Δ~(S)=2⟨R,TCS⟩−⟨CS,TCS⟩+2⟨A,ZCS⟩.\Delta(S)-\widetilde\Delta(S)=2\langle R,TC_S\rangle-\langle C_S,TC_S\rangle+2\langle A,ZC_S\rangle.

The first two terms capture changes in activation statistics and the diagonal approximation; the last captures the activation-error interaction.

The selection guarantee is conditional. With r=∣R∣Fr=\\|R\\|_F and cK=max⁡∣S∣≤K∣CS∣Fc_K=\max_{|S|\le K}\\|C_S\\|_F, every feasible set satisfies

∣Δ(S)−Δ~(S)∣≤∣T∣2(2rcK+cK2)+2∣Z⊤A∣FcK⏟ϵK.\left|\Delta(S)-\widetilde\Delta(S)\right|\le\underbrace{\\|T\\|_2(2rc_K+c_K^2)+2\\|Z^\top A\\|_F c_K}_{\epsilon_K}.

For fixed candidates and disjoint equal-cost blocks, selecting up to KK blocks with the largest positive surrogate gains gives the surrogate optimum S^\widehat S. If S⋆S^\star maximizes the actual gain under the same budget,

Δ(S⋆)−Δ(S^)≤2ϵK.\Delta(S^\star)-\Delta(\widehat S)\le 2\epsilon_K.

This bounds the loss in compensation benefit when the omitted terms are small. Bitwidth or smoothing alone cannot ensure a small ϵK\epsilon_K. The bound also holds when both optima use an exact-KK constraint.

The experiments cover 72 conditions. We tested three layers and six projection types in each of Qwen3-14B and Qwen3-32B under W4A4/W4A8. The suite includes 432 evaluations at a fixed 10% block budget and 61 additional positive-gain selection analyses. The table reports four methods at a 10% budget, excluding the 61 analyses.

Only the scoring method changes. Each condition fixes the dense base, residual quantizer, candidate blocks, calibration and held-out data, and 10% budget. All methods scored each block from the same uncompensated calibration base. Method (iii) used the complete layer residual, including interactions outside each candidate block; method (iv) used exact single-block gains under Eq. (6).

The table measures joint compensation on held-out data. We applied each selected set jointly and measured output MSE. Values are mean percentage reductions from the uncompensated base; positive means improvement.

Scoring method14B W4A414B W4A832B W4A432B W4A8
(i) Pre-quantization activations + diagonal Hessian2.17%4.56%2.44%4.47%
(ii) Quantized activations + diagonal Hessian2.14%4.56%2.42%4.47%
(iii) Quantized activations + full Hessian, weight-error term only2.87%6.73%3.38%7.83%
(iv) Direct single-block gain under Eq. (6)3.10%6.74%2.10%5.92%

The existing score gives positive average gains. All four model/precision groups show positive mean reductions. Quantized diagonal statistics change these means by at most 0.03 percentage points. Retaining the full Hessian improves mean reductions in every group, showing the value of off-diagonal terms for these selections.

Single-block gains need not add. Method (iv) outperforms method (iii) for 14B but underperforms it for 32B. For fixed candidates and evaluation data, writing CS=∑b∈SCbC_S=\sum_{b\in S}C_b gives

Δ(S)=∑b∈SΔ(b)−2∑b,b′∈S,,b<b′⟨ZCb,ZCb′⟩.\Delta(S)=\sum_{b\in S}\Delta(\\{b\\})-2\sum_{b,b'\in S,\\,b\lt b'}\langle ZC_b,ZC_{b'}\rangle.

Interactions between corrections can therefore offset their individual gains. Calibration gains may also change on held-out data; the table does not separate these effects.

The evidence supports average gains. The results evaluate compensation under Eq. (6), but do not establish improvement in every condition, measure ϵK\epsilon_K, or quantify score–gain correlations. The guarantee remains conditional.

Pre-quantization statistics permit weight reuse. With smoothing and weight/residual quantizers fixed, the same quantized weights and residual mask can be reused across activation quantizers, such as A8/A4 or different group sizes. Its accuracy cost remains an empirical question.

Reviewer_ZKTaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MosaicQuant targets W4A4/8 quantization and compensates quantization errors with additional local residual blocks. Both the residual blocks and the dense quantized weights remain 4-bit, enabling custom kernels for acceleration. This trades additional storage and computation for accuracy. Residual block selection uses a GPTQ-inspired Hessian criterion, while activation handling relies on SmoothQuant-style equivalent transformations.

Critical strengths
  1. The design is clean, combining 4-bit residual blocks with corresponding kernels to trade additional storage and computation for accuracy.
  2. The residual block selection has a theoretical basis and supporting analysis.
Critical weaknesses
  1. The experiments do not report the actual bits per weight (bpw), making the comparisons difficult to assess. See Q1.
  2. The residual block sizes/shapes and their overall proportion across the model are not clearly specified, even residual blocks are actually central to the method.
  3. The size of the additional storage needs to be evaluated. If other methods achieve comparable accuracy at the same bpw by using smaller group sizes, the benefit of residual blocks would be substantially reduced, especially given their dedicated kernel requirements.
  4. Appendix C.1 explicitly states the authors ran FlatQuant, yet it appears nowhere in the accuracy tables.
Questions for authors

1.The appendix only states: "For W4A8, we use dynamic per-token activation quantization and per-channel weight quantization. For W4A4, activations and weights use per-group and per-channel symmetric quantization, respectively." However, the W4A4 activation group size, the sparse residual scale configuration, and the quantization configurations of the baselines are not specified. Could the authors provide these details and MosaicQuant’s actual bpw? These omissions make me skeptical when assessing the experimental results.

2.The kernel evaluation does not isolate the benefit of the overlapped pipeline from kernel fusion. Separate ablations are needed to establish what the proposed scheduling contributes beyond fusion alone.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer ZKTa

Q1&W1&W2: Quantization Configuration, Residual Allocation, and Effective BPW

1. Activation quantization. W4A4 and W4A8 use dynamic per-token symmetric quantization with groups of 128 elements along the input dimension.

2. Weight and residual quantization. Dense INT4 weights use one FP16 scale per output channel and 128-element input group. Each INT4 residual block (128×128) uses 128 FP16 row scales (256 bytes). Block-column indices and row pointers use INT32. No zero-point tensors are stored.

Within each Transformer block, the 10% residual-block budget is applied independently to each of the seven projection matrices: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.

3. Effective bit-width. At the default 10% residual budget, the INT4 configuration uses 4.5377 bits per weight (bpw). We count dense and residual values, FP16 scales, and sparse indices, and divide by the number of elements in the original quantized weight matrices:

bpw=8(Bweights+Bscales+Bindices)Nquantized weights.\mathrm{bpw}=\frac{8(B_{\mathrm{weights}}+B_{\mathrm{scales}}+B_{\mathrm{indices}})}{N_{\mathrm{quantized\ weights}}}.

MethodResidual budgetEffective bpw
MosaicQuant (INT4)5%4.3314
MosaicQuant (INT4)10% (default)4.5377
MosaicQuant (INT4)20%4.9504

4. FlatQuant configuration. In the added W4A4 comparison, FlatQuant uses weight groups of 64 within each channel, activation groups of 128, and FP32 scales for both. All compared methods use an FP16 KV cache, which stores attention keys and values during generation. FlatQuant's weight values and group scales require 4.5 bpw; W3 gives the accounting and comparison.

5. Other baseline configurations. RTN, GPTQ, AWQ, SmoothQuant, QuaRot, and SpinQuant use symmetric weight quantization with group size 64, while ResQ and Atom use group size 128; all eight use FP16 scales. Quantized activations follow MosaicQuant's dynamic per-token symmetric group-128 scheme with FP16 scales, subject to the mixed-precision exceptions below; W4A16 retains 16-bit activations, and all methods use KV16. Quantization covers all seven projection matrices. Counting weight values and one FP16 scale per 64 weights, RTN, GPTQ, AWQ, SmoothQuant, QuaRot, and SpinQuant require 4+16/64=4.25004+16/64=4.2500 bpw. For ResQ, the published 1/8-INT8, 7/8-INT4 allocation gives a nominal 4(7/8)+8(1/8)+16/128=4.6254(7/8)+8(1/8)+16/128=4.625 bpw for matrices with this split. For an input dimension of 4096, Atom retains 128 input channels in INT8 and requires 4.125+4(128/4096)=4.25004.125+4(128/4096)=4.2500 bpw. These analytical costs cover weight values and group scales only; model-level averages require parameter-weighted aggregation, and any stored transformations, reordering indices, and padding must be accounted for separately. The accuracy experiments use local implementations: GPTQ uses group-wise Hessians with damping 0.01; AWQ and SmoothQuant use channel scaling (α=0.5\alpha=0.5) followed by W4 RTN, without AWQ's official clipping search; QuaRot, SpinQuant, ResQ, and Atom use reference adapters rather than full reproductions of the official pipelines. We will document these configurations in the revised appendix.

AuthorsOpenReview ↗

Author Response to Reviewer ZKTa

Q2: Kernel Fusion and Overlapped Pipeline Ablation

1. Ablation design. We compared three variants, each using the same separate kernel for activation quantization and layout packing:

  • Base (B, 4 kernels): Three additional kernels perform dense computation, sparse residual computation, and output accumulation separately.
  • Fusion-only (F, 2 kernels): One compute kernel combines these three operations, without explicit prefetching of the next tile.
  • Fusion+Pipeline (P, 2 kernels): The same fused computation additionally loads packed operands and scales for the next tile into registers ahead of use.

Thus, F and P have the same kernel count and differ in pipelined prefetching.

2. Experimental setup. We used a single RTX 4090 (24 GB) with CUDA 12.8, covering qkv_fused, o_proj, gate_up_fused, and down_proj at batch sizes b∈{4,8,16,32}b\in\{4,8,16,32\}. Decode (token generation) used M=bM=b input rows; prefill (prompt processing) used M=2048bM=2048b, giving 32 configurations in total.

All variants shared identical quantized inputs, weights, scales, residual masks, and indices. Dense weights and residuals used INT4, with A4 activations, group size 128, and approximately 10% active 128×128 residual blocks. Scales were FP16, accumulation was FP32, and outputs were BF16.

3. Additional pipeline benefit. P achieves geometric-mean speedups of 1.349× for decode and 1.243× for prefill over F, showing an additional benefit from pipelined prefetching beyond fusion.

B/F, F/P, and B/P are ratios of measured latencies. Each table entry is the geometric mean across the four projection groups at the indicated batch size.

WorkloadBatchFusion-only vs. Base (B/F)Fusion+Pipeline vs. Fusion-only (F/P)Total speedup (B/P)
Decode41.318×1.289×1.698×
Decode81.332×1.315×1.751×
Decode161.203×1.418×1.707×
Decode321.326×1.380×1.829×
Prefill41.896×1.239×2.350×
Prefill81.866×1.273×2.375×
Prefill161.950×1.227×2.393×
Prefill321.935×1.233×2.386×

P outperforms F in all 32 evaluated configurations. Across the 16 configurations in each workload, fusion alone gives geometric-mean speedups of 1.293× for decode and 1.911× for prefill over B. The combined speedups are 1.745× and 2.376×, respectively.

Full per-configuration latencies and matrix dimensions are reported in Appendix C.3, Table 9. All 32 configurations have F/P > 1, and speedups are computed before display rounding.

W4: FlatQuant Accuracy Comparison

We added FlatQuant results on Qwen3-14B and Qwen3-32B under the same W4A4 evaluation protocol. FlatQuant used weight groups of 64, activation groups of 128, and FP32 scales for both. All methods used an FP16 KV cache (KV16), rather than KV4, to keep cache precision consistent.

MosaicQuant (FP4) achieves 0.66 and 1.24 percentage points higher average accuracy than FlatQuant on 14B and 32B, respectively, with lower WikiText-2 perplexity. FlatQuant outperforms MosaicQuant (INT4) on both metrics.

ModelMethod (W4A4)Avg. Acc. (%) ↑Wiki2 PPL ↓
Qwen3-14BFlatQuant72.849.17
MosaicQuant (INT4)71.869.29
MosaicQuant (FP4)73.509.00
Qwen3-32BFlatQuant73.378.17
MosaicQuant (INT4)72.278.51
MosaicQuant (FP4)74.618.02

We will include these results in the revised accuracy tables.

AuthorsOpenReview ↗

Author Response to Reviewer ZKTa

W3: Residual Storage Overhead and Accuracy at Similar BPW

1. Additional storage from residual compensation. The dense 4-bit base, with one FP16 scale per 128 weights, uses 4 + 16/128 = 4.125 bpw. At the default 10% residual budget, MosaicQuant uses 4.5377 bpw. Residual values, their scales, and sparse indices therefore add approximately 0.4127 bpw, or 10.0% over the dense base's weight storage.

The FP4 variant uses the same grouping, residual block size, FP16 scales, and INT32 sparse metadata as INT4. Both store their quantized values in 4 bits, so they have the same effective bpw at the same residual budget.

2. Comparison at similar storage budgets. We evaluated smaller-group FlatQuant to test the alternative suggested by the reviewer. With 4-bit weights, a weight group size of 64, and one FP32 scale per group, FlatQuant requires 4 + 32/64 = 4.5 bpw for weight values and scales. MosaicQuant's 4.5377 bpw is 0.84% higher than this baseline.

All methods use W4A4, an FP16 KV cache, and the same evaluation protocol.

ModelMethodEffective bpwAvg. Acc. (%) ↑Wiki2 PPL ↓
Qwen3-14BFlatQuant4.500072.849.17
MosaicQuant (INT4)4.537771.869.29
MosaicQuant (FP4)4.537773.509.00
Qwen3-32BFlatQuant4.500073.378.17
MosaicQuant (INT4)4.537772.278.51
MosaicQuant (FP4)4.537774.618.02

BPW refers to quantized weight storage, excluding activations and the KV cache.

3. What the comparison shows. At these similar reported storage budgets, MosaicQuant (FP4) achieves higher accuracy and lower perplexity on both models, while MosaicQuant (INT4) trails FlatQuant. The accuracy–storage advantage in this comparison therefore applies to the FP4 variant.

FlatQuant uses learned affine transformations, while MosaicQuant uses channel-wise smoothing and residual correction. MosaicQuant's FP4 variant uses non-uniform quantization levels.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of feedback and improvements

Dear Area Chair,

Thank you for your time and consideration. We summarize MosaicQuant’s contributions, the reviewers’ assessments, and the progress made during the discussion.

Core Contributions and Recognized Strengths

  • Unified 4-bit residual compensation. MosaicQuant combines a dense 4-bit base with sparse 4-bit corrections under W4A4/W4A8. ZKTa recognizes the clean residual-and-kernel design, while EVkM and cGkF recognize the practical value of integrating quantization with low-bit GPU execution.

  • Benefit-based block allocation. Δ-Hessian ranks blocks by estimated error reduction after residual quantization. ZKTa notes its theoretical basis and supporting analysis. At the same 10% INT4 residual budget, replacing channel-wise compensation with block allocation lowers perplexity from 14.5 to 13.2 on Qwen3-4B and from 13.1 to 10.8 on LLaMA3-8B (Table 3).

  • Shared low-bit execution. ZipperEngine reuses activation tiles and fuses dense computation, residual compensation, and output accumulation. EVkM and cGkF recognize the algorithm–system integration and practical GPU evaluation. Pipelining adds geometric-mean speedups of 1.243× for prefill and 1.349× for decode over fusion alone (Table 9).

The paper also reports end-to-end throughput improvements of up to 1.24× over the 16-bit baseline on an RTX 4090.

Key Issues and Responses

  • Experimental Settings and Storage Accounting (ZKTa, EVkM). We added detailed quantization and baseline configurations, effective-BPW accounting, FlatQuant comparisons, and 5%/10%/20% residual-budget evaluations. At similar reported BPW, FP4 MosaicQuant outperforms FlatQuant, while INT4 trails it; separate preparation-time measurements provide an additional practical comparison.

  • Theoretical Scope and Selection Evidence (EVkM). We clarify that exact Top-K optimality applies to the diagonal surrogate and provide conditional bounds and activation-quantization analyses. The full-Hessian variant improves mean reconstruction in all four reported settings. Our fixed-budget INT4 ablation supports the combined block-allocation design without relying on FP4.

  • Systems Analysis and Broader Evaluation (ZKTa, cGkF, EVkM). We separate fusion from pipelining across 32 configurations, explain tile compatibility, and provide broader layer/projection analyses and MATH-500/MBPP evaluations. These additions are documented in the revised manuscript and discussion responses.

Discussion Progress

  • ZKTa. ZKTa welcomes the detailed settings but retains the original rating, citing representation complexity, novelty, and the INT4 comparison with FlatQuant. Our subsequent reply provides preparation-time measurements and distinguishes MosaicQuant’s selection and execution design from MBOK and VPTQ.

  • EVkM. EVkM maintains the original rating, citing the limited theoretical guarantee and uncertainty about selection-specific benefits. Our follow-up distinguishes the complete-method FlatQuant comparison from the fixed-budget INT4 ablation and adds measured preparation costs.

  • cGkF. cGkF states that the broader analysis, tile-compatibility clarification, and budget evaluation substantially address the original concerns. The reviewer recognizes the integrated framework as a meaningful contribution and maintains Weak Acceptance.

Detailed point-by-point responses and supporting results are available in the reviewer discussion threads. Thank you again for your time and effort in handling our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.