- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies post-training quantization for diffusion language models (DLMs). The authors argue that DLMs process two structurally different token groups, i.e., visible tokens as conditioning context and masked tokens as prediction targets, so the standard layer-wise MSE reconstruction objective that treats all tokens equally is mismatched with the true quantization goal. Through a first-order error propagation analysis, they bound the final masked-token error by an input-dependent weighted combination of the group-wise reconstruction errors of the Q, K, V and O projections. They then propose MV-Quant, which replaces the MSE objective in AWQ with a worst-group (min-max) objective, and prove that it yields a tighter error bound under the same error tolerance. Experiments on LLaDA-1.5, Fast-dLLM v2 and SDAR over seven benchmarks show improved average accuracy over RTN, GPTQ and AWQ.
Critical strengths
- The paper points out that DLMs contain two token groups with different roles, i.e., visible tokens and masked tokens, which are treated equally by the standard MSE objective, and turns this observation into a simple calibration objective. Since MV-Quant only replaces the reconstruction objective of AWQ, it is easy to implement and introduces no extra inference cost.
- The paper provides an error propagation analysis with detailed proofs in the appendix, and evaluates the method on three DLMs and seven benchmarks covering math, code, knowledge and instruction following. MV-Quant achieves the best average accuracy among the quantized models in all reported settings.
Critical weaknesses
- Sec. 3 states that β differs significantly across Q, K, V and O and shifts dynamically across sequences, which is the main motivation for the worst-group objective. Could the authors provide empirical measurements or visualizations of α and β on real models to support this claim?
- The analysis and the objective (Theorem 3.1 and Eq. (9)) only involve , , and , and the transformer layer in Sec. 2.3 does not include the FFN. How are the FFN layers quantized in MV-Quant, with the standard MSE objective or with the proposed one?
- Some implementation details are missing. The paper only states that pileval is used for calibration. How are the masked and visible tokens constructed during calibration (e.g., what masking ratio is used), how many calibration samples are used, and what group size is adopted? The evaluation setup only mentions “standard evaluation protocols” without giving the generation length, denoising steps or number of shots.
- There is no ablation study. For example, how does the proposed max objective compare with a weighted sum of the masked-token and visible-token reconstruction errors under different weights? How sensitive are the results to the masking ratio used in calibration? In addition, Eq. (9) is not normalized by the number of tokens in each group; would such normalization affect the results?
- The average improvement over AWQ on LLaDA-1.5 8B is only +0.3 (Table 3), much smaller than that on Fast-dLLM v2 (+1.5) and SDAR (+1.6). Since LLaDA is the only model trained from scratch among the three, could the authors explain the reason for this difference?
- Table 5 only compares FP and MV-Quant. Since MV-Quant only replaces the objective of AWQ, an AWQ row and the calibration time of both methods should be reported.
- Since MV-Quant is presented as a general quantization objective, can it also be combined with other PTQ methods such as GPTQ, and does it bring similar improvements?
Questions for authors
Please refer to the weakness part.
5 comments
Part2
W3: Some implementation details are missing. The paper only states that pileval is used for calibration. How are the masked and visible tokens constructed during calibration (e.g., what masking ratio is used), how many calibration samples are used, and what group size is adopted? The evaluation setup only mentions "standard evaluation protocols" without giving the generation length, denoising steps or number of shots.
Thank you for pointing this out. We provide the exact implementation details below and have included a full hyperparameter table in Sec. 5.1 and the Appendix:
- Quantization Configuration: Tables 1–3 use W4A16 (4-bit asymmetric weight-only quantization with zero-point, group size = 128, and 16-bit FP16 activations), while Table 4 uses W3A16 (group size = 128). Following AWQ, both per-channel scaling ratios and weight clipping thresholds are searched over a 20-point uniform grid.
- Calibration Data & Mask Construction: Following AWQ, we sample 128 text sequences from pileval. We independently sample and set the masking probability (expected masking ratio ). Each position is independently masked with
[MASK]via . - Evaluation Setup: All baselines and MV-Quant strictly follow the official evaluation protocol of each backbone. Under dynamic parallel decoding with a confidence threshold of , the number of denoising steps is adaptive rather than fixed. Specifically: For Fast-dLLM v2 7B, MMLU and GPQA are evaluated under 5-shot likelihood, GSM8K, MATH, and IFEval under 0-shot generation with a maximum generation length of 2,048 tokens, and HumanEval and MBPP under 0-shot greedy decoding with a maximum length of 768 tokens. For SDAR 8B, we set a maximum generation length of 4,096 tokens, adopting 5-shot prompting for MMLU and 0-shot prompting for all other tasks. For LLaDA-1.5 8B, we set the generation length and block size to for MMLU (5-shot), for GSM8K (4-shot), GPQA (5-shot), and IFEval (0-shot), for MATH (0-shot), and for HumanEval (0-shot) and MBPP (3-shot).
W4: There is no ablation study. For example, how does the proposed max objective compare with a weighted sum of the masked-token and visible-token reconstruction errors under different weights? How sensitive are the results to the masking ratio used in calibration? In addition, Eq. (9) is not normalized by the number of tokens in each group; would such normalization affect the results?
First, regarding the max objective and weighted sum: comparing AWQ and MV-Quant in Tables 1–4 already directly compares the equal-weighted sum (, Eq. 8) against our objective (Eq. 9) under identical settings.
For the -weighted sum, first, computing the exact weights is computationally expensive due to cross-layer Jacobians (see W1). Moreover, as shown by our empirical measurements in W1, differs across , varies across layers, and shifts dynamically across sequences within , so no fixed static weight can align across all layers and inputs. This directly motivates our parameter-free min-max objective as a tractable surrogate.
Second, regarding the calibration masking ratio and token-count normalization: during DLM inference, all masking ratios appear across denoising stages. Following the standard training procedure of masked DLMs (e.g., LLaDA), we construct calibration inputs using the same random masking process: for each calibration sequence, we sample and independently replace each token with [MASK] with probability , covering diverse masking ratios across the denoising process without relying on a fixed masking ratio. Since , the expected masking ratio over the calibration dataset is 0.5, so the aggregated numbers of masked and visible tokens are balanced. Therefore, neither group dominates Eq. (9) due to group size, and token-count normalization is unnecessary. Whichever group has a larger value in Eq. (9) reflects higher per-token reconstruction error.
Part3
W5: The average improvement over AWQ on LLaDA-1.5 8B is only +0.3 (Table 3), much smaller than that on Fast-dLLM v2 (+1.5) and SDAR (+1.6). Since LLaDA is the only model trained from scratch among the three, could the authors explain the reason for this difference?
Fast-dLLM v2 and SDAR are initialized from pre-trained autoregressive models, which process only clean visible tokens during pre-training, and are subsequently fine-tuned into diffusion models by introducing an additional [MASK] token. This induces a stronger distributional discrepancy between visible- and masked-token activations, causing uniform MSE (AWQ) to suffer larger group imbalance and yielding larger gains for MV-Quant at 4-bit. In contrast, LLaDA-1.5 is trained from scratch purely with masked diffusion, so its masked and visible representations co-evolve more smoothly. Since recent state-of-the-art DLMs predominantly adopt this AR-initialized fine-tuning paradigm (e.g., DiffusionGemma, Fast-dLLM v2, and SDAR), this further highlights the advantage of MV-Quant.
W6: Table 5 only compares FP and MV-Quant. Since MV-Quant only replaces the objective of AWQ, an AWQ row and the calibration time of both methods should be reported.
Compared with AWQ, MV-Quant only replaces the scalar MSE loss with the min-max loss, while keeping all other components, calibration data (), grid search steps (), and optimization procedures unchanged. Consequently, MV-Quant introduces no calibration overhead, requiring virtually the same PTQ calibration time as AWQ. Both AWQ and MV-Quant take approximately 15 minutes for 7B/8B models).
During inference, because both AWQ and MV-Quant pack weights into the exact same 4-bit format (W4A16, group size 128) and execute the identical INT4 kernel, AWQ has the same model memory footprint and kernel-level inference throughput as MV-Quant in Table 5.
W7: Since MV-Quant is presented as a general quantization objective, can it also be combined with other PTQ methods such as GPTQ, and does it bring similar improvements?
Thank you for your suggestion. We integrate our MV objective into two distinct PTQ frameworks GPTQ and Quant-dLLM, while keeping all other quantization and evaluation settings unchanged. As shown in Table B below, Quant-dLLM + MV also improves the performance of LLaDA 8B under 2-bit quantization from 56.3 to 57.1. Moreover, as shown in Table A below, GPTQ + MV improves the average performance of GPTQ over the seven benchmarks from 60.1 to 60.5. Together with AWQ, these results confirm that MV generalizes across activation-aware AWQ, Hessian-based GPTQ, and extreme low-bit Quant-dLLM frameworks.
Table A: Combining the MV objective with GPTQ (4-bit) on Fast-dLLM v2 7B.
| Method | GSM8K | Math | HumanEval | MBPP | GPQA | MMLU | IFEval | Avg. |
|---|---|---|---|---|---|---|---|---|
| GPTQ | 80.5 | 52.9 | 63.0 | 60.7 | 27.9 | 66.4 | 69.5 | 60.1 |
| GPTQ + MV | 81.2 | 53.0 | 63.0 | 61.0 | 28.3 | 67.1 | 70.1 | 60.5 |
Table B: Combining the MV objective with Quant-dLLM (2-bit) on LLaDA-8B-Base.
| Benchmark | Quant-dLLM | +MV |
|---|---|---|
| PIQA | 68.1 | 68.2 |
| ARC-C | 36.4 | 37.5 |
| ARC-E | 66.0 | 68.9 |
| HellaSwag | 44.0 | 44.5 |
| WinoGrande | 66.9 | 66.5 |
| Average | 56.3 | 57.1 |
Part1
In practice, computing the exact theoretical αi(l) and βi(l) in Theorem 3.1 is expensive in both memory and time, as Al and Bl require chaining full cross-layer Jacobians T(n)⋯T(l+1) across all subsequent layers:
This cost is also a reason why we adopt the parameter-free worst-group objective (Eq. 9) as a tractable surrogate for Eq. (7). However, to directly address your concern, we randomly select two layers (Layer 1 and Layer 14) on Fast-dLLM v2 7B and compute their coefficients αi(l) and βi(l). As shown in the table below, the results support the claims in Sec. 3: αi(l) and βi(l) vary noticeably across {WQ,WK,WV,WO}, exhibit clear group imbalance, and shift substantially across different input sequences.
For the FFN/MLP layers, we also apply the proposed MV-Quant objective rather than the standard uniform MSE loss. Although the FFN itself operates token-wise without direct token-to-token interactions, the quantization error introduced by an FFN layer becomes part of the hidden-state error and is subsequently propagated through the following Transformer layers via the propagation matrices T(l) in Eq. (5). After decomposing this error into masked- and visible-token components, the two components propagate through the distinct cross-layer operators Al and Bl in Proposition 3.2, and thus contribute asymmetrically to the final masked-token error.