acceptodds
Withdrawn from ICLR 2027

MUQuant: Mask State Adaptive Dual-Center Quantization for Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Block-diffusion language models repeatedly process partially masked blocks during generation. Within a decoding step, masked and unmasked tokens can have distinct activation centers, making a shared quantization origin poorly suited to either population. Calibration on fully unmasked text also misses the activation mixtures encountered during denoising. **MUQuant** addresses these two sources of W4A4 error with a state-conditioned representation. **MuZero** estimates separate centers for masked and unmasked tokens, and **MuCal** calibrates weights on the reconstructed activations collected along denoising rollouts. **MuEngine** then computes the center and residual contributions in one packed four-bit multiplication, avoiding separate weight passes. Across three diffusion language-model families, this approach improves quantized accuracy while preserving the decoding procedure. On Nemotron-Labs-Diffusion-8B, the evaluated NVFP4 implementation achieves up to **3.24× speedups over BF16**.

Primary area:
generative models
Keywords:
Quantization, Diffusion Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BACmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MUQuant, a W4A4 quantization framework for diffusion language models. Its key observation is that masked and unmasked tokens can have different channel-wise activation means within the same denoising step. MuZero computes a separate center for each group at every forward pass, quantizes the centers, and then quantizes each token’s residual relative to its reconstructed center. MuCal collects intermediate states along denoising trajectories and uses the reconstructed activations to build calibration statistics for GPTQ. MuEngine computes the center and residual contributions using a single packed 4-bit matrix multiplication and then combines their outputs. The paper reports higher average F1 scores than the evaluated quantization baselines on three models across six LongBench QA tasks. It includes results for both INT4 and NVFP4, ablations on token-state grouping, deployment latency measurements, and additional results on JevBench.

Critical strengths
  1. The method is closely integrated with its system implementation. MuZero uses the existing mask states to compute up to two shared centers. MuEngine then computes the center and residual contributions together, avoiding an additional full pass over the weights to support the improved representation. The paper explains how the activations are reconstructed and considers the practical cost of low-bit execution, making it more convincing than work that only reports simulated quantization error.
  2. The improvements are reasonably consistent across models and supported by targeted ablations. In Table 1, MUQuant achieves higher INT4 macro F1 scores than QuaRot+GPTQ on all three models. The full NVFP4 method in Table 2 also improves macro F1 over this baseline across all three models. Table 3 compares state-based grouping with random and mixed grouping under the same two-center budget, providing direct evidence for the value of token-state information.
Critical weaknesses
  1. MuCal’s two design choices need separate ablations. The current comparison also changes whether weight calibration is performed. Please compare raw versus reconstructed Hessian inputs with MuZero, samples, and rollout states fixed, and separately vary calibration-state sources with the representation fixed. These controls would clarify MuCal’s necessity and contribution.
  2. Some comparisons and reporting details need clarification. DLLMQuant’s near-zero scores on DiffusionGemma require clearer adaptation details and failure analysis. Timing and decoding protocols also need more detail. Moreover, MUQuant+MuCal’s top-5 probabilities in Figure 9(c) sum to 1.400 and require a verifiable correction. These issues do not directly invalidate the main results, but prevent me from giving a stronger recommendation.
Questions for authors
  1. Could you compare ∑X⊤X\sum X^\top X versus ∑X~⊤X~\sum \widetilde{X}^\top \widetilde{X} for Hessian estimation, keeping MuZero and the rollout data fixed? Please also compare teacher-forced versus rollout calibration with the representation fixed, and report the number of calibration states, total calibration time, and baseline settings.

  2. Why does DLLMQuant obtain near-zero scores on DiffusionGemma under both INT4 and NVFP4? Was the original implementation used, which components were enabled, and how was it adapted to this model? Does this failure persist at higher precision or when components are enabled individually?

  3. Please verify whether values such as 0.946 and 0.448 in Figure 9(c) are probabilities from the same distribution, and clarify whether any correction affects Table 4. Please also provide the hardware, software stack, decoding settings, and protocol for repeated timing measurements used in Figure 7 to support reproduction of the reported speedups.

Reviewer_JoS6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MUQuant addresses differences between the activation centers of masked and unmasked tokens in diffusion language models by assigning each group a separate quantization center and representing both the center and residual at 4 bits. MuCal calibrates GPTQ weights using activations reconstructed from this representation during denoising rollouts. MuEngine packs the center and residual contributions for execution in one low-bit matrix multiplication. The authors evaluate INT4 and NVFP4 on LongBench QA subsets across three model families and report measured inference speed. For example, on an RTX 5090, DiffusionGemma-26B runs about 3.2× faster per step than NF4 W4A16, at roughly 1.9 GiB higher memory use.

Critical strengths
  1. The method incorporates masked/unmasked state differences into the activation representation rather than only using state information to reweight calibration. The center-grouping ablation supports the state-based grouping choice.
  2. The paper reports more than quantized accuracy: it measures inference time on an actual GPU and describes a packed execution path using a single multiplication.
Critical weaknesses
  1. The main LongBench results do not report variability across random seeds or repeated runs. Calibration relies primarily on 128 WinoGrande examples, and the sensitivity to the calibration set is unclear.
  2. The ablations do not fully separate the contributions of MuZero’s reconstructed representation and MuCal’s rollout calibration. The results also show that MUQuant does not outperform the baselines on every dataset or format.
Questions for authors
  1. Could you add a factorial ablation that uses the same rollout examples but calibrates GPTQ on the original activations rather than MuZero-reconstructed activations? This would help separate the benefits of deployment-state coverage from those of the dual-center representation.
  2. How many independent runs or decoding seeds underlie the main LongBench results and speed measurements? If repeated results are available, please report variability in macro F1 and latency.
Reviewer_UMaeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes a state adaptive dual center quantization framework MUQuant for diffusion language models, aimed at alleviating low bit quantization errors caused by inconsistent activation distributions under different mask states. Specifically, the author designed three main components: MuZero calculates different channel centers based on the MASK/UNMASK state of the token, and quantifies the centers and residuals separately; MuCal utilizes the activation in the actual denoising trajectory and performs GPTQ calibration on the reconstructed activation representation; MuEngine combines the center and residuals into a low bit matrix multiplication to reduce additional computational overhead.

Critical strengths

The problem has practical significance. The method design is relatively complete.

Critical weaknesses
  1. The independent contribution and necessity of dual center quantification have not been fully validated. Although MuZero can reduce quantization errors in experiments, it essentially introduces an additional shared representation. Compared to directly using a single 4-bit activation quantization, MuZero uses two quantization components to jointly reconstruct the original activation, so its performance improvement may come from both state information and stronger representation capabilities.The paper lacks comparison with more direct alternatives, such as a single global shared center, other adaptive center selection strategies, or grouping quantization methods with similar representation overhead. Therefore, it is currently difficult to determine to what extent the performance improvement comes from modeling the differences in MASK/UNMASK states, and to what extent it comes from additional central representations and residual encoding.
  2. The innovation of MuCal and its advantages over existing calibration methods are not clear enough. According to Section 4.2, its core weight quantification process still uses the standard GPTQ and does not change the optimization and weight update mechanism of GPTQ. However, the idea of constructing calibration data using different denoising states has been studied in methods such as DLLMQuant. Therefore, the most valuable part of MuCal should be to align weight calibration with the actual deployed quantitative activation representation.
  3. The improvement in computational efficiency cannot be fully attributed to the specific design of MuEngine. Figure 6 mainly compares the optimized MuEngine with the naive dual path implementation. This can demonstrate the importance of operator fusion for the proposed representation, but it does not directly prove that it has better computational efficiency compared to the standard single path W4A4 quantization implementation. On the other hand, Figure 7 mainly shows the inference delay difference between MUQuant's NVFP4 implementation and the BF16 model. However, low bit Tensor Core itself can provide significant computational advantages, so these accelerations cannot be fully attributed to the dual center fusion computing mechanism proposed in this paper.
Questions for authors

What are the specific advantages of MuZero compared to a single global shared center, other adaptive center selection strategies, and grouping quantization methods with similar additional representation overhead? Can unified ablation be provided on the three models? How much of the performance improvement of MuCal comes from denoising trajectory sampling, and how much comes from using MuZero reconstruction activation for Hessian computation? Can we separate the two? How much additional end-to-end computing overhead does MuEngine introduce compared to a regular single path W4A4 GEMM under the same 4-bit computing core and hardware configuration? Is the difference in channel center between MASK/UNMASK common at different network layers and denoising time steps? Is the proposed central estimation still stable when there are few samples of a certain token state? Can this method maintain performance advantages in tasks such as mathematical reasoning, code generation, and long text generation? Can you provide the variance of multiple experiments for results with small improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.