- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MUQuant, a W4A4 quantization framework for diffusion language models. Its key observation is that masked and unmasked tokens can have different channel-wise activation means within the same denoising step. MuZero computes a separate center for each group at every forward pass, quantizes the centers, and then quantizes each token’s residual relative to its reconstructed center. MuCal collects intermediate states along denoising trajectories and uses the reconstructed activations to build calibration statistics for GPTQ. MuEngine computes the center and residual contributions using a single packed 4-bit matrix multiplication and then combines their outputs. The paper reports higher average F1 scores than the evaluated quantization baselines on three models across six LongBench QA tasks. It includes results for both INT4 and NVFP4, ablations on token-state grouping, deployment latency measurements, and additional results on JevBench.
Critical strengths
- The method is closely integrated with its system implementation. MuZero uses the existing mask states to compute up to two shared centers. MuEngine then computes the center and residual contributions together, avoiding an additional full pass over the weights to support the improved representation. The paper explains how the activations are reconstructed and considers the practical cost of low-bit execution, making it more convincing than work that only reports simulated quantization error.
- The improvements are reasonably consistent across models and supported by targeted ablations. In Table 1, MUQuant achieves higher INT4 macro F1 scores than QuaRot+GPTQ on all three models. The full NVFP4 method in Table 2 also improves macro F1 over this baseline across all three models. Table 3 compares state-based grouping with random and mixed grouping under the same two-center budget, providing direct evidence for the value of token-state information.
Critical weaknesses
- MuCal’s two design choices need separate ablations. The current comparison also changes whether weight calibration is performed. Please compare raw versus reconstructed Hessian inputs with MuZero, samples, and rollout states fixed, and separately vary calibration-state sources with the representation fixed. These controls would clarify MuCal’s necessity and contribution.
- Some comparisons and reporting details need clarification. DLLMQuant’s near-zero scores on DiffusionGemma require clearer adaptation details and failure analysis. Timing and decoding protocols also need more detail. Moreover, MUQuant+MuCal’s top-5 probabilities in Figure 9(c) sum to 1.400 and require a verifiable correction. These issues do not directly invalidate the main results, but prevent me from giving a stronger recommendation.
Questions for authors
-
Could you compare versus for Hessian estimation, keeping MuZero and the rollout data fixed? Please also compare teacher-forced versus rollout calibration with the representation fixed, and report the number of calibration states, total calibration time, and baseline settings.
-
Why does DLLMQuant obtain near-zero scores on DiffusionGemma under both INT4 and NVFP4? Was the original implementation used, which components were enabled, and how was it adapted to this model? Does this failure persist at higher precision or when components are enabled individually?
-
Please verify whether values such as 0.946 and 0.448 in Figure 9(c) are probabilities from the same distribution, and clarify whether any correction affects Table 4. Please also provide the hardware, software stack, decoding settings, and protocol for repeated timing measurements used in Figure 7 to support reproduction of the reported speedups.