acceptodds
Under review as a conference paper at ICLR 2027

Mixed precision inference for Diffusion language models

Abstract

Diffusion Language Models (DLMs) have drawn increasing attention as an alternative to classical Large Language Models architecture due to their potential to significantly increase inference speed. However, they lack in practice some key optimizations. One such feature is mixed-precision inference, i.e., using a quantized or a full precision model at different steps of the generation process. This hybrid design enables one to benefit from the accelerated inference of quantized models while maintaining an output quality similar to that of original floating-point precision. As existing approaches are not applicable to this setup, we propose here the first mixed precision framework specifically targeted for DLMs. By modeling the quantization error, we are able to design a training-free, adaptive mechanism to successfully exploit the speed of a quantized model and the precision of a regular one. Our results show we can achieve a 66% speed increase without significant degradation of the model's performance. We also demonstrate this can be combined with other acceleration systems to further speed up the sampling process.

Primary area:
generative models
Keywords:
Diffusion large language models, quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AAakOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper switches between quantized and full-precision inference at different generation steps, using prediction uncertainty and changes in entropy. The aim is to recover generation quality while retaining much of the benefit of quantization. Experiments mainly use LLaDA-8B, with a supplementary Dream result and comparisons against static precision schedules.

Critical strengths

The method recovers much of the quality lost through quantization. On GSM8K, accuracy increases from 69.8 to 74.7, close to the FP16 result of 74.8. The comparison with several static precision schedules supports studying when full-precision computation is needed.

Critical weaknesses
  1. The reported 1.66× speedup is estimated rather than measured end to end. The paper does not establish the actual cost of keeping both models in memory, switching precision, or performing additional steps. It also lacks results on a competitive modern dLLM, so the practical advantage remains unproven.

  2. The ablations do not match the amount of full-precision computation or the number of additional steps. The quality recovery could therefore come from spending more computation rather than from the proposed selection signals. This prevents a clear assessment of the value of tracking changes in entropy.

Questions for authors
  1. With actual low-precision kernels on SDAR or other competitive dLLMs, what are the measured latency, throughput, and peak memory at matched quality, including switching and additional steps?

  2. Does the proposed method outperform the best static schedule when the full-precision budget and total computation are matched? Does tracking entropy changes still help under this comparison?

  3. Is the entropy window measured in generation steps or committed tokens, and which definition was used for the reported experiments?

Reviewer_16t2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a mixed-precision inference framework for DLMs. It uses uncertainty to estimate per-step quantization error and entropy velocity to indicate error propagation. Two thresholds dynamically control switching between quantized and fp16 kernel to balance accuracy and speed. Experiments on the LLaDA-8B Instruct model demonstrate the effectiveness of the proposed framework in balancing accuracy and speed.

Critical strengths
  1. This paper identifies error propagation in dlm. It uses uncertainty as a surrogate for per-step quantization error and entropy velocity as a surrogate for error propagation. Experiments demonstrate that the two surrogates are strongly correlated with the corresponding quantization error (kl divergence).
  2. This paper introduces two thresholds to dynamically switch between the fp16 kernel and the quantization kernel to accelerate inference while maintaining high accuracy. Experiments demonstrate the effectiveness of the proposed framework.
Critical weaknesses
  1. This paper didn't report the corresponding peak gpu memory. If the dlm needs to support both the fp16 kernel and the quantization kernel, gpu memory analysis is important to determine the storage requirements.
  2. The main results in table1 only cover LLaDA-8B-Instruct, and the evaluation on other models is limited. Additional results on different models and tasks would help demonstrate generalization.
Questions for authors
  1. Could you report the corresponding peak gpu memory? This would help assess the memory footprint of the proposed framework?
  2. Could you report the end-to-end inference decoding throughput comparing MP quant with Fast-dLLM?
  3. Could you add more results based on a different backbone model to table1 to demonstrate generalization? For example, report the accuracy of Dream-7B on the tasks in table1 under the same evaluation setting.
  4. Could you report the end-to-end inference speedup, decode speed and ttft in table2, which includes fast-dllm? The inference settings also need to be reported (e.g. prefill length, batch size). And provide detailed end-to-end inference times for fig 6.
  5. Could you add inference time and peak gpu memory measurements for different batch sizes or input lengths? And report the corresponding decode speed, ttft and speedup. This would help assess the generalizability of the method across different inference scenarios.
Reviewer_QAE8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates mixed-precision inference for diffusion language models (DLMs), with the goal of reducing inference cost while maintaining generation quality. The authors analyze quantization errors throughout the denoising process and propose two complementary indicators: token uncertainty for estimating local quantization error and entropy velocity for estimating error propagation risk. Based on these observations, the paper introduces a training-free scheduling framework that dynamically switches between quantized and full-precision models. The method also incorporates token rejection and additional denoising steps to mitigate potentially harmful commitments. Experiments on LLaDA and Dream, across several reasoning, coding, and instruction-following benchmarks, show that mixed precision can recover much of the quality lost to aggressive quantization while reducing the estimated inference cost.

Critical strengths
  1. Mixed-precision inference is a meaningful direction for accelerating DLMs, particularly because their iterative denoising process may exhibit varying sensitivity to quantization across steps. The paper addresses a practical challenge that has received relatively limited attention in diffusion language model inference.

  2. The distinction between local quantization error and downstream error propagation provides a useful conceptual framework. The analyses of token uncertainty, block-boundary effects, and entropy evolution offer insights into the behavior of quantized DLMs.

  3. The proposed method does not require additional training and can be applied to different precision levels.

Critical weaknesses
  1. The paper reports approximately 1.66× inference acceleration. However, Appendix C.1 indicates that the acceleration is estimated using BOPs and roofline/Amdahl modeling, with an assumed 2.05× speedup for W4A4 inference. This is a significant limitation for an inference-acceleration paper.

  2. The effect of adaptive token rejection is not adequately separated from precision scheduling. The proposed method rejects uncertain token commitments and performs additional denoising steps to recover delayed tokens.

  3. Entropy velocity is presented as a key signal for predicting the propagation of quantization errors. However, the ablation in Table 4 shows that using the entropy-velocity threshold alone does not improve accuracy over the quantized baseline. Although combining uncertainty and entropy velocity yields better results, the ablation does not report matched execution budgets or equivalent numbers of full-precision and quantized steps.

  4. Theorem 3.1 relates expected KL divergence to token uncertainty under bounded, homogeneous, and decorrelated logit perturbations. While the derivation is informative, actual quantization errors may exhibit substantial correlations and systematic structure.

Questions for authors
  1. Can the authors provide measured end-to-end latency and tokens per second on supported hardware, including the cost of switching precision and performing additional denoising steps? How closely do these measurements match the estimated 1.66× speedup?

  2. How much of the performance improvement comes from dynamic precision selection, and how much comes from rejecting uncertain commitments and increasing the number of refinement steps? Can these effects be evaluated independently under a matched computation budget?

  3. Does entropy velocity consistently improve the accuracy-latency Pareto frontier over uncertainty-only scheduling when both approaches use the same FP16 budget? How sensitive is its predictive power to the history-window size and block boundaries?

  4. How well do the bounded and decorrelated perturbation assumptions describe the actual logit errors produced by the evaluated quantization methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.