acceptodds
Under review as a conference paper at ICLR 2027

ModTrain: Training Modulated Diffusion Models for Extreme Low-Bit Quantization

Abstract

Diffusion models have redefined generative AI, but their multi-step inference remains a critical bottleneck for efficient deployment. Quantization offers a powerful remedy, yet extremely low-bit activation quantization remains challenging even for state-of-the-art quantization-aware training (QAT), as its errors are difficult to absorb during training. Modulated diffusion models offer a promising path by exploiting temporal redundancy across denoising steps, but have so far been limited to post-training quantization. We present ModTrain, a training framework for modulated diffusion models. Training these models is non-trivial because historical states couple consecutive denoising steps, complicating data construction, gradient computation, and activation alignment. ModTrain addresses these issues with sequential forward sampling for efficient data construction, history-aware backpropagation for correct gradient flow, and full-precision alignment to mitigate activation distribution shift. ModTrain is orthogonal to existing QAT methods and integrates seamlessly with them. Experiments on LSUN and ImageNet show that ModTrain outperforms post-training MoDiff, reducing activation precision from 4 to 2 bits, and consistently improves QAT baselines at extremely low precision, with up to a 10.90 FID improvement. ModTrain advances extremely low-bit inference and may further motivate hardware support for such computation.

Primary area:
generative models
Keywords:
Quantized-Aware Training, Diffusion Models, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x6ceOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ModTrain, a training framework for modulated diffusion models under extremely low-bit quantization. While prior modulated quantization methods exploit temporal redundancy between neighboring denoising steps, they have mainly been studied in a post-training setting. Extending them to quantization-aware training is non-trivial because the modulated computation explicitly depends on historical activations and outputs.

Critical strengths
  1. The gains under extremely low precision are substantial.
  2. The framework shows useful compatibility with different quantization settings.
  3. This paper is clear and well written.
Critical weaknesses
  1. The central approximation made by SFS is not sufficiently validated. SFS constructs x_t and x_t' according to the forward diffusion process, whereas the historical states encountered at inference are produced through a recursive quantized reverse trajectory.
  2. The proposed history-aware backpropagation lacks the most important empirical ablation.
  3. Efficiency evaluation is incomplete for a paper motivated by practical low-bit deployment. In particular, ModTrain apparently requires two coupled forward computations during training, and the historical branch must retain gradients. Therefore, compared with conventional QAT, training-time computation and memory could be considerably higher.
  4. The evidence for architectural generality is weaker than the paper's framing suggests. The strongest extreme-low-bit experiments are conducted with LDMs on 256×256 LSUN/ImageNet. However, the model is relatively outdated. The paper does not validate the effectiveness of the proposed method on more advanced DiT-based models, such as FLUX.
Questions for authors
  1. How close are SFS-generated historical states to the historical states obtained from an actual quantized reverse trajectory?
4 comments

We appreciate the reviewer's recognition of the writing, generality, and the effectiveness of our work. Below, we address the reviewer's concerns through further clarifications and additional experimental results.

Weakness 1 & Question 1. The central approximation made by SFS is not sufficiently validated. SFS constructs xtx_t and xt′x_{t'} according to the forward diffusion process, whereas the historical states encountered at inference are produced through a recursive quantized reverse trajectory. How close are SFS-generated historical states to the historical states obtained from an actual quantized reverse trajectory?

Answer: The challenges of the reviewer points out showing the value of the introduction of FPA, which is designed to bridge the training–inference mismatch in historical caches arising from quantization errors. To further address the reviewer's concern, we measure the average relative ℓ2\ell_2 distance between historical and current activations across all layers and timesteps on LSUN-Churches with 200 denoising steps. These results demonstrate that SFS produces activation discrepancies comparable in magnitude to those encountered during inference. We also add the results to Section 4.3 of the revised paper as an ablation study.

Historical state obtainedRelative ℓ2\ell_2 distance
SFS Training10.17%
Quantized inference (Δt=5\Delta t=5)9.28%

Weakness 2. The proposed history-aware backpropagation lacks the most important empirical ablation.

Answer: We emphasize that history-aware backpropagation is theoretically motivated rather than merely an empirical design choice. Since the modulated output depends on both current and historical activations, ignoring gradients through historical states leads to an incomplete gradient of the training objective.

Following the reviewer's suggestion, we conduct an additional ablation on LSUN-Churches at W2A2, removing history-aware backpropagation while keeping the forward modulation mechanism unchanged. Removing history-aware backpropagation dramatically degrades FID from 7.85 to 206.71, demonstrating the critical importance of correctly accounting for historical gradients when training modulated diffusion models, particularly in the ultra-low-bit regime. This result strongly supports our theoretical motivation. We include this additional ablation in Appendix A of the revised paper.

ConfigurationPrecisionFID ↓
ModTrain w/o history-aware BPW2A2206.71
ModTrainW2A27.85

Weakness 3. Efficiency evaluation is incomplete for a paper motivated by practical low-bit deployment. In particular, ModTrain apparently requires two coupled forward computations during training, and the historical branch must retain gradients. Therefore, compared with conventional QAT, training-time computation and memory could be considerably higher.

Answer: Although our primary objective is to improve inference efficiency, ModTrain also facilitates training. We acknowledge that constructing historical states and retaining their gradients introduce additional computational and memory overhead. However, modulation reduces the quantization errors that QAT needs to overcome, while FPA stabilizes model optimization. Consequently, ModTrain can be effectively trained with a smaller batch size while achieving better generation quality.

For example, on LSUN-Churches, standard LSQ requires 16.1 hours on an NVIDIA RTX A6000 for 200K iterations with a batch size of 16, whereas ModTrain requires only 10.2 hours with a batch size of 8 under the same number of iterations. These results demonstrate that the additional computational overhead does not translate into increased overall training cost, highlighting the training and data efficiency of ModTrain. We include the efficiency studies in Appendix G of our revised paper.

Weakness 4. The evidence for architectural generality is weaker than the paper's framing suggests. The strongest extreme-low-bit experiments are conducted with LDMs on 256×256 LSUN/ImageNet. However, the model is relatively outdated. The paper does not validate the effectiveness of the proposed method on more advanced DiT-based models, such as FLUX.

Answer: Our primary objective is to establish a general framework for training modulated diffusion models, rather than to optimize quantization for every recent architecture. To demonstrate its applicability beyond convolutional diffusion models, we evaluate ModTrain on DiT-XL/2 in our main paper.

To further address the reviewer's concern, we conduct additional 50,000-image evaluations and few-step sampling results using DPM-Solver++ to demonstrate that ModTrain substantially mitigates quantization-induced degradation in diffusion transformers, approaching full-precision performance under the standard 50,000-image evaluation and remaining effective with only 25 sampling steps. Given the substantial computational resources required for quantization-aware training of larger and more recent architectures, such as FLUX, we leave this extension to future work.

MethodPrecisionFID (50K images) ↓IS (50K images) ↑FID (25 steps) ↓
FPW32A322.27278.242.81
LSQW4A48.25160.779.42
ModTrainW4A42.98250.015.09

Thank you for the detailed explanations and experiments. However, I could not find the relevant efficiency studies in Appendix G of the revised paper. Although the proposed method improves performance, it may require substantial training resources, which weakens its advantages. Furthermore, the experiments are primarily conducted on outdated diffusion models. Although the authors have added results on DiT-XL/2 in the rebuttal, these results are still insufficient to demonstrate the generalizability and effectiveness of the method on more practically relevant models, such as FLUX and PixArt. Based on these concerns, I maintain my original score.

We sincerely thank the reviewer for the valuable feedback. We appreciate the opportunity to clarify the following concerns.

Concern 1. Missing training cost analysis in Appendix G.

Response: We apologize for inadvertently omitting the training time table from Appendix G in our revised manuscript. As the provided examples of training costs in our previous response to Weakness 3 show, our training costs are less than the standard QAT baselines. To address this concern, we have uploaded an updated version of the paper that includes the training-time table in Appendix G.

Concern 2. Lack of experiments on recent models such as FLUX.

Response: We agree that evaluating our method on more recent models, such as FLUX, would further strengthen its practical relevance and demonstrate its potential benefits. However, our primary objective is not to evaluate our method across every state-of-the-art model, but rather to demonstrate its effectiveness under limited computational resources. We believe that our experiments on both U-Net- and DiT-based architectures already provide strong evidence of the generalizability of our approach across different model architectures.

Regarding implementation, the substantial computational requirements of these more advanced models make it challenging to conduct additional experiments within the rebuttal period, given our limited computational resources. For example, running 10,000 training iterations on a single NVIDIA A6000 GPU would require at least 75 GPU hours. Consequently, we are unable to provide reliable experimental results on these models within the current rebuttal period. Nevertheless, we recognize the value of such evaluations and will consider extending our experiments to more advanced models in future work.

Reviewer_sh1uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ModTrain extends modulated diffusion from post-training quantization to quantization-aware training. It combines correlated noisy training pairs, gradients through historical states, and an auxiliary loss for the full-precision-activation branch. Results on LSUN and ImageNet show clear gains at low precision, especially with 2-bit activations. A W4A4 experiment on DiT also suggests that the approach can work beyond U-Net backbones.

Critical strengths
  1. The paper tackles a concrete problem: standard QAT does not directly account for the historical states used in modulated inference.
  2. The low-bit results are convincing. On LSUN-Churches, W2A2 FID improves from 14.09 to 7.85, and W1A2 from 19.93 to 9.03.
  3. Testing both LSQ and BinaryDM is useful. The weight-only training followed by MoDiff comparison also helps show that weight training alone does not explain the gains.
Critical weaknesses
  1. I had trouble telling which component drives the improvement. The ablations vary the timestep gap and alignment weight, but do not isolate the sampling strategy or history gradients. Since removing FPA performs worse than LSQ, a plain QAT+FPA baseline seems especially important.
  2. The explanation for closing the training–inference gap is plausible, but not directly tested. Training histories come from full-precision activations; inference histories contain quantization effects. Proposition 1 shows correlated samples with correct marginals, which does not establish that their joint distribution matches inference.
  3. Proposition 2 needs correction. Equation (39) is missing a square on the error multiplier. A positive learning rate is also insufficient for descent: with (v=Q(x_2-x_1)+x_1), the condition is (0\leq\eta|v|_2^2\leq2). The single-layer result should also be interpreted more cautiously.
  4. The broader claims run ahead of the experiments. DiT is tested only at W4A4, and most experiments use 200–250 sampling steps. Performance at 2 bits on DiT and with shorter schedules remains unclear.
  5. Some baseline choices need clarification. The dynamic-quantization baseline changes MoDiff’s channel-wise quantization to tensor-wise quantization without showing the quality impact. ImageNet also lacks a direct LSQ QAT comparison.
  6. The deployment benefit remains uncertain. Kernel timings cover 4- and 8-bit settings rather than the main 2-bit setting, and modulation adds memory traffic. End-to-end latency, peak memory, and training overhead would make the practical tradeoff clearer.
Questions for authors
  1. Could you add QAT+FPA, alternative pair sampling, and detached-history ablations under comparable training budgets?
  2. Can you measure how closely training histories match actual inference caches, and whether SFS and FPA reduce that gap?
  3. Could you correct Proposition 2 and state the assumptions needed for its guarantee?
  4. Does the method still work well with W2A2 DiT and substantially fewer sampling steps?
  5. How much does tensor-wise quantization change the MoDiff baseline? Could you also add the direct LSQ baseline on ImageNet?
  6. What are the end-to-end latency and memory costs at W2A2, and the extra training cost over standard QAT?
5 comments

We appreciate the reviewer's recognition of the research topic, writing, and effectiveness of our work. Below, we address the reviewer's concerns through further clarifications and additional experimental results.

Weakness 1 & Question 1. I had trouble telling which component drives the improvement. The ablations vary the timestep gap and alignment weight, but do not isolate the sampling strategy or history gradients. Since removing FPA performs worse than LSQ, a plain QAT+FPA baseline seems especially important. Could you add QAT+FPA, alternative pair sampling, and detached-history ablations under comparable training budgets?

Answer: ModTrain is a unified framework whose components address complementary challenges in training modulated quantization. Removing any of these components can therefore substantially degrade performance. Following the reviewer's suggestion, we conduct additional ablations on LSUN-Churches to isolate their contributions.

To isolate the contribution of the sampling strategy, we replace sequential forward sampling with a pair of two independent forward samples (ModTrain w/o SFS). This degrades FID from 7.85 to 41.02, demonstrating the importance of preserving temporal correlations when constructing historical states. Similarly, ignoring historical gradients leads to an incomplete gradient of the training objective. Removing history-aware backpropagation dramatically degrades FID from 7.85 to 206.71, confirming its critical importance.

ConfigurationPrecisionFID ↓
ModTrain w/o SFSW2A241.02
ModTrain w/o history-aware BPW2A2206.71
ModTrainW2A27.85

Regarding FPA, applying it alone to standard LSQ substantially degrades FID from 5.31 to 39.58, whereas ModTrain achieves 4.69. Together with our existing ablation showing performance degradation when removing FPA from ModTrain, these results indicate that the improvements cannot be attributed to FPA alone. Instead, FPA plays a complementary role in facilitating the optimization of modulated quantization.

ConfigurationPrecisionFID ↓
Standard LSQW4A45.31
LSQ + FPA (λ=0.3\lambda=0.3)W4A439.58
ModTrainW4A44.69

Overall, these ablations demonstrate the importance of each component and the effectiveness of their combined design for training modulated diffusion models. We include these additional results in Section 4.3 and Appendix A of the revised paper.

Weakness 2 & Question 2. The explanation for closing the training–inference gap is plausible, but not directly tested. Training histories come from full-precision activations; inference histories contain quantization effects. Proposition 1 shows correlated samples with correct marginals, which does not establish that their joint distribution matches inference. Can you measure how closely training histories match actual inference caches, and whether SFS and FPA reduce that gap?

Answer: This discrepancy shows the advantages of our combined design: while SFS efficiently constructs correlated historical states, FPA is specifically introduced to mitigate the remaining training–inference gap by aligning the modulated model with its full-precision counterpart. However, even in standard diffusion training, noisy states are constructed through the stochastic forward process rather than by unrolling the learned reverse trajectory. Consequently, there is no canonical one-to-one correspondence between training and inference states. Rather than requiring exact matching, our design aims to approximate the activation discrepancies encountered during inference while mitigating quantization-induced trajectory deviations.

To further address the reviewer's concern, we empirically validate two complementary properties of our design: (1) SFS approximates the changes between historical and current activations. We measure the average relative ℓ2\ell_2 distance between historical and current activations across all layers and timesteps on LSUN-Churches with 200 denoising steps. (2) FPA mitigates quantization-induced trajectory deviations compared to full-precision inference. The results provide complementary empirical evidence for our design: SFS captures comparable activation changes during training and inference, while FPA substantially reduces deviations of the quantized reverse trajectory from its full-precision counterpart. We include these additional analyses in Section 4.3 of the revised paper.

AnalysisConfigurationRelative ℓ2\ell_2 distance/error
Activation changesSFS training10.17%
Activation changesQuantized inference9.28%
Trajectory errorModTrain w/o FPA78%
Trajectory errorModTrain w/ FPA14%

Weakness 3 & Question 3. Proposition 2 needs correction. Equation (39) is missing a square on the error multiplier. A positive learning rate is also insufficient for descent: with (v=Q(x2−x1)+x1)(v=Q(x_2-x_1)+x_1), the condition is (0≤η∣v∣22≤2)(0\leq\eta|v|_2^2\leq2). The single-layer result should also be interpreted more cautiously. Could you correct Proposition 2 and state the assumptions needed for its guarantee?

Answer: We thank the reviewer for pointing out this typo. We correct it by adding the missing square and specifying the condition in Appendix A of the revised paper.

Weakness 4 & Question 4. The broader claims run ahead of the experiments. DiT is tested only at W4A4, and most experiments use 200–250 sampling steps. Performance at 2 bits on DiT and with shorter schedules remains unclear. Does the method still work well with W2A2 DiT and substantially fewer sampling steps?

Answer: We clarify that our DiT experiments aim to demonstrate the applicability of ModTrain beyond convolutional diffusion architectures. W4A4 is already a challenging regime for DiT-XL/2, as evidenced by our additional 50,000-image evaluation: standard LSQ substantially degrades FID from 2.27 to 8.25, whereas ModTrain improves it to 2.98, recovering 88.1% of the FID degradation. Due to the limited rebuttal period, we have not completed W2A2 experiments on DiT and acknowledge this as an important direction for future work.

To address the concern regarding shorter sampling schedules, we conduct additional experiments using DPM-Solver++ with only 25 sampling steps on DiT-XL/2. Despite the larger temporal intervals between adjacent sampling steps, ModTrain consistently mitigates quantization-induced degradation, reducing FID from 9.42 to 5.09 under the 25-step schedule. We incorporate these results into Section 4.2 of the revised paper and clarify the scope of our claims.

MethodPrecisionFID (original schedule, 50K images) ↓FID (25 steps) ↓
FPW32A322.272.81
LSQW4A48.259.42
LSQ + ModTrainW4A42.985.09

Weakness 5 & Question 5. Some baseline choices need clarification. The dynamic-quantization baseline changes MoDiff’s channel-wise quantization to tensor-wise quantization without showing the quality impact. ImageNet also lacks a direct LSQ QAT comparison. How much does tensor-wise quantization change the MoDiff baseline? Could you also add the direct LSQ baseline on ImageNet?

Answer: To ensure a fair comparison between MoDiff and ModTrain, we adopt tensor-wise activation quantization for both methods, which is also more hardware-friendly. To quantify the impact of this choice, we additionally compare tensor-wise and channel-wise MoDiff under identical weight quantization configurations in the PTQ setting. On LSUN-Churches at W2A2, tensor-wise quantization yields an FID of 62.71, whereas channel-wise quantization achieves 38.65. Notably, ModTrain still outperforms MoDiff even when the latter benefits from finer-grained channel-wise quantization, further highlighting the effectiveness of training modulated diffusion models.

Following the reviewer's suggestion, we additionally evaluate standard LSQ on ImageNet at W2A2. LSQ achieves an FID of 26.72, whereas ModTrain achieves 13.23. This further demonstrates the effectiveness of ModTrain over standard QAT in the ultra-low-bit regime. We include these additional comparisons in Section 4.2 of the revised paper.

MethodDatasetPrecisionActivation granularityFID ↓
MoDiff (PTQ)LSUN-ChurchesW2A2Tensor-wise62.71
MoDiff (PTQ)LSUN-ChurchesW2A2Channel-wise38.65
ModTrainLSUN-ChurchesW2A2Tensor-wise7.85
MethodDatasetPrecisionFID ↓
Standard LSQImageNetW2A226.72
ModTrainImageNetW2A213.23

Weakness 6 & Question 6. The deployment benefit remains uncertain. Kernel timings cover 4- and 8-bit settings rather than the main 2-bit setting, and modulation adds memory traffic. End-to-end latency, peak memory, and training overhead would make the practical tradeoff clearer. What are the end-to-end latency and memory costs at W2A2, and the extra training cost over standard QAT?

Answer: We clarify that native W2A2 arithmetic is not supported by the GPU Tensor Cores used in our experiments. Therefore, our W2A2 results demonstrate the algorithmic potential of ultra-low-bit diffusion inference, while our W4A4 evaluations demonstrate practical acceleration on existing hardware. The effectiveness of W2A2 quantization also provides motivation for future hardware supporting lower-bit arithmetic.

Regarding memory overhead, our method explicitly exploits a memory–computation trade-off. Despite increased HBM traffic, our modulated kernels introduce only 3.0–6.9% latency overhead over standard quantized kernels (Table 7). To further characterize this trade-off, we conduct end-to-end evaluations on an NVIDIA A40 using LDM-8 on LSUN-Churches with 200 sampling steps. ModTrain introduces only 2.3% additional latency over standard LSQ while substantially improving generation quality. Despite the additional cache memory, it retains a 1.76× end-to-end speedup and lower peak memory than FP16.

MethodPrecisionLatency (ms/img) ↓Peak memory (GB) ↓FID ↓
FPFP16311.81.124.36
LSQW4A4173.20.415.31
ModTrainW4A4177.21.084.69

Finally, although our primary focus is inference efficiency rather than training cost, ModTrain does not increase training difficulty. Instead, modulation reduces the quantization errors that QAT needs to overcome, while FPA stabilizes model optimization. Consequently, ModTrain can be effectively trained with half the batch size of standard QAT under the same number of training iterations. For example, on LSUN-Churches, standard LSQ requires 16.1 hours on an NVIDIA RTX A6000 for 200K iterations with a batch size of 16, whereas ModTrain requires only 10.2 hours with a batch size of 8. We include the efficiency studies in Appendix G of our revised paper.

Reviewer_gFziOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ModTrain, a QAT framework designed for modulated diffusion models (MoDiff), which quantize temporal activation differences across consecutive denoising steps to achieve extreme low-bit quantization (down to W2A2 and W1A2). Because modulated diffusion relies on historical activation and output caches from previous steps, standard step-independent QAT cannot be directly applied without unrolling the entire generation trajectory.

To address this, the authors introduce three components:

  • SFS samples pairs of adjacent noisy states (xt,xt′)(x_t, x_{t'}) in the forward diffusion process to simulate historical caches without unrolling the reverse process;
  • a history-aware backpropagation scheme that retains the historical caches in the computation graph to prevent parameter-cache misalignment during weight updates; and
  • an auxiliary loss on the historical state xt′x_{t'} with full-precision activations to mitigate distribution shift between training and recursive inference. Experiments on LSUN-Churches, LSUN-Bedrooms, and ImageNet using LDM and DiT backbones demonstrate improvements over PTQ-based MoDiff and standard QAT baselines (LSQ and BinaryDM) at 2-bit and 3-bit activation precisions.
Critical strengths
  • Investigating quantization-aware training for temporal-difference (modulated) diffusion models is a well-motivated direction to overcome the severe activation quantization bottleneck at 2-3 bits.
  • Sequential Forward Sampling provides a computationally tractable mechanism to construct paired historical states during training, avoiding the prohibitive memory and compute cost of unrolling multi-step denoising trajectories.
  • The empirical gains at extreme low bitwidths (specifically W2A2 and W1A2) on LDM architectures are substantial compared to both post-training MoDiff and standard history-agnostic QAT baselines.
  • The method is orthogonal to specific quantizer parameterizations and integrates cleanly with existing QAT methods like LSQ and BinaryDM.
Critical weaknesses
  • The core premise of modulated diffusion, i.e., caching intermediate activations a^t+1(l)\hat{a}^{(l)}_{t+1} and outputs o^t+1(l)\hat{o}_{t+1}^{(l)} for every linear and convolutional layer across timesteps, introduces a massive memory footprint and HBM bandwidth overhead. As shown in Table 8, HBM traffic increases by 1.8x to 3.1x compared to standard quantized kernels, which directly contradicts the primary goal of low-bit quantization on memory-constrained edge devices. Furthermore, the paper lacks end-to-end wall-clock generation latency and peak GPU memory comparisons against standard QAT.
  • It seems that there is a fundamental train-test mismatch in how historical states are constructed: during training, SFS generates historical states at t′t' using a single-step full-precision activation pass without error compensation (explicitly dropping Eq. 1), whereas during inference, historical caches are accumulated recursively over 200 quantized steps via error compensation. While the authors use a large Δt\Delta t (e.g., 200–300) and FPA to heuristically bridge this gap, a Δt\Delta t of 200–300 out of T=1000T=1000 is vastly larger than the actual adjacent step interval (Δt=5\Delta t = 5 for 200 DDIM steps), making the theoretical justification of "neighboring step similarity" inconsistent with the actual hyperparameter choices.
  • Evaluation on modern architectures and fast samplers is weak. The DiT-XL/2 experiment is only evaluated at W4A4 (where standard methods already perform reasonably well) rather than the paper's target W2A2 regime, and uses only 5,000 images. Additionally, all experiments rely on 200-250 sampling steps; it remains unclear if modulated quantization holds up under practical 10-25 step regimes with modern solvers where activation differences between steps are much larger.
Questions for authors

Please refer to the weaknesses. I am willing to raise my score if my concerns are adequately addressed during the rebuttal.

4 comments

We appreciate the reviewer's recognition of the research topic, novelty, and effectiveness of our work. Below, we address the reviewer's concerns through further clarifications and additional experimental results.

Weakness 1. The core premise of modulated diffusion, i.e., caching intermediate activations a^t+1(l)\hat{a}_{t+1}^{(l)} and o^t+1(l)\hat{o}_{t+1}^{(l)} for every linear and convolutional layer across timesteps, introduces a massive memory footprint and HBM bandwidth overhead. As shown in Table 8, HBM traffic increases by 1.8x to 3.1x compared to standard quantized kernels, which directly contradicts the primary goal of low-bit quantization on memory-constrained edge devices. Furthermore, the paper lacks end-to-end wall-clock generation latency and peak GPU memory comparisons against standard QAT.

Answer. We would like to clarify that our primary objective is to accelerate diffusion inference through low-bit computation rather than to minimize memory consumption. This question shows the value of our research: since diffusion models are compute-bound, we exploit a memory–computation trade-off, conceptually similar to KV caching in LLMs, to improve inference efficiency. Despite the increased HBM traffic (Table 8), our method retains speedups of 2.17 compared to 2.23 for the standard quantization kernels in W4A4 settings. This demonstrates that our kernel design effectively mitigates the latency impact of additional memory accesses in compute-intensive diffusion workloads.

To further address the reviewer's concern, we evaluate end-to-end generation latency and peak GPU memory on an NVIDIA A40 using LDM-8 on LSUN-Churches with 200 denoising steps. It shows that the acceleration of our method also applies to end-to-end generation. We add the kernel studies to Appendix G in our revised version.

MethodPrecisionLatency (ms/img) ↓Peak memory (GB) ↓FID ↓
Full precisionFP16311.81.124.36
Standard QATW4A4173.2 (1.81×)0.415.31
OursW4A4177.2 (1.76×)1.084.69

Weakness 2. It seems that there is a fundamental train-test mismatch in how historical states are constructed: during training, SFS generates historical states at t′t' using a single-step full-precision activation pass without error compensation (explicitly dropping Eq. 1), whereas during inference, historical caches are accumulated recursively over 200 quantized steps via error compensation. While the authors use a large Δt\Delta t (e.g., 200–300) and FPA to heuristically bridge this gap, a Δt\Delta t of 200–300 out of T=1000T=1000 is vastly larger than the actual adjacent step interval ($\Delta t =5 $ for 200 DDIM steps), making the theoretical justification of "neighboring step similarity" inconsistent with the actual hyperparameter choices.

Answer: As the reviewer points out, our choice of Δt\Delta t and the introduction of FPA are designed to bridge the training–inference mismatch in historical caches arising from quantization errors. Moreover, we emphasize that even with a larger Δt\Delta t, SFS provides more controllable variations than independently sampling two states from the forward process. This is because independent samples may have substantially different or even opposite noise directions, whereas SFS preserves temporal correlations by constraining their relative noise directions.

To further address the reviewer's concern, we conduct an ablation using independent forward samples (ModTrain w/o SFS) while maintaining the same Δt\Delta t. Furthermore, we measure the average relative ℓ2\ell_2 distance between historical and current activations across all layers and timesteps on LSUN-Churches with 200 denoising steps. These results demonstrate the importance of sequential sampling and show that, despite the larger training interval, SFS produces activation discrepancies comparable in magnitude to those encountered during inference. We add the results to the revised paper in Section 4.3.

ConfigurationFID (W2A2) ↓
ModTrain w/o SFS41.02
ModTrain7.85
Historical state obtainedRelative ℓ2\ell_2 distance
Training (Δt=200\Delta t=200)10.17%
Quantized inference (Δt=5\Delta t=5)9.28%

Weakness 3. Evaluation on modern architectures and fast samplers is weak. The DiT-XL/2 experiment is only evaluated at W4A4 (where standard methods already perform reasonably well) rather than the paper's target W2A2 regime, and uses only 5,000 images. Additionally, all experiments rely on 200-250 sampling steps; it remains unclear if modulated quantization holds up under practical 10-25 step regimes with modern solvers where activation differences between steps are much larger.

Answer: We clarify that W4A4 is already a challenging quantization regime for DiT-XL/2. As shown in our original 5,000-image evaluation, standard LSQ substantially degrades generation quality, whereas ModTrain recovers most of the lost performance. We further strengthen this evidence with an additional 50,000-image evaluation. These results demonstrate that W4A4 quantization for DiT is nontrivial and that ModTrain consistently improves over standard QAT.

MethodPrecisionFID (5K) ↓IS (5K) ↑FID (50K) ↓IS (50K) ↑
FPW32A328.81278.582.27278.24
LSQW4A413.19160.778.25160.77
ModTrainW4A49.31249.962.98250.01

To further address the reviewer's concern regarding shorter sampling schedules, we evaluate DiT-XL/2 using DPM-Solver++ with only 25 sampling steps. Despite the larger temporal intervals between adjacent sampling steps, ModTrain substantially improves FID from 9.42 to 5.09, demonstrating its effectiveness under practical few-step sampling regimes. We include these additional results in Section 4.2 of the revised paper.

MethodPrecisionFID (25 steps) ↓
FPW32A322.81
LSQW4A49.42
ModTrainW4A45.09

Due to the limited rebuttal period, we have not completed additional W2A2 experiments on DiT-XL/2 and leave this extension for future work. Nevertheless, our results demonstrate the effectiveness of ModTrain on diffusion transformers under both the standard 50,000-image evaluation and fast sampling schedules.

Thank you for the rebuttal. I really appreciate the authors' efforts and think this paper is interesting. While the rebuttal has addressed some of my concerns, my primary remaining concern is about the experiments (i.e., Weakness 3). Specifically, (1) the methods are largely evaluated on older models (e.g., LDM) and small-scale datasets (e.g., LSUN); and (2) DiT is evaluated only under the W4A4 setting and compared with a relatively dated baseline, LSQ (ICLR'2020), which was originally proposed for CNN QAT. I understand that dedicated QAT methods for image generative models are scarce, but some advanced frameworks or methods from LLMs (e.g., ParetoQ) could be directly adapted as stronger baselines. Additionally, as noted by other reviewers, strengthening the evaluation would make the paper much more convincing.

I will maintain my initial rating at this stage.

Thank you for your thoughtful feedback and for acknowledging our efforts in the rebuttal. We understand your remaining concerns regarding the experimental evaluation, particularly the comparison with more recent and advanced QAT methods.

We agree that incorporating stronger QAT baselines, such as ParetoQ, would further strengthen our empirical evaluation. However, our primary objective is to develop a computationally efficient QAT approach for image generative models, rather than to comprehensively benchmark against all existing QAT methods. While recent QAT methods developed for LLMs provide valuable insights, their direct applicability and effectiveness for diffusion-based image generation are not necessarily established, given the differences in model architectures and generation processes.

We believe that our current experiments across both U-Net- and DiT-based architectures provide meaningful evidence of the effectiveness and generalizability of our method, particularly under limited computational resources.

We appreciate your suggestion and acknowledge that extending the comparisons to more advanced QAT baselines would be a valuable direction for strengthening this work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their time and constructive feedback. We are encouraged that the reviewers recognize several strengths of our work. Reviewers acknowledge the importance of extending modulated diffusion to quantization-aware training, addressing the fundamental challenge that conventional QAT does not account for the historical states required by modulated inference (gFzi, sh1u). They also recognize the novelty and computational efficiency of our Sequential Forward Sampling strategy, which enables the construction of correlated historical states without the prohibitive cost of unrolling the denoising trajectory (gFzi). Furthermore, all reviewers acknowledge the substantial improvements achieved by ModTrain under extremely low-bit quantization, particularly at W2A2 and W1A2, over PTQ-based MoDiff and conventional QAT baselines (gFzi, sh1u, x6ce). The reviewers also appreciate the compatibility of our framework with different quantization methods and settings, including LSQ and BinaryDM (gFzi, sh1u, x6ce), as well as our empirical evidence demonstrating that the improvements cannot be attributed solely to weight optimization (sh1u). Finally, the reviewers appreciate the clarity of our presentation (sh1u, x6ce).

We have carefully addressed the reviewers' comments in our point-by-point responses, including extensive additional experiments and analyses that further support our claims. In particular, we provide new component-wise ablations demonstrating the importance of Sequential Forward Sampling, history-aware backpropagation, and the complementary role of FPA, as well as quantitative analyses validating the ability of our framework to approximate activation discrepancies encountered during inference and mitigate quantization-induced trajectory deviations. We further strengthen our evaluation with additional baseline comparisons, standard 50,000-image evaluations on DiT-XL/2, and experiments with only 25 sampling steps using DPM-Solver++, demonstrating the effectiveness of ModTrain beyond convolutional diffusion architectures and under practical few-step sampling regimes. Moreover, we provide end-to-end latency, peak memory, and training cost evaluations to better characterize the practical efficiency and memory–computation trade-off of our method. We have also corrected and clarified our theoretical analysis and revised the manuscript accordingly, with all changes highlighted in blue in the revised PDF. We hope our responses address the reviewers' concerns, and we would be happy to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.