- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents ModTrain, a training framework for modulated diffusion models under extremely low-bit quantization. While prior modulated quantization methods exploit temporal redundancy between neighboring denoising steps, they have mainly been studied in a post-training setting. Extending them to quantization-aware training is non-trivial because the modulated computation explicitly depends on historical activations and outputs.
Critical strengths
- The gains under extremely low precision are substantial.
- The framework shows useful compatibility with different quantization settings.
- This paper is clear and well written.
Critical weaknesses
- The central approximation made by SFS is not sufficiently validated. SFS constructs x_t and x_t' according to the forward diffusion process, whereas the historical states encountered at inference are produced through a recursive quantized reverse trajectory.
- The proposed history-aware backpropagation lacks the most important empirical ablation.
- Efficiency evaluation is incomplete for a paper motivated by practical low-bit deployment. In particular, ModTrain apparently requires two coupled forward computations during training, and the historical branch must retain gradients. Therefore, compared with conventional QAT, training-time computation and memory could be considerably higher.
- The evidence for architectural generality is weaker than the paper's framing suggests. The strongest extreme-low-bit experiments are conducted with LDMs on 256×256 LSUN/ImageNet. However, the model is relatively outdated. The paper does not validate the effectiveness of the proposed method on more advanced DiT-based models, such as FLUX.
Questions for authors
- How close are SFS-generated historical states to the historical states obtained from an actual quantized reverse trajectory?
4 comments
Weakness 4. The evidence for architectural generality is weaker than the paper's framing suggests. The strongest extreme-low-bit experiments are conducted with LDMs on 256×256 LSUN/ImageNet. However, the model is relatively outdated. The paper does not validate the effectiveness of the proposed method on more advanced DiT-based models, such as FLUX.
Answer: Our primary objective is to establish a general framework for training modulated diffusion models, rather than to optimize quantization for every recent architecture. To demonstrate its applicability beyond convolutional diffusion models, we evaluate ModTrain on DiT-XL/2 in our main paper.
To further address the reviewer's concern, we conduct additional 50,000-image evaluations and few-step sampling results using DPM-Solver++ to demonstrate that ModTrain substantially mitigates quantization-induced degradation in diffusion transformers, approaching full-precision performance under the standard 50,000-image evaluation and remaining effective with only 25 sampling steps. Given the substantial computational resources required for quantization-aware training of larger and more recent architectures, such as FLUX, we leave this extension to future work.
| Method | Precision | FID (50K images) ↓ | IS (50K images) ↑ | FID (25 steps) ↓ |
|---|---|---|---|---|
| FP | W32A32 | 2.27 | 278.24 | 2.81 |
| LSQ | W4A4 | 8.25 | 160.77 | 9.42 |
| ModTrain | W4A4 | 2.98 | 250.01 | 5.09 |
Thank you for the detailed explanations and experiments. However, I could not find the relevant efficiency studies in Appendix G of the revised paper. Although the proposed method improves performance, it may require substantial training resources, which weakens its advantages. Furthermore, the experiments are primarily conducted on outdated diffusion models. Although the authors have added results on DiT-XL/2 in the rebuttal, these results are still insufficient to demonstrate the generalizability and effectiveness of the method on more practically relevant models, such as FLUX and PixArt. Based on these concerns, I maintain my original score.
We sincerely thank the reviewer for the valuable feedback. We appreciate the opportunity to clarify the following concerns.
Concern 1. Missing training cost analysis in Appendix G.
Response: We apologize for inadvertently omitting the training time table from Appendix G in our revised manuscript. As the provided examples of training costs in our previous response to Weakness 3 show, our training costs are less than the standard QAT baselines. To address this concern, we have uploaded an updated version of the paper that includes the training-time table in Appendix G.
Concern 2. Lack of experiments on recent models such as FLUX.
Response: We agree that evaluating our method on more recent models, such as FLUX, would further strengthen its practical relevance and demonstrate its potential benefits. However, our primary objective is not to evaluate our method across every state-of-the-art model, but rather to demonstrate its effectiveness under limited computational resources. We believe that our experiments on both U-Net- and DiT-based architectures already provide strong evidence of the generalizability of our approach across different model architectures.
Regarding implementation, the substantial computational requirements of these more advanced models make it challenging to conduct additional experiments within the rebuttal period, given our limited computational resources. For example, running 10,000 training iterations on a single NVIDIA A6000 GPU would require at least 75 GPU hours. Consequently, we are unable to provide reliable experimental results on these models within the current rebuttal period. Nevertheless, we recognize the value of such evaluations and will consider extending our experiments to more advanced models in future work.
We appreciate the reviewer's recognition of the writing, generality, and the effectiveness of our work. Below, we address the reviewer's concerns through further clarifications and additional experimental results.
Answer: The challenges of the reviewer points out showing the value of the introduction of FPA, which is designed to bridge the training–inference mismatch in historical caches arising from quantization errors. To further address the reviewer's concern, we measure the average relative ℓ2 distance between historical and current activations across all layers and timesteps on LSUN-Churches with 200 denoising steps. These results demonstrate that SFS produces activation discrepancies comparable in magnitude to those encountered during inference. We also add the results to Section 4.3 of the revised paper as an ablation study.
Answer: We emphasize that history-aware backpropagation is theoretically motivated rather than merely an empirical design choice. Since the modulated output depends on both current and historical activations, ignoring gradients through historical states leads to an incomplete gradient of the training objective.
Following the reviewer's suggestion, we conduct an additional ablation on LSUN-Churches at W2A2, removing history-aware backpropagation while keeping the forward modulation mechanism unchanged. Removing history-aware backpropagation dramatically degrades FID from 7.85 to 206.71, demonstrating the critical importance of correctly accounting for historical gradients when training modulated diffusion models, particularly in the ultra-low-bit regime. This result strongly supports our theoretical motivation. We include this additional ablation in Appendix A of the revised paper.
Answer: Although our primary objective is to improve inference efficiency, ModTrain also facilitates training. We acknowledge that constructing historical states and retaining their gradients introduce additional computational and memory overhead. However, modulation reduces the quantization errors that QAT needs to overcome, while FPA stabilizes model optimization. Consequently, ModTrain can be effectively trained with a smaller batch size while achieving better generation quality.
For example, on LSUN-Churches, standard LSQ requires 16.1 hours on an NVIDIA RTX A6000 for 200K iterations with a batch size of 16, whereas ModTrain requires only 10.2 hours with a batch size of 8 under the same number of iterations. These results demonstrate that the additional computational overhead does not translate into increased overall training cost, highlighting the training and data efficiency of ModTrain. We include the efficiency studies in Appendix G of our revised paper.