acceptodds
Under review as a conference paper at ICLR 2027

I-DiT: Integer-Only Quantization for Diffusion Transformers

Abstract

Diffusion transformers achieve remarkable generative performance but remain challenging to deploy on resource-constrained hardware due to their high computational and memory costs. These devices often rely on specialized AI accelerators whose efficiency comes primarily from low-precision integer-only datapaths. However, existing diffusion quantization methods still rely heavily on floating-point operations, and integer-only frameworks are not readily applicable to modern diffusion transformers due to fixed quantization scales, lack of tiled attention support, and sensitivity to activation outliers. For this, we present I-DiT, the first framework for fully integer-only inference of diffusion transformers without costly fine-tuning. First, we propose a dynamic scale adjustment method for integer-only frameworks, that adapts to varying activation ranges during runtime. Second, we introduce log2-based online softmax that enables numerically stable and memory-efficient tiled attention in integer. Lastly, we reformulate dyadic requantization to support finer-grained activation quantization, localizing the impact of outliers while preserving integer-only execution. Experiments across PixArt, SD3, and FLUX demonstrate that I-DiT preserves generation quality while enabling efficient fully integer-only inference for large-scale diffusion models.

Primary area:
generative models
Keywords:
Diffusion Transformers, Integer-Only Quantization, Edge AI, Post-Training Quantization, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SPnGOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces I-DiT, the first fully integer-only inference framework for diffusion transformers. It proposes three methods tailored for integer-only inference: dynafit for dynamic scale adaptation, a log2-domain integer online softmax for memory-efficient tiled attention, and a factored dyadic requantization for fine-grained activation quantization. Experiments across PixArt, SD3, and FLUX demonstrate strong performance, largely preserving FP16 generation quality under W8A8 settings.

Critical strengths

1.This paper presents the first integer-only inference framework for DiTs and achieves remarkable state-of-the-art performance. 2.The proposed method is technically solid and successfully addresses several critical challenges in building a fully integer-only inference framework. I think the framework is important so I would give a relative positive assessment.

Critical weaknesses

None

Questions for authors

1.The evaluation uses a single sampler configuration for each model, but Dynafit's calibrated scales are tied to a specific denoising trajectory. It remains unclear whether these scales generalize across different samplers (e.g., DDIM vs. DPM-Solver), which may lead to different activation dynamics. Could the authors provide results under at least two distinct sampling configurations? 2. Could authors discuss further on the potential implication of integer-only framework and illustrate why integer-only is important

2 comments
AuthorsOpenReview ↗

Response to reviewer SPnG

1. Do the calibrated scales generalize across different samplers?

→\to Thank you for an interesting experiment idea. We added cross-sampler experiments using PixArt-α\alpha and PixArt-Σ\Sigma. Whilst keeping the same calibrated base scales from the original sampler, we only swap the sampler to a different one during inference. All other settings are kept identical. We report FID scores of 5k prompts sampled from MJHQ dataset.

Table E. FID comparison across different samplers.

ModelDPM-Solver++DDIMUniPC
PixArt-α19.1718.9119.59
PixArt-Σ17.9218.5417.98

The results show that the calibrated scales generalize well to different samplers. Since our method dynafit adjusts the quantization range during inference, it provides flexibility to accommodate activation changes induced by different samplers. We also added a new experiment section in the revised manuscript to highlight this experiment. Thank you again for a great suggestion.

[1] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in Proceedings of the International Conference on Learning Representations (ICLR), 2021.

[2] W. Zhao, L. Bai, Y. Rao, J. Zhou, and J. Lu, “UniPC: A unified predictor-corrector framework for fast sampling of diffusion models,” in Advances in Neural Information Processing Systems (NeurIPS), 2023.

2. Potential implication of integer-only framework and its importance.

→\to Its potential implication is to broaden the hardware platforms on which diffusion backbones can run while preserving generation quality. Concrete deployments already demonstrate demand for efficient local generation. For example, Qualcomm deployed INT8-quantized Stable Diffusion 1.5 on a smartphone powered by Snapdragon 8 Gen 2, generating a 512×512 image in under 15 seconds without requiring an Internet connection [1]. Arm demonstrated diffusion transformer generating 128×128 face images on an Alif Ensemble E8 development board, with neural-network execution accelerated by the Ethos-U85 NPU and sampling controlled by a Cortex-M55 CPU [2].

Supporting these applications on a wider range of edge devices requires exploiting the integer computation capabilities of their accelerators, as many achieve their efficiency through dedicated integer arithmetic. For example, Google’s Edge TPU delivers 4 TOPS of fixed-point integer computation at approximately 2 W [3], while Arm’s Ethos-U85 provides up to 2048 INT8 multiply–accumulate operations per cycle, corresponding to approximately 4 TOPS of integer arithmetic at 1 GHz [4]. To execute a complete backbone on such integer accelerators, nonlinear operations and requantization must also use supported integer arithmetic, as remaining floating-point operations must be converted or executed on a separate processor. I-DiT addresses these arithmetic gaps for diffusion transformers, providing a foundation for deployment subject to each platform’s operator support and memory capacity.

[1] Z. Asghar and J. Hou, "World’s first on-device demonstration of Stable Diffusion on an Android phone," Qualcomm OnQ Blog, Feb. 23, 2023. Accessed: Nov. 11, 2026. [Online]. Available: https://www.qualcomm.com/news/onq/2023/02/worlds-first-on-device-demonstration-of-stable-diffusion-on-android

[2] A. Naik, "Generative AI comes to embedded: Powered by Arm Ethos-U NPU," Arm Community, Nov. 3, 2026. Accessed: Nov. 11, 2026. [Online]. Available: https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/generative-ai-comes-to-embedded-powered-by-arm-ethos-u-npu

[3] Google Coral, "USB Accelerator datasheet," datasheet. Accessed: Nov. 11, 2026. [Online]. Available: https://dlcdnwebimgs.asus.com/files/media/6dd0bb60-207c-4ddf-bb34-934447605ce9/coral-usb-accelerator-datasheet.pdf

[4] Arm, "Arm Ethos-U85 product brief," product brief. Accessed: Nov. 11, 2026. [Online]. Available: https://armkeil.blob.core.windows.net/developer/Files/pdf/datasheets/arm-ethos-u85-product-brief.pdf

Interesting work, thanks

Reviewer_vAqDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes I-DiT, a post-training quantization framework for integer-only Diffusion Transformer inference. It combines Dynafit for dynamic scale adjustment using integer bit-length information, a log₂-domain integer online softmax to prevent scale accumulation across attention tiles, and fine-grained dyadic requantization to reduce quantization-parameter storage. Experiments on PixArt-Σ, PixArt-α, SD3-medium, and FLUX.1-schnell report competitive generation quality relative to floating-point models, while FPGA evaluations examine the hardware feasibility of integer-only attention.

Critical strengths
  1. A practically valuable framework for integer-only DiT inference. This paper addresses the practical challenges of deploying Diffusion Transformers on integer-only hardware by proposing a comprehensive quantization framework. By integrating dynamic scale adaptation, integer-only online softmax, and fine-grained requantization, the proposed method enables integer-only inference of DiT backbones. Compared with existing approaches that primarily quantize linear layers while retaining floating-point operations for nonlinear functions, this work offers practical value in terms of system completeness and hardware deployment.

  2. Extensive evaluation of generation quality across different architectures. The paper evaluates the proposed method on several representative diffusion architectures, including PixArt-Σ, PixArt-α, SD3-medium, and FLUX.1-schnell, demonstrating generation quality close to that of their floating-point counterparts across multiple datasets.

Critical weaknesses

It is unclear what the two core components add over existing methods. Dynafit's integer scale adjustment needs a direct comparison with I-LLM's DI-MatMul; the integer online softmax needs a direct comparison with I-FlashAttention and QFlash, which also address tiled integer attention and scale management.

[1] I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models.

[2] I-FlashAttention: Fully Integer Fused Attention for Efficient Vision Transformers.

[3] QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention.

Questions for authors
  1. How does the online softmax differ from I-FlashAttention and QFlash? Please compare scale-management rules, output error, and runtime under matched numerical settings.

  2. What does Dynafit add over I-LLM's DI-MatMul? Please compare accuracy and scale-update overhead, and explain the quality degradation from Dynafit alone and some softmax combinations in Appendix F, Table 8.

  3. How does I-DiT perform against the original quantizers retaining floating-point nonlinearities? Appendix A states that the reported baselines also use I-ViT approximations. Please label these adapted versions and report quality and comparable inference costs for the original methods at matched bit widths and sampling settings.

6 comments
AuthorsOpenReview ↗

Response to reviewer vAqD (1/3)

1. Comparison with I-FlashAttention and QFlash in terms of scale-management rules, output error, and runtime.

→\to Thank you for highlighting these related works. They appear to be an earlier preliminary paper (I-FlashAttention) and a later expanded work (QFlash) from the same authors, and seem to share the same underlying design in their proposed integer online softmax. Thereby we will only compare against QFlash[1], which is the more recent and expanded version.

We have added a dedicated discussion in the revised manuscript (Appendix C), including FID comparison on image quality and latency. We also note, for context, that it was published at IJCAI in August 2026 and therefore falls within ICLR’s contemporaneous-work window, for which experimental comparisons are optional. We provide these additional results to help clarify the relationship between the two methods.

1. Scale-management rules: Both QFlash and our integer online softmax prevent exponential scales from compounding across tiles, but differ in where and how scale alignment occurs. QFlash achieves this through explicit scale conversions. It requantizes each tile’s exponential weights to INT8 before PV multiplication uses its proposed ScaleRelease() operation when applying the running-maximum correction to the previous numerator and denominator. This operation compensates for the scale carried by the integer exponential correction factor, preventing each update from introducing another accumulated scale factor.

Our method instead fixes the exponential representation scale before accumulation. DyPow() directly represents both exponential weights and running-maximum correction factors using the same input-independent power-of-two scale (Eq. 13). Consequently, the extra representation factor introduced during an update is always removed by the same fixed rounded shift. This builds scale alignment into the online recurrence, avoiding a scale-dependent compensation multiplier and its associated approximation at this step. The rule remains unchanged across input score scales, including those arising from token-dependent query quantization.

2. Output error & runtime: We compare denoiser output error and runtime. For QFlash, we used the official implementation from the authors. FID is measured using 5k text prompts from MJHQ, denoiser output error (NMSE) is measured across 64 samples from them, and latency is measured on A6000 GPU.

Table C. Comparison against QFlash using denoiser error and latency.

ModelMethodFIDMean NMSE (%)Worst NMSE (%)Latency (denoiser)
PixArt-ΣQFlash20.310.693.231.16 s
Ours17.920.623.011.79 s
SD3-mediumQFlash265.519.9957.331.45 s
Ours23.380.462.271.85 s

Our method achieves lower FID on both models: 17.92 versus 20.31 on PixArt- and 23.38 versus 265.51 on SD3-medium. In particular, replacing attention with QFlash in SD3-medium backbone produces severely corrupted images (Figure 7, Appendix C). Our method also achieves lower mean and worst-case NMSE on both models. This mainly comes from maintaining higher precision for the numerically-sensitive exponential weights using a common dyadic scale, whereas QFlash requantizes them to INT8 to utilize INT8 tensor cores on GPU. While this allows QFlash to achieve lower latency, it comes at the cost of large degradation in generation quality. We include this study in Appendix C, titled “Comparison with Concurrent Work.”, along with a qualitative comparison of generated images (Figure 7).

[1] S. Oh, Y. Kwon, and J. Lee, “QFlash: Bridging quantization and memory efficiency in vision transformer attention,” in Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence (IJCAI), 2026.

AuthorsOpenReview ↗

Response to reviewer vAqD (2/3)

2. What does Dynafit add over I-LLM's DI-MatMul?

→\to The key distinction lies in how much runtime work is required for dynamic requantization. Following their notation, DI-MatMul maps the current accumulator range to the output integer range through QY=round⁡((P−pmin⁡)(2n−1)/(pmax⁡−pmin⁡))Q_Y=\operatorname{round}((P-p_{\min})(2^n-1)/(p_{\max}-p_{\min})). At each such requantization, the denominator (pmax⁡−pmin⁡p_{\max}-p_{\min}) must be determined from the current quantization group, and every output element must be normalized by that runtime-dependent range. This requires division or an equivalent reciprocal-based implementation; it cannot generally be reduced to changing a bit-shift exponent alone.

Dynafit avoids this runtime range normalization by retaining the calibrated dyadic multiplier and adapting only the effective shift. And its runtime range measurement selects a bounded exponent correction through bit-length comparison rather than becoming a divisor in the elementwise requantization path (equation 10). Thus, adaptation requires only updating the shift exponent, preserving the existing multiply-and-shift computation path. Dynafit leverages the pre-calibrated scales to enable a simpler adaptation path while preserving useful flexibility to expand or contract the representable range.

We also provide a direct comparison to I-LLM’s DI-MatMul. During implementation, we found that the equations in I-LLM paper contain inconsistencies that make it hard to implement their method from scratch. Specifically, equation(6,7) for solving the output scale do not match their initial definition in equation(4). Therefore, we make our best reasonable effort and follow the explicit objective in equation(4) and re-derive equation(6,7) for this experiment.

We evaluate denoiser-output error on identical inputs against FP32 outputs using 64 uniformly sampled MJHQ prompts per model, and report both mean and worst NMSE. Quantization latency is measured on an RTX A6000 GPU. It includes range reduction, scale selection, and requantization, averaged over all quantization calls in each model.

Table A. Comparison between Dynafit (ours) and DI-MatMul (I-LLM)

ModelDynafitDI-MatMul
Mean NMSE (%)Worst NMSE (%)Latency (μs)Mean NMSE (%)Worst NMSE (%)Latency (μs)
PixArt-Σ0.623.0175.2817.07140.2092.33
SD3-medium0.462.2763.782.555.6275.52

Dynafit achieves lower denoiser-output error with lower quantization overhead on both models. On PixArt-Σ\Sigma, its mean and worst-case NMSE are 0.62% and 3.01%, compared with 17.07% and 140.20% for DI-MatMul. On SD3-medium, the corresponding values are 0.46% and 2.27%, compared with 2.55% and 5.62%. These results show lower error both on average and in the worst observed case. Dynafit also has 18.5% and 15.5% lower average quantization latency, respectively. This is consistent with its simpler runtime adaptation. Thus, in the evaluated setting, Dynafit’s shift-based adaptation provides higher accuracy with lower overall quantization overhead.

3. Explain the quality degradation from Dynafit alone and some softmax combinations.

→\to Thank you for highlighting these results. We explain the two cases separately.

DynaFit alone. Without sufficient granularity, an extreme outlier can cause DynaFit to expand the shared quantization range. This reduces clipping but increases the quantization step size, leaving fewer bins to distinguish typical activation values. Quality can therefore decrease when the added rounding error outweighs the reduction in clipping. This trade-off explains why degradation is not universal: DynaFit alone slightly worsens PixArt-Σ\Sigma, but improves SD3-medium, increasing IR from −1.85 to −1.73 and reducing FID from 121.54 to 53.18. The different outcomes likely reflect differences in outlier patterns and severity across the two models.

DynaFit with integer softmax. The same loss of precision can affect attention score differences, such as distinct scores rounding to the same value, changing the relative weights that softmax assigns. Integer exponentiation and normalization introduce additional approximation error on top of these perturbed scores. Thus, although our integer softmax stabilizes tile-wise accumulation, it cannot recover distinctions lost through coarse quantization. Consistent with this interpretation, adding DynaFit improves both models once fine-grained quantization ($\sec$4.3) localizes the influence of outliers.

We believe this is a very insightful discussion that should be included in the paper. In the revised manuscript, we updated Appendix H (previously Appendix F) to discuss these trade-offs in more detail. We thank the reviewer for prompting this discussion.

AuthorsOpenReview ↗

Response to reviewer vAqD (3/3)

4. Comparison against baselines with fp nonlinearities.

→\to Thank you for requesting this clarification. As specified in the manuscript’s experimental settings, only baselines from the diffusion quantization literature (Q-Diffusion, Q-DiT, TDQ, PTQD, and PTQ4DM) are adapted to satisfy the integer-only execution constraint using I-ViT approximations. We agree that the distinction between these adapted versions and the original methods could be made more explicit. In our revised manuscript, we labeled the adapted baselines with a marker (†\dagger) in the comparison table to avoid confusion, and more explicitly explained their adaptation in the experimental setup.

We additionally evaluate the original baselines retaining floating-point operations, alongside a corresponding I-DiT variant that retains floating-point nonlinearities and attention while keeping dynafit and fine-grained dyadic quantization. The table below reports generation quality (FID), denoiser-forward latency, full generation latency, and peak memory under matched evaluation settings. FID is measured using 5k samples from MJHQ dataset, and latency and memory are measured on A6000 GPU, all using PixArt-Σ\Sigma.

Table D. Comparison against FP-retained baselines using PixArt-Σ\Sigma on MJHQ.

MethodFP-retainedINT-Only
FIDLatency (denoiser)Latency (full gen.)Peak memoryFIDLatency (denoiser)Latency (full gen.)Peak memory
Q-Diffusion22.901.22 s25.04 s18.84 GB94.212.97 s65.78 s16.85 GB
Q-DiT16.461.17 s24.21 s16.76 GB51.022.84 s62.42 s18.27 GB
TDQ18.521.09 s22.42 s18.87 GB102.412.94 s64.39 s16.93 GB
PTQD22.951.23 s25.30 s18.84 GB94.442.95 s65.41 s16.85 GB
PTQ4DM19.250.98 s20.53 s18.85 GB57.252.95 s65.49 s16.72 GB
I-DiT (Ours)16.011.03 s21.50 s16.78 GB17.921.79 s39.86 s21.61 GB

I-DiT achieves the lowest FID under both matched execution settings. When floating-point nonlinearities and attention are retained, it achieves an FID of 16.01 versus 16.46–22.95 for the original baselines at comparable inference costs. Although designed for integer-only execution, dynafit and fine-grained quantization also address activation range variation and outlier-induced quantization error in GEMMs, so their benefits remain relevant when nonlinearities and attention retain floating-point precision.

Note that the integer-only evaluations appear slower than their fp-retained counterparts because on GPU, much of the integer arithmetic is simulated using FP32. This underlies our inclusion of the FPGA evaluation in Table 4, which assesses the proposed integer attention through native execution on a pure-integer datapath.

We also included this table in the experiment section of the revised manuscript as additional analysis. Thank you again for this helpful suggestion.

Reviewer_mcqeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents I-DiT, a framework for fully integer-only inference of diffusion transformers without costly fine-tuning. To address these challenges, the authors propose Dynafit for runtime quantization scale adjustment, a log2-domain integer online softmax for memory-efficient tiled attention, and fine-grained dyadic requantization to reduce the impact of activation outliers. Experiments on several diffusion transformers demonstrate substantial improvements in generation quality over the evaluated integer-only baselines, along with promising hardware efficiency results on an FPGA.

Critical strengths
  1. Important and practical problem. The paper addresses the valuable problem of enabling fully integer-only inference for diffusion transformers, which is particularly relevant for deployment on resource-constrained hardware.

  2. Clear motivation and technical presentation. The paper is well written, with clear explanations of the challenges faced by existing integer-only quantization methods and the motivation behind each proposed technique. The methodology is generally easy to follow.

  3. Effective technical solutions with encouraging experimental results. The proposed techniques directly address the identified challenges, including dynamic activation ranges, numerical instability in integer-only online softmax, and fine-grained quantization under dyadic constraints. The experimental results demonstrate substantial improvements in generation quality over the evaluated integer-only baselines, with additional evidence of hardware efficiency.

Critical weaknesses
  1. Unclear higher-precision settings. The paper states that W8A8 is used by default, with higher precision for selected operations, but does not clearly specify these exceptions.

  2. Dynafit buffering and computational overhead. Eq. (9–11) requires a group-wide maximum before determining the final output scale. Therefore, intermediate results must remain available for requantization. If the buffer is too large, do we need to compute the GEMM twice?

  3. Insufficient justification for Dynafit. There are existing works using dynamic integer-only matrix multiplication [1]. Could the authors show the benefit of the proposed method with calibration reference?

  4. Numerical error from fine-grained dyadic requantization. Eq. (15) applies two successive multiply-and-shift operations. Intermediate rounding could introduce additional quantization error compared with a single merged dyadic multiplier. Could the authors demonstrate improved end-to-end accuracy from fine-grained quantization?

[1] Hu, Xing, et al. "I-llm: Efficient integer-only inference for fully-quantized low-bit large language models." arXiv preprint arXiv:2405.17849 (2024).

Questions for authors

Please refer to the weakness.

2 comments
AuthorsOpenReview ↗

Response to reviewer mcqe (1/2)

1. Unclear higher-precision settings.

→\to We apologize for the missing detail. As specified, all GEMM operations use W8A8 as default. We use 16-bit integer activations for elementwise multiplication and addition in AdaLN and for residual additions to limit quantization error in modulation and residual pathways. Using wider integer precision for such intermediate computations is an established practice in integer-only frameworks [1,2]. Thus we follow this mixed-integer-precision design principle, while introducing no floating-point operations. Since this is an important detail, we explicitly listed these precision settings in the experiment section in the revised manuscript. Thank you for highlighting this detail.

[1] Z. Li and Q. Gu, “I-ViT: Integer-only quantization for efficient vision transformer inference,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

[2] S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, “I-BERT: Integer-only BERT quantization,” in Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.

2. Dynafit buffering and computational overhead.

→\to The reviewer is correct that intermediate outputs must remain available until the group-wide maximum is determined. However, this does not require buffering the entire output tensor. With our per-token activation quantization, each scale group is a single output row. GEMM can therefore be partitioned along the token dimension, with the number of concurrently processed rows chosen to fit the available accumulator or output buffer. The required storage is modest on a per-group basis. For PixArt-Σ\Sigma, one INT32 output row occupies 4.5 KiB for a 1152-channel hidden projection, and 18 KiB for the 4608-channel MLP expansion. For comparison, Huawei Atlas A2/A3 provides a 128 KB matrix-result buffer [1], while Arm’s Ethos-U85 configurations with 512–2048 MACs/cycle provide 32–128 KiB of accumulator RAM [2]. If even one complete row (group) cannot be retained and spilling intermediate outputs to external memory is unavailable, then recomputation becomes necessary. However, this dependency is not unique to Dynafit. It arises in any dynamic quantization methods that determine a group’s scale from its runtime values before converting wider intermediate results to a narrower representation [3-5]. Dynafit simplifies the scale-selection arithmetic, but does not eliminate this shared buffering–recomputation tradeoff.

[1] Huawei Technologies Co., Ltd., "Hardware constraints," CANN Ascend C Operator Development Guide. [Online]. Available: https://www.hiascend.com/doc_center/source/en/CANNCommunityEdition/850/opdevg/Ascendcopdevg/atlas_ascendc_10_00048.html.

[2] Arm Ltd., "Ethos-U85 architecture configuration," Ethos-U Vela compiler, source code distributed by NXP. [Online]. Available: https://github.com/nxp-imx/ethos-u-vela/blob/lf-6.18.20_2.0.0/ethosu/regor/architecture/ethosu85/ethos_u85.cpp.

[3] T. Zhao et al., "ViDiT-Q: Efficient and accurate quantization of diffusion transformers for image and video generation," in Proceedings of the International Conference on Learning Representations (ICLR), 2025.

[4] S. Ashkboos et al., "QuaRot: Outlier-free 4-bit inference in rotated LLMs," in Advances in Neural Information Processing Systems (NeurIPS), 2024.

[5] Y. Lin et al., "QServe: W4A8KV4 quantization and system co-design for efficient LLM serving," in Proceedings of Machine Learning and Systems (MLSys), 2025.

AuthorsOpenReview ↗

Response to reviewer mcqe (2/2)

3. Comparison against I-LLM, which uses dynamic integer-only matrix multiplication.

→\to The key distinction lies in how much runtime work is required for dynamic requantization. Following their notation, DI-MatMul maps the current accumulator range to the output integer range through QY=round⁡((P−pmin⁡)(2n−1)/(pmax⁡−pmin⁡))Q_Y=\operatorname{round}((P-p_{\min})(2^n-1)/(p_{\max}-p_{\min})). At each such requantization, the denominator (pmax⁡−pmin⁡p_{\max}-p_{\min}) must be determined from the current quantization group, and every output element must be normalized by that runtime-dependent range. This requires division or an equivalent reciprocal-based implementation; it cannot generally be reduced to changing a bit-shift exponent alone.

Dynafit avoids this runtime range normalization by retaining the calibrated dyadic multiplier and adapting only the effective shift. And its runtime range measurement selects a bounded exponent correction through bit-length comparison rather than becoming a divisor in the elementwise requantization path (equation 10). Thus, adaptation requires only updating the shift exponent, preserving the existing multiply-and-shift computation path. Dynafit leverages the pre-calibrated scales to enable a simpler adaptation path while preserving useful flexibility to expand or contract the representable range.

We also provide a direct comparison to I-LLM’s DI-MatMul. During implementation, we found that the equations in I-LLM paper contain inconsistencies that make it hard to implement their method from scratch. Specifically, equation(6,7) for solving the output scale do not match their initial definition in equation(4). Therefore, we make our best reasonable effort and follow the explicit objective in equation(4) and re-derive equation(6,7) for this experiment.

We evaluate denoiser-output error on identical inputs against FP32 outputs using 64 uniformly sampled MJHQ prompts per model, and report both mean and worst NMSE. Quantization latency is measured on an RTX A6000 GPU. It includes range reduction, scale selection, and requantization, averaged over all quantization calls in each model.

Table A. Comparison between Dynafit (ours) and DI-MatMul (I-LLM)

ModelDynafitDI-MatMul
Mean NMSE (%)Worst NMSE (%)Latency (μs)Mean NMSE (%)Worst NMSE (%)Latency (μs)
PixArt-Σ0.623.0175.2817.07140.2092.33
SD3-medium0.462.2763.782.555.6275.52

Dynafit achieves lower denoiser-output error with lower quantization overhead on both models. On PixArt-Σ\Sigma, its mean and worst-case NMSE are 0.62% and 3.01%, compared with 17.07% and 140.20% for DI-MatMul. On SD3-medium, the corresponding values are 0.46% and 2.27%, compared with 2.55% and 5.62%. These results show lower error both on average and in the worst observed case. Dynafit also has 18.5% and 15.5% lower average quantization latency, respectively. This is consistent with its simpler runtime adaptation. Thus, in the evaluated setting, Dynafit’s shift-based adaptation provides higher accuracy with lower overall quantization overhead.

4. Numerical error from intermediate rounding in fine-grained dyadic requantization.

→\to We directly compare merged and factorized dyadic requantization while keeping the quantization granularity and other settings fixed. FID is evaluated using 5k MJHQ prompts. Denoiser output error (NMSE) is measured against the same floating-point reference with identical inputs and averaged over 64 distinct MJHQ prompts. We also measure the incremental peak memory for each linear layer and report its mean across all layers.

Table B. Comparison between merged and factorized dyadic requantization formulation.

ModelDyadic requant.FIDNMSE (model, %)Memory
PixArt-Σmerged17.610.630.19 GB
factorized (ours)17.920.620.10 GB
PixArt-αmerged19.270.680.19 GB
factorized (ours)19.170.690.10 GB
SD3-mediummerged23.450.450.14 GB
factorized (ours)23.380.460.07 GB

The factorized formulation achieves comparable FID and denoiser accuracy while reducing the measured mean incremental layer peak memory by approximately 50%. Across the three models, the reported average NMSE differs by only 0.01 percentage points from the merged formulation. FID is also comparable, with small changes in either direction. These results indicate that intermediate rounding has a limited net effect on accuracy, while retaining separate token-wise and channel-wise scale factors substantially reduces memory consumption.

The additional rounding has a limited effect on the final accuracy because its rounding error is subsequently scaled by DN(SX/SY)DN(S_X /S_Y ) and can reinforce or partially offset existing quantization errors. This provides a possible explanation for the small NMSE changes in either direction.

We have included this analysis in the experimental section of the revised manuscript. We thank the reviewer for the thoughtful suggestion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to the Reviewers

We sincerely thank all the reviewers for taking the time to review, and for their thoughtful and constructive feedback. We are encouraged by their recognition of the practical value of our framework (Reviewers mcqe, vAqD), its technical soundness and effectiveness in addressing key challenges in integer-only inference (Reviewers mcqe, SPnG), and its strong generation quality across diverse diffusion architectures (Reviewers vAqD, SPnG). We also appreciate the positive feedback on the clarity of our motivation and technical presentation (Reviewer mcqe). The reviewers’ questions and suggestions are helpful in identifying where further clarification and evidence are needed, and we try our best to address them below.

Revisions are highlighted in blue in the updated manuscript. The main changes are:

  • Clarified the bit-width settings in the experimental setup.
  • Explicitly marked adapted baselines with †\dagger and added an explanatory footnote to the main table.
  • Added experiments on dyadic requantization factorization, cross-sampler generalization, and comparison under conventional quantization settings to the main experimental section.
  • Added a dedicated comparison with QFlash (Appendix C) and I-LLM (Appendix D).
  • Expanded the explanation of quality degradation in the ablation study, now in Appendix H (previously Appendix F).

We look forward to the reviewers’ responses and are happy to answer any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.