- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose Chameleon, a dynamic format adapter for post-training quantization (PTQ) of diffusion models. Instead of using a fixed numerical format, the proposed framework adaptively selects formats for different tensors based on computationally inexpensive statistical measures, including empirical activation kurtosis and closed-form diffusion signal-to-noise ratio (SNR). The framework considers variations across layers, weight channels, and diffusion timesteps where applicable. Its performance is evaluated on three diffusion model architectures using the COCO-2014 dataset.
Critical strengths
- Adaptive numerical format selection. The paper considers numerical format selection across different layers, weight channels, and diffusion timesteps, providing greater flexibility than conventional PTQ methods that use fixed numerical formats.
- Consistent improvements across different architectures. The experimental results show that the proposed framework improves quantized image generation quality across three different diffusion model architectures.
Critical weaknesses
- Limited methodological novelty and insufficient justification of parameter selection. The implementation of the proposed framework is relatively straightforward, relying mainly on an exhaustive search over a predefined set of numerical formats. Such an approach may become computationally expensive when the number of candidate formats or optimization variables increases. Moreover, the selection of the thresholds and is not sufficiently justified. Although the authors provide some explanations, it remains unclear how these thresholds are determined theoretically or systematically. If they are mainly selected through empirical trial and error, the contribution appears to be more of an empirical engineering study than a methodological advance.
- Inconsistent formatting in Table 1. In each model block of Table 1, two FID values are highlighted in bold, while only one CLIP value is highlighted. The authors should clarify the highlighting criteria and use consistent formatting.
- Minor notation issue. In lines 238 and 240, equation references should preferably use numbers in parentheses for consistency with standard academic writing conventions, such as "Eq. (2)" rather than "equation 2".
Questions for authors
- Is the selection of the numerical format sensitive to the choice of optimization objective, particularly when alternative error norms are used instead of the squared norm? The authors should investigate whether different objectives lead to different format selections and affect the quantization performance.
4 comments
Response to Reviewer 6x29 (2/2)
(W1-d) Why route to FP8 E5M2? At the shipped calibrated range, E5M2 is not chosen for per-value fidelity: on real SDXL activations its relative error is about twice E4M3's (4.5% vs 2.3%). Its effect on generation is nonetheless clear. We changed only the (layer, bucket) slots the rule routes to E5M2 (SDXL W4A8, 5k, A5000):
| E5M2-routed slots use | FID ↓ | CLIP ↑ |
|---|---|---|
| E5M2 (shipped rule) | 19.35 | 26.55 |
| E4M3 at the same calibrated range | 20.13 | 26.47 |
| E4M3 + random noise with E5M2's error energy | 19.74 | 26.63 |
Replacing E5M2 with the more accurate E4M3 costs 0.78 FID. Injecting zero-mean noise of exactly E5M2's error energy recovers about half of that loss, so about half of E5M2's benefit behaves like a regularizer in the noise-dominated, heavy-tailed regime. We have revised §3.3 and Appendix F to describe E5M2 this way and added the table.
(W2) Table 1 bolding. Fixed. Within each backbone and bit width we now bold the best quantized FID and the best quantized CLIP, with FP16 as the unbolded reference; the rule is stated in the caption.
(W3) Equation references. Fixed throughout: all references now read "Eq. (N)".
(Q1) Is format selection sensitive to the error norm? We reran SDXL W4A8 weight-format selection ranking the candidate formats by L1, L∞ and cosine distance instead of squared L2, holding the FP16 fallback fixed so that only the ranking changes (5k images; two platforms):
| Ranking objective | FID ↓ (GB200) | FID ↓ (A5000) | CLIP ↑ (GB200) |
|---|---|---|---|
| L2 (shipped) | 19.44 | 19.35 | 26.59 |
| L1 | 19.34 | 19.09 | 26.58 |
| cosine | 19.48 | 19.38 | 26.55 |
| L∞ | 19.97 | 19.75 | 26.46 |
L1, L2 and cosine agree to within 0.15 FID on GB200 and 0.3 on A5000 (seed-to-seed spread 0.14), so selection is not tied to the L2 choice. Only L∞ is consistently worse (+0.4–0.5 FID): ranking formats by their single worst element over-weights rare outliers. The table is added to the appendix.
We hope these results address the reviewer's concerns, in particular about soundness: the thresholds are robust, have a measured basis, and the rule outperforms error-based selection. We would be grateful if the reviewer would consider updating the assessment, and we are happy to answer any further questions.
Maintain my original score
Thanks for the detailed response. The additional experiments suggest that the chosen hyperparameters and in the original submission are not necessarily optimal. Moreover, the error norm used for the ranking objective may not be the most appropriate choice. These observations indicate that the paper lacks a sufficiently thorough comparison and analysis of alternative configurations and selection criteria. Therefore, I maintain my original score.
Response to Reviewer 6x29: robustness rather than optimality
Thank you for reading our response. We agree the study shows the shipped constants are not uniquely optimal; that was not our claim. Its purpose was to test whether Chameleon's conclusions depend on them, and they do not: across all six threshold variants FID stays within 19.39–19.52 (shipped: 19.44), a spread comparable to calibration-seed noise (SD up to 0.14), and every threshold and error-norm configuration, including L∞, still outperforms MoFQ-style selection under identical settings (worst variant 19.97 vs 20.08 on GB200; 19.75 vs 20.06 on A5000). The one consistent improvement the study surfaced, L1 instead of L2 for weight-format ranking (−0.10 / −0.26 FID), we will adopt as the default and report in the final version, alongside the full sensitivity tables already in Appendix A.14. We believe a method whose ranking against baselines is invariant to its hyperparameters is a stronger result than one tuned to a single setting, and we hope this clarifies the point.
Response to Reviewer 6x29 (1/2)
We thank the reviewer for the detailed and constructive review, and for recognizing the flexibility of selecting formats across layers, weight channels and timesteps, and the consistent improvements across three diffusion architectures. Below we address each weakness and question. All changes in the revised PDF are in blue.
(W1-a) Is the method an exhaustive search, and does it scale? There is no search on the activation side. For each (layer, timestep bucket), the rule reads two running moments (kurtosis) and the closed-form schedule SNR, then looks up the format: about 200 KB of state and no candidate fake-quantizations. The only search is offline on weights (|formats| × |group sizes| fake-quantizations per layer, done once). End-to-end calibration of all three backbones takes 54 minutes on one A100 (Appendix, calibration cost table).
More importantly, the rule is not an approximation of exhaustive error minimization; it is a different criterion, and a better one. Selecting each layer's activation format by its calibration error (the MoFQ-style arg-min) is worse than Chameleon's rule: FID 20.14 vs 19.35 (SDXL W4A8, 5k, A5000), and 20.08 vs 19.44 on a second platform (GB200). Error-based selection picks INT8 for 86% of layers; the κ/SNR rule keeps heavy-tailed activations in floating-point formats, which generation quality rewards.
(W1-b) How were κ and τ chosen? Are they trial and error? The thresholds were fixed before evaluation and not tuned per model. To show the method does not hinge on them, we moved each one independently (SDXL W4A8, 5k):
FID moves by at most 0.13 across the grid, comparable to the run-to-run spread of one configuration across calibration seeds (0.05–0.14). The thresholds sit on a flat region.
(W1-c) A principled basis for κ_high = 5. The reviewer's question prompted us to redo the generalized-Gaussian (GG) analysis with exactly the shipped static scaling rules, and to separate two error measures. Energy-weighted MSQE is dominated by a handful of outliers; under it the INT8/E4M3 crossover lies at κ ≈ 6–9 (depending on the calibration sample count), so it is not the right basis for the gates. The quantity that matters for generation is how well a format preserves the bulk of the distribution, and on that measure the crossover falls at κ ≈ 5, matching the shipped gate:
Above κ ≈ 5, INT8's step size is set by the outliers and an increasing share of ordinary activations rounds to zero; κ_low = 3 is a conservative bound below that point. We have replaced the energy-weighted MSQE statement with this bulk-error analysis wherever it appeared (§1, §3.3, Appendix B and Appendix F). The thresholds themselves are unchanged.