acceptodds
Under review as a conference paper at ICLR 2027

Chameleon: Dynamic Format Adapter for Efficient Diffusion

62 views

Abstract

A very popular way to enable image generation on memory-constrained accelerators using modern spatiotemporal generative models is post-training quantization (`PTQ`). However, every existing diffusion `PTQ` scheme shares an unstated commitment: the *number format* is fixed in advance, and only the scale, zero point, or the per-layer bit-width is allowed to move. This is unfortunate, because at a fixed bit-width the best format depends on the distribution being encoded, and that distribution differs from tensor to tensor: across weight channels, across layers, and — in a denoising model — along the *diffusion timestep*, where activations slide from heavy-tailed and noise-dominated at to tightly clustered and structured at . One format for the whole network is therefore mispriced for most of its tensors, and re-tuning a scale cannot repair a format mismatch. In this work, we propose a `PTQ` framework, which we call **Chameleon**, that holds the bit-width fixed and treats the number format itself as a discrete variable, chosen per weight channel and per *(layer, timestep bucket)* activation tensor. Chameleon draws activation formats from a palette of {`INT8`, `FP8 E4M3`, `FP8 E5M2`, `MXFP8`, `MXINT8`} and weight formats from {`INT8`, `MXINT8`} at bits or {`INT4`, `NF4`, `FP4 E2M1`, `MXINT4`, `MXFP4`} at bits, choosing activation formats ahead of time from two cheap statistics — empirical activation kurtosis and closed-form diffusion SNR — stored in a per-(layer, timestep bucket) lookup table, and weight formats offline by reconstruction error. An architectural fork adapts the same selection layer to each model family, so each model exercises the palette as far as its own statistics warrant: weights and per-(layer, bucket) activations on multi-step `SDXL`, weights alone where a single sampler step leaves no timestep axis. We evaluate Chameleon across three diffusion regimes — multi-step `SDXL`, single-step `SDXL-Turbo`, and the `PixArt`- Diffusion Transformer — on COCO-2014. Chameleon achieves the best `FID` for all six backbone bit-width settings, while preserving semantic alignment with the text prompts (`CLIP` within of the `FP16` reference in every setting, and the best of all quantized methods at ). Adapting the format, rather than only its scale, recovers distributional fidelity at 4-bit weights while holding prompt adherence at the `FP16` level.

Primary area:
generative models
Keywords:
Post-training quantization, Diffusion models, adaptive number formats, Timestep aware activation quantization, Efficient Edge Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6x29OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Chameleon, a dynamic format adapter for post-training quantization (PTQ) of diffusion models. Instead of using a fixed numerical format, the proposed framework adaptively selects formats for different tensors based on computationally inexpensive statistical measures, including empirical activation kurtosis and closed-form diffusion signal-to-noise ratio (SNR). The framework considers variations across layers, weight channels, and diffusion timesteps where applicable. Its performance is evaluated on three diffusion model architectures using the COCO-2014 dataset.

Critical strengths
  1. Adaptive numerical format selection. The paper considers numerical format selection across different layers, weight channels, and diffusion timesteps, providing greater flexibility than conventional PTQ methods that use fixed numerical formats.
  2. Consistent improvements across different architectures. The experimental results show that the proposed framework improves quantized image generation quality across three different diffusion model architectures.
Critical weaknesses
  1. Limited methodological novelty and insufficient justification of parameter selection. The implementation of the proposed framework is relatively straightforward, relying mainly on an exhaustive search over a predefined set of numerical formats. Such an approach may become computationally expensive when the number of candidate formats or optimization variables increases. Moreover, the selection of the thresholds κ\kappa and τ\tau is not sufficiently justified. Although the authors provide some explanations, it remains unclear how these thresholds are determined theoretically or systematically. If they are mainly selected through empirical trial and error, the contribution appears to be more of an empirical engineering study than a methodological advance.
  2. Inconsistent formatting in Table 1. In each model block of Table 1, two FID values are highlighted in bold, while only one CLIP value is highlighted. The authors should clarify the highlighting criteria and use consistent formatting.
  3. Minor notation issue. In lines 238 and 240, equation references should preferably use numbers in parentheses for consistency with standard academic writing conventions, such as "Eq. (2)" rather than "equation 2".
Questions for authors
  1. Is the selection of the numerical format sensitive to the choice of optimization objective, particularly when alternative error norms are used instead of the squared ℓ2\ell_2 norm? The authors should investigate whether different objectives lead to different format selections and affect the quantization performance.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6x29 (1/2)

We thank the reviewer for the detailed and constructive review, and for recognizing the flexibility of selecting formats across layers, weight channels and timesteps, and the consistent improvements across three diffusion architectures. Below we address each weakness and question. All changes in the revised PDF are in blue.

(W1-a) Is the method an exhaustive search, and does it scale? There is no search on the activation side. For each (layer, timestep bucket), the rule reads two running moments (kurtosis) and the closed-form schedule SNR, then looks up the format: about 200 KB of state and no candidate fake-quantizations. The only search is offline on weights (|formats| × |group sizes| fake-quantizations per layer, done once). End-to-end calibration of all three backbones takes 54 minutes on one A100 (Appendix, calibration cost table).

More importantly, the rule is not an approximation of exhaustive error minimization; it is a different criterion, and a better one. Selecting each layer's activation format by its calibration error (the MoFQ-style arg-min) is worse than Chameleon's rule: FID 20.14 vs 19.35 (SDXL W4A8, 5k, A5000), and 20.08 vs 19.44 on a second platform (GB200). Error-based selection picks INT8 for 86% of layers; the κ/SNR rule keeps heavy-tailed activations in floating-point formats, which generation quality rewards.

(W1-b) How were κ and τ chosen? Are they trial and error? The thresholds were fixed before evaluation and not tuned per model. To show the method does not hinge on them, we moved each one independently (SDXL W4A8, 5k):

SettingFID ↓CLIP ↑
Shipped (κ_low=3, κ_high=5, τ=(0.2, 2))19.4426.59
κ_low = 2.5 / 3.519.40 / 19.4226.59 / 26.58
κ_high = 4.5 / 6.019.52 / 19.4026.59 / 26.58
τ = (0.1, 1) / (0.5, 5)19.39 / 19.4426.60 / 26.59

FID moves by at most 0.13 across the grid, comparable to the run-to-run spread of one configuration across calibration seeds (0.05–0.14). The thresholds sit on a flat region.

(W1-c) A principled basis for κ_high = 5. The reviewer's question prompted us to redo the generalized-Gaussian (GG) analysis with exactly the shipped static scaling rules, and to separate two error measures. Energy-weighted MSQE is dominated by a handful of outliers; under it the INT8/E4M3 crossover lies at κ ≈ 6–9 (depending on the calibration sample count), so it is not the right basis for the gates. The quantity that matters for generation is how well a format preserves the bulk of the distribution, and on that measure the crossover falls at κ ≈ 5, matching the shipped gate:

  • Bulk SQNR (|x| ≤ 99th percentile, shipped static scales): at κ = 4.5, INT8 34.2 dB vs FP8-E4M3 31.4 dB; at κ = 5.3, INT8 31.2 dB vs FP8-E4M3 31.6 dB.
  • Share of nonzero activations that INT8 rounds to zero: 1.7% at κ = 3, 6.4% at κ = 6, 35% at κ = 25.

Above κ ≈ 5, INT8's step size is set by the outliers and an increasing share of ordinary activations rounds to zero; κ_low = 3 is a conservative bound below that point. We have replaced the energy-weighted MSQE statement with this bulk-error analysis wherever it appeared (§1, §3.3, Appendix B and Appendix F). The thresholds themselves are unchanged.

AuthorsOpenReview ↗

Response to Reviewer 6x29 (2/2)

(W1-d) Why route to FP8 E5M2? At the shipped calibrated range, E5M2 is not chosen for per-value fidelity: on real SDXL activations its relative error is about twice E4M3's (4.5% vs 2.3%). Its effect on generation is nonetheless clear. We changed only the (layer, bucket) slots the rule routes to E5M2 (SDXL W4A8, 5k, A5000):

E5M2-routed slots useFID ↓CLIP ↑
E5M2 (shipped rule)19.3526.55
E4M3 at the same calibrated range20.1326.47
E4M3 + random noise with E5M2's error energy19.7426.63

Replacing E5M2 with the more accurate E4M3 costs 0.78 FID. Injecting zero-mean noise of exactly E5M2's error energy recovers about half of that loss, so about half of E5M2's benefit behaves like a regularizer in the noise-dominated, heavy-tailed regime. We have revised §3.3 and Appendix F to describe E5M2 this way and added the table.

(W2) Table 1 bolding. Fixed. Within each backbone and bit width we now bold the best quantized FID and the best quantized CLIP, with FP16 as the unbolded reference; the rule is stated in the caption.

(W3) Equation references. Fixed throughout: all references now read "Eq. (N)".

(Q1) Is format selection sensitive to the error norm? We reran SDXL W4A8 weight-format selection ranking the candidate formats by L1, L∞ and cosine distance instead of squared L2, holding the FP16 fallback fixed so that only the ranking changes (5k images; two platforms):

Ranking objectiveFID ↓ (GB200)FID ↓ (A5000)CLIP ↑ (GB200)
L2 (shipped)19.4419.3526.59
L119.3419.0926.58
cosine19.4819.3826.55
L∞19.9719.7526.46

L1, L2 and cosine agree to within 0.15 FID on GB200 and 0.3 on A5000 (seed-to-seed spread 0.14), so selection is not tied to the L2 choice. Only L∞ is consistently worse (+0.4–0.5 FID): ranking formats by their single worst element over-weights rare outliers. The table is added to the appendix.

We hope these results address the reviewer's concerns, in particular about soundness: the thresholds are robust, have a measured basis, and the rule outperforms error-based selection. We would be grateful if the reviewer would consider updating the assessment, and we are happy to answer any further questions.

Reviewer_6x29OpenReview ↗

Maintain my original score

Thanks for the detailed response. The additional experiments suggest that the chosen hyperparameters κ\kappa and τ\tau in the original submission are not necessarily optimal. Moreover, the error norm used for the ranking objective may not be the most appropriate choice. These observations indicate that the paper lacks a sufficiently thorough comparison and analysis of alternative configurations and selection criteria. Therefore, I maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer 6x29: robustness rather than optimality

Thank you for reading our response. We agree the study shows the shipped constants are not uniquely optimal; that was not our claim. Its purpose was to test whether Chameleon's conclusions depend on them, and they do not: across all six threshold variants FID stays within 19.39–19.52 (shipped: 19.44), a spread comparable to calibration-seed noise (SD up to 0.14), and every threshold and error-norm configuration, including L∞, still outperforms MoFQ-style selection under identical settings (worst variant 19.97 vs 20.08 on GB200; 19.75 vs 20.06 on A5000). The one consistent improvement the study surfaced, L1 instead of L2 for weight-format ranking (−0.10 / −0.26 FID), we will adopt as the default and report in the final version, alongside the full sensitivity tables already in Appendix A.14. We believe a method whose ranking against baselines is invariant to its hyperparameters is a stronger result than one tuned to a single setting, and we hope this clarifies the point.

Reviewer_MuJSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Chameleon proposes a post-training quantization method for diffusion models that, after fixing the quantization bit width, adjusts the quantization format to accommodate different data distributions. The continuous denoising process in diffusion models causes the distributions of activation data to vary across different time steps. Additionally, data distributions differ across layers and weight channels. The authors incorporate these three factors into the selection of quantization formats. By utilizing activation kurtosis, which is a statistical measure reflecting the tail characteristics of data distributions, in conjunction with SNR, these two metrics jointly determine the quantization format for activation data according to fixed rules. Through calibration, quantization format is fixed in a LUT for lookup during inference. For weights, the optimal weight quantization format is selected per channel based on SQNR. Additionally, different quantization format configuration methods are designed for different architectural. The authors conducted experiments on the SDXL, SDXL-Turbo, and PixArt-α\alpha models. The results show that, under both the W8A8 and W4A8 configurations, Chameleon achieves significant improvements in image generation quality.

Critical strengths
  1. Introduced a method for selecting the appropriate quantization format based on different data distributions. Use a fixed statistical measure of the data to determine the quantization format and store it in a LUT for inference.

  2. Designed different quantization format configuration methods tailored to the characteristics of various architectures. Additionally designed a joint search for group size and quantization format.

Critical weaknesses
  1. The experiment is based on fake-quantization and did not include end-to-end latency measurements. Therefore, the actual deployment efficiency and overhead have not yet been quantified. The lack of a unified format for activations and weights may increase the difficulty of actual deployment.

  2. The paper lacks comparisons with several recent PTQ baselines, such as ViDiT-Q and FP4DiT which are mentioned in related work. Including comparisons with recent baselines would strengthen the paper’s positioning.

Questions for authors
  1. Can the authors report at least one comparison result with a recent baseline method?

  2. In SDXL-Turbo, since the activation processing is consistent with MixDQ, can the authors confirm which weight format predominates in the quantization format selection? Which layers’ quantization format replacements can most improve generation quality?

1 comment
AuthorsOpenReview ↗

Response to Reviewer MuJS

Thank you for the detailed and encouraging review. We appreciate the recognition of selecting the quantization format from data statistics stored in a lookup table, and of the architecture-specific configurations, including the joint search over group size and format. Below we address the weaknesses and questions. All changes in the revised PDF are in blue.

(W1) Fake quantization only; no latency; mixed formats may be hard to deploy. We implemented and measured real kernels for Chameleon's mixed per-layer formats.

Exact Chameleon numerics on RTX 6000 Ada (FP8 tensor cores), Triton kernel:

  • Weights are stored as 4-bit codes. INT4-asym and FP4 decode exactly into FP8 tensor-core operands; NF4 uses FP16 MMA.
  • Activations are per-tensor FP8 (E4M3/E5M2), cast with one round-to-nearest-even.
  • One kernel serves every weight format through a 32-entry lookup table, so mixing formats across layers adds no kernel variants and no runtime cost.
PixArt-α 1024², 20 steps, batch 1ms / imagevs FP16Transformer weights
FP16 (cuBLAS)2,7761.00×1,165 MB
Fake quantization (as in the paper)4,7120.59×—
Chameleon, real kernel2,5551.09×392 MB (3.0× smaller)
Quantized linear layers only (CUDA graphs)1,206 vs 1,4321.19×

Hardware-native variant on Blackwell (GB200). Using power-of-two block scales (MX, E8M0, block 32), every format in the palette runs as one cuBLAS MXFP8 GEMM. It is bit-exact against its own reference, and the GEMMs are 1.74× faster than FP16 (4.00 vs 6.95 ms per step). End to end it matches FP16 (494 vs 493 ms), because on GB200 attention and elementwise work take 70% of each step; Chameleon quantizes the linear layers, and attention quantization is orthogonal. The hardware-native scaling costs +0.93 FID relative to exact Chameleon and still beats Q-DiT at the same group size.

Fidelity. Real kernels match fake quantization per layer (≤ 9e-4 on Ada; bit-exact on GB200). FID reproduces exactly across A100 and GB200.

Why the DiT first. One kernel reproduces it exactly (one format per linear, per-tensor FP8 activations); the UNet paths also need per-channel mixed weight formats, convolutions and INT8/MXFP8 activations, so their kernels are future work.

These results are in a new appendix, and §4.1 now points to them.

(W2-Q1) Comparison with a recent baseline. We ran ViDiT-Q on PixArt-α with our prompts, seeds and scoring. Relative to each method's own FP16 reference, Chameleon improves FID by 3.30 (W8A8) and 5.39 (W4A8), versus 0.83 and 3.90 for ViDiT-Q. Details, and FP4DiT/PTQ4DiT, are in our reply to Reviewer 16Hj (Q2).

(Q2) SDXL-Turbo: which weight format dominates, and which layers matter most? Format statistics. All 777 quantized UNet layers are 4-bit. The joint (format, group size) search selects INT4-asym with g = 32 for 769 layers (99.0%) and NF4 with g = 32 for 8 (7 of them in the up-blocks); FP4, MXINT4 and MXFP4 are never chosen. In the one-step regime the palette therefore reduces almost entirely to group-wise asymmetric INT4, and Chameleon's gain over MixDQ on SDXL-Turbo comes from that choice of format and granularity rather than from mixing many formats. We now state this explicitly. SDXL-Turbo is the one backbone where this happens: on SDXL the per-layer activation choice is worth 0.82 FID over the best single format, and per-channel weight selection draws on several formats (removing MXFP4 alone costs 0.38 FID); on PixArt-α the (g, f) search selects NF4 for 69 of 282 layers, and at equal group size adaptive formats beat every fixed format.

Layer sensitivity. We replaced Chameleon's choice with per-output-channel symmetric INT4 in one block group at a time, keeping every layer at 4 bits (5k):

Block group switched to per-channel INT4ΔFID vs Chameleon
FFN−0.27
mid block+0.13
attention+0.46
down blocks+0.54
ResNet blockscollapse (FID > 225)
up blockscollapse (FID > 244)

The sensitive layers are the up-block and ResNet layers that receive skip-concatenated inputs: at a uniform 4 bits, per-output-channel scaling cannot represent them. MixDQ avoids this collapse with mixed precision (its "W4" configuration keeps 106 of 410 up-block layers at 8 bits, averaging 4 bits overall). Chameleon keeps every layer at 4 bits and remains stable through group-wise asymmetric scaling. The format and granularity choice in the skip-connected decoder is therefore where quality is won on SDXL-Turbo. The statistics and the sensitivity table are added to the appendix, together with a new figure of the few-step path (Figure 4 in the revised supplement) that shows the per-layer (g, f) choices, the sensitivity map and MixDQ's bit allocation side by side.

We thank the reviewer again for the positive assessment and hope the new latency and deployment results strengthen it. We are happy to answer any further questions.

Reviewer_16HjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Chameleon, a PTQ framework that selects number formats at a fixed bit width. Weight formats are selected offline by quantization error, while activation formats are chosen per layer and timestep bucket using kurtosis and diffusion SNR. The framework adapts these choices to SDXL, SDXL-Turbo, and PixArt-α, and reports improved FID over the evaluated quantization baselines at W8A8 and W4A8.

Critical strengths
  1. The format-adaptation idea has practical relevance. Choosing number formats according to differences in tensor distributions provides a reasonable direction beyond adjusting only scales and bit widths. The framework applies this idea to both weights and activations across different diffusion architectures.

  2. The complete method shows useful improvements without gradient-based optimization. At W4A8, Chameleon reduces FID from 21.41 to 14.43 over Q-Diffusion on SDXL and from 24.50 to 21.29 over MixDQ on SDXL-Turbo. The reported calibration cost is at most 45 minutes on one A100.

Critical weaknesses
  1. The methodological novelty has not been sufficiently substantiated. MoFQ already explores layer-wise format selection, TDQ adjusts quantization parameters across diffusion timesteps, and FP4DiT studies floating-point quantization for diffusion transformers. Although these works are cited, the paper does not clearly establish the additional value of its format-selection strategy. Direct comparisons with FP4DiT, ViDiT-Q, and PTQ4DiT are also missing.

  2. The necessity of timestep-dependent format switching is not sufficiently demonstrated. SDXL-Turbo uses no temporal switching, and PixArt-α selects E5M2 for 97–100% of its quantized linear layers in every bucket. The main text reports that layer-wise format selection improves SDXL FID by 0.82, while the timestep axis adds only 0.12. These results support choosing suitable formats more clearly than they support switching formats along the diffusion trajectory.

Questions for authors
  1. What does the proposed format-selection strategy add beyond layer-wise format selection and timestep-dependent quantization parameters in methods such as MoFQ and TDQ?

  2. How does Chameleon compare with FP4DiT, ViDiT-Q, and PTQ4DiT under the same bit widths and sampling settings? FP4DiT is particularly relevant to the claimed benefit over existing floating-point quantization.

  3. How much improvement requires changing formats across timesteps rather than fixing one format per layer?

  4. How much of the PixArt-α improvement comes from reducing the weight group size from 128 to 32? Please compare integer, fixed floating-point, and adaptive-format quantization at the same group size.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 16Hj (1/2)

We thank the reviewer for the thoughtful and detailed review. The reviewer recognized the practical relevance of adapting the number format to tensor distributions, beyond scales and bit widths, and the improvements Chameleon achieves without gradient-based optimization. Below we address each weakness and question. All changes in the revised PDF are in blue.

(W1-Q1) What does Chameleon add beyond MoFQ and TDQ? MoFQ chooses one format per layer (INT vs FP) by quantization error; TDQ keeps one integer format and learns per-timestep scales. To measure the additional value directly, we reproduced both ideas in our harness (SDXL W4A8, 5k, all rows on the same GPUs):

MethodWhat adaptsFID ↓CLIP ↑
TDQ-style: all INT8, per-timestep-bucket scalesscale per timestep23.9226.18
MoFQ-style, activations: per-layer INT8/FP8-E4M3 by calibration errorformat per layer, by error20.1426.47
MoFQ-style, full: + per-layer INT4/FP4 weights by errorformat per layer, by error20.0626.50
Chameleonformat per layer/channel, κ/SNR rule19.3526.55
  • Versus MoFQ (−0.7 FID). Applying MoFQ's selection to weights as well changes little (20.14 → 20.06; 370 of 372 layers pick FP4). The difference is the selection criterion: error-based selection sends 86% of activation layers to INT8, whereas Chameleon's robustness-based rule keeps heavy-tailed activations in FP formats. Selecting by calibration error is a worse objective for generation quality than the κ/SNR statistics.
  • Versus TDQ-style scale adaptation (−4.6 FID). Our TDQ proxy is training-free (TDQ learns its scales). Each timestep bucket gets its own calibrated INT8 range, widened by 1.5× because calibrated per-bucket ranges are exceeded at inference time (without the widening the model collapses). Even with the widened ranges, adapting only the scale per timestep is clearly worse than adapting the format at the same bit width. This supports the paper's premise that a scale cannot repair a format mismatch.

Beyond these, Chameleon differs in scope: one framework covers per-channel weights (UNet), joint group-size/format search (DiT) and few-step models, with a mixed INT/FP/NF/MX palette and a single deployable kernel (see our reply to MuJS).

FP4DiT, ViDiT-Q and PTQ4DiT: see (Q2) in part 2.

(W2-Q3) Is timestep-dependent switching necessary? How much requires changing formats across timesteps? We thank the reviewer for this question; it led to the most informative experiment of the rebuttal. We repeated the bucket ablation with three independent calibration seeds per arm (SDXL W4A8, 5k, GB200):

Timestep bucketsSeed 0Seed 1000Seed 2000Mean ± SD
B = 1 (one format per layer)19.1819.2519.2819.24 ± 0.05
B = 10 (shipped)19.4419.3719.6419.48 ± 0.14

At 8-bit activations, a single format per layer is as good as or better than switching across ten buckets; the 0.12 FID timestep gain in the submission's ablation is within seed-to-seed variation. At 6-bit activations (W4A6), where the format choice is tighter, a moderate number of buckets helps:

W4A6 activations, SDXL, 5kFID ↓CLIP ↑
Single format (all FP6-E2M3)20.0326.64
κ/SNR rule, B = 119.6226.84
κ/SNR rule, B = 519.4026.73
κ/SNR rule, B = 1019.7026.71

Conclusion and revision. The reviewer's reading is correct: the consistent gain comes from choosing the format per layer (0.3–0.6 FID over the best single format at A6; 0.82 at A8 in the paper's ablation). We have reframed the abstract and introduction so that training-free, per-layer/per-channel format selection is the core contribution. Timestep bucketing is retained as an optional axis of the same rule: it costs nothing at inference (one table lookup), it gave the best result at W4A6 (B = 5), and on PixArt-α the calibrated table reduces to a static choice (E5M2 in 97–100% of layers), as the reviewer noted. The seed study and the W4A6 study are added to the appendix.

AuthorsOpenReview ↗

Response to Reviewer 16Hj (2/2)

(Q2) Comparison with ViDiT-Q, FP4DiT and PTQ4DiT. ViDiT-Q has its own PixArt-α pipeline (its own FP16 reference and DPM-Solver sampler), so we ran it with our prompts, seeds and scoring and compare each method's change relative to its own FP16 reference:

PixArt-αFP16W8A8 (Δ vs FP16)W4A8 (Δ vs FP16)
ViDiT-Q (own pipeline, 5k)31.0130.18 (−0.83)27.11 (−3.90)
Chameleon (Table 1)27.6324.33 (−3.30)22.24 (−5.39)

For W4A8, ViDiT-Q's released PixArt configurations use 6-bit weights and per-channel weight scales only; per-channel W4 under its pipeline collapses (FID 140.2), so we added group-wise (g = 128) W4 weight scales to ViDiT-Q's quantizer (matching Q-DiT's setting) to obtain a fair W4A8 baseline. Chameleon improves on its FP16 reference by more at both bit widths. (FID values across the two pipelines are not directly comparable, since the FP16 references and sample sizes differ; the Δ column is the fair comparison.)

FP4DiT (Chen et al., TMLR 2025) reports PixArt-α at 512×512 (10k COCO-2014 prompts, its own pipeline), whereas we evaluate the 1024×1024 model, so we again compare changes relative to each reference. From FP4DiT's Table 2 (full precision: FID 34.05): FP4DiT W4A8 30.37 (Δ −3.68), W4A6 40.61 (Δ +6.56); in the same table, Q-DiT gives 36.17 / 43.91 and ViDiT-Q 39.11 / 56.54. Chameleon improves on its FP16 reference by 5.39 at W4A8 (Table 1), and moving from W4A8 to W4A6 costs it only +0.10 FID under our protocol, whereas FP4DiT loses 10.2 FID between those two settings. These are published numbers under a different resolution and protocol, not reruns in our harness. PTQ4DiT targets class-conditional DiT-XL/2 on ImageNet, outside our text-to-image setting; we cite its published results.

(Q4) How much of the PixArt-α gain comes from reducing the group size from 128 to 32? We compared integer, fixed-format and adaptive-format quantization at equal group size (PixArt-α W4A8, 5k):

  • At equal g = 32, Chameleon beats Q-DiT by 1.2 FID. About 0.9 comes from activation routing (Q-DiT INT4 vs Chameleon INT4-only weights) and about 0.3 from adaptive weight formats (INT4-only → adaptive). Fixed FP4-only and NF4-only are each worse than adaptive.
  • Reducing Q-DiT's group size from 128 to 32 makes it 2.7 FID worse, so a smaller group does not by itself explain Chameleon's gain.
  • At 8 bits, Q-DiT changes by only 0.2 FID between g = 128 and g = 32, and Chameleon is 3.3–3.5 FID better at either group size.
  • At Q-DiT's default g = 128 under the paper's full protocol (Table 1), Chameleon reaches 22.24 FID vs Q-DiT's 23.53.

The full integer/fixed-FP/adaptive table at equal group size is added to the appendix.

We hope these results clarify Chameleon's contribution relative to MoFQ and TDQ and answer the reviewer's questions. We would be grateful if the reviewer would consider updating the assessment, and we welcome further questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of the rebuttal: new experiments and revisions

We sincerely thank all three reviewers for their careful reading and constructive feedback. The reviews led us to run a substantial set of new experiments, which we summarize here; each reviewer-specific reply gives the details. All changes in the revised PDF are marked in blue.

Protocol note. To fit the rebuttal period, new experiments use 5,000 COCO val2014 captions (same prompt order, seeds, clean-FID and CLIP-L/14 scoring as the paper) instead of the paper's 24,576. FID depends on sample size, so 5k numbers are compared only with each other, and every row in a table was run on the same GPU type and software stack.

1. Real low-precision kernels, not only fake quantization (MuJS). We implemented and measured kernels for Chameleon's mixed per-layer formats.

  • RTX 6000 Ada (FP8 tensor cores), exact Chameleon numerics: 1.09× end-to-end speedup over FP16 on PixArt-α 1024², 1.19× on the quantized linear layers, 3.0× smaller transformer weights (392 vs 1,165 MB), outputs within 9e-4 of the fake-quant reference.
  • One kernel serves every weight format through a 32-entry lookup table, so mixing formats per layer costs nothing extra at runtime.
  • Blackwell (GB200), hardware-native MX variant: GEMMs 1.74× faster than FP16 and bit-exact against its own reference.

2. Direct comparison with MoFQ and TDQ (16Hj). SDXL W4A8, 5k, same GPUs:

MethodWhat adaptsFID ↓CLIP ↑
TDQ-style (INT8, per-timestep scales, training-free)scale23.9226.18
MoFQ-style, activations (per-layer INT8/FP8 by error)format, by error20.1426.47
MoFQ-style, weights + activationsformat, by error20.0626.50
Chameleonformat, by κ/SNR rule19.3526.55

3. The thresholds are not hand-tuned (6x29). Moving any of the four thresholds changes FID by at most 0.13 (19.39–19.52 vs 19.44), and the format selection is insensitive to the error norm (L1/L2/cosine within 0.15 FID). We also give a principled basis for κ_high = 5 (where INT8 stops preserving the bulk of the distribution better than FP8 under the shipped scaling rules).

4. The value of switching formats over timesteps (16Hj). With three calibration seeds per arm, most of the benefit comes from choosing the format per layer: a single timestep bucket is as good as or better than ten at A8. We have revised the framing accordingly: per-layer/per-channel format selection is the core contribution, and timestep bucketing is retained as an optional axis (it gave the best result at W4A6 with five buckets).

5. Recent baselines (16Hj, MuJS). ViDiT-Q on PixArt-α under its own pipeline with our prompts, seeds and scoring; FP4DiT and PTQ4DiT added to the comparison with their published numbers.

6. Group size vs. format on PixArt-α (16Hj). At equal group size (g = 32) Chameleon beats Q-DiT by 1.2 FID, while shrinking Q-DiT's group from 128 to 32 makes it 2.7 FID worse; the gain is not group size.

7. SDXL-Turbo format statistics and layer sensitivity (MuJS).

Revisions to the paper (blue text):

  • Reframed abstract/introduction around per-layer format selection (timestep bucketing as an optional axis).
  • Replaced the generalized-Gaussian MSQE argument for the kurtosis gates with the measured bulk-error analysis (six places).
  • Clarified the role of E5M2 with the new E5M2/E4M3/noise experiment.
  • Corrected the all-INT8 ablation discussion with the new range-headroom result.
  • New appendix with all rebuttal experiments: threshold and error-norm sensitivity, MoFQ/TDQ comparisons, timestep-bucket study, group-size study, ViDiT-Q, real-kernel latency, SDXL-Turbo statistics.
  • Table 1 bolding rule and equation references fixed.
AuthorsOpenReview ↗

Gentle reminder before the response period closes tonight (AoE)

Dear Reviewers 6x29, 16Hj and MuJS,

The author response period closes tonight (November 18) at 23:59 AoE, and we have not yet heard from any of you. We would be very grateful for a short reply, so that we know whether our responses resolve your concerns or whether anything else is needed. The headline changes in the revised PDF (all edits in blue; new Appendix A.14) are:

  • Real kernels (MuJS). A mixed-format kernel with exact Chameleon numerics runs PixArt-α 1.09× faster than FP16 end to end on an RTX 6000 Ada, with 3× smaller transformer weights (392 vs 1,165 MB). A hardware-native MX variant makes the GEMMs 1.74× faster on GB200.
  • Robustness of the routing rule (6x29). A threshold grid moves FID by at most 0.08 (19.39–19.52 vs 19.44); L1, L2 and cosine error norms give equivalent results; three calibration seeds quantify the noise floor.
  • Stronger baselines (16Hj, MuJS). Chameleon beats MoFQ-style and TDQ-style selection under identical settings; at equal group size (g = 32) it beats Q-DiT by 1.2 FID; ViDiT-Q, run in its own pipeline, gains 3.90 FID over its FP16 reference at W4A8 versus 5.39 for Chameleon; published FP4DiT numbers are discussed.
  • Revised claims (6x29). The format analysis is redone under the scaling rules we actually ship, the E5M2 mechanism is restated, and the timestep refinement is now framed as optional: most of the gain is per-layer, and timestep buckets help mainly at 6-bit activations.
  • SDXL-Turbo (MuJS). Per-layer format statistics, a block-wise sensitivity study, and a new figure of the few-step path.

Even a one-line acknowledgment would help close the loop. Thank you again for your time and reviews.

The Authors

Area_Chair_qC9iOpenReview ↗

Reminder: Please Respond to Author Rebuttals

Dear Reviewers,

As the discussion period is coming to an end, please review the authors’ responses and provide a brief follow-up on whether your concerns have been addressed.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.