Chameleon: Dynamic Format Adapter for Efficient Diffusion

A very popular way to enable image generation on memory-constrained accelerators using modern spatiotemporal generative models is post-training quantization (`PTQ`). However, every existing diffusion `PTQ` scheme shares an unstated commitment: the *number format* is fixed in advance, and only the scale, zero point, or the per-layer bit-width is allowed to move. This is unfortunate, because at a fixed bit-width the best format depends on the distribution being encoded, and that distribution differs from tensor to tensor: across weight channels, across layers, and — in a denoising model — along the *diffusion timestep*, where activations slide from heavy-tailed and noise-dominated at $t\to T$ to tightly clustered and structured at $t\to 0$. One format for the whole network is therefore mispriced for most of its tensors, and re-tuning a scale cannot repair a format mismatch. In this work, we propose a `PTQ` framework, which we call **Chameleon**, that holds the bit-width fixed and treats the number format itself as a discrete variable, chosen per weight channel and per *(layer, timestep bucket)* activation tensor. Chameleon draws activation formats from a palette of {`INT8`, `FP8 E4M3`, `FP8 E5M2`, `MXFP8`, `MXINT8`} and weight formats from {`INT8`, `MXINT8`} at $8$ bits or {`INT4`, `NF4`, `FP4 E2M1`, `MXINT4`, `MXFP4`} at $4$ bits, choosing activation formats ahead of time from two cheap statistics — empirical activation kurtosis and closed-form diffusion SNR ${\bar{\alpha }}_t/(1-{\bar{\alpha }}_t)$ — stored in a per-(layer, timestep bucket) lookup table, and weight formats offline by reconstruction error. An architectural fork adapts the same selection layer to each model family, so each model exercises the palette as far as its own statistics warrant: weights and per-(layer, bucket) activations on multi-step `SDXL`, weights alone where a single sampler step leaves no timestep axis. We evaluate Chameleon across three diffusion regimes — multi-step `SDXL`, single-step `SDXL-Turbo`, and the `PixArt`-$\alpha$ Diffusion Transformer — on COCO-2014. Chameleon achieves the best `FID` for all six backbone $\times$ bit-width settings, while preserving semantic alignment with the text prompts (`CLIP` within $0.24$ of the `FP16` reference in every setting, and the best of all quantized methods at $W_4{A}_8$). Adapting the format, rather than only its scale, recovers distributional fidelity at 4-bit weights while holding prompt adherence at the `FP16` level.