- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents CentriQ, a calibration-free post-training quantization framework for diffusion transformers. The paper observes that adaptive LayerNorm introduces a token-dependent channel mean into DiT activations, and argues that conventional Hadamard rotations may concentrate this DC component into a few coordinates, thereby dominating the activation quantization range. CentriQ subtracts the per-token channel mean before rotation and restores its contribution exactly through a full-precision rank-1 correction. The remaining centered activation is processed by a factorized Hadamard transform and dynamically quantized. The paper further introduces RoDS, an L_p-based grouped weight quantizer designed for heavy-tailed DiT weights.
Critical strengths
- The paper identifies an interesting failure mode of Hadamard-based activation quantization in adaLN-based DiTs.
- At W4A4, the component ablation does show that centering is useful.
- The paper is clear and well presented.
Critical weaknesses
- Mean-centered DiT quantization is already closely related to existing work. HadaNorm already studies mean-centered transformations for DiT quantization, while ConvRot explicitly addresses the mean component using regular group-wise Hadamard transforms.
- The main calibration-free baseline comparison is not sufficiently strong. Table 1 primarily compares CentriQ against RTN, NF4, and HQQ combined with plain per-token activation quantization. These are not competitive calibration-free DiT activation quantizers. Stronger related methods such as ConvRot and OrbitQuant are discussed by the paper itself but are absent from the main fidelity comparison.
- The “calibration-free” claim should be qualified more carefully. The paper states that the rotation layout and clipping percentile were selected using a disjoint 1,000-prompt MJHQ development split.
- The practical efficiency benefit is weak. On FLUX.1-schnell, the reported end-to-end latency improves only from 1.77 s in BF16 to 1.69 s with W4A4 CentriQ.
Questions for authors
- Can the authors report transformer-only latency, activation preprocessing latency, rotation latency, INT4 GEMM latency, epilogue latency, peak memory, and throughput separately?
8 comments
W4. The efficiency benefit is weak. The latency in Section 5.1 came from an unoptimized prototype without the engine-level fusions of SVDQuant's Nunchaku engine. We have now optimized CentriQ the same way and run both 4-bit methods in the same engine on the same layers, with identical settings for all methods (1024², 4 steps, batch 1, mean ± std):
| Transformer (s) | BF16 | BF16 + compile | SVDQuant | CentriQ |
|---|---|---|---|---|
| A100, INT4 (150 imgs) | 1.762 ± 0.008 | 1.407 ± 0.005 | 1.245 ± 0.005 | 1.289 ± 0.005 |
| RTX PRO 6000, NVFP4 (50 imgs) | 1.166 ± 0.001 | 1.006 ± 0.001 | 0.646 ± 0.000 | 0.664 ± 0.000 |
| End to end, A100 / RTX | 2.000 / 1.338 | 1.644 / 1.178 | 1.480 / 0.817 | 1.522 / 0.834 |
CentriQ is within 2.8–3.5% of calibrated SVDQuant and 1.37× (A100) / 1.76× (RTX) faster than BF16.
Q1. Per-component latency, memory, throughput.
| A100, ms per image | BF16 | SVDQuant | CentriQ |
|---|---|---|---|
| Centering + rotation (one fused kernel) | – | – | 22.8 |
| Activation quantization | – | 118.7 | 122.8 |
| INT4 GEMM + fused epilogue | – | 428.8 | 441.9 |
| BF16 GEMM | 956.1 (all layers) | 262.1 | 263.7 |
| Attention | 302.6 | 297.6 | 299.0 |
| Norm / RoPE / elementwise | 503.5 | 132.4 | 132.7 |
Centering and rotation run as one fused kernel, and the epilogue is fused into the GEMM, so neither can be timed separately. Peak memory: 36.3 GB (BF16), 26.6 GB (SVDQuant), 26.8 GB (CentriQ). Throughput: 0.66 / 1.20 img/s on A100 / RTX (BF16: 0.50 / 0.75). All added to the revised paper.
We hope these results address the reviewer's concerns, and we would be glad to clarify any remaining issue.
Kind request to check our responses
Dear Reviewer, thank you for your detailed review. Your feedback have helped us strengthen the paper. As the discussion period is coming to an end, we wanted to kindly check whether you have had a chance to look at our response. Following your suggestions, we have:
- Stronger baselines: added ConvRot- and HadaNorm-style activation paths and calibrated SVDQuant to the main comparison. CentriQ is clearly better than both alternatives on PixArt-Σ and at 2-bit activations.
- OrbitQuant: compared on its own protocol.
- Calibration-free: qualified the claim, and showed the settings transfer unchanged to FLUX.1-dev.
- Efficiency: reported per-component latency, memory and throughput.
We hope these additions address your concerns, and we would be grateful if you could take them into account in your final assessment. We would be glad to clarify any remaining points.
Rebuttal by Authors
Thank you for your detailed and constructive feedback. We greatly appreciate your recognition that our paper identifies an interesting failure mode of Hadamard-based activation quantization in adaLN-based DiTs, that centering is useful at W4A4, and that the paper is clear and well presented. We address your major questions and concerns below.
W1. Overlap with HadaNorm and ConvRot. We agree centering is not new and will say so in Section 2. The difference is which mean is removed. HadaNorm removes a per-channel mean over tokens (with calibrated scales), which leaves the per-token adaLN offset in place. ConvRot mixes only within small regular-Hadamard blocks, so the mean is not spread across the full width. CentriQ removes the per-token mean exactly (Wx = W(x − μₜ1) + μₜ·W1) and then rotates globally, without data.
W2. The calibration-free baselines in Table 1 are weak; ConvRot and OrbitQuant are missing. We agree that RTN, NF4 and HQQ with plain per-token activations are weak baselines. Following the reviewer's suggestion, we added the closest calibration-free activation quantizers to the fidelity comparison. We reimplemented ConvRot (its release uses NVFP4/FP8, not our INT4 grid) and HadaNorm (no public code) under identical weights, INT4 grid and clip, and also reproduced calibrated SVDQuant in our own harness (MJHQ-5k):
On PixArt-Σ, CentriQ is clearly better than both alternatives (FID 19.10 vs 36.57 and 42.57). On FLUX at 2-bit activations, the gap is also large (FID 35.59 vs 42.58 and 43.17). On FLUX at W4A4, all methods are close, and calibrated SVDQuant is slightly ahead on FID, IR and LPIPS. The SVDQuant rows are our reproduction in the same pipeline, not the published numbers. We have added these results to the revised paper (Table 5, in red).
OrbitQuant reports only GenEval and uses a Lloyd–Max codebook rather than an integer grid, so we compare on its own protocol: at W2A4, CentriQ scores 0.647 vs 0.604 for OrbitQuant. In addition, on a 1,000-prompt paired check, an OrbitQuant-style activation path (permuted block-Hadamard on our INT4 grid) reaches PixArt-Σ FID-1k 131.7, vs 47.7 for CentriQ.
W3. Qualifying "calibration-free". CentriQ collects no activation statistics; scales and W·1 are closed-form. The disjoint development split was used once to fix two global settings shared by all models. These settings transfer unchanged to a model we never tuned on (FLUX.1-dev, 500 MJHQ prompts):
CentriQ matches the model-specific calibrated baseline without any data from the new model. (FID on 500 images is higher than FID on 5,000 and is not comparable to the MJHQ-5k numbers above.) We have added this definition of calibration-free to Section 1 (in red).