- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies minority-generation degradation in quantized diffusion models and attributes it to amplified errors in high-noise clean predictions used for minority scoring. It proposes selective activation-bit allocation and quantization-consistent low-rank adaptation trained to match full-precision predictions. Experiments across ADM, LDM, DiT, and Stable Diffusion report generation and score/label recovery with several minority samplers, alongside ablations and deployment measurements.
Critical strengths
-
The paper identifies a useful evaluation blind spot: preserving standard generation does not ensure reliable minority-scoring signals. Intermediate measurements support the observed repair: SGMS score concordance rises from 0.31 to 0.82, and W6A6 DiT score rank correlation with full precision rises from 0.484 to 0.950.
-
The complete repair is tested across several architectures, precisions, and minority samplers. Table 1 reduces W6A6 cFID from 27.89 to 19.46 on CelebA and from 36.84 to 28.33 on LSUN Bedrooms; component ablations and trade-off curves provide additional evidence of empirical utility.
Critical weaknesses
-
The theoretical conditions do not fully cover the experiments: Eq.8-9 analyze DDPM, while DDIM is also tested; Appendix D assumes a globally Lipschitz discrepancy, but LDM SGMS uses squared L2; and the two-stage bound is applied to model-dependent re-corrupted inputs without controlling the distribution shift.
-
The benefit over generic quantization repair is unclear. The Plain-objective control is limited to CelebA, and site selection lacks a matched-bit comparison between clean-prediction and noise-prediction criteria. It is therefore unclear whether the task-specific choices are needed.
Questions for authors
-
Under what assumptions do the guarantees cover the tested DDIM schedules and LDM SGMS? Please address squared L2's Lipschitz condition and the distribution of model-dependent re-corrupted inputs.
-
What benefit does the minority-targeted repair provide over generic quantization recalibration or teacher-matching fine-tuning? Please compare them using the same calibration data, training budget, and extra-bit budget, and explain which minority-specific choices account for any additional gains.
8 comments
Official Comment by Authors (2)
[W2] The benefit over generic quantization repair is unclear. The Plain-objective control is limited to CelebA, and site selection lacks a matched-bit comparison between clean-prediction and noise-prediction criteria. It is therefore unclear whether the task-specific choices are needed.
[Q2] What benefit does the minority-targeted repair provide over generic quantization recalibration or teacher-matching fine-tuning? Please compare them using the same calibration data, training budget, and extra-bit budget, and explain which minority-specific choices account for any additional gains.
We thank the reviewer for these suggestions. We ran all requested comparisons under the same calibration data, training budget and extra-bit budget.
The repair makes two minority-specific choices, both following from the observation in Sec. 4 that the quantization error is amplified in the high-noise clean estimate consumed by minority samplers (Eq. 6). First, the adapter objective weights the error in clean-prediction space (Eq. 13). Second, the site criterion ranks quantizers by their effect on the high-noise clean-prediction error (Eq. 11). The comparisons below replace each choice with its generic counterpart while keeping the rest fixed.
Objective. Tables R11 and R12 extend the component ablation with recalibration and teacher matching (Plain), and are added to Tables 3(a) and 3(b) of the revised PDF, so the Plain control now covers LDM-4 as well.
Table R11. Component ablation on CelebA with TFMQ and SGMS (Table 3(a) in the revised PDF). Recal.: AdaRound re-calibration of the base PTQ with the high-noise-weighted clean-prediction objective of Eq. (13), without adapter or extra bits. Plain: the same adapter trained with uniform noise-prediction distillation ().
| Bits | Grid | Adapter | cFID↓ | sFID↓ | Prec.↑ | Rec.↑ |
|---|---|---|---|---|---|---|
| W6A6 | × | × | 33.39 | 19.35 | 0.758 | 0.514 |
| × | Recal. | 32.21 | 19.02 | 0.762 | 0.525 | |
| ✓ | × | 30.54 | 18.32 | 0.779 | 0.558 | |
| × | Plain | 31.54 | 18.34 | 0.773 | 0.543 | |
| × | ✓ | 29.65 | 18.27 | 0.768 | 0.561 | |
| ✓ | Plain | 27.57 | 18.19 | 0.772 | 0.565 | |
| ✓ | ✓ | 25.69 | 17.97 | 0.780 | 0.577 | |
| W4A8 | × | × | 22.31 | 18.24 | 0.765 | 0.578 |
| × | Recal. | 22.26 | 18.15 | 0.770 | 0.580 | |
| ✓ | × | 22.17 | 17.71 | 0.786 | 0.588 | |
| × | Plain | 21.97 | 17.88 | 0.764 | 0.602 | |
| × | ✓ | 21.91 | 17.66 | 0.790 | 0.579 | |
| ✓ | Plain | 21.67 | 17.62 | 0.794 | 0.578 | |
| ✓ | ✓ | 21.34 | 17.51 | 0.797 | 0.592 |
Table R12. Component ablation on LDM-4 ImageNet with DMQ and SGMS (Table 3(b) in the revised PDF). Recal. and Plain as in Table R11.
| Bits | Grid | Adapter | cFID↓ | sFID↓ | Prec.↑ | Rec.↑ |
|---|---|---|---|---|---|---|
| W4A6 | × | × | 60.47 | 14.84 | 0.932 | 0.161 |
| × | Recal. | 59.60 | 14.30 | 0.938 | 0.162 | |
| ✓ | × | 58.41 | 12.12 | 0.957 | 0.157 | |
| × | Plain | 58.42 | 11.89 | 0.935 | 0.167 | |
| × | ✓ | 57.68 | 11.63 | 0.951 | 0.166 | |
| ✓ | Plain | 56.47 | 11.14 | 0.953 | 0.168 | |
| ✓ | ✓ | 55.14 | 10.57 | 0.956 | 0.172 | |
| W4A8 | × | × | 61.75 | 13.29 | 0.948 | 0.144 |
| × | Recal. | 61.10 | 13.20 | 0.950 | 0.148 | |
| ✓ | × | 60.91 | 13.86 | 0.961 | 0.159 | |
| × | Plain | 60.30 | 13.86 | 0.959 | 0.156 | |
| × | ✓ | 59.75 | 13.19 | 0.955 | 0.163 | |
| ✓ | Plain | 58.59 | 13.07 | 0.958 | 0.166 | |
| ✓ | ✓ | 57.17 | 12.78 | 0.962 | 0.171 |
Recalibration yields the smallest cFID improvement in all four settings. With the same adapter and budget, the clean-prediction objective outperforms Plain teacher matching in cFID in every setting, both with and without grid refinement, on both models.
Official Comment by Authors (3)
Site selection. Tables R13 and R14 compare the two criteria at a matched bit budget, with , the extra bit, the adapter, the loss, and held fixed. "Clean" is the saliency of Eq. (11), and "Noise" is the same first-order saliency on the noise-prediction error over all timesteps. Table R15 lists the selected sites.
Table R13. Site-selection criterion at matched bit budget on CelebA with TFMQ and SGMS, (Table 14 in the revised PDF).
| Bits | Grid | Adapter | Clean (ours) cFID↓ / sFID↓ | Noise cFID↓ / sFID↓ |
|---|---|---|---|---|
| W6A6 | ✓ | × | 30.54 / 18.32 | 32.02 / 18.91 |
| ✓ | Plain | 27.57 / 18.19 | 29.33 / 18.52 | |
| ✓ | ✓ | 25.69 / 17.97 | 27.45 / 18.34 | |
| W4A8 | ✓ | × | 22.17 / 17.71 | 22.26 / 17.97 |
| ✓ | Plain | 21.67 / 17.62 | 21.98 / 17.81 | |
| ✓ | ✓ | 21.34 / 17.51 | 21.65 / 17.75 |
Table R14. Site-selection criterion at matched bit budget on LDM-4 ImageNet with DMQ and SGMS, (Table 14 in the revised PDF).
| Bits | Grid | Adapter | Clean (ours) cFID↓ / sFID↓ | Noise cFID↓ / sFID↓ |
|---|---|---|---|---|
| W4A6 | ✓ | × | 58.41 / 12.12 | 59.62 / 13.45 |
| ✓ | Plain | 56.47 / 11.14 | 57.62 / 12.01 | |
| ✓ | ✓ | 55.14 / 10.57 | 56.69 / 11.52 | |
| W4A8 | ✓ | × | 60.91 / 13.86 | 61.45 / 13.96 |
| ✓ | Plain | 58.59 / 13.07 | 59.44 / 13.13 | |
| ✓ | ✓ | 57.17 / 12.78 | 58.38 / 12.98 |
Table R15. Activation sites selected by each criterion (Table 15 in the revised PDF).
| Model | Rank | Clean (ours) | Noise |
|---|---|---|---|
| ADM CelebA () | 1 | input_blocks.0.0 | input_blocks.0.0 |
| 2 | output_blocks.15.0.skip_connection | output_blocks.11.2.out_layers.3 | |
| 3 | output_blocks.14.0.skip_connection | output_blocks.12.0.in_layers.2 | |
| 4 | output_blocks.13.0.skip_connection | output_blocks.13.0.in_layers.2 | |
| LDM-4 ImageNet () | 1 | input_blocks.0.0 | input_blocks.0.0 |
| 2 | output_blocks.11.0.skip_connection | output_blocks.11.0.out_layers.3 | |
| 3 | output_blocks.11.0.out_layers.3 | output_blocks.9.0.skip_connection | |
| 4 | output_blocks.11.0.in_layers.2 | input_blocks.1.0.out_layers.3 | |
| 5 | output_blocks.10.0.skip_connection | output_blocks.8.1.ff.net.2 | |
| 6 | output_blocks.10.0.out_layers.3 | input_blocks.5.1.ff.net.2 |
The clean-prediction criterion is better in every setting, and the margin is present already with the grid alone (Adapter ×), so it comes from the selected sites rather than the adapter. The two criteria select partly different sites (Table R15), consistent with the analysis that the sites most relevant to the high-noise clean estimate can differ from those ranked highest by the noise-prediction error. These results are added as Appendix J of the revised PDF.
In summary, each minority-specific choice outperforms its generic counterpart under the same budget, so the additional gain comes from measuring the error where the quantized model fails, at high noise and in clean-prediction space.
Official Comment by Authors (1)
We thank Reviewer
1S5ofor the constructive questions on our theoretical analysis and ablations. Below, we provide point-by-point responses to your concerns and questions.DDPM and DDIM. First, we clarify that Eqs. (8)–(9) provide insight into why quantization largely preserves standard sampling despite large errors in the clean estimate at high noise levels. They are not a guarantee of the repair, which is given in Appendix D and does not depend on the reverse sampler.
While Eqs. (8)–(9) are written for the ancestral DDPM update, the same insight carries over to the DDIM update, which we add to the revised Appendix A (Eq. (15)).
For the same input xt, substituting ϵθ=(xt−αtx^0)/1−αt into the deterministic DDIM update gives xt−1=λtDDIMx^0+γtDDIMxt with λtDDIM=αt−1−αt(1−αt−1)/(1−αt) and a denoiser-independent γtDDIM=(1−αt−1)/(1−αt).
Hence, the quantization error again enters only through x^0, with net factor ∣λtDDIM∣(1−αt)/αt:
αt−1(1−αt)/αt−1−αt−1=1−βt(1−αt+1−αt−βt)βt,
where βt=1−αt/αt−1 is the effective step parameter of the possibly respaced transition. This never exceeds the DDPM factor βt/(1−βt1−αt) of Eq. (9), so the bound βt/(1−βt) carries over with no additional assumption. For the 20-step DDIM schedule of LDM-4, the Tweedie amplification in the high-noise regime is at least 3 while the update factor stays below 0.6, so the same separation between the standard update and the minority score holds.
Squared ℓ2. We thank the reviewer for catching this. Since squared ℓ2 is not globally Lipschitz, we keep Proposition 1 unchanged and add a moment-based bound for this case to Appendix D (in blue).
Let u=x^0ϕ−x0, v=x^0θ−x0, and Ds(x0;ϕ)=E[∥x^0ϕ−x^0θ∥2] be the mean squared clean-prediction gap at level s. Factorizing ∥u∥2−∥v∥2=(∥u∥−∥v∥)(∥u∥+∥v∥) and applying the reverse triangle and Cauchy–Schwarz inequalities give (Eq. (26) in the revised PDF)
Lθ^,ϕ(x0;s)−Lθ(x0;s)≤2(Lθ+Lθ^,ϕ)Ds(x0;ϕ)≤6Lθ+4DsDs,
which has the form of Eq. (21) (Eq. (22) in the revised PDF) with the Lipschitz constant replaced by a moment of the scores. The only additional assumption is that the reconstruction residuals have finite second moments under the relevant input distributions, with no bounded-domain assumption on the latents. The same decomposition extends to the two-stage self-guidance score, whose connection to the adapter objective additionally relies on the regularity and distribution-transfer assumptions in Appendix D.
Model-dependent re-corrupted inputs. The revised Appendix D now states explicitly the assumption that the clean-prediction gap at level s, which Eq. (18) (Eq. (19) in the revised PDF) controls under the forward law qαs(xs∣x0), transfers to the re-corrupted inputs x~ϕ of Eq. (24) (Eq. (27) in the revised PDF). High-noise re-corruption makes this assumption mild: with αs≈0.1, differences in the clean estimates enter the re-corrupted input scaled by αs≈0.3, so these inputs stay close to the forward law. Empirically, Fig. 3(b) evaluates the two-stage SGMS score with each model re-corrupting its own clean estimate, and the repair raises concordance with the full-precision score from 0.31 to 0.82 at these model-dependent inputs.