acceptodds
Under review as a conference paper at ICLR 2027

Restoring Minority Generation in Low-Bit Diffusion Models with a Plug-in Repair

Abstract

Efficient deployment of diffusion models has motivated extensive research on low-bit quantization, which can substantially reduce memory and computation while maintaining generation quality under standard sampling. However, we argue that this preservation is not guaranteed to extend to the task of minority generation, which specifically targets generation from low-density but valid regions of the data distribution. We show that quantization error becomes significant in clean estimates at high noise levels, on which existing minority samplers heavily rely yet standard sampling places only a small weight. Based on this analysis, we propose a lightweight plug-in repair that aligns the high-noise clean estimates of quantized models with those of their full-precision counterparts. Our repair combines selective bit allocation with a quantization-consistent low-rank adapter that remains on the deployed quantization grid, and is activated only when computing these estimates, leaving the ordinary sampling path unchanged. Experiments across diverse diffusion architectures, bit widths, and minority samplers show that our repair substantially improves minority generation with negligible overhead while retaining the efficiency of low-bit inference. Code is available here.

Primary area:
generative models
Keywords:
Diffusion Models, Diffusion Model Quantization, Low-Bit Inference, Minority Generation, Minority Sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1S5oOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies minority-generation degradation in quantized diffusion models and attributes it to amplified errors in high-noise clean predictions used for minority scoring. It proposes selective activation-bit allocation and quantization-consistent low-rank adaptation trained to match full-precision predictions. Experiments across ADM, LDM, DiT, and Stable Diffusion report generation and score/label recovery with several minority samplers, alongside ablations and deployment measurements.

Critical strengths
  1. The paper identifies a useful evaluation blind spot: preserving standard generation does not ensure reliable minority-scoring signals. Intermediate measurements support the observed repair: SGMS score concordance rises from 0.31 to 0.82, and W6A6 DiT score rank correlation with full precision rises from 0.484 to 0.950.

  2. The complete repair is tested across several architectures, precisions, and minority samplers. Table 1 reduces W6A6 cFID from 27.89 to 19.46 on CelebA and from 36.84 to 28.33 on LSUN Bedrooms; component ablations and trade-off curves provide additional evidence of empirical utility.

Critical weaknesses
  1. The theoretical conditions do not fully cover the experiments: Eq.8-9 analyze DDPM, while DDIM is also tested; Appendix D assumes a globally Lipschitz discrepancy, but LDM SGMS uses squared L2; and the two-stage bound is applied to model-dependent re-corrupted inputs without controlling the distribution shift.

  2. The benefit over generic quantization repair is unclear. The Plain-objective control is limited to CelebA, and site selection lacks a matched-bit comparison between clean-prediction and noise-prediction criteria. It is therefore unclear whether the task-specific choices are needed.

Questions for authors
  1. Under what assumptions do the guarantees cover the tested DDIM schedules and LDM SGMS? Please address squared L2's Lipschitz condition and the distribution of model-dependent re-corrupted inputs.

  2. What benefit does the minority-targeted repair provide over generic quantization recalibration or teacher-matching fine-tuning? Please compare them using the same calibration data, training budget, and extra-bit budget, and explain which minority-specific choices account for any additional gains.

8 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We thank Reviewer 1S5o for the constructive questions on our theoretical analysis and ablations. Below, we provide point-by-point responses to your concerns and questions.

[W1] The theoretical conditions do not fully cover the experiments: Eq.8-9 analyze DDPM, while DDIM is also tested; Appendix D assumes a globally Lipschitz discrepancy, but LDM SGMS uses squared L2; and the two-stage bound is applied to model-dependent re-corrupted inputs without controlling the distribution shift.

[Q1] Under what assumptions do the guarantees cover the tested DDIM schedules and LDM SGMS? Please address squared L2's Lipschitz condition and the distribution of model-dependent re-corrupted inputs.

DDPM and DDIM. First, we clarify that Eqs. (8)–(9) provide insight into why quantization largely preserves standard sampling despite large errors in the clean estimate at high noise levels. They are not a guarantee of the repair, which is given in Appendix D and does not depend on the reverse sampler.

While Eqs. (8)–(9) are written for the ancestral DDPM update, the same insight carries over to the DDIM update, which we add to the revised Appendix A (Eq. (15)).

For the same input xt\mathbf{x}_t, substituting ϵθ=(xt−αtx^0)/1−αt\boldsymbol\epsilon_\theta=(\mathbf{x}_t-\sqrt{\alpha_t}\hat{\mathbf{x}}_0)/\sqrt{1-\alpha_t} into the deterministic DDIM update gives xt−1=λtDDIMx^0+γtDDIMxt\mathbf{x}_{t-1}=\lambda_t^{\rm DDIM}\hat{\mathbf{x}}_0+\gamma_t^{\rm DDIM}\mathbf{x}_t with λtDDIM=αt−1−αt(1−αt−1)/(1−αt)\lambda_t^{\rm DDIM}=\sqrt{\alpha_{t-1}}-\sqrt{\alpha_t(1-\alpha_{t-1})/(1-\alpha_t)} and a denoiser-independent γtDDIM=(1−αt−1)/(1−αt)\gamma_t^{\rm DDIM}=\sqrt{(1-\alpha_{t-1})/(1-\alpha_t)}.

Hence, the quantization error again enters only through x^0\hat{\mathbf{x}}_0, with net factor ∣λtDDIM∣(1−αt)/αt|\lambda_t^{\rm DDIM}|\sqrt{(1-\alpha_t)/\alpha_t}:

∣αt−1(1−αt)/αt−1−αt−1∣=βt1−βt(1−αt+1−αt−βt),\Big|\sqrt{\alpha_{t-1}(1-\alpha_t)/\alpha_t}-\sqrt{1-\alpha_{t-1}}\Big|=\frac{\beta_t}{\sqrt{1-\beta_t}\big(\sqrt{1-\alpha_t}+\sqrt{1-\alpha_t-\beta_t}\big)},

where βt=1−αt/αt−1\beta_t=1-\alpha_t/\alpha_{t-1} is the effective step parameter of the possibly respaced transition. This never exceeds the DDPM factor βt/(1−βt1−αt)\beta_t/\big(\sqrt{1-\beta_t}\sqrt{1-\alpha_t}\big) of Eq. (9), so the bound βt/(1−βt)\sqrt{\beta_t/(1-\beta_t)} carries over with no additional assumption. For the 20-step DDIM schedule of LDM-4, the Tweedie amplification in the high-noise regime is at least 33 while the update factor stays below 0.60.6, so the same separation between the standard update and the minority score holds.

Squared ℓ2\ell_2. We thank the reviewer for catching this. Since squared ℓ2\ell_2 is not globally Lipschitz, we keep Proposition 1 unchanged and add a moment-based bound for this case to Appendix D (in blue).

Let u=x^0ϕ−x0\mathbf{u}=\hat{\mathbf{x}}_0^{\phi}-\mathbf{x}_0, v=x^0θ−x0\mathbf{v}=\hat{\mathbf{x}}_0^{\theta}-\mathbf{x}_0, and Ds(x0;ϕ)=E[∥x^0ϕ−x^0θ∥2]\mathcal{D}_s(\mathbf{x}_0;\phi)=\mathbb{E}\big[\lVert\hat{\mathbf{x}}_0^{\phi}-\hat{\mathbf{x}}_0^{\theta}\rVert^2\big] be the mean squared clean-prediction gap at level ss. Factorizing ∥u∥2−∥v∥2=(∥u∥−∥v∥)(∥u∥+∥v∥)\lVert\mathbf{u}\rVert^2-\lVert\mathbf{v}\rVert^2=(\lVert\mathbf{u}\rVert-\lVert\mathbf{v}\rVert)(\lVert\mathbf{u}\rVert+\lVert\mathbf{v}\rVert) and applying the reverse triangle and Cauchy–Schwarz inequalities give (Eq. (26) in the revised PDF)

∣Lθ^,ϕ(x0;s)−Lθ(x0;s)∣≤2(Lθ+Lθ^,ϕ)Ds(x0;ϕ)≤6Lθ+4DsDs,\big|\mathcal{L}_{\hat\theta,\phi}(\mathbf{x}_0;s)-\mathcal{L}_{\theta}(\mathbf{x}_0;s)\big| \le \sqrt{2(\mathcal{L}_{\theta}+\mathcal{L}_{\hat\theta,\phi})}\sqrt{\mathcal{D}_s(\mathbf{x}_0;\phi)} \le \sqrt{6\mathcal{L}_{\theta}+4\mathcal{D}_s}\sqrt{\mathcal{D}_s},

which has the form of Eq. (21) (Eq. (22) in the revised PDF) with the Lipschitz constant replaced by a moment of the scores. The only additional assumption is that the reconstruction residuals have finite second moments under the relevant input distributions, with no bounded-domain assumption on the latents. The same decomposition extends to the two-stage self-guidance score, whose connection to the adapter objective additionally relies on the regularity and distribution-transfer assumptions in Appendix D.

Model-dependent re-corrupted inputs. The revised Appendix D now states explicitly the assumption that the clean-prediction gap at level ss, which Eq. (18) (Eq. (19) in the revised PDF) controls under the forward law qαs(xs∣x0)q_{\alpha_s}(\mathbf{x}_s\mid\mathbf{x}_0), transfers to the re-corrupted inputs x~ϕ\tilde{\mathbf{x}}^\phi of Eq. (24) (Eq. (27) in the revised PDF). High-noise re-corruption makes this assumption mild: with αs≈0.1\alpha_s\approx0.1, differences in the clean estimates enter the re-corrupted input scaled by αs≈0.3\sqrt{\alpha_s}\approx0.3, so these inputs stay close to the forward law. Empirically, Fig. 3(b) evaluates the two-stage SGMS score with each model re-corrupting its own clean estimate, and the repair raises concordance with the full-precision score from 0.31 to 0.82 at these model-dependent inputs.

AuthorsOpenReview ↗

Official Comment by Authors (2)

[W2] The benefit over generic quantization repair is unclear. The Plain-objective control is limited to CelebA, and site selection lacks a matched-bit comparison between clean-prediction and noise-prediction criteria. It is therefore unclear whether the task-specific choices are needed.

[Q2] What benefit does the minority-targeted repair provide over generic quantization recalibration or teacher-matching fine-tuning? Please compare them using the same calibration data, training budget, and extra-bit budget, and explain which minority-specific choices account for any additional gains.

We thank the reviewer for these suggestions. We ran all requested comparisons under the same calibration data, training budget and extra-bit budget.

The repair makes two minority-specific choices, both following from the observation in Sec. 4 that the quantization error is amplified in the high-noise clean estimate consumed by minority samplers (Eq. 6). First, the adapter objective weights the error in clean-prediction space (Eq. 13). Second, the site criterion ranks quantizers by their effect on the high-noise clean-prediction error (Eq. 11). The comparisons below replace each choice with its generic counterpart while keeping the rest fixed.

Objective. Tables R11 and R12 extend the component ablation with recalibration and teacher matching (Plain), and are added to Tables 3(a) and 3(b) of the revised PDF, so the Plain control now covers LDM-4 as well.

Table R11. Component ablation on CelebA with TFMQ and SGMS (Table 3(a) in the revised PDF). Recal.: AdaRound re-calibration of the base PTQ with the high-noise-weighted clean-prediction objective of Eq. (13), without adapter or extra bits. Plain: the same adapter trained with uniform noise-prediction distillation (wt≡1w_t\equiv1).

BitsGridAdaptercFID↓sFID↓Prec.↑Rec.↑
W6A6××33.3919.350.7580.514
×Recal.32.2119.020.7620.525
✓×30.5418.320.7790.558
×Plain31.5418.340.7730.543
×✓29.6518.270.7680.561
✓Plain27.5718.190.7720.565
✓✓25.6917.970.7800.577
W4A8××22.3118.240.7650.578
×Recal.22.2618.150.7700.580
✓×22.1717.710.7860.588
×Plain21.9717.880.7640.602
×✓21.9117.660.7900.579
✓Plain21.6717.620.7940.578
✓✓21.3417.510.7970.592

Table R12. Component ablation on LDM-4 ImageNet with DMQ and SGMS (Table 3(b) in the revised PDF). Recal. and Plain as in Table R11.

BitsGridAdaptercFID↓sFID↓Prec.↑Rec.↑
W4A6××60.4714.840.9320.161
×Recal.59.6014.300.9380.162
✓×58.4112.120.9570.157
×Plain58.4211.890.9350.167
×✓57.6811.630.9510.166
✓Plain56.4711.140.9530.168
✓✓55.1410.570.9560.172
W4A8××61.7513.290.9480.144
×Recal.61.1013.200.9500.148
✓×60.9113.860.9610.159
×Plain60.3013.860.9590.156
×✓59.7513.190.9550.163
✓Plain58.5913.070.9580.166
✓✓57.1712.780.9620.171

Recalibration yields the smallest cFID improvement in all four settings. With the same adapter and budget, the clean-prediction objective outperforms Plain teacher matching in cFID in every setting, both with and without grid refinement, on both models.

AuthorsOpenReview ↗

Official Comment by Authors (3)

Site selection. Tables R13 and R14 compare the two criteria at a matched bit budget, with KK, the extra bit, the adapter, the loss, and wmgw_{\rm mg} held fixed. "Clean" is the saliency of Eq. (11), and "Noise" is the same first-order saliency on the noise-prediction error over all timesteps. Table R15 lists the selected sites.

Table R13. Site-selection criterion at matched bit budget on CelebA with TFMQ and SGMS, K=4K=4 (Table 14 in the revised PDF).

BitsGridAdapterClean (ours) cFID↓ / sFID↓Noise cFID↓ / sFID↓
W6A6✓×30.54 / 18.3232.02 / 18.91
✓Plain27.57 / 18.1929.33 / 18.52
✓✓25.69 / 17.9727.45 / 18.34
W4A8✓×22.17 / 17.7122.26 / 17.97
✓Plain21.67 / 17.6221.98 / 17.81
✓✓21.34 / 17.5121.65 / 17.75

Table R14. Site-selection criterion at matched bit budget on LDM-4 ImageNet with DMQ and SGMS, K=6K=6 (Table 14 in the revised PDF).

BitsGridAdapterClean (ours) cFID↓ / sFID↓Noise cFID↓ / sFID↓
W4A6✓×58.41 / 12.1259.62 / 13.45
✓Plain56.47 / 11.1457.62 / 12.01
✓✓55.14 / 10.5756.69 / 11.52
W4A8✓×60.91 / 13.8661.45 / 13.96
✓Plain58.59 / 13.0759.44 / 13.13
✓✓57.17 / 12.7858.38 / 12.98

Table R15. Activation sites selected by each criterion (Table 15 in the revised PDF).

ModelRankClean (ours)Noise
ADM CelebA (K=4K=4)1input_blocks.0.0input_blocks.0.0
2output_blocks.15.0.skip_connectionoutput_blocks.11.2.out_layers.3
3output_blocks.14.0.skip_connectionoutput_blocks.12.0.in_layers.2
4output_blocks.13.0.skip_connectionoutput_blocks.13.0.in_layers.2
LDM-4 ImageNet (K=6K=6)1input_blocks.0.0input_blocks.0.0
2output_blocks.11.0.skip_connectionoutput_blocks.11.0.out_layers.3
3output_blocks.11.0.out_layers.3output_blocks.9.0.skip_connection
4output_blocks.11.0.in_layers.2input_blocks.1.0.out_layers.3
5output_blocks.10.0.skip_connectionoutput_blocks.8.1.ff.net.2
6output_blocks.10.0.out_layers.3input_blocks.5.1.ff.net.2

The clean-prediction criterion is better in every setting, and the margin is present already with the grid alone (Adapter ×), so it comes from the selected sites rather than the adapter. The two criteria select partly different sites (Table R15), consistent with the analysis that the sites most relevant to the high-noise clean estimate can differ from those ranked highest by the noise-prediction error. These results are added as Appendix J of the revised PDF.

In summary, each minority-specific choice outperforms its generic counterpart under the same budget, so the additional gain comes from measuring the error where the quantized model fails, at high noise and in clean-prediction space.

Reviewer_CMauOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Minority samplers (SGMS, MG) score from a Tweedie clean estimate formed at high noise, and the Tweedie map multiplies the denoiser's quantization error by (1−αt)/αt≥3\sqrt{(1-\alpha_t)/\alpha_t} \ge 3; standard sampling is immune because the reverse-update mean weights by λt\lambda_t, leaving a net factor bounded by βt/(1−βt)\sqrt{\beta_t/(1-\beta_t)} (Eqs. 6 and 9 — I checked the algebra, Fig. 6 confirms). The repair gives one extra bit to the top-KK (3–10) activation quantizers ranked by a one-pass first-order saliency, and attaches a grid-confined low-rank adapter plus a per-channel affine that folds into scale and zero point, distilled under wt=min⁡((1−αt)/αt,wmax)w_t = \min((1-\alpha_t)/\alpha_t, w_{max}). Validated on ADM, LDM-4, DiT-XL/2 and SD v1.4 at two bit widths with SGMS, MG and the guidance-free BnS, plus a Triton int8 runtime on an A6000 and a Jetson Orin Nano.

Critical strengths
  1. The diagnosis is correct, quantitative and portable. Eq. (6)'s amplification against Eq. (9)'s attenuation explains at the coefficient level why standard sampling survives quantization while minority sampling collapses; I verified the algebra. It transfers to samplers the authors never ran, and it is the main reason I land on accept.

  2. Deployment-compatible by construction, and actually built. The grid-confined update and the affine folded into scale and zero point avoid breaking low-bit inference, and Appendix C is a real int8 runtime — Triton kernels, tensor-core GEMM with epilogue dequantization, CUDA graphs — measured on two devices. Rare for a PTQ paper, and it is what makes the latency column mean anything.

Critical weaknesses
  1. No recalibration-only baseline, so the machinery's contribution is unmeasured. Base PTQ is calibrated on standard trajectories with noise-prediction-space reconstruction. Once the failure is localized to the clean estimate at high noise, the cheap fix is to recalibrate with a high-noise-weighted Tweedie objective, or calibrate only on t∈Thnt \in T_{hn}. The one control (Table 3, "Plain", wt≡1w_t \equiv 1) still carries the full adapter and the extra bits, so it isolates the weighting, not the machinery. Without that row I cannot tell whether the saliency scan, the LoRA and the bit refinement earn their complexity.

  2. No standard-sampling metrics anywhere for the repaired model. LadaptL_{adapt} distills the FP noise prediction over all timesteps with only wmax=10w_{max} = 10 damping the most amplified ones, so the adapter is substantially a generic quantization-error compensator. One table of FID/sFID (CLIPScore for SD) for FP / quant / quant+Ours with the repair enabled throughout would settle whether the gain is minority-specific. Cheap, absent, and until it exists I cannot rule out a stronger general PTQ method wearing a minority framing.

Questions for authors
  1. (Clarification) Add a row recalibrating the base PTQ with a high-noise-weighted Tweedie objective, no adapter, no extra bits. How much of the gap does it close alone?
  2. (Clarification) Report standard-sampling FID/sFID (CLIPScore for SD) for FP / quant / quant+Ours with the repair on throughout. If standard generation also improves, the framing needs to change.
  3. (Clarification) Standard deviations over at least three seeds, or bootstrap intervals, for Tables 1, 2 and 12. Several headline deltas sit at noise level: DiT W8A8 cFID 37.72 → 37.24; SD W4A8 Pick 21.284 → 21.303, LL 0.7883 → 0.7876, IR 0.260 → 0.263.
  4. (Clarification) Report precision, recall, density and coverage plus a per-sample fidelity proxy. Precision recovers about 8% of the gap on ADM CelebA W6A6 (0.851 → 0.775 → 0.781) and degrades on LDM W4A6 (0.966 → 0.960 → 0.955), while cFID fully recovers — so restoration cannot rest on cFID.
  5. (Clarification) In the A8→A9 configurations, what fraction of GEMM FLOPs falls back to fp16-operand mode, and what is the latency at batch size > 1? Table 4 is batch 1 and launch-bound.
8 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We thank Reviewer CMau for the thoughtful comments and the precise control experiments requested. Below we provide point-by-point responses.

[W1] No recalibration-only baseline, so the machinery's contribution is unmeasured. [...] The one control (Table 3, "Plain", wt≡1w_t\equiv1) still carries the full adapter and the extra bits, so it isolates the weighting, not the machinery. Without that row I cannot tell whether the saliency scan, the LoRA and the bit refinement earn their complexity.

[Q1] Add a row recalibrating the base PTQ with a high-noise-weighted Tweedie objective, no adapter, no extra bits. How much of the gap does it close alone?

We first state why the correction is a separate plug-in rather than a change to the quantized weights, since this is the design choice the reviewer questions.

Quantized diffusion models are already on par with full precision on standard-sampling metrics such as FID (Sec. 3, Table R5). This is the result of careful calibration: PTQ is sensitive to how each weight is rounded, and PTQ methods optimize the rounding to minimize the reconstruction error against the full-precision model [1, 2, 3], on states collected along the standard sampling trajectory. Re-optimizing these weights for a different objective could disturb this calibrated behavior. Since standard generation is what the quantized model already does well and minority generation is where it degrades (Sec. 4), we keep the base weights exactly as calibrated and solve the problem on the sampling side instead. For score-based samplers every reverse update runs on the stock quantized model, and the repaired weights of Eq. (12) form a second set that is used only when computing the minority score. This is why the correction takes the form of an adapter: it can be trained and attached without touching the deployed model.

That said, the control the reviewer proposes is a fair test of whether this machinery is needed, and we ran it exactly as described. The "Recal." row (Grid ×) in Tables R3 and R4 re-optimizes the weight rounding of the deployed PTQ checkpoint with the high-noise-weighted clean-prediction objective of Eq. (13), using the same calibration data and training budget as our adapter, with no adapter and no extra bits. The recalibrated model is then used for the entire sampling process. These rows are added to Table 3(a) and 3(b) of the revised PDF.

Table R3. Component ablation on CelebA with TFMQ and SGMS (Table 3(a) in the revised PDF). Recal.: AdaRound re-calibration of the base PTQ with the high-noise-weighted clean-prediction objective of Eq. (13), without adapter or extra bits. Plain: the same adapter trained with uniform noise-prediction distillation (wt≡1w_t\equiv1).

BitsGridAdaptercFID↓sFID↓Prec.↑Rec.↑
W6A6××33.3919.350.7580.514
×Recal.32.2119.020.7620.525
✓×30.5418.320.7790.558
×Plain31.5418.340.7730.543
×✓29.6518.270.7680.561
✓Plain27.5718.190.7720.565
✓✓25.6917.970.7800.577
W4A8××22.3118.240.7650.578
×Recal.22.2618.150.7700.580
✓×22.1717.710.7860.588
×Plain21.9717.880.7640.602
×✓21.9117.660.7900.579
✓Plain21.6717.620.7940.578
✓✓21.3417.510.7970.592

Table R4. Component ablation on LDM-4 ImageNet with DMQ and SGMS (Table 3(b) in the revised PDF). Recal. and Plain as in Table R3.

BitsGridAdaptercFID↓sFID↓Prec.↑Rec.↑
W4A6××60.4714.840.9320.161
×Recal.59.6014.300.9380.162
✓×58.4112.120.9570.157
×Plain58.4211.890.9350.167
×✓57.6811.630.9510.166
✓Plain56.4711.140.9530.168
✓✓55.1410.570.9560.172
W4A8××61.7513.290.9480.144
×Recal.61.1013.200.9500.148
✓×60.9113.860.9610.159
×Plain60.3013.860.9590.156
×✓59.7513.190.9550.163
✓Plain58.5913.070.9580.166
✓✓57.1712.780.9620.171

Recalibration alone yields only a small reduction in cFID. In all four settings, both the adapter alone (Grid ×, Adapter ✓) and grid refinement alone (Grid ✓, Adapter ×) achieve lower cFID than recalibration. Combining them achieves the lowest cFID in every setting.

Unlike recalibration, which must serve the minority score and every reverse update with a single weight set, the repair keeps the stock weights for the reverse updates and applies the correction only when the minority score is estimated.

AuthorsOpenReview ↗

Official Comment by Authors (2)

[W2] No standard-sampling metrics anywhere for the repaired model. Ladapt\mathcal{L}_{\rm adapt} distills the FP noise prediction over all timesteps with only wmax⁡=10w_{\max}=10 damping the most amplified ones, so the adapter is substantially a generic quantization-error compensator. One table of FID/sFID (CLIPScore for SD) for FP / quant / quant+Ours with the repair enabled throughout would settle whether the gain is minority-specific. Cheap, absent, and until it exists I cannot rule out a stronger general PTQ method wearing a minority framing.

[Q2] Report standard-sampling FID/sFID (CLIPScore for SD) for FP / quant / quant+Ours with the repair on throughout. If standard generation also improves, the framing needs to change.

Our goal is not to improve the quantized model in general, but to restore the one quantity it predicts poorly: the clean estimate at high noise. The weight in Ladapt\mathcal{L}_{\rm adapt} is the squared amplification factor of Eq. (6), so the objective is the clean-prediction error of Eq. (10) and concentrates on high-noise timesteps rather than compensating the quantization error uniformly.

We ran the requested experiment. Table R5 reports standard sampling with the repair enabled throughout, without any minority sampler, on ADM and LDM-4.

Table R5. Standard sampling with the repair enabled throughout, without any minority sampler, evaluated against the full training distribution (Table 18 in the revised PDF).

ModelBitsMethodcFID↓sFID↓Prec.↑Rec.↑
ADM CelebAFPFull Precision35.1426.360.8060.228
W6A6TFMQ29.9126.380.7640.316
TFMQ + Ours36.0530.840.7950.229
W4A8TFMQ31.4825.350.7980.257
TFMQ + Ours33.9026.930.8050.247
LDM-4 ImageNetFPFull Precision13.888.980.9460.512
W4A6DMQ11.326.680.9160.581
DMQ + Ours10.888.550.9090.582
W4A8DMQ12.527.370.9330.552
DMQ + Ours12.109.320.9320.547

Standard generation does not consistently improve. The stock quantized models are already on par with full precision, and the repair mainly moves them back toward it, e.g., to the full-precision cFID, precision, and recall on CelebA. This is expected, since standard generation is not where the quantized model fails, while the gains in Tables 1 and 2 appear where the high-noise clean estimate is used for minority generation. These results and further discussion are added as Appendix M of the revised PDF.

AuthorsOpenReview ↗

Official Comment by Authors (3)

[Q3] Standard deviations over at least three seeds, or bootstrap intervals, for Tables 1, 2 and 12. Several headline deltas sit at noise level: DiT W8A8 cFID 37.72 → 37.24; SD W4A8 Pick 21.284 → 21.303, LL 0.7883 → 0.7876, IR 0.260 → 0.263.

As per your suggestion, we have added bootstrap 95% confidence intervals for the DiT rows of Table 1 and for Table 2, which contain all the deltas listed above (Tables R6 and R7, added as Appendix K of the revised PDF). The intervals are computed with B=500B=500 bootstrap resamples of the 10,000 generated samples, with the reference set held fixed.

Table R6. MG on DiT-XL/2 (ImageNet 256×256) with bootstrap 95% confidence intervals (Table 16 in the revised PDF).

BitsMethodcFID↓sFID↓Prec.↑Rec.↑
FPFull Precision + MG37.26 ± 0.6213.72 ± 0.150.902 ± 0.0050.302 ± 0.008
W6A6TQ-DiT + MG42.04 ± 0.5315.07 ± 0.150.858 ± 0.0060.255 ± 0.007
TQ-DiT + MG + Ours38.38 ± 0.5414.15 ± 0.150.872 ± 0.0060.317 ± 0.007
W8A8TQ-DiT + MG37.72 ± 0.6413.90 ± 0.150.887 ± 0.0050.324 ± 0.008
TQ-DiT + MG + Ours37.24 ± 0.6213.65 ± 0.150.896 ± 0.0050.335 ± 0.008

Table R7. BnS on Stable Diffusion v1.4 (MS-COCO) with bootstrap 95% confidence intervals (Table 17 in the revised PDF).

BitsMethodCLIP↑Pick↑IR↑Prec.↑Rec.↑Den.↑Cov.↑LL↓
FPFull Precision + BnS0.3143 ± 0.000621.310 ± 0.0110.261 ± 0.0090.676 ± 0.0090.711 ± 0.0080.734 ± 0.0280.1625 ± 0.0080.7636 ± 0.0015
W4A6DMQ + BnS0.3113 ± 0.000620.982 ± 0.0120.085 ± 0.0090.610 ± 0.0090.728 ± 0.0080.643 ± 0.0280.1369 ± 0.0080.7285 ± 0.0015
DMQ + Ours + BnS0.3139 ± 0.000621.032 ± 0.0110.178 ± 0.0090.698 ± 0.0090.701 ± 0.0080.760 ± 0.0280.1439 ± 0.0080.7195 ± 0.0015
W4A8DMQ + BnS0.3138 ± 0.000621.284 ± 0.0110.260 ± 0.0090.669 ± 0.0090.587 ± 0.0080.759 ± 0.0280.1519 ± 0.0080.7883 ± 0.0015
DMQ + Ours + BnS0.3148 ± 0.000621.303 ± 0.0110.263 ± 0.0090.682 ± 0.0090.601 ± 0.0080.767 ± 0.0270.1534 ± 0.0080.7876 ± 0.0015

At the lower activation bit width, where the degradation is large, the gains exceed the intervals on cFID, sFID, precision and recall for DiT at W6A6, and on CLIPScore, PickScore, ImageReward, precision, density and log-likelihood for Stable Diffusion at W4A6. At 8-bit activations the stock quantized model is already close to full precision, so the remaining gap is small and the differences there are correspondingly small. The repaired rows still improve on every metric and fall within the full-precision intervals on cFID, sFID, PickScore and ImageReward.

Reviewer_73bgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that low-bit diffusion models can preserve ordinary sampling yet degrade generation of rare valid samples, because high-noise clean estimates amplify quantization error. Its repair combines selective activation precision increases with a quantization-consistent low-rank correction trained using capped, clean-prediction-weighted distillation. It targets the minority-signal path, or the full trajectory for BnS. Experiments cover multiple architectures, quantizers, and samplers.

Critical strengths
  1. An important evaluation blind spot with a clear explanation. Equations (6) and (9) distinguish clean-estimate error from reverse-update error. Reconstruction and score-fidelity diagnostics connect this analysis to minority-sampling failures, providing an insight beyond the particular repair.

  2. A targeted repair with broad supporting evidence. Component and uniform-weighting ablations support the design; broader evaluations span ADM, LDM, DiT, and Stable Diffusion. Appendix K includes quality–rarity/precision–recall sweeps, and measured latency remains close to the quantized baseline.

Critical weaknesses
  1. Low latency overhead does not capture all deployment costs. Repair requires per-model/bit-configuration training; the timed score-branch implementation holds base and repaired weights, increasing A6000 memory from 1.43 to 1.76 GB. Appendix C also uses INT8 products for W4A8 and FP16 operands at selected refined activation sites. These disclosed distinctions should be more prominent.

  2. The score bound does not directly cover one evaluated discrepancy. Appendix D assumes a globally Lipschitz discrepancy and proves the L2-norm case, but Appendix G.4 uses squared L2 for LDM SGMS. Squared L2 lacks that global property on unbounded latents; this case needs bounded-domain assumptions or a moment-based extension. The appendix already acknowledges that score fidelity does not guarantee guidance-gradient fidelity. This appears repairable without overturning the empirical results.

Questions for authors
  1. What is the repair-training wall-clock cost per configuration? Please summarize the disclosed storage/runtime-precision and resident-copy distinctions alongside Table 4.

  2. How does the Appendix D bound extend to the squared-L2 discrepancy actually used for LDM SGMS, and what additional assumptions are needed?

  3. Please label the BnS results as boost-only in the main tables, consistent with Appendix G.4–G.5, and distinguish this evidence from validation of the full boost-and-skip method.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We thank Reviewer 73bg for the constructive feedback and the positive assessment of our analysis and experiments. Below, we provide point-by-point responses to your concerns and questions.

[W1] Low latency overhead does not capture all deployment costs. Repair requires per-model/bit-configuration training; the timed score-branch implementation holds base and repaired weights, increasing A6000 memory from 1.43 to 1.76 GB. Appendix C also uses INT8 products for W4A8 and FP16 operands at selected refined activation sites. These disclosed distinctions should be more prominent.

[Q1] What is the repair-training wall-clock cost per configuration? Please summarize the disclosed storage/runtime-precision and resident-copy distinctions alongside Table 4.

We thank the reviewer for this helpful point. Appendix G.3 (Table 9 in the revised PDF) already reports the iterations, batch size, and calibration set size, and we now add the wall-clock training time below and in the revision.

Training cost. As reported in Appendix G.3, the repair is trained once per model and bit configuration (5,000 iterations, effective batch size 16, single RTX A6000). We now add the wall-clock times:

Table R1. Wall-clock cost on a single RTX A6000.

ModelBitsBaseline PTQPTQ calibration wall-clockRepair training: NcalN_{\rm cal} / iterationsRepair training wall-clock (Ours)
ADM CelebA 64×64W6A6 / W4A8TFMQ15 h 36 m10,000 / 5,0001 h 57 m
LDM-4 ImageNet 256×256W4A6 / W4A8DMQ29 h 20 m256 / 5,0002 h 16 m

PTQ calibration is the cost of the baseline quantization itself, shown for reference. Our repair adds a one-time cost of about two hours, roughly one tenth of that, on the same GPU. We agree that inference cost is not the whole picture, but the conclusion of Table 4 (Table 4(a) in the revised PDF), that the deployment latency overhead is within timing noise, still holds.

Peak memory and runtime precision. As described in Appendices C and C.1, the W4A8 weights are 4-bit AdaRound codes stored in an INT8 container and executed with INT8 multiplies and INT32 accumulation. The KK activation sites refined from A8 to A9 do not fit INT8 tensor cores, so the same GEMM kernel runs with FP16 operands for those layers only. The peak memory increase in Table 4 arises mainly because SGMS uses the base weights for generation and the repaired weights for the score branch, keeping one additional low-bit weight set resident.

Since Table 4 is measured at batch size 1, we additionally report larger batch sizes in Table R2 (new Appendix C.2, Table 6 in the revised PDF). The repaired model stays within about 1% of the stock quantized model in latency at every batch size, and its memory overhead decreases in relative terms from 23% to 6% as the batch size grows. Checkpoint storage remains small in factor form (Size column of Table 1; Table 11 (Table 12 in the revised PDF)).

Table R2. Sampling latency and peak memory on LDM-4 ImageNet with SGMS and DMQ on an RTX A6000 at different batch sizes. The batch-1 row is that of Table 4.

BatchMethodTime (s/image)SpeedupPeak Mem.
1Full precision1.2901.00×2.45 GB
W4A80.4862.65×1.43 GB
W4A8 + Ours0.4872.65×1.76 GB
2Full precision1.0671.00×3.93 GB
W4A80.3592.97×2.38 GB
W4A8 + Ours0.3612.95×2.75 GB
4Full precision0.9081.00×4.74 GB
W4A80.3072.95×3.15 GB
W4A8 + Ours0.3102.93×3.63 GB
8Full precision0.8181.00×6.10 GB
W4A80.2713.02×4.67 GB
W4A8 + Ours0.2733.00×5.23 GB
16Full precision0.7731.00×9.14 GB
W4A80.2553.03×7.72 GB
W4A8 + Ours0.2583.00×8.40 GB
32Full precision0.7501.00×15.25 GB
W4A80.2463.05×13.82 GB
W4A8 + Ours0.2493.01×14.62 GB

We agree that Table 4 alone does not show the storage format, runtime precision, and resident weight copy, which were disclosed only in the appendix. In the revised PDF, Section 6.2 now includes a paragraph "Deployment cost beyond latency" summarizing them with pointers to the appendices, and Table R1 is added next to the latency results as Table 4(b). Both additions are marked in blue.

AuthorsOpenReview ↗

Official Comment by Authors (2)

[W2] The score bound does not directly cover one evaluated discrepancy. Appendix D assumes a globally Lipschitz discrepancy and proves the L2-norm case, but Appendix G.4 uses squared L2 for LDM SGMS.

[Q2] How does the Appendix D bound extend to the squared-L2 discrepancy actually used for LDM SGMS, and what additional assumptions are needed?

Since squared ℓ2\ell_2 is not globally Lipschitz, we keep Proposition 1 unchanged and add a moment-based bound for this case to Appendix D (in blue). Let u=x^0ϕ−x0\mathbf{u}=\hat{\mathbf{x}}_0^{\phi}-\mathbf{x}_0, v=x^0θ−x0\mathbf{v}=\hat{\mathbf{x}}_0^{\theta}-\mathbf{x}_0, and Ds(x0;ϕ)=E[∥x^0ϕ−x^0θ∥2]\mathcal{D}_s(\mathbf{x}_0;\phi)=\mathbb{E}\big[\lVert \hat{\mathbf{x}}_0^{\phi}-\hat{\mathbf{x}}_0^{\theta}\rVert^2\big] be the mean squared clean-prediction gap at level ss. Factorizing ∥u∥2−∥v∥2=(∥u∥−∥v∥)(∥u∥+∥v∥)\lVert\mathbf{u}\rVert^2-\lVert\mathbf{v}\rVert^2=(\lVert\mathbf{u}\rVert-\lVert\mathbf{v}\rVert)(\lVert\mathbf{u}\rVert+\lVert\mathbf{v}\rVert) and applying the reverse triangle and Cauchy–Schwarz inequalities give (Eq. (26) in the revised PDF)

∣Lθ^,ϕ(x0;s)−Lθ(x0;s)∣≤2(Lθ+Lθ^,ϕ)Ds(x0;ϕ)≤6Lθ+4DsDs.\big|\mathcal{L}_{\hat\theta,\phi}(\mathbf{x}_0;s)-\mathcal{L}_{\theta}(\mathbf{x}_0;s)\big| \le \sqrt{2(\mathcal{L}_{\theta}+\mathcal{L}_{\hat\theta,\phi})}\sqrt{\mathcal{D}_s(\mathbf{x}_0;\phi)} \le \sqrt{6\mathcal{L}_{\theta}+4\mathcal{D}_s}\sqrt{\mathcal{D}_s}.

This has the form of Eq. (21) (Eq. (22) in the revised PDF), with the Lipschitz constant replaced by a moment of the scores. The only additional assumption is that the reconstruction residuals have finite second moments under the relevant input distributions. No bounded-domain assumption on the latents is needed, and no decoder factor enters since LDM-4 SGMS evaluates dd directly on latents. The same decomposition extends to the two-stage self-guidance score, whose connection to the adapter objective additionally relies on the regularity and distribution-transfer assumptions in Appendix D.

[Q3] Please label the BnS results as boost-only in the main tables, consistent with Appendix G.4–G.5, and distinguish this evidence from validation of the full boost-and-skip method.

We thank the reviewer for raising this, as our wording in Appendix G.4 ("we do not use the timestep skipping") can be read as departing from the method. We clarify that timestep skipping in Boost-and-Skip [1] is a hyperparameter with an explicit selection rule (Appendix C of [1]), and our setting follows that rule. In [1], the skip length Δt\Delta_t is chosen so that the signal coefficient at the first denoising step, α(Tskip)\alpha(T_{\mathrm{skip}}) with Tskip=T−ΔtT_{\mathrm{skip}}=T-\Delta_t, is not negligible (α(Tskip)>0.005\alpha(T_{\mathrm{skip}})>0.005 for high-resolution data). When α(T)\alpha(T) already satisfies this, Δt=0\Delta_t=0 is the prescribed choice, as used in [1] for LSUN-Bedrooms with (γ2,Δt)=(7.5,0)(\gamma^2,\Delta_t)=(7.5,0). In our setups, α(T)=αˉT≈0.0064\alpha(T)=\sqrt{\bar\alpha_T}\approx0.0064 for DiT and ≈0.068\approx0.068 for Stable Diffusion, both above this threshold (our αt\alpha_t denotes αˉt\bar\alpha_t, whereas α(t)\alpha(t) in [1] denotes αˉt\sqrt{\bar\alpha_t}). Our results therefore correspond to Boost-and-Skip with Δt=0\Delta_t=0 selected by its own rule.

For Stable Diffusion, we further use a deterministic DDIM sampler with a small γ\gamma, which adapts Boost-and-Skip to deterministic sampling (Appendix G.5). This choice is consistent with the analysis in [1]: without the contraction of stochastic sampling, the boost is transmitted to the output without attenuation, so a small γ\gamma suffices while a large one leaves the data manifold.

In the revised PDF, we state this in the captions of Table 2 and Table 12 (Table 13 in the revised PDF), add the selection rule and the resulting α(T)\alpha(T) to Appendix G.4, and revise the related wording in Appendices G.4 and G.5 accordingly.

[1] Um, Kim, and Ye. Boost-and-Skip: A Simple Guidance-Free Diffusion for Minority Generation. ICML 2025.

Dear Reviewer 73bg,

Thank you again for the constructive review. As the discussion period is coming to an end, we would like to ask whether our responses have addressed your concerns. If so, we would appreciate it if you could consider updating your assessment. Otherwise, we would be happy to clarify any remaining points.

Best regards,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their careful reading and valuable feedback.

We appreciate that the reviewers found the paper to identify an important evaluation blind spot in quantized diffusion models (73bg, 1S5o), to give a clear and quantitative diagnosis of why minority generation degrades while standard sampling is preserved (73bg, CMau), to propose a deployment-compatible repair backed by a real int8 runtime (CMau), and to provide broad empirical evidence across architectures, bit widths, and minority samplers (73bg, CMau, 1S5o).

The main changes in the revised PDF are listed below and marked in blue. Detailed responses are given to each reviewer individually.

Summary of Key Revisions

  • Recalibration-only and teacher-matching controls, with the ablation extended to LDM-4 (Table 3).
  • Site-selection criterion compared at a matched bit budget (Appendix J).
  • Standard sampling with the repair enabled, bootstrap confidence intervals, and additional fidelity and diversity metrics (Appendices K, M, and N).
  • Deployment cost beyond latency, including training time and larger batch sizes (Section 6.2, Table 4(b), Appendix C.2).
  • Extended analysis for DDIM, the squared ℓ2\ell_2 discrepancy, and re-corrupted inputs (Appendices A and D).
  • Clarified Boost-and-Skip setting (Tables 2 and 13, Appendices G.4 and G.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.