acceptodds
Under review as a conference paper at ICLR 2027

Score Bridge Distillation: Student-Adaptive Targets for One-Step Diffusion Models

Abstract

Distribution matching distillation accelerates diffusion model sampling by training a few-step or one-step student with the score difference between the teacher and student distributions, without requiring sample-wise correspondence between their generated outputs. However, the score differences evaluated at different noise levels can induce conflicting update directions for the student, causing them to partially cancel and thereby slowing convergence and limiting the final generation quality. Existing progressive distillation methods rely on a sequence of interme- diate distributions fixed before distillation; as the student evolves, this predefined path can become increasingly misaligned with its current distribution. To address this limitation, we propose Score Bridge Distillation (SBD), which periodically reconstructs teacher-guided bridge targets from the current student score estimate, thereby recalibrating the intermediate target as the student evolves while contin- uously guiding it toward the teacher distribution. SBD alternates between two stages: Bridge Score Adaptation and Bridge Distribution Matching. Bridge Score Adaptation constructs a bridge by adapting the current student score toward the teacher. Bridge Distribution Matching then trains the student to approximate this distribution. As the student distribution evolves, SBD repeatedly reconstructs the bridge distribution so that the training target at each stage remains adapted to the student’s current state. This progressive matching process reduces gradient conflicts across noise levels, enabling the student to approach the teacher distribution more effectively. Extensive experiments show that SBD achieves faster convergence and better generation quality than existing distillation methods.

Primary area:
generative models
Keywords:
distribution matching distillation, diffusion distillation, score matching, one-step generation, diffusion models, generative models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_FBarOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Summary

This paper proposes Score Bridge Distillation (SBD), a method for one-step diffusion distillation based on distribution matching. Motivated by conflicting student-update gradients across noise levels, SBD periodically constructs a teacher-guided bridge target from the current student score estimate through teacher alignment and real-data denoising. The student is then trained to match the resulting intermediate distribution. The authors analyze cross-noise gradient conflicts and evaluate SBD on unconditional image generation, class-conditional ImageNet generation, and text-to-image generation with SDXL, reporting reduced gradient interference and improved generation quality.

Critical strengths

Strengths

  • The core idea of adapting the matching target to the student’s current distribution is conceptually simple and easy to follow.
  • The method achieves substantial improvements on several benchmarks. The evaluation covers class-conditional generation, unconditional generation, and high-resolution text-to-image generation with SDXL, providing broad empirical support for its effectiveness.
Critical weaknesses

Weaknesses

  • Section 6.3 would benefit from deeper analysis of the dataset-dependent ablation results. For example, removing teacher alignment has a much larger impact on FFHQ than on CIFAR-10, yet this difference is not sufficiently explained.
  • Although the compute-matched ablation is informative, SBD’s additional runtime and memory overhead remain unquantified. Reporting training GPU-hours and peak GPU memory relative to the baseline would clarify the practical efficiency trade-off.
Questions for authors

Questions

  • Table 4 shows that combining teacher alignment and real-data denoising yields the best results. Since real-data denoising alone can, in principle, learn the data score, could the authors explain the additional benefit of teacher alignment?
  • The component gains in Table 4 vary substantially across datasets. For example, teacher alignment provides a much larger improvement on FFHQ than on CIFAR-10. Could the authors report comparable component ablations on ImageNet and SDXL to assess whether these findings generalize?
4 comments
AuthorsOpenReview ↗

W1 and Q2. Component ablations across datasets

Thank you for the suggestion. We added component ablations on ImageNet and SDXL to examine how the two supervision terms contribute across different tasks.

Bridge adaptation objectiveCIFAR-10 FID ↓FFHQ FID ↓ImageNet FID ↓SDXL FID ↓
Real-data denoising only3.3610.551.7624.58
Teacher alignment only3.547.571.3119.27
Full SBD3.296.141.2018.52

The different ablation trends can be explained by the difficulty of learning an effective bridge within a limited number of adaptation steps. The real-data term requires the bridge to learn a denoising mapping from noisy–clean training pairs, whereas teacher alignment directly transfers predictions from a pretrained model. The latter allows the bridge to reuse the teacher's denoising knowledge during the short adaptation stage. On low-resolution CIFAR-10, real-data supervision alone is sufficient to produce an effective bridge, so teacher alignment provides a smaller additional gain. On higher-resolution or more complex generation tasks, learning the target mapping from real-data supervision alone within the same limited adaptation budget is more difficult, making teacher alignment more beneficial. The new results support this explanation: on FFHQ, ImageNet, and SDXL, teacher alignment alone outperforms real-data denoising alone.

The full objective achieves the lowest FID on all four tasks, showing that real-data supervision and teacher guidance are complementary even though their relative contributions vary across tasks. We will add these results and analysis to Section 6.3 and revise the current general statement about component necessity to discuss these task-dependent effects.

AuthorsOpenReview ↗

W2. Training time and memory cost

Thank you for raising the issue of training cost. We added measurements of training time and peak GPU memory. For the same number of student updates, SBD increases total training time by 8.0%–28.6%, including the full cost of bridge adaptation.

DatasetBaseline GPU hoursSBD GPU hours, including bridge adaptationTime increaseBaseline peak memory (GB/GPU)SBD peak memory (GB/GPU)
CIFAR-105054+8.0%4.8785.298
FFHQ4448+9.1%11.57512.040
AFHQv24550+11.1%10.68611.152
LSUN-Bedroom2127+28.6%≈28.0≈32.1
LSUN-Cat2126+23.8%≈28.0≈31.8

Simply training DMD for longer does not reproduce the gains obtained with this additional compute. In the equal-GPU-time controls, DMD is trained with the same compute budget as SBD. On FFHQ and CIFAR-10, this extended DMD training reaches FIDs of 11.03 and 4.27, respectively, compared with 6.14 and 3.29 for SBD. In these two settings, using the additional compute for bridge adaptation is therefore more effective than allocating it to further DMD training. We will add these resource statistics to the revised manuscript and discuss the trade-off between generation quality and training cost together with the equal-GPU-time results.

AuthorsOpenReview ↗

Q1. Benefit of teacher alignment beyond real-data denoising

Thank you for the question. With sufficient training, real-data denoising can recover the data score. In SBD, however, the bridge is initialized from the current fake-score model in each cycle and adapted for finite steps. The key is therefore to construct an effective denoising target within this limited adaptation budget. Teacher alignment directly transfers the denoising mapping learned by the pretrained teacher, allowing the bridge to reuse this knowledge rather than learning the target mapping solely from sampled real-data supervision. This guidance is especially valuable for more complex distributions and tasks requiring finer visual detail.

The ablation results support this explanation. On CIFAR-10, real-data denoising alone achieves an FID of 3.36, which improves to 3.29 when teacher alignment is added. On FFHQ, ImageNet, and SDXL, teacher alignment improves FID from 10.55, 1.76, and 24.58 to 6.14, 1.20, and 18.52, respectively. The full objective also outperforms teacher alignment alone, indicating that supervision from real data and pretrained knowledge from the teacher provide complementary benefits.

Thank you for your response. My concerns have been fully addressed.

Reviewer_mTrxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper attempts to address the issue of conflicting student update directions across different noise levels in DMD distillation. It proposes Score Bridge Distillation (SBD), which dynamically learns a Bridge Model that, ideally, captures a linear mixture of the student and teacher distributions and uses it in place of the signal from the teacher model. The paper conducts experiments on models and datasets such as SDXL, demonstrating the superiority of the proposed method over other baselines.

Critical strengths
  • The problem studied in this paper is indeed worth addressing. Compared with Disback, which uses checkpoints at fixed intervals, SBD is more dynamic and enables the student to learn more effectively.
  • The paper empirically demonstrates that SBD can alleviate the issue of conflicting student update directions across different noise levels. The corresponding evaluation metrics introduced in the paper are reasonable and convincing.
Critical weaknesses
  • How is CFG handled when the Bridge Model fits the teacher model's distribution? The paper does not seem to discuss this point. If, at this stage of training, the Bridge Model fits a CFG-enabled teacher model without itself using CFG, does this mean that the Bridge Model is effectively distilling CFG from the teacher? Could this step introduce additional errors?
  • The experiments do not evaluate the method on modern flow-matching models, but only on DDPM-based models such as SDXL. Given that flow matching has almost completely superseded DDPM, experiments on flow-matching models (e.g., SD3.5) would provide more convincing evidence.
Questions for authors

See “Critical Weaknesses.”

3 comments
AuthorsOpenReview ↗

W1. CFG handling during bridge adaptation

Thank you for pointing out that the CFG setting requires further clarification. During bridge adaptation, the teacher-alignment term uses the CFG teacher output with guidance strength w=8w=8:

TCFG(y,σ,c)=T(y,σ,∅)+w[T(y,σ,c)T(y,σ,∅)].T_{\mathrm{CFG}}(y,\sigma,c)=T(y,\sigma,\varnothing)+w\left[T(y,\sigma,c)T(y,\sigma,\varnothing)\right].

The bridge takes this combined output as its supervision target and learns the teacher’s guided prediction through a single conditional forward pass. Subsequent distribution matching directly uses the bridge output without applying CFG again. At a fixed guidance strength, TCFGT_{\mathrm{CFG}} is a deterministic function of the input and condition. Therefore, using a single conditional prediction introduces no additional error at the level of the CFG formulation; applying CFG again would instead duplicate the guidance. As in other distillation methods, finite model capacity and a finite number of adaptation steps may leave an approximation residual. In SBD, however, finite adaptation is used to construct an intermediate target adapted to the current student, rather than to reproduce the teacher exactly. Prior CFG distillation work likewise trains a single conditional network to learn the combined guided output [1].

[1] Chenlin Meng et al. On Distillation of Guided Diffusion Models. CVPR, 2023.

AuthorsOpenReview ↗

W2. Evaluation on a flow-matching model

Thank you for the suggestion. We added a one-step generation experiment with the 8B-parameter flow-matching model SD3.5 Large, using SenseFlow[2] as the baseline, to evaluate SBD on a modern flow-matching model.

Adapting SBD to flow matching. For the interpolation yt=(1−t)x+tϵy_t=(1-t)x+t\epsilon, a velocity prediction vAv_A corresponds to the clean-sample prediction

DA(yt,t,c)=yt−t vA(yt,t,c).D_A(y_t,t,c)=y_t-t\,v_A(y_t,t,c).

We perform bridge adaptation in the clean-sample prediction space. Specifically, we initialize the bridge from the current fake-score model, adapt it using real-data denoising and teacher-alignment supervision, and then freeze it and use it in place of the teacher in the distribution-matching term. The resulting normalized matching field is

dB=DF(yt,t,c)−DB(yt,t,c)mean⁡∣x−DB(yt,t,c)∣.d_B=\frac{D_F(y_t,t,c)-D_B(y_t,t,c)}{\operatorname{mean}|x-D_B(y_t,t,c)|}.

CFG handling. The teacher first combines conditional and unconditional velocity predictions:

vTCFG=vT(yt,t,∅)+w[vT(yt,t,c)−vT(yt,t,∅)].v_T^{\mathrm{CFG}}=v_T(y_t,t,\varnothing)+w\left[v_T(y_t,t,c)-v_T(y_t,t,\varnothing)\right].

We then convert it to DTCFG=yt−t vTCFGD_T^{\mathrm{CFG}}=y_t-t\,v_T^{\mathrm{CFG}} and use this clean-sample prediction for teacher alignment during bridge adaptation. The bridge distills this guided teacher output into a single conditional prediction, which is used directly during subsequent distribution matching without applying CFG again.

Experimental results. SenseFlow and SenseFlow + SBD use the same student initialization, training data, CFG configuration, and base update schedule. Both retain the IDA, ISG, and GAN components of SenseFlow. SBD adds bridge adaptation and replaces the teacher target in the distribution-matching term, together with the target-dependent normalization. We evaluate both methods on 10,000 prompts from COCO 2014 using the same one-step inference setting:

MethodInference stepsCLIP ↑HPSv2 ↑AESv2 ↑PickScore ↑ImageReward ↑
SenseFlow (reported in paper)10.33320.28035.42122.321.0651
SenseFlow + SBD10.34520.28465.58322.951.1193

[2] Xingtong Ge, Xin Zhang, Tongda Xu, Yi Zhang, Xinjie Zhang, Yan Wang, and Jun Zhang. SenseFlow: Scaling Distribution Matching for Flow-based Text-to-Image Distillation. ICLR, 2026.

Thank the authors for their reply. My concerns have largely been addressed, so I decide to raise my score to wa.

Reviewer_pNtbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Score Bridge Distillation (SBD) for one-step diffusion generation. Each cycle copies the current fake-score network, adapts it using real-data denoising and teacher alignment, then uses the frozen result as the student's matching target. Rebuilding this bridge makes the target depend on the evolving student. Experiments cover unconditional generation, class-conditional ImageNet and SDXL text-to-image generation. Ablations and gradient measurements investigate why the bridge improves training.

Critical strengths
  1. The target construction has a clear purpose and meaningful controls. Table 4 tests student initialization, periodic rebuilding and both adaptation losses. It also includes norm-matched and GPU-time-matched DMD controls. These controls make a stronger case for the bridge than an improvement in final FID alone. The large-batch and fixed-mixture comparisons in Appendix C.3 provide additional support.

  2. The reported improvements extend across generation settings without changing inference. On FFHQ, SBD obtains FID 6.14 versus 10.88 for DisBack; on ImageNet, it obtains 1.20 versus 1.38 for DisBack and 1.28 for longer-trained DMD2 (Tables 2-3). The SDXL improvement is smaller, with FID 18.52 versus 18.96 for DisBack (Table 1). Together, these results suggest a useful training modification with an unchanged one-step generator.

Critical weaknesses
  1. The gradient diagnostics do not measure the implemented matching update directly. Appendix B.5, Eq. (10), uses unnormalized denoiser differences with equal weights over eight noise quantiles. The actual loss uses a target-dependent normalization and an adversarial term (Appendix B.1, Eqs. (8)-(9)). These differences matter when interpreting retention as the fraction of the training update that survives cancellation. Higher agreement among bridge gradients also does not by itself establish better alignment with the teacher objective. The controls support the method, but the causal explanation in Section 6.2 is stronger than these measurements establish.

  2. The evidence for the large-scale gains is less complete than the small-scale ablations. The ImageNet improvement over longer-trained DMD2 and the SDXL FID improvement are reported without variation across training runs or corresponding large-scale GPU-time comparisons. Appendix B.3 also leaves some SDXL settings unspecified, including the learning rate and GPU configuration. Given the small SDXL margins and the additional adaptation stage, these details affect how confidently the results establish a quality and training-efficiency advantage at scale. The existing FFHQ/CIFAR-10 compute controls are useful but do not settle this question.

Questions for authors
  1. Do the conflict and retention improvements in Figure 4 persist when computed using the normalized matching field in Eq. (8), with the training noise weighting? Appendix C.2 already reports bridge-teacher gradient cosines at shared checkpoints; please clarify whether those measurements use the implemented normalization and how they relate to the unnormalized conflict diagnostics.

  2. For the ImageNet and SDXL comparisons, can you provide total GPU time including bridge adaptation, the complete training configuration, and any available estimates of run-to-run or evaluation variability? Please clarify which baselines were rerun under the same setup and which results came from released checkpoints or prior reports.

2 comments
AuthorsOpenReview ↗

W1 and Q1. Gradient Diagnostics: Implementation and Interpretation

Thank you for your questions.

  1. Figure 4 uses normalized distribution-matching gradients at each noise level. The normalization term was inadvertently omitted from Equation (10). We will correct the equation as follows:

    gσ=∇θE[1d⟨Gθ(z,c),sg⁡(Fψ(y,σ,c)−A(y,σ,c)WA(x,y,σ,c))⟩],g_\sigma=\nabla_\theta\mathbb E\left[\frac{1}{d}\left\langle G_\theta(z,c),\operatorname{sg}\left(\frac{F_\psi(y,\sigma,c)-A(y,\sigma,c)}{W_A(x,y,\sigma,c)}\right)\right\rangle\right],

    where x=Gθ(z,c)x=G_\theta(z,c), y=sg⁡(x)+σϵy=\operatorname{sg}(x)+\sigma\epsilon, and WAW_A is defined exactly as in Equation (8) . This normalization was already included in the calculations, so the correction does not affect the values reported in Figure 4.

  2. DMD's per-timestep weighting. At each noise level, Figure 4 computes the normalized distribution-matching gradient used in training, which already incorporates DMD's noise-dependent weighting. DMD defines wt=σt2αtWAw_t=\frac{\sigma_t^2}{\alpha_t W_A}, and the corresponding generator gradient contains wtαt(sF−sA)w_t\alpha_t(s_F-s_A). We use the EDM perturbation y=x+σϵy=x+\sigma\epsilon, for which αt=1\alpha_t=1. Moreover, the denoiser prediction and score satisfy D=y+σ2sD=y+\sigma^2s. Therefore,wtαt(sF−sA)=σ2WAFψ−Aσ2=Fψ−AWA.w_t\alpha_t(s_F-s_A)=\frac{\sigma^2}{W_A}\frac{F_\psi-A}{\sigma^2}=\frac{F_\psi-A}{W_A}. Thus, the normalized denoiser difference used in our code is exactly the DMD-weighted matching field: wtw_t is already incorporated, and no additional noise-dependent coefficient is required.

  3. Aggregation across noise levels. DMD uniformly samples timesteps during training, so its expected training gradient averages the gradients from different timesteps under this uniform sampling. To estimate the resulting cross-noise conflict, we likewise compute the actual matching gradients, including wtw_t, at eight representative noise levels and aggregate them with equal weights. Thus, Figure 4 already uses the normalized, training-weighted gradients requested in the question.

  4. GAN loss. The CIFAR-10 and FFHQ experiments in Figure 4 follow DisBack's setting and do not use GAN loss, allowing us to isolate the effect of distribution-matching distillation. Since distribution matching is the only generator objective in these settings, the gradients analyzed in Figure 4 are exactly the gradients used to update the generator during training. To examine whether the same pattern holds with adversarial training, we repeat the cross-noise gradient analysis for DMD2 and SBD on ImageNet 64×6464\times64, where both methods are trained with GAN loss. SBD retains more consistent cross-noise distribution-matching gradients when combined with adversarial training. | Method | Conflict rate ↓\downarrow | Retention ratio ↑\uparrow | |---|---:|---:| | DMD2 | 24.9% | 75.2% | | SBD | 16.8% | 83.4% |

  5. Gradient calculations in Appendix C.2. The bridge–teacher gradient cosine similarities in Appendix C.2 are also computed from the normalized distribution-matching gradients used in training. They compare the update directions induced by the two matching targets at the same checkpoint.

  6. Interpretation of the results in 6.2. We have therefore revised Section 6.2 by replacing “The gradient statistics account for this gap” with “These quality gains are accompanied by more coherent cross-noise DM gradients,” and replacing “a larger share of every update moves the student toward the teacher” with “the diagnosed gradients undergo less cancellation when aggregated.” The revised text states precisely what Figure 4 supports: SBD achieves lower FID together with reduced cross-noise conflict and cancellation.

AuthorsOpenReview ↗

W2 and Q2. Evidence for large-scale gains

Thank you for pointing out that the large-scale experiments lacked details on training cost and variability. We provide the baseline sources, complete training configurations, total compute including bridge adaptation, and available variability estimates below.

Baseline sources. The standard and longer-trained ImageNet results and the SDXL result for DMD2 are taken from the original paper and its public checkpoints; we did not retrain DMD2 under the SBD training setup. We will identify the source of each result in the table notes to distinguish comparisons with published baselines from controlled comparisons under matched settings.

Complete training configurations. We will expand Appendix B.3, particularly the SDXL learning rate and GPU configuration. The main settings are as follows:

ConfigurationImageNet 64×6464\times64SDXL
GPU model and count4 × A10032 × A100
Global batch size3232
Student/fake-score optimizer1e-55e-7
Student/fake-score learning rate and schedule1e-55e-7

Training cost. The table below reports the total training cost, including all bridge adaptation stages. GPU hours are calculated by multiplying wall-clock training time by the number of GPUs.

SettingFID ↓Total GPU hoursBridge adaptation GPU hours
DMD2: ImageNet 64×6464\times641.51336N/A
SBD: ImageNet 64×6464\times641.2019258
DMD2: SDXL19.013840N/A
SBD: SDXL18.521856536

Result variability. We report all currently available repeated-run statistics and distinguish independent training runs from repeated evaluations of a fixed checkpoint.

SettingRepetition typeRepetitionsMean FID ± standard deviation
ImageNet: DMD2Repeated checkpoint evaluations51.544 ± 0.087
ImageNet: DMD2 (longer training)Repeated checkpoint evaluations51.298 ± 0.029
ImageNet: SBDIndependent training runs51.210 ± 0.153
SDXL: DMD2Repeated checkpoint evaluations519.357 ± 0.479
SDXL: SBDIndependent training runs518.398 ± 0.331
Reviewer_5LZ6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

The authors argue that in DMD training, update signals at different noise levels can conflict, resulting in noisy training signals. Hence, they propose score bridge distillation, which replaces the fixed teacher score with a periodically refreshed bridge score. The bridge is initialized from the current fake-score model and briefly adapted using both real-data denoising and teacher-alignment objectives. It is then frozen to guide student training and rebuilt as the student evolves. The authors report reduced cross-noise gradient conflict, faster convergence, and improved generation quality.

Critical strengths
  1. A wide range of experiments covering text-to-image, unconditional generation, and class-conditional generation.
  2. The motivation makes sense (conflicting update directions across noise levels are a potential obstacle), and demonstrates the effectiveness of the SBD by showing more coherent gradient updates ("conflicts" and "retention")
Critical weaknesses
  1. Additional training complexity: introduces an auxiliary bridge denoiser, requiring three main networks to be optimized in alternating stages. This further complicates the already involved DMD framework and adds more hyperparameters to tune, raising concerns about the tuning burden and scalability to larger models.

  2. Inconsistent baselines. SDXL and ImageNet include comparisons with DMD2, while other experiments use baselines labelled DMD. I am concerned that the larger gains over DMD may not hold against a well-tuned DMD2 baseline, which would weaken the evidence for the method’s effectiveness.

  3. Modest gains in conditional generation. The improvements in text-to-image and class-conditional generation are relatively modest. also lacks visual comparisons with the baselines.

  4. Unclear effect on the final convergence behavior. The experiments show more coherent gradients and improvements over DMD, but it remains unclear whether repeatedly matching intermediate bridge targets introduces persistent bias. I am concerned about how this affects convergence to the intended target distribution and whether it could ultimately limit the attainable generation quality.

  5. Presentation clarity. I found the presentation difficult to follow, particularly Figures 1 and 2.

Questions for authors
  1. Adversarial training and implementation details. Figure 2 includes an adversarial loss, but its use across the experiments is unclear. (is GAN used when comparing to DMD?)

  2. Could the authors provide the full training objectives, loss weights, and update schedules for each setting, and clarify which components and settings are shared with the DMD/DMD2 baselines?

  3. DMD2 reports improved stability when the fake-score model receives more updates per generator update, allowing it to better track the evolving generator distribution. I wonder whether some of the observed gradient conflict could also arise from inaccurate fake-score estimates.

9 comments
AuthorsOpenReview ↗

W1. Additional training complexity

Thank you for raising concerns about training complexity, hyperparameter tuning, and scalability.

Training procedure. DMD already alternates between optimizing the student generator and the fake-score model. SBD retains the original optimization procedure for these two networks and adds a periodic bridge adaptation stage. After every MM distribution-matching iterations, we reinitialize the bridge model with the weights of the current fake-score model and adapt it for KK steps using the teacher score and real data as targets. We then freeze the bridge model, use it in place of the teacher as the matching target, and continue the original DMD training procedure. The bridge model uses the same architecture, optimizer type, and learning rate as the fake-score model.

Hyperparameters and tuning burden. The default bridge adaptation steps KK, reconstruction interval MM, and teacher-alignment weight λ\lambda are 200, 1000, and 1, respectively. Setting λ=1\lambda=1 corresponds to directly adding the real-data denoising and teacher-alignment losses. Appendix Tables 5–6 show that, with all other settings fixed, every tested value of K∈{100,200,400,800}K\in\{100,200,400,800\} outperforms the corresponding DMD baseline on FFHQ and CIFAR-10. For M∈{500,1000,2000}M\in\{500,1000,2000\}, the FIDs are 7.54, 6.14, and 7.83 on FFHQ, and 3.49, 3.29, and 3.58 on CIFAR-10. All are better than the DMD results of 11.23 and 4.32, respectively. Both datasets achieve their best tested results with the default K=200K=200, M=1000M=1000 setting.

Scalability to large models. We have applied SBD to SDXL, whose denoising backbone has approximately 2.6B parameters, for one-step text-to-image generation at 1024×10241024\times1024 resolution. In the 10K COCO-2014 evaluation in Table 1, with both methods using one inference step, SBD improves DMD2's FID from 19.01 to 18.52 and also improves Patch-FID, CLIP, PickScore, HPSv2, and MPS. This provides direct evidence of its applicability to billion-parameter, high-resolution generative models.

AuthorsOpenReview ↗

W2 and Q1. Baseline consistency and GAN loss

Thank you for raising questions about baseline consistency and the use of GAN loss. We clarify the rationale for our baseline choices below.

Baseline selection and GAN loss. For CIFAR-10, FFHQ, AFHQv2, and LSUN, we follow DisBack's experimental setup to focus on the effect of distribution-matching distillation itself. We use DMD without GAN loss as the baseline and train SBD under the same setting. For ImageNet and SDXL, we include GAN loss and compare SBD with the stronger DMD2 baseline.

Additional DMD2 comparison. To determine whether the observed gains hold against a stronger baseline, we additionally compare SBD with DMD2 on FFHQ and AFHQv2. Both methods use the same teacher, training data, student initialization, GAN loss configuration, and evaluation protocol.

MethodFFHQ 64×6464\times64 FID ↓AFHQv2 64×6464\times64 FID ↓
DMD25.234.07
SBD4.383.32

SBD reduces DMD2's FID by 0.85 on FFHQ and 0.75 on AFHQv2, showing that its gains persist against the stronger DMD2 baseline when both methods use the same GAN loss configuration.

AuthorsOpenReview ↗

W3. Gains in conditional generation

Thank you for the suggestion. To illustrate the quality differences in conditional generation more clearly, we reran the DisBack experiment and added nine groups of text-to-image comparisons. In these examples, SBD produces more complete object structures and clearer local details, including the contours of train cars, the beak of a black swan, and the layered structure of a knight's armor. These visual results complement the quantitative comparison with DisBack in the original Table 1, where SBD and DisBack achieve FIDs of 18.52 and 18.96, respectively, under the same one-step generation setting.

The gains over DMD2 also extend across multiple evaluation metrics. On SDXL, SBD achieves a lower FID (18.52 vs. 19.01) and improves Patch-FID, CLIP, PickScore, HPSv2, and MPS. On ImageNet 64×6464\times64, SBD achieves an FID of 1.20, compared with 1.51 for the standard DMD2 model and 1.28 for its longer-trained variant. Together, the new visualizations and quantitative results provide complementary evidence for conditional generation: the improvements in aggregate metrics are reflected in more complete object structures and clearer local details in individual samples.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.