acceptodds
Under review as a conference paper at ICLR 2027

Energy-Guided Distribution Matching Distillation for Diverse Few-Step Visual Generation

Abstract

Diffusion distillation enables high-quality generation in only a few sampling steps, but aggressive acceleration often substantially reduces sample diversity. In particular, Distribution Matching Distillation (DMD) optimizes a reverse-KL objective whose mode-seeking behavior can collapse the teacher's rich multimodal distribution. Existing diversity-preserving methods mitigate this issue through trajectory-level regularization, forcing individual student trajectories to reproduce teacher trajectories under a prescribed noise correspondence. We argue that such pairing is unnecessarily restrictive: diversity should instead be preserved at the distribution level. We introduce Energy-Guided Distribution Matching Distillation (EG-DMD), which complements DMD with population-level teacher supervision for improved mode coverage. At each training step, EG-DMD jointly rolls out multiple student and teacher trajectories to an intermediate diffusion time and matches their empirical distributions through a Sinkhorn-divergence energy. Unlike paired regression, this objective optimizes the coupling between student and teacher populations, preserving teacher modes without enforcing a particular noise-to-sample correspondence. Across diffusion models with diverse architectures, EG-DMD consistently improves the diversity–quality frontier over DMD and existing diversity-preserving baselines while retaining four-step inference. On SDXL, EG-DMD improves ImageReward over the strongest baseline by 8.9%, while increasing DINO and CLIP diversity by 13.3% and 23.6%, respectively. These results establish population-level energy guidance as a simple, data-free, and architecture-agnostic principle for high-fidelity, diverse few-step generation.

Primary area:
generative models
Keywords:
diffusion distillation, few-step generation, energy guidance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3ZePOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the loss of sample diversity in few-step DMD. The proposed EG-DMD supplements the standard reverse-KL-based DMD objective with a population-level loss. For each prompt, the method independently samples multiple student and teacher trajectories, rolls them out to an intermediate diffusion time, extracts features using a frozen intermediate block of the teacher, and matches the resulting empirical feature distributions using the Sinkhorn divergence. The first student transition is supervised by this energy loss, while the remaining trajectory is optimized using the standard DMD objective. Experiments on SD3.5-Medium, SDXL, and MiniT2I-B/16 suggest that EG-DMD improves DINO/CLIP diversity while retaining competitive quality and preference scores at four-step inference.

Critical strengths

1.The distinction between matching a prescribed trajectory correspondence and matching two sample populations is well motivated. The contraction example is useful for explaining why paired supervision can be unnecessarily restrictive. 2. The ablations are generally informative. The paper examines the number of samples per prompt, energy weight, proxy time, entropic regularization, latent sharing, feature projection, and gradient detachment. The MMD comparison also supports the broader claim that the gain is not entirely specific to Sinkhorn divergence. Meanwhile, the appendix contains more theoretical and systems detail than is typical for this type of empirical work.

Critical weaknesses
  1. My main concern is that EG-DMD uses (R=4) student and teacher populations per prompt, with each teacher sample requiring a partial multi-step rollout. The appendix reports that the resulting training time per generator update is approximately (2.5x) that of the (R=1) setting, or about (1.9x) with the proposed detached-rollout optimization. However, the baseline numbers in Table 1 are taken from Wu et al. (2026), and the comparison is controlled by training iterations rather than by wall-clock time, teacher NFEs, or total numbers of generated training samples. Figure 4 apparently uses (R=4) rollouts for DMD and DP-DMD, but this is not a replacement for retraining the main baselines under the same total training budget. I would like to see at least one main-model comparison in which DMD and DP-DMD receive the same teacher-NFE or wall-clock budget as EG-DMD.
  2. The empirical success of EG-DMD appears to depend heavily on the teacher feature encoder rather than on the Sinkhorn population-matching objective alone. In Table 5, removing the feature encoder causes a substantial degradation in diversity: DINO diversity drops from 0.191 to 0.129 on Pick-a-Pic and from 0.261 to 0.208 on COCO, accompanied by lower ImageReward. This is a much larger effect than several of the other design choices studied in the paper.
Questions for authors

1.Can the authors provide a comparison against DMD and DP-DMD matched by total teacher NFEs or wall-clock training time? Specially, I would like to see actually training time for different baselines. 2. Could the authors clarify how essential the teacher feature encoder is to EG-DMD’s gains? Table 5 suggests that removing it substantially degrades both diversity and quality. Have the authors compared different teacher layers, random projections, or independent pretrained feature extractors?

5 comments

Thank you for the detailed review and suggestions! We provide a point-to-point response to your comments below.

W1. My main concern is that EG-DMD uses (R=4) student and teacher populations per prompt, with each teacher sample requiring a partial multi-step rollout.

Thank you for raising this point. For vanilla DMD, additional compute does not recover diversity, and its diversity decreases during training (Fig. 4, right). We provide additional results of the baseline DP-DMD with R=4R=4 rollouts per prompt with the same teacher NFE as follows. We also report the training time for each method and include the result of the training-efficient version of EG-DMD depicted in Appendix C.

The numbers in each cell refer to the evaluation on Pick-a-Pic/COCO2014-10K.

DINO DivCLIP DivVQ-R1MIQAImgRPicSTraining Time
DP-DMD (R=1R=1)0.179/0.2500.182/0.1824.646/4.6891.017/1.0321.142/0.98821.76/22.418h12min
DP-DMD (R=4R=4)0.168/0.2370.175/0.1694.654/4.7021.008/1.0421.138/0.98521.80/22.3618h51min
EG-DMD (R=4R=4)0.191/0.2610.188/0.1904.688/4.7571.045/1.0711.156/1.01521.78/22.4119h48min
EG-DMD (efficient)0.187/0.2640.190/0.1914.682/4.7451.051/1.0741.153/1.00821.87/22.4615h31min

In particular, when scaling DP-DMD to R=4R=4 to match the training compute, its quality is essentially unchanged while a slight decrease in diversity scores is instead observed. This is because DP-DMD trains on multiple rollouts independently, which leads to a less noisy estimate of the DP-DMD loss and therefore strengthens the variance shrinkage effect (our Proposition 4.1) and decreases the diversity. Differently, our EG-DMD is designed to leverage the distribution-level signal within a group of rollouts to preserve high diversity while maintaining the sample quality. Remarkably, our efficient-training variant that only performs gradient update on a single trajectory also achieves better performance than DP-DMD with R=4R=4, while being more training-efficient.

W2. The empirical success of EG-DMD appears to depend heavily on the teacher feature encoder rather than on the Sinkhorn population-matching objective alone.

Thank you for raising the point. Your understanding is correct that EG-DMD relies on performing distribution matching in the feature space. In fact, we would like to highlight that leveraging teacher feature encoder to provide effective OT energy guidance signal is one of the key findings in this paper. It is well known that directly solving the OT problem in the high-dimensional latent space is prohibitively noisy, and in this work we effectively address this problem by a simple design of projecting it into the teacher feature space. Therefore, we hope it could be viewed as one of the technical contributions in this paper, instead of being a limitation of the approach. More importantly, we have also empirically verified it to be effective and generalizable across different model architectures.

To further help understand the role of the feature encoder, we provide two additional experiments.

First, we ablate a variant of DP-DMD that performs paired regression in the same teacher feature space as EG-DMD. The results are as follows.

ImageReward (Pic)ImageReward (COCO)DINO Div (Pic)DINO Div (COCO)
paired regression w/o feat. enc.1.1380.9850.1680.237
paired regression w/ feat. enc.1.1310.9900.1570.229
energy guidance (reference)1.1561.0150.1910.261

Notably, it is observed that applying the same feature encoder on top of paired regression offers no empirical benefit and even leads to lower diversity. This further supports that the role of feature encoder is to reduce variance in the signal provided by either trajectory-matching (DP-DMD) or distribution-matching (EG-DMD) objective by constraining the update via the feature encoder's Jacobian. In this way, the variance shrinking effect of trajectory-matching DP-DMD loss gets amplified and further reduces the diversity, while the distribution matching objective of EG-DMD instead benefits from the feature encoder to produce more accurate update direction with a limited number of samples.

Besides, per your advice, we also provide the results on different choices of the feature space. We focus the analysis on choosing different layers from the teacher model instead of leveraging external feature extractors to ensure a fair comparison with the baseline methods.

ImgR (Pic)ImgR (COCO)DINO Div (Pic)DINO Div (COCO)
mid block (reference)1.1561.0150.1910.261
first block1.1420.9770.1360.224
last block1.1491.0030.1920.258

Specifically, our default choice of the middle block as feature map achieves the best performance. Using the first block only provides shallow features that are insufficient to reduce the variance, while using the last block yields similar performance as the middle block while incurring unnecessary compute. We therefore advocate our default choice, which also remains effective across a range of models we used in this work.

Q1. Can the authors provide a comparison against DMD and DP-DMD matched by total teacher NFEs or wall-clock training time?

Please kindly refer to our response to W1.

Q2. Could the authors clarify how essential the teacher feature encoder is to EG-DMD’s gains? Table 5 suggests that removing it substantially degrades both diversity and quality. Have the authors compared different teacher layers, random projections, or independent pretrained feature extractors?

Please kindly refer to our response to W2.

We have incorporated the additional results and analyses into the revised manuscript, all marked in red. We sincerely hope our response could well address your concerns!

AuthorsOpenReview ↗

Look forward to your feedback

Dear Reviewer 3ZeP,

We sincerely thank you for the detailed review and advice.

As the discussion period is coming to the end soon, we are eager to hear your feedbacks on our rebuttal, where we have provided a point-to-point response with additional results. We have also revised the manuscript accordingly. We are more than willing to clarify or discuss more if needed.

Thank you again!

Best,

Authors

Reviewer_bVVbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the issue of sample diversity collapse—caused by the mode-seeking behavior of the reverse-KL divergence in few-step distillation methods like Distribution Matching Distillation (DMD)—this paper proposes EG-DMD. Unlike prior approaches that enforce step-by-step alignment between student and teacher trajectories, EG-DMD constructs surrogate distributions for both the student and teacher models. It employs the Sinkhorn divergence as an auxiliary energy-based loss, thereby enhancing the diversity of the distillation process. The effectiveness of this method has been validated across several models.

Critical strengths
  1. The motivation for the proposed method is clear, and the paper includes theoretical analysis comparing "distribution-level" versus "trajectory-level" alignment. It formally identifies a flaw in paired regression: even when the marginal distributions of the student and teacher models are identical, the paired loss can still decrease strictly along the direction of contraction. The approach of distribution-level alignment also offers a degree of novelty.
  2. The methodological framework is well-designed, and the experimental evaluation is comprehensive. By employing an intermediate proxy, the method reduces the reliance on image datasets; it utilizes Sinkhorn divergence as a loss function and incorporates feature projection to enhance training stability. Overall, the framework is sound and effective. Furthermore, the experiments include comparisons across different model families, which strengthens the credibility of the paper.
Critical weaknesses
  1. There are deficiencies in diversity assessment and super parameter selection. The current diversity index only calculates the cosine similarity of pairwise features per prompt, and each prompt only generates 9 images, lacking some distribution-level indicators to directly measure the diversity of the generated distribution. In Tables 1 and 3, the absolute value of the diversity gap between EG-DMD and DP-DMD for each method is very small, and the quality index gap is also very small. Is there any statistical bias? There is a lack of multiple seed variances/confidence intervals/error bars. Is distribution alignment really sufficient on R=4? Although the paper says that computing with mid-level features can make up for this shortcoming, ablation also shows a significant drop in performance when encoding without features, but is it possible that the observed gains may mainly come from "progressive alignment in teacher feature space" rather than distribution-level alignment. At the same time, λ differs greatly among the three models (0.15/0.005/0.02). It is not explained how it was selected. Is it based on the test set parameter adjustment?
  2. There is no comparison with the latest model. Although experiments were conducted on SD3.5, there is still a lack of experiments on recent popular models, such as Zimage, QwenImage-1, Flux2, etc., which were all released half a year ago. And some models themselves will open source their own distilled versions, such as Zimage-Turbo, Flux2-klein. Does EG-DMD have advantages over these open source distillation models?
Questions for authors
  1. Based on what validation protocol were the hyperparameters—such as λ (0.15/0.005/0.02), τ, and ε—selected?
  2. Is it possible that the observed gains stem primarily from "step-wise alignment within the teacher's feature space" rather than "distribution-level alignment"?
  3. Could you include additional diversity metrics—such as TIE (Truncated Inception Entropy) or TCE (Truncated CLIP Entropy)—as well as error bars?
  4. The paper would be more compelling if it included performance results for more recent models, particularly comparisons with open-source distilled models such as Zimage-Turbo and Flux2-klein.
3 comments

Thank you for the detailed review and suggestions! We provide a point-to-point response to your comments below.

W1.1. Is there any statistical bias? There is a lack of multiple seed variances/confidence intervals/error bars.

Thank you for the comment. We rerun the main experiment with SD3.5-M and 3 seeds (42, 100, and 1000) and report the mean and standard deviation below:

DINO DivCLIP DivVQ-R1MIQAImgRPicS
Pick-a-Pic0.193 (0.002)0.194 (0.004)4.682 (0.005)1.049 (0.003)1.154 (0.006)21.80 (0.016)
COCO2014-10K0.260 (0.002)0.192 (0.002)4.748 (0.008)1.069 (0.004)1.020 (0.004)22.38 (0.021)

It is observed that the results are relatively stable across different seeds. Comparing with the results in Table 1, our EG-DMD consistently outperforms DP-DMD by achieving much higher diversity score under multiple seeds with comparable or better quality metrics.

Besides, we would also like to kindly refer the reviewer to the user study we performed in Figure 7 in Appendix C, which demonstrates a clear and convincing improvement in both diversity and user preference, aligning closely with our main results.

W1.2. Is distribution alignment really sufficient on R=4? Is it possible that the observed gains may mainly come from "progressive alignment in teacher feature space" rather than distribution-level alignment?

Thank you for the question! We provide additional ablation experiments which perform the DP-DMD style paired-regression but in the same teacher feature space as EG-DMD. Note that all ablations here adopt the R=4R=4 setting, which ensures a controlled training budget.

ImageReward (Pic)ImageReward (COCO)DINO Div (Pic)DINO Div (COCO)
paired regression w/o feat. enc.1.1380.9850.1680.237
paired regression w/ feat. enc.1.1310.9900.1570.229
energy guidance (reference)1.1561.0150.1910.261

Notably, it is observed that performing paired-regression in the same teacher feature space does not lead to better performance. In fact, the diversity even drops compared with the raw latent space paired regression. This is expected, since the role of the feature encoder is to reduce variance by constraining the optimization to the latent manifold induced by the encoder's Jacobian. Since paired regression induces a variance contraction effect as we have analyzed in the paper (see Proposition 4.1), such an artifact will be magnified even more severely when the feature encoder is present, hence the decreased diversity. Therefore, this result also directly verifies that the observed gain does indeed come from the distribution-level alignment in the feature space, since doing paired-regression in the same feature space yields no empirical benefits.

We thank the reviewer for raising this ablation which in fact leads to a more favorable observation that further strengthens the analysis in our paper. We have also revised the manuscript to include this result and the corresponding analysis.

W1.3. λ differs greatly among the three models (0.15/0.005/0.02). It is not explained how it was selected. Is it based on the test set parameter adjustment?

Thank you for the question! When we were doing the experiments, we first focused on doing ablation studies over the hyperparameters λ\lambda, τ\tau, and ε\varepsilon using SD3.5-M as the base model to try to identify the correct range for these hyperparameters. This leads to the results we presented in Table 5 in the main paper. Then, for each model, we tune λ\lambda and ε\varepsilon within the range using a held out validation set with 500 prompts from DiffusionDB.

Notably, the scale difference in λ\lambda (0.15/0.005/0.02) is due to the fact that different models (SD3.5-M, SDXL, and MiniT2I) produce very different gradient scales in the DMD loss due to their parameterization. In practice, before training, we pre-computed the gradient norm ratio η=∥∥∇θLDMD∥∥/∥∥∇θLEG∥∥\eta=\|\|\nabla_\theta\mathcal{L}_\mathrm{DMD}\|\|/\|\|\nabla_\theta\mathcal{L}_\mathrm{EG}\|\| on a batch of 32 training samples and normalize λ\lambda by η\eta such that the choices of λ\lambda on other models match the optimal range of λ\lambda on SD3.5-M. Finally, we reported the raw values of λ\lambda (0.15/0.005/0.02) instead of the η\eta-normalized scale for consistency. We have included this detail in Appendix B in the updated manuscript.

When deploying our approach to a potential new model, we recommend following the same hyperparameter selection procedure we adopted in the paper to avoid the need of exhaustive hyperparameter tuning.

W2. There is no comparison with the latest model. Does EG-DMD have advantages over these open source distillation models?

Thank you for raising this point. We would like to first respectfully point out that we have covered a wide range of base models in the current experiments, including SD3, SD3.5-M, SDXL, and the very recent pixel-space diffusion model MiniT2I. The experiment results have consistently verified the efficacy of our approach across models with different architectures, i.e., UNet, latent-space DiT, and pixel-space DiT. The comparison is also performed in a controlled setting with the same dataset and training iteration.

Per your advice, we apply our approach on the Z-Image model and draw a comparison with the official Z-Image-Turbo checkpoint. We keep the same training recipe, dataset, and iterations in the same experiment setting in our paper. We do highlight, however, that this is only a system-level comparison since the actual training setup is indeed very different from the official Z-Image-Turbo and it is prohibitive for us to reproduce their setup, including dataset, training steps, etc. We follow the default inference setting for the base model and Turbo.

On Pick-a-Pic:

DINO divCLIP divVQ-R1MIQAImgRPickScore
Base0.2750.2514.6331.0591.02321.40
Turbo0.1460.1404.7721.0721.01821.65
EGDMD0.2020.2064.6171.0481.00521.54

On COCO2014-10K:

DINO divCLIP divVQ-R1MIQAImgRPickScore
Base0.3000.2184.7271.0581.06822.54
Turbo0.1470.1224.8141.0690.95022.93
EGDMD0.2210.1744.7061.0450.98222.57

We observe that Z-Image-Turbo significantly degrades diversity. Our EG-DMD achieves much higher diversity scores at slightly lower perceptual quality than Turbo, which was trained on much larger data with a different pipeline. The results again verify the wide applicability of our approach.

Q1. Based on what validation protocol were the hyperparameters—such as λ (0.15/0.005/0.02), τ, and ε—selected?

Please kindly refer to our response to W1.3.

Q2. Is it possible that the observed gains stem primarily from "step-wise alignment within the teacher's feature space" rather than "distribution-level alignment"?

Please kindly refer to our response to W1.2.

Q3. Could you include additional diversity metrics—such as TIE (Truncated Inception Entropy) or TCE (Truncated CLIP Entropy)—as well as error bars?

Thank you for the advice for including TIE and TCE as additional diversity metrics. We follow the setting in the original work [1], sample 32 seeds per prompt, and compute TIE and TCE using the top-20 eigenvalues. We provide the results below (std in the parenthesis):

On SD3.5-M:

Pick-a-Pic TIEPick-a-Pic TCECOCO TIECOCO TCE
DMD33.90 (0.15)10.86 (0.18)31.65 (0.16)12.94 (0.14)
DP-DMD36.79 (0.12)15.97 (0.13)33.86 (0.14)15.73 (0.12)
EG-DMD38.50 (0.11)17.10 (0.13)35.06 (0.14)17.55 (0.11)

On SDXL:

Pick-a-Pic TIEPick-a-Pic TCECOCO TIECOCO TCE
DMD39.29 (0.09)16.80 (0.10)36.50 (0.14)17.02 (0.11)
DP-DMD40.47 (0.08)18.33 (0.09)37.77 (0.13)18.62 (0.13)
EG-DMD41.04 (0.08)19.42 (0.10)38.74 (0.11)20.22 (0.12)

Our EG-DMD clearly shows higher TIE and TCE diversity scores, which is consistent with the main results and strengthens the conclusion.

[1] Ibarrola and Grace. Measuring Diversity in Co-creative Image Generation. 2024.

Q4. The paper would be more compelling if it included performance results for more recent models, particularly comparisons with open-source distilled models such as Zimage-Turbo and Flux2-klein.

Please kindly refer to our response to W2.

We have incorporated the additional results and analyses into the revised manuscript, all marked in red. We sincerely hope our response could well address your concerns!

AuthorsOpenReview ↗

Look forward to your feedback

Dear Reviewer bVVb,

We sincerely thank you for the detailed review and advice.

As the discussion period is coming to the end soon, we are eager to hear your feedbacks on our rebuttal, where we have provided a point-to-point response with additional results. We have also revised the manuscript accordingly. We are more than willing to clarify or discuss more if needed.

Thank you again!

Best,

Authors

Reviewer_EPRaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EG-DMD, which augments DMD with Sinkhorn distribution matching in a teacher feature space instead of the fixed-pair auxiliary supervision used in prior diversity-preserving approaches. Gradient separation assigns different objectives to the first and subsequent student steps. Experiments across three text-to-image models, ablations, and a user study support improved quality–diversity trade-offs for four-step generation.

Critical strengths
  1. The method is well motivated. The analysis shows that fixed-pair supervision can induce contraction even when the marginals agree, providing a meaningful rationale for population matching.
  2. Evaluation spans different architectures and both latent- and pixel-space models. Ablations and a user study provide additional empirical support for the method’s practical value.
Critical weaknesses
  1. The independent contribution of population matching is not fully isolated. Although Table 5 evaluates shared latents and the removal of teacher features, these ablations do not replace a paired-regression baseline in the same teacher feature space. Matching population size and gradient settings would help distinguish the contributions of feature supervision and population matching.
  2. The advantage under equal training computation remains unclear. The authors control training iterations and major settings, analyze costs in Appendix C, and match population sizes in Figure 4. However, the main results do not establish performance under matched GPU-time or teacher-computation budgets. Given the additional teacher rollouts, the relationship between performance gains and training cost remains incompletely characterized.
Questions for authors
  1. How does fixed-pair feature regression compare with Sinkhorn matching under the same teacher features, R=4, gradient settings, and training budget?
  2. Do the main improvements over DMD and DP-DMD persist under matched total GPU-time or teacher-network evaluation budgets?
  3. Implementation clarification: Should Eq. (16) and Step 6 of Algorithm 1 use the detached intermediate student state rather than the original latent z_i? How are the first and subsequent steps counted within the total four-step sampling budget?
3 comments

Thank you for the detailed review and suggestions! We provide a point-to-point response to your comments below.

W1. The independent contribution of population matching is not fully isolated. Although Table 5 evaluates shared latents and the removal of teacher features, these ablations do not replace a paired-regression baseline in the same teacher feature space.

Thank you for the question! We provide additional ablation experiments which perform the DP-DMD style paired-regression but in the same teacher feature space as EG-DMD. Note that all ablations here adopt the R=4R=4 setting, which ensures a controlled training budget.

ImageReward (Pic)ImageReward (COCO)DINO Div (Pic)DINO Div (COCO)
paired regression w/o feat. enc.1.1380.9850.1680.237
paired regression w/ feat. enc.1.1310.9900.1570.229
energy guidance (reference)1.1561.0150.1910.261

Notably, it is observed that performing paired-regression in the same teacher feature space does not lead to better performance. In fact, the diversity even drops compared with the raw latent space paired regression. This is expected, since the role of the feature encoder is to reduce variance by constraining the optimization to the latent manifold induced by the encoder's Jacobian. Since paired regression induces a variance contraction effect as we have analyzed in the paper (see Proposition 4.1), such an artifact will be magnified even more severely when the feature encoder is present, hence the decreased diversity. Therefore, this result also directly verifies that the observed gain does indeed come from the distribution-level alignment in the feature space, since doing paired-regression in the same feature space yields no empirical benefits.

We thank the reviewer for raising this ablation which in fact leads to a more favorable observation that further strengthens the analysis in our paper. We have also revised the manuscript to include this result and the corresponding analysis.

W2. The advantage under equal training computation remains unclear. The authors control training iterations and major settings, analyze costs in Appendix C, and match population sizes in Figure 4. However, the main results do not establish performance under matched GPU-time or teacher-computation budgets.

Thank you for raising this point. For vanilla DMD, additional compute does not recover diversity, and its diversity decreases during training (Fig. 4, right). We provide additional results of the baseline DP-DMD with R=4R=4 rollouts per prompt with the same teacher NFE as follows. We also report the training time (on 1 H100 node) for each method and include the result of the training-efficient version of EG-DMD depicted in Appendix C.

The numbers in each cell refer to the evaluation on Pick-a-Pic/COCO2014-10K.

DINO DivCLIP DivVQ-R1MIQAImgRPicSTraining Time
DP-DMD (R=1R=1)0.179/0.2500.182/0.1824.646/4.6891.017/1.0321.142/0.98821.76/22.418h12min
DP-DMD (R=4R=4)0.168/0.2370.175/0.1694.654/4.7021.008/1.0421.138/0.98521.80/22.3618h51min
EG-DMD (R=4R=4)0.191/0.2610.188/0.1904.688/4.7571.045/1.0711.156/1.01521.78/22.4119h48min
EG-DMD (efficient variant)0.187/0.2640.190/0.1914.682/4.7451.051/1.0741.153/1.00821.87/22.4615h31min

In particular, when scaling DP-DMD to R=4R=4 to match the training compute, its quality is essentially unchanged while a slight decrease in diversity scores is instead observed. This is because DP-DMD trains on multiple rollouts independently, which leads to a less noisy estimate of the DP-DMD loss and therefore strengthens the variance shrinkage effect (our Proposition 4.1) and decreases the diversity. Differently, our EG-DMD is designed to leverage the distribution-level signal within a group of rollouts to preserve high diversity while maintaining the sample quality. Remarkably, our efficient-training variant that only performs gradient update on a single trajectory also achieves better performance than DP-DMD with R=4R=4, while being more training-efficient.

Q1. How does fixed-pair feature regression compare with Sinkhorn matching under the same teacher features, R=4, gradient settings, and training budget?

Please kindly refer to our response to W1.

Q2. Do the main improvements over DMD and DP-DMD persist under matched total GPU-time or teacher-network evaluation budgets?

Please kindly refer to our response to W2.

Q3. Implementation clarification: Should Eq. (16) and Step 6 of Algorithm 1 use the detached intermediate student state rather than the original latent z_i? How are the first and subsequent steps counted within the total four-step sampling budget?

Yes, both Eq. (16) and Step 6 of Algorithm 1 detach the gradient after the first student sampling step while enabling the gradient for the remaining 3 steps. Therefore, in total there are still only 4 steps, while the gradient is detached immediately after the first step. We have specified this implementation detail in the paper (see line 302-304 and Step 6 description of Algorithm 1). We also updated the manuscript to explicitly clarify that we perform 4 sampling steps in total, with the first step's gradient detached and the remaining 3 steps' gradient enabled.

We have incorporated the additional results and analyses into the revised manuscript, all marked in red. We sincerely hope our response could well address your concerns!

AuthorsOpenReview ↗

Look forward to your feedback

Dear Reviewer EPRa,

We sincerely thank you for the detailed review and advice.

As the discussion period is coming to the end soon, we are eager to hear your feedbacks on our rebuttal, where we have provided a point-to-point response with additional results. We have also revised the manuscript accordingly. We are more than willing to clarify or discuss more if needed.

Thank you again!

Best,

Authors

Comments on the paper

1 comment

Dear reviewers,

This is a kind reminder that the discussion with the authors ends on 18th Nov. Please take a moment to leave a comment to acknowledge the authors’ responses and engage in the discussion if you have further questions, particularly on points that may affect your assessment.

Thank you!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.