- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the loss of sample diversity in few-step DMD. The proposed EG-DMD supplements the standard reverse-KL-based DMD objective with a population-level loss. For each prompt, the method independently samples multiple student and teacher trajectories, rolls them out to an intermediate diffusion time, extracts features using a frozen intermediate block of the teacher, and matches the resulting empirical feature distributions using the Sinkhorn divergence. The first student transition is supervised by this energy loss, while the remaining trajectory is optimized using the standard DMD objective. Experiments on SD3.5-Medium, SDXL, and MiniT2I-B/16 suggest that EG-DMD improves DINO/CLIP diversity while retaining competitive quality and preference scores at four-step inference.
Critical strengths
1.The distinction between matching a prescribed trajectory correspondence and matching two sample populations is well motivated. The contraction example is useful for explaining why paired supervision can be unnecessarily restrictive. 2. The ablations are generally informative. The paper examines the number of samples per prompt, energy weight, proxy time, entropic regularization, latent sharing, feature projection, and gradient detachment. The MMD comparison also supports the broader claim that the gain is not entirely specific to Sinkhorn divergence. Meanwhile, the appendix contains more theoretical and systems detail than is typical for this type of empirical work.
Critical weaknesses
- My main concern is that EG-DMD uses (R=4) student and teacher populations per prompt, with each teacher sample requiring a partial multi-step rollout. The appendix reports that the resulting training time per generator update is approximately (2.5x) that of the (R=1) setting, or about (1.9x) with the proposed detached-rollout optimization. However, the baseline numbers in Table 1 are taken from Wu et al. (2026), and the comparison is controlled by training iterations rather than by wall-clock time, teacher NFEs, or total numbers of generated training samples. Figure 4 apparently uses (R=4) rollouts for DMD and DP-DMD, but this is not a replacement for retraining the main baselines under the same total training budget. I would like to see at least one main-model comparison in which DMD and DP-DMD receive the same teacher-NFE or wall-clock budget as EG-DMD.
- The empirical success of EG-DMD appears to depend heavily on the teacher feature encoder rather than on the Sinkhorn population-matching objective alone. In Table 5, removing the feature encoder causes a substantial degradation in diversity: DINO diversity drops from 0.191 to 0.129 on Pick-a-Pic and from 0.261 to 0.208 on COCO, accompanied by lower ImageReward. This is a much larger effect than several of the other design choices studied in the paper.
Questions for authors
1.Can the authors provide a comparison against DMD and DP-DMD matched by total teacher NFEs or wall-clock training time? Specially, I would like to see actually training time for different baselines. 2. Could the authors clarify how essential the teacher feature encoder is to EG-DMD’s gains? Table 5 suggests that removing it substantially degrades both diversity and quality. Have the authors compared different teacher layers, random projections, or independent pretrained feature extractors?
5 comments
W2. The empirical success of EG-DMD appears to depend heavily on the teacher feature encoder rather than on the Sinkhorn population-matching objective alone.
Thank you for raising the point. Your understanding is correct that EG-DMD relies on performing distribution matching in the feature space. In fact, we would like to highlight that leveraging teacher feature encoder to provide effective OT energy guidance signal is one of the key findings in this paper. It is well known that directly solving the OT problem in the high-dimensional latent space is prohibitively noisy, and in this work we effectively address this problem by a simple design of projecting it into the teacher feature space. Therefore, we hope it could be viewed as one of the technical contributions in this paper, instead of being a limitation of the approach. More importantly, we have also empirically verified it to be effective and generalizable across different model architectures.
To further help understand the role of the feature encoder, we provide two additional experiments.
First, we ablate a variant of DP-DMD that performs paired regression in the same teacher feature space as EG-DMD. The results are as follows.
| ImageReward (Pic) | ImageReward (COCO) | DINO Div (Pic) | DINO Div (COCO) | |
|---|---|---|---|---|
| paired regression w/o feat. enc. | 1.138 | 0.985 | 0.168 | 0.237 |
| paired regression w/ feat. enc. | 1.131 | 0.990 | 0.157 | 0.229 |
| energy guidance (reference) | 1.156 | 1.015 | 0.191 | 0.261 |
Notably, it is observed that applying the same feature encoder on top of paired regression offers no empirical benefit and even leads to lower diversity. This further supports that the role of feature encoder is to reduce variance in the signal provided by either trajectory-matching (DP-DMD) or distribution-matching (EG-DMD) objective by constraining the update via the feature encoder's Jacobian. In this way, the variance shrinking effect of trajectory-matching DP-DMD loss gets amplified and further reduces the diversity, while the distribution matching objective of EG-DMD instead benefits from the feature encoder to produce more accurate update direction with a limited number of samples.
Besides, per your advice, we also provide the results on different choices of the feature space. We focus the analysis on choosing different layers from the teacher model instead of leveraging external feature extractors to ensure a fair comparison with the baseline methods.
| ImgR (Pic) | ImgR (COCO) | DINO Div (Pic) | DINO Div (COCO) | |
|---|---|---|---|---|
| mid block (reference) | 1.156 | 1.015 | 0.191 | 0.261 |
| first block | 1.142 | 0.977 | 0.136 | 0.224 |
| last block | 1.149 | 1.003 | 0.192 | 0.258 |
Specifically, our default choice of the middle block as feature map achieves the best performance. Using the first block only provides shallow features that are insufficient to reduce the variance, while using the last block yields similar performance as the middle block while incurring unnecessary compute. We therefore advocate our default choice, which also remains effective across a range of models we used in this work.
Q1. Can the authors provide a comparison against DMD and DP-DMD matched by total teacher NFEs or wall-clock training time?
Please kindly refer to our response to W1.
Q2. Could the authors clarify how essential the teacher feature encoder is to EG-DMD’s gains? Table 5 suggests that removing it substantially degrades both diversity and quality. Have the authors compared different teacher layers, random projections, or independent pretrained feature extractors?
Please kindly refer to our response to W2.
We have incorporated the additional results and analyses into the revised manuscript, all marked in red. We sincerely hope our response could well address your concerns!
Look forward to your feedback
Dear Reviewer 3ZeP,
We sincerely thank you for the detailed review and advice.
As the discussion period is coming to the end soon, we are eager to hear your feedbacks on our rebuttal, where we have provided a point-to-point response with additional results. We have also revised the manuscript accordingly. We are more than willing to clarify or discuss more if needed.
Thank you again!
Best,
Authors
Thank you for the detailed review and suggestions! We provide a point-to-point response to your comments below.
Thank you for raising this point. For vanilla DMD, additional compute does not recover diversity, and its diversity decreases during training (Fig. 4, right). We provide additional results of the baseline DP-DMD with R=4 rollouts per prompt with the same teacher NFE as follows. We also report the training time for each method and include the result of the training-efficient version of EG-DMD depicted in Appendix C.
The numbers in each cell refer to the evaluation on Pick-a-Pic/COCO2014-10K.
In particular, when scaling DP-DMD to R=4 to match the training compute, its quality is essentially unchanged while a slight decrease in diversity scores is instead observed. This is because DP-DMD trains on multiple rollouts independently, which leads to a less noisy estimate of the DP-DMD loss and therefore strengthens the variance shrinkage effect (our Proposition 4.1) and decreases the diversity. Differently, our EG-DMD is designed to leverage the distribution-level signal within a group of rollouts to preserve high diversity while maintaining the sample quality. Remarkably, our efficient-training variant that only performs gradient update on a single trajectory also achieves better performance than DP-DMD with R=4, while being more training-efficient.