acceptodds
Under review as a conference paper at ICLR 2027

RISE-DMD: Reward-Improved Self-Evolving Distribution Matching Distillation for Black-Box Reward Optimization

Abstract

Distribution Matching Distillation (DMD) can distill a multi-step diffusion model into a deterministic few-step generator, but introducing black-box rewards into DMD poses a key challenge: black-box rewards provide only scalar evaluations of final samples and do not directly specify the target distribution or marginal velocity field required by DMD. We propose RISE-DMD (Reward-Improved Self-Evolving Distribution Matching Distillation) and introduce Reward Score to learn a reward-improved marginal velocity field. Reward Score is trained on on-policy samples generated by the current student and their reward feedback, after which DMD uses the learned field to update the few-step student. The updated student then generates data for the next training round, allowing the target distribution to evolve together with the student distribution. Theoretically, we characterize the population-optimal Reward Score field and the corresponding reward-tilted target distributions at each diffusion time, and provide a regularized reward-optimization interpretation under a fixed-reference constraint. In the main SD3.5 Medium experiment, RISE-DMD outperforms the compared methods on all six preference and quality metrics, three of which are not used for training. Results on Sana 1.6B, DrawBench, and OCR further demonstrate the method's applicability across model backbones and prompt distributions, as well as its ability to optimize non-differentiable task rewards using only scalar feedback.

Primary area:
generative models
Keywords:
Distribution Matching Distillation, black-box reward optimization, few-step diffusion models, reward tilting, marginal velocity fields

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_TdS5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RISE-DMD optimizes a deterministic few-step diffusion generator using black-box scalar rewards. It trains an auxiliary Reward Score model on student-generated samples with a positive-negative Flow Matching objective adapted from DiffusionNFT and a penalty toward a fixed reference/teacher model. The student is then updated through DMD, using the Reward Score EMA to supply the target velocity field and a separate Fake Score model to estimate the current student's marginal velocity field. Fresh student samples drive the next round of target learning. The authors characterize the population optimum and its target densities at each noise level, and report improvements on SD3.5 Medium, Sana, DrawBench, and an OCR task.

Critical strengths

S1. The method addresses a clear limitation of reward-weighted DMD. When the student and reference fields match, weighting the original DMD gradient by reward still gives a zero update. Learning a different target field provides a way to incorporate black-box feedback while retaining four-step generation.

S2. RISE-DMD obtains the best values on all six SD3.5 Medium metrics in Table 1, including comparisons with DMDR and TDM-R1. Results on Sana and DrawBench provide evidence beyond the main backbone and prompt set, and the gains extend to metrics excluded from the training reward.

S3. The population analysis helps explain the training objective. With an ideal current-student center, the analysis characterizes how reward changes the target density at each noise level. Its KL-regularized interpretation also makes the roles of the current student and the fixed reference easy to understand.

Critical weaknesses

W1. Some important training details remain unclear. Appendix A.6 provides reward-processing and batch settings, but does not give the values of β\beta and λref\lambda_{\mathrm{ref}}, the EMA schedules, or several optimizer and LoRA settings. Appendix A.3 also mentions a "practical branch normalization"; it is unclear whether this means the reward normalization documented in Appendix A.5 or an additional operation. A complete configuration table and focused sensitivity ablations for β\beta, λref\lambda_{\mathrm{ref}}, and the EMA schedule would help readers assess the effect of these choices. Also, The validation and checkpoint-selection protocol could be more explicit. Appendix A.5 states that the normalization configuration was selected considering all six metrics, including those excluded from the training reward. It is unclear whether this selection used a separate validation set or the final evaluation prompts. The checkpoint-selection procedure is also unspecified. Clarifying the selection data and criteria would help readers interpret the reported evaluation results.

W2. Some gains over strong baselines are modest; for example, Aesthetic improves from 6.422 to 6.521 over TDM-R1 in Table 1. The table reports only point estimates. 95% confidence intervals, computed from the existing evaluation outputs or additional repeated experiments, would help quantify uncertainty in the mean differences over the evaluation prompts.

W3. Under the stated ideal-field assumption, Eq. (16) characterizes a valid target density at each noise level. However, these targets need not be the noisy marginals of one clean distribution. Consider the unregularized case with a fixed reward r(x0)r(x_0), let NtN_t denote forward noising operator. For a clean tilt q0∝p0e(2/β)rq_0\propto p_0e^{(2/\beta)r}, the actual noisy marginal and the target in Eq. (16) are

\qquad q_t\propto p_t e^{(2/\beta)\mathbb E[r\mid x_t]}.$$ These expressions generally differ, so the per-time construction does not by itself establish a compatible clean-space target. If the targets are incompatible, one clean student cannot match them all exactly. Appendix A.9 explicitly assumes cross-time compatibility and limits its convergence result to that idealized setting. A brief discussion of how DMD fits the targets when compatibility does not hold, and how that fit relates to the intended clean target, would strengthen the theoretical interpretation.
Questions for authors

Q1. Could you provide a complete configuration table, including the implemented loss normalization and optimizer step counts or update frequencies for each component, and critical sensitivity ablations for β\beta, λref\lambda_{\mathrm{ref}}, and the EMA schedule?

Q2. Could you report prompt-level 95% confidence intervals for the main differences against the strongest baselines, using the existing evaluation outputs?

Q3. Which prompts and criteria were used to select normalization settings and evaluation checkpoints? Were these prompts separate from the final evaluation sets?

Q4. How should the DMD update be interpreted when the per-time targets are not compatible with one clean distribution, and what can be said about the resulting clean distribution?

5 comments
AuthorsOpenReview ↗

Author Response (1/4)

We sincerely thank the reviewer for the careful and constructive review, and in particular for the precise theoretical question on cross-time compatibility and the concrete suggestions on configuration, uncertainty and selection protocol. They helped us make both the analysis and the experimental protocol clearer and more rigorous. We address each point below. Unless stated otherwise, all numbers are means ± std over three generation seeds.

W1 & Q1: Configuration and sensitivity

Branch normalization. We apologize for the ambiguous wording: this is a separate operation from reward normalization. Reward normalization acts on advantages (group std, clip 1.0). Branch normalization is the adaptive loss weighting of DiffusionNFT (Sec. 3.3, "self-normalized x₀ regression"), applied separately to the positive and negative branches: L±=∥x0±−x0∥2/sg(mean∣x0±−x0∣)L^\pm=\|x_0^\pm-x_0\|^2/\mathrm{sg}(\mathrm{mean}|x_0^\pm-x_0|), with x0±=xt−σv±x_0^\pm=x_t-\sigma v^\pm.

Hyperparameters and compute (to be added to the appendix):

ItemValue
Training rewardPickScore/26 + CLIPScore/100 + HPSv2.1, prompt-centered
β / λ_ref / Reward Score LR1.0 / 0.01 / 5e-5
Advantagegroup std, clip 1.0
Reward Score EMAType-1 (main): min⁡(0.001i,0.5)\min(0.001i,0.5); Type-2 (OCR): 0 for i≤75i\le75, then min⁡(0.0075(i−75),0.999)\min(0.0075(i-75),0.999)
Field time samplingshifted uniform, shift 3 (σ=3u/(1+2u)\sigma=3u/(1+2u), u∼U(0,1)u\sim\mathcal U(0,1)), drawn independently per field
LoRArank 32, alpha 64; separate student / Fake Score / Reward Score adapters
Reward Score init.copied from Fake Score, EMA synchronized
Rollouts per update48 groups × 24 = 1,152, 4-NFE Euler (shift 3)
Per updateone optimizer step each for Fake Score, Reward Score, student; two supervision times per sample
Schedule / computeDMD 1,500 steps + 900 RISE-DMD updates; 212.6 GPU-h on 8×A100
AuthorsOpenReview ↗

Author Response (2/4)

Sensitivity. For the EMA schedule, Fig. 9 compares Type-1 and Type-2: Type-1 improves faster, while the slower Type-2 is more stable over long training. For λ_ref, β and the number of field supervision times, we ran one-factor-at-a-time ablations on Pick-a-Pic. Each run starts from the same DMD checkpoint, changes a single hyperparameter from the default (λ_ref = 1e-2, β = 1, two supervision times), and is evaluated at a fixed endpoint after 200 updates. Each sweep is run on one machine together with its own default run. LPIPS measures per-prompt diversity.

Reference weight λ_ref.

λ_refPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedRewardLPIPS
022.452 ± 0.0030.2815 ± 0.00030.3484 ± 0.00026.646 ± 0.0041.384 ± 0.0040.7090 ± 0.00040.563
1e-422.544 ± 0.0040.2838 ± 0.00010.3483 ± 0.00016.637 ± 0.0011.439 ± 0.0020.7130 ± 0.00090.538
1e-2 (default)22.672 ± 0.0040.2906 ± 0.00030.3432 ± 0.00016.529 ± 0.0041.401 ± 0.0050.7097 ± 0.00130.594
0.121.837 ± 0.0100.2839 ± 0.00050.3009 ± 0.00016.201 ± 0.0021.044 ± 0.0030.6850 ± 0.00100.654

λ_ref controls the trade-off between reward and diversity. Removing or weakening the reference term (λ_ref ≤ 1e-4) raises HPSv2.1 and Aesthetic, but lowers PickScore and CLIPScore and clearly reduces diversity (ΔLPIPS −0.031 and −0.056; both 95% CIs exclude zero), a sign of reward over-optimization. A too strong reference term (λ_ref = 0.1) slows reward learning and lowers all six metrics; its higher LPIPS reflects artifacts rather than useful diversity. The default gives the best PickScore and CLIPScore, with the highest diversity among the well-performing settings.

Policy coefficient β.

βPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedReward
0.122.599 ± 0.0010.2865 ± 0.00010.3363 ± 0.00016.531 ± 0.0041.414 ± 0.0030.7163 ± 0.0026
1 (default)22.423 ± 0.0050.2882 ± 0.00040.3285 ± 0.00016.372 ± 0.0051.334 ± 0.0060.7094 ± 0.0012
1022.607 ± 0.0030.2855 ± 0.00030.3465 ± 0.00026.553 ± 0.0071.430 ± 0.0080.7010 ± 0.0024

RISE-DMD is not sensitive to β. Over two orders of magnitude, the metrics change by at most 0.18 in PickScore and Aesthetic, without a monotonic trend. β = 0.1 and β = 10 are even slightly better than the default on most metrics, so the main results were not obtained by tuning β.

Number of field supervision times.

TimesPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedRewardLPIPSTrain GPU-h
2 (default)22.672 ± 0.0040.2906 ± 0.00030.3432 ± 0.00016.529 ± 0.0041.401 ± 0.0050.7097 ± 0.00130.59447.2
422.743 ± 0.0050.2911 ± 0.00040.3426 ± 0.00026.502 ± 0.0071.432 ± 0.0040.7182 ± 0.00180.56668.0
822.520 ± 0.0070.2832 ± 0.00050.3477 ± 0.00026.648 ± 0.0011.445 ± 0.0060.7107 ± 0.00170.569109.6

More supervision times give modest and mixed changes at a substantially higher cost: four times improve PickScore, CLIPScore, ImageReward and UnifiedReward, and eight times improve HPSv2.1, Aesthetic and ImageReward, while training compute grows by 1.4× and 2.3×. Two supervision times are therefore a cost-effective default.

AuthorsOpenReview ↗

Author Response (3/4)

W2 & Q2: Confidence intervals

We thank the reviewer for this suggestion. We re-generated all main-table checkpoints with three generation seeds and computed paired prompt-level 95% confidence intervals: for each of the 2,048 prompts, the paired differences are averaged over seeds, and prompts are resampled with replacement (10,000 replicates).

MetricRISE-DMD − TDM-R195% CI
PickScore+0.583[0.561, 0.606]
CLIPScore+0.005[0.004, 0.006]
HPSv2.1+0.015[0.014, 0.015]
Aesthetic+0.100[0.089, 0.111]
ImageReward+0.076[0.064, 0.089]
UnifiedReward+0.035[0.032, 0.038]

The Aesthetic gain highlighted by the reviewer is modest but clearly nonzero. Against all five baselines, 29 of 30 intervals are entirely positive; the exception is CLIPScore against DMDR ([−0.0009, +0.0009]), which we report as a tie. The full three-seed main table is:

MethodPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedReward
DMD21.930 ± 0.0080.2832 ± 0.00050.3230 ± 0.00036.091 ± 0.0031.305 ± 0.0070.6901 ± 0.0017
DMD221.970 ± 0.0100.2854 ± 0.00010.3262 ± 0.00026.154 ± 0.0021.315 ± 0.0030.6918 ± 0.0012
DMDR22.818 ± 0.0020.2921 ± 0.00020.3516 ± 0.00026.362 ± 0.0021.458 ± 0.0060.7199 ± 0.0015
RT-DM21.781 ± 0.0060.2842 ± 0.00030.3175 ± 0.00035.986 ± 0.0071.281 ± 0.0020.6845 ± 0.0008
TDM-R122.401 ± 0.0030.2870 ± 0.00030.3383 ± 0.00026.422 ± 0.0041.443 ± 0.0030.6894 ± 0.0013
RISE-DMD22.984 ± 0.0110.2921 ± 0.00030.3532 ± 0.00026.522 ± 0.0041.519 ± 0.0040.7247 ± 0.0011
Reviewer_3XBZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RISE-DMD, a framework for optimizing few-step diffusion generators using black-box scalar rewards. It presents a Reward Score model that learns reward-improved marginal velocity fields via positive–negative flow matching; these learned fields then guide DMD updates within a self-evolving training loop. The paper also offers a theoretical interpretation based on reward-tilted distributions and KL regularization. Experiments on SD3.5 Medium, Sana 1.6B, and DrawBench show gains in generation quality and preference metrics. An OCR experiment further confirms the method’s ability to optimize non-differentiable rewards.

Critical strengths
  • The paper has a solid theoretical foundation and is clearly presented. It does a good job combining DMD and DiffusionNFT into a novel framework, with well-worked-out mathematical derivations to support it. The overall framework is illustrated clearly, and the paper is well organized and easy to follow.

  • The experimental results are convincing. The paper shows consistent improvements across multiple evaluation metrics and model backbones. In particular, the OCR experiment shows a big improvement, from around 0.4 to nearly 0.9, which really highlights how effective the proposed method is for black-box reward optimization.

Critical weaknesses
  • First, there’s a gap between the theoretical analysis and the practical implementation. The mathematical derivations are well developed, but they rely on somewhat idealized assumptions—especially the assumption that the previous Reward Score EMA accurately represents the current student distribution. This may not really hold during iterative training, and its practical impact isn’t sufficiently investigated.

  • Second, the experimental analysis of stability and key components feels insufficient. The OCR experiment shows pretty severe reward collapse under one EMA setting, which raises concerns about training stability. More thorough ablations, multiple-seed evaluations, and computational cost comparisons would make the reported improvements more reliable.

Questions for authors
  • What is the main motivation for using the Reward Score EMA rather than the Fake Score as the previous-field estimate?

  • How important is updating the Reward Score at every iteration? Would less frequent updates achieve comparable performance?

  • In the OCR experiment, how does the improvement in OCR accuracy affect other generation quality metrics?

5 comments
AuthorsOpenReview ↗

Author Response (1/4)

We sincerely thank the reviewer for the positive assessment and the constructive comments. The questions on the previous-field estimate, the update frequency and the OCR trade-offs were particularly helpful: they led us to run a lower-frequency Reward Score variant, and to examine the gap between the analysis and the practical algorithm more carefully. We address each point below. Unless stated otherwise, all numbers are means ± std over three generation seeds.

W1: Gap between theory and practice

We agree that the assumption that the previous Reward Score EMA equals the current student field holds only approximately in practice, and we thank the reviewer for pointing this out. We address the gap in two ways:

  • Bounded effect. App. A.8 already quantifies the effect of this mismatch. If the previous field hh deviates from the student field vϕv_\phi, the population-optimal Reward Score target changes by at most β+1β+λref∥h−vϕ∥\frac{\beta+1}{\beta+\lambda_{\rm ref}}\|h-v_\phi\|, i.e. about 2∥h−vϕ∥2\|h-v_\phi\| in our setting. The error enters the target in a controlled way and is not amplified.
  • Measured directly. We probed the EMA and the Fake Score on student samples from held-out prompts. In most checkpoints, the Fake Score is the closer estimate of the student's marginal field. Thus the EMA is not used because it is a more accurate estimate; Q1 explains why it is nevertheless the better previous field.

We will mark the ideal-field assumption explicitly as an idealization in App. A.8, and add the probe results.

AuthorsOpenReview ↗

Author Response (2/4)

W2: Stability, ablations, seeds and compute

Stability. The OCR collapse in Fig. 9 occurs with the fast Type-1 EMA. DiffusionNFT, from which both schedules are inherited, reports the same instability for this schedule on OCR (its App. C). We find that a longer DMD warm-up substantially improves stability: keeping all other settings unchanged and extending the warm-up from 1,500 to 4,000 steps, the Type-1 run improves steadily to an online OCR reward of about 0.95, and the stable interval grows from about 300 to about 950 updates. The slower Type-2 schedule remains stable in Fig. 9.

Ablations and seeds. The paper already ablates the previous field (Fig. 4), the reward-field objective (Fig. 5) and the reward normalization (Table 5). The new experiments below add the update frequency (Q2) and OCR baselines (Q3). All main results are now reported with three generation seeds on the 2,048 Pick-a-Pic test prompts:

MethodPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedReward
DMD21.930 ± 0.0080.2832 ± 0.00050.3230 ± 0.00036.091 ± 0.0031.305 ± 0.0070.6901 ± 0.0017
DMD221.970 ± 0.0100.2854 ± 0.00010.3262 ± 0.00026.154 ± 0.0021.315 ± 0.0030.6918 ± 0.0012
DMDR22.818 ± 0.0020.2921 ± 0.00020.3516 ± 0.00026.362 ± 0.0021.458 ± 0.0060.7199 ± 0.0015
RT-DM21.781 ± 0.0060.2842 ± 0.00030.3175 ± 0.00035.986 ± 0.0071.281 ± 0.0020.6845 ± 0.0008
TDM-R122.401 ± 0.0030.2870 ± 0.00030.3383 ± 0.00026.422 ± 0.0041.443 ± 0.0030.6894 ± 0.0013
RISE-DMD22.984 ± 0.0110.2921 ± 0.00030.3532 ± 0.00026.522 ± 0.0041.519 ± 0.0040.7247 ± 0.0011

Paired prompt-level 95% bootstrap CIs (RISE-DMD minus each baseline) are entirely positive in 29 of 30 comparisons; the exception is CLIPScore against DMDR ([−0.0009, +0.0009]).

Compute. RISE-DMD takes 212.6 GPU-hours on 8×A100 for 900 updates, including Reward Score and Fake Score training. We also compare with a two-stage baseline at a matched budget: DiffusionNFT is applied to the teacher with the same reward, and the result is distilled into a four-step student with DMD.

ModelSamplingPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedRewardGPU-h (8×A100)
DiffusionNFT teacher (190 updates)25 steps21.970 ± 0.0020.305 ± 0.0000.324 ± 0.0005.971 ± 0.0061.491 ± 0.0000.686 ± 0.001212.0
DiffusionNFT → DMD (+4,000 steps)4 NFE21.935 ± 0.0090.308 ± 0.0000.335 ± 0.0005.777 ± 0.0041.459 ± 0.0030.690 ± 0.001212.0 + 17.7
RISE-DMD (900 updates)4 NFE22.984 ± 0.0110.292 ± 0.0000.353 ± 0.0006.522 ± 0.0041.519 ± 0.0040.725 ± 0.001212.6

RISE-DMD is better on five of six metrics at a lower total cost.

AuthorsOpenReview ↗

Author Response (3/4)

Q1: Why the Reward Score EMA rather than the Fake Score as the previous field

Motivation. If DMD fits accurately, the student matches the previous Reward Score EMA, so the EMA is itself the current student field and can directly serve as voldv_{\rm old} in the next round. The Fake Score, in contrast, has to re-learn the student field from fresh samples in every round, which introduces an additional fitting error into the construction of v±v^\pm (App. A.8).

Empirical evidence. The ablation in Fig. 4 compares the two choices on Pick-a-Pic over 200 updates from the same checkpoint. Values are means over the first → last 20 updates.

voldv_{\rm old}Online rewardReward-DMD field change
Reward Score EMA1.44 → 1.500.117 → 0.203
Fake Score1.44 → 1.450.031 → 0.032

With the Fake Score as voldv_{\rm old}, the Reward-DMD field change that drives the student update is lower than with the EMA at 199 of 200 steps and stays almost flat, and the reward rises far more slowly. This suggests that the Fake Score, which closely tracks the current student (consistent with the probe in W1), repeatedly pulls the Reward Score back towards the student field, so the reward-driven correction is reset in every round. With the EMA as voldv_{\rm old}, the correction accumulates across rounds and the reward update direction stays more consistent.

Q2: Less frequent Reward Score updates

We thank the reviewer for this insightful question, and have run the experiment on OCR. In the low-frequency variant, each cycle generates three fresh rollout buffers. Only the first is scored and used to update the Reward Score, while all three update the Fake Score and the student. All runs start from the same DMD checkpoint.

Reward Score updateReward / Student updatesTrain GPU-hOCRPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedReward
Every step400 / 40095.40.925 ± 0.00121.863 ± 0.0100.2835 ± 0.00020.3164 ± 0.00026.272 ± 0.0041.311 ± 0.0050.6806 ± 0.0011
Every step800 / 800190.80.943 ± 0.00221.796 ± 0.0080.2852 ± 0.00020.3004 ± 0.00016.274 ± 0.0031.210 ± 0.0060.6677 ± 0.0011
Every 3rd rollout400 / 1,200208.10.933 ± 0.00422.131 ± 0.0030.2813 ± 0.00040.3179 ± 0.00006.473 ± 0.0041.337 ± 0.0060.6915 ± 0.0012
  • Same number of global steps. With the same 400 Reward Score updates, the low-frequency variant improves OCR (+0.008) and five of the six general-quality metrics, with a slightly lower CLIPScore.
  • Comparable training cost. At a comparable cost, the every-step run completes 800 updates. It reaches a slightly higher OCR score (+0.010), but loses general quality relative to the low-frequency variant (HPSv2.1 −0.018, ImageReward −0.127, UnifiedReward −0.024). The low-frequency variant therefore achieves a better balance between the task reward and image quality.

Updating the Reward Score at every step is thus not necessary. A slowly updated Reward Score can be projected into the student several times, which makes better use of each reward evaluation and keeps the target more stable, in line with Q1.

Reviewer_hG2EOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method to learn a reward-improved velocity field from a black-box feedback that distills a multi-step diffusion model into a few-step generator. I think this kind of work is important, especially considering the downstream tasks while performing the optimization.

P.S. I have to admit that I am not familiar with this topic, so my review might not be accurate to help assess and improve the submission.

Critical strengths
  • The motivation is clearly stated.
  • experiment is thorough and show the improved performance of the proposed method.
Critical weaknesses
  • The meaning of some figure is difficult to understand.

    • Fig. 3: I wonder what this figure trying to convey? Should we only evaluate the quality of each generated image on its own, or we should trying to see how the generated images match the one generated by the teacher generator?
  • Lack of enough qualitative results.

    • Overall, I think the evaluation replies on mostly quantitative results, which imho sometimes quite misleading. These metrics are mostly just a proxy for real human perceptions. Therefore, I think the authors will need to provide more qualitative results, performing user evaluations, and discuss the failure cases on qualitative results more thoroughly.
    • Following the above points, I am confused how exactly we should evaluate the results. I think it is important to evaluate the performance by referring to the teacher generators. But I think this is not presented in the current paper.
Questions for authors
  • While in intro, the authors mentioned that it is important to consider about downstream tasks. But across the method and experiment, I did not see it is considered properly. I am not sure what exactly the downstream tasks the authors mentioned and how they incorporate it?
    • In my humble opinion, using those common reward functions PickScore, HPS, CLIP-score does not counted as "considering downstream tasks" since they are all just trained model which served as pseudo target reward and not really black-box function people usually discussed.
Reviewer_fE5WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RISE-DMD, a reward fine-tuning stage for DMD-distilled few-step generators that requires only scalar (black-box) rewards. In each round, the four-step student generates on-policy samples, which are then scored. A Reward Score velocity field is trained on these samples using DiffusionNFT's positive–negative flow-matching objective, together with a regularizer toward the frozen teacher. Its EMA then replaces the teacher score as the target in the DMD update of the student. The paper shows that the optimal Reward Score corresponds to a reward-tilted version of the student's noisy marginals. Starting from a DMD-distilled SD3.5 Medium student, RISE-DMD outperforms DMD, DMD2, DMDR, RT-DM and TDM-R1 on all six reported metrics (e.g., HPSv2.1 rises from 0.323 to 0.353). The paper also reports results on Sana 1.6B, on DrawBench, and with an OCR-based reward.

Critical strengths
  1. The paper gives a clear diagnosis of reward-weighted DMD. Reweighting the original DMD gradient does not amount to matching a reward-tilted target, and the gradient vanishes once the student matches the teacher. RISE-DMD addresses this with a principled target and a clean population-level analysis.

  2. The gains are consistent across settings, including three metrics and one prompt set that were not used for training, and they are obtained with access only to scalar rewards.

Critical weaknesses
  1. The necessity of the on-the-fly design is not established. RISE-DMD still trains a full diffusion velocity field, the Reward Score (presumably initialized from the teacher), with DiffusionNFT-style RL. Its idealized fixed point (App. A.9) is the KL-regularized, reward-tilted teacher distribution p_ref·exp(2r/λ_ref)/Z. The obvious alternative is therefore two-stage: run RL directly on the teacher (e.g., with DiffusionNFT or Flow-GRPO), then distill the result with DMD, or use the RL-tuned teacher as the real score. The paper neither runs nor discusses this baseline, and it shows no advantage of the interleaved design in final quality, compute or stability. Because the core components come from DiffusionNFT and DMD, the contribution hinges on this comparison.

  2. The black-box and stability claims are under-supported. The main training rewards (PickScore, CLIPScore and HPSv2.1) are all differentiable, so the main experiment does not test the black-box setting that motivates the method. The only non-differentiable experiment (OCR) has no baselines. Moreover, Fig. 9 shows that the Type-1 EMA run, for which the main text reports an OCR score of 0.889 after 300 updates, collapses to near-zero reward shortly afterwards. The two EMA schedules ("Type-1" and "Type-2") are never defined.

  3. The experimental protocol is under-specified. Results come from single runs without error bars. The reward-normalization setting used in Table 1 was selected by comparing evaluation metrics (Table 5). The Sana and DrawBench experiments compare only against DMD and RT-DM, and the paper does not state how the baselines were implemented or which rewards they used. Key hyperparameters (β, λ_ref, EMA decay, LoRA rank, Reward Score initialization) and training compute are not reported. Finally, there is no analysis of diversity or reward hacking, even though the target is an exponential tilt of the student distribution.

Questions for authors
  1. Necessity. Could you compare against a two-stage baseline? This baseline would apply DiffusionNFT (or Flow-GRPO) to the teacher with the same rewards and a comparable reference regularization, and then distill the result with DMD (or use it as the DMD real score). The comparison should be at matched total compute, measured in GPU hours and NFEs and including the training of the Reward Score and Fake Score. What concrete advantage does the interleaved design offer: cheaper four-step rollouts, higher reward for the deployed few-step model, or better stability? How is the Reward Score initialized? Can RISE-DMD start from an undistilled student, so that distillation and reward optimization are merged into one stage?

  2. Black-box setting and stability. Could you compare against RT-DM and TDM-R1 on the OCR task, and report final rather than peak results? Please define the two EMA schedules. How often does the collapse in Fig. 9 occur, and what prevents it in the main experiments?

  3. Protocol. Could you report mean ± std over several seeds, explain how the baselines were run and with which rewards, and provide the missing hyperparameters and compute? Could you also add diversity metrics (e.g., FID or per-prompt diversity) as a function of λ_ref?

5 comments
AuthorsOpenReview ↗

Author Response (1/4)

We sincerely thank the reviewer for the careful reading and the insightful, constructive questions. The suggested two-stage comparison and the questions on stability and protocol were particularly helpful: they led us to run the two-stage baseline, add OCR baselines, study the stability behaviour more carefully, and complete the experimental details. We address each point below. Unless stated otherwise, all numbers are means ± std over three generation seeds.

W1 & Q1: Necessity of the interleaved design

Two-stage baseline at a matched budget. We set the budget to the measured cost of the RISE-DMD reward stage, which already includes training the Reward Score and the Fake Score. Within this budget, DiffusionNFT is applied to the SD3.5 Medium teacher with the same reward as RISE-DMD (PickScore / 26 + CLIPScore / 100 + HPSv2.1, following the DiffusionNFT multi-reward setting). The resulting teacher is then used as the DMD real score and distilled into a four-step student for 4,000 steps (per-GPU batch size 12), which is longer than the DMD warm-up of RISE-DMD. This covers both variants suggested by the reviewer. Evaluation uses the same 2,048 Pick-a-Pic test prompts.

ModelSamplingPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedRewardGPU-h (8×A100)
DiffusionNFT teacher (190 updates)25 steps21.970 ± 0.0020.305 ± 0.0000.324 ± 0.0005.971 ± 0.0061.491 ± 0.0000.686 ± 0.001212.0
DiffusionNFT → DMD (+4,000 steps)4 NFE21.935 ± 0.0090.308 ± 0.0000.335 ± 0.0005.777 ± 0.0041.459 ± 0.0030.690 ± 0.001212.0 + 17.7
RISE-DMD (900 updates)4 NFE22.984 ± 0.0110.292 ± 0.0000.353 ± 0.0006.522 ± 0.0041.519 ± 0.0040.725 ± 0.001212.6

RISE-DMD is better on five of six metrics at a lower total cost, while the two-stage student has a higher CLIPScore. We see three concrete advantages of the interleaved design:

  1. No distillation loss. As the first two rows show, distilling the reward-optimized teacher into four steps gives back part of its gain. RISE-DMD optimizes the deployed four-step generator directly.
  2. Cheaper rollouts. A rollout costs 4 NFEs instead of 25 solver steps, so within the same budget RISE-DMD learns from 1,036,800 on-policy rollouts, compared with 218,880 for DiffusionNFT.
  3. The Reward Score is a target, not a sampler. In DiffusionNFT, the trained field is the generator itself, and its errors accumulate along the 25-step trajectory. In RISE-DMD, the Reward Score is only queried at random times to provide the DMD target direction, and its errors are averaged over many DMD updates. Two supervision times per rollout sample are therefore sufficient.

Reward Score initialization. The Reward Score is copied from the Fake Score (not from the teacher) at the end of the DMD warm-up, and its EMA is synchronized at that point. Because the two fields are identical at the start, the initial Reward-DMD gradient is zero, and the student departs smoothly from the DMD solution as the Reward Score learns the reward direction.

Starting from an undistilled student. We appreciate this interesting suggestion. RISE-DMD shifts the student towards the high-reward part of its own four-step distribution, so it needs a student that can already generate in four steps and whose samples include high-reward images. An undistilled model does not have four-step generation ability, so its four-step samples contain essentially no high-reward images, and on-policy rollouts would give the Reward Score little useful signal. The DMD warm-up provides exactly this starting point. Our stability study (W2 & Q2 below) supports this: a longer warm-up, which covers the teacher distribution more completely, substantially improves training stability. We therefore keep DMD distillation as the first stage.

AuthorsOpenReview ↗

Author Response (2/4)

W2 & Q2: Black-box setting, OCR baselines and stability

Black-box setting. Although PickScore, CLIPScore and HPSv2.1 are differentiable, RISE-DMD uses only their scalar outputs and never backpropagates through the reward models. The OCR reward (PaddleOCR with normalized edit distance) is non-differentiable.

OCR baselines (fixed endpoint, not peak). All methods start from the same DMD checkpoint (OCR 0.425) and are trained for 400 updates. OCR is evaluated on 1,018 OCR prompts. The Δ columns are changes on the 2,048 Pick-a-Pic test prompts, and ΔLPIPS measures per-prompt diversity.

MethodOCRΔPickScoreΔHPSv2.1ΔAestheticΔImageRewardΔUnifiedRewardΔLPIPS
RT-DM0.681 ± 0.016−0.010+0.002−0.013+0.059+0.006−0.038
TDM-R10.946 ± 0.004−0.885−0.028+0.039−0.134−0.057−0.059
RISE-DMD0.925 ± 0.001+0.002−0.004+0.214+0.026−0.009−0.009

RISE-DMD comes close to TDM-R1 on OCR while largely preserving general quality and diversity. TDM-R1's OCR gain comes with clear drops in both.

EMA schedules. We apologize for omitting the definitions. The update is θˉi=diθˉi−1+(1−di)θi\bar\theta_i=d_i\bar\theta_{i-1}+(1-d_i)\theta_i, where ii counts Reward Score updates. Type-1 is the DiffusionNFT default, di=min⁡(0.001i,0.5)d_i=\min(0.001i,0.5). Type-2 is the conservative schedule DiffusionNFT uses for OCR: di=0d_i=0 for i≤75i\le75 and di=min⁡(0.0075(i−75),0.999)d_i=\min(0.0075(i-75),0.999) afterwards. DiffusionNFT itself reports that its default schedule "suffers from instability" on OCR (App. C), and its soft-update ablation (Sec. 4.4, Fig. 8) shows that an old policy that follows the online model too quickly destabilizes training.

A longer DMD warm-up substantially improves stability. The OCR run in Fig. 9 starts from a 1,500-step DMD checkpoint. Keeping all other settings unchanged and only extending the warm-up to 4,000 steps, the Type-1 run improves steadily to an online OCR reward of about 0.95, and the stable interval grows from about 300 to about 950 updates. The extra cost is small (17.6 GPU-h, versus 212.6 for the reward stage), because the DMD stage uses a small batch. We attribute this to distribution coverage. After a short warm-up, the student mainly fits the high-probability modes of the teacher, and rare high-reward regions are almost never sampled by on-policy rollouts, so the reward signal cannot point towards them. A longer warm-up covers the teacher distribution more completely, which gives the Reward Score a learnable signal.

Full-run result. As the reviewer requested, we report the full run rather than the peak. Even with the longer warm-up, the Type-1 run collapses after about 950 updates: its online OCR reward over the last 100 of 1,000 updates averages 0.39. The student latent variance stays normal throughout, so this is a reward collapse rather than a degenerate generator. We attribute it to the fast Type-1 EMA. With the decay capped at 0.5, the previous field tracks the online Reward Score too closely, so its errors repeatedly enter v±v^\pm and accumulate. This is consistent with what DiffusionNFT reports for the same schedule on OCR. The slower Type-2 schedule remains stable in our Fig. 9.

Further improvement. Supervising the fields at more times per rollout sample would reduce estimation variance and may further improve long-horizon stability. This adds only field-training cost, since the rollouts remain four-NFE. Our implementation supports it, but the experiment could not be completed within the rebuttal period.

We will add the OCR baselines, the EMA definitions, the warm-up study and the full-run result to the revision.

AuthorsOpenReview ↗

Author Response (3/4)

W3 & Q3: Protocol, hyperparameters, compute and diversity

Mean ± std over seeds. Main-table results over three generation seeds on the 2,048 test prompts:

MethodPickScoreCLIPScoreHPSv2.1AestheticImageRewardUnifiedReward
DMD21.930 ± 0.0080.2832 ± 0.00050.3230 ± 0.00036.091 ± 0.0031.305 ± 0.0070.6901 ± 0.0017
DMD221.970 ± 0.0100.2854 ± 0.00010.3262 ± 0.00026.154 ± 0.0021.315 ± 0.0030.6918 ± 0.0012
DMDR22.818 ± 0.0020.2921 ± 0.00020.3516 ± 0.00026.362 ± 0.0021.458 ± 0.0060.7199 ± 0.0015
RT-DM21.781 ± 0.0060.2842 ± 0.00030.3175 ± 0.00035.986 ± 0.0071.281 ± 0.0020.6845 ± 0.0008
TDM-R122.401 ± 0.0030.2870 ± 0.00030.3383 ± 0.00026.422 ± 0.0041.443 ± 0.0030.6894 ± 0.0013
RISE-DMD22.984 ± 0.0110.2921 ± 0.00030.3532 ± 0.00026.522 ± 0.0041.519 ± 0.0040.7247 ± 0.0011

The seed-to-seed standard deviations are small compared with the differences between methods. RISE-DMD has the highest mean on five of six metrics and ties DMDR on CLIPScore. We will add independent training seeds in the revision.

Choice of reward normalization. DiffusionNFT uses global-std normalization. We normalize within each 24-sample prompt group so that prompts with different reward scales contribute comparably, which we expect to generalize better across prompts. Table 5 shows that the results are not very sensitive to this choice: no configuration is best on all metrics, and all three improve over DMD on all six. We will revise App. A.5 accordingly. On DrawBench, a separate prompt set, RISE-DMD is also best on all six metrics.

Baselines and rewards. All baselines, including those on Sana, share with RISE-DMD the codebase, backbone, LoRA topology, four-step sampler, DMD initialization and training reward. DMDR backpropagates differentiable rewards through DMD; RT-DM reward-weights the per-sample DMD loss; TDM-R1 projects the student with a group-DPO surrogate reward field.

Hyperparameters and compute (to be added to the appendix):

ItemValue
Training rewardPickScore/26 + CLIPScore/100 + HPSv2.1, prompt-centered
β / λ_ref / Reward Score LR1.0 / 0.01 / 5e-5
Advantagegroup std, clip 1.0
Reward Score EMAType-1 (main): min⁡(0.001i,0.5)\min(0.001i,0.5); Type-2 (OCR): 0 for i≤75i\le75, then min⁡(0.0075(i−75),0.999)\min(0.0075(i-75),0.999)
Field time samplingshifted uniform, shift 3 (σ=3u/(1+2u)\sigma=3u/(1+2u), u∼U(0,1)u\sim\mathcal U(0,1)), drawn independently per field
LoRArank 32, alpha 64; separate student / Fake Score / Reward Score adapters
Reward Score init.copied from Fake Score, EMA synchronized
Rollouts per update48 groups × 24 = 1,152, 4-NFE Euler (shift 3)
Per updateone optimizer step each for Fake Score, Reward Score, student; two supervision times per sample
Schedule / computeDMD 1,500 steps + 900 RISE-DMD updates; 212.6 GPU-h on 8×A100

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.