acceptodds
Under review as a conference paper at ICLR 2027

Displacement Matching: One Objective for Reward Learning and Distillation in Few-Step Generators

21 views

Abstract

Few-step generators synthesize images with a handful of deterministic updates, but post-training them faces two difficulties: the deployed sampler lacks a stochastic transition likelihood for standard policy gradients, and reward alignment and distillation lack a unified formulation. We take a distributional view, expressing both forms of supervision as multiplicative tilts of the generator's terminal distribution. Forward noising associates clean outputs with noisy observations, yielding a clean-sample posterior at each noise level. The terminal tilt shifts these posterior means, defining a displacement in the student's clean-prediction space. We introduce *Displacement Matching* (DM), which learns this posterior displacement as an additive prediction update, unifying pure distillation, sequential reward alignment, and joint training in one regression. DM estimates the generator's posterior mean separately from re-noised terminal samples, since a few-step head represents a finite-interval jump and need not equal that mean. The resulting regression accommodates black-box rewards without requiring trajectory likelihoods or differentiation through the rollout. Our population analysis connects these updates to deterministic deployment, establishing first-order reward improvement up to an explicit finite-step residual and a unique teacher fixed point for raw distillation. At matched reward-query budgets, four-step DM improves alignment and distillation quality while preserving fidelity, outperforming SOTA methods with 0.362 vs. 0.345 HPS on SD3.5-M and 0.93 vs. 0.81 GenEval on Z-Image.

Primary area:
generative models
Keywords:
Reinforcement Learning, Diffusion Models, Few-Step Generation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_qznkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Displacement Matching (DM), a post-training method for few-step text-to-image generators, covering both few-step distillation from a multi-step teacher and reward alignment with black-box rewards.

The core contribution is to take a forward-noising based RL method (DiffusionNFT), and naturally combine it with an already forward-noising based few-step distillation method (DMD).

Validation is performed on 4-step distillation, and several rewards (HPSv2.1, CLIPScore, PickScore, GenEval). Baselines consist of representative prior methods on (i) few-step distillation, (ii) reward alignment, (iii) joint few-step distillation and reward alignment. Methods are compared at a matched number of reward queries. The proposed method outperforms all baselines across all rewards.

Critical strengths
  1. Strong empirical results. Across two backbones, DM improves across all trained and held-out rewards over recent few-step RL and joint distillation+RL baselines. It is abundant in breadth, so if the comparison is fair, it would strongly support acceptance.
Critical weaknesses
  1. Clarity. The manuscript is very difficult to follow despite careful reading, to the extent that it is unreadable. To name a few,
  • “Few-step generators lack a learning interface that is both faithful to deployment and practical” (L54–55) conveys nothing beyond the following sentence.
  • I cannot comprehend what the authors are trying to communicate in the first three sentences of Section 2.2 (L124-126).
  • “Clean law” / “clean (probability) law” (L136, L140) is never defined.

The poor ordering of content exacerbates this problem. To name a few,

  • There is no mention of diffusion / flow-matching models in Section 1; the paper dives straight into their few-step distillations.
  • The paper does not introduce its notation for diffusion / flow-matching before Section 2.1.
  • Section 2.2 begins by referencing Section 2.4.
  1. Unclear novelty. As a direct consequence of the absence of a 'Preliminaries' section, well-established results and the paper's contributions are not distinguished, and instead intertwined throughout Section 2. Notably, it does not cite previous work for well-established works (e.g., Section 2.1). Since this paper does not use the exact same notation as prior work it compares against, it is unclear which contributions are novel.

  2. Unclear connection between theory and practial implementation. Several assumptions have not been qualified well enough. For instance, the root-mean-square gap of (P2*) is O(hK)O(h_K), which is not small for 4-step disillation. The GAN-distilled initialization does not guarantee the flow-map assumption (Section 3.3 L320), but performs as well as the flow-distilled version (Table 2).

Questions for authors
  1. Novelty (affects assessment): Which parts of the formulation are novel contributions, and which parts can be deduced directly from prior work? To properly address this question, authors shall properly cite previous works across both reward alignment [1-2] and joint distillation & reward alignment [3-7] literature.

  2. Fairness of evaluation (affects assessment): DiffusionNFT, AdvantageFlow, and Flow-GRPO were designed for multi-step models. Therefore, a more detailed description is granted for the evaluation protocol used to obtain results in Table 2 for these methods.

Results were compared at a matched number of reward queries, but a fair comparison grants a matched amount of wall-clock time. There is only an indirect measurement of wall-clock time on a subset of methods (Table 6). Does DM outperform existing methods when the total compute budget is controlled?

  1. Connection between theory and practical implementation (affects assessment): Given [Weakness 3], which empricial results of DM merit Theorem 1 and the flow-map assumption?

[1] https://openreview.net/forum?id=3mJ9eTU319

[2] https://openreview.net/forum?id=jqsxOdmX88

[3] TDM-R1

[4] GDMD

[5] MeanFlowNFT

[6] DMDR

[7] RTDMD

4 comments
AuthorsOpenReview ↗

Response to Reviewer toU3 (1/2)

We thank the reviewer for recognizing the strong empirical results and for identifying the questions about novelty, evaluation fairness, and the role of the theory.

W1. Presentation

We agree that these passages need clearer wording. The intended meanings are:

  • L54-55: We want to improve the actual few-step sampler using black-box rewards, without requiring trajectory likelihoods or backpropagation through sampling. We agree that the following sentence already explains this point.
  • Section 2.2, L124-126: We want to change the distribution of generated images by changing the model's predictions at each sampling step. Reward alignment favors higher-reward images, while distillation moves this distribution toward the teacher.
  • “Clean law,” L136 and L140: This means the probability distribution of the final generated samples, before adding noise for training.

We have revised Sections 2.1 and 2.2 to clarify these concepts and added an overview figure (Fig. 2).

W2 / Q1. Why DM is more than a combination of reward learning and DMD

The contribution is a correction that adding DMD to an existing reward loss does not provide. We agree that regression, EMA, and distribution matching build on prior work. DM addresses a specific error in using them for few-step reward learning. The model's prediction is trained to make a finite sampling step. The reward update instead needs the average clean output associated with a re-noised input, namely the posterior mean. These quantities need not agree. Eq. (47) identifies the bias caused by using the prediction as the posterior.

DM estimates the posterior from the generator's own samples and uses it in the reward residual $ A(X_0-\hat\mu) $. The same estimate is reused for the teacher update. With an exact posterior, the reward residual has the intended first-order conditional mean. It can be learned from scalar rewards without reward gradients or backpropagation through the sampling process. This is the specific correction and integration that our experiments test.

The cited methods address related goals, but a shared training objective alone does not resolve this issue. The likelihood-estimation view of Choi et al. [1] and the common score-regression view of Lee et al. [2] do not by themselves establish that a finite-step prediction equals its output posterior. TDM-R1, GDMD, MeanFlowNFT, DMDR, and RTDMD already combine distillation and reward optimization. DM's contribution is more specific than performing these tasks together. The matched forward-process control retains the head-substitution bias. Direct reward backpropagation takes another route, but requires differentiable rewards. DM uses a separately estimated posterior to correct the reward residual while retaining support for black-box rewards. Tables 2 and 3 also show that DM improves both trained and held-out rewards over the compared joint methods.

The controlled comparisons show why this distinction matters. With the same EMA and reward processing, changing the reward reference from the head to the posterior improves MNR from 1.441 to 1.568 in Table 4. In our independently rerun joint experiment, both methods also receive the same teacher supervision and posterior-estimator training. The correction improves MNR from 1.482 to 1.572, DPG from 84.22 to 85.69, and ImageReward from 1.4528 to 1.5105. Thus the gains persist when the control already combines reward learning, distillation, and EMA.

The pure-distillation comparison provides additional evidence. DMD2-style + EMA and DM both obtain 0.66 GenEval, while DM improves DPG from 84.21 to 84.90. Our main evidence for the posterior correction is the controlled reward comparison above.

AuthorsOpenReview ↗

Response to Reviewer toU3 (2/2)

Q2. Four-step adaptation and compute

FlowGRPO, DiffusionNFT, and AdvantageFlow are sequential baselines: starting from the same flow-distilled checkpoint as our sequential flow-base DM variant, we apply each method's own reward-training objective with four-step sampling for 1,000 iterations at 512 px. They retain their respective optimization formulations. Table 2 evaluates all methods with the deployed four-step CFG-free sampler. Training settings can differ from evaluation settings. For example, the timed FlowGRPO configuration below uses CFG. The joint baselines instead keep distillation active during reward optimization. We will clarify this distinction in a future revision.

Measured training costs. We measure iteration times at 512 px on eight dedicated H200 GPUs with 1,152 samples per iteration. The following are actual iteration times, without normalizing methods to the same number of update timesteps:

ConfigurationSeconds / iteration
DM, joint79.0
DiffusionNFT, four-step configuration39.7
FlowGRPO, four steps + CFG, β=0.01\beta=0.0157.1
RTDMD, reward backpropagation on67.0
TDM-R174.6
DMDR87.1
GDMD104.3
MeanFlowNFT109.0
AdvantageFlow209.3

The component measurements also give a similar cost comparison to Table 6. When normalized to four update timesteps, the corresponding measurements are 81.3 / 84.1 / 161.5 s for DM / RTDMD / TDM-R1. This normalized comparison is reported separately from actual elapsed training cost.

Joint training at common time budgets. We compare DM with the joint RL-and-distillation baselines on the same held-out validation prompt set, using the same reward metric, MNR = HPS + CLIP + PickScore/26. The time column gives wall-clock hours on eight H200 GPUs. This validation set differs from the evaluation set used for Table 2.

Wall-clock time (h)DMRTDMDTDM-R1MeanFlowNFTGDMDDMDR
1 h1.4661.4511.4431.3891.4111.423
4 h1.5341.4961.4781.4211.4551.452
8 h1.5691.5101.4821.4381.4681.466
12 h1.5601.5201.4741.4531.4731.465
16 h1.5701.5291.4841.4651.4981.475

DM achieves the highest validation MNR among these joint-training methods at every reported common time budget, including 1.534 at 4 hours and 1.570 at 16 hours. Thus its advantage in this comparison persists when methods receive the same estimated elapsed training time, despite DM's higher per-iteration cost than RTDMD and TDM-R1.

W3 / Q3. Connection to four-step sampling and the GAN initialization

Theorem 1 explains when fitting the proposed update changes the final output distribution in the intended direction. Its assumptions allow us to follow that change through the sampling steps. The remaining approximation error can still matter at four steps, so the theorem does not guarantee improvement for every practical model.

The posterior used by DM is defined from the generator's own outputs. We can estimate it whether the starting generator was trained by flow distillation or GAN distillation. The GAN initialization therefore does not invalidate the construction of the reward target. It also does not establish the theorem's flow-map assumption.

The experiments answer the practical questions separately. The controlled comparisons show that the posterior correction improves training. The local response measurements show positive reward changes on the actual four-step sampler. Table 2 shows that DM works from both initializations: the flow-base and GAN-base variants reach HPS 0.358 / 0.360, ImageReward 1.52 / 1.51, and DPG 85.77 / 85.53. Together, these results support the usefulness of the method beyond the idealized assumptions. We do not use the GAN results as evidence that those assumptions hold.

We also examine whether the marginal-distribution agreement required by (P1) holds approximately in this setting. We compare visited sampler states with independently re-noised outputs for both the frozen anchor and the trained student. Using 16,384 prompts per model with separate training and test prompts, a small CNN and a logistic regression classifier achieve 50.1/50.5% test accuracy. These near-chance results provide empirical support for this agreement at the tested checkpoint.

Reference

[1] Choi et al. (2026), likelihood estimation for diffusion RL.

[2] Lee et al. (2026), Reward Score Matching (RSM).

[3] Dong et al. (2026), GDMD

I thank the authors for a detailed response. While reviewing the updated manuscript, I found that the authors have revealed their names and affiliations. As such, I am unable to continue reviewing under the double-blind policy. I have notified the Area Chair and Program Chairs but have not yet received a response. I will update my assessment after receiving a response.

We apologize for the issue. We immediately removed the problematic version, including its revision history, to ensure that it does not influence others’ assessments.

Reviewer_toU3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes Displacement Matching (DM), a regression framework for distillation and reward alignment of deterministic few-step generators. The method samples terminal outputs from a frozen or EMA anchor, re-noises these samples, and separately trains an estimator of the posterior mean induced by the generated distribution. Relative to the anchor prediction, the student learns a distillation increment given by the teacher–student posterior-mean difference and a reward increment given by the advantage-weighted posterior residual. The paper provides a first-order reward-response analysis with an explicit finite-step residual and a teacher fixed-point characterization for the raw population distillation update. Experiments evaluate pure distillation, sequential reward alignment, and joint training on four-step SD3.5-M and 6B Z-Image.

Critical strengths

The paper explicitly distinguishes a finite-step generator’s prediction from the posterior mean associated with its terminal distribution and uses a separately estimated posterior to construct supervision. The derivation that converts scalar rewards into prediction increments through posterior reweighting and conditional-covariance identities is coherent. Regressing increments relative to the anchor prediction also allows distillation and reward supervision to share a training interface suitable for deterministic few-step generators without requiring reward derivatives (Sections 2.2–2.3).

Critical weaknesses

W1. Limited novelty of the proposed displacement-regression framework

The abstract and introduction emphasize a unified displacement-regression objective for distillation and reward alignment, which may give the impression that the underlying optimization formulation is itself substantially novel. However, several key components have clear precedents:

  • DMD and DMD2 already use the teacher–fake score difference as the central distribution-distillation signal. More explicitly, Algorithm2 in DMD and Eq. (6) of rCM constructs a stop-gradient MSE surrogate based on this difference.
  • Consistency Models have already adopted EMA target networks and regression against fixed prediction targets.
  • AdvantageFlow employs advantage-weighted forward-process supervision and rollout regularization for reward optimization.

Although DM is not strictly equivalent to these methods, particularly because it regresses prediction increments at re-noised queries whereas DMD backpropagates through the network call producing the generated sample, the paper does not sufficiently isolate the independent contribution of this distinction. The reported improvements may also partly result from EMA anchoring, target normalization, and other training choices. Therefore the paper should more clearly identify its essential novelty relative to prior work, rather than relying primarily on the unified displacement-regression formulation.

W2. The method presentation is unnecessarily difficult to follow and lacks an overall framework diagram

Section 2 is unnecessarily difficult to follow due to its heavy use of abstract terminology and the introduction of multiple distributions, posterior means, and displacement fields before explaining the actual training procedure. This presentation increases the cognitive burden and makes several relatively straightforward operations appear more complicated than necessary. The paper also lacks a comprehensive method overview diagram.

Questions for authors

Q1. What is the essential algorithmic novelty beyond existing DMD and EMA-based regression methods?

Given that DMD/rCM already use the teacher–fake score difference to construct stop-gradient update targets, and that Consistency Models have adopted EMA-based target regression (see W1), could the authors further clarify the essential algorithmic novelty of DM?

Q2. Could the authors simplify the Method section and provide an overall framework diagram?

Could the authors add an intuitive algorithm overview at the beginning of Section 2 and why posterior displacement is used as an additive update to the finite-step head? Additionally, an overall framework diagram would be helpful to illustrate the roles of the four models, the generation and re-noising of training samples, the construction of distillation and reward targets, the stop-gradient locations, and the update relationship between the Student and EMA Anchor. This would substantially improve the accessibility of the method.

Q3. How well do the theoretical assumptions hold for the practical four-step generator?

Theorem 1 relies on P1, P2*, and the finite-step error εK\varepsilon _ K, but practical four-step students may not satisfy the exact probability-flow-map assumption. In particular, at fixed K=4K=4, the residual O(ηεK)O(\eta\varepsilon _ K) is of the same first order as the ideal reward gain, so reducing η\eta alone does not guarantee reward improvement.

2 comments
AuthorsOpenReview ↗

Response to Reviewer toU3

We thank the reviewer for the detailed comparison with prior work. The key contribution is a correction to the reward update for few-step generators. Our new experiments test whether it still helps when EMA and teacher supervision are already present.

W1 / Q1. What is new beyond DMD, EMA, and forward-process regression?

DM separates the prediction used for sampling from the posterior mean needed for reward learning. A few-step head predicts a finite sampling step and need not equal the posterior mean of its own outputs. Using the head as this mean introduces the reward-update bias identified in Eq. (47). DM estimates the posterior separately and uses the residual A(X0−μ^)A(X_0-\hat\mu), while retaining the anchor prediction as the regression baseline. The same estimate supplies the teacher correction. This construction applies to clean-prediction students, including GAN-distilled ones.

The matched controls isolate the contribution of this correction. Proposition 3 connects the head-based control to matched adaptive AdvantageFlow. With the same EMA and reward normalization, replacing the head reference with the posterior estimate improves MNR from 1.441 to 1.568 and DPG from 85.03 to 85.77. Our independently rerun joint comparison also keeps teacher supervision, EMA, and the posterior-estimator training budget fixed. Changing only the reward residual improves MNR from 1.482 to 1.572 and ImageReward from 1.4528 to 1.5105. These results show that the correction contributes beyond EMA and teacher supervision.

DM also changes where the supervision is applied. DMD applies the teacher/fake score difference through the network call that generated the sample. DM trains the prediction at an independently re-noised terminal sample, using the anchor prediction plus a fixed correction as its target. In Table 4, replacing re-noised queries with visited rollout states reduces MNR from 1.568 to 1.451 and DPG from 85.77 to 85.23. Thus query construction also matters. Although score differences, EMA, and fixed-target regression have precedents, DM combines a separately estimated posterior with this training procedure to support reward and teacher supervision in one loss, without reward derivatives or backpropagation through sampling.

Additional distillation control with EMA

We compare three pure-distillation configurations at the same 1,000-iteration budget.

ConfigurationGenEvalDPG
DMD2-style reference0.6684.11
DMD2-style + EMA0.6684.21
DM0.6684.90

DM and the EMA control both use EMA evaluation weights. The reference uses student weights. Adding EMA improves early GenEval progress, with 0.65 versus 0.60 at iteration 100. At the final budget, DM still improves DPG by 0.69 with comparable GenEval.

The DMD2-style controls start their targets from the student's prediction. They use four trajectory-anchor steps and query the score models after re-noising that prediction. DM starts from the anchor prediction and queries the posterior models at the same re-noised input used to train the student. We call this a DMD2-style controlled adaptation because it is not a reproduction of the full DMD2 system. It provides additional evidence for the update design. The reward-residual comparisons above isolate the posterior correction more directly.

W2 / Q2. Presentation

We have added an overview figure (Fig. 2) and rewritten Sections 2.1 and 2.2 with clearer explanations and lighter notation.

Q3. Assumptions and four-step sampling

The theorem asks whether learning the proposed prediction change produces the intended reward change after all sampling steps are run. It assumes that re-noised training inputs have the same distribution as the sampler's intermediate states, and that each sampling step exactly follows the underlying probability flow. At four steps, the approximation error can be large enough that a smaller update alone does not guarantee reward improvement.

We therefore also measure the effect on the actual four-step sampler. The fitted local responses are 0.423, 0.389, 0.431, and 0.510 for HPS, PickScore, CLIPScore, and combined reward. A positive response means the update increases the normalized reward in these measurements. These results show that the update is useful in practice.

We also examine whether the marginal-distribution agreement required by (P1) holds approximately in this setting. We compare visited sampler states with independently re-noised outputs for both the frozen anchor and the trained student. Using 16,384 prompts per model with separate training and test prompts, a small CNN and a logistic regression classifier achieve 50.1/50.5% test accuracy. These near-chance results provide empirical support for this agreement at the tested checkpoint.

Thank you for the detailed rebuttal and additional experiments.

While the authors provide a DMD2-style controlled comparison, I remain concerned about the novelty of the pure-distillation component. In particular, the final distillation objective remains closely related to the existing DMD/DMD2 stop-gradient surrogate, with the main differences appearing to be incremental modifications, such as EMA-based sampling and anchor-relative regression.

Nevertheless, I recognize the value of providing a unified perspective that brings reward maximization and distillation into a common posterior-displacement framework. The clearer explanations in Section 2 and the newly added framework figure make the method more accessible.

Given these improvements, I am willing to raise my rating.

Reviewer_7oHYOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes displacement matching (DM), which unifies reward alignment and distillation in few-step generators through a single regression objective. DM expresses both forms of supervision as multiplicative tilts of the generator’s terminal distribution and uses forward noising to translate them into posterior-mean displacements. DM learns these displacements as additive prediction updates, estimating the generator’s posterior mean separately from the few-step head. The method accommodates black-box rewards without trajectory likelihoods or differentiation through the rollout. Experiments on four-step SD3.5-M and Z-Image show improvements in training rewards and held-out metrics.

Critical strengths
  1. Unified framework. DM formulates reward alignment and distillation through posterior displacements, supporting pure distillation, sequential alignment, and joint training with one regression objective.
  2. Well-motivated design. DM distinguishes the few-step head’s prediction from the generator’s posterior mean and estimates the latter separately. This choice is motivated by the analysis and supported by the DM-self ablation.
  3. Comprehensive evaluation. Experiments cover SD3.5-M and Z-Image, multiple training modes, and both training rewards and held-out metrics. Ablations and local-response diagnostics further support the proposed design.
Critical weaknesses
  1. The finite-step residual in Theorem 1 is of the same order as the expected reward gain, so the result does not by itself guarantee improvement for four-step sampling. Could the authors extend the local-response diagnostics beyond a linear advantage to the rewards used in training?
  2. The design ablations in Section 3.5 focus on sequential training on SD3.5-M. Additional ablations for joint training and Z-Image would help establish whether separate posterior estimation remains important with teacher supervision and at a larger model scale.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7oHY

We thank the reviewer for the positive assessment and for suggesting tests of the practical scope of the posterior correction.

W1. Response with the actual training rewards

We extend the local-response diagnostic to HPSv2.1, PickScore, CLIPScore, and the actual training combination on the deployed four-step SD3.5-M sampler. The normalization statistics are computed from the same evaluation group's anchor rollouts and frozen during evaluation, with paired initial noise for anchor and updated rollouts. The LoRA update is fitted for 1,500 steps, and we estimate the intercept from $ \eta\in{0.05,0.1,0.15,0.2} $.

RewardFitted response $ b_0 $
HPSv2.10.423
PickScore0.389
CLIPScore0.431
Combined0.510

The positive responses show that the deployed update moves the normalized training rewards in the intended direction beyond the synthetic linear diagnostic. These measurements directly address practical realization, while retaining the finite-step qualification in Theorem 1.

W2. Posterior correction with teacher supervision

We independently rerun a matched joint-training control on SD3.5-M. The table below reports these new runs. Both arms use the same teacher branch, EMA, posterior-estimator training, and other training settings. Only the reward-branch reference changes:

Δcontrol=λRA(X0−fanchor)+λD(μT−μ^)\Delta_{\mathrm{control}}=\lambda_R A(X_0-f_{\mathrm{anchor}})+\lambda_D(\mu_T-\hat{\mu})

ΔDM=λRA(X0−μ^)+λD(μT−μ^)\Delta_{\mathrm{DM}}=\lambda_R A(X_0-\hat{\mu})+\lambda_D(\mu_T-\hat{\mu})

The branch weights and teacher-update schedule are identical between the two arms.

Joint variantMNRDPGImageRewardAesthetic
Reward-self control1.48284.221.45286.385
DM1.57285.691.51056.536

Here MNR is HPS + CLIP + PickScore/26. DM improves MNR by 0.090 and DPG by 1.47, with gains in both held-out metrics. This directly shows that the posterior correction remains useful when teacher supervision is already present.

For larger-scale applicability, Table 3 provides complementary evidence on 6B Z-Image: DM reaches 0.93 GenEval and 1.55 ImageReward, compared with best reported baseline values of 0.81 and 1.44, respectively. The new joint ablation isolates the design contribution on SD3.5-M. The Z-Image comparison shows the effectiveness of the full method at larger scale.

Thanks for your reply! I have no further questions, as I have already given a relatively positive score, I decide to maintain my score.

Reviewer_1n1zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Displacement Matching (DM), a regression framework for reward alignment and distillation of deterministic few-step generators. Terminal-distribution tilts are represented through posterior-mean displacements obtained by forward noising. The key distinction is that a finite-step prediction head need not equal the posterior mean of its terminal distribution. DM estimates this posterior separately and learns an additive update relative to the rollout head, accommodating both scalar rewards and teacher supervision. The paper provides conditional population-level analysis and evaluates pure distillation, sequential alignment, and joint training on SD3.5-M and Z-Image, together with mechanism ablations and local-response diagnostics.

Critical strengths
  1. The posterior–head distinction identifies a meaningful source of bias, and the paper provides a convincing matched control. Appendix F establishes the exact equivalence between DM-self and matched adaptive AdvantageFlow, then identifies the correction introduced by separately estimating the posterior mean. Table 4 isolates this substitution: replacing the posterior estimate with the frozen prediction head reduces MNR from 1.568 to 1.441 and DPG from 85.77 to 85.03. This combination of analysis and controlled comparison substantiates the central methodological contribution.

  2. The empirical results demonstrate useful breadth and improvements beyond the optimized rewards. The framework supports three training modes across two backbones and also works from an adversarially distilled initialization. On SD3.5-M, joint training improves HPS from the strongest reported baseline’s 0.345 to 0.362, while also improving held-out ImageReward from 1.39 to 1.53. The common initialization for sequential comparisons and the reported training-cost analysis strengthen the practical evidence. Together, these results support the usefulness of the shared scalar/vector regression interface.

Critical weaknesses
  1. The deployment analysis and local diagnostics do not yet establish how accurately DM realizes the intended update in the practical four-step setting. Theorem 1 assumes exact forward and flow consistency, and the residual O(ηεK)O(\eta\varepsilon_K) remains first-order in the update size. Consequently, taking a smaller η\eta does not alone ensure that the residual is small relative to the predicted improvement at fixed K=4K=4. The measured response of 0.86±0.040.86\pm0.04 is encouraging, but does not separate finite-step, posterior-estimation, and optimization errors. Additionally, the claimed quantitative prediction in Fig. 2b needs clarification: agreement is highlighted at ϵ=0.1\epsilon=0.1, while the plotted errors at larger corruption strengths remain approximately similar. The prediction’s derivation and validity range are not sufficiently explained to assess this evidence.

  2. Important details needed to assess the headline Z-Image comparison are unspecified. Appendix A.3 introduces a “GenEval training prompt set,” whereas evaluation uses the official benchmark prompts, without documenting training-set construction or overlap checks. GenEval is explicitly a training reward here, so separation of evaluation prompts is particularly important. Moreover, all baselines reportedly follow the same curriculum containing non-differentiable GenEval feedback, although Section 4 describes the default DMDR implementation as requiring differentiable rewards. The adaptations and reward objectives used for these baselines must be specified. These omissions do not establish an unfair comparison, but they limit confidence in interpreting the reported 0.93 versus 0.81 improvement.

Questions for authors
  1. Local-response clarification: Please derive the prediction t21−tϵ\frac{t^2}{1-t}\epsilon shown in Fig. 2b and specify its validity range. How should the approximately similar errors at ϵ=0.25\epsilon=0.25 and 0.50.5 be interpreted relative to this prediction? Please also define the plotted error and uncertainty, and specify the update-fitting and advantage-normalization protocol.

  2. Evaluation clarification: How is the GenEval training prompt set constructed? Please provide its size and the checks used to establish separation from the official evaluation prompts. This would clarify whether the reported result measures generalization to unseen prompts under the optimized evaluator.

  3. Baseline clarification: How is non-differentiable GenEval feedback incorporated into each Z-Image baseline, particularly DMDR? Please specify any objective changes, omitted reward components, or surrogate models, together with the reward aggregation and query-budget accounting.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1n1z (1/2)

We thank the reviewer for recognizing the controlled comparison of posterior choices and for the specific questions about response accuracy and evaluation.

W1. Reward changes in the four-step sampler

Theorem 1 describes the reward change for a small update and includes an error from using a finite number of sampling steps. At fixed $ K $, reducing $ \eta $ alone does not make this error negligible relative to the expected gain. We therefore measure the reward change on the actual sampler.

We extend the diagnostic beyond the original linear advantage to all training rewards. The new experiment uses the four-step SD3.5-M DM. We fit the response intercept from $ \eta\in{0.05,0.10,0.15,0.20} $.

RewardFitted response $ b_0 $
HPSv2.10.423
PickScore0.389
CLIPScore0.431
Combined training reward0.510

All four fitted response intercepts are positive, providing empirical evidence that the deployed parameter update moves the normalized training rewards in the intended direction beyond the synthetic linear diagnostic.

We also compare the regression budget and posterior choice in an experiment using four update sizes:

ConfigurationCombined-reward $ b_0 $
Estimated posterior (1,500 regression steps)0.510
Estimated posterior (3,000 regression steps)0.527
Anchor head as posterior (1,500 regression steps)0.254

Doubling the regression budget changes the response only slightly, while using the anchor head approximately halves it. Together with Table 4, this shows that posterior choice matters for both the measured response and training performance.

Q1. Posterior-error diagnostic: prediction and measurement

We address Q1 in Response (2/2) due to character limit.

W2 / Q2. GenEval training prompts and overlap check

We use the GenEval-style training file released with DiffusionNFT, “dataset/geneval/train_metadata.jsonl.” It contains 50,000 entries and 33,199 unique prompts. Evaluation uses the 553 official GenEval prompts, with four generated samples per prompt.

After lowercasing and removing punctuation, 0 / 553 evaluation prompts overlap with the training prompts. This check establishes separation at the normalized-text level. GenEval is used as a training reward, so the result measures performance on unseen prompt texts under an evaluator used during training. We distinguish this from ImageReward and Aesthetic, which are not optimized during training.

Q3. GenEval feedback in the Z-Image baselines

Our DMDR baseline uses its Flow-GRPO-based variant, which accepts scalar GenEval feedback without differentiating through the evaluator. GDMD likewise uses reward scores to weight its optimization objective without requiring reward derivatives. Thus non-differentiability does not prevent either baseline from using GenEval. The stage-2 reward is the sum of the specified reward components, and we count each reward-evaluation invocation as one query. We will clarify these details in a future revision.

AuthorsOpenReview ↗

Response to Reviewer 1n1z (2/2)

Q1. Posterior-error diagnostic: prediction and measurement

The experiment uses a linear-advantage JVP target. Under Xt=(1−t)X0+tεX_t=(1-t)X_0+t\varepsilon, Cov⁡(X0∣Xt=x,c) u=t21−tJμ(x,c)u.\operatorname{Cov}(X_0\mid X_t=x,c)\,u=\frac{t^2}{1-t}J_\mu(x,c)u. Writing sGs_G for the training-group standard deviation, the implemented diagnostic direction is d0=t2(1−t)sGJ^μud_0=\frac{t^2}{(1-t)s_G}\widehat J_\mu u, estimated by finite differences. The controlled corruption μ^ϵ=(1−ϵ)μ^+ϵx\hat\mu_\epsilon=(1-\epsilon)\hat\mu+\epsilon x gives dϵ=(1−ϵ)d0+ϵt21−tusG.d_\epsilon=(1-\epsilon)d_0+\epsilon\frac{t^2}{1-t}\frac{u}{s_G}. the added target component has coefficient 0.1×0.92/(1−0.9)=0.810.1\times0.9^2/(1-0.9)=0.81. This is the 0.81 in the submission: it is the coefficient of the injected component in the direction, not a prediction of the deployed response. Predicting the response requires an additional step, namely how the target change passes through the fitted model and the sampler, which the submission did not account for. The sentence "9× response error … 0.81 measured vs. 0.81 predicted" therefore compared target-level quantities and described them with response terminology; the 9× and 4σ figures refer to the target-level change. We will remove them from the response discussion and replace them with the response-level sweep below. For the same reason, the single-strength agreement in the original panel does not by itself establish proportionality across corruption strengths.

To measure this effect directly, we add the diagnostic direction to the selected level's anchor prediction in the four-step deterministic sampler, without fitting a LoRA update. This new experiment uses a different checkpoint and procedure from the original Fig. 2b (Fig. 3b in the revised manuscript). For this new experiment at t=0.9t=0.9, the response to the calibrated direction u/sGu/s_G is bu=0.075b_u=0.075, and the uncorrupted reference is b0(0)=0.126b_0(0)=0.126. Here uu has unit norm and sGs_G is the anchor-group standard deviation of ⟨X0,u⟩\langle X_0,u\rangle. The calibrated local prediction is Δb0(ϵ)≈ϵ[t21−tbu−b0(0)]=0.4815ϵ.\Delta b_0(\epsilon)\approx\epsilon\left[\frac{t^2}{1-t}b_u-b_0(0)\right]=0.4815\epsilon.

Corruption ϵ\epsilonPredicted excessMeasured direct-field excess
0.100.04820.048
0.250.12040.119
0.500.24080.236

The agreement extends across the tested corruption strengths, including ϵ=0.5\epsilon=0.5. The regression-based calculation gives the same increasing trend: its intercepts are 0.106, 0.126, 0.148, 0.217, and 0.362 at ϵ=0,0.05,0.1,0.25,0.5\epsilon=0,0.05,0.1,0.25,0.5, respectively.

The target changes linearly with ϵ\epsilon, but separately fitted models need not reproduce those changes equally well. The original points alone do not explain why the larger-ϵ\epsilon responses are similar. The new experiment measures the sampler's response to a directly added direction. It tests the prediction under that controlled setting without assigning a cause to the original behavior. We will clarify this distinction in a future revision.

A second comparison shows why denoising loss alone is not enough to explain the reward response. In the new experiment, replacing the fake adapter by the base model raises sampled DSM loss from 0.237 to 0.560, while the regression-based response remains 0.106. By contrast, ϵ=0.1\epsilon=0.1 raises sampled DSM loss only to 0.257, yet changes response to 0.148. Thus a larger denoising loss increase need not produce a larger response change: the direction of posterior error matters.

Definitions and uncertainty. The signed response is S(η)=1ηmean⁡[Rθ(η)−gmgs],S(\eta)=\frac{1}{\eta}\operatorname{mean}\left[\frac{R_{\theta(\eta)}-g_m}{g_s}\right], where gm,gsg_m,g_s are the mean and standard deviation computed from the same evaluation group's anchor rollouts and are held fixed across update strengths. Anchor and updated rollouts use paired initial noise. Because gmg_m is the mean of these same anchor rewards, the group-averaged expression equals the paired updated-minus-anchor reward difference divided by ηgs\eta g_s. We fit S(η)=b0+b1ηS(\eta)=b_0+b_1\eta. “Excess” means the corrupted intercept minus the corresponding uncorrupted reference. Here b0b_0 is the signed response intercept. The absolute response error is ∣b0−1∣|b_0-1|, which is a different quantity.

The original experiment uses six step sizes from 0.025 to 0.2. The new experiment with a directly added direction uses η∈{0.05,0.1,0.15,0.2}\eta\in\{0.05,0.1,0.15,0.2\}. At each step size, SE is estimated with 2,000 bootstrap resamples of 96 prompt groups, preserving the paired anchor/updated rollouts. Intercept SE is calculated by propagating these pointwise uncertainties through diagonal weighted least squares. Covariance across step sizes is not included. Error bars denote SE, and any reported 95% interval is ±1.96 SE\pm1.96\,\mathrm{SE}. The original detection threshold is the reference intercept plus three SE.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful reviews. We appreciate their recognition of the following strengths:

  • The distinction between the posterior and the prediction head identifies “a meaningful source of bias”, supported by “a convincing matched control” (1n1z).
  • The combination of analysis and controlled comparison “substantiates the central methodological contribution” (1n1z).
  • DM offers a “Unified framework” and “Well-motivated design”, supporting pure distillation, sequential alignment, and joint training with one regression objective (7oHY).
  • The derivation connecting scalar rewards to prediction increments “is coherent”, and the shared regression interface supports deterministic few-step generators without requiring reward derivatives (toU3).
  • The experiments demonstrate “useful breadth and improvements beyond the optimized rewards” (1n1z). qznk also highlights “Strong empirical results” across two backbones and both trained and held-out rewards.

Our responses focus on the contribution of the posterior correction, its effect on the actual four-step sampler, and the fairness of the comparisons.

DM addresses a specific problem that simply combining reward learning and distillation does not resolve. A few-step model's prediction tells the sampler how to make its next step. It need not equal the average clean output associated with a re-noised input, which is the posterior mean needed for the reward update. Using the prediction in its place introduces the bias identified in Eq. (47). DM estimates this posterior separately and uses it to correct the reward update. The same estimate also supports distillation. This gives a shared training objective that accepts black-box rewards without differentiating the reward or backpropagating through the sampling process.

The controlled experiments show that this correction matters. In the original sequential comparison, changing only the posterior reference improves MNR from 1.441 to 1.568. Our independently rerun joint comparison keeps the same EMA, teacher supervision, and posterior-estimator training in both methods. The correction improves MNR from 1.482 to 1.572, DPG from 84.22 to 85.69, and ImageReward from 1.4528 to 1.5105. These gains remain after the control already has EMA and a teacher loss.

The new measurements also address practical effectiveness and compute. The four-step sampler gives positive local responses for HPSv2.1, PickScore, CLIPScore, and their combination. On the same held-out validation set, DM achieves higher reward than the compared joint-training baselines at every reported time budget. We also clarify the GenEval prompt separation and how the baselines use its non-differentiable feedback.

We have rewritten Sections 2.1 and 2.2 to introduce the basic concepts and explain the method more clearly, and added an overview figure. Text revisions are highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.