acceptodds
Under review as a conference paper at ICLR 2027

Aligning One-Step Generative Models with Reward-Weighted Transport Distillation

19 views

Abstract

One-step generators enable high-quality visual generation with a single network evaluation, but their post-training is difficult: general implicit generators provide neither tractable likelihoods nor denoising trajectories, and many rewards are non-differentiable. We introduce *Reward-Weighted Transport Distillation* (RWTD), a post-training method that requires only generated samples and scalar reward evaluations. Rather than aligning solely to the conventional reward-tilted reference distribution, RWTD constructs an adaptive target that mixes separately tilted current and reference distributions. The current component incorporates improvements discovered during training, while the reference component anchors the target to the pretrained generator. RWTD realizes this target through feature-space optimal transport and fixed-point regression. Theoretical analysis shows that the fixed-point distributions of RWTD interpolate between off-policy reward tilting of the reference and on-policy tilting of the current model, providing a principled approach to balancing reward adaptation with retention of prior knowledge. Empirically, RWTD substantially improves the GenEval score of the one-step SANA Sprint 1.6B backbone from 0.73 to 0.80, while separate preference alignment experiments demonstrate strong cross-reward generalization that yields balanced improvements and preservation of compositional capabilities.

Primary area:
generative models
Keywords:
t2i, post-training, rlhf, alignment, one-step-generation, sdxl-turbo, sana-sprint

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FicZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a single-step RL fine-tuning method. The main contributions are introduction of mixture target distribution to balance off-policy and on-policy objective function, and Sinkhorn fixed-point iteration update rule, explored in previous one-step generative modeling for RL fine-tuning. The experiment results, though not comprehensive, are reasonable, and some of the proposed components seem to bring additional improvements.

Critical strengths

S1. The use of mixture target distribution is a simple, but effective practice to balance off/on-policy training in RL fine-tuning. Although it does introduce a new hyperparameter ρ\rho which would need further tuning for each application, model, reward, etc.

S2. The problem is timely and practical issue in industry. It can bring significant impacts to the generative modeling community for solving single-step generation and downstread reward alignment.

Critical weaknesses

W1. Novelty. Sinkhorn divergence as the objective of a Wasserstein gradient flow has already been used in prior work [1, 2]. The paper applies it to reward alignment, but its use for generative modeling itself is not new. The authors should position the contribution more precisely with respect to and clarify what is novel beyond the application setting.

W2. The theoretical results may be misleading. The Sinkhorn divergence does not guarantee convergence to the target distribution. As a result, the propositions in the paper do not characterize the method that is actually implemented.

Additionally, the proposed method does not give the same terminal distribution as the original optimal distribution. Then is it fair to compare with the baselines that fit towards the reward-tilted distribution that is different to the proposed method? This connects to W5. The mixed fixed point is a different distribution from a tempered reference tilt in theory. How should we interpret comparing these results?

Additionally, even if empirical performances are better, can we obtain a similar result from other techniques? This connects to W3.

W3. Missing comparison with other methods that mix on-policy and off-policy signals. Several recent RL fine-tuning methods, such as Diffusion-NFT [3] and dual forward KL [4], start from different objectives and derivations but arrive at a similar conclusion: combining samples from the current policy with off-policy samples. Although these methods are not designed for one-step generators, comparing against them, or at least adapting their mixing scheme to the same scheme, would clarify where the proposed method stand in the literature.

W4. Runtime comparisons. While joint reward alignment and distillation can be useful, how does this approach compare with reward alignment then distillation or the other way around? The proposed algorithm seems costly, considering drawing multiple particles from two models, and encoding them to solve Sinkhorn in the latent space. Can the authors show the performances and compute required for each approach and when one can be preferred the other?

W5. The mixture target controlled by ρ\rho. The mixture target is effective, as shown in Tab. 6(b), but comparable performance appears achievable by alternatively tuning the reward temperature β\beta . This does not invalidate the method, but it raises a practical question: why should a practitioner adopt the mixture formulation instead of tuning β\beta directly? Also as mentioned previously, can these be seen as fair comparison while their terminal distributions are different?

[1] Han et al., One-Step Generative Modeling via Wasserstein Gradient Flows

[2] He et al., Sinkhorn-Drifting Generative Models

[3] Zheng et al., DiffusionNFT: Online Diffusion Reinforcement with Forward Process

[4] He et al., Unifying Stable Optimization and Reference Regularization in RLHF

Questions for authors

None. Please see the weaknesses.

Reviewer_HiADOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Reward-Weighted Transport Distillation (RWTD) for aligning one-step image generators using potentially non-differentiable scalar rewards. It separately applies exponential reward weighting to the current and frozen reference distributions, then mixes them into an adaptive target. Entropic optimal transport in a frozen visual feature space constructs detached regression targets, which are used to update generator LoRA parameters. The analysis characterizes fixed points of an idealized distributional recursion and their invariance to partial steps under exact transport. Experiments cover GenEval [4] alignment and separate HPSv2 preference alignment. GenEval training improves SANA Sprint 1.6B [5] from 0.73 to 0.80 and SDXL Turbo from 0.55 to 0.61. The HPSv2 experiments show improvements in the optimized reward alongside the other reported preference and semantic metrics. Ablations examine reference mixing, transport assignments, and sample diversity.

Critical strengths

1. Useful practical results for one-step alignment with non-differentiable rewards. The method requires neither reward gradients nor generator likelihoods, making it applicable to detector-based feedback such as GenEval. Improvements on both backbones exceed the reported DrPO [1] results, with comparable training cost on SANA. The contribution is therefore supported by practical training results in addition to its target-distribution formulation.

2. Ablations provide concrete evidence for transport assignments and reference mixing. Table 3 keeps the reward-weighted target fixed while comparing transport-based and independent assignments. Transport performs better on PickScore and CLIP. Figure 3b and Table 7 show that reference mixing raises LPIPS from 0.414 under purely current-distribution training to 0.580. Although this remains below the base model's 0.652, it supports the role of reference samples in mitigating distributional concentration.

Critical weaknesses

1. It remains unclear whether higher reward scores come at the cost of other capabilities. The GenEval increase from 0.73 to 0.80 shows better performance on that task. However, the paper does not evaluate preference scores on general prompts, image-text alignment, or diversity for this trained model. The relevant evaluations concern a separate model trained with HPSv2 and cannot establish what is preserved after GenEval training.

The HPSv2 comparisons also reach different reward levels. RWTD scores 32.45, while DRaFT [3] and FAV [2] score 37.02 and 35.74. The latter methods optimize the training reward more strongly, which may itself cause larger losses elsewhere. Table 8 includes earlier checkpoints, but their HPSv2 scores are still 34.87 and 34.21. The results therefore do not distinguish better capability retention from simply pushing the training reward less far.

2. A missing control makes it difficult to identify the benefit of adding current-model samples. The central design adds current-model samples to reference-model samples when constructing the training target. Figure 3a shows better GenEval performance than using reference samples alone, reaching 0.80 versus approximately 0.77.

However, the method also has a parameter β that controls how strongly higher-reward samples are favored. The theoretical analysis shows that adding the current-model component also increases the emphasis on high rewards. A relevant control is therefore to use reference samples alone and tune β separately to see whether comparable gains are achievable. Table 6 does not report this comparison. The two constructions need not be mathematically equivalent, but the experiments do not establish how much the proposed design adds over a well-tuned reference target.

Questions for authors
  1. Capability retention after GenEval training. Could you evaluate the same checkpoint that achieves GenEval 0.80 using the paper's existing PartiPrompts preference metrics, CLIP, and per-prompt LPIPS, alongside the base model and corresponding baselines, to determine whether the compositional gain accompanies losses elsewhere?
  2. An independently tuned reference-only target. Could you tune β separately for ρ = 1 and compare its best results with mixed targeting under the same training budget? Please compare preference gains and diversity as well as GenEval, to distinguish adaptive mixing from stronger reward weighting.
  3. Comparison at similar reward gains. When HPSv2 is close to RWTD's 32.45, how do DRaFT [3] and FAV [2] perform on CLIP, PickScore, GenEval, and per-prompt diversity? The earliest saved checkpoints in Table 8 still have higher HPSv2. Could smaller update strengths or earlier checkpoints provide a comparison at similar reward gains?
  4. The role of the partial transport step in actual optimization. Under Equations 9 and 10, constructing targets and taking one regression update at the same parameter point as in Algorithm 1 gives ∇L_η = η∇L_1. Thus η only rescales the gradient, and AdamW need not reduce parameter updates proportionally. What makes η implement the conservative partial transport described in the paper, beyond rescaling the loss? Does the implementation take multiple optimization steps on fixed targets, or use other operations that depend on gradient scale?
1 comment

Dear Reviewer HiAD,

We thank you again for your detailed review. As the rebuttal period only has a few days remaining, we wanted to kindly ask if your concerns have been addressed. We have provided several new results that demonstrate

(1) RWTD's capability retention after GenEval training,

(2) RWTD's improvements relative to tuning β\beta for the reference-only variants, and

(3) RWTD's gains on held-out metrics (i.e. CLIP, PickScore, GenEval) compared to DRaFT and FAV at matched HPSv2 optimization.

We also have appropriately toned down our interpretation of the partial transport step parameter η\eta given your insightful comment in how it rescales the gradient.

We believe these new results and clarifications have addressed the weaknesses and questions that you brought up. If anything remains unclear, we'd be happy to provide further clarification. Thank you for your time!

Reviewer_atTMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reward-Weighted Transport Distillation for post-training implicit one-step generators using samples and scalar rewards. RWTD mixes separately reward-tilted current and reference particles, computes feature-space optimal transport, and regresses the generator toward partial transport targets. It improves SANA Sprint GenEval from 0.73 to 0.80 and reports balanced preference-alignment results.

Critical strengths
  1. RWTD provides a useful black-box interface that does not require generator likelihoods, denoising trajectories, or reward gradients.
  2. Separately tilting and mixing current and reference particles is an intuitive mechanism for balancing adaptation with reference anchoring.
  3. The gains on two one-step backbones are strong, and the paper includes informative target, diversity, hyperparameter, early-stopping, and computational-cost analyses.
Critical weaknesses
  1. The main GenEval experiment optimizes a reward derived from the same detector-and-rule pipeline used for evaluation. Disjoint prompts do not exclude evaluator overfitting, and no human or independent compositional evaluation is provided.
  2. The theory assumes exact distributional updates and optimal transport maps, whereas practical RWTD uses finite particles, entropic feature-space transport, barycentric targets, and neural regression. The fixed-point results therefore do not directly describe the implemented algorithm.
  3. Cross-reward generalization is evaluated only with automated metrics and a single reported run. Multi-seed uncertainty and human preference judgments would strengthen the preservation claim.
Questions for authors
  1. Does the GenEval improvement persist under blinded human evaluation or an independent evaluator that does not reuse the training reward pipeline?
  2. Can the authors empirically connect the practical feature-regression update to the population fixed-point analysis?
  3. Could the authors report multi-seed uncertainty for the main GenEval and preference-alignment results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.