acceptodds
Withdrawn from ICLR 2027

High-Quality Demonstrations in Diffusion Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) improves diffusion models by rewarding generations that satisfy task objectives. Yet task rewards capture only part of visual quality, while the model's generations may continue to fall short of task requirements. High-quality external prompt–image pairs, which we call demonstrations, provide holistic visual supervision beyond what task rewards capture. To incorporate this supervision into RL, we introduce Demonstration-Augmented NFT (Demo-NFT), which combines two sources of image targets in DiffusionNFT's objective: prompt-matched external demonstrations and native samples generated by the diffusion model being optimized. Their rewards jointly determine the advantages used to weight these targets. However, large reward gaps can skew this weighting toward demonstrations. Native-referenced reward capping calibrates demonstration rewards against those of native samples. With 5% demonstration targets, Demo-NFT improves QJudge, an auxiliary metric of image quality, by 8.84 points on average over NFT across FLUX and SD3.5 on compositional generation and text rendering, while achieving better or comparable task performance. Using only the task reward, Demo-NFT also outperforms a multi-reward NFT baseline combining task and HPSv3 rewards on the task metric, QJudge, and PickScore in all four settings. We further examine supervised mixing and shared SFT initialization, and extend the evaluation to Qwen-Image and SenseNova. These results show that Demo-NFT improves multiple aspects of image quality using a single task reward.

Primary area:
generative models
Keywords:
Reinforcement Learning, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PQprOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Demo-NFT extends DiffusionNFT by replacing a small fraction of native training targets with prompt-matched, high-quality GPT-image-2 demonstrations. Both sources share the existing NFT objective and joint advantage estimation. Demonstration rewards are capped at the highest retained native reward for each prompt before computing advantages.

With 5% demonstrations, all four auxiliary metrics improve over NFT across the four main settings, while task scores improve in two settings and decrease by less than 0.01 in the others. Capping generally improves task accuracy but can reduce auxiliary quality: at 5%, it lowers all four auxiliary metrics on both OCR models. Gains in image quality persist after shared SFT initialization, although GenEval2 task scores decrease, indicating a task–quality trade-off. (Tables 1, 4, and 6)

Critical strengths

S1. Consistent auxiliary-quality gains. Demonstrations improve QJudge, HPSv3, PickScore, and Aesthetic over native-only NFT in all four main settings. Relative to Base, QJudge gains are substantial on GenEval2 but modest on OCR, where demonstrations mainly recover quality lost during NFT. Preserving quality while improving task accuracy remains practically valuable. (Table 1)

S2. Relevant controls and calibration analysis. Comparisons with supervised mixing, multi-reward NFT, and shared-SFT initialization examine plausible alternative explanations. Capping improves task scores in six of eight configurations, with negligible differences in the remaining two. Fixed-input diagnostics help isolate its effects on advantages and gradients, although the overall task–quality benefit is not uniform. (Tables 4 and 6; Appendix E)

Critical weaknesses

W1. The incremental mechanism is not sufficiently isolated from demonstration supervision strength. Demo-NFT retains NFT’s objective, adds external image targets, and introduces reward capping. Simplicity is not itself a weakness, but the strongest evidence establishes the benefit of demonstrations rather than a clear advantage of the proposed integration.

The supervised-mixing comparison remains inconclusive because matching demonstration ratios does not match loss or gradient magnitudes, as Appendix D acknowledges. At a matched 10% ratio, DDRL* exceeds Demo-NFT on all five metrics for SD3.5 GenEval2, whereas Demo-NFT achieves higher task scores in the other three settings. A demonstration-loss coefficient sweep would clarify whether these differences arise from the integration strategy or supervision strength. (Tables 1 and 3; Appendix D)

Additionally, 99.95% of demonstration rewards reach the cap in the SD3.5 OCR 5% diagnostic run. This raises the question of whether demonstration-specific reward evaluation is necessary in that regime. It does not imply equivalence to SFT: joint advantages remain dependent on prompt/batch statistics, and the objective remains NFT. (Appendix E)

W2. The evidence supports specific operating points rather than a generally superior task–quality trade-off. Capping can improve task accuracy while reducing auxiliary quality, and continued demonstrations after shared SFT reduce GenEval2 task scores by 0.0344 on FLUX and 0.0228 on SD3.5. These are meaningful trade-offs rather than uniform improvements. (Tables 4 and 6)

The multi-reward baseline uses equal-weight task/HPSv3 advantages without a mixing-weight sweep. Demo-NFT outperforms this configuration on task score, QJudge, and PickScore, but the baseline achieves higher HPSv3. Alternative weights could narrow these differences. Single-run results without significance testing further limit conclusions from small gains. PickScore also truncates OCR prompts, measuring prefix-conditioned preference rather than full-prompt compliance. (Table 1; Appendices A and B)

Questions for authors

Q1. At a matched demonstration ratio, how sensitive is DDRL* versus Demo-NFT to the demonstration-loss coefficient? A small sweep would help isolate integration strategy from supervision strength.

Q2. Where nearly all demonstration rewards reach the cap, does directly assigning the best retained-native reward produce comparable results?

Q3. How sensitive are the comparisons to the task/HPSv3 mixing weight? Please distinguish superiority over the tested equal-weight configuration from superiority over tuned multi-reward mixtures.

Q4. Can repeated runs or uncertainty estimates establish whether the smaller task and auxiliary-metric differences are reliable?

Reviewer_aA5EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose Demo-NFT, which adds high-quality external prompt–image pairs ("demonstrations") as training targets in DiffusionNFT. The DiffusionNFT objective learns from clean images and does not need their sampling trajectories. Demo-NFT replaces a small fraction of native samples with prompt-matched demonstrations and scores both with the task reward. It then limits each demonstration reward to the best native reward for the same prompt ("native-referenced reward capping") and estimates the advantages of both sources together for the shared NFT loss.

Critical strengths
  • The method is simple and practical. The trajectory-free objective of NFT lets external images enter RL directly, without the image inversion that policy-gradient methods need.
  • The paper identifies a real problem - the difference in reward scale between demonstrations and native samples changes the shared advantage baseline.
  • The evaluation is broad. It covers four backbones of different architectures, two tasks with different failure modes, and four quality metrics in addition to the task metric.
Critical weaknesses
  • The main mechanism is weighted supervised fine-tuning, and the paper does not show that it is better than simpler alternatives. The method adds a small fraction of GPT-image-2 images to DiffusionNFT batches and gives each the advantage of the best native sample, so better automated quality scores are expected. The DDRL* baseline (plain supervised flow matching on the same images) gives similar gains (Sec. 5.2). The "Beyond scalar rewards" argument (Sec. 3.3) explains why demonstrations improve quality, but not why Demo-NFT's method (joint advantages plus the cap) is better than simple fine-tuning on the same images. A fixed-advantage variant and the SFT-only checkpoint, which can resolve this, are missing.
  • Several key design choices are heuristic and not justified. 1) DiffusionNFT's derivation assumes the targets are samples from the reference policy (sec 3.3 ): Use of external demonstrations violates this but the paper doesn't discuss how the loss formulation changes or remains same. 2) Unclear difference from supervised fine-tuning (Sec. 3.3): Because the cap almost always applies, demonstrations effectively receive a fixed per-prompt weight, close to weighted SFT as in DDRL*. 3) Unjustified loss normalization (Sec. 3.3): Dividing the MSE by each target's mean absolute error may down-weight demonstrations with large residuals, acting as an implicit second cap. No reason or ablation is given. 4) Unexplained base-model penalty (Sec. 3.3): The penalty toward the frozen base model has no motivation or ablation and may limit the effect of demonstrations.
Questions for authors
  1. How does Demo-NFT compare with a variant that gives each demonstration a fixed advantage (for example, the best native advantage)? How does it compare with SFT-only and DDRL* at matched demonstration ratios, over multiple seeds? If the results are similar, what does joint advantage estimation add?
  2. What does the NFT objective optimize when some targets are not samples of the reference policy? How much weight do demonstrations get on prompts that the model fails consistently?
  3. Why is the best native reward the correct cap level? How do the cap level, the loss normalization and the frozen-base penalty each change the results?
Reviewer_J5tcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Demo-NFT, which incorporates prompt-matched external demonstrations into DiffusionNFT’s training targets. Demonstrations and native samples share the NFT objective and jointly determine advantages. A native-referenced reward cap limits demonstration rewards to the maximum retained native reward. Experiments examine whether this combination improves visual quality while maintaining task performance.

Critical strengths
  1. Practically relevant problem and promising results. Task-specific rewards can overlook visual attributes such as typography and composition. Demo-NFT improves all four auxiliary metrics over native-only NFT across the four main model–task settings, using demonstrations for 5% of training targets.
  2. Useful controls. The shared-SFT initialization experiment helps distinguish continued demonstration use from initialization effects. The ratio and capping ablations also reveal task–quality trade-offs rather than uniformly favorable results.
Critical weaknesses
  1. The benefit of Demo-NFT’s weighting scheme remains unclear. The supervised-mixing baseline uses a fixed loss weight, leaving open whether a tuned baseline could achieve the same gains with the same demonstrations.
  2. The evaluation does not establish reliable improvements. Each configuration uses one training run, and checkpoints are selected on test scores. This can overstate gains and leaves their reproducibility uncertain.
Questions for authors

1.Does Demo-NFT outperform supervised mixing after tuning the supervised-loss coefficient under the same demonstration data, training resources, and validation protocol? Would a simpler fixed demonstration-weighting scheme recover similar performance? 2. Can a stronger multidimensional MLLM judge used as the training reward recover the quality improvements without demonstrations? The task-plus-HPSv3 baseline does not fully address this alternative explanation for the motivation. 3. Do the main conclusions persist across multiple training seeds with validation-based checkpoint selection and an untouched test set? Please report uncertainty for both task and quality metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.