acceptodds
Under review as a conference paper at ICLR 2027

GuidanceNFT: NFT Based Reward Fine-Tuning for Guided Diffusion Models

15 views

Abstract

Guidance and reward fine-tuning improve diffusion generation in complementary ways. Combining the two, however, can undermine both benefits: applying classifier-free guidance after DiffusionNFT (NFT) can reduce the optimized reward, while incorporating guidance during NFT fine-tuning may cause collapse. We identify a mismatch in NFT's regression center as a source of this instability. The center must match the conditional mean velocity obtained by re-noising generated samples; using a mismatched center introduces a bias that can rotate or reverse the reward-driven update. We propose GuidanceNFT, which learns this center online by flow matching on guided samples and corrects both NFT regression terms while preserving the policy's reward-driven displacement. The method supports explicit and intrinsic guidance without additional inference passes. On SD3.5-M, GuidanceNFT improves the optimized reward over standard NFT across three training objectives and outperforms NFT on all seven evaluation metrics for GenEval and OCR specialization. In particular, OCR reward increases from 0.971 to 0.975 while held-out GenEval success rises from 0.035 to 0.488, indicating improved prompt fidelity beyond the training objective. Across ImageNet generation, OpenWebText, and GSM8K reasoning, GuidanceNFT improves generation quality and task performance under intrinsic guidance. These results show that online center correction enables effective reward fine-tuning of guided diffusion models.

Primary area:
generative models
Keywords:
diffusion models, reward fine-tuning, classifier-free guidance, diffusion language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SK2ZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GuidanceNFT, a reward fine-tuning method designed to address the instability of DiffusionNFT when combined with classifier-free guidance (CFG) or built-in guidance mechanisms. The authors identify a mismatch between NFT's regression center and the conditional mean velocity induced by re-noising guided samples, which introduces bias into reward-driven updates and may lead to training collapse. To address this issue, GuidanceNFT trains an auxiliary flow-matching model to estimate and correct the regression center online. Experiments across image generation, language modeling, and mathematical reasoning demonstrate improved training stability and reward optimization, without introducing additional inference passes.

Critical strengths

The paper proposes a simple yet effective extension of DiffusionNFT that improves the stability of reward fine-tuning under guidance. The method is supported by theoretical analysis and experiments across multiple generative tasks, without introducing additional inference overhead.

Critical weaknesses
  1. Additional training overhead. The proposed method requires training an auxiliary flow-matching network to estimate the regression center, introducing substantial computational overhead. The reported training iteration time is approximately 1.73–1.77 times that of standard NFT.
  2. Others see Questions.
Questions for authors
  1. The authors identify regression center mismatch as a key factor contributing to training instability and collapse in guided NFT. However, the experiments primarily demonstrate a correlation between increasing center mismatch and performance degradation, rather than establishing a causal relationship. It is also possible that training instability causes the generated distribution to shift, which in turn increases the center mismatch. Could the authors provide further evidence to clarify whether center mismatch is a primary cause of training collapse or a consequence of it? For example, a controlled experiment that varies the regression center while keeping the rollout distribution fixed would help validate the proposed explanation.
  2. The ImageNet experiments primarily rely on PickScore to evaluate generation quality. However, improvements in preference-based rewards do not necessarily indicate better distributional fidelity or diversity. Could the authors additionally report standard image generation metrics, such as FID and Inception Score (IS), for the pretrained model, NFT, and GuidanceNFT under matched sampling settings? This would help clarify whether the improvements in reward optimization also translate into better overall generation quality.

I would be willing to reconsider my rating if the authors can provide convincing evidence or clarification addressing these concerns.

4 comments

W1. Additional training overhead. The proposed method requires training an auxiliary flow-matching network to estimate the regression center, introducing substantial computational overhead. The reported training iteration time is approximately 1.73–1.77 times that of standard NFT.

Thank you for raising this point. By fitting an auxiliary center network, GuidanceNFT enables stable reward fine-tuning and continued improvement on guided rollouts, whereas NFT peaks and then deteriorates with further training. As shown in the table below, GuidanceNFT surpasses NFT’s peak performance across image and language generation. The additional computation therefore delivers gains that NFT does not achieve with the same training time.

MethodSiT PickScore ↑MG PickScore ↑ELF Entropy / PPL†GSM8K Pass@1 ↑
NFT20.64520.3353.74 / 3.4141.68
GuidanceNFT21.70221.6995.14 / 16.6043.03

† On ELF, GuidanceNFT avoids the entropy collapse that accompanies NFT’s low perplexity.

This extra cost can also be reduced. As shown in our response to Reviewer VkE2 (Q2), using one fitting timestep per center update instead of ten reduces the total training cost from 1.73× to 1.40× that of NFT, while improving CFG-4 GenEval from 0.91 to 0.94.

With the same GPU-time budget on SD3.5-M, GuidanceNFT completes only about 70% or less as many updates as NFT, yet GuidanceNFT achieves higher GenEval and Learned3 rewards and comparable OCR reward, as shown below.

SD3.5-M: matched compute

MethodCFGGenEvalOCRLearned3
NFT10.8980.9721.518
NFT40.6670.1871.495
GuidanceNFT40.9500.9691.523

We also note that GuidanceNFT adds no inference cost, so the additional computation is confined to training.

Q1. Could the authors provide further evidence to clarify whether center mismatch is a primary cause of training collapse or a consequence of it? For example, a controlled experiment that varies the regression center while keeping the rollout distribution fixed would help validate the proposed explanation.

We thank the reviewer for the insightful question. We trained GuidanceNFT and standard NFT on the same CFG-1.5 rollouts sampled from the frozen pretrained SiT. Even with a fixed rollout distribution, standard NFT without a matched center degrades, while our method continues to improve. This shows that rollout-distribution drift is not required for the observed degradation.

SiT-XL/2: fixed-rollout PickScore

CFGMethod@100@150@200
1NFT19.41219.31318.561
1GuidanceNFT19.51319.93420.389
1.5NFT19.84119.84318.944
1.5GuidanceNFT19.98420.33720.691

GuidanceNFT achieves higher PickScores than NFT at both CFG scales and all three evaluated checkpoints.

We also analyzed the center mismatch using fixed rollouts in our toy tree model (see toy-model figure). In most regions, the center bias opposes the intended update with comparable magnitude. Applying this bias alone for ten steps shifts samples away from the original distribution (panel (c)). When added to the intended update, the bias distorts the velocity field, resulting in a distribution with degraded reward (panel(d)).

Together, the fixed-rollout experiment and the bias-only intervention support a causal role for center mismatch in training degradation.

Q2. Could the authors additionally report standard image generation metrics, such as FID and Inception Score (IS), for the pretrained model, NFT, and GuidanceNFT under matched sampling settings? This would help clarify whether the improvements in reward optimization also translate into better overall generation quality.

We thank the reviewer for suggesting the additional evaluation. We report PickScore, FID, and IS at matched training steps and under identical sampling settings. For both SiT and Model Guidance, GuidanceNFT achieves higher PickScore and IS than standard NFT.

Standard FID is computed against the original data distribution, so it is expected to increase during reward fine-tuning, which intentionally shifts the generated distribution toward higher-reward regions. To account for this shift, we also report reward-tilted FID (tFID), computed against the following reward-tilted reference distribution:

pβ(x∣c)  ∝  pdata(x∣c) exp⁡(β z(x,c)).p_\beta(x\mid c)\;\propto\;p_{\rm data}(x\mid c)\,\exp\big(\beta\, z(x,c)\big).

Here, z(x,c)z(x,c) is the PickScore z-score of image xx within class cc.

We report the minimum tilted FID over the evaluated β\beta values. The full curves for comparison at matched β\beta are shown here. NFT has higher tFID than GuidanceNFT at all evaluated β\beta values across the three settings, except at β=0\beta=0 for SiT with CFG 1.5.

SiT-XL/2: generation quality

CFGMethodPickScore ↑IS ↑tFID ↓FID ↓
1Pretrained19.885115.3 ± 4.111.2811.28
NFT20.398111.0 ± 5.617.6417.72
GuidanceNFT20.907200.4 ± 8.87.7311.11
1.5Pretrained20.321218.7 ± 6.34.284.48
NFT20.645181.9 ± 5.411.5813.64
GuidanceNFT21.113270.1 ± 5.26.7613.93

Model Guidance: generation quality

MethodPickScore ↑IS ↑tFID ↓FID ↓
Pretrained20.338240.6 ± 7.23.934.05
NFT20.33594.8 ± 3.830.6130.61
GuidanceNFT21.017231.4 ± 5.313.3017.52

Thank you for the detailed responses and additional experiments. The fixed-rollout experiments provide more convincing evidence for the causal role of center mismatch in training degradation. The additional FID and IS results also strengthen the empirical validation beyond preference-based rewards.

Although the additional training cost remains a limitation, the matched-compute results help justify the overhead. Overall, the authors have addressed most of my concerns, and I am willing to raise my rating to 3 (Weak Accept).

Thank you for your thoughtful review and for raising your score. The fixed-rollout experiment you suggested helped us better show the causal role of center mismatch, and the FID results gave us stronger evidence of generation quality. We are also glad that the equal-compute results addressed your concern about the training overhead.

Reviewer_z3gGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GuidanceNFT to make NFT reward fine-tuning work with guided diffusion models. It learns a regression center from generated endpoints and corrects both NFT branches. Experiments show promising stabilization and reward improvements across image generation, language generation, and reasoning, although several comparisons use different training or evaluation configurations.

Critical strengths
  1. The analytical diagnosis is useful and clearly stated. The fixed-population derivation explains how center error can change the intended update.
  2. The paper proposes a simple and effective fix in the reported image experiments: learn the center online and apply the same correction to both NFT branches.
  3. The experiments cover several guidance mechanisms and assess more than the optimized reward. In particular, the OCR specialist improves separate GenEval success from 0.035 to 0.488 while retaining a high OCR reward, and the language experiments examine diversity rather than treating very low perplexity alone as success.
Critical weaknesses

The practical need for the auxiliary center estimator is not sufficiently established against a simpler CFG-style training baseline. A relevant alternative is ordinary NFT with unguided rollouts, randomly dropping the text condition during the NFT objective to jointly adapt conditional and unconditional predictions, followed by CFG at inference.

Questions for authors

How does GuidanceNFT compare with NFT using conditioning dropout on unguided rollouts and CFG at inference?

3 comments

W1. The practical need for the auxiliary center estimator is not sufficiently established against a simpler CFG-style training baseline. A relevant alternative is ordinary NFT with unguided rollouts, randomly dropping the text condition during the NFT objective to jointly adapt conditional and unconditional predictions, followed by CFG at inference.

Q1. How does GuidanceNFT compare with NFT using conditioning dropout on unguided rollouts and CFG at inference?

Thank you for suggesting this baseline. We tested this variant in three settings. We use unguided rollouts and a default conditioning dropout rate of 0.1 in the NFT objective, followed by CFG at inference. GuidanceNFT generally outperforms conditioning dropout on ELF-B, SiT, and SD3.5-M, as detailed below.

(a) ELF-B, OWT

We first tested this variant on ELF-B using unguided rollouts and dropout rates of 0.1 and 0.5. With a dropout rate of 0.5, training collapses into repetitive text, with 12%–16% of the word 4-grams in each sample being repeats. With a rate of 0.1, it trails GuidanceNFT on both matched-entropy PPL and matched-PPL entropy. The PPL–entropy trade-off curves are shown here. Therefore, we use a dropout rate of 0.1 for the experiments below.

ELF-B: dropout comparison on OWT

MethodMatched entropy = 5.15 nats: PPL ↓Matched PPL = 17: Entropy (nats) ↑
Pretrained24.0not reached
NFT18.25.134
Dropout (0.1)20.55.115
Dropout (0.5)not reachednot reached
GuidanceNFT16.15.176

We compare PPL at matched entropy (5.15 nats) and entropy at matched PPL (17).

(b) SiT experiments

We also tested the dropout variant on SiT. Its reward is lower than that of standard NFT, and its diversity, measured by IS and reward-tilted FID (tFID, details in Appendix F), is also worse. GuidanceNFT outperforms it on every metric.

SiT-XL/2: dropout comparison

MethodCFG 1 PickScore ↑CFG 1 IS ↑CFG 1 tFID ↓CFG 1.5 PickScore ↑CFG 1.5 IS ↑CFG 1.5 tFID ↓
Pretrained19.885115.311.2820.321218.74.28
NFT20.398111.017.6420.645181.911.58
Dropout20.372105.118.9520.630172.712.28
GuidanceNFT20.907200.47.7321.113270.16.76

(c) SD3.5-M

On SD3.5M, we evaluated the dropout variant using OCR. It partially mitigates the collapse of standard NFT at CFG 4, but its training reward is lower than that of standard NFT at CFG 1 and much lower than that of GuidanceNFT. On held-out data, its performance is mixed relative to standard NFT at CFG 4, while GuidanceNFT outperforms the dropout variant on every metric.

SD3.5-M: OCR training

MethodCFGOCR rewardGenEvalPickScoreCLIPHPSv2AestheticImageReward
NFT10.9710.03219.600.2070.1374.21−1.69
NFT40.2300.26120.230.2420.1884.60−0.87
Dropout40.7240.31920.360.2490.1834.46−0.91
GuidanceNFT40.9750.49621.810.2740.2545.11+0.40

Overall, this variant improves NFT's performance with guidance in some cases, but with a decreased training reward. GuidanceNFT achieves better reward and generation quality than conditioning dropout across the evaluated image and language settings.

Thank authors for the response. The additional results have addressed my concern and I hold my rating to weak acceptance.

Thank you for your response and for suggesting the conditioning dropout baseline. We are glad that our comparison with it addressed your concern.

Reviewer_VkE2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the instability of DiffusionNFT. It identifies a mismatch between NFT's regression center and the mean velocity of re-noised guided samples, which can bias or even reverse the intended reward-driven update. The proposed GuidanceNFT uses an auxiliary Flow Matching model to estimate the correct center online and adjust NFT's positive and negative regression terms. Experiments demonstrate improvements across image generation, language generation, and mathematical reasoning.

Critical strengths
  1. The paper identifies a clear and important problem in guided reward fine-tuning. The theoretical analysis explains how regression-center mismatch introduces bias into NFT's gradient.
  2. The proposed solution is well motivated by the analysis. Learning the regression center from guided samples directly addresses the identified mismatch, while preserving NFT's original reward-driven optimisation.
  3. The improvements are demonstrated across multiple settings, including SD3.5, SiT, Model Guidance, language generation, and GSM8K reasoning.
Critical weaknesses

no critical weakness

Questions for authors
  1. Could the authors investigate how perturbing the learned regression center affects training? For example, adding different levels of noise to the estimated center and examining the effects on gradient direction, training stability, and reward improvement would help validate the proposed explanation of center mismatch.
  2. Given the additional training cost reported in Appendix E, it would also be helpful to understand whether less frequent center updates could achieve similar performance.
3 comments

Q1. Could the authors investigate how perturbing the learned regression center affects training? For example, adding different levels of noise to the estimated center and examining the effects on gradient direction, training stability, and reward improvement would help validate the proposed explanation of center mismatch.

Thank you for this suggestion. We tested two ways to perturb the learned regression center.

(i) Noise. As suggested, we added Gaussian noise of different scales to the fitted center. The perturbed gradients remain closely aligned with the GuidanceNFT gradient, but both noise levels reduce the final reward. The training curves are shown here.

SD3.5-M: center noise

MethodGrad. cosineGenEval (CFG 1)GenEval (CFG 4)Mean Δ vs. GuidanceNFT
GuidanceNFT (c = 0)10.75340.81730.00000
GuidanceNFT + noise (c = 0.05)1.0000.65270.7141-0.10195
GuidanceNFT + noise (c = 0.1)0.9990.71130.7691-0.04515
NFT0.7590.56160.6039-0.20260

Noise scale cc is relative to the GuidanceNFT center; Grad. cosine compares parameter gradients with unperturbed GuidanceNFT. Mean Δ is the average difference from GuidanceNFT across CFG 1 and CFG 4.

(ii) Interpolated center. We also controlled the scale of the center mismatch error by using λvold+(1−λ)vϕ\lambda v^{\rm old}+(1-\lambda)v_\phi as the center, where λ=0\lambda=0 is GuidanceNFT and λ=1\lambda=1 is NFT. As λ\lambda decreases, the gradient aligns more closely with the GuidanceNFT gradient, and the reward improves. The training curves are shown here.

SD3.5-M: center interpolation

MethodGrad. cosineGenEval (CFG 1)GenEval (CFG 4)Mean Δ vs. GuidanceNFT
GuidanceNFT (λ = 0)10.75340.81730.00000
Interpolated center (λ = 0.33)0.9670.71040.7567-0.05180
Interpolated center (λ = 0.66)0.8540.64570.7261-0.09945
NFT (λ = 1)0.7590.56160.6039-0.20260

Mean Δ is the average difference from GuidanceNFT across CFG 1 and CFG 4.

Q2. Given the additional training cost reported in Appendix E, it would also be helpful to understand whether less frequent center updates could achieve similar performance.

Thank you for the question. We tested two ways to reduce center-fitting cost: refreshing the center less frequently during training and reducing forward-pass frequency per update.

(i) Reduced center-update frequency. We reduced the center-update frequency during training. Here, a schedule of k/mk/m denotes mm center-fitting optimization steps every kk policy updates. Updating the center every 10 training steps with the same total number of center-fitting optimization steps maintains a high PickScore. However, reducing the number of optimization steps lowers the reward to near the NFT level.

SD3.5-M: refresh schedules (CFG 4)

MethodSchedulePickScore ↑s/updateCost (×)
NFT—22.3081821.00×
GuidanceNFT1/122.5003241.78×
GuidanceNFT10/1022.6933241.78×
GuidanceNFT10/522.3372871.58×
GuidanceNFTFixed22.3262491.37×

(ii) Reduced forward-pass frequency. We also reduced the forward-pass frequency within each center update. Using one diffusion timestep instead of ten reduces the total cost from 1.73× to 1.40×, while improving the reward from 0.91 to 0.94.

SD3.5-M: fitting timesteps

MethodTimestepsGenEval (CFG 1)GenEval (CFG 4)Cost (×)
NFT—0.85850.68191.00×
GuidanceNFT100.88090.90541.73×
GuidanceNFT50.87620.91321.56×
GuidanceNFT10.90630.93961.40×

Given the limited rebuttal period, we will further investigate why fewer fitting timesteps improve the reward—including whether higher gradient variance helps training—and extend this study to other rewards in the revision.

Reviewer_VkE2OpenReview ↗

Official Comment by Reviewer VkE2

Thanks for the additional experiments! The results are interesting, especially that fewer fitting timesteps lead to better performance. It would be great to see more analysis on this. Happy to keep my positive score.

Thank you for your positive feedback and for keeping your score! We are glad the additional experiments on center perturbation and reduced fitting cost were helpful, and we will further analyze the improvement of fewer fitting timesteps.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. In the rebuttal, we addressed their concerns with additional experiments, and all three reviewers now rate the paper as weak accept.

Strengths identified by reviewers

  • The paper identifies a clear and important problem in guided reward fine-tuning, and the analysis explains how regression-center mismatch biases NFT's update (VkE2, z3gG, SK2Z).
  • The proposed fix is simple and effective (z3gG, SK2Z) and well motivated by the analysis (VkE2).
  • Experiments cover image generation, language generation, and reasoning (VkE2, z3gG, SK2Z), and evaluate generation quality beyond the optimized reward, such as held-out GenEval and text diversity (z3gG). The method adds no inference cost (SK2Z).

Main concerns and our responses

  • Causal role of center mismatch (SK2Z, VkE2). We isolate the center effect by training GuidanceNFT and NFT on the same fixed rollouts from the pretrained SiT. Even with a fixed rollout distribution, NFT still degrades while GuidanceNFT continues to improve. We also perturb the learned center by adding Gaussian noise of different scales or by interpolating it toward NFT's center, and both perturbations reduce the final reward.
  • Conditioning-dropout baseline (z3gG). We compare GuidanceNFT with NFT using conditioning dropout on unguided rollouts, with CFG at inference for both. GuidanceNFT achieves better reward and generation quality than this baseline on ELF-B, SiT, and SD3.5-M.
  • Training overhead (SK2Z, VkE2). GuidanceNFT surpasses NFT's peak performance across image and language generation. Using one fitting timestep instead of ten reduces the total training cost from 1.73× to 1.40× that of NFT, while improving GenEval from 0.91 to 0.94. With the same GPU-time budget on SD3.5-M, GuidanceNFT still achieves higher GenEval and Learned3 rewards.
  • Generation quality beyond preference rewards (SK2Z). We report FID and IS for the ImageNet experiments. GuidanceNFT achieves higher PickScore and IS than NFT, and lower FID against a reward-tilted reference (tFID).

Reviewer feedback after rebuttal

  • Reviewer SK2Z raised their rating to weak accept after seeing the fixed-rollout, FID/IS, and matched-compute results, stating that "the authors have addressed most of my concerns."
  • Reviewer z3gG stated that "the additional results have addressed my concern" after seeing the conditioning-dropout comparison.
  • Reviewer VkE2 kept their positive score and found the reduced-cost results interesting, "especially that fewer fitting timesteps lead to better performance."

We have included these results in the revised paper, and we thank the reviewers and the area chair again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.