- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GuidanceNFT, a reward fine-tuning method designed to address the instability of DiffusionNFT when combined with classifier-free guidance (CFG) or built-in guidance mechanisms. The authors identify a mismatch between NFT's regression center and the conditional mean velocity induced by re-noising guided samples, which introduces bias into reward-driven updates and may lead to training collapse. To address this issue, GuidanceNFT trains an auxiliary flow-matching model to estimate and correct the regression center online. Experiments across image generation, language modeling, and mathematical reasoning demonstrate improved training stability and reward optimization, without introducing additional inference passes.
Critical strengths
The paper proposes a simple yet effective extension of DiffusionNFT that improves the stability of reward fine-tuning under guidance. The method is supported by theoretical analysis and experiments across multiple generative tasks, without introducing additional inference overhead.
Critical weaknesses
- Additional training overhead. The proposed method requires training an auxiliary flow-matching network to estimate the regression center, introducing substantial computational overhead. The reported training iteration time is approximately 1.73–1.77 times that of standard NFT.
- Others see Questions.
Questions for authors
- The authors identify regression center mismatch as a key factor contributing to training instability and collapse in guided NFT. However, the experiments primarily demonstrate a correlation between increasing center mismatch and performance degradation, rather than establishing a causal relationship. It is also possible that training instability causes the generated distribution to shift, which in turn increases the center mismatch. Could the authors provide further evidence to clarify whether center mismatch is a primary cause of training collapse or a consequence of it? For example, a controlled experiment that varies the regression center while keeping the rollout distribution fixed would help validate the proposed explanation.
- The ImageNet experiments primarily rely on PickScore to evaluate generation quality. However, improvements in preference-based rewards do not necessarily indicate better distributional fidelity or diversity. Could the authors additionally report standard image generation metrics, such as FID and Inception Score (IS), for the pretrained model, NFT, and GuidanceNFT under matched sampling settings? This would help clarify whether the improvements in reward optimization also translate into better overall generation quality.
I would be willing to reconsider my rating if the authors can provide convincing evidence or clarification addressing these concerns.
4 comments
Q2. Could the authors additionally report standard image generation metrics, such as FID and Inception Score (IS), for the pretrained model, NFT, and GuidanceNFT under matched sampling settings? This would help clarify whether the improvements in reward optimization also translate into better overall generation quality.
We thank the reviewer for suggesting the additional evaluation. We report PickScore, FID, and IS at matched training steps and under identical sampling settings. For both SiT and Model Guidance, GuidanceNFT achieves higher PickScore and IS than standard NFT.
Standard FID is computed against the original data distribution, so it is expected to increase during reward fine-tuning, which intentionally shifts the generated distribution toward higher-reward regions. To account for this shift, we also report reward-tilted FID (tFID), computed against the following reward-tilted reference distribution:
Here, is the PickScore z-score of image within class .
We report the minimum tilted FID over the evaluated values. The full curves for comparison at matched are shown here. NFT has higher tFID than GuidanceNFT at all evaluated values across the three settings, except at for SiT with CFG 1.5.
SiT-XL/2: generation quality
| CFG | Method | PickScore ↑ | IS ↑ | tFID ↓ | FID ↓ |
|---|---|---|---|---|---|
| 1 | Pretrained | 19.885 | 115.3 ± 4.1 | 11.28 | 11.28 |
| NFT | 20.398 | 111.0 ± 5.6 | 17.64 | 17.72 | |
| GuidanceNFT | 20.907 | 200.4 ± 8.8 | 7.73 | 11.11 | |
| 1.5 | Pretrained | 20.321 | 218.7 ± 6.3 | 4.28 | 4.48 |
| NFT | 20.645 | 181.9 ± 5.4 | 11.58 | 13.64 | |
| GuidanceNFT | 21.113 | 270.1 ± 5.2 | 6.76 | 13.93 |
Model Guidance: generation quality
| Method | PickScore ↑ | IS ↑ | tFID ↓ | FID ↓ |
|---|---|---|---|---|
| Pretrained | 20.338 | 240.6 ± 7.2 | 3.93 | 4.05 |
| NFT | 20.335 | 94.8 ± 3.8 | 30.61 | 30.61 |
| GuidanceNFT | 21.017 | 231.4 ± 5.3 | 13.30 | 17.52 |
Thank you for the detailed responses and additional experiments. The fixed-rollout experiments provide more convincing evidence for the causal role of center mismatch in training degradation. The additional FID and IS results also strengthen the empirical validation beyond preference-based rewards.
Although the additional training cost remains a limitation, the matched-compute results help justify the overhead. Overall, the authors have addressed most of my concerns, and I am willing to raise my rating to 3 (Weak Accept).
Thank you for your thoughtful review and for raising your score. The fixed-rollout experiment you suggested helped us better show the causal role of center mismatch, and the FID results gave us stronger evidence of generation quality. We are also glad that the equal-compute results addressed your concern about the training overhead.
Thank you for raising this point. By fitting an auxiliary center network, GuidanceNFT enables stable reward fine-tuning and continued improvement on guided rollouts, whereas NFT peaks and then deteriorates with further training. As shown in the table below, GuidanceNFT surpasses NFT’s peak performance across image and language generation. The additional computation therefore delivers gains that NFT does not achieve with the same training time.
† On ELF, GuidanceNFT avoids the entropy collapse that accompanies NFT’s low perplexity.
This extra cost can also be reduced. As shown in our response to Reviewer VkE2 (Q2), using one fitting timestep per center update instead of ten reduces the total training cost from 1.73× to 1.40× that of NFT, while improving CFG-4 GenEval from 0.91 to 0.94.
With the same GPU-time budget on SD3.5-M, GuidanceNFT completes only about 70% or less as many updates as NFT, yet GuidanceNFT achieves higher GenEval and Learned3 rewards and comparable OCR reward, as shown below.
SD3.5-M: matched compute
We also note that GuidanceNFT adds no inference cost, so the additional computation is confined to training.
We thank the reviewer for the insightful question. We trained GuidanceNFT and standard NFT on the same CFG-1.5 rollouts sampled from the frozen pretrained SiT. Even with a fixed rollout distribution, standard NFT without a matched center degrades, while our method continues to improve. This shows that rollout-distribution drift is not required for the observed degradation.
SiT-XL/2: fixed-rollout PickScore
GuidanceNFT achieves higher PickScores than NFT at both CFG scales and all three evaluated checkpoints.
We also analyzed the center mismatch using fixed rollouts in our toy tree model (see toy-model figure). In most regions, the center bias opposes the intended update with comparable magnitude. Applying this bias alone for ten steps shifts samples away from the original distribution (panel (c)). When added to the intended update, the bias distorts the velocity field, resulting in a distribution with degraded reward (panel(d)).
Together, the fixed-rollout experiment and the bias-only intervention support a causal role for center mismatch in training degradation.