acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Classifier-free Guidance in Reward-based Diffusion Post-training

Abstract

Reward-based diffusion post-training updates model parameters from task rewards, yet the role of classifier-free guidance (CFG) in this process remains unclear. We identify an Inference–Learning Mismatch: guidance can improve a fixed model's outputs while stronger guided collection weakens subsequent learning. This contrast motivates using CFG to organize reward measurements rather than prescribe a training direction. We construct local reward-query coordinates from the conditional–unconditional residual and compare symmetric same-state interventions to identify reward-preferred changes. The resulting reward differences provide directional credit tied to a known intervention, complementing the sample-level credit of individual images. To combine these signals in a parameter update, we introduce Fiber-Augmented Negative-Aware Fine-Tuning (FANFT), which pulls local responses into parameter space and fuses them with global NFT credit according to agreement between disjoint estimates. Frozen-policy tests show that these coordinates predict held-out preferences better than matched random coordinates, while training comparisons connect directional writing to improved learning on condition-sensitive rewards. Across tasks, the gains depend on how well the coordinate represents reward-improving directions. These findings place reward–coordinate alignment at the center of CFG's post-training role: the residual organizes local queries, and global reward supervision complements the directional information they provide.

Primary area:
generative models
Keywords:
Diffusion model, Post-training, Reinforcement Learning, Classifier-free Guidance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o4i7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FANFT, which extends DiffusionNFT by introducing local directional supervision alongside image-level reward learning. The method uses CFG-derived coordinates and same-state symmetric interventions to identify reward-preferred velocity directions, then integrates these signals with the global NFT gradient through agreement-controlled fusion.

Critical strengths
  1. FANFT presents an interesting approach to reward-based learning by complementing image-level reward supervision with locally measured directional information. The use of same-state symmetric interventions and agreement-controlled gradient fusion is a meaningful contribution.
  2. The frozen-policy experiments support the informativeness of CFG-derived coordinates, while the training ablations provide evidence for the usefulness of directional updates.
Critical weaknesses
  1. Absence of qualitative results. The paper provides no generated-image examples, making it difficult to assess whether higher reward scores correspond to meaningful visual improvements or introduce quality trade-offs, particularly given the weaker PickScore results.
  2. The incremental contribution and generalizability of CFG-based directional learning remain somewhat limited. Random 2D performs nearly as well as FANFT on OCR, and the improvement over Antithetic NFT is modest on GenEval. The results suggest that the benefits depend strongly on the reward task, while the effectiveness of directional exploration at later diffusion steps remains unexplored.
Questions for authors
  1. Could the authors provide qualitative comparisons to demonstrate the visual improvements and potential trade-offs associated with FANFT?
  2. How much of FANFT's improvement is specifically attributable to CFG-derived coordinates, given the competitive Random 2D results on OCR?
11 comments
AuthorsOpenReview ↗

Response to Reviewer o4i7

We thank the reviewer. We address the two questions with matched images and equal-budget tests of the reward information supplied by CFG-derived coordinates.

Anonymous supplementary material: https://anonymous.4open.science/r/Response-to-Reviewer-o4i7/ — The page contains all 240 matched image groups, target/non-target reward tables, frozen OCR coordinate tests, confirmation on 48 new prompts, and clean/positive/negative intervention panels, with experiment settings and figures.

Q1. Visual improvements and trade-offs

We provide 240 matched Base/DiffusionNFT/FANFT groups (720 images), with 60 groups for each of GenEval, OCR, HPSv2 and PickScore. Each group shares its prompt, initial latent and inference settings: 40 steps, CFG 4.5, and 512 × 512 resolution. The galleries allow direct comparison of object arrangements, rendered text and appearance.

For example, GenEval Case 01 requests a blue book to the left of a yellow vase and a red apple to its right; OCR Case 02 specifies the phrase “Jasmine at Dusk.” Presenting each prompt with all three outputs makes the requested content directly inspectable.

A separate fixed evaluation uses 64 matched prompt–latent pairs per task:

RewardDiffusionNFTFANFTPaired difference [95% CI]
GenEval0.957030.98698+0.02995 [-0.00781, 0.07812]
OCR0.950650.96802+0.01738 [-0.00335, 0.03746]
HPSv234.6695434.24803-0.42151 [-0.77370, -0.09061]
PickScore0.873000.87467+0.00167 [-0.00343, 0.00676]

Intervals use paired prompt bootstrap with the checkpoints fixed. On the same images, non-target PickScore differences for GenEval- and OCR-trained models are -0.00224 [-0.00707, 0.00269] and -0.00371 [-0.01058, 0.00292], respectively. The page also reports complementary preference scores for HPSv2- and PickScore-trained models. HPSv2 is score × 100; PickScore is logit / 26.

The galleries show content and appearance changes, while the paired scores quantify their target-reward and preference profiles under shared inference conditions.

Q2. CFG-derived coordinates versus Random 2D

The OCR training AUCs, 0.95804 for FANFT and 0.95696 for Random 2D, compare complete training configurations. To isolate the coordinate's contribution, we replace the CFG-derived first axis with a matched random axis while fixing the model state, prompt, latent, two-dimensional basis size, shared second axis, action norm and query budget.

The test uses 48 prompts × 3 states × 4 held-out directions = 576 pairs per design. A/B directions fit the local reward response; separate C directions test it. Mean signed branch advantage is 0.02091 for CFG versus 0.00599 for random, a paired gain of +0.01492 [95% CI: 0.00738, 0.02319]. Positive advantage means the prediction points toward the higher-reward branch. The ratio of means is 3.49× [2.00, 7.40] at equal query budget.

Correct-direction yield is 153/576 (26.56%) for CFG versus 84/576 (14.58%) for random: +11.98 pp [6.94, 17.19], or a 1.82× ratio [1.41, 2.51]. Ties and abstentions contribute zero, and all pairs remain in the denominator. These measures therefore compare useful information across the full matched budget.

We then fixed the decision rule before testing 48 previously unused prompts, with 144 states per design. Mean reward gain over clean images is +0.03459 for CFG versus +0.01267 for random, yielding +0.02192 [0.00266, 0.04174] for the paired difference. Every state enters the estimate, including states where the rule retains the clean output. Thus, the additional coordinate information supports better local reward decisions on new prompts.

The two experiments have complementary roles: training controls measure the value of directional writing; frozen coordinate replacement identifies the information supplied by CFG. Clean/+σu/−σu panels visualize the tested intervention at the first conditional velocity, followed by deterministic 10-step DPM2 with behavior CFG 1. Together, the matched images and coordinate tests connect visible generation behavior with FANFT's local reward-query construction.

If you have any further questions, please feel free to discuss them with us!

AuthorsOpenReview ↗

Follow-up to Reviewer o4i7: Manuscript Revisions

Thank you again for your questions. We have incorporated the qualitative comparisons and CFG-specific coordinate tests from our earlier response into the revised manuscript. Revisions are marked in blue; all references below use the revised PDF's numbering.

Q1. Generated-image comparisons and reward-dependent effects. We added Section 5.5 (p. 9) and Figure 5 (p. 10). The figure presents six matched cases with SD3.5-M (Base), DiffusionNFT, and FANFT in rows: three GenEval object/spatial-relation cases and three OCR spelling/phrase-completeness cases. Each column uses the same prompt, initial latent, and inference settings across methods.

Appendix E.6 (pp. 35–36) describes the gallery protocol and the separate 64-pair-per-task checkpoint evaluation. Tables 22–23 (p. 36) report target and non-target rewards with paired confidence intervals, while Table 24 (p. 36) gives the full prompts and gallery identifiers for the main figure. Figure 22 (p. 37) adds matched examples across all four reward tasks, including HPSv2 and PickScore. Figure 23 (p. 38) shows the image changes produced by clean, positive, and negative local interventions at matched solver states. The revised PDF therefore places the generated outputs, their prompts, and the accompanying reward measurements together.

Q2. The specific contribution of CFG-derived coordinates. We added the OCR matched-coordinate findings to Section 3.2 (pp. 4–5) and revised Section 5.4 (pp. 8–9) to distinguish local coordinate information from complete training outcomes.

Appendix E.3 and Figure 20 (p. 33) specify the equal-query-budget CFG-versus-random comparison, including the matched state, dimension, shared auxiliary axis, action norm, held-out directions, and prompt-cluster uncertainty. Figure 21 (p. 34) reports the fixed-rule confirmation on 48 new prompts. These additions directly locate the CFG-specific evidence alongside the coordinate ablation in Table 3 (p. 9).

We also incorporated the five-seed no-write and Antithetic NFT training comparisons in Table 2 (p. 9), with details in Appendices E.1–E.2 (pp. 31–33). This lets the manuscript distinguish the information provided by the residual coordinate from the learning benefit of applying directional credit, including the comparison against Antithetic NFT noted in your review.

These additions connect the image-level evidence, the controlled coordinate tests, and the training comparisons within the paper itself.

AuthorsOpenReview ↗

Follow-up on Qualitative Results and Revised Manuscript

Dear Reviewer o4i7,

Thank you again for your thoughtful review and helpful suggestions. In our earlier responses, we provided matched Base/DiffusionNFT/FANFT image comparisons, target and complementary reward evaluations, and a frozen OCR experiment comparing CFG-derived coordinates with matched random coordinates at equal query budgets. We also evaluated a fixed local-decision rule on new prompts. The revised manuscript now incorporates these results, including a six-case comparison in the main text and additional images, signed-intervention examples, and experimental details in the appendix. Changes are highlighted in blue.

We would greatly appreciate your feedback on whether the visual comparisons and controlled coordinate tests address your questions, or whether further clarification would be useful. We would welcome any additional questions or suggestions before the discussion period closes so that we can address them within the remaining discussion time.

Your suggestions have helped us make the generated-image comparisons more informative and separate the CFG residual’s contribution to local reward information from the training benefit of directional writing.

We sincerely appreciate your time and consideration.

Best regards,

The Authors

Reviewer_rg5JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper finds that stronger classifier-free guidance can improve a fixed model's samples while making reward-based post-training worse when used for sample collection. FANFT uses the conditional-unconditional residual to choose velocity perturbations, tests their effect on reward, and adds the resulting local update to DiffusionNFT. The main training gains are on GenEval and OCR.

Critical strengths
  1. I found the frozen-model tests convincing. The coordinate replacements use the same dimension and perturbation size, and predictions are checked on held-out directions. These tests show that the CFG residual gives useful information about the local reward response.

  2. The measurement-only and paired-only controls are helpful. They keep the paired samples and global supervision, so they test whether actually using the measured response improves training beyond collecting those samples.

Critical weaknesses
  1. I can't tell how stable the training gains are. In Table 10, the AUC improvements over measurement-only are about 0.54 percentage points on GenEval and 1.67 on OCR, but the table does not give variation across training runs. The frozen-model bootstrap answers a different question and does not resolve this concern.

  2. It is also hard to tell how much of the full-system improvement comes from the local update itself. Antithetic NFT already closes much of the gap in the ablations. The short guidance sweep also favors CFG 2 on several rewards, while the full-system baseline uses CFG 1. That sweep does not establish the best setting for the full training run, but it makes the baseline choice worth checking.

Questions for authors
  1. How many independent runs were used for Tables 1, 2, and 10? Could you report the variation in the paired differences?

  2. Does FANFT still improve on antithetic NFT when the collection scale is chosen using the guidance sweep and compute is matched?

16 comments
AuthorsOpenReview ↗

Response to Reviewer rg5J

We thank the reviewer. The additional experiments address training variability and comparison with guidance-adjusted Antithetic NFT at matched GPU budgets.

Anonymous supplementary material: https://anonymous.4open.science/r/Response-to-Reviewer-rg5J/ — Figure S1 shows five-seed trajectories, paired AUC differences and uncertainty; Figure S2 shows CFG 1/2 Antithetic baselines, GPU-hour curves and fixed-budget rewards. The page also lists the original run counts.

Q1. Independent runs and variation in paired gains

The submitted tables use one seed-42 trajectory per method-task cell: Table 1 contains 8 trajectories and Table 2 contains 14. Table 10 reuses two FANFT trajectories from Table 2 and adds four controls. Tables 2 and 10 therefore represent 18 distinct training trajectories.

We extended GenEval and OCR to five independent training seeds per configuration for FANFT, Measurement-only, and Antithetic NFT at CFG 1 and 2. For each run, normalized AUC is the trapezoidal reward integral divided by window length: epochs 10–190 for GenEval and 10–120 for OCR. We subtract each control's AUC from FANFT's AUC for the same seed and compute the mean, SD and 95% Student-t interval across the five paired differences (df=4).

TaskFANFT minus controlMean gain (pp)SD (pp)95% CI (pp)
GenEvalMeasurement-only+0.6360.124[0.482, 0.790]
GenEvalAntithetic CFG 1+0.3930.109[0.258, 0.529]
GenEvalAntithetic CFG 2+0.3470.128[0.188, 0.506]
OCRMeasurement-only+1.1760.305[0.798, 1.555]
OCRAntithetic CFG 1+0.7570.310[0.371, 1.142]
OCRAntithetic CFG 2+1.1360.160[0.937, 1.335]

All six paired AUC confidence intervals are above zero. Measurement-only retains paired collection and global NFT supervision while disabling directional writing, directly testing the value of the local update. The Antithetic comparisons evaluate FANFT against a baseline that already uses antithetic pairing. S1 displays each trajectory and paired difference, relating the average gains to variation across independent runs.

Q2. Guidance-adjusted baselines at matched compute

The submitted standard-NFT sweep motivates testing both collection CFG 1 and 2: their AUCs over epochs 0–80 are 0.7886/0.8833 for GenEval and 0.9048/0.8163 for OCR. We retain both Antithetic settings and fix evaluation CFG at 4.5.

For each independent seed, we compare rewards at 36 GPU-hours for GenEval and 24 for OCR, interpolating within the run's recorded range. The five-seed results are:

MethodGenEval, mean ± SDOCR, mean ± SD
FANFT0.92976 ± 0.001200.96316 ± 0.00744
Antithetic CFG 10.92524 ± 0.001530.95456 ± 0.00811
Antithetic CFG 20.92375 ± 0.000580.95010 ± 0.00388

We also pair rewards by training seed at these budgets and compute 95% Student-t intervals from the five differences:

TaskFANFT minus baselineMean gain (pp)Paired 95% CI (pp)
GenEvalAntithetic CFG 1+0.452[0.303, 0.602]
GenEvalAntithetic CFG 2+0.602[0.468, 0.735]
OCRAntithetic CFG 1+0.860[-0.899, 2.618]
OCRAntithetic CFG 2+1.306[0.540, 2.072]

GPU time includes rollout sampling, reward-model scoring, training updates, and FANFT's additional gradient calculations. The matched budgets therefore count the extra computation used by the local update. S2 presents the full curves and per-seed budget values.

FANFT has higher mean rewards than both Antithetic configurations on both tasks, with the paired intervals reporting fixed-budget uncertainty. Combined with Q1, the evidence separates two questions: Measurement-only isolates directional writing; the Antithetic comparisons evaluate the complete method after accounting for pairing, guidance choice and GPU time.

If you have any further questions, please feel free to discuss them with us!

AuthorsOpenReview ↗

Follow-up to Reviewer rg5J: Manuscript Revisions

Thank you again for your questions. We have incorporated the independent-seed and matched-compute evidence from our earlier response into the revised manuscript. Revisions are marked in blue; all references below use the revised PDF's numbering.

Q1. Independent runs and variation in paired differences. Section 4.3 (pp. 6–7) and Table 2 (p. 9) now report the five-independent-seed comparisons on GenEval and OCR. Appendix E.1 (pp. 31–32) documents the run counts for the submitted results, identifies reused trajectories, and distinguishes these counts from the additional five-seed comparisons. The original coordinate and credit-channel tables are now Tables 3 and 11.

Equation (31) and Table 16 (p. 32) give the seed-paired calculation, comparison windows, means, standard deviations, and 95% Student-t intervals. All six AUC intervals against Measurement-only and the two Antithetic configurations are positive. The revised presentation explicitly distinguishes uncertainty across independent optimization runs from prompt-cluster uncertainty in the frozen-policy experiments.

Q2. Guidance-adjusted Antithetic NFT at matched compute. We revised Section 5.2 (p. 7) and added the equal-budget results to Table 2 (p. 9). The comparison includes Antithetic NFT at collection CFG 1 and 2, evaluated at CFG 4.5, with common budgets of 36 GPU-hours on GenEval and 24 on OCR.

Appendix E.2 (pp. 32–33) specifies the within-run interpolation protocol. Table 17 (p. 32) reports reward means and standard deviations at the common budgets, while Table 18 (p. 32) reports seed-paired differences and their confidence intervals. Thus, the manuscript now presents training-window AUC and fixed-budget reward as separate, explicitly defined comparisons.

The GPU-hour accounting paragraph on p. 33 states that the recorded time includes rollout sampling, reward-model inference, basis construction, local pullback differentiation, and parameter updates. It also separates the standard-DiffusionNFT checkpoint-acquisition comparison in Figure 4 from the Antithetic comparisons in Tables 17–18. This makes both the baseline and the compute scope explicit.

The revised manuscript therefore contains the run accounting, paired uncertainty estimates, and guidance-adjusted compute comparisons underlying our earlier response.

AuthorsOpenReview ↗

Follow-up on Additional Experiments and Revised Manuscript

Dear Reviewer rg5J,

Thank you again for your careful review and constructive feedback. In our earlier responses, we reported additional comparisons using five independent training seeds per configuration, with seed-paired AUC differences and confidence intervals. We also compared FANFT with Antithetic NFT at collection CFG 1 and 2 under matched GPU-hour budgets, added confidence intervals for the fixed-budget differences, and clarified that GPU time includes sampling, reward scoring, and the additional gradient calculations. These results are now included in the revised manuscript, with changes highlighted in blue.

We would greatly appreciate your feedback on whether the additional experiments and revised presentation address your questions about training variability and baseline comparisons. We would also welcome any further questions or suggestions before the discussion period closes, allowing us to respond during the remaining discussion.

Your feedback has helped us strengthen the training-level evidence and distinguish the contribution of directional writing from antithetic pairing, collection-guidance choices, and computational cost.

We sincerely appreciate your time and consideration.

Best regards,

The Authors

Reviewer_Wr2UOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the role of CFG in reward-based diffusion post-training. It finds that stronger CFG improves frozen-model outputs but can impair subsequent learning under the tested NFT configurations. The authors use the conditional–unconditional residual to construct local reward-query coordinates and obtain directional credit through symmetric positive and negative velocity interventions at the same state. FANFT pulls this information into parameter space through derivatives of the velocity prediction, then combines it with the global NFT update through controlled fusion based on gradient agreement between disjoint probe sets. Validation includes frozen-policy held-out tests against matched random coordinates, coefficient transfer across states and prompts, training ablations that retain paired collection while disabling directional writing, and full-system comparisons across four rewards. The frozen-policy results provide strong evidence for the information carried by residual-based coordinates. Training improves on GenEval and OCR, remains close on HPSv2, and favors NFT on PickScore.

Critical strengths
  • Using the CFG residual to define query coordinates and rewards to determine the preferred action sign provides a distinct credit-assignment mechanism. The matched random replacements and held-out direction tests in Figure 2, Appendix C.2, and Table 8 strongly support the residual coordinates’ information value.
  • The no-write controls in Figure 3 isolate the added value of directional writing while retaining paired collection and global supervision. The four-reward comparison reports improvements, comparable performance, and degradation, providing evidence for both training utility and task dependence.
Critical weaknesses
  • The reward–coordinate alignment explanation remains plausible but lacks direct validation. No comparable alignment measure is evaluated across rewards or shown to predict training gains. Frozen-policy PickScore probes reveal useful local information, yet PickScore post-training underperforms NFT. Although the settings differ, the link between informative queries and improved training remains incomplete.
  • The full-system comparison does not establish the advantage over suitably tuned NFT or fully characterize its scope. Table 6 favors collection CFG 2 over CFG 1 for GenEval and HPSv2 within the reported short window, but the full-system baseline uses CFG 1. Missing cross-reward evaluations and matched-prompt image comparisons also leave potential trade-offs in other capabilities and visual quality unclear.
Questions for authors
  • Why does the local coordinate advantage in the frozen-policy PickScore experiment fail to translate into better full-system training performance? Given the differences between these settings, which explanations are supported by existing evidence, and which remain hypotheses? Which results directly support reward–coordinate alignment as an explanation for training gains?
  • Table 6 identifies CFG 2 as the better fixed collection scale for some tasks. What motivates using CFG 1 in the full-system comparison? Which FANFT gains remain when compared with a more suitable fixed CFG for each task?
  • FANFT and Random 2D achieve very similar OCR AUCs in Table 2. Do the authors agree that this result more directly supports the utility of local directional credit, while providing limited evidence for the distinctive advantage of the CFG residual anchor? Should the conclusions in Section 5.4 be qualified accordingly?
  • Could the authors provide targeted cross-reward evaluations, such as non-target reward scores for models post-trained on GenEval or OCR, alongside matched-prompt outputs from the base model, NFT, and FANFT? These results would help determine whether target-reward gains reflect the intended content improvements and whether they involve trade-offs in visual quality or other capabilities. Image comparisons between positive and negative local interventions would also clarify what the probes change.
  • Please provide the key configurations needed for reproduction, including LoRA settings, the optimizer, the behavior-model averaging schedule, NFT hyperparameters, prompt splits, and reward normalization. Please also clarify whether the GPU-hours in Figure 4 include the actual collection, training, basis construction, and local gradient computation costs incurred by each method.
7 comments
AuthorsOpenReview ↗

Response to Reviewer Wr2U

We thank the reviewer for the detailed questions. All intervals below are 95% confidence intervals.

Anonymous supplementary material: https://anonymous.4open.science/r/Response-to-Reviewer-Wr2U-063D/ — W1–W2: prediction and updates; S1–S2: five-seed training and compute; W6: guidance; W3/Q4: images and cross-reward scores; Q5: configurations.

Q1. Local PickScore information and training

The frozen test measures reward prediction; training also depends on which directions enter updates. On 48 GenEval prompts and 144 frozen states, CFG improves conditional accuracy over matched random coordinates by +10.59 pp for PickScore [4.86, 16.15] and +10.59 pp for HPSv2 [5.56, 15.45]. Evaluation directions are held out; conditional accuracy excludes ties and abstentions (W1).

The PickScore trace records nonzero local corrections in seven of 235 logged epochs. In three matched one-step tests, the gate selects zero local contribution, so Full FANFT and Measurement-only produce identical updates and rewards (W2). This verifies the gate's no-write behavior.

Across five independent training seeds per configuration, FANFT improves AUC over Measurement-only by +0.636 pp on GenEval [0.482, 0.790] and +1.176 pp on OCR [0.798, 1.555]. The control retains paired collection and global NFT supervision (S1). W1 measures local alignment, W2 update selection, and S1 accumulated learning: the frozen PickScore test and full training assess different stages.

Q2. Collection guidance and matched compute

CFG 1 provides a common collection setting. The short standard-NFT sweep motivates testing Antithetic NFT at both CFG 1 and 2, with evaluation CFG 4.5. Across five independent seeds, fixed-budget FANFT-minus-baseline differences are:

Task/budgetAntithetic CFGMean gain (pp)Paired CI (pp)
GenEval/36 GPU-h1+0.452[0.303, 0.602]
GenEval/36 GPU-h2+0.602[0.468, 0.735]
OCR/24 GPU-h1+0.860[-0.899, 2.618]
OCR/24 GPU-h2+1.306[0.540, 2.072]

Scores are interpolated within each run's observed range; intervals use five seed-paired differences and Student-t (df=4). GPU time includes sampling, reward scoring, training updates, and FANFT's extra gradient calculations. S2 shows the curves.

W6 tests reward and learning signals with a fixed model: 16 prompts × 4 latents per task and CFG {1, 2, 4.5} × {10, 20} steps. On 20-step PickScore, CFG 4.5 versus 1 raises reward by 0.07800 [0.05806, 0.09994], while CLIP-based diversity and reward contrast decrease. The reward-derived loss signal, mean |r − 0.5|, falls 41.8%. Here r is NFT's mixing value, computed with the implemented normalization and reset history. Thus, immediate reward and the signal used by the loss can move in opposite directions.

Q3. CFG-specific information on OCR

The OCR AUCs, 0.95804 for FANFT and 0.95696 for Random 2D, compare trained configurations. To isolate coordinate information, we replace the first axis while matching the frozen state, dimension, second axis, action norm and query budget.

Across 576 held-out pairs, CFG gives 3.49× the mean signed branch advantage: 0.02091 versus 0.00599, with paired difference +0.01492 [0.00738, 0.02319]. Correct-direction yield increases by 11.98 pp [6.94, 17.19], counting all pairs. On 48 new prompts, a fixed rule yields a CFG-minus-random reward-gain difference of +0.02192 [0.00266, 0.04174]. Accordingly, §5.4 distinguishes the training value of directional writing from the coordinate's measured local information.

Q4. Images and cross-reward effects

We provide 240 matched Base/DiffusionNFT/FANFT groups (720 images), plus clean/positive/negative interventions at shared states. A separate 64-pair cohort per task uses 40 steps and CFG 4.5. FANFT-minus-NFT target estimates are: GenEval +0.02995 [-0.00781, 0.07812]; OCR +0.01738 [-0.00204, 0.03808]. On the same images:

Training taskHPSv2 difference [CI]PickScore difference [CI]
GenEval-0.17430 [-0.54931, 0.18562]-0.00224 [-0.00713, 0.00293]
OCR-1.74952 [-2.57498, -0.92519]-0.00371 [-0.01050, 0.00277]

These prompt-bootstrap estimates distinguish content fidelity from preference (HPSv2: score × 100; PickScore: logit / 26). W3 shows the corresponding images.

Q5. Reproduction settings

Q5 specifies SD3.5 Medium, 512 × 512, LoRA rank 32/alpha 64, and AdamW (lr 3e-4, betas 0.9/0.999, epsilon 1e-8, weight decay 1e-4, clipping norm 1). Batch, accumulation and sample counts match between methods within each task. Collection/evaluation use 10/40 DPM2 steps and CFG 1/4.5. NFT beta is 0.1 for OCR and 1 otherwise; EMA flags are true for GenEval/PickScore and false for OCR/HPSv2. Diagnostic allocations are reported separately from the training budgets defined in Q2.

If you have any further questions, please feel free to discuss them with us!

AuthorsOpenReview ↗

Follow-up to Reviewer Wr2U: Manuscript Revisions

Following our earlier response, we have incorporated the corresponding analyses and results into the revised manuscript. Revisions are marked in blue. All references below use the revised PDF's numbering.

Q1. Connecting local reward prediction to parameter updates. We revised Section 5.3 (p. 7) and added Appendix E.4 (pp. 34–35), including Tables 19–20 (p. 34). These passages distinguish local reward prediction, admission of a local correction, and accumulated training outcomes. They bring together the frozen PickScore/HPSv2 measurements, the PickScore update-admission record, and the matched one-step controls. The five-seed directional-writing results are incorporated in Section 4.3 (pp. 6–7) and Table 2 (p. 9), connecting this discussion to the training comparisons.

Q2. Guidance-adjusted baselines and collection-time learning information. Section 5.2 (p. 7) and Table 2 (p. 9) now present the five-seed comparisons with Antithetic NFT at collection CFG 1 and 2. Appendix E.2 (pp. 32–33), Tables 17–18 (p. 32) gives the fixed-budget means, standard deviations, and seed-paired confidence intervals at 36 GPU-hours for GenEval and 24 for OCR. We also expanded Section 2.2 (p. 3) and added Appendix E.5 and Table 21 (p. 35) to explain the guidance diagnostic through its measured immediate reward, within-prompt diversity, reward contrast, and loss-level credit.

Q3. Separating coordinate information from the OCR training comparison. The new OCR matched-coordinate results are summarized in Section 3.2 (pp. 4–5). Appendix E.3 and Figures 20–21 (pp. 33–34) provide the equal-query-budget comparison and the fixed-rule evaluation on 48 new prompts. We revised Section 5.4 (pp. 8–9) to distinguish the information supplied by the CFG anchor from the outcomes of complete training configurations. The coordinate ablation is now Table 3 (p. 9).

Q4. Matched images, cross-reward evaluation, and signed interventions. We added Section 5.5 (p. 9) and Figure 5 (p. 10), showing six matched Base/DiffusionNFT/FANFT examples: three GenEval composition cases and three OCR text-rendering cases. Appendix E.6 (pp. 35–36), Tables 22–24 (p. 36) reports target and non-target scores on the separate fixed-checkpoint cohort and supplies the complete prompts for Figure 5. Figure 22 (p. 37) extends the visual comparison to all four rewards, and Figure 23 (p. 38) shows clean, positive, and negative interventions from the same solver states.

Q5. Reproduction settings and GPU-hour accounting. Appendix E.7 (pp. 36–37) and Table 25 (p. 37) now collect the LoRA, optimizer, batching, accumulation, NFT, and EMA settings, with a cross-reference to Appendix A for reward normalization and the update definitions. The evaluation cohorts are specified in Appendices E.3–E.6. The explicit accounting paragraph in Appendix E.2 (p. 33) states that training GPU-hours include rollout generation, reward scoring, basis construction, local pullback differentiation, and global updates. It also distinguishes standard-DiffusionNFT acquisition results from the matched-budget Antithetic comparisons.

These revisions place the evidence discussed in our earlier response directly in the manuscript, together with the corresponding explanations and experimental protocols.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewer Wr2U,

Thank you again for your thoughtful review and constructive questions. In our earlier point-by-point responses, we presented additional evidence connecting local reward prediction, update admission, and training outcomes. We also provided guidance-adjusted training comparisons, a CFG-by-sampling-step analysis, matched image and cross-reward evaluations, and further details on the training settings and GPU-hour accounting. These results and clarifications have now been incorporated into the revised manuscript, with changes highlighted in blue.

We would greatly appreciate your feedback on whether these responses and manuscript revisions address your concerns, or whether any points would benefit from further clarification. Before the discussion period closes, we would welcome any additional questions or suggestions so that we can respond within the remaining discussion time.

Your questions have helped us clarify the relationship between informative local reward queries and their use in parameter updates, examine how collection guidance affects the learning signal, and present the image-level effects and experimental protocols more clearly.

We sincerely appreciate your time and consideration.

Best regards,

The Authors

Comments on the paper

4 comments

Dear Reviewers,

The authors have posted their responses. Could you please read the rebuttal and let us know whether it resolves the questions and concerns raised in your review? If any point remains unclear, feel free to follow up with the authors during the discussion period.

Please update your review and final rating at your earliest convenience.

Thank you very much for your time and contribution.

Best regards, AC

AuthorsOpenReview ↗

Clarifying the Scientific Contribution, Evidence Chain, and Roles of Our Controls

Dear Reviewers,

Thank you for your thoughtful reviews. Following our point-by-point rebuttals, we have incorporated the additional experiments into the revised PDF (changes marked in blue). We would like to clarify how the scientific question, the method, and the controls fit together.

1. Scientific question and novelty. Our paper asks: What role can classifier-free guidance (CFG) play in reward-based diffusion post-training? Figure 1 identifies an inference–learning mismatch: guidance that improves outputs from a fixed model does not necessarily improve subsequent learning when used to collect training samples. This motivates a different role for CFG: the CFG residual defines where to query local reward; measured reward differences determine which direction to learn. FANFT implements this idea using same-state positive/negative interventions and controlled integration of directional credit with global DiffusionNFT supervision (Sections 3–4). Our contribution is therefore both an empirical understanding of CFG during post-training and a method that uses this understanding.

2. Why we constructed Random 2D and Antithetic NFT. These are diagnostic controls introduced in this paper, not independent algorithms proposed by other papers. Random 2D retains local directional probing and learning but replaces the CFG-derived coordinate with a random one; it tests coordinate informativeness. Antithetic NFT uses paired forward-process noise views without FANFT's local directional reward update; it tests pairing versus measured directional credit. Measurement-only retains paired collection and global NFT learning but disables local directional writing; it tests the value of using the measured signal. These controls deliberately share components with FANFT. Accordingly, their performance gaps quantify specific component contributions, while standard DiffusionNFT is the reference for overall system-level improvement.

3. Evidence from mechanism to learning. The revised manuscript establishes three connected results:

  • CFG supplies useful local reward information. At equal query budgets, the CFG coordinate achieves 3.49× the mean held-out OCR signed branch advantage of a matched random coordinate. A fixed decision rule evaluated on 48 new prompts gives a paired reward-gain difference of +0.02192 [95% CI: +0.00266, +0.04174] (Section 3.2; Appendix E.3).
  • Directional writing improves training. Across five independent seeds, FANFT exceeds Measurement-only in normalized AUC by +0.636 pp [0.482, 0.790] on GenEval and +1.176 pp [0.798, 1.555] on OCR (Table 2). This comparison isolates the use of local directional credit.
  • The complete method improves over DiffusionNFT. In the submitted configuration comparison, GenEval AUC increases from 0.86135 to 0.92600 (+6.47 pp), and OCR AUC from 0.92446 to 0.95804 (+3.36 pp) (Table 3). Five-seed comparisons against Antithetic NFT at CFG 1 and 2 additionally report fixed-budget reward differences and paired confidence intervals (Appendix E.2). GPU-hour budgets include rollout sampling, reward scoring, basis construction, and extra local-gradient computation.

Thus, the similar OCR AUCs of FANFT and Random 2D describe two full training configurations that both use local directional credit; the controlled coordinate-replacement experiment separately establishes what the CFG residual adds.

In summary, the paper's contribution is a connected scientific finding: CFG changes the information available during reward-based collection, its residual provides informative local reward-query directions, and measured directional credit can improve diffusion post-training. The revised PDF now brings the mechanism tests, independent-seed comparisons, matched-compute evidence, and visual examples together (Figure 5; Appendix E).

We appreciate your time and consideration, and hope this clarification helps assess the paper's novelty and experimental evidence as a whole.

Best regards,

The Authors

AuthorsOpenReview ↗

General Response

Dear Area Chair, Senior Area Chairs, and Program Chairs,

We sincerely thank you and the reviewers for your time and constructive feedback. Throughout the rebuttal, we conducted targeted experiments directly addressing the reviewers' concerns and further clarifying the scientific contribution of our paper.

1. Point-by-point responses and experimental evidence

Wr2U — CFG's role in learning. We identified an Inference–Learning Mismatch: stronger CFG improves immediate generation but can reduce the reward-derived learning signal by 41.8%. Five-seed comparisons further demonstrate the value of directional writing, improving AUC over Measurement-only by +0.636 pp on GenEval and +1.176 pp on OCR.

rg5J — Training robustness and computational fairness. We conducted five independent seeds per configuration, with all six paired AUC comparisons yielding positive 95% confidence intervals. We additionally evaluated guidance-adjusted Antithetic NFT at CFG 1/2 under matched GPU-hour budgets, accounting for FANFT's additional computation.

o4i7 and rg5J — CFG-specific benefits over Random 2D. We conducted controlled coordinate-replacement experiments demonstrating a 3.49× advantage in held-out OCR directional reward response.

More importantly, our latest three-seed paired OCR training experiment directly measures the difference in trained checkpoint quality. FANFT achieves higher best evaluation rewards than Random 2D in all three paired runs, with gains of +0.231, +0.144, and +0.210 pp, respectively. The mean improvement is +0.195 ± 0.045 pp, with a 95% confidence interval of [0.082, 0.308] pp. This provides direct training-level evidence that CFG-specific coordinate information can translate into improved OCR checkpoints.

We also provided 240 matched image groups, cross-reward evaluations, and reproducibility details.

2. New mechanistic findings and generalization

We further measured the Pearson correlation between predicted and actual local reward responses on held-out directions:

RewardCFG 2DRandom 2D
GenEval0.8410.458
OCR0.7180.658
HPSv20.8530.828
PickScore0.8400.900

These results explain the task-dependent benefits of CFG through reward–coordinate alignment.

We then extended FANFT to reward-gradient-aligned exploration, improving HPSv2 from 34.49 to 38.52 and PickScore from 22.64 to 24.08, with approximately 15× and 29× earlier target attainment, respectively.

In joint GenEval–HPSv2 training, reward-aligned exploration also improves both objectives, from 0.88/29.52 to 0.90/31.02 at epoch 50.

These results establish the broader applicability of reward-aligned exploration beyond CFG.

3. Scientific contribution

Our work addresses a fundamental question: When, why, and how should CFG be used during reward-based diffusion post-training?

We establish three central findings:

  • Discovery: Better inference-time guidance does not necessarily produce better learning signals.
  • Mechanism: CFG's usefulness depends on its alignment with the target reward, explaining its different effects across tasks.
  • Solution and future direction: Reward-aligned exploration provides a principled approach to selecting informative training directions, extending naturally to adaptive subspace selection and multi-reward optimization.

Our central message is that CFG should not be applied blindly during post-training. Researchers should evaluate its relevance to the target reward and select exploration directions accordingly.

4. Respectful request

We appreciate Wr2U's continued weak acceptance and o4i7's statement that they would not oppose acceptance. In response to rg5J's remaining questions, we have provided additional mechanistic evidence and the new three-seed OCR training comparison.

We would be sincerely grateful if the committee could consider the complete rebuttal, particularly these latest results, when forming its final assessment.

Thank you very much for your careful consideration.

Sincerely,
The Authors

AuthorsOpenReview ↗

Supplementary Evidence for Our Rebuttal

As a follow-up to our general rebuttal summary, we would like to share three anonymous repositories containing the additional experiments and supporting evidence provided during rebuttal:

These repositories directly correspond to the reviewers' original questions. Their key findings have also been incorporated into the revised manuscript, with changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.