acceptodds
Under review as a conference paper at ICLR 2027

Field Constrained Bargaining DiffusionNFT

Abstract

Diffusion alignment increasingly requires optimizing several criteria at once, such as visual quality, prompt fidelity, text rendering, and compositional correctness. Existing approaches commonly aggregate these criteria with fixed reward weights or balance their stochastic parameter gradients. Both views are incomplete for Diffusion Negative-aware FineTuning (DiffusionNFT), whose primitive learning signal is a scalar optimality that reweights the current clean-image distribution. A reward vector therefore has to be converted into a scalar target, and a target that appears attractive in reward space may still require a large modification of the forward velocity field. We formulate multi-reward DiffusionNFT as a constrained target-selection problem. For the linear optimality family , we show that the exact distribution-level reward gain is , where is the covariance of the centered rewards. We further derive an NFT-native field Gram matrix for which the population-optimal velocity displacement is exactly . These two geometries lead to Field-Constrained BargainingNFT (FCB-NFT), a small convex program that maximizes Nash reward surplus under a velocity-field displacement budget and valid-optimality constraints. We also provide a finite-sample conservative estimator of that avoids learning a conditional field probe. On two four-reward suites, FCB-NFT alone improves every reward, with a worst normalized gain of while baselines stay imbalanced or collapse. Ablations trace this gain to the structure of beyond reward covariance.

Primary area:
generative models
Keywords:
Diffusion Models, Multi-Reward Alignment, Multi-Objective Optimization, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_gLVeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper frames multi-reward DiffusionNFT as the problem of selecting a single scalar optimality target. The authors use the reward covariance to predict the reward gains of a candidate target, constrain the velocity-field change the target requires with a residual-weighted Gram matrix, and solve a per-group Nash bargaining problem to choose the combination coefficients. On two four-reward suites with SD3.5-Medium, the method achieves the best worst-channel normalized gain.

Critical strengths

S1: The covariance gain map directly explains why fixed weights fail: nonnegative weights can still lower a reward through negative correlations between rewards. The raw-sum result in Table 1 (worst gain −0.34) is consistent with this analysis.

S2: On the compressibility suite, raw sum, group-z sum, reward-only bargaining, and MARBLE all collapse to black images, while FCB-NFT improves all four rewards. Table 8 further shows that even with the controller off, the residual-weighted matrix avoids collapse while the scale-matched covariance does not. This is an interesting finding.

S3: The scale-matched covariance and shuffled-q controls are well designed, separating the effect of the matrix shape from its scale and from finite-sample noise.

Critical weaknesses

W1: The motivation for the field Gram matrix M does not match the estimator M ̂ used in practice. The paper motivates M by its ability to capture whether reward-induced field responses reinforce, oppose, or are orthogonal to each other. However, M ̂ (Eq. 19) only weights reward outer products by a scalar residual energy and contains no directional information; it is essentially a residual-weighted covariance. In Table 4, the deviation of M ̂ from the scale-matched covariance is only 1.5 to 1.8 times the noise floor. Moreover, the field budget \Delta is calibrated with M̂ itself (Eq. 56), so the "conservative estimate" carries limited meaning.

W2: The feedback controller in Appendix C has a large effect on the results: in Table 8, turning it off drops the worst gain from 0.43 to 0.29. Yet the main text mentions it in only one sentence. Apart from reward-only bargaining, no baseline has a comparable mechanism, and all ablations on the OCR suite keep the controller on. This makes it difficult to attribute the improvement to the field constraint.

W3: In Table 3, M ̂ differs from the scale-matched covariance by only about 20%, yet the worst gain rises from 0.60 to 0.89, and the paper does not explain this. The mechanism proposed in Appendix D for the compressibility suite (that residuals from low-frequency changes correlate with the compressibility reward) is also not verified with data.

W4: Balance-oriented baselines are missing, such as Tchebycheff / max-min aggregation, geometric-mean scalarization, and a weighted sum with a variance trust region. In addition, the evaluation uses only the four reward models used in training and covers a single base model and prompt distribution, with no held-out metrics or human evaluation.

Questions for authors

Q1: Since M ̂(Eq. 19) is a scalar residual-weighted covariance with no directional information, can you state precisely which quantity in your method actually captures the "reinforce / oppose / orthogonal" structure that motivates the field Gram matrix? If none does, what is the remaining justification for calling M ̂a field Gram matrix?

Q2: Given that Δ is calibrated with (M ) ̂itself (Eq. 56), what independent evidence shows the resulting budget is conservative rather than self-consistent by construction? Please give a calibration check that does not use M ̂.

Q3: Table 4 reports M ̂'s deviation from scale-matched covariance at only 1.5–1.8× the noise floor. Does this imply M ̂is statistically distinguishable from a plain covariance? If so, what test establishes that, and if not, how should the reader interpret the estimator?

Q4: Table 8 shows worst gain drops from 0.43 to 0.29 when the controller is off. Can you report the field-constraint ablation with the controller off, so the improvement can be attributed to the field constraint rather than the controller?

Q5: In Table 3, a ~20% difference between M ̂and scale-matched covariance coincides with worst gain rising from 0.60 to 0.89. What mechanism explains this magnitude of gain from such a small estimator difference? Can you provide an ablation or diagnostic that isolates the cause?

Q6: For the compressibility suite (Appendix D), can you verify with data that residuals from low-frequency changes correlate with the compressibility reward, as the proposed mechanism claims? Without this, what is the basis for the mechanism?

Q7: Can you add the balance-oriented baselines you omitted—Tchebycheff / max-min aggregation, geometric-mean scalarization, and weighted sum with a variance trust region—so the field constraint can be compared against established balancing methods?

Q8: The evaluation uses only the four training reward models, one base model, and one prompt distribution, with no held-out metrics or human evaluation. Can you report held-out reward models and at least one additional base model / prompt distribution to establish generalization?

2 comments
AuthorsOpenReview ↗

Response to Reviewer gLVe

We thank the reviewer for the detailed questions.

Q1–Q2. Directional structure and conservativeness.

The directional interpretation belongs to the exact field Gram matrix $ M $ in Section 4.2. Section 4.3 calls $ \hat M $ an empirical field metric: it estimates the residual-weighted population upper bound and captures reward–residual-energy associations. We will make this distinction explicit in the abstract and conclusion: the estimator targets the population upper bound, and the training ablations use M^\hat M. General Response 3, G8 shows a controlled case where the bound preserves an interaction sign missed by covariance and selects a target closer to the exact-M solution. The inequality $ M\preceq\bar M $ follows from conditional Jensen under the stated assumption, before budget calibration. The toy checks population costs independently of the chosen budget. The upper-bound guarantee applies at the population level. The training metric estimates this bound from samples.

Q3–Q5. Noise, attribution, and the size of the reward gap.

Table 4 describes the matrix deviations relative to the shuffled-q noise floor. Training evidence for the metric’s contribution comes from General Response 1, G1: full metric versus covariance gives OCR Worst 0.85 versus 0.58 without feedback, and 0.89 versus 0.60 with feedback. The submitted shuffled-q control gives 0.61. The field budget binds in 100% of groups (Table 3), and Table 4 locates the additional structure primarily in off-diagonal interactions. Changing these terms can alter the preferred coefficient direction. G8 illustrates target-direction sensitivity, while General Response 1, G1 isolates the training benefit.

Q6. Image-structure diagnostics.

On compressibility rollouts at iterations 100/300/1,000, mean within-group correlations between residual energy q and image high-frequency energy are 0.02/0.17/0.28. At iteration 1,000, high-frequency energy correlates with compressibility at −0.55, and q with compressibility at −0.31. Pooled q–high-frequency correlation is 0.33. These measurements show an association between residual energy and image structure relevant to compressibility that strengthens later in training. We will revise Appendix D to describe this measured association and the metric-dependent controller effects in G6.

Q7. Balancing alternatives.

General Response 2, G3 reports percentile-min, percentile geometric mean, and weighted-sum/variance-budget baselines, compared with FCB-off. FCB-off has higher Worst than all three added alternatives on OCR and both completed alternatives on compressibility. G3 specifies the sample-level reward transformations and reports the percentile-min run's early degeneration and numerical failure.

Q8. Held-out evaluation.

General Response 1, G2 adds two evaluators not used for training and a second evaluation prompt distribution. FCB-NFT improves both external scores over GroupZ on both distributions.

I thank the authors for the constructive rebuttal. The added evidences to Q3–Q8 are helpful. However, the core soundness concern—the mismatch between the field Gram matrix motivation and the trained residual-weighted covariance M̂, which lacks directional information and is clarified rather than resolved. The controller attribution (Q4) also remains incomplete. I keep my scores unchanged.

Reviewer_Y9DFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for selecting the scalar optimality used in multi-reward DiffusionNFT training. It constructs a target distribution through a linear combination of centered rewards and leverages the fact that reward gains under this distribution can be computed from the reward covariance. It further expresses the required velocity-field displacement under the idealized population DiffusionNFT objective as a quadratic form. Combining these two geometries, the proposed Nash-bargaining convex program allocates improvements across rewards. In practice, the method employs an empirical field metric based on velocity-residual energy and a realized-gain feedback controller.

On two reward suites using SD3.5-Medium, FCB-NFT reports the highest worst-channel normalized gains of 0.89 and 0.43, respectively. Scale-matched covariance and shuffled-q controls, along with metric/controller ablations, provide evidence supporting the effectiveness of the proposed components.

Critical strengths
  • The proposed formulation provides a concrete target-selection approach tailored to the DiffusionNFT learning signal. It makes it theoretically clear what is being improved and what is being constrained, based on the reward gains induced by linear optimality and the assumption that vold(z)=E[v∣z]v_{\mathrm{old}}(z)=\mathbb{E}[v\mid z]. It also incorporates correlations between rewards into coefficient selection. This provides a meaningful alternative to simple reward aggregation.

  • The paper includes experiments that distinguish the effects of the field metric and controller, while demonstrating low computational overhead. In Table 3 on the OCR suite, the scale-matched covariance and shuffled-q controls achieve Worst scores of 0.60 and 0.61, respectively, compared with 0.89 for the full field metric. Since these comparisons change only the field matrix while keeping the controller enabled, they suggest that the structure introduced by residual weighting matters beyond simple scale differences. In Table 8 on the compressibility suite, the full metric with the controller disabled achieves 0.29, the covariance metric with the controller enabled achieves 0.06, and the full metric with the controller enabled achieves 0.43. These results indicate that the overall improvement cannot be attributed solely to the controller. The reported training time also increases only from 73.2 to 74.9 seconds per step, corresponding to approximately 2.3% overhead, which is a practical advantage.

Critical weaknesses
  • The paper appropriately distinguishes target-distribution gains from realized training gains and introduces a feedback controller to address their mismatch. However, the relationship between the theoretical quantities and actual training outcomes remains insufficiently characterized. The quantity Car0\frac{Ca}{r_0} represents the reward gains of the reweighted target distribution, while the population-optimal field response relies on the assumption vold(z)=E[v∣z]v_{\mathrm{old}}(z)=\mathbb{E}[v\mid z]. Although the ablations support the empirical usefulness of the proposed metric, they do not establish how accurately these theoretical quantities describe the realized updates or how departures from this assumption affect their usefulness.

  • The evidence for the reproducibility and generalizability of the reported performance gains is limited. Both experiments use the same SD3.5-Medium model and OCR prompt distribution, differing only in their reward configurations. Each cell in Table 8 is explicitly described as a single run, and the main tables do not sufficiently report uncertainty across training seeds. Although evaluating on 1,018 held-out prompts with identical initial noise across models is a strength, this does not replace an assessment of training-seed variability. In particular, Worst is a minimum-based metric and is also sensitive to specialist-based normalization. For example, the specialist–base difference for CLIPScore is only 0.055, meaning that a raw score difference of 0.01 translates into approximately 0.18 in normalized gain. The coefficient grid, validation split, and selection criterion for the tuned fixed sum baseline, as well as the configurations of MARBLE and APEX, should also be described more clearly.

Questions for authors
  • How many training seeds were used for the main tables and Table 3? Were any additional repeated runs conducted for the results in Table 8? What were the tuning and selection procedures for the tuned fixed sum, MARBLE, and APEX baselines?

  • Could the authors quantify the relationship between predicted target gains and realized gains in subsequent rollouts, accounting for the rollout model’s EMA update? For example, per-reward calibration plots, slopes, or correlations across field-displacement budgets would help assess whether the theoretical quantities remain informative about actual training outcomes.

  • Have the authors conducted experiments on any models other than SD3.5-Medium?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Y9DF

We thank the reviewer for the reproducibility and calibration questions.

Q1. Seeds and baseline configurations.

The original main tables, Table 3, and Table 8 report individual runs. General Response 2, G4 adds three-seed evaluation for FCB-NFT and GroupZ. Their average Worst scores are 0.894 ± 0.006 versus 0.771 ± 0.017, and average Mean scores are 0.938 ± 0.023 versus 0.858 ± 0.025. G4 also lists the Worst gain of each individual seed. All four raw-reward averages favor FCB-NFT, with per-reward sample standard deviations reported.

The tuned fixed sum uses a coefficient grid with 0.5 increments. MARBLE uses unit-L2 gradient normalization, a solver interval of 10 iterations, and coefficient EMA 0.7. APEX equally weights learning potential, conflict, and progress need, with two-stage z-score normalization and per-epoch updates. We will add the fixed-sum search range, selected coefficients, validation split, and selection criterion to Appendix B, together with the adapted baseline schedules. G3 defines the added balancing baselines.

Regarding normalization, for every multi-reward method in submitted Table 1, the lowest normalized gain occurs in a reward other than CLIPScore. Using the best multi-reward improvements in that table as denominators preserves the ordering, with Worst approximately 0.90 for FCB-NFT versus 0.86 for GroupZ. Raw rewards and external evaluators provide complementary comparisons (G2 and G4).

Q2. Predicted versus realized gains.

General Response 3, G7 gives the full protocol: 25-iteration windows, three-iteration endpoint averages, and historical rescaling from the preceding four windows. Outcomes are measured on pre-update EMA rollouts. The EMA ramp ends at iteration 208, and the evaluation uses the 31 complete windows starting at iteration 225 or later. Target gains correlate with subsequent realized improvements for HPSv2 and PickScore (Spearman 0.48 and 0.50), and historical rescaling substantially reduces raw RMSE. The table reports all four channels alongside zero-gain and past-mean forecasts. G8 complements this training diagnostic with an independent calculation of population field geometry.

Q3. Other models.

Given our computational budget, we focused the experiments on SD3.5-Medium. To assess generalization beyond the original evaluation setting, G2 adds evaluations on 2,048 general-purpose prompts and two reward evaluators held out from training, ImageReward and LAION Aesthetic v2. FCB-NFT outperforms GroupZ on all five reported metrics on the general-purpose prompts.

Thank you for your response. The controller-off comparison convinced me that the field metric helps balance the OCR rewards. I’m keeping my rating at weak acceptance because the link to actual training gains is still uncertain. In the OCR run, predicting zero gain gave a lower RMSE than the rescaled forecast for all four rewards.

Reviewer_tLWFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses multi-reward fine-tuning with DiffusionNFT, where several rewards must be combined into a single scalar optimality per image. It derives the reward gains induced by a linear combination of rewards, which follow from the reward covariance, and a field matrix measuring how much the target moves the velocity field. It then selects the combination through a convex Nash bargaining program under a field-displacement budget (FCB-NFT). On SD3.5-Medium with two four-reward suites, FCB-NFT improves all rewards and achieves the best worst-channel normalized gain.

Critical strengths

The target problem is clear, and so is the approach.

Since rewards enter DiffusionNFT only through a scalar optimality, the paper frames multi-reward training as choosing that target. It predicts each reward’s gain from the reward covariance and the cost of the target from the NFT field response. The resulting design is clean: a small convex program that leaves the NFT loss unchanged and adds little overhead.

Critical weaknesses

Weakness 1. The “exact” geometry does not describe the update that is actually trained, and the gap is closed by a hand-designed controller that appears only in the appendix.

Proposition 1 and Theorem 1 are exact only for the reweighted target under the current rollout distribution, and Theorem 1 further assumes that the old model is the conditional mean of the forward process. Training instead takes one AdamW step on a LoRA with an EMA anchor, so the realized reward changes differ from the prediction. The paper itself notes that the field displacement measures the requested population-optimal response, independently of whether finite optimization reaches that field. Appendix C states that predicted and realized gains “diverge systematically” and that, without correction, the compressibility suite ends in blank images. Table 8, however, shows that this collapse occurs only when the field metric is also replaced by the scale-matched covariance. The fix is a feedback controller with at least six hand-set constants (three design quantities and three fixed constants, App. C), which the main text mentions only as calibrating priorities during training. Its effect is large: Table 8 shows that it moves the worst gain from 0.29 to 0.43 with the field metric, and from collapse to 0.06 without it. On the OCR suite the controller is never ablated, and all comparisons in Table 3 are run with it on (App. D). In addition, the matrix used in practice is an estimate of a population upper bound M̄, computed from 12-image groups with one probe per image. Target selection depends on its shape, and whether that shape matches M is never checked, even on a toy problem.

Weakness 2. The results show a different trade-off rather than a clear improvement over most baselines.

Evaluating only SD3.5-Medium on two reward suites is understandable given the cost. Within this setting, however, FCB-NFT does not dominate most baselines. On the OCR suite it dominates group-z summation and reward-only bargaining. It does not dominate the tuned fixed sum, MARBLE, APEX, global normalization or raw summation, all of which reach higher OCR (0.953, 0.944, 0.928, 0.929 and 0.962 vs. 0.878) at lower preference scores. Against these it moves along the trade-off rather than beyond it. On the compressibility suite it clearly avoids the collapse of several baselines, yet its compressibility is still lower than that of APEX and the tuned fixed sum (−0.122 vs. −0.060 and −0.090). The headline metric, the worst normalized gain, favors balanced operating points by design, so being best on it mainly reflects which point is selected. The priority sweep in Table 9 confirms this. Varying only the base OCR priority moves the worst gain between 0.42 and 0.89, and the default priority is exactly its maximizer. The claim of superiority therefore rests largely on the choice of metric rather than on a better trade-off.

Weakness 3. Insufficient presentation and discussion.

Figure 1 is hard to read. Text is so small and the role of each colors are not clearly distinguishable. Qualitative results are also very limited. The only example is Fig. 2, which shows six prompts on the compressibility suite, and there are no qualitative results for the OCR suite or in the appendix. More examples are needed to judge whether the balanced gains are visible in the images. In addition, multi-reward alignment is not specific to DiffusionNFT, but the paper does not discuss how its approach extends to other reward fine-tuning methods. The paper should at least discuss, for example in the related work, how the approach relates to the two main families. Other reward-value-based (zeroth-order) methods such as Flow-GRPO [1] likewise reduce rewards to a scalar advantage, so the target-selection view may carry over. However, both the gain map and the field metric are derived for NFT’s reweighting and loss. Reward-gradient-based (first-order) methods such as Adjoint Matching [2] and Efficient Adjoint Matching [3] use reward gradients directly, so neither the gain map nor M applies as is. Without this discussion, the contribution reads as a fix for one training algorithm rather than a general approach to multi-reward alignment.

References

[1] Liu et al., Flow-GRPO: Training Flow Matching Models via Online RL, NeurIPS 2025.

[2] Domingo-Enrich et al., Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control, ICLR 2025.

[3] Shin et al., Efficient Adjoint Matching for Fine-tuning Diffusion Models, NeurIPS 2026.

Questions for authors

Please see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer tLWF

We thank the reviewer for distinguishing attribution, trade-off selection and the scope of the theory. Shared numerical results are in General Responses 1–3.

W1. The field-metric benefit persists without the controller.

General Response 1, G1 adds the OCR ablation. With the complete controller disabled, the full metric reaches Worst 0.85, versus 0.58 for covariance and 0.57 for identity. On compressibility, the full metric alone avoids collapse and reaches 0.29, versus −0.51 for covariance/off. These comparisons isolate a benefit independent of adaptive priorities and channel removal.

We will correct Appendix C's collapse statement to identify covariance/off as the collapsing configuration in the 2×2 ablation. With the full metric, feedback further reallocates improvement from compressibility to preference rewards, raising Worst from 0.29 to 0.43. We will describe both components in the main text.

Connecting population targets to training outcomes. The gain map evaluates the reweighted target, and the field response describes the population NFT optimum. G7 compares target gains with subsequent EMA-rollout improvements using historical scaling and 25-iteration windows. Target gains have Spearman correlations of 0.48 for HPSv2 and 0.50 for PickScore with realized improvement. G5 additionally reports late-training efficiency variation: SD(log₂η) is 0.25–0.29 on OCR and 0.30–0.44 in the compressibility run.

The practical metric agrees with exact geometry in a controlled setting. G8 uses independently computed conditional fields. The population bound selects a target 5.8° from exact-M, versus 62.8° for covariance. With B=12 and L=1, the empirical metric is closer in 74% of 2,000 groups. This provides direct evidence that residual weighting retains field information useful for target selection in the toy.

W2. Evidence for balanced improvement.

Our goal is balanced multi-reward improvement, so Worst directly measures the least-improved reward. The evidence also extends beyond this metric: FCB-NFT improves all four raw rewards over GroupZ and reward-only bargaining on the OCR suite.

Against the tuned fixed sum, MARBLE, APEX, and global normalization, the results reflect a trade-off: OCR decreases by 0.050–0.075, while PickScore increases by 0.046–0.070, HPSv2 by 0.047–0.066, and CLIPScore by 0.011–0.024. Mean normalized gain also increases to 0.93, compared with 0.66–0.73 for these four methods. Thus, the improvement is visible in both the minimum and average normalized gains, supporting stronger balanced performance at the evaluated operating points.

The default base priorities are all 1, representing the symmetric Nash setting. Table 9 shows how changing this preference changes reward allocation. We will make the symmetric default and the role of the preference sweep explicit in Appendix E.

W3. Presentation and scope.

We will enlarge Figure 1 and its legend, add matched OCR qualitative comparisons and further compressibility examples, and identify the methods represented by blank outputs. Value-based methods such as Flow-GRPO share the scalar-signal selection problem, but transferring the gain map and field metric requires a derivation for their update rule. Reward-gradient methods such as Adjoint Matching and Efficient Adjoint Matching instead expose gradient-space combination choices. We will expand this discussion to distinguish the broader motivation from the present NFT-specific construction.

Thank you for the response. The controller-off ablation (G1) and the toy check of M̄ (G8) address my concern about attribution, and I agree that all-ones priorities are a natural default. However, two points remain.

1. Predicted vs. realized gains. In G7, the rescaled forecast does no better than simply predicting zero gain on any channel (e.g., RMSE 0.017 vs. 0.016 for OCR), and the rank correlation is almost zero for OCR and CLIPScore. So the target geometry does not really describe the trained update, and the benefit seems to come from the empirical side.

2. Trade-off. As the response also notes, FCB-NFT gives up some OCR for higher preference scores compared with the tuned fixed sum, MARBLE and APEX. The new seeds, held-out evaluators and general prompts are only compared with GroupZ, so it is still unclear whether FCB-NFT is better than the strongest baselines.

For these reasons, I will keep my score.

Reviewer_CxBiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to combine multiple rewards into one scalar optimality signal for DiffusionNFT. It derives an exact covariance map from linear target coefficients to reward gains under the current reweighted rollout distribution, and a Gram matrix for the displacement of the population-optimal velocity field. FCB-NFT selects a target by maximizing Nash surplus subject to a field budget and valid-optimality constraints. On two four-reward suites, it reports the highest worst-channel normalized gain among the compared methods. I lean “weak acceptance” because the DiffusionNFT-specific target-selection formulation, sound population analysis, and substantial balanced gains with targeted ablations provide useful new knowledge. This is a close decision: the controller's contribution to the final outcome needs clearer operational evidence, and evaluation on another model or prompt distribution would strengthen the claim of broader utility.

Critical strengths
  • The target-selection perspective is well motivated for DiffusionNFT and is distinct from parameter-gradient balancing methods such as the paper which is published on https://arxiv.org/abs/2605.06507.
  • The covariance gain identity and population field-displacement derivation are clear under their stated assumptions. The paper appropriately distinguishes target gains from gains after finite neural-network training.
  • The experiments include relevant scalarization and adaptive baselines, a conflicting compressibility setting, and useful metric and controller ablations. The reported worst-channel gains are substantial: 0.89 on OCR and 0.43 on compressibility.
Critical weaknesses
  1. Appendix C explains why balanced predicted gains can diverge from realized gains, and Appendix D usefully separates the controller from the field metric on the compressibility suite. The metric alone avoids collapse in that ablation, while adding the controller changes the reward allocation and raises the worst-channel gain from 0.29 to 0.43. The paper does not report the stability of realization-efficiency estimates or how often disagreement floors are relaxed, channels are removed, or a group falls back to zero coefficients. These details matter when attributing the final outcome to the proposed target-selection geometry.
  2. Both suites use SD3.5-Medium and the OCR prompt distribution, sharing three rewards and changing the fourth. JPEG compressibility is a useful conflict stress test, but an independent prompt distribution, model, or quality assessment beyond the optimized reward channels would better establish transfer and practical image quality.
Questions for authors
  1. Appendix C (pp. 18–19) introduces a realized-gain feedback controller because predicted gains can diverge from training outcomes, and Appendix D, Table 8 (p. 19) shows that adding it to the field metric changes the compressibility-suite worst-channel gain from 0.29 to 0.43. How stable are its realization-efficiency estimates, and how often are disagreement floors relaxed, channels removed, or zero coefficients used? Can you report these diagnostics to clarify the controller's contribution?
  2. Does the balanced gain persist with a different model, a different prompt distribution, and an independent quality evaluation beyond the optimized rewards?
1 comment
AuthorsOpenReview ↗

Response to Reviewer CxBi

We thank the reviewer for the constructive assessment.

Q1. Controller contribution and operation.

General Response 1, G1 separates the field metric and feedback on both suites. The new OCR controller-off result is 0.85 with the full metric versus 0.58 with covariance. Compressibility improves from 0.29 without feedback to 0.43 with it. Thus the metric establishes a stronger operating point, and feedback further adjusts allocation on compressibility.

General Response 3, G6 reports controller operating frequencies on both suites.. Solves succeed without floor relaxation in 100% / 99.93% of groups, respectively. Reported relaxation and zero-fallback rates are each 0% / 0.04%. OCR is removed from the active objective in 24% / 27% of groups, with its coefficient set to zero and its reward-floor constraint retained. G5 directly quantifies efficiency stability: late-phase SD(log₂η) is 0.25–0.29 on OCR and 0.30–0.44 on compressibility, with per-channel priority dynamics and bound occupancy reported. We will move the controller into the main text.

Q2. Transfer and independent evaluation.

General Response 1, G2 adds ImageReward and LAION Aesthetic v2, neither used for training. FCB-NFT exceeds GroupZ on both evaluators on OCR prompts, and on all five reported metrics on 2,048 general-purpose prompts. These are evaluations of models trained on the OCR split, supporting transfer of measured quality gains.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response 4/4: Controller and geometry diagnostics

G8. Independent field-geometry check.

We examine a 2-D, two-component Gaussian-mixture rectified flow, with xt=(1−t)x0+tϵx_t=(1-t)x_0+t\epsilon, four analytic rewards (two Gaussian bumps and two tanh projections), and the trainer's 25-step probe grid. Analytic posteriors and Gauss–Hermite quadrature with 20 nodes per dimension provide numerical evaluations of the conditional-field moments. The empirical metric estimates the population upper bound $ \bar M=\mathbb E[q\tilde R\tilde R^\top] $. We compare it with the exact conditional-field Gram matrix M.

DiagnosticPopulation bound $ \bar M $Scale-matched covariance $ \bar q C $
Interaction signs matching exact M6/65/6
Angle to the exact-M Nash target5.8°62.8°

Angles use common z-units. R2/R3 have clean-reward correlation −0.18, normalized exact field interaction +0.56 and bound interaction +0.35. Across 2,000 sampled B=12, L=1 groups in this setting, the empirical metric selects a closer target than covariance in 74%, with median angle 5.5°.

In the analytic toy, the population upper bound covers the independently computed exact field cost in every tested nonnegative coefficient direction. Under 96 smooth old-field perturbations with 10% relative RMS magnitude, the difference between the recomputed bound and reference field-cost matrices remains positive semidefinite. We will include the full toy specification and perturbation protocol in the revised appendix.

AuthorsOpenReview ↗

General Response 3/4: Controller and geometry diagnostics

This comment collects the shared implementation and analytical diagnostics. G1 in General Response 1 provides the controlled training attribution.

G6. Controller rule and operation.

From iteration 25 onward, the controller tracks a 25-iteration realization ratio and applies

$ \eta_k^{raw}=\operatorname{clip}(G_k^{real}/G_k^{pred},0.02,5),\quad \eta_k\leftarrow0.95\eta_k+0.05\eta_k^{raw}, $

$ u_k=K\eta_k^{-1}/\sum_j\eta_j^{-1},\quad \tilde\omega_k=\operatorname{clip}(u_k,1/2,2),\quad \omega_k=b_k\tilde\omega_k. $

Before applying $ b_k $ (1 by default), $ \tilde\omega_k $ is capped at 1 at a reward ceiling. Table 7 states this rule in compact form. In full, the ratio is clipped so that $ \eta_k $ stays positive when a window has zero or negative realized gain, and priorities are normalized across channels, so only relative efficiencies enter the program. We will state both explicitly in Algorithm 1 and Tables 6–7 and describe the controller in the main text. The lower clip is active in approximately 50% of the checked channel–iteration observations. G5 reports the variation of the smoothed efficiency estimates and issued priorities during late training.

DiagnosticOCR suiteCompressibility suite
Feasible without floor relaxation100%99.93%
Floor relaxation0%0.04%
Zero-coefficient fallback0%0.04%
OCR removed from active objective24%27%
OCR ceiling cap binding52%69%

These rates cover 1,000 iterations and are rounded independently. Feasibility is assessed after active-channel selection. Solver outcomes and removal are per group, while ceiling-cap binding is per iteration. All 48,000 solves in the reported OCR run succeed without relaxation/fallback. Removed channels have coefficient zero and retain reward-floor constraints. G5 reports the late-phase efficiency variation and priority dynamics for each channel.

On compressibility, feedback under the full metric changes normalized gains by 0.74→0.43 for compressibility, 0.43→0.62 for PickScore and 0.29→0.65 for HPSv2. Identity/off retains OCR 0.975 and compressibility −0.037, but HPSv2 falls to 0.122 versus base 0.192. This configuration avoids pure-black collapse but sacrifices HPSv2, whereas identity/on collapses. The controller’s effect therefore depends on the field metric.

G7. Predicted versus realized gain (OCR suite).

We use the 1,000-iteration OCR run and non-overlapping 25-iteration windows [s, s+25). The predicted gain PkP_k sums the group-averaged target gains issued by updates between the endpoint blocks. The realized gain YkY_k is the difference in mean rollout reward between the three-iteration blocks [s,s+3)[s, s+3) and [s+22,s+25)[s+22, s+25), with 576 samples per iteration. Rollouts are sampled from the EMA model before each update, so YkY_k measures changes in the sampling policy. The EMA decay is zero before iteration 75 and ramps to 0.999 over iterations 75–208. Historical scaling uses the gains observed after this EMA smoothing.

The rescaled forecast is η^kPk\hat\eta_k P_k, where η^k=clip⁡(∑Yk/∑Pk,0.02,5)\hat\eta_k=\operatorname{clip}(\sum Y_k/\sum P_k,0.02,5) uses the previous four windows. This offline scaling uses only preceding outcomes. Baselines predict zero gain or the mean realized gain over the same historical windows. Evaluation covers the 31 complete late-training windows starting at iterations 225, 250, …, 975; RMSE is in specialist-minus-base units.

ChannelRMSE: raw → rescaledZero-gain RMSEPast-mean RMSESpearman(P, Y)
HPSv20.068 → 0.0170.0160.0180.48
PickScore0.109 → 0.0380.0360.0400.50
OCR0.179 → 0.0170.0160.0170.01
CLIPScore0.266 → 0.0730.0680.078−0.06

Target gains carry information about the ordering of subsequent realized improvements for HPSv2 and PickScore, with Spearman correlations 0.48 and 0.50. Historical rescaling substantially reduces the raw prediction error on all four channels, bringing RMSE close to the simple forecasting baselines.

AuthorsOpenReview ↗

General Response 2/4: Balancing baselines, seeds and controller dynamics

This comment collects the shared baseline, seed and controller-dynamics results. The controller-off attribution and external evaluations are in General Response 1 (G1–G2).

G3. Balance-oriented baselines.

Entries are Worst / Mean normalized gain. New baselines are single runs with no controller. FCB-off is the direct comparison.

MethodOCR Worst / MeanCompressibility Worst / Mean
GroupZ0.76 / 0.86−0.51 / −0.02
Tuned fixed sum0.57 / 0.730.08 / 0.49
APEX0.51 / 0.660.24 / 0.59
Percentile-min scalarization0.00 / 0.38Failed run
Percentile geometric mean0.58 / 0.73−0.51 / −0.02
Weighted sum + variance trust region0.53 / 0.650.13 / 0.60
FCB-NFT, controller off0.85 / 0.890.29 / 0.59
FCB-NFT, controller on0.89 / 0.930.43 / 0.67

The first two new baselines map each reward to its mid-rank percentile among 576 rollouts, then apply a per-image minimum or equal-weight geometric mean before standard NFT mapping. The variance baseline solves $ \max_{a\ge0}w^\top\hat C a $ under $ a^\top\hat C a\le\Delta $ and the optimality box, using tuned fixed-sum weights and the same single-reward budget-calibration principle.

On compressibility, percentile-min reaches OCR 0 at step 25 and encounters fp16 overflow/NaN at step 668; all methods use the same fp16 training recipe. FCB-off has higher Worst than all three added alternatives on OCR and both completed alternatives on compressibility.

G4. Three training seeds.

We report results from three additional training seeds on the OCR suite. All entries report mean ± sample standard deviation across three training seeds. For each seed, Worst and Mean are computed from the four training rewards using the fixed base/specialist normalization in Eq. (24), then summarized across seeds. ImageReward and Aesthetic v2 are reported separately as external evaluators.

MethodOCRPickScoreHPSv2CLIPScoreImageRewardAesthetic v2Avg. WorstAvg. Mean
GroupZ0.852 ± 0.0060.893 ± 0.0030.328 ± 0.0070.321 ± 0.0021.323 ± 0.0265.665 ± 0.0160.771 ± 0.0170.858 ± 0.025
FCB-NFT0.879 ± 0.0050.918 ± 0.0040.337 ± 0.0040.325 ± 0.0021.445 ± 0.0285.842 ± 0.0260.894 ± 0.0060.938 ± 0.023

Per-seed Worst normalized gains are:

MethodSeed 1Seed 2Seed 3Binding channel in all three seeds
GroupZ0.7830.7520.777PickScore
FCB-NFT0.9000.8880.894OCR

All four training-reward averages and both external-evaluator averages favor FCB-NFT.

G5. Controller dynamics.

We summarize controller logs from training runs on the OCR and compressibility suites. Controller phases are warm-up 25–74, EMA ramp 75–208, and late training 209–999. The table reports late-training statistics; the initial fixed-priority iterations 0–24 are excluded.

SuiteChannelMedian ηSD(log₂η)Median ωMean per-step |Δlog₂ω|Lower / upper-bound occupancy (%)
OCRCLIPScore0.210.250.670.0593 / 0
OCRHPSv20.160.270.860.0550 / 0
OCROCR0.080.261.000.3410 / 4
OCRPickScore0.180.290.750.0520 / 0
CompressibilityJPEG0.110.441.430.0501 / 1
CompressibilityHPSv20.590.400.500.00297 / 0
CompressibilityOCR0.090.301.000.3330 / 9
CompressibilityPickScore0.390.320.500.01380 / 0

Late-training SD(log₂η) is 0.25–0.29 on OCR and 0.30–0.44 on compressibility. Priorities track changes in relative efficiency: in the compressibility run, JPEG receives priority 0.5 throughout warm-up and ramp, then has median priority 1.43 in late training. The larger per-step OCR-priority changes reflect switching of the ceiling cap between the efficiency-based priority and 1.

AuthorsOpenReview ↗

General Response 1/4: Main results and external evaluation

We thank the reviewers for their constructive feedback and recognition of FCB-NFT's contributions. The reviews highlight our clear and well-motivated target-selection formulation (CxBi, tLWF), explicit analysis of reward gains and field responses (CxBi, Y9DF), and well-designed ablations supporting the value of the field metric (Y9DF, gLVe). Reviewers tLWF and Y9DF also emphasize the clean design and low computational overhead.

Our rebuttal addresses three shared concerns: isolating the field metric's contribution from the controller, clarifying the connection between population geometry and practical training, and strengthening the evidence for balanced gains, reproducibility and generalization. We provide controller-off ablations, analytical and calibration diagnostics, additional balancing baselines, three-seed results, and evaluations beyond the training rewards and prompt distribution. The shared evidence is collected in the General Response comments, followed by point-by-point replies to each reviewer.

G1. Field metric × controller.

Worst normalized gain is shown below. GroupZ denotes the group-z-score sum baseline. Each cell is a single run. OCR controller-off cells and compressibility identity/off are new. Other cells are from submitted Tables 3 and 8.

Field matrixOCR, controller offOCR, controller onCompressibility, offCompressibility, on
Identity, reward-only bargaining0.570.58−0.45−0.51
Scale-matched covariance0.580.60−0.510.06
Full field metric0.850.890.290.43

Without feedback, the full metric reaches OCR Worst 0.85, versus 0.58 for covariance and GroupZ's 0.76. Its Mean is 0.89, versus 0.71 for covariance/off and 0.86 for GroupZ. On compressibility, the metric alone avoids collapse and reaches 0.29, exceeding covariance/on (0.06). The field metric establishes this advantage. With the full metric, feedback further reallocates improvement toward preference rewards on compressibility.

Controller off fixes priorities at 1 and retains every channel with nonzero group variance. Both priority adaptation and controller-based channel selection are disabled. The relaxation ladder and budget-calibration rule remain unchanged.

G2a. External reward evaluators.

ImageReward v1.0 and LAION Aesthetic v2 are held out from training. We evaluate single-run final checkpoints on 1,018 paired OCR prompts.

MethodImageRewardAesthetic v2
Base−0.3035.251
Raw sum−1.5804.285
Global normalization1.0365.354
GroupZ1.3355.681
Reward-only bargaining1.2015.641
FCB-NFT, controller off1.4735.795
FCB-NFT, controller on1.4565.823

FCB/on exceeds GroupZ by 0.121 / 0.142, scoring higher on 61% / 74% of prompts. FCB/off also exceeds GroupZ on both evaluators.

G2b. Transfer to 2,048 general-purpose prompts.

Both fine-tuned models were trained on OCR prompts. Evaluation uses the DiffusionNFT PickScore test split.

ModelPickScoreHPSv2CLIPScoreImageRewardAesthetic v2
Base0.7620.2130.244−0.4535.538
GroupZ0.8540.3370.3011.3126.493
FCB-NFT0.8970.3440.3101.3606.624
FCB-NFT − GroupZ+0.043+0.007+0.009+0.048+0.131
Prompts scoring higher with FCB-NFT55%56%56%63%66%

FCB-NFT improves all five metrics over GroupZ, supporting transfer of the quality gains to general-purpose prompts. These prompts contain no specified target text, so we evaluate quality and fidelity metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.