acceptodds
Under review as a conference paper at ICLR 2027

Elucidating the Design Space of Regression-based Diffusion Reinforcement Learning

Abstract

A nascent family of methods that forgoes the policy gradient and reweights a supervised regression instead has garnered momentum in reinforcement learning for diffusion and flow models. DiffusionNFT, FlowAWR, and RAM are representative regimes with contrasting motivations. It is yet opaque what, if anything, they share. We substantiate that each is the solution of one divergence-constrained reward-maximization problem, and they are differentiated only by the convex generator that defines the constraint. Under the unified modeling framework, we unravel the relaxations that prior art made during building the advantage-embedded regression target: approximating the KKT condition and/or posterior normalizer for the linear and exponential tilt shapes DiffusionNFT and FlowAWR respectively, while preserving the exact sparsemax projection onto the probability simplex for linear tilt leads to another superior model type in this work. Beyond the theoretical underpinnings, we further empirically investigate the design space and shed light on the training recipe for regression-style diffusion RL. Retaining the merits discovered during our exploration gives rise to DiffusionRFT, our paradigm that converges faster, trains more stably, and attains the top performance.

Primary area:
generative models
Keywords:
diffusion models, reinforment learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_748BOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents an interesting unified perspective on regression-based diffusion RL, with meaningful theoretical insights and well-motivated experimental analyses. The connection between different reward weighting schemes and the proposed sparsemax-based improvement is particularly valuable. However, the novelty beyond existing policy optimization frameworks could be better clarified. Given that theoretical unification is the main contribution, the paper would benefit from more rigorous mathematical presentation, including explicit assumptions, lemmas, and theorems, as well as a summary diagram or table. Moreover, the experimental comparisons need stronger controls, and broader evaluations across different backbones and samplers would make the conclusions more convincing.

Critical strengths
  1. The unified perspective is interesting and potentially valuable. The paper connects several diffusion RL methods through ff-divergence regularization and derives sparsemax from Pearson-χ2\chi^2. I particularly like that the analysis leads to a concrete algorithmic improvement rather than simply reinterpreting existing methods.
  2. The experiments are closely connected to the theoretical insights. For example, the authors measure when the dense-linear approximation becomes invalid and relate this to training collapse. This helps explain why the proposed method works, rather than just reporting better rewards.
Critical weaknesses
  1. The novelty of the proposed framework could be better justified. Reward-weighted regression, advantage-weighted policy optimization, and sparsemax have been studied in prior work, such as Reward-Weighted Regression, Advantage-Weighted Regression , and From Softmax to Sparsemax. The paper would benefit from a clearer discussion or a schematic figure of what is fundamentally new beyond combining these existing ideas.

  2. The unification claim seems somewhat overstated. While the methods share a similar idealized policy improvement objective, their practical implementations differ in important ways. The paper does not fully establish that these algorithms can be unified solely through different reward weighting functions.

  3. The experimental comparisons may not fully isolate the source of improvement. For example, sparsemax may improve stability partly because it reduces the effective update size, rather than because of the proposed theoretical motivation.

  4. The experiments are mostly limited to SD3.5-Medium. It remains unclear whether the proposed design principles generalize to different backbones, samplers, or reward models.

  5. The main contribution of this work appears to be its theoretical unification of existing methods. However, the mathematical presentation lacks sufficient rigor and structure, with few clearly stated assumptions, lemmas, or theorems.

Questions for authors
  1. What is the main conceptual novelty compared with existing reward-weighted regression and sparsemax-based methods? Could the authors clarify which insights are genuinely new?

  2. How much of the improvement from sparsemax comes from the projection itself, rather than simply reducing the effective update size? Could the authors compare against clipping baselines with matched gradient norms or update magnitudes?

  3. How fair are the comparisons with existing methods when they are reimplemented within the proposed unified framework? Could the authors include comparisons against the original implementations with their recommended settings?

  4. Do the main findings hold on another diffusion backbone or with a different sampler? In particular, are the advantages of sparsemax and rolling anchors consistent across different training settings?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 748B (Part 1)

We sincerely thank the reviwer for your positive feedback. We have added further clarification regarding the issues mentioned in your comments.


W1 / Q1. Comparison to prior work

RWR, AWR and sparsemax are related work that our draft cites, but not in the same domain of diffusion generative models.

  • sparsemax (Martins & Astudillo, 2016) is a mathematical tool from attention mechanisms. We do not claim it. Instead, we show that in diffusion RL it is derived as the KKT solution of the χ2\chi^2-constrained improvement problem, where the constraint that becomes active is the feasibility condition (Eq. 20) that a dense linear weight violates on 86.8% of training steps.
  • AWR / RWR (Peng et al., 2019) come from robotics, where the regression target and reward share the same variable. In diffusion the target lives on xtx_t while the reward is on clean endpoints, so the weight must pass through the noising kernel. Eq. 14 shows the flow-optimal target differs from AWR by a state-dependent factor rt=z(xt)/Zr_t=z(x_t)/Z that has no scalar-action counterpart. The diffusion-domain instantiation of this family is AWM (Xue et al., 2026), which under a unified protocol even underperforms DiffusionNFT on GenEval. Please refer to RAM (Bergmeister et al., 2026), Figure 4 for the performance comparison.

W2. The unification and differences

The paper separates exactly the two levels the reviewer distinguishes: Sections 2–3 unify the form (at the level of the population regression target, all three methods solve one reward-maximization problem under an ff-divergence constraint, differing only in the convex generator), while Section 4 varies the implementation axes the target leaves free — loss space and normalization, anchor, timestep law, baseline — one at a time.

We do not claim the algorithms differ only by a weighting function at the level of implementation. We add this table in the draft, placing the unified target beside the axes Section 4 tests:

AxisDiffusionNFT (official)FlowAWR (arXiv v1)DiffusionRFT
Tilt (divergence)χ2\chi^2 interior branch (dense linear)exponential (KL)sparsemax (exact χ2\chi^2)
Effective temperatureCβσC\beta\sigma (5σ5\sigma on GenEval)reward std (=σ=\sigma)σ\sigma
Clippingz∈[−5,5]z\in[-5,5] (active on 0.2%–1% of samples)nonenone (feasibility by construction)
Loss space / normalizationxx-space, per-samplevv-spacexx-space, per-sample
Regression anchorrolling voldv_{\rm old} (EMA)rolling voldv_{\rm old} (EMA)rolling voldv_{\rm old} (EMA)
Timestep lawtrajectorytrajectorytrajectory
Endpoint normalizer Z(c)Z(c)estimated as Rˉ(c)\bar R(c)estimated as Z(c)Z(c)exact: Z=1Z=1 from the water-filling constraint by construction
Baselineconstantconstantb∗b^*

One row deserves comment. The endpoint normalizer is a group average the other two tilts must estimate; for the projected tilt the constraint that defines it already enforces Epold[ρ]=1\mathbb{E}_{p^{\rm old}}[\rho]=1, so Z=1Z=1 exactly. That is a property of the construction, not of tuning — and we do not overstate it: the posterior normalizer z(xt)z(x_t) is set to one in every member of the family, ours included, so what the projection removes is the endpoint-level approximation only.

Revision. The abstract is rewritten; Section 2 gains the table.


W3 / Q2. Is sparsemax simply taking smaller effective steps?

No. Two families of runs hold update magnitude fixed (GenEval, constant baseline, single seed) — dense weights across temperatures, and clipped baselines that freeze the advantage scale:

Configurationearly E∣A∣\mathbb{E}\lvert\mathcal{A}\rvert (ep 10)Outcome
dense @ σ\sigma0.59collapses to 0.718 by ep 150
dense @ 2σ2\sigma0.29declines from ep 180, 0.808 at ep 200
dense @ 5σ5\sigma (= DiffusionNFT)0.12stable
lower clip max⁡(A,−1)\max(\mathcal{A},-1) @ σ\sigma0.54fully degenerate by ep 70
symmetric clip ∥A∥≤1\|\mathcal{A}\|\le1 @ σ\sigma0.50stalls at 0.72–0.73
sparsemax @ σ\sigma0.51stable

The decisive cell is dense @ 2σ2\sigma: over ep 70–130, before it collapses, its E∣A∣\mathbb{E}|\mathcal{A}| averages 0.165 — below sparsemax's 0.197 over the same window — and it collapses while sparsemax does not. The clipped baselines freeze the advantage scale and keep every weight non-negative, yet the lower clip collapses earliest of all; the projection differs in re-solving the threshold after clipping, so the advantages still sum to zero. At matched gradient norms the picture is unchanged: clipping never triggers for dense @ 5σ5\sigma, the lower clip or the clip-free DiffusionNFT variant (max 0.87, 0.59, 0.94), so those three are already compared under one cap, while dense @ 2σ2\sigma triggers it on 17% of steps before collapsing and is not saved.

AuthorsOpenReview ↗

Response to Reviewer 748B (Part 2)

W4 / Q4. Other backbones, samplers and rewards

Samplers. For each method's checkpoint near its evaluation peak, we re-evaluated on the full GenEval test set with a first-order Euler sampler, a second-order Heun (ODE) sampler, and the SDE sampler used during training (40 steps, CFG 1.0, fixed seed):

GenEvalEulerHeunSDE
DiffusionNFT0.9420.9430.941
FlowAWR0.9380.9370.936
DiffusionRFT0.9760.9800.977

Scores move by at most ±0.004 and the ordering is unchanged, so the gains do not depend on the sampler.

Reward models. Appendix G already reports PickScore. The conclusion is consistent with the main paper.

Backbone. We chose SD3.5-Medium because all the methods we compare against use it as their primary platform, which lets us run their official code under their recommended configurations — and is what makes the curve-level comparison possible. In addition, we do not afford a second backbone exploration within the discussion period. We list it as a limitation rather than report a rushed result. The theoretical results use only the structure of the flow-matching target and the advantage distribution, which is expected to be backbone-agnostic.


W5. Rigor mathematical presentation

We thankfully agree and we have restructured Sections 2–3 into explicit assumptions and propositions, stated in the order the derivation needs them, with proofs retained in Appendices B–F:

  • Assumption 1 (standing). A1: the experience buffer is frozen within one update, so poldp^{\rm old} and R(c,x0)R(c,x_0) do not depend on θ\theta. A2: the noising kernel is Gaussian, q(xt∣x0)=N((1−t)x0,t2I)q(x_t|x_0)=\mathcal{N}((1-t)x_0,t^2I), and the anchor is the corresponding marginal velocity vold(xt,c,t)=E[ut∣xt,c]v^{\rm old}(x_t,c,t)=\mathbb{E}[u_t\mid x_t,c]. A3: the expectations used are integrable and z(xt,c,t)>0z(x_t,c,t)>0 a.s.
  • Proposition 1 (tilt to velocity target), Eq. 9: for any endpoint reweighting ww, the velocity target depends on the tilt only through κ=w/z\kappa=w/z. Proof in Appendix B.
  • Proposition 2 (z→Zz\to Z substitution), Eq. 14: the substitution rescales the displacement by rt=z/Z>0r_t=z/Z>0 without changing its direction, with E[rt]=1\mathbb{E}[r_t]=1 and Var⁡(rt)≤E[A2]\operatorname{Var}(r_t)\le\mathbb{E}[\mathcal{A}^2] decreasing towards the noisy end. Proof in Appendix D.
  • Lemma 1 (single-sample unbiasedness), Eq. 15: the per-sample objective differs from the exact-target one by a constant independent of θ\theta, namely the conditional variance of the drawn target, so gradients and minimizer are unchanged. Proof in Appendix E.
  • Proposition 3 (optimal tilt under a divergence constraint), Eqs. 17–18: the maximizer exists, is unique, and is fixed by ρ=(f′)−1((R−λ)/γ)\rho=(f')^{-1}((R-\lambda)/\gamma) — a KL constraint gives the exponential tilt, a Pearson-χ2\chi^2 constraint gives the water-filling solution whose group form is sparsemax (Eq. 21). Proof in Appendix F.
  • Proposition 4 (feasibility of the interior branch), Eq. 20: the dense linear weight solves the χ2\chi^2 problem if and only if min⁡iA>−b\min_i\mathcal{A}>-b holds for every sample.
  • Proposition 5 (identification of DiffusionNFT): whenever the clip in Eq. 11 is inactive, the official DiffusionNFT weight coincides sample by sample with the interior branch at γ=Cβσ\gamma=C\beta\sigma and b=1b=1; since a group-standardized reward obeys ∣zi∣≤G−1|z_i|\le\sqrt{G-1}, Eq. 20 holds for the whole linear family exactly when G≤1+(βC)2G\le1+(\beta C)^2. This is what makes the heuristic testable: at its released 5σ5\sigma the criterion holds on every logged step, and the same weight at γ=σ\gamma=\sigma violates it on 86.8% of them and collapses.

The Remark closing Section 3 now reads off these propositions rather than restating them.


Q3. Is the comparison fair?

Yes, our principle: where official code exists, use it under its recommended settings; where it does not, implement the paper faithfully.

  • DiffusionNFT. Our codebase is built on the official repository; the dashed curve in Fig. 1 is the official loss under the official GenEval configuration, as are the PickScore experiments (W4). We can now also state the relation precisely: the official loss is second-order equivalent to our dense weight at γ=5σ\gamma=5\sigma (3uSR Q3), and training that configuration from scratch in our code reproduces the official curve.
  • RAM. Fig. 3a includes the official code under its recommended settings alongside an independent reproduction; the two agree.
  • FlowAWR. No code was public at submission time, so we implemented its tilt and adaptive temperature from arXiv v1. To isolate the tilt, that cell uses the unified loss geometry; the paper's vv-space loss corresponds to the vv-prediction results in Fig. 2. Checking the later-released code item by item, its loss geometry, anchor and timestep law agree with ours.
AuthorsOpenReview ↗

Follow-up Before the Discussion Period Ends

Dear Reviewer,

Thank you again for your careful review and thoughtful feedback. We have sincerely tried to address your concerns through our responses, additional experiments, and draft revisions in blue. As the discussion period draws to a close, if time permits, we would greatly appreciate learning which concerns remain unresolved or where our explanations are still insufficient. Your perspective would help us identify what we may have missed, and we would welcome the opportunity of further clarification.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_3uSROpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper studies regression-based reinforcement learning for diffusion models through a shared formulation of divergence-constrained reward maximization. The proposed method, DiffusionRFT, combines sparsemax weighting with a variance-minimizing baseline. The paper also investigates the choice of regression parameterization.

Critical strengths

The comparison of three weighting rules across two baselines (Figure 1) helps distinguish the contribution of the sparsemax projection from that of the proposed baseline. Isolating these effects makes the empirical findings more informative and provides useful practical guidance for diffusion fine-tuning.

Critical weaknesses
  1. I found the paper challenging to follow, even after several careful readings. The abstract and introduction would benefit from a clearer narrative and more precise descriptions of the contributions. For example, “leads to another superior model type in this work” (line 21) leaves the model type and the basis for its superiority unclear. Similarly, “Beyond the theoretical underpinnings, we further empirically investigate the design space and shed light on the training recipe for regression-style diffusion RL” (lines 23–24) would be more informative if it explicitly identified the aspects of the design space being studied and the main findings.

  2. Much of the KL/exponential-weighting derivation appears to overlap with flowAWR. While the χ2\chi^2/KKT derivation and sparsemax projection introduce additional technical developments, I would appreciate a clearer explanation of the conceptual novelty beyond these design choices and why it is consequential.

  3. The paper does not include visual examples, which makes it difficult to assess how the reported improvements translate into generation quality. Representative qualitative comparisons, potentially complemented by a user study, would strengthen the evidence for the practical benefits of the proposed changes.

Questions for authors
  1. Section 4.2, starting at line 460, states that for the RAM variant, “timesteps are drawn from a uniform distribution.” However, Appendix D of the original RAM paper states: “During training we sample the timestep t from the linear density p(t) = 2t on [0, 1], biasing toward values near the noise endpoint t=1 where generation begins. This notably enhances training stability over uniform sampling.” This difference may affect the interpretation of the results discussed at line 468. Could the authors clarify the choice of uniform sampling and provide an additional comparison using RAM’s recommended p(t)=2tp(t)=2t sampling strategy, with the other settings held fixed?

  2. As I understand it, flowAWR uses advantage scaling for GenEval: “rescale the centered advantages by their within-group standard deviation to adapt the correction magnitude.” It also includes additional safeguards against extreme amplification and training instability. In the submission, I could not find details on scaling and normalization beyond Equation (13). Could the authors clarify the procedure used in their flowAWR implementation and whether it follows these aspects of the original training recipe?

  3. For DiffusionNFT, Equation (11) appears to omit clipping and differs from the formulation in the original paper. Could the authors clarify these differences and whether clipping is included in their implementation? Since clipping may affect the subsequent derivations and conclusions, it would be helpful to explain how it is handled in the analysis. Providing the implementation code in the supplementary material would also help clarify these questions.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3uSR (Part 1)

We thank the reviewer for a careful review and the three questions about the implementation details. We address the concerns one-by-one below.


Q1. The loss timestep law attributed to RAM

RAM's paper uses p(t)=2tp(t)=2t and U[0,1]\mathcal{U}[0,1] is one by our cosntruction. We implement 2t2t, as the paper and your reading specify. It is stable, which sharpens rather than weakens L467–468: the axis turns on coverage of the low-noise band, and 2t2t is an intermediate dose the submission lacked.

The affected scope is one label. The curve named "RAM's sampling scheme" in Fig. 4 and at L461 is our own uniform law, and nothing else in the paper uses it: both RAM curves in Fig. 3a run RAM's official law — our port matches its sample_timesteps element-by-element under a common seed — so the frozen-anchor result at L452–456 is untouched, and Fig. 4 remains valid as the uniform-versus-trajectory comparison it actually measures. We correct the label and add 2t2t.

The three laws, by the fraction of loss evaluations in t<0.43t<0.43, the band the trajectory law never visits. The official law is a linear weight over indices of the 1000-step grid after SD3.5's shift=3, which pushes the induced noise levels sharply up:

Timestep lawP(t<0.43)P(t<0.43)median tt
RAM official code, p(t)=2tp(t)=2t, shift=33.9%0.879
RAM paper, p(t)=2tp(t)=2t18.4%0.707
U[0,1]\mathcal{U}[0,1] (our Fig. 4)42.9%0.500
Trajectory law (our default)0%0.818

Why this strengthens L467–468. The submission argued from a two-point contrast, 0% against 42.9%. With 2t2t it becomes a graded one — 0%, 3.9% and 18.4% stable, 42.9% collapsing.

Revision. L461 and the attribution at L460–468; Fig. 4 gains the 2t2t curve; Appendix fig. 8.


Q2. Advantage scaling and safeguards in FlowAWR

Neither safeguard exists in the version before our ICLR submission. Note that the cited FlowAWR is a strictly contemporary work under ICLR review.

We implemented faithfully arXiv:2606.30376v1 (2026-06-29), the only version available before ICLR submission deadline. The arXiv v1 sets the temperature to the reward std of the experience buffer and states that the exponential tilt is bounded "without requiring heuristic clipping operations", which is what we did exactly. "rescale the centered advantages by their within-group standard deviation to adapt the correction magnitude" appears only in arXiv:2606.30376v2 (2026-10-09), posted with the first public code.

One observation, stated factually: the per-group rescaling drops the guarantee A≥−1\mathcal{A}\ge-1, so the endpoint weight 1+A1+\mathcal{A} can go negative — the infeasible region of Eq. 20. Running FlowAWR's own stat_tracking.py on GenEval-like rewards (24 per group, three binary checks), 3–15% of informative samples have A<−1\mathcal{A}<-1 depending on pass rate, and the [−3,3][-3,3] clip is what bounds them. The v2 recipe is not the exact KL solution, rather one plus a heuristic rescaling and clip.

AuthorsOpenReview ↗

Response to Reviewer 3uSR (Part 2)

Q3. Clipping of DiffusionNFT

Eq. 11 should carry the clip and we are adding it, but the clip is not what our argument rests on: measured, it is active on at most 1.04% of samples, and removing it entirely changes nothing during training. Chasing it down produced a sharper result instead — what keeps released DiffusionNFT feasible is its temperature, which we can now identify exactly as γ=Cβσ\gamma=C\beta\sigma.

(1) What the implementation does. The advantage is standardized with the group mean and the global batch std (sample.global_std=True), clipped to [−C,C][-C,C] with C=5C=5, and mapped by r=clip(z/(2C)+1/2,0,1)r=\mathrm{clip}(z/(2C)+1/2,0,1). Eq. 11 omits the clip and L182 says group std; both are corrected.

(2) Measured rather than bounded. Appendix F argues the clip is inert via ∣z∣≤G−1<C|z|\le\sqrt{G-1}<C — a bound that holds for the group std, not the global std — so we replaced the inference with measurement. Resuming the submission's DiffusionNFT run from five checkpoints (steps 30/120/300/510/630), the clipped fraction is 0, 0.26%, 0.17%, 1.04%, 0.78%, and only the lower bound activates; trained from scratch with both clips removed, the run peaks at 0.948 (ep 270) against 0.938 for the original over the same window, with no instability. Appendix F's conclusion survives, its route does not.

(3) What does carry the weight. Near f=foldf=f_{\rm old} the DiffusionNFT loss and our dense χ2\chi^2 objective are the same objective at γ=Cβσ\gamma=C\beta\sigma — 5σ5\sigma for GenEval's β=1\beta=1 — and feasibility follows from that temperature rather than from the clip: 0.13% of samples get a negative weight at 5σ5\sigma against 5–9% at γ=σ\gamma=\sigma, where the dense weight collapses by ep 150. The full temperature sweep is in our reply to WBHo's W1.

Revision. Eq. 11 includes the clip and the global std; L182 is corrected; Appendix F's inference is replaced by the two qualifications above and by measurement, and the identification of released DiffusionNFT with the dense χ2\chi^2 weight at γ=Cβσ\gamma=C\beta\sigma is now stated as Proposition 5 in Section 3.3.


W1. The presentation

We have revised accordingly. The introduction gains a numbered contribution list, and a summary table 1 places each method on every design axis.


W2. The KL/exponential part overlaps with FlowAWR

The KL corner is FlowAWR and we say so at L247–248, it is a special case we recover, not a contribution we claim. As in reviewerr 748B's strengths: "The unified perspective is interesting and potentially valuable. The paper connects several diffusion RL methods through ff-divergence regularization and derives sparsemax from Pearson-χ2\chi^2. I particularly like that the analysis leads to a concrete algorithmic improvement rather than simply reinterpreting existing methods". Our contribution is the generator-agnostic machinery around diffusion RL optimization, and none of the following is visible from the KL case alone:

  1. Staring from an exact identification of DiffusionNFT, with a testable prediction. Its linear weight is the interior KKT branch of the χ2\chi^2 solution, valid only while all weights are non-negative (Eq. 20). We place the released DiffusonNFT at γ=5σ\gamma=5\sigma. We then test the prediction: the same weight at γ=σ\gamma=\sigma violates feasibility on 5–9% of samples and collapses.
  2. The exact solution and its consequences. The χ2\chi^2 solution is a sparsemax projection onto the simplex, feasible at any temperature, whose advantage dispersion decays as the reward saturates. Hence only the projected tilt is stable at the natural temperature under a constant baseline.
  3. A general tilt-to-velocity map. For an arbitrary endpoint tilt ww, only the normalized weight κ=w/z\kappa=w/z enters the velocity target, with z(xt)=E[w∣xt]z(x_t)=\mathbb{E}[w\mid x_t] state-dependent. It is the form for any convex generator including FlowAWR as the exponential case, and we also analyze replacing z(xt)z(x_t) by the endpoint normalizer ZZ (Appendix D).

For clarity, we also add a summary table 1, where DiffusionRFT requires the least heuristic design choices and approximation.


W3. Qualitative results

Added. With identical initial noise (per-prompt seed 0) and sampling settings (40-step Euler, CFG 1.0, as at evaluation), we compare checkpoints near each method's peak. The visible differences are in attribute binding: for "a white handbag and a red giraffe" DiffusionNFT renders the giraffe in its natural color, for "a brown hot dog and a purple pizza" FlowAWR renders an ordinary pizza, while DiffusionRFT binds correctly in these examples. Spatial relations show the same pattern: for "a wine glass above a kite" both baselines place the kite above the glass and score 0.750, while DiffusionRFT gets the ordering right.

Revision. A new appendix with qualitative comparisons.

Reviewer_3uSROpenReview ↗

Response to authors

Thank you for the detailed rebuttal. You clarified most of my questions.

Regarding the qualitative results - I find it difficult to judge the quality of the generated images against FlowAWR except the chosen examples above. Also, in most other cases FlowAWR seems more aligned with the prompt that DiffusionRFT and with a better visual quality.

Have the authors observed any reward hacking or training instabilities that could explain these results? What justifies using DiffusionRFT over the baselines?

Did the authors try regular visual alignment metrics like HPSv2/3 or PickScore and can tell whether they have shown an improved visual quality over the baselines?

Reviewer_WBHoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper unifies regression-based diffusion RL through divergence-regularized reward maximization. KL regularization yields exponential weighting, while Pearson chi-squared regularization yields sparsemax, with DiffusionNFT’s dense linear weighting as its interior solution. The resulting DiffusionRFT combines sparsemax with empirically selected training choices, evaluated using SD3.5-Medium on GenEval and PickScore.

Critical strengths
  1. The divergence-based formulation usefully connects existing methods. The feasibility condition in Eq. 20 and the posterior-normalizer analysis in Appendix D clarify both their relationships and their approximations.
  2. The tilt-by-baseline grid and ablations of prediction space, anchors, and timestep sampling provide informative evidence beyond final performance, making the design-space analysis a valuable contribution.
Critical weaknesses
  1. The contribution of sparsemax projection relative to temperature selection is not sufficiently isolated. According to Eq. 20 and Appendix F, the original DiffusionNFT configuration (G=24, βC=5) already satisfies the positivity constraint, where dense linear weighting coincides with sparsemax. In contrast, the reported 86.8% constraint violation occurs at the more aggressive temperature γ=σ. Therefore, it remains unclear whether the improvements primarily arise from the projection itself or from enabling more aggressive reward optimization. A temperature sensitivity study under matched training configurations would better establish the practical contribution of sparsemax.
  2. The stability claims lack sufficient statistical validation. All experiments use a single random seed (Appendix G), despite training stability and collapse prevention being central contributions. Appendix H.3 further reveals substantial sensitivity to extremely small baseline perturbations, making single-seed conclusions particularly uncertain. Multi-seed evaluations of representative configurations would be important to substantiate the claimed robustness.
Questions for authors
  1. Since the original DiffusionNFT configuration already satisfies the positivity constraint, how does sparsemax compare with dense linear weighting across different temperatures under otherwise identical training settings? Do the improvements persist when both methods use their best stable temperatures?
  2. Can the authors provide multi-seed results for the main configurations in Figure 1, including final reward variance and collapse frequency, to substantiate the stability improvements?
  3. Appendix H.3 shows that the variance reduction from the proposed baseline is extremely small, yet its stabilization effect is substantial. Could the authors further clarify whether this benefit arises from finite-batch bias or optimization sensitivity rather than variance reduction itself?
2 comments
AuthorsOpenReview ↗

Response to Reviewer WBHo (Part 1)

We are grateful for your thoughtful and constructive feedback, and for the time and care you invested in reviewing our work. Your comments have helped us sharpen our claims and strengthen the supporting evidence.


W1 / Q1. The effect of temperature tuning

Temperature sweep (everything but γ\gamma fixed: GenEval, constant baseline, single seed, same code):

TiltNegative-weight samplesOutcome
dense @ γ=σ\gamma=\sigma5–9%peak 0.904 @ep 90 → 0.718 @ep 150
dense @ γ=2σ\gamma=2\sigma2.3%peak 0.912 @ep 130 → 0.808 @ep 200
dense @ γ=5σ\gamma=5\sigma0.13%stable
sparsemax @ γ=σ\gamma=\sigma0stable

The “Negative-weight samples” column is per-sample, whereas the 86.886.8% quoted in the original paper is per-step — the fraction of steps on which Eq. 20 fails for at least one of the G=24G=24 samples in a group — so a 5–9% per-sample rate puts almost every step in that set. All the above runs are included in Fig. 1 now.

Your premise is confirmed: at 5σ5\sigma infeasibility is essentially never active and the dense tilt is stable, and collapse is postponed monotonically with temperature, the GenEval threshold lying between 2σ2\sigma and 5σ5\sigma.

At each tilt's best considered stable temperature, sparsemax is better — as Fig. 1 shows, sparsemax @ σ\sigma against dense @ 5σ5\sigma for the DiffusionNFT.

What the projection guarantees. (i) It is the exact solution of the χ2\chi^2-constrained problem, so its weights are admissible at any temperature, and it reduces to the dense tilt wherever the truncation is inactive, it therefore cannot be worse than a tuned dense tilt. (ii) It needs no tuning: we ran γ=σ\gamma=\sigma on every task, whereas all three baselines set a temperature per task. Other works resort to tuning: DiffusionNFT 5σ5\sigma on GenEval and 0.5σ0.5\sigma on OCR, RAM a reward multiplier of 100 and 1000.


W2 / Q2. The stability with statistical validation

According to your suggestions, we have run a second seed on the pairing that tests the stability claim directly — dense + constant bb versus sparsemax + constant bb, both at γ=σ\gamma=\sigma, same temperature and baseline, the tilt as the only variable — plus the dense + b∗b^* cell you single out in Q3.

Measured as the largest drawdown below a run's own peak, the criterion test shows below:

Configurationseed 42seed 1 (to ep 200)collapsed
dense + constant bb @ σ\sigma0.1860.3222 / 2
sparsemax + constant bb @ σ\sigma0.0010.0050 / 2
dense + b∗b^*0.0110.0110 / 2

Both dense + constant bb runs peak at exactly ep 90 and fall from ep 110. The seeds differ only in how deep the collapse goes, not in whether or when it happens. Neither sparsemax run nor either dense + b∗b^* run exceeds a drawdown of 0.011, more than an order of magnitude below the collapse. So the collapse is not one unlucky seed, and the stability of the projected tilt is not one lucky seed. Limited by afforable computing resource, we might keep adding seeds and report the frequency as a count.

The same three rows also calibrate how large a difference has to be before it means anything. Taking the mean paired per-epoch difference between the two seeds of each row: 0.011 for dense + b∗b^* and 0.026 for sparsemax + constant bb, against 0.297 for dense + constant bb — and that last spread is entirely collapse depth, since both of its seeds turn at ep 90. We therefore treat a difference of order 0.01 in a terminal or peak number as run-to-run variation and do not argue from it.

AuthorsOpenReview ↗

Response to Reviewer WBHo (Part 2)

Q3. Relation between stablization and variance reduction

b∗b^* does one thing, and whether that thing matters depends on the preset. On a preset that is feasible, it is the control variate Eq. 23 designs it to be: it reduces variance and buys a modest, consistent speed-up. On the dense tilt at γ=σ\gamma=\sigma, a preset that violates Eq. 20 and that our paper argues against using, the same shift appears to decide between collapse and no collapse, which no variance argument of that size can explain. There the outcome is governed by optimization sensitivity, and we do not claim b∗b^* repairs that preset. A control variate cannot make an infeasible tilt feasible, Eq. 20 is violated for the entire run: not collapsing is not the same as being feasible. As written in L474, which your question is really about: the two mechanisms are complementary rather than interchangeable — the projection decides whether a tilt is admissible, b∗b* advances one that already is. Switching b=1→b∗b=1\to b* with everything else fixed:

CellEffect of b=1→b∗b=1\to b^*Same order as the variance account?
sparsemax @ σ\sigma (feasible, our proposal)+0.012+0.012 @ep 470, median +0.020+0.020yes
exp tilt (feasible)small, positiveyes
dense @ σ\sigma (infeasible)0.718→0.9640.718\to0.964, i.e. +0.246+0.246no — 20× too large

The variance criterion removes ≈10−4\approx10^{-4} of the second moment, a reward contribution on the order of +0.01+0.01, which is the size of the effect on the feasible presets. What Eq. 23 does not cover is the dense cell, as Appendix H itself notes.

Plus, the excess is not finite-batch bias. b∗b* is estimated on the micro-batch it reweights, so it does carry an O(1/B)O(1/B) self-normalization bias, and at our micro-batch size that term is not negligible — which is why the question is well posed. But replacing b∗b* by the constant equal to its logged median, −1.5×10−3-1.5\times10^{-3}, recovers 90% of the gap (0.941, against 0.964 with b∗b* and 0.718 without). A fixed constant is not estimated from the batch and has no such bias, so whatever does the work survives its removal.

Revision. (i) After Eq. 23 we state the scope: the variance criterion accounts for what b∗b* does on a tilt that is already stable, not for the dense cell at γ=σ\gamma=\sigma. (ii) Stabilizes becomes does not collapse where it refers to the dense cell (L419–420, Fig. 1 caption, L483), with the feasibility diagnostic stated alongside it. (iii) appendix H.3 adds the direct measurement of the factorization error behind the 10−410^{-4} figure (median 2.8% over 50 steps) and the multi-seed results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.