- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents an interesting unified perspective on regression-based diffusion RL, with meaningful theoretical insights and well-motivated experimental analyses. The connection between different reward weighting schemes and the proposed sparsemax-based improvement is particularly valuable. However, the novelty beyond existing policy optimization frameworks could be better clarified. Given that theoretical unification is the main contribution, the paper would benefit from more rigorous mathematical presentation, including explicit assumptions, lemmas, and theorems, as well as a summary diagram or table. Moreover, the experimental comparisons need stronger controls, and broader evaluations across different backbones and samplers would make the conclusions more convincing.
Critical strengths
- The unified perspective is interesting and potentially valuable. The paper connects several diffusion RL methods through -divergence regularization and derives sparsemax from Pearson-. I particularly like that the analysis leads to a concrete algorithmic improvement rather than simply reinterpreting existing methods.
- The experiments are closely connected to the theoretical insights. For example, the authors measure when the dense-linear approximation becomes invalid and relate this to training collapse. This helps explain why the proposed method works, rather than just reporting better rewards.
Critical weaknesses
-
The novelty of the proposed framework could be better justified. Reward-weighted regression, advantage-weighted policy optimization, and sparsemax have been studied in prior work, such as Reward-Weighted Regression, Advantage-Weighted Regression , and From Softmax to Sparsemax. The paper would benefit from a clearer discussion or a schematic figure of what is fundamentally new beyond combining these existing ideas.
-
The unification claim seems somewhat overstated. While the methods share a similar idealized policy improvement objective, their practical implementations differ in important ways. The paper does not fully establish that these algorithms can be unified solely through different reward weighting functions.
-
The experimental comparisons may not fully isolate the source of improvement. For example, sparsemax may improve stability partly because it reduces the effective update size, rather than because of the proposed theoretical motivation.
-
The experiments are mostly limited to SD3.5-Medium. It remains unclear whether the proposed design principles generalize to different backbones, samplers, or reward models.
-
The main contribution of this work appears to be its theoretical unification of existing methods. However, the mathematical presentation lacks sufficient rigor and structure, with few clearly stated assumptions, lemmas, or theorems.
Questions for authors
-
What is the main conceptual novelty compared with existing reward-weighted regression and sparsemax-based methods? Could the authors clarify which insights are genuinely new?
-
How much of the improvement from sparsemax comes from the projection itself, rather than simply reducing the effective update size? Could the authors compare against clipping baselines with matched gradient norms or update magnitudes?
-
How fair are the comparisons with existing methods when they are reimplemented within the proposed unified framework? Could the authors include comparisons against the original implementations with their recommended settings?
-
Do the main findings hold on another diffusion backbone or with a different sampler? In particular, are the advantages of sparsemax and rolling anchors consistent across different training settings?
8 comments
Response to Reviewer 748B (Part 2)
W4 / Q4. Other backbones, samplers and rewards
Samplers. For each method's checkpoint near its evaluation peak, we re-evaluated on the full GenEval test set with a first-order Euler sampler, a second-order Heun (ODE) sampler, and the SDE sampler used during training (40 steps, CFG 1.0, fixed seed):
| GenEval | Euler | Heun | SDE |
|---|---|---|---|
| DiffusionNFT | 0.942 | 0.943 | 0.941 |
| FlowAWR | 0.938 | 0.937 | 0.936 |
| DiffusionRFT | 0.976 | 0.980 | 0.977 |
Scores move by at most ±0.004 and the ordering is unchanged, so the gains do not depend on the sampler.
Reward models. Appendix G already reports PickScore. The conclusion is consistent with the main paper.
Backbone. We chose SD3.5-Medium because all the methods we compare against use it as their primary platform, which lets us run their official code under their recommended configurations — and is what makes the curve-level comparison possible. In addition, we do not afford a second backbone exploration within the discussion period. We list it as a limitation rather than report a rushed result. The theoretical results use only the structure of the flow-matching target and the advantage distribution, which is expected to be backbone-agnostic.
W5. Rigor mathematical presentation
We thankfully agree and we have restructured Sections 2–3 into explicit assumptions and propositions, stated in the order the derivation needs them, with proofs retained in Appendices B–F:
- Assumption 1 (standing). A1: the experience buffer is frozen within one update, so and do not depend on . A2: the noising kernel is Gaussian, , and the anchor is the corresponding marginal velocity . A3: the expectations used are integrable and a.s.
- Proposition 1 (tilt to velocity target), Eq. 9: for any endpoint reweighting , the velocity target depends on the tilt only through . Proof in Appendix B.
- Proposition 2 ( substitution), Eq. 14: the substitution rescales the displacement by without changing its direction, with and decreasing towards the noisy end. Proof in Appendix D.
- Lemma 1 (single-sample unbiasedness), Eq. 15: the per-sample objective differs from the exact-target one by a constant independent of , namely the conditional variance of the drawn target, so gradients and minimizer are unchanged. Proof in Appendix E.
- Proposition 3 (optimal tilt under a divergence constraint), Eqs. 17–18: the maximizer exists, is unique, and is fixed by — a KL constraint gives the exponential tilt, a Pearson- constraint gives the water-filling solution whose group form is sparsemax (Eq. 21). Proof in Appendix F.
- Proposition 4 (feasibility of the interior branch), Eq. 20: the dense linear weight solves the problem if and only if holds for every sample.
- Proposition 5 (identification of DiffusionNFT): whenever the clip in Eq. 11 is inactive, the official DiffusionNFT weight coincides sample by sample with the interior branch at and ; since a group-standardized reward obeys , Eq. 20 holds for the whole linear family exactly when . This is what makes the heuristic testable: at its released the criterion holds on every logged step, and the same weight at violates it on 86.8% of them and collapses.
The Remark closing Section 3 now reads off these propositions rather than restating them.
Q3. Is the comparison fair?
Yes, our principle: where official code exists, use it under its recommended settings; where it does not, implement the paper faithfully.
- DiffusionNFT. Our codebase is built on the official repository; the dashed curve in Fig. 1 is the official loss under the official GenEval configuration, as are the PickScore experiments (W4). We can now also state the relation precisely: the official loss is second-order equivalent to our dense weight at (3uSR Q3), and training that configuration from scratch in our code reproduces the official curve.
- RAM. Fig. 3a includes the official code under its recommended settings alongside an independent reproduction; the two agree.
- FlowAWR. No code was public at submission time, so we implemented its tilt and adaptive temperature from arXiv v1. To isolate the tilt, that cell uses the unified loss geometry; the paper's -space loss corresponds to the -prediction results in Fig. 2. Checking the later-released code item by item, its loss geometry, anchor and timestep law agree with ours.
Follow-up Before the Discussion Period Ends
Dear Reviewer,
Thank you again for your careful review and thoughtful feedback. We have sincerely tried to address your concerns through our responses, additional experiments, and draft revisions in blue. As the discussion period draws to a close, if time permits, we would greatly appreciate learning which concerns remain unresolved or where our explanations are still insufficient. Your perspective would help us identify what we may have missed, and we would welcome the opportunity of further clarification.
Thank you again for your time and consideration.
Best regards,
The Authors
Response to Reviewer 748B (Part 1)
We sincerely thank the reviwer for your positive feedback. We have added further clarification regarding the issues mentioned in your comments.
W1 / Q1. Comparison to prior work
RWR, AWR and sparsemax are related work that our draft cites, but not in the same domain of diffusion generative models.
W2. The unification and differences
The paper separates exactly the two levels the reviewer distinguishes: Sections 2–3 unify the form (at the level of the population regression target, all three methods solve one reward-maximization problem under an f-divergence constraint, differing only in the convex generator), while Section 4 varies the implementation axes the target leaves free — loss space and normalization, anchor, timestep law, baseline — one at a time.
We do not claim the algorithms differ only by a weighting function at the level of implementation. We add this table in the draft, placing the unified target beside the axes Section 4 tests:
One row deserves comment. The endpoint normalizer is a group average the other two tilts must estimate; for the projected tilt the constraint that defines it already enforces Epold[ρ]=1, so Z=1 exactly. That is a property of the construction, not of tuning — and we do not overstate it: the posterior normalizer z(xt) is set to one in every member of the family, ours included, so what the projection removes is the endpoint-level approximation only.
Revision. The abstract is rewritten; Section 2 gains the table.
W3 / Q2. Is sparsemax simply taking smaller effective steps?
No. Two families of runs hold update magnitude fixed (GenEval, constant baseline, single seed) — dense weights across temperatures, and clipped baselines that freeze the advantage scale:
The decisive cell is dense @ 2σ: over ep 70–130, before it collapses, its E∣A∣ averages 0.165 — below sparsemax's 0.197 over the same window — and it collapses while sparsemax does not. The clipped baselines freeze the advantage scale and keep every weight non-negative, yet the lower clip collapses earliest of all; the projection differs in re-solving the threshold after clipping, so the advantages still sum to zero. At matched gradient norms the picture is unchanged: clipping never triggers for dense @ 5σ, the lower clip or the clip-free DiffusionNFT variant (max 0.87, 0.59, 0.94), so those three are already compared under one cap, while dense @ 2σ triggers it on 17% of steps before collapsing and is not saved.