acceptodds
Under review as a conference paper at ICLR 2027

DIFFUSIONRVGE: REINFORCEMENT LEARNING VIA REGRESSION-BASED VALUE GRADIENT ESTIMATION

Abstract

Reinforcement learning improves diffusion models using rewards on generated images, but repeated denoising and gradient estimation make training expensive. We introduce DiffusionRVGE, an efficient method that estimates useful update directions by regressing centered rewards on centered final denoised images. It requires neither reward derivatives nor adjoint backpropagation through the sampling trajectory. A unified first-variation formulation gives a gradient-matching update with or without a trajectory-KL penalty, and clean-image regression supplies a preconditioned approximation to its value-gradient target. A supporting analysis explains the directional advantage of clean-image over noisy-state regression. The implementation adapts rollout bandwidth and query weights to reward continuity: local rollouts suit smooth rewards, whereas broader groups provide informative contrasts for discrete rewards. Trained jointly on five rewards, one SD3.5-Medium model matches or exceeds DiffusionNFT's joint model on seven of eight metrics, reaching GenEval 0.94, OCR 0.92, and PickScore 23.82 at about one fifth of its estimated training compute. In single-reward training, Diffusion-RVGE reaches matched reward thresholds 2.1-6.7x faster than reproduced DiffusionNFT on identical hardware. Ablations support the clean-image estimator, direct direction transfer, and reward-dependent rollout choices. Our code is provided in the supplementary material.

Primary area:
reinforcement learning
Keywords:
diffusion models, reinforcement learning, value gradient estimation, flow matching, reward optimization, clean-image regression, derivative-free optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_suVSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a new reward fine-tuning method for diffusion models. Their method is based on a regression loss for a direction that aims to approximate the gradient of the value function. They mathematically derive a direction that is positively aligned with the gradient of the value function through a minimization problem that does not require reward gradients or backward integration. They experiment with fine-tuning Stable Diffusion 3.5 on various sets of rewards, both differentiable and nondifferentiable, individually and jointly, using different estimators with a similar underlying approach and adapting their method accordingly. They show better performance and efficiency than DiffusionNFT and FlowGRPO, popular fine-tuning methods, and ablate some of the design choices made for each setup.

Critical strengths

The proposed theoretical regression target is novel. The justification for its positive alignment with the gradient of the actual expected reward is neat: even though it relies on an assumption, the alignment holds when the corruption process is Gaussian, as at initialization or under an exponential tilt. It is a nice, elegant way to obtain an approximate reward gradient through a minimization problem without relying on reward gradients or adjoint methods.

Critical weaknesses
  1. The paper lacks clarity, particularly in its presentation of the background and its contextualization of the proposed method relative to other methods, e.g. the lack of a technical comparison with FDFO or heavy notation used to introduce the value function. It introduces theorems and fragments of theoretical analysis that do not seem connected to or necessary for the narrative. There is no clear explanation of the relationship between the theory and what is actually done in practice, e.g. Proposition 2 assumes posterior samples at a fixed noisy state, which neither implemented estimator provides. Several arbitrary practical choices are introduced without enough motivation in the main paper, although they are evaluated empirically through ablations. The paper also uses nonstandard terminology, “rollout bandwidth,” “trajectory bank,” “query weights”, without sufficiently explaining it, making the method unnecessarily difficult to follow.
  2. The experiments use DiffusionNFT as the main baseline but do not include subsequent methods that improve upon it, such as Reinforce Adjoint Matching [1]. The differences between the theoretical formulation and the practical implementation make it difficult to determine whether the efficiency gains come from a better estimator of the value-function gradient or from other design choices orthogonal to it, in particular setting the reference penalty coefficient to zero and reward dependent trajectory weights.

[1] https://arxiv.org/abs/2605.10759

Questions for authors
  1. Could you add newer baselines that improve upon DiffusionNFT, e.g., RAM and AdvantageFlow [2] ?

  2. While Section 6.4 ablates trajectory weighting within the proposed method, does its efficiency advantage over the baselines persist under matched trajectory weighting and reference regularization?

  3. Can you clarify the importance of the directional analysis and noisy state regression discussions, as well as their corresponding ablations?

  4. Can you clarify the missing conditioning on the state in the practical estimation of the value-function gradient?

  5. Can you add a theoretical comparison with FDFO, of which the proposed method appears to be a generalization to multiple samples?

[2] https://arxiv.org/abs/2605.26013

3 comments

Thank you for the detailed review and the concrete suggestions.

W1. Clarity, organization, and terminology.

Thank you for these suggestions. We have reorganized and simplified Section 3 to clarify the progression from deriving the policy gradient to estimating ∇Vt\nabla V_t for the update. The presentation now follows three steps:

  1. Policy gradient (Theorem 1, Sec. 3.1): give a unified first-variation derivation for all τ≥0\tau\ge0, retaining the path-tilting derivation only in Appendix A.2 to provide intuition.
  2. Value-gradient estimation (Sec. 3.2): estimate ∇Vt\nabla V_t by clean-image regression. Proposition 2 extends the population regression analysis from the reference policy to old policies obtained by terminal reweighting of the reference path law.
  3. Reward-adaptive rollout (Sec. 4): choose local rollouts from the main branch or independent rollouts according to reward variation, within the same regression framework.

New Figure 1 illustrates one update from sampling through regression to the policy loss, with Algorithm 1 providing the full procedure.

We have also clarified the value-function notation and the relationship between the population assumptions and finite-rollout implementation. Detailed proofs and implementation settings are collected in the appendices. Table 5 maps each theoretical result to the design choice it supports, its scope and its test. The remaining practical choices are motivated where they are introduced and tested in ablations: the rollout type by reward continuity (Sec. 4; Fig. 5), the update weights by GenEval's saturation (Sec. 4; Table 4), and ridge with row normalization by coefficient stability (App. B.5); all other hyperparameters are shared across rewards (Table 11). We also replaced the nonstandard terms: "trajectory bank" → local rollouts from the main branch; "independent-seed group" → independent rollouts; "rollout bandwidth" is replaced by the choice between the two, stated in words; "query weights" → update weights aia_i. Section 4 follows sampling, joint regression, and policy update without additional names for the sample sets.

W2 / Q1. Additional baselines: RAM and AdvantageFlow.

RAM. We ran its official code on all three rewards using the same 8×8\timesA800 hardware and evaluation sets. Ours reaches OCR and GenEval 0.90 in 3.2 and 22.8 GPU-hours, versus RAM's 20.7 and 59.8 (Fig. 2; Tables 3 and 9). Ours also matches or exceeds RAM's published final scores with half or less of the compute (Table 9):

RewardRAM, publishedOurs (Table 2)RAM reproduction, matched GPU-h
PickScore23.95 (≈222 GPU-h)23.99 (110)23.69 (111)
OCR0.97 (51)0.98 (19.5)0.91 (21.4)
GenEval0.97 (≈259)0.98 (129)0.93 (134.5)

Entries show reward (GPU-hours). Published scores retain RAM's protocol; their schedule costs are measured or estimated on our 8×8\timesA800 hardware (≈\approx: estimates from the reported 300/270 steps). Our PickScore and GenEval reproductions cover about half the published schedules and are read at compute close to ours.

AdvantageFlow. Since its code is not publicly available, we evaluate our checkpoints using its evaluation protocol and compare with the curves reported in its paper. Ours reaches PickScore/26 0.89 and OCR 0.90 in 34.8 and 3.2 GPU-hours, versus 47.7 and 20.1 for AdvantageFlow. This comparison uses our A800 runs and its published A100 results (Table 10; App. D.7).

Q2. Trajectory weighting and reference regularization.

PickScore and OCR use uniform update weights (ai=1a_i=1), without lowest-reward selection; their performance advantages are reported in W2. GenEval's discrete scores often saturate, motivating updates on the lowest-six trajectories while retaining the other rollouts of the same prompt for regression. With uniform updates, ours reaches GenEval 0.80/0.90 in 17.2/58.7 GPU-hours, versus official DiffusionNFT's 30.3/102.0. With uniform weights and no reference penalty for either method, reaching 0.80 takes 17.2 versus 35.7 GPU-hours for KL-free DiffusionNFT (Table 4). Under these matched settings, ours reaches OCR 0.90 in 3.2 versus 16.9 GPU-hours for DiffusionNFT (App. D.5).

Q3. Importance of directional analysis and noisy-state regression ablation.

Thank you for this question. Under a local linear approximation, the change in expected rollout reward is Vt(x+δ)−Vt(x)≈⟨δ,∇Vt⟩V_t(x+\delta)-V_t(x)\approx\langle\delta,\nabla V_t\rangle. The unregularized population coefficient of noisy-state regression therefore approaches ∇Vt\nabla V_t as the perturbations shrink, without the clean-image covariance factor Σt†\Sigma_t^\dagger (App. A.1). Our clean-image regression yields the preconditioned direction gt=Σt†∇Vtg_t=\Sigma_t^\dagger\nabla V_t (Proposition 2). Its motivation is the geometry of clean-image variation: the Gaussian subspace example shows how low-dimensional coordinates improve directional estimation with few reward evaluations (Eq. 7), and the ablation tests this using identical perturbations and reward differences (Fig. 3). We have clarified this reasoning in Sections 3.2 and 6.1.

Q4. Conditioning on xtx_t in practical estimation.

Thank you for highlighting this point. Eq. (6) conditions on a fixed state xtx_t. In principle, its conditional moments can be estimated by Monte Carlo from several independent stochastic completions of the same xtx_t, but doing this at every training state multiplies the rollout cost, a key practical constraint in diffusion RL. Our reward-adaptive rollout avoids this cost in two ways, which we have clarified in Section 4 (theory in App. B.3):

  • Independent rollouts with the same prompt (discrete rewards, GenEval). At the noise end the conditioning is exact: since α1=0\alpha_1=0, X1X_1 carries no information about the clean image, so the completions of a prompt from independent initial noises are posterior samples at t=1t=1, and their fit is the slope of Eq. (6) at t=1t=1 for any output law (Prop. B.3(a)). The approximation is the reuse of this slope at later states. For a Gaussian output law, the reused slope is the average of the state-conditional slopes and still has positive inner product with the state-averaged value gradient (Prop. B.3(b)); its per-state error is zero at t=1t=1 and grows toward the data end.
  • Local rollouts from the main branch (continuous rewards, PickScore and OCR). Each training state is fitted on the ODE completions of the other states on its own stochastic trajectory, centered at its own completion. These are not posterior samples: the fit estimates the reward slope at the state's completion, with the perturbation spread in place of the posterior spread (App. B.3).

Section 6.3 (Fig. 5) supports this choice. On GenEval, whose scores often remain constant under local perturbations, local fits reach 0.84–0.85 at 28 GPU-hours versus 0.92 for independent rollouts; on PickScore, local rollouts lead independent ones by 0.4–0.7 at matched budgets.

Q5. Theoretical comparison with FDFO.

Thank you for this suggestion. We have expanded the theoretical comparison in Section 2. FDFO uses paired denoised-image differences, while our method jointly regresses reward differences on multiple image differences, accounting for correlations among them. With two nearby trajectories, our row-normalized ridge fit on the single image difference is a scalar multiple of FDFO's secant direction; jointly regressing over the independent rollouts of a prompt, together with update weights, extends the estimator to discrete rewards such as GenEval, which the FDFO paper does not report. Proposition 2 characterizes the population target of this joint fit: under its Gaussian-noising assumptions, gt=Σt†∇Vtg_t=\Sigma_t^\dagger\nabla V_t is a covariance-preconditioned value-gradient direction.

Our existing PickScore comparison shows ours reaching 23.20 in 14.3 GPU-hours, versus 109.3 for FDFO (Fig. 2; original Fig. 1).

Thank you again for the thorough review. We hope these changes address your concerns, and we are happy to provide more detail.

Reviewer_5F9WOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper seeks to reduce the training cost of reinforcement learning (RL) with diffusion models, where a learned control term is added to the drift of the sampling SDE. To avoid reward gradients or differentiating back through the diffusion model's sampling trajectory, the paper proposes a method of estimating effective gradient update directions through clean-image linear regression: predicting the deviations from the reward mean using the deviations of denoised samples from the Tweedie's estimate. The paper shows that, at the reference policy (the control term is zero, as it may be when it is initialized), the slope vector from the regression is a preconditioned direction for a gradient needed to compute the loss that optimizes the RL objective. The experiments show empirically how their preconditioned direction aligns with the true gradient, and show that the method leads to competitive or improved metrics in significantly fewer GPU-hours of training.

Critical strengths
  1. The proposed method avoids reward gradients and differentiating through the sample trajectories.
  2. The proposed method meets or exceeds the performance of other methods on multiple metrics with significantly fewer GPU-hours for training.
  3. The paper provides thorough ablations on training design decisions (in particular, for how the banks Bi\mathcal{B}_i are constructed).
Critical weaknesses
  1. It is not clear theoretically how the direction found by linear regression gtg_t relates to ∇Ftθold,τ\nabla F_t^{\theta_{\mathrm{old}},\tau} when the reference policy is not zero (uθold≠0u_{\theta_{\mathrm{old}}} \neq 0); however, Figure 3 is empirical evidence that the direction aligns well throughout training.
Questions for authors
  1. In Figure 2, why does the gradient alignment of the direction found by clean-image regression decrease as the query index qq increases?
1 comment

Thank you for the positive and careful review.

W1. How does the regression direction relate to the value gradient away from the reference policy?

Thank you for raising this point and for noting the empirical evidence. Proposition 2 now analyzes ∇Vt\nabla V_t under old policies with terminal tilting. This structure holds if the preceding iteration attains the exact positive-τ\tau trajectory-KL optimum (App. A.2). In practice, the old policy need not reach this optimum or preserve terminal tilting; Appendix B.2 characterizes the resulting population error. Off-policy sampling from the reference path law satisfies the Gaussian-noising structure, with the identity applying to VtrefV_t^{\mathrm{ref}}.

As you observed, Figure 4 (original Figure 3) supports the effectiveness of the directly transferred directions during training. Figure 3a (original Figure 2) additionally measures their positive alignment with the numerical ODE reward gradient after 200 updates.

Q1. Why does the clean-image alignment in Fig. 2 (now Fig. 3) decrease as qq increases?

Thank you for this question. Larger time indices correspond to smaller noise times tt, closer to the data end. Under our local approximation, the clean-image and noisy-state differences are related by the Jacobian of the remaining denoising map. As t→0t\to0, this map approaches the identity, so the two sets of image differences become similar and the finite-sample advantage of clean-image regression diminishes. Its alignment therefore approaches the noisy-state baseline.

Presentation. A new Figure 1 shows one update of the algorithm, Table 5 maps each theoretical result to its scope and test, and the nonstandard terms are renamed (general response).

Thank you again for the encouraging assessment.

Reviewer_JByPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers reward tilting of flow/diffusion models formulated as an exponential tilt of the base path measure. Noting that the value gradient is generally not differentiable and expensive to compute, the authors propose to use an anisotropic preconditioner for an approximation. Techniques to stabilize training via reward centering are also proposed. The paper validates the algorithm on various image experiments and show that it outperforms a reproduction of DiffusionNFT.

Critical strengths
  • The experiments show a direct compute efficiency gain over DNFT on matched hardware with helpful ablations provided in the appendix.
  • It's handy that the method works with non-differentiable rewards (though, below, I don't think there's good theoretical justification for the approach yet, so I think it's important we work toward what we mean more clearly by "the method").
Critical weaknesses

Importantly, I don't think the justifies that algorithm that was actually run in the experiments.

  • The first theorem defines the optimum only for τ>0\tau > 0, and on top of that the actual training loss given in eq 6 as I understand it comes for a totally separate argument about first variations. The experiments all use τ=0\tau = 0, so it's hard to understand the trajectory KL tilting argument here
  • Also, there's no actual preconditioner here. What is referred to as preconditioner is actually just removing a term from the chain rule in the gradient of the value function, as I understand it. E.g. if I have ∇xVt=κtct=(∇xmt) Σt†ct\nabla_x V_t = \kappa_t c_t = (\nabla_x m_t)\, \Sigma_t^\dagger c_t then using κtΣt†ct\kappa_t \Sigma_t^\dagger c_t instead just means not actually doing the chain rule properly? So the update done is not actually the value gradient, and it's quite hard to characterize because including the Jacobian makes the performance worse apparently.
  • Also, each reward setup has its own recipe. Are they fairly optimized against DNFT, or does this result in performance favorability to the proposed method?
Questions for authors
  • What do the theory sections justify at τ=0\tau = 0?
  • How does proposition 2 carry over to policy rollouts at $theta_old \neq ref$?
  • Why does the reproduction of DNFT not match their stated performance values? It looks pretty hobbled in comparison.
3 comments

Thank you for the precise questions. Section 3.1 now clarifies the roles of the path-tilting interpretation and the unified first-variation derivation.

Q1 / W1. What does the theory justify at τ=0\tau=0? Original Eq. (6) comes from a separate first-variation argument.

Our presentation did not clearly explain the roles of the two arguments. For τ>0\tau>0, path tilting gives an optimal-control interpretation of the loss as regression toward the optimal control drift. We introduced this perspective to motivate the loss and provide intuition.

The mathematical basis for the update is the unified first-variation derivation, already included in Appendix A.1 of the original submission and valid for every τ≥0\tau\ge0. In the revision, we promote this existing result to Theorem 1 (Sec. 3.1; proof in App. A.1) and place the positive-τ\tau path-tilting result in App. A.2. This makes the basis for the reward-only experiments explicit while retaining the optimal-control interpretation for positive regularization. At τ=0\tau=0, the gradient of our loss at θold\theta_{\rm old} is therefore exactly the policy gradient of the expected reward, with no optimality or θold=θref\theta_{\rm old}=\theta_{\rm ref} assumption, and the same proof covers the reward-only ODE (App. A.1).

Choice of τ\tau. The framework allows both τ=0\tau=0 and τ>0\tau>0, with τ\tau controlling the strength of KL regularization; the main experiments use τ=0\tau=0 to prioritize reward-optimization speed. Adding a per-step KL to the pretrained model to our OCR run (App. G.2) preserves the pretrained scenes, with text written on signs and surfaces (Fig. 11; PickScore 20.87 vs. 19.40 without the KL, pretrained 20.03), while still reaching OCR 0.97, 3.0/2.9/3.4× faster than DiffusionNFT to 0.80/0.90/0.95.

W2. Is omitting the Jacobian a preconditioner, and why does including it reduce performance?

We agree that omitting the Jacobian changes the gradient. The regression yields a covariance-preconditioned value-gradient direction, and we should have made this distinction clearer. With conditional clean-image covariance Σt\Sigma_t and posterior mean mtm_t, Proposition 2 and the regression in Eq. (6) give

Dmt=κtΣt,gt=Σt†∇xVt.Dm_t=\kappa_t\Sigma_t,\qquad g_t=\Sigma_t^\dagger\nabla_xV_t.

The regression normal equations implicitly apply this inverse-covariance action. Since Σt†\Sigma_t^\dagger is positive definite on the identifiable subspace Range⁡(Σt)\operatorname{Range}(\Sigma_t), which contains ∇xVt\nabla_xV_t, we have ⟨gt,∇xVt⟩>0\langle g_t,\nabla_xV_t\rangle>0 whenever ∇xVt≠0\nabla_xV_t\ne0. Thus the population direction is a preconditioned ascent direction in this model (Sec. 3.2; App. B.1), and Prop. B.1 bounds its angle with ∇xVt\nabla_xV_t through cond(Σt)\mathrm{cond}(\Sigma_t).

When clean-image variation concentrates in a low-dimensional subspace, the covariance and this Jacobian can be nearly rank deficient. Regression compensates their unequal scaling of gradient components, consistent with classical preconditioning (Boyd and Vandenberghe, 2004, Secs. 9.4.1 and 9.4.4).

Our probes support the reward-relevant subspace motivation: at the first sampled state, seven clean-image differences capture more of the unit ODE reward-gradient direction than seven noisy-state differences (projection norms 0.176 vs 0.0099 in a 65,536-dimensional latent; App. D.3).

We also test Jacobian transport directly using the numerical ODE completion, whose Jacobian differs from the posterior-mean Jacobian above. With the rollout recipe fixed and directions norm-matched per sample and timestep, each pullback update costs 2.1 times as much. After 69 updates (81 GPU-hours), pullback reaches PickScore 23.18, versus 23.48 for direct transfer at equal updates and 23.65 at equal compute (Sec. 6.2; Fig. 4; App. D.2). This supports using the regularized regression direction directly as an efficient optimization choice.

W3. Each reward has its own recipe. Is the comparison with DiffusionNFT fair?

RL reward functions differ substantially. They may be differentiable or accessible only through black-box evaluations, and their scores may vary smoothly or remain locally constant. Reward-adaptive rollout is one of our contributions: local rollouts from the main branch capture PickScore/OCR variation, while independent rollouts of the same prompt cross GenEval's reward plateaus. Both use the same clean-image regression. Figure 5 in the revision (Figure 4 in the original) supports local rollouts for PickScore, independent rollouts for GenEval, and either construction for OCR.

On GenEval, all three arms below use 24 independent rollouts per prompt. Table 4 reports GPU-hours to each test accuracy and best accuracy within 137 GPU-hours (one seed per arm):

MethodUpdate selectionGPU-h to 0.80GPU-h to 0.90Best
Ours, full recipeLowest 6 of 2411.522.80.97
Ours, uniformAll 2417.258.70.91
DiffusionNFT, officialAll 2430.3102.00.90

With uniform updates, ours remains 1.8× and 1.7× faster at the two thresholds, supporting the effectiveness of our preconditioned clean-image regression update under the same sampling format. Lowest-six update selection further improves efficiency and accuracy.

Core hyperparameters are shared across our reward settings (Table 11). DiffusionNFT uses its official reward-specific configurations. Comparisons use the same hardware and evaluation sets and measured GPU-hours, including sampling cost.

Q2. How does Proposition 2 carry over when θold≠θref\theta_{\rm old}\ne\theta_{\rm ref}?

Thank you for highlighting this point. We now state Proposition 2 directly for old policies that are terminal reweightings of the reference path law. Such policies change the clean-image prior while preserving the Gaussian noising likelihood. This structure holds if the preceding iteration attains the exact positive-τ\tau trajectory-KL optimum (App. A.2). In practice, the old policy need not reach this optimum or preserve terminal tilting; Appendix B.2 characterizes the resulting population error and gives a sufficient ascent condition for VtV_t. Off-policy sampling from the reference path law satisfies the Gaussian-noising structure, with the identity applying to VtrefV_t^{\mathrm{ref}}.

As complementary empirical evidence, Figure 3a (original Figure 2) shows positive alignment with the numerical ODE reward gradient after 200 updates, while Figure 4 (original Figure 3) shows reward improvement during training with direct transfer.

Q3. Why doesn't the DiffusionNFT reproduction match its published values?

Thank you for raising this concern. We ran DiffusionNFT's official code with its released reward-specific configurations on the same hardware and under compute budgets comparable to ours. Table 2 also reports DiffusionNFT's published results from longer training schedules: with 129, 19.5, and 110 GPU-hours, our single-reward checkpoints match DiffusionNFT's published GenEval, exceed its OCR, and are within 0.04 of its PickScore, and our five-reward model matches or exceeds DiffusionNFT's joint model on seven of eight metrics at about one fifth of its estimated compute (Table 1).

Our DiffusionNFT reproductions on PickScore, OCR, and GenEval cover 260, 50, and 210 iterations, compared with the published 2000, 150, and 1000. These shorter runs serve the learning-curve comparison at comparable GPU-hours.

We additionally repeated the PickScore reproduction with a different seed. At 110 GPU-hours, the two DiffusionNFT runs score 23.51 and 23.50, respectively (Table 8; App. D.6), supporting the consistency of the PickScore reproduction. The second run also reproduces the DiffusionNFT curve published by AdvantageFlow within 0.001 (PickScore/26) over 20–60 GPU-hours (App. D.6).

Thank you again; these points substantially improved the paper.

Comments on the paper

1 comment

We thank the reviewers for careful and expert reviews. They found the method "a nice, elegant way to obtain an approximate reward gradient through a minimization problem without relying on reward gradients or adjoint methods" (suVS), with "a direct compute efficiency gain over DNFT on matched hardware" (JByP) and thorough ablations (5F9W). We answer each reviewer's questions in their own thread; this note only lists what changed in the revised PDF (changes in blue; all numbers below refer to it).

  • Clearer organization of Section 3. We first derive the policy gradient (Theorem 1), then explain how to estimate ∇Vt\nabla V_t for the update. Section 3.1 now uses a unified first-variation proof for all τ≥0\tau\ge0; the path-tilting derivation is retained only in Appendix A.2 to provide intuition. Proposition 2 extends the regression analysis from the reference policy to old policies obtained by terminal reweighting of the reference path law.
  • New Figure 1 shows one update of the algorithm. The submission's Figures 1–4 are now Figures 2–5.
  • New theory in Appendix B, each result answering a specific question: Prop. B.1 (the chain-rule reading of Prop. 2, with an angle bound), B.2 (policies other than the reference), B.3 (independent rollouts of the same prompt and conditioning on the state). Table 5 maps every result to its scope and test.
  • New experiments: RAM on all three rewards (Fig. 2, Tables 3 and 9), AdvantageFlow on its own protocol (App. D.7), a matched-design comparison with DiffusionNFT (Table 4, Sec. 6.4), a second run of the official DiffusionNFT code (App. D.6), and a τ>0\tau>0 OCR run (App. G.2).
  • Terminology: "trajectory bank" → local rollouts from the main branch; "independent-seed group" → independent rollouts; "rollout bandwidth" is replaced by the choice between the two, stated in words; "query weights" → update weights.
  • Code and checkpoint: the supplementary material contains our code and the checkpoint of the five-reward model in Table 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.