acceptodds
Under review as a conference paper at ICLR 2027

ROOT: Resolving Coefficient Ambiguity in Endpoint Reward Tuning for Flow Models

Abstract

Forward-process reward tuning uses rewards on behavior-generated endpoints to define a reward-improved endpoint distribution, then re-noises these endpoints to learn its induced conditional flow. We identify coefficient ambiguity in stochastic velocity training, where positive weights defining the same endpoint distribution and induced flow can produce different updates. Condition-wise reward centering exposes this ambiguity by rescaling exponential weights without changing the target at a fixed tilt. We show that the target and behavior distributions uniquely determine a mean-one target-to-behavior density ratio. ROOT uses this probability-defined ratio as the coefficient for stochastic velocity learning, making its construction invariant to condition-wise rescaling of endpoint-equivalent weights. Its single-pass component, RootSolve, yields a behavior-centered finite-tilt objective whose conditional population optimum matches the target flow under behavior-flow consistency. Its multi-pass component, RootTrace, reuses scored rollouts through moving-reference updates that retain reward-dependent corrections on later passes while removing the pull toward the original behavior prediction. On Stable Diffusion 3.5 Medium, ROOT reaches matched DiffusionNFT training-reward thresholds with – and – fewer outer iterations across single- and multi-pass settings, respectively, and achieves comparable final performance under sequential multi-reward training with about % fewer iterations.

Primary area:
generative models
Keywords:
Reward Tuning, Flow Matching, Endpoint Density Ratios

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DLUVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ROOT for reward tuning of flow models. ROOT uses the target-to-behavior density ratio as the unique mean-one canonical coefficient to construct a calibrated velocity training field, RootSolve, and further proposes RootTrace. Overall, the paper has some good motivations, but there are also some flaws in the core method.

Critical strengths

The overall writing is good and the paper is fairly polished. In particular, the second contribution is relatively clear and well articulated.

Critical weaknesses
  1. Limited novelty and lack of attribution. The paper’s novelty is primarily demonstrated against ITM (Implicit Tilt Matching), which is a baseline constructed by the authors rather than an established method. The original DiffusionNFT does not suffer from the coefficient-ambiguity issue emphasized in the paper. Moreover, the paper does not provide an attribution analysis explaining why ROOT outperforms DiffusionNFT overall. Is the gain mainly due to the difference between the probability-ratio coefficient p and DiffusionNFT’s bounded optimality coefficient? If so, the novelty appears limited.

  2. RootTrace is well motivated (but the design is too simple), the Figure 4 only reports the K=2 setting. As the model is updated, the method reuses a fixed p without any corresponding trust-region constraint. The consistent degradation at K=4 clearly indicates an instability that is not analyzed.

  3. Some figures (e.g., Figures 4 and 5) cover only a limited number of training steps. For an RL method, it is important to demonstrate stable convergence over longer training horizons, rather than only short-term gains.

  4. As a new RL method, the paper compares against a limited set of baselines. It should include comparisons with RL/post-training methods such as FlowGRPO-Guard[1], TempFlowGRPO[2], FlowAWR[3], and related approaches[4].

[1] GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping

[2] When Timing Matters for GRPO in Flow Models

[3] FLOWAWR: ONLINE ADAPTIVE FLOW REINFORCEMENT VIA ADVANTAGE-WEIGHTED RECTIFICATION

[4] Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View

Questions for authors

Please see the weaknesses.

9 comments
AuthorsOpenReview ↗

Response 1/3 — Novelty and attribution (W1)

Novelty and attribution (W1)

We thank the reviewer for the careful reading, and we respectfully disagree with this assessment of novelty.

First, identifying coefficient ambiguity is precisely our central contribution. DiffusionNFT [6] avoids this ambiguity by directly specifying a bounded optimality coefficient through its positive–negative policy construction. However, it does not address which coefficient is determined by a prescribed reward-improved endpoint distribution. We identify this missing link. Endpoint-equivalent coefficients can produce different stochastic updates (Theorem 1), while the target-to-behavior density ratio ρ\rho provides the unique mean-one representative (Proposition 1). RootSolve then realizes this coefficient with a conditionally exact finite-tilt objective (Theorem 2). The difference in coefficients is therefore a consequence of deriving the optimization objective from the target distribution, rather than an empirical choice of weighting functions. This provides a principled solution to forward-process reward tuning without additional models or likelihood estimation.

Second, DiffusionNFT is our primary baseline, with direct comparisons in Figures 3–4 and Table 1. ITM is adapted from the implicit formulation in Tilt Matching [5] to our matched training protocol. We include it as a controlled comparison for coefficient attribution.

Third, besides the comparison of ROOT and DiffusionNFT, our controlled comparisons examine three distinct design choices. ROOT vs. ITM (Figure 5) isolates coefficient identification within the same stochastic field form, ROOT vs. UDR (Figure 5) isolates the stochastic realization while fixing ρ\rho, and ROOT vs. repeated RootSolve (Figure 4) isolates the benefit of moving-reference reuse. These comparisons distinguish the roles of coefficient identification, stochastic realization, and multi-pass optimization.

Having identified the coefficient-ambiguity problem, we further demonstrate empirically that selecting the appropriate coefficient is central to ROOT's improvement. Beyond coefficient identification, RootSolve provides a conditionally exact finite-tilt objective, while RootTrace further extends the framework to efficient multi-pass reuse of reward-scored endpoints. Together, these theoretical findings, optimization methods, and controlled empirical results establish ROOT's novelty and effectiveness.

AuthorsOpenReview ↗

Response 2/3 — RootTrace design and reuse depth (W2)

RootTrace design and reuse depth (W2)

We appreciate these questions on RootTrace and clarify the fixed-ratio approximation, its control, and the effect of reuse depth. In the revised manuscript, we revised the motivation in Section 3.3, added Appendix D.4, added the staged-ratio analysis in Appendix D.5, and added the reuse-depth discussion in Appendix F.4.

Fixed ratio and moving reference. The ratio ρ=q⋆/pb\rho=q^\star/p_b is defined relative to the frozen behavior distribution and remains valid when reusing the same scored endpoints. In fact, repeatedly applying RootSolve with fixed ρ\rho preserves the conditional target exactly (Theorem 2). The limitation lies in optimization efficiency. RootSolve retains the behavior-restoring term v−vbv-v_b on every pass, which pulls the updated model toward the original behavior prediction and counteracts reward-driven updates. RootTrace addresses this limitation by moving the reference to the detached current prediction while retaining the full reward correction (ρ−1)(v−ut)(\rho-1)(v-u_t). This allows subsequent passes to apply reward-dependent corrections at the updated model rather than repeatedly restoring the behavior field.

This moving reference introduces a target shift, which we explicitly characterize in Appendix D.4. Preserving conditional exactness would require the current-to-behavior density ratio rk=pθk−1/pbr_k=p_{\theta_{k-1}}/p_b, yielding the corrected field ρ(v−ut)−rk(vˉk−1−ut)\rho(v-u_t)-r_k(\bar v_{k-1}-u_t). Computing rkr_k for deterministic flow samplers requires costly likelihood estimation, so RootTrace adopts rk≈1r_k\approx1. Importantly, the resulting reuse updates have a clear optimization interpretation. Appendix D.4 establishes that each reuse pass follows the gradient of the contrastive risk C=Lq⋆−Lb\mathcal C=\mathcal L_{q^\star}-\mathcal L_b, where Lq⋆\mathcal L_{q^\star} and Lb\mathcal L_b are the target and behavior flow-matching risks. Under Assumption 1, their gradients agree at the behavior solution because ∇Lb=0\nabla\mathcal L_b=0 there. Thus, moving-reference reuse retains the target-improving first-order direction near the behavior field while removing the behavior-restoring gradient that limits repeated RootSolve.

Finite reuse. Appendix D.4 further relates contrastive descent to target-risk improvement through the following identity (Eq. (88)).

=\Delta\mathcal C+\frac12\Delta\mathbb E_b\|v-v_b\|^2.$$ Consequently, a reuse update decreases the target risk whenever the contrastive-risk reduction exceeds the increase in the behavior-displacement term. Near the behavior solution, this displacement is second-order, whereas its accumulated effect can reduce the benefit of deeper reuse. This analysis motivates finite reuse with periodic behavior refresh. RootTrace refreshes the behavior model, density ratios, and scored endpoints at every outer iteration, limiting how long the moving-reference approximation is applied to a fixed batch. We further evaluate $K\in\{1,2,4\}$ in Appendix F.4. At a fixed budget of 200 optimizer updates, these settings use 200, 100, and 50 freshly scored rollout batches, respectively. The lower reward at $K=4$ is consistent with the diminishing benefit of deeper reuse and the reduced frequency of fresh rollouts. The experiment does not exhibit the training collapse suggested by the reviewer. We use $K=2$ to demonstrate effective finite reuse without introducing additional trust-region constraints or likelihood estimation. Finally, we compare against staged-ratio RootSolve, $\rho_k=1+\frac{k}{K}(\rho-1)$, which preserves conditional exactness for each intermediate target while distributing the reward correction across passes. This alternative underperforms both repeated RootSolve and RootTrace across all four tasks (Figure 4 and Appendix D.5). This comparison supports our design choice of applying the full reward correction at the moving reference, rather than retaining behavior restoration or splitting the correction across passes.
AuthorsOpenReview ↗

Response 3/3 — Training horizon and additional baselines (W3, W4)

Training horizon (W3)

We thank the reviewer for emphasizing the training horizon. Figures 4–5 are short mechanism studies under matched budgets. Our longer-horizon evidence comes from the sequential multi-reward training in Section 4.3. We advanced to the next stage after observing convergence of the training rewards, which occurred after approximately 500 iterations in Stage 1 and 300 iterations in Stage 2, and terminated Stage 3 after another 200 iterations when its rewards had also converged. These observations determined our three-stage, 1k-iteration schedule (Table 4). For a controlled comparison, we trained DiffusionNFT using the same three-stage schedule. ROOT (1k) outperforms DiffusionNFT (1k) across all ten metrics and achieves performance comparable to the released DiffusionNFT (1.7k) checkpoint (Table 1).

Additional baselines (W4)

We thank the reviewer for suggesting these related works. As stated in the ICLR 2027 Reviewer Guidelines, direct comparisons with contemporaneous or arXiv-only works are not required. Nevertheless, we have expanded Section 5 to discuss their contributions and clarify their connections to ROOT. We would also like to emphasize that our baseline selection is guided by the specific theoretical and algorithmic claims of this paper.

GRPO-Guard [1] and TempFlow-GRPO [2] improve trajectory-based policy optimization through regulated importance-ratio clipping and temporal credit assignment, respectively. These methods operate on trajectory-level policy gradients, with a sampling and likelihood interface different from forward-process training. FlowAWR [3] derives advantage-weighted velocity rectification from KL-regularized reward maximization, sharing ETM's reward-tilting motivation and residual-correction form. The Unified Path-Space View [4] connects trajectory-based and forward-matching RL through variance-reduced value-gradient estimation. These works address related optimization problems, while ROOT specifically investigates coefficient identification, conditional finite-tilt exactness, and multi-pass reuse of reward-scored endpoints.

Importantly, our experiments are designed to directly isolate and validate these contributions. We adopt DiffusionNFT, an established forward-process RL method, as our primary baseline, with direct comparisons across four reward objectives and a matched 1k-iteration multi-reward schedule (Figures 3–4 and Table 1). Beyond this comparison, ITM and its scale-aligned variant examine coefficient identification, UDR and WFM examine stochastic-field construction with the same normalized ratio ρ\rho, and repeated RootSolve and staged-ratio RootSolve evaluate the benefit of moving-reference reuse (Figures 4–5). These controlled comparisons isolate the sources of ROOT's improvement under matched training configurations. Together with Theorems 1–2 and the finite-reuse analysis in Appendix D, they provide direct theoretical and empirical evidence for ROOT's central claims. We believe the existing matched comparisons and controlled ablations provide substantial evidence for ROOT's theoretical novelty and algorithmic effectiveness.

According to the reviewer's comments, we have revised Sections 3.3, 4.2, 4.3, and 5 and Appendices D.4, D.5, and F.4. We hope these revisions and clarifications address the reviewer's concerns, and we would be glad if the reviewer could consider raising the score.

References:

[1] Wang et al., GRPO-Guard: Mitigating implicit over-optimization in flow matching via regulated clipping., (CVPR 2026).

[2] He et al., TempFlow-GRPO: When timing matters for GRPO in flow models., (ICLR 2026).

[3] Fu et al., FlowAWR: Online adaptive flow reinforcement via advantage-weighted rectification., (arXiv 2026).

[4] Xu et al., Designing reinforcement learning for diffusion models: A unified path-space view., (NeurIPS 2026).

[5] Potaptchik et al., Tilt matching for scalable sampling and fine-tuning., (ICML 2026).

[6] Zheng et al., DiffusionNFT: Online diffusion reinforcement with forward process., (ICLR 2026).

[7] Liu et al., Flow-GRPO: Training flow matching models via online RL., (NeurIPS 2025).

Reviewer_7RzuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies coefficient representation in forward-process reward tuning. Different positive weights can define the same reward-tilted endpoint distribution while producing different stochastic updates when training uses a behavior-residual control variate. The authors introduce RootSolve, which fixes the coefficient using the mean-one target-to-behavior density ratio. Under behavior-flow consistency and additional assumptions, they prove that its conditional population optimum matches the target flow. RootTrace moves the reference prediction on subsequent reuse passes to remove the behavior-restoring term. Experiments on SD3.5-M include four single-reward tasks, ITM and UDR comparisons, reuse-depth and tilt ablations, and multi-reward checkpoint evaluations. These results support the method’s practical utility. The main novelty lies in separating coefficient identification from stochastic realization and introducing the corresponding rollout-reuse strategy. RootSolve shares the stochastic-field form of existing ITM.

Critical strengths
  • The paper distinguishes the target distribution from its stochastic realization, establishes representation dependence within the studied update family, and provides a representation-invariant coefficient choice.
  • Comparisons with ITM, UDR, and repeated RootSolve isolate distinct design choices. RootTrace’s gains under matched reuse budgets and the final multi-reward performance provide substantive support for acceptance.
Critical weaknesses
  • The ITM–RootSolve comparison changes both coefficient representation and mean update scale. Figure 5 lacks the scale-aligned control corresponding to Theorem 1, leaving the source of ROOT’s advantage insufficiently isolated.
  • RootSolve’s exactness relies on the population density ratio and behavior-flow consistency. Subsequent RootTrace updates generally do not preserve the same target root, and the resulting shift remains insufficiently analyzed. Efficiency evidence lacks runtime measurements, while Table 1 compares checkpoints trained with different stage schedules.
Questions for authors
  • After aligning ITM’s overall field scale and allowing reasonable learning-rate adjustments, do the collapse and ROOT’s advantage in Figure 5 persist?
  • Please provide the exact formula for detached adaptive loss weighting and specify the variables on which it depends. Does it preserve the conditional optimum in Theorem 2? Also, how do the authors interpret and control the shift introduced by subsequent RootTrace updates, which generally no longer have the original tilted target flow as their root?
  • Please report the per-iteration runtime, total training GPU-hours, and reward curves against GPU-hours for ROOT and DiffusionNFT on identical hardware. For Table 1, how does DiffusionNFT perform under the same three-stage, 1,000-iteration schedule? This would help distinguish algorithmic gains from differences in training schedules.
  • Some text-containing examples in Figures 9 and 11 still exhibit text errors or unnatural integration into the image. Although these multi-reward checkpoints differ from the setting in Figure 4(d), could the authors provide independent test results corresponding to the single-reward training curves to verify whether faster OCR reward improvement also improves text accuracy and overall perceptual quality?
  • Why does Table 1 report HPSv2.1 only on DrawBench without results on the test set corresponding to that single-reward task? How should the current efficiency conclusions be scoped with respect to other backbone models, such as FLUX or Qwen-Image, and different training paradigms, such as Flow-GRPO?
7 comments
AuthorsOpenReview ↗

Response 1/3 — ITM scale-alignment and learning-rate controls (W1, Q1)

ITM scale-alignment and learning-rate controls (W1, Q1)

We thank the reviewer for highlighting the need to separate coefficient representation from overall field scaling. To address this concern, we added a scale-aligned ITM control (ITM-A), which multiplies the entire stochastic field by 1/a^(c)1/\hat{a}(c), where a^(c)\hat{a}(c) is the group mean of wi=eτAiw_i=e^{\tau A_i}. This correction follows the normalization relationship in Theorem 1 and aligns ITM's overall field scale with ROOT while retaining its original coefficient representation. All other training settings, including the optimizer and learning rate, remain unchanged in this scale-alignment control.

The updated Figure 5 (Appendix F.4) shows that, at the matched learning rate, ITM-A closely follows the original ITM trajectory. Both initially improve reward but subsequently collapse on all four tasks, whereas RootSolve continues improving. We further evaluated ITM-A with reduced learning rates (Table R1). Lower learning rates delay reward degradation, but the tested runs have not achieved sustained reward improvement over the evaluated training horizon.

These findings are consistent with Theorem 1. Although scale normalization aligns the conditional mean fields under the stated assumptions, their sample-wise realizations still differ by the residual (a(c)−1−1)(ut−vb)(a(c)^{-1}-1)(u_t-v_b). This residual has zero conditional mean under Assumption 1 but affects individual stochastic updates, leading to different optimization dynamics. Together, the scale-alignment and learning-rate controls show that ROOT's advantage persists beyond overall field scaling and the tested learning-rate adjustments, supporting the importance of identifying the appropriate coefficient representation for stochastic forward-process training.

MethodLRIter. 1–2526–5051–7576–100
ITM-A3×10−43\times10^{-4}0.0040.0000.0000.000
ITM-A1×10−41\times10^{-4}0.0550.0000.0000.000
ITM-A3×10−53\times10^{-5}0.1000.0670.0100.002
ITM-A1×10−51\times10^{-5}0.0820.1060.1050.082
ROOT3×10−4\mathbf{3\times10^{-4}}0.5160.8000.8570.894

Table R1. ROOT versus ITM-A with different learning rates on single-reward single-pass (K=1K=1) GenEval training. The numbers are training rewards averaged over the given ranges of outer iterations.

AuthorsOpenReview ↗

Response 2/3 — Exactness, adaptive weighting, and training efficiency (W2, Q2, Q3)

Exactness and adaptive loss weighting (W2 part 1, Q2)

We appreciate the reviewer's careful reading of the exactness conditions. We first clarify the shift in later RootTrace passes, and then discuss the detached adaptive loss weighting.

Intuitively, repeating RootSolve on the same scored endpoints keeps its conditional optimum (Theorem 2), so multi-pass reuse is already valid. The question is whether it is efficient. Our insight is that the behavior-restoring term v−vbv-v_b, which is necessary for exactness, keeps pulling the updated model back toward the behavior prediction and limits reward-driven progress under a finite update budget. RootTrace therefore moves the reference to the detached current prediction and keeps the full reward correction (ρ−1)(v−ut)(\rho-1)(v-u_t) (Section 3.3, Eqs. (16) and (18), and Figure 4).

The price is exactness on later passes. An exact correction would need the importance ratio rk=pθk−1/pbr_k=p_{\theta_{k-1}}/p_b of the live model on the old endpoints, which gives (ρ−rk)(v−ut)(\rho-r_k)(v-u_t) and recovers the target flow. For deterministic flow samplers, however, rkr_k requires ODE divergence integration or likelihood proxies with extra compute or approximation error, and estimating it without additional budget remains open. RootTrace therefore uses rk≈1r_k\approx1 as a feasible implementation. Appendix D.4 shows what this choice optimizes. Reuse descends the contrastive risk C=Lq⋆−Lb\mathcal C=\mathcal L_{q^\star}-\mathcal L_b, where Lb\mathcal L_b and Lq⋆\mathcal L_{q^\star} are the flow-matching risks of the model under the behavior and the reward-improved endpoint distributions. At the behavior solution, the gradients of CC and Lq⋆L_{q^\star} coincide. Thus, RootTrace follows a target-aligned first-order direction near the behavior model, while Eq. (89) characterizes when a finite reuse update also decreases the target risk.

Detached adaptive loss weighting almost follows DiffusionNFT [1]. We provide its exact formula and dependencies in Appendix F.1. The positive, detached weight depends only on (xt,t,x0)(x_t,t,x_0) and on stop-gradient predictions, not on the reward, ρ\rho, or the prompt group. It preserves samplewise roots, but does not in general preserve the conditional optimum of Theorem 2, because it depends on the sampled endpoint x0x_0. We therefore treat it as a practical normalization strategy, supported by DiffusionNFT's empirical findings and applied consistently across matched comparisons, separate from the exactness guarantee of RootSolve. Ablation study of this adaptive loss weighting is added in Appendix F.4.

Training efficiency and matched baseline (W2 part 2, Q3)

We thank the reviewer for asking for runtime evidence. We timed both methods on 16 A800 GPUs. Over the three stages of the multi-reward schedule, a ROOT outer iteration takes 152.7, 159.5, and 165.7 s versus 152.9, 158.5, and 164.7 s for DiffusionNFT. Both methods use the same rollouts, reward scoring, and a single forward-backward pass over the same noised endpoints. ROOT only changes the per-sample coefficient of the loss and adds no computation. The nearly identical per-iteration runtimes allow the single-pass reward curves in Figure 3 to be compared directly in GPU hours. For the 1,000-iteration multi-reward schedule, this amounts to about 700 GPU hours (about 44 hours on 16 GPUs) for either method. We describe the training cost in Section 4.1 and Appendix F.2. Appendix F.4 (Figure 6(d)) additionally reports GPU-hour efficiency across ROOT's reuse depths K∈{1,2,4}K \in \{1,2,4 \}.

We have also added a DiffusionNFT baseline trained under the identical 3-stage, 1000-iteration schedule (Section 4.3, Table 1). Under this matched schedule, ROOT outperforms DiffusionNFT on all ten metrics. Moreover, ROOT at 1k iterations achieves comparable performance to the released DiffusionNFT [1] checkpoint trained for approximately 1.7k iterations. These results show that ROOT achieves comparable overall generation quality with 41% fewer training iterations, without evidence of a systematic quality decline across the evaluated metrics.

Reference: [1] Zheng et al., DiffusionNFT: Online Diffusion Reinforcement with Forward Process (ICLR 2026).

AuthorsOpenReview ↗

Response 3/3 — Held-out evaluation and scope (Q4, Q5)

Held-out evaluation (Q4, Q5 part 1)

We thank the reviewer for highlighting the distinction between OCR reward improvement and overall generation quality. To illustrate the OCR-only training dynamics, we provide a visualization tracking the first 50 training iterations of ROOT and DiffusionNFT using fixed prompts and seeds. The visualization is available at https://anonymous.4open.science/w/ocr_showcase-0813/index.html. The results show that ROOT rapidly improves OCR scores, consistent with Figure 4(d). As training progresses, both methods gradually sacrifice non-textual visual content to favor OCR accuracy. This reflects a limitation of OCR-only supervision, where higher reward does not necessarily imply better overall visual quality. With reward models that more faithfully reflect human judgments of text accuracy and visual quality, we expect ROOT's efficient reward optimization to translate into stronger generation quality.

The examples highlighted by the reviewer are generated by multi-reward checkpoints, whose overall quality is more reliably assessed through aggregate evaluation across diverse prompts and metrics than through individual examples. Table 1 provides this broader comparison, including perceptual-quality and preference measures. We have also added the HPSv2.1 metric on the Pick-a-Pic test set as a new column in Table 1 (Section 4.3). ROOT (0.352) matches the released DiffusionNFT checkpoint (0.353) with about 41% fewer iterations and exceeds the same-schedule DiffusionNFT (0.336).

Scope beyond SD3.5-M and Flow-GRPO (Q5 part 2)

We thank the reviewer for raising the question of generalizability. ROOT is formulated within the forward-process flow-matching framework and does not rely on architecture-specific properties of SD3.5-M. Our experiments follow DiffusionNFT's SD3.5-M setup to ensure a controlled comparison under the same training paradigm. Importantly, recent works have extended DiffusionNFT-based training to other backbones. For example, UniWorld-V2 [3] applies it to FLUX.1 Kontext and Qwen-Image-Edit, while A Unified Path-Space View [4] evaluates DiffusionNFT on Qwen-Image. These results support the broader applicability of forward-process reward tuning, and we expect ROOT's coefficient correction to extend to these backbones as well.

Regarding Flow-GRPO [2], our work focuses on resolving coefficient ambiguity in forward-process reward tuning, whereas Flow-GRPO employs trajectory-based policy optimization. We therefore select DiffusionNFT as the primary baseline to isolate ROOT's contributions within the same training paradigm. DiffusionNFT has already reported up to 25×25\times higher training efficiency than Flow-GRPO on SD3.5-M, and ROOT further improves upon this established forward-process baseline. We have clarified this scope in Related Work (Section 5).

According to the reviewer's comments, we have revised Sections 4.1, 4.3, and 5 and Appendices D.4, F.1, F.2, F.3, and F.4. We hope these revisions and clarifications address the reviewer's concerns, and we would be glad if the reviewer could consider raising the score.

References:

[1] Zheng et al., DiffusionNFT: Online diffusion reinforcement with forward process,. (ICLR 2026).

[2] Liu et al., Flow-GRPO: Training flow matching models via online RL., (NeurIPS 2025).

[3] Li et al., UniWorld-V2: Reinforce image editing with diffusion negative-aware finetuning and MLLM implicit feedback., (arXiv 2025).

[4] Xu et al., Designing reinforcement learning for diffusion models: A unified path-space view., (NeurIPS 2026).

Reviewer_L1qwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ROOT, an endpoint-reward tuning framework for flow models based on normalized target-to-behavior ratios. It addresses coefficient ambiguity, where weights defining the same endpoint distribution can produce different stochastic velocity updates. Experiments on SD3.5-M show improved training-reward efficiency and comparable final multi-reward performance with fewer outer iterations.

Critical strengths
  1. The paper elegantly uses normalized target-to-behavior ratios to eliminate the dependence of ITM-style stochastic updates on endpoint-equivalent coefficient scaling.
  2. RootSolve provides a conditionally exact objective for learning the reward-improved flow under behavior-flow consistency, while RootTrace addresses the restrictive behavior anchor during rollout reuse.
  3. Experiments across multiple reward objectives demonstrate effective reward optimization and substantially fewer training iterations to reach comparable performance.
Critical weaknesses
  1. The conditional exactness theorem applies to RootSolve and the first pass of RootTrace, but does not establish target correctness or stability for the full multi-pass procedure.
  2. The theoretical scope is relatively narrow: the analysis focuses on ITM-style gradient constructions, whereas ordinary weighted flow matching does not exhibit the same residual ambiguity after scale alignment.
  3. Despite stronger reported metrics than the CFG baseline, some ROOT examples in Appendix F.5 appear less faithful in artistic style than DiffusionNFT, such as the impressionist portrait in Figure 10. A controlled perceptual evaluation is needed to determine whether these differences reflect a systematic efficiency–quality trade-off.
Questions for authors
  1. Beyond the pass-specific loss in Eq. (17), what endpoint-level reward or distributional improvement criterion justifies RootTrace’s reuse updates, given that they generally do not preserve the target flow as a conditional fixed point?
  2. The ITM and UDR comparisons provide useful ablations, but does ROOT retain its advantage over ITM after explicit mean-field scale alignment and independent learning-rate tuning?
  3. Beyond the empirical reuse-depth study in Figure 7, can the authors establish a finite-reuse stability or error bound with explicit conditions on the step size and number of passes (K)?
5 comments
AuthorsOpenReview ↗

Response 1/3 — Multi-pass RootTrace: correctness and stability (W1, Q1, Q3)

Multi-pass RootTrace: correctness and stability (W1, Q1, Q3)

We thank the reviewer for raising the question of target correctness in multi-pass reuse. We emphasize that the subsequent RootTrace passes do not maintain conditional exactness, and that this is part of our design. We have clarified the motivation for this design and the conditions under which reuse remains meaningful (Section 3.3 and Appendix D.4).

Intuitively, repeating RootSolve on the same scored endpoints keeps its conditional optimum (Theorem 2), so multi-pass reuse is already valid. The question is whether it is efficient. Our insight is that the behavior-restoring term v−vbv-v_b, which is necessary for exactness, keeps pulling the updated model back toward the behavior prediction and limits reward-driven progress under a finite update budget. RootTrace therefore moves the reference to the detached current prediction and keeps the full reward correction (ρ−1)(v−ut)(\rho-1)(v-u_t) (Section 3.3, Eqs. (16) and (18), and Figure 4).

The price is exactness on later passes. An exact correction would need the importance ratio rk=pθk−1/pbr_k=p_{\theta_{k-1}}/p_b of the live model on the old endpoints, which gives (ρ−rk)(v−ut)(\rho-r_k)(v-u_t) and recovers the target flow. For deterministic flow samplers, however, rkr_k requires ODE divergence integration or likelihood proxies with extra compute or approximation error, and estimating it without additional budget remains open. RootTrace therefore uses rk≈1r_k\approx1 as a feasible implementation. Appendix D.4 shows what this choice optimizes. Reuse descends the contrastive risk C=Lq⋆−Lb\mathcal C=\mathcal L_{q^\star}-\mathcal L_b, where Lb\mathcal L_b and Lq⋆\mathcal L_{q^\star} are the flow-matching risks of the model under the behavior and the reward-improved endpoint distributions. At the behavior solution, the gradients of CC and Lq⋆L_{q^\star} coincide. Thus, RootTrace follows a target-aligned first-order direction near the behavior model, while Eq. (89) characterizes when a finite reuse update also decreases the target risk.

The analysis also clarifies the role of reuse depth. Since C\mathcal C is not generally lower-bounded and the behavior displacement may accumulate with KK, the relationship between contrastive descent and target-risk improvement is local. We evaluate K=2K=2 to demonstrate the effectiveness of finite reuse, refreshing ρ\rho, vbv_b, and the scored endpoints at every outer iteration (Appendix F.4). We further investigate a staged-ratio scheme ρk=1+kK(ρ−1)\rho_k=1+\frac{k}{K}(\rho-1) using RootSolve across multiple passes. While this scheme preserves conditional exactness for each intermediate target, it underperforms both repeated RootSolve and RootTrace on all four tasks under the same budget (Figure 4 and Appendix D.5).

AuthorsOpenReview ↗

Response 2/3 — Weighted flow matching and perceptual quality (W2, W3)

Scope relative to weighted flow matching (W2)

We appreciate the reviewer's question on the scope of our analysis. Our analysis focuses on the calibrated stochastic field motivated by implicit Tilt Matching (ITM) [1], which has a variance advantage over ordinary weighted flow matching (WFM) under the small-tilt conditions studied in [1]. At the un-tilted behavior solution, WFM retains the stochastic residual vb−utv_b-u_t, whereas ITM cancels it samplewise. This motivates our focus on the calibrated field (Appendix E).

Ordinary WFM is indeed homogeneous in its coefficient and does not exhibit the same residual ambiguity after scale alignment. Theorem 1 instead characterizes endpoint-equivalent coefficients within the calibrated stochastic-field family, and ROOT resolves this ambiguity using the normalized target-to-behavior ratio.

We additionally compare WFM using the same normalized ratio ρ\rho as ROOT (updated Figure 5 in Appendix F.4). In the unweighted population analysis, WFM and RootSolve share the same conditional target under Assumption 1, while their stochastic fields differ by the behavior residual ut−vbu_t-v_b. Under the matched practical training protocol, WFM exhibits reward collapse while ROOT continues improving. This highlights the importance of stochastic-field construction even with an identical endpoint coefficient.

Perceptual quality and efficiency trade-off (W3)

We thank the reviewer for the observation. While individual examples may exhibit differences in artistic style, the systematic quality–efficiency trade-off is better assessed through the aggregate evaluation across diverse prompts and metrics in Table 1. To further strengthen this comparison, we have added a DiffusionNFT baseline trained under the identical three-stage, 1k-iteration schedule (Section 4.3, Table 1). Under this matched budget, ROOT outperforms DiffusionNFT on all ten metrics. Furthermore, ROOT at 1k iterations achieves comparable overall performance to the released DiffusionNFT checkpoint trained for approximately 1.7k iterations, using 41% fewer training iterations. These results support ROOT's efficiency gains without indicating a systematic decline in the evaluated quality metrics. More broadly, we believe that when reward models faithfully reflect human perceptual preferences, ROOT's improved reward optimization efficiency can translate into corresponding gains in perceptual quality. For completeness, we have also updated Appendix F.5 to include qualitative comparisons with both the matched-budget DiffusionNFT (1k) model and the released checkpoint.

Reference: [1] Potaptchik et al., Tilt Matching for Scalable Sampling and Fine-Tuning (ICML 2026).

AuthorsOpenReview ↗

Response 3/3 — Scale-aligned ITM and learning-rate controls (Q2)

Scale-aligned ITM and learning-rate controls (Q2)

We appreciate the reviewer's suggestion to further isolate the source of ROOT's advantage. Following Theorem 1, we have added a scale-aligned ITM control (ITM-A), which multiplies the entire stochastic field by 1/a^(c)1/\hat a(c), where a^(c)\hat a(c) is the group mean of wi=eτAiw_i=e^{\tau A_i}. This aligns ITM's overall field scale with RootSolve while preserving its original coefficient representation. We use identical training configurations to isolate the effect of scale alignment.

The updated Figure 5 (Appendix F.4) shows that ITM-A closely follows the original ITM trajectory. Both initially improve reward but subsequently collapse on all four rewards, whereas RootSolve continues improving. This observation is consistent with Theorem 1. Although scale alignment makes the conditional mean fields equivalent under the stated assumptions, the stochastic estimators still differ by the sample-dependent residual (a(c)−1−1)(ut−vb)(a(c)^{-1}-1)(u_t-v_b) in Eq. (7). This residual has zero conditional mean under the ideal behavior-field assumption, yet affects individual stochastic updates. Consequently, matching the mean field does not imply equivalent optimization dynamics, as also demonstrated by the controlled experiment.

We further evaluated ITM-A with independently adjusted learning rates while keeping the remaining training settings fixed (Table R1). Lower learning rates delay reward degradation, but none of the tested settings has produced sustained reward improvement over the evaluated training horizon. In contrast, RootSolve continues improving under the original training protocol (Figure 5). These results show that ROOT's advantage cannot be explained solely by overall field scaling or by the tested learning rates. This observation is consistent with Theorem 1. Even after mean-field scale alignment, the stochastic estimators differ in their sample-wise updates.

MethodLRIter. 1–2526–5051–7576–100
ITM-A3×10−43\times10^{-4}0.0040.0000.0000.000
ITM-A1×10−41\times10^{-4}0.0550.0000.0000.000
ITM-A3×10−53\times10^{-5}0.1000.0670.0100.002
ITM-A1×10−51\times10^{-5}0.0820.1060.1050.082
ROOT3×10−4\mathbf{3\times10^{-4}}0.5160.8000.8570.894

Table R1. ROOT versus ITM-A with different learning rates on single-reward single-pass (K=1K=1) GenEval training. The numbers are training rewards averaged over the given ranges of outer iterations.

According to the reviewer's comments, we have revised Sections 3.3, 4.2, and 4.3 and Appendices D.4, D.5, E, F.4, and F.5. We hope these revisions and clarifications address the reviewer's concerns, and we would be glad if the reviewer could consider raising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.