- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ROOT for reward tuning of flow models. ROOT uses the target-to-behavior density ratio as the unique mean-one canonical coefficient to construct a calibrated velocity training field, RootSolve, and further proposes RootTrace. Overall, the paper has some good motivations, but there are also some flaws in the core method.
Critical strengths
The overall writing is good and the paper is fairly polished. In particular, the second contribution is relatively clear and well articulated.
Critical weaknesses
-
Limited novelty and lack of attribution. The paper’s novelty is primarily demonstrated against ITM (Implicit Tilt Matching), which is a baseline constructed by the authors rather than an established method. The original DiffusionNFT does not suffer from the coefficient-ambiguity issue emphasized in the paper. Moreover, the paper does not provide an attribution analysis explaining why ROOT outperforms DiffusionNFT overall. Is the gain mainly due to the difference between the probability-ratio coefficient p and DiffusionNFT’s bounded optimality coefficient? If so, the novelty appears limited.
-
RootTrace is well motivated (but the design is too simple), the Figure 4 only reports the K=2 setting. As the model is updated, the method reuses a fixed p without any corresponding trust-region constraint. The consistent degradation at K=4 clearly indicates an instability that is not analyzed.
-
Some figures (e.g., Figures 4 and 5) cover only a limited number of training steps. For an RL method, it is important to demonstrate stable convergence over longer training horizons, rather than only short-term gains.
-
As a new RL method, the paper compares against a limited set of baselines. It should include comparisons with RL/post-training methods such as FlowGRPO-Guard[1], TempFlowGRPO[2], FlowAWR[3], and related approaches[4].
[1] GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping
[2] When Timing Matters for GRPO in Flow Models
[3] FLOWAWR: ONLINE ADAPTIVE FLOW REINFORCEMENT VIA ADVANTAGE-WEIGHTED RECTIFICATION
[4] Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
Questions for authors
Please see the weaknesses.
9 comments
Response 2/3 — RootTrace design and reuse depth (W2)
RootTrace design and reuse depth (W2)
We appreciate these questions on RootTrace and clarify the fixed-ratio approximation, its control, and the effect of reuse depth. In the revised manuscript, we revised the motivation in Section 3.3, added Appendix D.4, added the staged-ratio analysis in Appendix D.5, and added the reuse-depth discussion in Appendix F.4.
Fixed ratio and moving reference. The ratio is defined relative to the frozen behavior distribution and remains valid when reusing the same scored endpoints. In fact, repeatedly applying RootSolve with fixed preserves the conditional target exactly (Theorem 2). The limitation lies in optimization efficiency. RootSolve retains the behavior-restoring term on every pass, which pulls the updated model toward the original behavior prediction and counteracts reward-driven updates. RootTrace addresses this limitation by moving the reference to the detached current prediction while retaining the full reward correction . This allows subsequent passes to apply reward-dependent corrections at the updated model rather than repeatedly restoring the behavior field.
This moving reference introduces a target shift, which we explicitly characterize in Appendix D.4. Preserving conditional exactness would require the current-to-behavior density ratio , yielding the corrected field . Computing for deterministic flow samplers requires costly likelihood estimation, so RootTrace adopts . Importantly, the resulting reuse updates have a clear optimization interpretation. Appendix D.4 establishes that each reuse pass follows the gradient of the contrastive risk , where and are the target and behavior flow-matching risks. Under Assumption 1, their gradients agree at the behavior solution because there. Thus, moving-reference reuse retains the target-improving first-order direction near the behavior field while removing the behavior-restoring gradient that limits repeated RootSolve.
Finite reuse. Appendix D.4 further relates contrastive descent to target-risk improvement through the following identity (Eq. (88)).
=\Delta\mathcal C+\frac12\Delta\mathbb E_b\|v-v_b\|^2.$$ Consequently, a reuse update decreases the target risk whenever the contrastive-risk reduction exceeds the increase in the behavior-displacement term. Near the behavior solution, this displacement is second-order, whereas its accumulated effect can reduce the benefit of deeper reuse. This analysis motivates finite reuse with periodic behavior refresh. RootTrace refreshes the behavior model, density ratios, and scored endpoints at every outer iteration, limiting how long the moving-reference approximation is applied to a fixed batch. We further evaluate $K\in\{1,2,4\}$ in Appendix F.4. At a fixed budget of 200 optimizer updates, these settings use 200, 100, and 50 freshly scored rollout batches, respectively. The lower reward at $K=4$ is consistent with the diminishing benefit of deeper reuse and the reduced frequency of fresh rollouts. The experiment does not exhibit the training collapse suggested by the reviewer. We use $K=2$ to demonstrate effective finite reuse without introducing additional trust-region constraints or likelihood estimation. Finally, we compare against staged-ratio RootSolve, $\rho_k=1+\frac{k}{K}(\rho-1)$, which preserves conditional exactness for each intermediate target while distributing the reward correction across passes. This alternative underperforms both repeated RootSolve and RootTrace across all four tasks (Figure 4 and Appendix D.5). This comparison supports our design choice of applying the full reward correction at the moving reference, rather than retaining behavior restoration or splitting the correction across passes.Response 3/3 — Training horizon and additional baselines (W3, W4)
Training horizon (W3)
We thank the reviewer for emphasizing the training horizon. Figures 4–5 are short mechanism studies under matched budgets. Our longer-horizon evidence comes from the sequential multi-reward training in Section 4.3. We advanced to the next stage after observing convergence of the training rewards, which occurred after approximately 500 iterations in Stage 1 and 300 iterations in Stage 2, and terminated Stage 3 after another 200 iterations when its rewards had also converged. These observations determined our three-stage, 1k-iteration schedule (Table 4). For a controlled comparison, we trained DiffusionNFT using the same three-stage schedule. ROOT (1k) outperforms DiffusionNFT (1k) across all ten metrics and achieves performance comparable to the released DiffusionNFT (1.7k) checkpoint (Table 1).
Additional baselines (W4)
We thank the reviewer for suggesting these related works. As stated in the ICLR 2027 Reviewer Guidelines, direct comparisons with contemporaneous or arXiv-only works are not required. Nevertheless, we have expanded Section 5 to discuss their contributions and clarify their connections to ROOT. We would also like to emphasize that our baseline selection is guided by the specific theoretical and algorithmic claims of this paper.
GRPO-Guard [1] and TempFlow-GRPO [2] improve trajectory-based policy optimization through regulated importance-ratio clipping and temporal credit assignment, respectively. These methods operate on trajectory-level policy gradients, with a sampling and likelihood interface different from forward-process training. FlowAWR [3] derives advantage-weighted velocity rectification from KL-regularized reward maximization, sharing ETM's reward-tilting motivation and residual-correction form. The Unified Path-Space View [4] connects trajectory-based and forward-matching RL through variance-reduced value-gradient estimation. These works address related optimization problems, while ROOT specifically investigates coefficient identification, conditional finite-tilt exactness, and multi-pass reuse of reward-scored endpoints.
Importantly, our experiments are designed to directly isolate and validate these contributions. We adopt DiffusionNFT, an established forward-process RL method, as our primary baseline, with direct comparisons across four reward objectives and a matched 1k-iteration multi-reward schedule (Figures 3–4 and Table 1). Beyond this comparison, ITM and its scale-aligned variant examine coefficient identification, UDR and WFM examine stochastic-field construction with the same normalized ratio , and repeated RootSolve and staged-ratio RootSolve evaluate the benefit of moving-reference reuse (Figures 4–5). These controlled comparisons isolate the sources of ROOT's improvement under matched training configurations. Together with Theorems 1–2 and the finite-reuse analysis in Appendix D, they provide direct theoretical and empirical evidence for ROOT's central claims. We believe the existing matched comparisons and controlled ablations provide substantial evidence for ROOT's theoretical novelty and algorithmic effectiveness.
According to the reviewer's comments, we have revised Sections 3.3, 4.2, 4.3, and 5 and Appendices D.4, D.5, and F.4. We hope these revisions and clarifications address the reviewer's concerns, and we would be glad if the reviewer could consider raising the score.
References:
[1] Wang et al., GRPO-Guard: Mitigating implicit over-optimization in flow matching via regulated clipping., (CVPR 2026).
[2] He et al., TempFlow-GRPO: When timing matters for GRPO in flow models., (ICLR 2026).
[3] Fu et al., FlowAWR: Online adaptive flow reinforcement via advantage-weighted rectification., (arXiv 2026).
[4] Xu et al., Designing reinforcement learning for diffusion models: A unified path-space view., (NeurIPS 2026).
[5] Potaptchik et al., Tilt matching for scalable sampling and fine-tuning., (ICML 2026).
[6] Zheng et al., DiffusionNFT: Online diffusion reinforcement with forward process., (ICLR 2026).
[7] Liu et al., Flow-GRPO: Training flow matching models via online RL., (NeurIPS 2025).
Response 1/3 — Novelty and attribution (W1)
Novelty and attribution (W1)
We thank the reviewer for the careful reading, and we respectfully disagree with this assessment of novelty.
First, identifying coefficient ambiguity is precisely our central contribution. DiffusionNFT [6] avoids this ambiguity by directly specifying a bounded optimality coefficient through its positive–negative policy construction. However, it does not address which coefficient is determined by a prescribed reward-improved endpoint distribution. We identify this missing link. Endpoint-equivalent coefficients can produce different stochastic updates (Theorem 1), while the target-to-behavior density ratio ρ provides the unique mean-one representative (Proposition 1). RootSolve then realizes this coefficient with a conditionally exact finite-tilt objective (Theorem 2). The difference in coefficients is therefore a consequence of deriving the optimization objective from the target distribution, rather than an empirical choice of weighting functions. This provides a principled solution to forward-process reward tuning without additional models or likelihood estimation.
Second, DiffusionNFT is our primary baseline, with direct comparisons in Figures 3–4 and Table 1. ITM is adapted from the implicit formulation in Tilt Matching [5] to our matched training protocol. We include it as a controlled comparison for coefficient attribution.
Third, besides the comparison of ROOT and DiffusionNFT, our controlled comparisons examine three distinct design choices. ROOT vs. ITM (Figure 5) isolates coefficient identification within the same stochastic field form, ROOT vs. UDR (Figure 5) isolates the stochastic realization while fixing ρ, and ROOT vs. repeated RootSolve (Figure 4) isolates the benefit of moving-reference reuse. These comparisons distinguish the roles of coefficient identification, stochastic realization, and multi-pass optimization.
Having identified the coefficient-ambiguity problem, we further demonstrate empirically that selecting the appropriate coefficient is central to ROOT's improvement. Beyond coefficient identification, RootSolve provides a conditionally exact finite-tilt objective, while RootTrace further extends the framework to efficient multi-pass reuse of reward-scored endpoints. Together, these theoretical findings, optimization methods, and controlled empirical results establish ROOT's novelty and effectiveness.