- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Differentiable Flow Policy Optimization (DFPO), an RL method for flow-matching policies trained in differentiable simulators. The core motivation is the use of flow-based or similar multi-step generation policies in vision-based settings in differentiable simulators. In such cases, the main issues are that backpropagating through rendering is usually infeasible or too expensive, so the observation path has to be truncated, which biases the analytical gradient. Backpropagating through the multi-step generation process of the policy is also costly. Likelihood-ratio gradients avoid both problems, but they have high variance and require the policy’s likelihood, which is intractable for flow policies.
DFPO adds a likelihood-ratio policy gradient (LRPG), estimated with the FPO++ surrogate, to a truncated gradient, and claims that it corrects the truncation bias. The paper also compares three ways of propagating the simulator’s action gradients through the flow policy. The method is evaluated on an LQR problem and six Genesis tasks with state and RGB observations.
Critical strengths
- Using differentiable simulation for vision-based generative policies is a relevant problem.
- The method is simple, and the comparison of the three propagation channels, including compute and memory profiles is useful for practitioners.
- The evaluation covers both state-based and RGB-based settings.
Critical weaknesses
My main issue is that of limited scientific novelty. It seems to me that the DFPO combines several existing components, already studied in prior work. The outcome seems closer to a systems/engineering contribution rather than a fully-developed method. I also have some reservations about experiments and methodology, and some presentation concerns.
- The method section is dense with equations but gives little intuition. For example, the three propagation channels and the FPO++ surrogate (Eqs. 16–18) are stated without explaining how they differ conceptually, and adjoint matching is only described in words. Can you please add intuitive explanations?
- Upon doing a literature search, it seems to me that combining first-order and likelihood-ratio gradients has been studied before. Parmas et al. (2018) blend the two at each time step with inverse-variance weights. Suh et al. (2022) interpolate them in differentiable simulators with a variance-based weight. Son et al. (2023) incorporate analytical gradients into PPO with an adaptive weight. Onoda et al. (2026) switch between or inverse-variance-weight the two. Only Suh et al. is cited, and only in passing. What is the contribution beyond these methods, and why are they not used as baselines?
- The motivation is not supported theoretically. While the authors acknowledge that the combined gradient is not unbiased, they state that the LRPG “corrects” the truncation bias. Adding an unbiased estimate to a biased one does not remove the bias; it only dilutes it, at the cost of higher variance. The two terms are also summed with fixed weights, while the BPTT term alone reaches very high gradient norms. If a global gradient-norm clip of 1.0 is applied, what is the effective contribution of each term? Why are they not weighted, e.g. by their variance as in prior work?
- The LQR study does not test the proposed method. It uses Gaussian policies, so neither the flow policy nor the FPO++ surrogate is evaluated.
- The baselines seem undertuned and the evaluation is weak. The first-order baselines use DFPO’s horizon (H = 8 for state, H = 16 for RGB) instead of the H = 32 used in the SHAC and RFO papers. Gradient-based methods also appear to be evaluated in a different simulator build than FPO and PPO. Are all experiments under comparable dynamics?
Questions for authors
Please see weaknesses.
2 comments
Thank you for your responses.
Novelty: Thanks for the clarifications. While this addresses my concerns partially, I still do not find the proposed algorithm convincingly novel. I will reserve my judgement for the reviewer discussion phase.
Intuition & Writing: The added section helps to make the paper clear. I would still suggest adding a notation section in the preliminaries and only labeling equations that are mention in the text. This would help make the paper look less busy.
Fig 4: Could you please provide some intuition on the situations where DFPO leads to an improvement, and those where it does not. It would help to contextualize why your gradient underperforms in some cases.
The rest of my concerns are addressed and I will maintain my score until the reviewer-AC discussions.
We thank the reviewer for the detailed feedback. We have substantially revised the method presentation and added theoretical analysis and experiments to address these concerns.
W1. Limited novelty and relation to prior work
We acknowledge that combining analytical and likelihood-ratio gradients has been explored in prior work. However, our main contributions extend beyond combining existing estimators, addressing two challenges specific to differentiable RL with flow policies:
(1) Sample-efficient differentiable RL for RGB-based flow policies. Non-differentiable rendering prevents full analytical gradient propagation, while iterative flow sampling makes policy differentiation computationally expensive. DFPO addresses both challenges, achieving substantially higher sample efficiency than LRPG-only and truncated APG-only methods.
(2) Direct One-step Transport (DOT). We introduce a new method for transporting action-level policy gradients to flow-policy parameters. Unlike BPTT, which differentiates through the entire flow trajectory, or Adjoint Matching (AM), which propagates adjoints through stored flow states, DOT transports the gradient through a single velocity-field evaluation, avoiding expensive multi-step differentiation. Across 12 tasks, BPTT, AM, and DOT achieve aggregate normalized scores of 10.3, 10.2, and 9.5, respectively. Meanwhile, DOT reduces computation time to approximately 1/5 (state-based) and 1/3 (RGB-based) compared with BPTT.
While Parmas et al., Suh et al., Son et al., and Onoda et al. investigate combining gradient estimators, our setting differs in that the analytical gradient is truncated by non-differentiable observations, while the LRPG is approximated through a flow-matching surrogate rather than an unbiased likelihood-ratio estimator. We will expand the related-work discussion to clarify these distinctions.
W2. Method intuition
We agree that the original presentation lacked intuition. We have rewritten Section 3.1 to explain the three gradient transport mechanisms conceptually.
W3. Bias, theoretical motivation, and weighting
We agree that simply adding an unbiased LRPG to a biased TAPG does not eliminate truncation bias. Moreover, the FPO++ surrogate used in our flow-policy setting is not necessarily unbiased, making conventional unbiased-estimator variance arguments insufficient.
To address this, we added Proposition 1 (Sec. 3.3), establishing a sufficient bound under which the combined gradient is an ascent direction and guarantees improvement with an appropriate step size. The bound explicitly characterizes how gradient agreement affects optimization: alignment reinforces the update, while conflicting gradients may compromise it.
Motivated by this analysis, we introduce cosine-similarity-based adaptive weighting. As shown in Fig. 6, our adaptive ratio substantially improves RGB-based performance AND performs comparably to fixed weighting in state-based tasks.
Regarding gradient magnitudes, BPTT's pre-clip norms frequently exceed the clipping threshold of 1.0, effectively normalizing its updates. This also highlights BPTT's sensitivity to gradient explosion and its limitation to one analytical-gradient update per rollout, whereas the LRPG surrogate supports repeated minibatch optimization.
W4. LQR study and flow-policy validation
We agree that the Gaussian-policy LQR study alone does not establish gradient quality for flow policies. We therefore added Fig. 4, directly evaluating update quality at flow-policy training checkpoints across all 12 state-based and RGB-based tasks, using the actual flow policy and lrpg surrogate.
DFPO outperforms both individual gradients in approximately 40% of cases, outperforms one in another 40%, and underperforms both in approximately 20%. This provides direct evidence that combining the signals frequently improves the optimization direction in our actual setting, complementing the controlled LQR analysis in Appendix H.
W5. Baseline tuning and simulator consistency
All methods are evaluated under identical simulator dynamics, physics parameters, and task configurations. The only implementation difference is that FPO and PPO do not construct or retain simulation gradient graphs, since they do not require differentiating through the simulator. This does not alter the forward dynamics.
We acknowledge that our shared rollout horizon differs from the original SHAC/RFO configurations and are rerunning these baselines with their original settings. These experiments are still ongoing due to the substantial GPU resources required for our adaptive-weighting experiments. Nevertheless, changing the horizon cannot eliminate the systematic bias introduced by truncating observation gradients, as further supported by our TAPG-only ablations (Fig. 5).