acceptodds
Withdrawn from ICLR 2027

Improving Truncated First-Order Gradients for Flow Policies with Surrogate Objective

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Differentiable simulation offers a way to improve the sample efficiency of reinforcement learning for robotics. However, correctly utlizing the first-order information from the simulator for training vision-based flow-matching policies remains challenging. Visual observations may be nondifferentiable with respect to the environment state, and iterative action generation in flow matching makes end-to-end backpropagation costly. Truncating the observation gradient path makes the analytical policy gradient (APG) tractable, but the resulting truncated APG is biased. Likelihood-ratio policy gradients (LRPGs) avoid this differentiable requirement, but can suffer from high variance. We introduce Differentiable Flow Policy Optimization (DFPO), a method that combines these complementary signals to improve policy updates under incomplete differentiation. DFPO uses LRPG, estimated with a flow-matching surrogate objective, to improve the update direction of the truncated APG. On classical control problems, we show that the combined update has low variance and is better aligned with the true policy gradient. We further evaluate DFPO on robotic tasks with visual and state observations. In the vision-based setting, DFPO consistently outperforms baselines using either truncated APG or LRPG alone. In the state-based setting, it achieves performance comparable to methods using the full APG. These results demonstrate that DFPO, which combines the truncated APG with the LRPG can support effective reinforcement learning of vision-based flow-matching policies.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Flow Policy, Differentiable Simulator

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7Wf5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Differentiable Flow Policy Optimization (DFPO), an RL method for flow-matching policies trained in differentiable simulators. The core motivation is the use of flow-based or similar multi-step generation policies in vision-based settings in differentiable simulators. In such cases, the main issues are that backpropagating through rendering is usually infeasible or too expensive, so the observation path has to be truncated, which biases the analytical gradient. Backpropagating through the multi-step generation process of the policy is also costly. Likelihood-ratio gradients avoid both problems, but they have high variance and require the policy’s likelihood, which is intractable for flow policies.

DFPO adds a likelihood-ratio policy gradient (LRPG), estimated with the FPO++ surrogate, to a truncated gradient, and claims that it corrects the truncation bias. The paper also compares three ways of propagating the simulator’s action gradients through the flow policy. The method is evaluated on an LQR problem and six Genesis tasks with state and RGB observations.

Critical strengths
  • Using differentiable simulation for vision-based generative policies is a relevant problem.
  • The method is simple, and the comparison of the three propagation channels, including compute and memory profiles is useful for practitioners.
  • The evaluation covers both state-based and RGB-based settings.
Critical weaknesses

My main issue is that of limited scientific novelty. It seems to me that the DFPO combines several existing components, already studied in prior work. The outcome seems closer to a systems/engineering contribution rather than a fully-developed method. I also have some reservations about experiments and methodology, and some presentation concerns.

  • The method section is dense with equations but gives little intuition. For example, the three propagation channels and the FPO++ surrogate (Eqs. 16–18) are stated without explaining how they differ conceptually, and adjoint matching is only described in words. Can you please add intuitive explanations?
  • Upon doing a literature search, it seems to me that combining first-order and likelihood-ratio gradients has been studied before. Parmas et al. (2018) blend the two at each time step with inverse-variance weights. Suh et al. (2022) interpolate them in differentiable simulators with a variance-based weight. Son et al. (2023) incorporate analytical gradients into PPO with an adaptive weight. Onoda et al. (2026) switch between or inverse-variance-weight the two. Only Suh et al. is cited, and only in passing. What is the contribution beyond these methods, and why are they not used as baselines?
  • The motivation is not supported theoretically. While the authors acknowledge that the combined gradient is not unbiased, they state that the LRPG “corrects” the truncation bias. Adding an unbiased estimate to a biased one does not remove the bias; it only dilutes it, at the cost of higher variance. The two terms are also summed with fixed weights, while the BPTT term alone reaches very high gradient norms. If a global gradient-norm clip of 1.0 is applied, what is the effective contribution of each term? Why are they not weighted, e.g. by their variance as in prior work?
  • The LQR study does not test the proposed method. It uses Gaussian policies, so neither the flow policy nor the FPO++ surrogate is evaluated.
  • The baselines seem undertuned and the evaluation is weak. The first-order baselines use DFPO’s horizon (H = 8 for state, H = 16 for RGB) instead of the H = 32 used in the SHAC and RFO papers. Gradient-based methods also appear to be evaluated in a different simulator build than FPO and PPO. Are all experiments under comparable dynamics?
Questions for authors

Please see weaknesses.

2 comments

We thank the reviewer for the detailed feedback. We have substantially revised the method presentation and added theoretical analysis and experiments to address these concerns.

W1. Limited novelty and relation to prior work

We acknowledge that combining analytical and likelihood-ratio gradients has been explored in prior work. However, our main contributions extend beyond combining existing estimators, addressing two challenges specific to differentiable RL with flow policies:

(1) Sample-efficient differentiable RL for RGB-based flow policies. Non-differentiable rendering prevents full analytical gradient propagation, while iterative flow sampling makes policy differentiation computationally expensive. DFPO addresses both challenges, achieving substantially higher sample efficiency than LRPG-only and truncated APG-only methods.

(2) Direct One-step Transport (DOT). We introduce a new method for transporting action-level policy gradients to flow-policy parameters. Unlike BPTT, which differentiates through the entire flow trajectory, or Adjoint Matching (AM), which propagates adjoints through stored flow states, DOT transports the gradient through a single velocity-field evaluation, avoiding expensive multi-step differentiation. Across 12 tasks, BPTT, AM, and DOT achieve aggregate normalized scores of 10.3, 10.2, and 9.5, respectively. Meanwhile, DOT reduces computation time to approximately 1/5 (state-based) and 1/3 (RGB-based) compared with BPTT.

While Parmas et al., Suh et al., Son et al., and Onoda et al. investigate combining gradient estimators, our setting differs in that the analytical gradient is truncated by non-differentiable observations, while the LRPG is approximated through a flow-matching surrogate rather than an unbiased likelihood-ratio estimator. We will expand the related-work discussion to clarify these distinctions.

W2. Method intuition

We agree that the original presentation lacked intuition. We have rewritten Section 3.1 to explain the three gradient transport mechanisms conceptually.

W3. Bias, theoretical motivation, and weighting

We agree that simply adding an unbiased LRPG to a biased TAPG does not eliminate truncation bias. Moreover, the FPO++ surrogate used in our flow-policy setting is not necessarily unbiased, making conventional unbiased-estimator variance arguments insufficient.

To address this, we added Proposition 1 (Sec. 3.3), establishing a sufficient bound under which the combined gradient is an ascent direction and guarantees improvement with an appropriate step size. The bound explicitly characterizes how gradient agreement affects optimization: alignment reinforces the update, while conflicting gradients may compromise it.

Motivated by this analysis, we introduce cosine-similarity-based adaptive weighting. As shown in Fig. 6, our adaptive ratio substantially improves RGB-based performance AND performs comparably to fixed weighting in state-based tasks.

Regarding gradient magnitudes, BPTT's pre-clip norms frequently exceed the clipping threshold of 1.0, effectively normalizing its updates. This also highlights BPTT's sensitivity to gradient explosion and its limitation to one analytical-gradient update per rollout, whereas the LRPG surrogate supports repeated minibatch optimization.

W4. LQR study and flow-policy validation

We agree that the Gaussian-policy LQR study alone does not establish gradient quality for flow policies. We therefore added Fig. 4, directly evaluating update quality at flow-policy training checkpoints across all 12 state-based and RGB-based tasks, using the actual flow policy and lrpg surrogate.

DFPO outperforms both individual gradients in approximately 40% of cases, outperforms one in another 40%, and underperforms both in approximately 20%. This provides direct evidence that combining the signals frequently improves the optimization direction in our actual setting, complementing the controlled LQR analysis in Appendix H.

W5. Baseline tuning and simulator consistency

All methods are evaluated under identical simulator dynamics, physics parameters, and task configurations. The only implementation difference is that FPO and PPO do not construct or retain simulation gradient graphs, since they do not require differentiating through the simulator. This does not alter the forward dynamics.

We acknowledge that our shared rollout horizon differs from the original SHAC/RFO configurations and are rerunning these baselines with their original settings. These experiments are still ongoing due to the substantial GPU resources required for our adaptive-weighting experiments. Nevertheless, changing the horizon cannot eliminate the systematic bias introduced by truncating observation gradients, as further supported by our TAPG-only ablations (Fig. 5).

Thank you for your responses.

Novelty: Thanks for the clarifications. While this addresses my concerns partially, I still do not find the proposed algorithm convincingly novel. I will reserve my judgement for the reviewer discussion phase.

Intuition & Writing: The added section helps to make the paper clear. I would still suggest adding a notation section in the preliminaries and only labeling equations that are mention in the text. This would help make the paper look less busy.

Fig 4: Could you please provide some intuition on the situations where DFPO leads to an improvement, and those where it does not. It would help to contextualize why your gradient underperforms in some cases.

The rest of my concerns are addressed and I will maintain my score until the reviewer-AC discussions.

Reviewer_U4sXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at training vision-based flow policies with a differentiable simulator when you can't backprop through the observations (e.g. rendering). Detaching the observation gives a truncated, biased analytic gradient (TAPG), so they simply add an FPO++-style likelihood-ratio gradient on top and use the sum as the update (DFPO). They also compare three ways of pushing the action gradient into the flow policy: plain BPTT through the flow, adjoint matching, and a cheap action-level regression (DOT). Validation is an LQR study of update quality, plus six Genesis tasks (five DiffRL locomotion/control tasks and a ping-pong balance task) in both state and 64x64 RGB settings, along with a compute/memory profile of the three variants.

Critical strengths

The setting is practical and under-explored: differentiable sims are mostly used for state-based policies, and this is a simple, working recipe for image-based flow policies where truncated first-order methods basically don't learn. The comparison of the three propagation channels, including the cost profile, is also useful on its own and would carry over to other sources of action gradients like a learned critic.

Critical weaknesses

The core idea is thin and not really positioned against prior work. Mixing pathwise and likelihood-ratio gradients is an old idea (e.g. Suh et al. 2022, which the paper cites, proposes exactly an interpolated first/zeroth-order estimator; PIPPS-style total propagation is another one). Here the two gradients are just summed with fixed weights of 1, with no argument for why that's the right combination, and the paper itself admits the sum isn't unbiased. I'd want to see at least a comparison to or discussion of these combined estimators, and some reasoning about the weighting.

The results are good, but harder to read as "one method" than the paper suggests. At matched env steps (Fig. 4), the best DFPO variant learns clearly faster than FPO on most state-based tasks and on RGB hopper, halfcheetah and pingpong, though not on RGB humanoid. But the best variant flips between settings (AM in state, BPTT in RGB), and DOT is often no better than FPO. Table 1 reports FPO/PPO as the best of standard and 4x-budget runs, which makes final numbers hard to interpret. Fig. 1's aggregate bars put full-gradient baselines (summed over 6 tasks) next to methods summed over 12 tasks. The tasks are also small (MLP policies, 64x64 images).

Questions for authors
  1. BPTT's pre-clip gradient reaches the thousands (Table 2) with a global clip of 1.0, but the LRPG gradient norm isn't reported. How do the two terms balance in practice? Did you try tuning or normalizing them?
  2. How does DFPO compare to an existing combined estimator like Suh et al.'s interpolation, applied with the same flow policy?
  3. The LQR study uses Gaussian MLP policies rather than flow policies. Is there any evidence that the improved update direction holds for the flow policy and the FPO++ surrogate you actually use?
3 comments

We thank the reviewer for the detailed feedback. We have substantially expanded the theoretical analysis and experimental evaluation in the revised manuscript.

W1. Gradient weighting and theoretical justification

We agree that the original fixed weighting lacked sufficient justification. We have added Proposition 1 (Sec. 3.3), establishing a sufficient condition under which the combined gradient is an ascent direction. The analysis shows that gradient alignment directly affects update quality, motivating adaptive weighting based on cosine similarity.

We also investigated multiple weighting strategies. Fig. 6 shows that our adaptive weighting substantially improves RGB-based performance and achieves comparable performance in state-based tasks.

W2. Performance interpretation and evaluation

DOT substantially outperforms both FPO and PPO after introducing adaptive weighting. We do not claim that DOT outperforms BPTT or AM. Rather, BPTT and AM offer stronger performance in different settings, while DOT provides a substantially more computationally efficient alternative with competitive aggregate performance (9.5 v.s. ~10). All three share the same DFPO framework and differ only in how action-level gradients are transported to policy parameters. Additionally, we illustrate the full training curve with the updated adaptive weighting under both 1× and 3× budgets in Fig.3.

Q1. Gradient magnitudes and normalization

BPTT's pre-clip gradient norms (10²–10⁹, Table 2) consistently exceed the global clipping threshold of 1.0, effectively normalizing each update to a unit-norm direction. In comparison, LRPG gradients have median pre-clip norms of 0.9–2.0 across tasks and are clipped in 38–93% of minibatch updates. Thus, both signals contribute updates of comparable magnitude after clipping.

However, this also highlights a key limitation of BPTT: its gradients are prone to explosion, and retaining the full computation graph restricts BPTT to a single gradient update per rollout. In contrast, LRPG can reuse collected trajectories for multiple minibatch updates, providing more stable optimization without repeatedly backpropagating through the entire simulation graph.

We therefore investigated alternative weighting strategies, including gradient normalization, and introduced cosine-similarity-based adaptive weighting to account for variations in gradient agreement, which substantially improves the performance of DFPO AM and DOT on RGB-based tasks. The new ablation in Fig. 6 demonstrates that adaptive weighting substantially improves training stability and performance in RGB-based tasks.

Q2. Comparison with Suh et al.

Suh et al.'s α-order estimator provides an important precedent for combining first- and zeroth-order information. However, directly applying its estimator to our setting is nontrivial: the full observation gradient is unavailable due to non-differentiable rendering, and flow-policy likelihoods are not analytically tractable. Our method instead combines truncated action-level analytical gradients with an FPO++ surrogate, while separately addressing gradient transport through iterative flow sampling.

We have strengthened the discussion of this distinction. Our new fixed-versus-adaptive weighting ablation in Fig. 6 also examines how different combinations affect optimization, although it is not a direct implementation of Suh et al.'s estimator.

Q3. Gradient quality for flow policies

We agree that LQR experiments with Gaussian policies alone are insufficient to validate gradient quality in our actual flow-policy setting.

To address this, we added Fig. 4, directly measuring update quality at flow-policy training checkpoints across all 12 state-based and RGB-based tasks, using the actual DFPO gradients and FPO++ surrogate.

DFPO outperforms both individual gradients in approximately 40% of cases, outperforms one in another 40%, and underperforms both in approximately 20%. These results directly demonstrate that combining the two signals frequently improves the optimization direction for flow policies, complementing the controlled LQR analysis in Appendix H.

Thank you for the additional experiments. Figure 4 addresses my concern about evaluating gradient quality using the actual flow policy, and the adaptive weighting results also strengthen the empirical evidence.

My main concern remains the comparison with existing gradient combination methods. I understand that directly applying the estimator from Suh et al. is not straightforward, but comparing only against a fixed 1:1 weighting is not yet sufficient to establish an advantage over existing weighting methods that have been tuned in the same setting.

Please also clarify the order of clipping for AM and DOT: are the two gradients clipped separately before combination, or is their sum clipped? This is important for understanding their respective contributions.

Thank you for the follow-up and appreciate the acknowledgment of our additional experiments.

Regarding weighting baselines, we apologize for the incomplete comparison. A full evaluation across 12 tasks, 2 variants (AM and DOT), and 5 seeds requires several days of computation, and our limited GPU resources prevent us from completing additional baselines within the discussion period. We will report additional state-based comparisons, if available, before the discussion period ends.

We have also evaluated several other fixed weighting ratios, including 3:1, 1:3, and 1:9. Interestingly, a carefully selected fixed ratio can outperform our current adaptive weighting on individual tasks. We therefore acknowledge that our adaptive weighting strategy is still preliminary and do not claim it is optimal. Developing a more principled and consistently effective weighting scheme is an important direction for future work.

Regarding clipping, for both AM and DOT, the TAPG and LRPG gradients are clipped separately before being combined. This prevents an exploding gradient in one component from dominating or overwhelming the other, allowing both signals to contribute to the final update.

Reviewer_fQ57OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Differentiable Flow Policy Optimization (DFPO), an RL algorithm for differentiable simulation environments that combines truncated analytical policy gradient (TAPG) and likelihood-ratio policy gradient (LRPG) for policy updates. TAPG is biased but has low variance, while the LRPG is unbiased but has high variance. Combining these two complementary policy update signals for flow policy training can lead to better performance, mitigating the drawbacks of each individual component. Empirical results on both state-based and visual input control environments show improved performance compared to baselines based on either TAPG or LRPG. Further experiments are also provided (e.g., the comparison among BPTT, adjoint matching and action-level regression variants for using first-order information).

Critical strengths
  1. The proposed method is practical and shows empirical benefits.
  2. The paper is well-written and easy to follow.
Critical weaknesses
  1. The novelty of the proposed method is limited. The main contribution is a simple combination of the two policy update signals (TAPG and LRPG). Each signal has already been well-studied in the literature (such as in adjoint matching and FPO++). Further, similar techniques have already been proposed in prior work [1].
  2. The paper lacks further analysis. For example, the update in Equation 12 is still biased. Why does this bias not affect the training performance much? Are there gradient conflict issues when combining the two signals?

[1] Parmas, Paavo, et al. "PIPPS: Flexible model-based policy search robust to the curse of chaos." International Conference on Machine Learning. PMLR, 2018.

Questions for authors
  1. Why is only the comparison with TAPG provided in Figure 2? Is there any specific reason that the results for U(GDFPO)−U(GLRPG)\mathcal{U}\left(G_{\mathrm{DFPO}}\right)-\mathcal{U}\left(G_{\mathrm{LRPG}}\right) are not provided?
  2. In Table 2, why does the policy inference time differ much across different variants? For a fair comparison, the network size and flow timesteps should be the same for the three variants, so shouldn't the inference time be similar?
  3. Why are the weighting coefficients for gTAPGg_{\mathrm{TAPG}} and gLRPGg_{\mathrm{LRPG}} fixed at 1 in the experiments? Do the two components have norms in the same range?
1 comment

We thank the reviewer for the constructive feedback. We have strengthened the theoretical analysis and added extensive experiments in the revised manuscript.

W1. Novelty

While combining analytical and likelihood-ratio gradients has been explored in prior work (e.g., PIPPS), our contributions go beyond combining two existing estimators, focusing on two challenges specific to differentiable RL with flow policies:

(1) Sample-efficient differentiable RL for RGB-based flow policies. Non-differentiable rendering prevents full analytical gradient propagation, while iterative flow sampling makes policy differentiation computationally expensive. DFPO addresses both challenges, achieving substantially higher sample efficiency than LRPG-only and truncated APG-only methods.

(2) Direct One-step Transport (DOT). We introduce a new method for transporting action-level policy gradients to flow-policy parameters. Unlike BPTT, which differentiates through the entire flow trajectory, or Adjoint Matching (AM), which propagates adjoints through stored flow states, DOT transports the gradient through a single velocity-field evaluation, avoiding expensive multi-step differentiation. Across 12 tasks, BPTT, AM, and DOT achieve aggregate normalized scores of 10.3, 10.2, and 9.5, respectively. Meanwhile, DOT reduces computation time to approximately 1/5 (state-based) and 1/3 (RGB-based) compared with BPTT.

These contributions distinguish DFPO from PIPPS, which studies combining gradient estimators but does not address gradient transport through iterative flow policies or optimization with non-differentiable visual observations.

W2. Bias and gradient conflicts

We acknowledge that the combined update remains biased. To address this concern, we added Proposition 1 in Section 3.3, establishing a sufficient condition under which the combined gradient is an ascent direction and guarantees improvement with an appropriate step size. This analysis explicitly characterizes gradient conflicts: alignment between TAPG and LRPG reinforces the update, while disagreement may compromise its effectiveness.

Motivated by this result, we introduce adaptive weighting based on gradient cosine similarity to dynamically balance the two signals. We further validate the effectiveness of their combination through direct update-quality measurements (Fig. 4), controlled LQR experiments (Appendix H), and ablation on robot control tasks (Fig. 6).

Q1. Comparison against individual gradients

We have added comprehensive comparisons against both TAPG and LRPG.

Fig. 4 evaluates update quality across training checkpoints on all 12 tasks. DFPO achieves a better update direction than both individual gradients in approximately 40% of cases, outperforms one of them in another 40%, and underperforms both in approximately 20%. These results demonstrate that the combination frequently improves the optimization direction despite its remaining bias.

Additionally, Appendix H provides detailed LQR experiments comparing DFPO against each individual gradient.

Q2. Policy inference time

All three variants use identical network architectures and flow integration steps. The reported differences arise from the additional computation required by each gradient transport mechanism.

DOT performs standard forward inference without constructing a gradient graph. BPTT must retain the computation graph throughout flow integration for subsequent backpropagation. AM additionally requires storing intermediate flow states and computing adjoint information along the trajectory, introducing the highest rollout overhead.

Therefore, the reported inference times reflect the practical computational cost of each method during rollout, rather than pure forward-pass latency. We have clarified this distinction in the revised manuscript.

Q3. Weighting coefficients

We explored several weighting strategies, including gradient normalization and adaptive weighting. Based on these experiments, we now adopt a cosine-similarity-based adaptive weighting strategy (Sec. 3.3), which adjusts the TAPG contribution according to its agreement with LRPG.

As shown in Figure 6, adaptive weighting substantially improves performance in RGB-based tasks and performs comparably to fixed weighting in state-based tasks. Fixed weighting leads to early plateaus or failures on several RGB tasks, whereas adaptive weighting enables stable learning across all six.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.