- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoVE-PPO, a reinforcement learning fine-tuning framework for flow-matching policies based on action probability densities computed through the change-of-variables (CoV) formula. Unlike existing approaches that rely on surrogate likelihood ratios or treat intermediate denoising steps as additional decisions, CoVE-PPO directly estimates action log-densities by solving the flow ODE in reverse and integrating the divergence of the velocity field, enabling PPO-based policy optimization. To improve computational efficiency and numerical stability, the authors share random probes in the Hutchinson trace estimator between the old and new policies to reduce the variance of likelihood-ratio estimation. They also analyze the stiffness of flow ODEs and adjust the placement of numerical integration nodes based on the spectral properties of the velocity-field Jacobian to improve ODE solver stability. Furthermore, the authors introduce an Explicit-Diagonal Coupling (EDC) flow parameterization that allows the velocity-field divergence to be computed directly from network outputs without additional Jacobian differentiation, along with a Split-EDC variant that enables exact density computation for the discrete flow. Experiments on simulated robotic manipulation tasks from Robomimic and MimicGen compare CoVE-PPO with existing policy optimization methods, while ablation studies examine the effects of probe sharing, numerical integration strategies, and flow parameterization.
Critical strengths
-
The paper addresses a clear and meaningful problem: making PPO optimization with change-of-variables action densities practical for flow policies. Its motivation and presentation clearly connect the computational and numerical obstacles to the proposed solutions, including probe sharing, adjusted integration nodes, and EDC parameterization. This contributes to my positive assessment because the paper demonstrates a viable optimization approach and identifies concrete mechanisms that make it practical in the evaluated settings.
-
The key designs have explicit mathematical foundations and corresponding empirical support. The probe-sharing analysis relates log-ratio estimation noise to differences between the old and new policy Jacobians, while the EDC construction enables direct divergence computation. Targeted ablations complement evaluations across state- and vision-based manipulation tasks. This connection between analysis and experiments increases my confidence that the reported improvements have identifiable technical causes and that the findings can inform subsequent research.
Critical weaknesses
- The experiments provide limited evidence about the computational and expressivity trade-offs as model size, action dimensionality, and action horizon increase. The evaluation primarily uses single-task policies with relatively short action chunks. Reverse integration and trace-related differentiation may become more costly at larger scales, while EDC imposes structural constraints whose impact on more complex policy distributions remains unclear. This limits my assessment of the approach’s broader practical value, since its favorable performance–cost trade-off has yet to be demonstrated beyond the evaluated settings.
- The empirical support for some central mechanisms has limited coverage. The main probe-sharing ablation is conducted only on Square, and the numerical integration ablation uses the same task with a single seed. These results leave uncertainty about whether the observed effectiveness of single-probe estimation and adjusted integration nodes is robust across tasks and training seeds. This matters to my overall assessment because these mechanisms underpin the practicality of direct CoV-based PPO; limited evidence of their robustness reduces confidence in the generality of the paper’s central findings.
Questions for authors
-
Appendix B reports computational costs at matched network sizes. How do update time and peak memory scale with action dimensionality, action horizon, and network size for the general and EDC variants, based on available measurements or complexity analysis?
-
Beyond matching initial BC success rates, what evidence clarifies EDC’s expressivity–efficiency trade-off, particularly the impact of the diagonal bound α on BC fitting and final RL performance?
-
Beyond the ablations in Figures 3–4 and the diagnostics in Appendix E, what evidence supports the continued reliability of single-probe estimation and the pretrained integration mesh as the policy changes during PPO fine-tuning?
3 comments
As the discussion period ends today, we'd like to check whether our revisions addressed your concerns. If any remain, we are happy to clarify. Thank you again for your time.
Thank you for the response. My main concern remains the trade off between policy expressivity and computational cost. The general variant still requires backpropagation through a reverse ODE solve, and EDC reduces that cost by restricting the velocity field, so I’m keeping my rating at 2.
Thank you for the review. We address each point below; the General Response (GR) has the full details.
Ablation robustness (W2).
See GR-B.
Reliability during fine-tuning (Q3). We measured both mechanisms during training (GR-A, App. H and F.2). Every 5 PPO iterations we compared the log-ratio against an exact trace reference and re-measured stiffness on the current policy.
This matches the analysis in Section 3.3. Shared-probe noise depends on how much the Jacobian changes within one PPO update, which the trust region keeps small at every point in training, not on the total change since BC. See GR-A.
Computational scaling (W1, Q1). We measure time and memory against action dimension and network width (GR-C, App. I).
Our chunk lengths of 4–8 match the settings of the baselines we compare against. We leave VLA-scale fine-tuning for future work, and the revised Limitations reflect this.
EDC expressivity and α (Q2). In Appendix E, we show BC training curves for a sweep of α compared to a general flow. More restrictive bounds (lower α) can reduce BC success, and less restrictive bounds lead to a stiffer flow. Stiffness, hμ, scales linearly with α. For our work, we chose α such that the flow would not be stiff, and that still maintained BC performance. Stiffer EDC flows with a higher bound could potentially benefit from the same stiffness mesh as the general field, however, we found our choice of α to be effective for both BC and RL fine-tuning.
We also show RL curves for the same sweep of α. We train on Square from performance matched BC checkpoints, and compare training curves and final performance. We observe that the stiffest flow, or highest α, has lower final performance. The least stiff flow starts lower, but eventually reaches performance near more moderate values of α.
We hope these additions address your concerns about robustness and scaling, and we are happy to answer further questions.