- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DEFlow, a flow policy trained with SAC, based on SAC-flow. It replaces explicit Euler sampling with implicit Euler and solves all intermediate actions jointly as an equilibrium system. It uses implicit differentiation for policy gradients and derives the final-action density for SAC’s entropy term. They try to addresses a real issue: more denoising steps do not always improve flow-policy performance. Experiments on docking, manipulation, and MuJoCo show benefits mainly in docking and early learning on several manipulation tasks, at additional computational cost.
Critical strengths
-
The paper points out a real numerical issue in flow policies. Using implicit Euler to improve stability with few sampling steps is a reasonable idea and has some value.
-
The matched explicit-Euler comparison and step-count ablations provide useful evidence that implicit sampling can help on precision tasks. I also appreciate the numerical checks of action densities and gradients.
Critical weaknesses
-
The method adds substantial computational cost, with training taking about three times as long as SAC. The gains are mainly in docking and early learning on some manipulation tasks, with no consistent advantage in MuJoCo or final manipulation success. I am not convinced these gains justify the extra computation in practice.
-
I found the theoretical derivations difficult to follow. Although the paper discusses some limitations, it could explain more clearly when implicit sampling is useful and when a simpler explicit sampler would be sufficient.
Questions for authors
Could the authors further discuss when DEFlow is worth using? In particular, which types of tasks benefit enough from implicit sampling to justify its additional computational cost compared with explicit sampling?
4 comments
Gentle Reminder: Discussion Period Approaching Deadline
Dear Reviewer,
Hope you are doing well. As the discussion period is drawing to a close, we would really appreciate it if you could review our rebuttal soon. We have responded point-by-point to your feedback and added new results as requested.
If you think our responses adequately resolve your concerns, we kindly ask you to consider updating your evaluation. Feel free to raise any remaining questions, and we will reply promptly.
We sincerely appreciate your valuable time.
Best regards, Authors
Thank you for the detailed response. Theoretical derivations are not my main area. I remain uncertain whether the gains justify the additional computation across a broader range of tasks. I maintain my original decision.
Dear Reviewer,
Thank you for your follow-up and for considering our rebuttal. We appreciate the opportunity to clarify where DEFlow can be the most useful, particularly when learning diverse and precise behaviors matters. If you have any further questions, please feel free to reply here. We’d be happy to clarify.
Best regards,
The authors
Response to reviewer agWk
Thank you for recognizing the numerical motivation and the evidence from the matched explicit-Euler comparisons.
W1: "The method adds substantial computational cost, with training taking about three times as long as SAC"
We understand these concerns and agree that DEFlow's additional computational cost must be justified by a concrete benefits:
Higher successful coverage under precision constraints. We claim that DEFlow has higher learning ability by achieving the highest mean successful coverage on all four MultiGoal docking variants (Table 1). For example, on wide-navigation/loose-docking, its U4 is 0.671 ± 0.002, compared with 0.634 ± 0.021 for the strongest baseline, DIME. Across the four variants, the mean relative improvement over DIME is approximately 10.1%. We believe the main value lies in achieving higher successful coverage on challenging precision-constrained tasks, where stiff learned flow dynamics can make accurate sampling with few integration steps difficult. Moreover we believe that implicit flow policies have a promising future: further advances in this direction could reduce computational costs while preserving their ability to learn diverse and precise behaviors in challenging environments.
Higher sample efficiency. Across the five MuJoCo and five manipulation tasks, DEFlow achieves higher mean return AUC than SAC-Flow on 8/10 tasks and both SAC and DIME on 7/10 tasks each, suggesting improved sample efficiency under the evaluated training recipes (Tables 7–8). Additionally, at 100k training steps, DEFlow achieves returns of 25.7 ± 6.8 on PickCube and 40.5 ± 0.4 on PushCube, compared with 18.3 ± 1.8 and 39.4 ± 1.0 for SAC, 17.2 ± 3.1 and 31.9 ± 1.7 for SAC-Flow, and 20.2 ± 1.8 and 32.6 ± 8.3 for DIME, respectively. Averaging the DEFlow-to-baseline return ratios across both tasks and all three baselines yields a mean relative improvement of 28.5% at this checkpoint. Thus, we are simply more sample efficient and we believe that in some cases where environment is too expensive this benefit can make the learning more efficient.
W2: "it could explain more clearly when implicit sampling is useful and when a simpler explicit sampler would be sufficient"
First of all, thank you for the time and noticable care you put into reading our paper. Your comment highlights something we should explain more clearly: when these benefits justify the additional computational cost. Intuitively, we want a policy to preserve several distinct, precise ways to succeed. Standard SAC has limited flexibility because it uses a single Gaussian before tanh squashing as a policy. Flow policies such as SAC-Flow are more expressive, but a few explicit sampling steps may struggle to preserve narrow target modes. Section 3 illustrates how coarse integration can distort the sampled distribution and weaken the learning signal. This motivates our use of implicit sampling.
Therefore, answering your question, we see DEFlow as most useful when a task admits several distinct successful solutions, requiring precise actions, and preserving these alternatives matters. And Docking directly tests this combination of precision and successful diversity, where DEFlow leads the entire docking suite. Manipulation tasks also suggest benefits in settings involving contact and precise object control, with DEFlow achieving the highest mean return AUC on four of five tasks under the evaluated training recipes, which is also reached by higher sample efficiency. At the same time MuJoCo locomotion tasks provide a broader comparison and we just show that DEFlow remains broadly competitive.