acceptodds
Under review as a conference paper at ICLR 2027

DEFlow: Deep Equilibrium Flow Maximum-Entropy Reinforcement Learning

Abstract

Expressive flow policies can represent multiple high-value actions, but sampling them with few integration steps while retaining tractable action densities remains challenging. We introduce DEFlow, a deep equilibrium flow policy that replaces explicit Euler sampling with implicit Euler integration and solves all intermediate actions jointly as an equilibrium system. We combine implicit differentiation with a discrete change-of-variables formula for the final-action density, enabling off-policy training within soft actor-critic without backpropagating through solver iterations. DEFlow achieves an average relative improvement of approximately 10% in mean successful coverage over the strongest baseline across four Multi-Goal docking variants. It also achieves the highest mean time-averaged return on four of five manipulation tasks, while remaining broadly competitive across five MuJoCo tasks. These results support implicit flow policies for tasks requiring both diversity and precision, at the cost of additional nonlinear and linear solves.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Flow-based policy, Soft actor critic, Deep Equilibrium Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_agWkOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DEFlow, a flow policy trained with SAC, based on SAC-flow. It replaces explicit Euler sampling with implicit Euler and solves all intermediate actions jointly as an equilibrium system. It uses implicit differentiation for policy gradients and derives the final-action density for SAC’s entropy term. They try to addresses a real issue: more denoising steps do not always improve flow-policy performance. Experiments on docking, manipulation, and MuJoCo show benefits mainly in docking and early learning on several manipulation tasks, at additional computational cost.

Critical strengths
  1. The paper points out a real numerical issue in flow policies. Using implicit Euler to improve stability with few sampling steps is a reasonable idea and has some value.

  2. The matched explicit-Euler comparison and step-count ablations provide useful evidence that implicit sampling can help on precision tasks. I also appreciate the numerical checks of action densities and gradients.

Critical weaknesses
  1. The method adds substantial computational cost, with training taking about three times as long as SAC. The gains are mainly in docking and early learning on some manipulation tasks, with no consistent advantage in MuJoCo or final manipulation success. I am not convinced these gains justify the extra computation in practice.

  2. I found the theoretical derivations difficult to follow. Although the paper discusses some limitations, it could explain more clearly when implicit sampling is useful and when a simpler explicit sampler would be sufficient.

Questions for authors

Could the authors further discuss when DEFlow is worth using? In particular, which types of tasks benefit enough from implicit sampling to justify its additional computational cost compared with explicit sampling?

4 comments
AuthorsOpenReview ↗

Response to reviewer agWk

Thank you for recognizing the numerical motivation and the evidence from the matched explicit-Euler comparisons.

W1: "The method adds substantial computational cost, with training taking about three times as long as SAC"

We understand these concerns and agree that DEFlow's additional computational cost must be justified by a concrete benefits:

  1. Higher successful coverage under precision constraints. We claim that DEFlow has higher learning ability by achieving the highest mean successful coverage on all four MultiGoal docking variants (Table 1). For example, on wide-navigation/loose-docking, its U4 is 0.671 ± 0.002, compared with 0.634 ± 0.021 for the strongest baseline, DIME. Across the four variants, the mean relative improvement over DIME is approximately 10.1%. We believe the main value lies in achieving higher successful coverage on challenging precision-constrained tasks, where stiff learned flow dynamics can make accurate sampling with few integration steps difficult. Moreover we believe that implicit flow policies have a promising future: further advances in this direction could reduce computational costs while preserving their ability to learn diverse and precise behaviors in challenging environments.

  2. Higher sample efficiency. Across the five MuJoCo and five manipulation tasks, DEFlow achieves higher mean return AUC than SAC-Flow on 8/10 tasks and both SAC and DIME on 7/10 tasks each, suggesting improved sample efficiency under the evaluated training recipes (Tables 7–8). Additionally, at 100k training steps, DEFlow achieves returns of 25.7 ± 6.8 on PickCube and 40.5 ± 0.4 on PushCube, compared with 18.3 ± 1.8 and 39.4 ± 1.0 for SAC, 17.2 ± 3.1 and 31.9 ± 1.7 for SAC-Flow, and 20.2 ± 1.8 and 32.6 ± 8.3 for DIME, respectively. Averaging the DEFlow-to-baseline return ratios across both tasks and all three baselines yields a mean relative improvement of 28.5% at this checkpoint. Thus, we are simply more sample efficient and we believe that in some cases where environment is too expensive this benefit can make the learning more efficient.

W2: "it could explain more clearly when implicit sampling is useful and when a simpler explicit sampler would be sufficient"

First of all, thank you for the time and noticable care you put into reading our paper. Your comment highlights something we should explain more clearly: when these benefits justify the additional computational cost. Intuitively, we want a policy to preserve several distinct, precise ways to succeed. Standard SAC has limited flexibility because it uses a single Gaussian before tanh squashing as a policy. Flow policies such as SAC-Flow are more expressive, but a few explicit sampling steps may struggle to preserve narrow target modes. Section 3 illustrates how coarse integration can distort the sampled distribution and weaken the learning signal. This motivates our use of implicit sampling.

Therefore, answering your question, we see DEFlow as most useful when a task admits several distinct successful solutions, requiring precise actions, and preserving these alternatives matters. And Docking directly tests this combination of precision and successful diversity, where DEFlow leads the entire docking suite. Manipulation tasks also suggest benefits in settings involving contact and precise object control, with DEFlow achieving the highest mean return AUC on four of five tasks under the evaluated training recipes, which is also reached by higher sample efficiency. At the same time MuJoCo locomotion tasks provide a broader comparison and we just show that DEFlow remains broadly competitive.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Approaching Deadline

Dear Reviewer,

Hope you are doing well. As the discussion period is drawing to a close, we would really appreciate it if you could review our rebuttal soon. We have responded point-by-point to your feedback and added new results as requested.

If you think our responses adequately resolve your concerns, we kindly ask you to consider updating your evaluation. Feel free to raise any remaining questions, and we will reply promptly.

We sincerely appreciate your valuable time.

Best regards, Authors

Thank you for the detailed response. Theoretical derivations are not my main area. I remain uncertain whether the gains justify the additional computation across a broader range of tasks. I maintain my original decision.

Dear Reviewer,

Thank you for your follow-up and for considering our rebuttal. We appreciate the opportunity to clarify where DEFlow can be the most useful, particularly when learning diverse and precise behaviors matters. If you have any further questions, please feel free to reply here. We’d be happy to clarify.

Best regards,
The authors

Reviewer_dkt6OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DEFlow, a deep-equilibrium flow policy for maximum-entropy reinforcement learning. The authors identify limitations of explicit Euler integration in flow-based policies, particularly when only a small number of integration steps are available and the target action distribution contains narrow or multiple modes. To address this issue, DEFlow replaces explicit Euler integration with implicit Euler updates and formulates the intermediate action trajectory as a joint deep-equilibrium system. The forward trajectory is obtained through a numerical root solver, while policy gradients are computed using implicit differentiation without backpropagating through the solver iterations. The authors further derive a discrete change-of-variables expression for the final-action log-density, enabling DEFlow to be trained within the Soft Actor-Critic framework. The proposed approach is evaluated on MultiGoal docking, MuJoCo locomotion, and ManiSkill/MetaWorld manipulation tasks. DEFlow achieves the highest mean successful coverage across four MultiGoal variants and the highest time-averaged return on four of five manipulation tasks. The paper also includes numerical analyses and comparisons against explicit Euler integration, demonstrating advantages under certain precision-constrained settings. Overall, the paper presents an interesting numerical approach to training expressive flow policies for maximum-entropy reinforcement learning.

Critical strengths
  1. A technically interesting formulation of implicit flow policies for maximum-entropy reinforcement learning. A major strength of this paper is its integration of implicit numerical methods with expressive flow-based policies. The authors formulate implicit Euler integration as a joint equilibrium system and derive the corresponding policy gradients through implicit differentiation. Furthermore, the discrete change-of-variables formulation enables direct evaluation of the final-action log-density, allowing the resulting policy to be trained within the maximum-entropy SAC framework. This is a meaningful technical contribution because obtaining tractable action densities and stable policy gradients remains challenging for multi-step generative policies. The mathematical derivations are generally well presented, and the paper discusses the assumptions required for their validity.
  2. Strong evidence of the benefits of implicit integration for precision-constrained multimodal action distributions. The MultiGoal docking experiments provide encouraging evidence that implicit integration can improve the ability of flow policies to represent multiple precise action modes. DEFlow achieves the highest mean successful coverage across all four evaluated variants, with particularly notable improvements under tight docking constraints. The matched explicit-Euler comparison and step-count analyses further support the argument that numerical integration choices can substantially influence the quality of generated action distributions.
Critical weaknesses
  1. Insufficient ablation to isolate the contribution of the joint deep-equilibrium formulation from implicit Euler integration. Although the paper introduces both implicit Euler integration and a joint deep-equilibrium formulation, the experimental comparisons do not clearly establish the necessity of solving all intermediate actions jointly. The principal controlled comparison replaces implicit Euler with explicit Euler, demonstrating the benefits of implicit integration. However, this experiment does not distinguish the benefits of implicit Euler discretization from those of the proposed joint equilibrium solver. Since the implicit Euler equations have a sequential structure, an alternative implementation could solve each implicit step independently, potentially using a sequential root solver or Newton iterations. Furthermore, the proposed method introduces additional computational costs, including nonlinear forward solving and dense linear backward solving. Without a comparison against sequential implicit Euler under matched computational budgets, it remains unclear whether the joint equilibrium formulation offers advantages beyond those provided by implicit integration itself.
  2. Limited practical performance improvements relative to the computational and numerical costs. Although DEFlow performs well on the precision-constrained MultiGoal docking benchmarks, its advantages are considerably less consistent on standard continuous-control and robotic manipulation tasks. On MuJoCo, DEFlow does not consistently outperform existing methods, and standard SAC achieves a higher aggregate return across the five evaluated tasks. On manipulation benchmarks, DEFlow achieves strong time-averaged returns but does not consistently outperform alternative methods in final task success. For example, SAC achieves substantially higher final success on PegInsertSide, while DIME achieves higher success on PickCube. Several manipulation tasks also exhibit low absolute success rates for DEFlow despite improvements in return. These limitations are particularly relevant given the additional computational requirements of the proposed method. The authors report approximately 40% lower training throughput than SAC-Flow and approximately three times the training time of standard SAC.
Questions for authors
  • How reliable are the policy gradients and action densities when the equilibrium solver does not converge?

The theoretical derivations assume an exact equilibrium solution, while the practical implementation uses finite tolerances, regularization, and clipping. The numerical audit reports that a substantial fraction of evaluation calls reach the solver iteration limit, and some tasks exhibit considerable discrepancies in actor gradients. Could the authors clarify how frequently solver non-convergence occurs during training and quantify its effect on the SAC objective? In particular, how sensitive are policy learning and performance to the solver tolerance, maximum iterations, and Jacobian regularization?

5 comments
AuthorsOpenReview ↗

Response to reviewer dkt6 (1/2)

Thank you for highlighting both the technical contribution of combining implicit flow policies with maximum-entropy learning and the evidence for successful diversity under precision constraints. Your request for a sequential implicit baseline identifies an important attribution gap.

W1: “Insufficient ablation to isolate the contribution of the joint deep-equilibrium formulation from implicit Euler integration.”

Equations (7) and (8) clarify this distinction: summing the sequential implicit-Euler updates gives the joint equations, while subtracting consecutive joint equations recovers the sequential updates. They therefore describe the same exact trajectories and endpoint policy when the same solution branch is selected. Our choice of a joint formulation with Anderson acceleration was motivated by its computational structure: each solver iteration evaluates the velocity field across all trajectory points in a batch and updates the entire trajectory together, without requiring each implicit step to converge before proceeding to the next. This provides an opportunity to exploit parallel computation.

To address the gap you identified, we have extended the experiments to compare:

  • Joint Anderson (DEFlow baseline): The original DEFlow solver, which applies Anderson acceleration to the joint fixed-point system over the entire implicit-Euler trajectory.
  • Sequential Anderson: A step-by-step solver that applies the same Anderson acceleration algorithm to the local fixed-point equation at each implicit-Euler step.

The comparison uses three training seeds on wide-navigation/tight-docking MultiGoal, with 200k transitions and 1,000 stochastic evaluation episodes with a rerun of our original work.

MethodSeedsU4 ↑Success (%) ↑A2 forward solve, ms ↓N3 forward solve, ms↓
Joint Anderson (DEFlow baseline)30.6652 ± 0.003396.17 ± 1.1019.55 (1.00×)19.81 (1.00×)
Sequential Anderson30.6197 ± 0.056892.13 ± 5.4166.82 (3.42×)66.63 (3.36×)

For both solvers, we measure the success and U4 (Eq. 17) forward-solve latency using the same fixed model weights and inputs on the same GPU, reporting the median of three synchronized runs after warmup. Joint Anderson achieved higher mean policy performance and lower observed variability, while sequential Anderson required 3.36–3.42× the forward-solve time. Overall, these results support our choice of joint Anderson as a practical solver for DEFlow in the tested setting. Compared with sequential Anderson, it achieved higher mean goal coverage and success, with lower variation across the three training seeds, while requiring approximately 3.4× less forward-solve time. Comparing joint and sequential Anderson helps isolate the effect of solving the trajectory jointly, and the observed results support this design choice in DEFlow.

W2: “Limited practical performance improvements relative to the computational and numerical costs.”

We understand these concerns and agree that DEFlow's additional computational cost must be justified by a concrete benefits:

  1. Higher successful coverage under precision constraints. We claim that DEFlow has higher learning ability by achieving the highest mean successful coverage on all four MultiGoal docking variants (Table 1). For example, on wide-navigation/loose-docking, its U4 is 0.671 ± 0.002, compared with 0.634 ± 0.021 for the strongest baseline, DIME. Across the four variants, the mean relative improvement over DIME is approximately 10.1%. We believe the main value lies in achieving higher successful coverage on challenging precision-constrained tasks, where stiff learned flow dynamics can make accurate sampling with few integration steps difficult. Moreover we believe that implicit flow policies have a promising future: further advances in this direction could reduce computational costs while preserving their ability to learn diverse and precise behaviors in challenging environments.

  2. Higher sample efficiency. Across the five MuJoCo and five manipulation tasks, DEFlow achieves higher mean return AUC than SAC-Flow on 8/10 tasks and both SAC and DIME on 7/10 tasks each, suggesting improved sample efficiency under the evaluated training recipes (Tables 7–8). Additionally, at 100k training steps, DEFlow achieves returns of 25.7 ± 6.8 on PickCube and 40.5 ± 0.4 on PushCube, compared with 18.3 ± 1.8 and 39.4 ± 1.0 for SAC, 17.2 ± 3.1 and 31.9 ± 1.7 for SAC-Flow, and 20.2 ± 1.8 and 32.6 ± 8.3 for DIME, respectively. Averaging the DEFlow-to-baseline return ratios across both tasks and all three baselines yields a mean relative improvement of 28.5% at this checkpoint. Thus, we are simply more sample efficient and we believe that in some cases where environment is too expensive this benefit can make the learning more efficient.

AuthorsOpenReview ↗

Response to reviewer dkt6 (2/2)

Q: “How reliable are the policy gradients and action densities when the equilibrium solver does not converge?”

Thank you for raising this concern. To answer this concern, we conducted ablation studies on Andweson setting (including tolerance and cap) to show how different the DEFlow performs on docking task with different tolerance and cap settings.

Anderson setting (tolerance / cap)Forward time (ms/batch)Log-density (nat), MedianLog-density (nat), P95Gradient (%), MedianGradient (%), P95Success (%)Median stopping stepsCap reached (batches)
1e-5 / 107.631.47e-49.19e-220731550.63 ± 6.86109/9
1e-5 / 2518.767.77e-77.54e-38.0259.096.17 ± 1.10259/9
1e-5 / 10045.234.09e-74.97e-60.3952.5796.60 ± 1.42613/9
1e-5 / 50044.734.10e-74.39e-60.3952.5798.20 ± 0.70611/9
1e-6 / 10064.223.95e-72.70e-60.03430.094696.53 ± 1.60833/9

Our experiments show that even when the equilibrium has not fully converged, the effect on the policy gradient and the action density is already small, and so is the effect on final performance. As is demonstrated in the table, at 1e−5/251\mathrm{e}{-5}/25, all nine batches still hit the iteration cap. Against the same tolerance once the median solve reaches 61 steps, the median log-density discrepancy only moves from 4.09×10−74.09\times10^{-7} to 7.77×10−77.77\times10^{-7} nat, and the 95th percentile of 7.54×10−37.54\times10^{-3} nat rescales the density by a factor of about 1.008. The median relative gradient error has already fallen from 207% at cap 10 to 8.02%, and closing the rest of that gap, down to 0.395%, leaves success at 96.17 ± 1.10%, 96.60 ± 1.42%, and 98.20±0.70%. Tolerance 1e−61\mathrm{e}{-6} gives 96.53±1.60%. Success reaches this high level at the same moderate budget. Those longer solves also reduce the log-density and gradient discrepancies, so the added steps tighten the numerics. The docking success is already obtained at cap 2525, at roughly one-third to one-half the forward time of the longer settings. This is why we keep 1e−5/251e−5/25: it secures the task result, and further steps trade a steady increase in solve time for smaller numerical residuals.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Approaching Deadline

Dear Reviewer,

Hope you are doing well. As the discussion period is drawing to a close, we would really appreciate it if you could review our rebuttal soon. We have responded point-by-point to your feedback and added new results as requested.

If you think our responses adequately resolve your concerns, we kindly ask you to consider updating your evaluation. Feel free to raise any remaining questions, and we will reply promptly.

We sincerely appreciate your valuable time.

Best regards, Authors

Reviewer_VrBXOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DEFlow replaces explicit flow-policy sampling with implicit Euler integration and jointly solves the intermediate actions as an equilibrium system. It combines implicit differentiation with a final-action change-of-variables density, exact under the stated equilibrium assumptions, to train an expressive policy within SAC. Results support improved successful diversity on docking and earlier reward learning on several manipulation tasks, with matched explicit-flow controls and numerical audits. Benefits come with additional solver cost and approximation error; MuJoCo performance is broadly competitive rather than consistently superior.

Critical strengths
  1. A coherent way to combine implicit flow sampling with maximum-entropy learning. The same implicit-step structure yields both policy sensitivities and a final-action density expression. This provides a principled alternative to explicit flow rollouts without requiring backpropagation through solver iterations or a path-entropy surrogate.
  2. Targeted evidence for successful diversity under precision constraints. The matched explicit-flow control and docking metric test whether policies preserve multiple successful strategies, not merely high return on a preferred route. DEFlow's higher mean coverage across all four variants makes the numerical design empirically relevant, while the disclosed costs and negative results help define its scope.
Critical weaknesses
  1. The practical density and gradient approximation needs stronger validation during learning. The exact formulas describe a converged implicit solution, whereas training uses finite solves, shifted matrices, and clipping. The numerical audit is useful and reassuring on four tasks, but PushWall's P95 gradient discrepancy remains 50.3% even with tighter evaluation. Comparing against an implicit reference does not fully validate the likelihood of the actual finite-iteration sampler, and unchanged final-policy success does not establish that these approximations are harmless during training.
  2. The practical advantage remains limited by compute cost and statistical uncertainty. DEFlow takes about three times as long to train as SAC and has lower throughput than SAC-Flow, but its gains are evaluated at fixed interaction budgets rather than equal elapsed time. Tight-docking improvements and the PushWall return estimate are uncertain across seeds, and the depth-robustness illustration uses one surviving paired seed.
Questions for authors
  1. How large are the discrepancies between the actual finite-solver sensitivities and the implicit gradients at representative training checkpoints, especially on PushWall? A targeted tighter-solver training comparison would help determine whether the remaining errors change learning, rather than only final evaluation.
  2. Do the docking or manipulation advantages persist under equal wall-clock budgets against SAC-Flow and DIME? Please clarify hardware and field-evaluation costs; return-versus-time curves from existing logs, if available, would directly address this trade-off.
5 comments
AuthorsOpenReview ↗

Response to reviewer VrBX (1/3)

Thank you for the careful and constructive review. We are glad that the numerical audit was useful and reassuring on four tasks and that the presentation was clear. Below we quote each of your points and answer it in turn, with new experiments for each.

W1 and Q1: “The practical density and gradient approximation needs stronger validation during learning.” / “A targeted tighter-solver training comparison would help determine whether the remaining errors change learning, rather than only final evaluation.”

In short, on docking the finite-iteration error at the solver setting of the paper's docking and manipulation runs, tolerance 1e-5 and cap 25, has only a small effect on learning. You are right that exactness describes the converged solution, as Sec. 4 and Sec. 6 state, and that unchanged final success does not by itself show that the approximations are harmless. Appendix C.1 notes that errors during learning and the likelihood of the finite-iteration sampler remain open. Thank you for suggesting a tighter-solver training comparison. We trained DEFlow with each solver setting on wide-navigation/tight-docking MultiGoal, with three seeds per setting and 200k transitions, and compared the settings on nine identical batches saved during training at about 25k, 100k and 200k transitions in each seed.

Anderson setting (tolerance / cap)Forward time (ms/batch)Log-density (nat), MedianLog-density (nat), P95Gradient (%), MedianGradient (%), P95Success (%)Median stopping stepsCap reached (batches)
1e-5 / 107.631.47e-49.19e-220731550.63 ± 6.86109/9
1e-5 / 2518.767.77e-77.54e-38.0259.096.17 ± 1.10259/9
1e-5 / 10045.234.09e-74.97e-60.3952.5796.60 ± 1.42613/9
1e-5 / 50044.734.10e-74.39e-60.3952.5798.20 ± 0.70611/9
1e-6 / 10064.223.95e-72.70e-60.03430.094696.53 ± 1.60833/9

Our experiments show that even when the equilibrium has not fully converged, the effect on the policy gradient and the action density is already small, and so is the effect on final performance. As is demonstrated in the table, at 1e−5/251\mathrm{e}{-5}/25, all nine batches still hit the iteration cap. Against the same tolerance once the median solve reaches 61 steps, the median log-density discrepancy only moves from 4.09×10−74.09\times10^{-7} to 7.77×10−77.77\times10^{-7} nat, and the 95th percentile of 7.54×10−37.54\times10^{-3} nat rescales the density by a factor of about 1.008. The median relative gradient error has already fallen from 207% at cap 10 to 8.02%, and closing the rest of that gap, down to 0.395%, leaves success at 96.17 ± 1.10%, 96.60 ± 1.42%, and 98.20±0.70%. Tolerance 1e−61\mathrm{e}{-6} gives 96.53±1.60%. Success reaches this high level at the same moderate budget. Those longer solves also reduce the log-density and gradient discrepancies, so the added steps tighten the numerics. The docking success is already obtained at cap 2525, at roughly one-third to one-half the forward time of the longer settings. This is why we keep 1e−5/251e−5/25: it secures the task result, and further steps trade a steady increase in solve time for smaller numerical residuals.

AuthorsOpenReview ↗

Response to reviewer VrBX (2/3)

W1: “Comparing against an implicit reference does not fully validate the likelihood of the actual finite-iteration sampler …”

We extended our numerical audit to directly examine the density calculation for the finite-iteration sampler used in training. This analysis complements the training comparisons presented above. Our density calculation follows the discrete change-of-variables formula in Sec. 4 and Appendix A.3: with ∂ak/∂ak−1=(I−Δt Jk)−1\partial a_k/\partial a_{k-1}=(I-\Delta t\,J_k)^{-1}, the density is log⁡πθ(aenv∣s)=log⁡p(z)+∑k=1Tlog⁡∣det⁡(I−Δt Jk)∣−∑ilog⁡[ci(1−tanh⁡2aT,i)]\log\pi_\theta(a_{\mathrm{env}}\mid s)=\log p(z)+\sum_{k=1}^{T}\log|\det(I-\Delta t\,J_k)|-\sum_i\log[c_i(1-\tanh^2 a_{T,i})], which is exact for the selected solution of the implicit-Euler equations. For the 25-iteration sampler actually used, we computed the exact log-density from the Jacobian of the unrolled iterations on 9,216 docking samples. The median absolute difference to the density used in training is 5.8e-6 nat, the P95 is 0.75 nat and the maximum is 15.5 nat. The difference follows the equation residual of the returned iterate, with a Spearman correlation of 0.70: for the 86% of samples with residual below 1e-4 the median difference is 5.4e-6 nat, and 97% of the samples that differ by more than 0.1 nat lie in the top residual decile. Our density calculation typically agrees closely with the numerical check on docking samples. Larger differences mainly occur when the solver has not yet solved the implicit equations accurately. These findings apply to the docking samples tested here. We examine the larger gradient discrepancies on PushWall separately in the below section.

W1 and Q1: “PushWall's P95 gradient discrepancy remains 50.3% even with tighter evaluation.” / “How large are the discrepancies … at representative training checkpoints, especially on PushWall?”

On docking, the discrepancies at representative training checkpoints are in the table above. We agree that PushWall remains a limitation of the finite-iteration approximation, Sec. 6 and Appendix C.1 already report it. PushWall is also the hardest task in our suite: at 500k steps no method solves it reliably, with sampled ever-success of at most 10.3%. DEFlow is the only method whose mean return rises substantially, to 457.8, but through one of three seeds and without solving the task. In the paper runs the solver stopped at its 25-iteration cap on 98–99% of logged training updates, and the tighter solver lowers the final-checkpoint P95 gradient discrepancy from 96.1% to 50.3%. We rerun the model and at the gathered checkpoints, the actor-gradient direction agrees only partly with an independent fp64 implicit reference, with a median cosine similarity of 0.39–0.53 across three seeds and 0.91–0.95 with the tighter solver. As you point out, gradient accuracy during training remains a limitation on PushWall.

W2 and Q2: “… its gains are evaluated at fixed interaction budgets rather than equal elapsed time.” / “Do the docking or manipulation advantages persist under equal wall-clock budgets against SAC-Flow and DIME?”

We measured training time and effective throughput and reported approximate cost comparisons in Sec. 6 to make the computational trade-off explicit. These measurements suggest that DEFlow’s training cost is of the same order as DIME’s and higher than that of SAC and SAC-Flow under our experimental conditions. Our existing runs used different GPU models and different numbers of concurrent jobs per GPU, so elapsed times reflect both algorithmic cost and differences in hardware and workload. A controlled equal-wall-clock comparison would require additional runs under matched conditions, which we could not conduct with the GPU resources available to us. Our performance comparisons therefore use equal environment-step budgets, and we do not claim an advantage at equal wall-clock time. Additionally, at fixed environment-step budgets, DEFlow achieves the highest mean successful coverage across all four docking variants, with an average relative improvement of about 10% over DIME (Table 1). Across the ten MuJoCo and manipulation tasks, it also achieves higher mean return AUC than SAC-Flow on 8/10 tasks and SAC and DIME on 7/10 tasks each under the evaluated training recipes (Tables 7–8). These gains may justify the additional computation when environment interactions are costly or when preserving diverse, precise, and successful behaviors is especially important.

AuthorsOpenReview ↗

Response to reviewer VrBX (3/3)

Q2: “Please clarify hardware and field-evaluation costs.”

Forward-solve times were measured on identical saved batches on one RTX 5090, exclusive for the solver-setting table and with no other job on the GPU for the joint-versus-sequential measurement. The docking study trained on RTX 4090 and RTX 5090 GPUs with up to two jobs per GPU, and the manipulation benchmark on RTX 4080 SUPER and RTX 5090 hosts, with several jobs per GPU in some campaigns. One action sample needs 1 network evaluation for SAC, 4 sequential evaluations for SAC-Flow, 5 for QSM and 16 for DIME, against at most 25 batched field evaluations for DEFlow, the solver cap of the paper's docking and manipulation runs, plus one Jacobian evaluation when the log-density is needed and one dense adjoint solve per actor update. DEFlow therefore costs more per sample than SAC, SAC-Flow and QSM and a comparable amount to DIME, in line with Sec. 6.

W2: “Tight-docking improvements … are uncertain across seeds”

More seeds support the docking advantage. We added seeds, the table reports successful coverage U4, whose maximum is 0.684. For tight docking, the three additional DEFlow values are the joint-Anderson runs of the 1e-5/25 setting above, and we ran three new DIME seeds.

TaskMethodPaper seeds (3), mean ± SDNew seeds (3)All six, mean ± SDDEFlow − DIME (Welch 95% CI)
Wide navigation / loose dockingDEFlow0.671 ± 0.0020.668, 0.675, 0.6680.671 ± 0.003+0.041 [+0.019, +0.064]
Wide navigation / loose dockingDIME0.634 ± 0.0210.637, 0.642, 0.5940.629 ± 0.022
Wide navigation / tight dockingDEFlow0.589 ± 0.1280.668, 0.661, 0.6660.627 ± 0.091+0.127 [+0.031, +0.223]
Wide navigation / tight dockingDIME0.502 ± 0.0110.494, 0.507, 0.4950.500 ± 0.008

On loose docking, every DEFlow seed exceeds every DIME seed. On tight docking, five of six DEFlow seeds reach 0.66–0.67 while all six DIME seeds lie between 0.494 and 0.514; the sixth DEFlow seed, 0.441, is the low paper seed, and we report it. The relative improvement of DEFlow over DIME on tight docking is 25% over six seeds, against 17% over the three paper seeds. The new seeds are re-runs on RTX 5090 GPUs, except the tight-docking DEFlow values, which come from the solver study on an RTX 4090.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal phase

Dear Area Chair,

Thank you for taking the time to consider our submission. We’d like to summarize what the reviewers concerned about and how our rebuttal addressed it.

All the reviewers (VrBX, agWk and dkt6) responded to our rebuttal and all of them either remained or raised their scores to 3 (weak acceptance).

Here are the specific concerns raised by the reviewers, how we responded and the final outcomes of discussion.

What does the joint solver add beyond implicit Euler? One of the main concerns dkt6 raised was whether the gains came from implicit discretization alone. We tested this directly by solving the same implicit-Euler equations with joint and sequential Anderson. On two matched-input batches, the joint solver took about 20 ms per batch versus 67 ms for the sequential solver. It also achieved higher mean docking coverage across three training runs per solver (0.665 vs. 0.620). These results give a concrete reason to solve jointly in DEFlow: lower measured forward latency and higher observed coverage in the tested setting. Reviewer dkt6 thinks our response addressed his concern and he updated his score to 3 accordingly.

How reliable are the density and gradients when the solver hits its cap? VrBX and dkt6 asked whether SAC can still estimate how likely a sampled action was and how to update the policy when the solver stops before reaching equilibrium. To address these concerns, we trained new policies on tight docking across five solver settings. We checked the action-density approximation by differentiating through the actual solver iterations and compared policy gradients on saved training batches with a high-precision equilibrium reference. Mean success rose from 50.6% at cap 10 to 96.2% at cap 25. Longer solves improved gradient accuracy, with mean success ranging from 96.5% to 98.2%. A cap of 25 achieved 96.17% mean success at less than half the forward-solve time of the higher-cap settings. So, our design choice was well supported by the docking results. VrBX confirmed that the new analyses addressed their concerns about the iteration cap and density approximation.

When does the additional computation pay off? VrBX, dkt6, and agWk all questioned whether DEFlow’s gains justify the additional computation cost. At fixed interaction budgets, DEFlow showed two strengths. It uniquely learned multiple precise ways to succeed, achieving the highest mean successful coverage on all four docking variants. It also learned faster in terms of environment interactions, reaching higher returns earlier in training. These are the settings where we believe DEFlow offers the clearest value. However, the compute trade-off remained a concern for VrBX and agWk, which made them keep their 3 (weak acceptance) scores.

In the revised paper, we added experiments comparing joint and sequential solvers for implicit Euler and testing how accurate the density and gradient estimates are when the solver reaches its iteration cap.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.