- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies safe optimization of flow policies in reinforcement learning. The authors argue that the internal surrogate objectives of flow-matching policies are mismatched with terminal-action performance, and accordingly propose a sensitivity-guided primal-dual framework that uses endpoint behavioral feedback to improve safety optimization for flow policies. Experiments are conducted on the Safety-Gymnasium navigation and Safe MuJoCo velocity-constraint benchmarks, together with an analytically solvable CMDP, in comparison with FPO-Lag, PPO-Lag, CSPO, ALGD, Epi-Flow, and other baselines.
Critical strengths
- The paper formalizes the surrogate-endpoint mismatch in the primal-dual safe optimization of flow policies.
- The proposed method is accompanied by supporting theory, including the local contraction of the multiplier update and the convergence of the coupled dynamics to active KKT points, with the response margin inducing only a steady-state offset in the multiplier.
- The proposed mechanism is transferable to π_{RL} fine-tuning.
Critical weaknesses
- The problem setups of several compared algorithms focus on per-step cost constraints, which deviates from the expected discounted cost constraint of the safe RL problem studied in this paper (Eq. (2)). This compromises the fairness of the comparison.
- The paper does not measure, on the real benchmarks, the quantitative relation between the multiplier and the endpoint response during FPO-Lag training.
- No sensitivity analysis of the algorithm's hyperparameters is provided.
Questions for authors
- What are the additional computational and memory overheads of Flow-Lag relative to FPO-Lag?
- On the real benchmarks, is Q^C, computed with the critic held fixed, correlated or consistent with the variation of the actual episodic cost?
- Can the proposed method guarantee absolute constraint satisfaction and absolute safety of the agent?
- How can long-term saturation and persistent drift of the multiplier be avoided?
4 comments
Reply to Reviewer oxYn [2/3]
Q1.What are the additional computational and memory overheads of Flow-Lag relative to FPO-Lag?
We thank the reviewer for this insightful question. Flow-Lag introduces additional computation through endpoint sensitivity estimation and the auxiliary cost critic. However, the overhead remains limited for two reasons.
First, Flow-Lag does not require additional rounds of policy optimization beyond the underlying FPO training procedure. Second, endpoint sensitivity is computed through automatic differentiation on a fixed calibration batch, without additional environment interactions or rollouts.
We conducted matched wall-clock benchmarks across four Safe MuJoCo environments with five random seeds each (40 runs in total). Both methods use identical actor architectures, rollout settings, batch sizes, and optimizer configurations. The following table summarizes the measured training time per iteration.
| Environment | FPO-Lag (s) | Flow-Lag (s) | Overhead | Mean |
|---|---|---|---|---|
| HalfCheetah | 5.00 | 5.18 | +3.36% | +4.03% |
| Walker2d | 6.42 | 6.57 | +1.79% | +2.59% |
| Ant | 9.01 | 9.24 | +2.26% | +2.42% |
| Swimmer | 5.19 | 5.32 | +2.22% | +2.69% |
| Overall | — | — | +3.08% | +2.93% |
We additionally measure process-level peak GPU memory and CPU resident set size (RSS) using three paired seeds per environment in the following table (Memory values are reported in MiB).
| Environment | FPO GPU | Flow GPU | GPU Overhead | FPO CPU RSS | Flow CPU RSS | CPU Overhead |
|---|---|---|---|---|---|---|
| HalfCheetah | 324 | 332 | +2.47% | 1213.9 | 1205.8 | -0.66% |
| Walker2d | 324 | 332 | +2.47% | 1204.5 | 1203.6 | -0.07% |
| Ant | 324 | 334 | +3.09% | 1219.5 | 1214.7 | -0.37% |
| Swimmer | 308 | 314 | +1.95% | 1199.7 | 1208.9 | +0.77% |
| Overall Mean | — | — | +2.50% | — | — | — |
Across the matched seed pairs, Flow-Lag incurs only 2.82% aggregate wall-clock overhead, with a 2.50% more GPU usage.
Q2. On the real benchmarks, is , computed with the critic held fixed, correlated or consistent with the variation of the actual episodic cost?
Yes. We conducted two additional validation experiments to examine the consistency of the learned cost estimators and whether fixed-critic predictions agree with actual episodic cost changes. The results are provided in the newly added Table 7 and Table 8 of Appendix C.11.
Q2.1 Consistency between and .
We conducted additional experiments to evaluate the consistency between the two cost estimators in the following table (Table 7 of Appendix C.11 in the revised manuscript).
Since is action-conditioned whereas is not, directly correlating their raw outputs would not constitute a meaningful comparison. We instead compare the two cost signals actually consumed by the algorithm: the action-relative cost advantage induced by the endpoint critic, and the -based GAE cost advantage used by the FPO actor update.
We conducted additional experiments to evaluate the agreement between the two learned cost estimators in the following table (Table 7 of Appendix C.11 in the revised manuscript), demonstrating consistency in their directional and ranking information.
| Environment | Sign Agreement | Spearman |
|---|---|---|
| Analytical CMDP | 94.4% | 0.994 |
| HalfCheetah | 81.3% | 0.744 |
| Walker2d | 82.8% | 0.888 |
Q2.2 Consistency with realized episodic cost
More directly, we evaluate whether the cost changes predicted by the fixed critic agree with the actual cost changes following policy updates.
For each policy-update pair , we compute the predicted cost difference while holding fixed. The corresponding actual cost change is independently measured through environment rollouts. We evaluate policy-update pairs per environment across three training seeds, with 100 evaluation episodes per policy using common random seeds. As shown in the following table, the fixed-critic predictions achieve 86.7-91.9% sign agreement with realized episodic cost changes. Pearson and Spearman correlations are positive across all these environments.
| Environment | Sign Agreement | Pearson | Spearman |
|---|---|---|---|
| HalfCheetah | 91.9% | 0.813 | 0.860 |
| Walker2d | 86.7% | 0.764 | 0.634 |
| Ant | 87.9% | 0.639 | 0.469 |
Reply to Reviewer oxYn [3/3]
Q3.Can the proposed method guarantee absolute constraint satisfaction and absolute safety of the agent?
Flow-Lag does not claim absolute or trajectory-wise safety guarantees. Our method follows the standard Constrained Markov Decision Process (CMDP) formulation defined in Eq. 2:
where
This expected cumulative-cost constraint is widely adopted in CMDP-based safe reinforcement learning, particularly in Lagrangian and primal-dual methods. All our compared baselines, except Epi-Flow, also follow expected cumulative-cost constraints.
Q4. How can long-term saturation and persistent drift of the multiplier be avoided?
Flow-Lag addresses multiplier-response saturation through the gain threshold . When the response gain is informative, the multiplier is corrected according to the observed endpoint response. When the gain becomes weak, Flow-Lag suppresses potentially unstable sensitivity-based corrections and applies only a small multiplier probe . If a cost-decreasing direction exists, bounded step calibration () further exploits it without relying solely on increased multiplier pressure. Theorem 1 establishes local stability under the stated regularity and informative-response assumptions.
We acknowledge that extreme cases may fall outside these conditions. For example, when the feasible policy set is empty or difficult to reach, policy updates may remain ineffective at reducing cost regardless of the multiplier. In such cases, sensitivity feedback cannot recover a cost-decreasing direction that the underlying policy optimizer fails to identify. This limitation is inherent to Lagrangian-based methods, rather than specific to Flow-Lag. The multiplier can only adjust the relative importance of the cost objective. If the policy optimizer cannot identify a feasible cost-reducing update, increasing the multiplier will not resolve the constraint violation.
Thank you for the author's carefully written response, which resolved my concerns. I have improved my score.
Reply to Reviewer oxYn [1/3]
Reply to Reviewer oxYn [1/3]
We sincerely thank you for your time and valuable suggestions. Your thoughtful feedback has helped us substantially strengthen our paper.
W1. Fairness of baselines
We thank the reviewer for this careful observation. We would like to clarify that except for Epi-Flow, the other compared methods all follow episode-level expected-cost constraints. To make this distinction explicit, we summarize their safety formulations below.
Our experiments already include multiple Lagrangian-based baselines and sensitivity mechanisms. We therefore include Epi-Flow as a representative non-Lagrangian flow-based safe RL method to provide a broader evaluation of Flow-Lag. Epi-Flow originally adopts an offline, hard state-constrained formulation. This distinction is inherent to the original method, and we intentionally preserve its problem formulation rather than modifying its underlying safety objective.
Importantly, all methods are evaluated in the same unmodified benchmark environments, using identical episodic reward and cost metrics. Thus, while their optimization objectives differ, the reported results provide a meaningful empirical comparison under a common evaluation protocol.
W2. Lack of the quantitative relation between the multiplier and the endpoint response during FPO-Lag training.
We thank the reviewer for this valuable suggestion. We have added a dedicated measurement in Fig. 19 of Appendix C.8 to quantitatively examine the multiplier-endpoint response relationship during FPO-Lag training.
Specifically, we measure the Lagrange multiplier λk, endpoint response Sk, and episodic cost across four Safe MuJoCo benchmarks with three random seeds, without modifying the original FPO-Lag optimization procedure.
As shown in Fig. 19, increasing the multiplier does not consistently produce a stronger negative endpoint response. In Walker2d and Ant, Sk fluctuates around zero despite continued multiplier escalation, while episodic costs frequently remain above the safety threshold. Similarly, Swimmer exhibits a nearly vanishing endpoint response even as the multiplier continues to increase.
These results provide direct empirical evidence that increasing the constraint penalty does not necessarily translate into effective endpoint cost reduction, which is precisely the failure mode that Flow-Lag is designed to address. Flow-Lag explicitly distinguishes whether this weak response can still be improved by adjusting the multiplier. If h^k≤−hmin, the revised multiplier is corrected to drive Sk toward the desired negative response −ϵ. Otherwise, Flow-Lag avoids ineffective multiplier escalation and applies only a small multiplier probe. When an existing cost-decreasing direction is available (Sk<0), it can further exploit that direction through bounded step calibration.
W3. No sensitivity analysis of the algorithm's hyperparameters is provided
We have conducted additional hyperparameter sensitivity experiments in Fig. 20 of Appendix C.10. We evaluate Flow-Lag on HalfCheetah and Walker2d, varying two key hyperparameters: the target endpoint response ϵ∈0.5ϵ0,ϵ0,2ϵ0 and the maximum step-calibration scale αmax∈2,4,8, where ϵ0=0.005 denotes the default response target.
As shown in Fig. 20, the default response target achieves a favorable reward-safety trade-off in both environments. A weaker target 0.5ϵ0 leads to insufficient safety correction on HalfCheetah, where the mean episodic cost exceeds the safety budget. Conversely, a stronger target 2ϵ0 reduces reward in both environments without providing additional safety improvements.
For the step cap, αmax=2 and 4 achieve comparable performance, whereas increasing the cap to 8 reduces reward and increases episodic cost in both environments. These results suggest that Flow-Lag maintains favorable performance under moderate hyperparameter choices, while excessively weak response targets or overly large step caps can degrade performance.
We further investigate the influence of different update scales α∈1.0,1.5,2.0. As shown in Fig. 18 (Appendix C.7), increasing the fixed update scale does not consistently improve reward or constraint satisfaction. The effects are non-monotonic and vary across environments. In particular, fixed scaling fails to achieve constraint satisfaction on HalfCheetah, Walker2d, and Swimmer, while its reward remains substantially below Flow-Lag on most of the stages.