acceptodds
Under review as a conference paper at ICLR 2027

Flow-Lag: Sensitivity-Aware Primal-Dual Optimization for Safe Flow Policies

Abstract

Flow-based policies can represent expressive action distributions and have recently shown strong performance in reinforcement learning. However, enforcing safety constraints for flow-based policies remains challenging. Most flow-policy optimization methods optimize surrogate objectives defined over the internal flow process, such as flow-matching or path-space objectives, rather than the deployed endpoint policy directly. This creates a surrogate-endpoint mismatch in which the policy is updated in surrogate objectives, while safety is ultimately determined by the endpoint actions executed in the environment. We show that this mismatch can substantially weaken safety correction in primal-dual reinforcement learning. Specifically, increasing the Lagrange multiplier can induce substantial changes in the flow policy while yielding only limited reductions in cost. To address this issue, we propose Flow-Lag, a sensitivity-aware primal-dual framework that propagates endpoint cost information through the flow dynamics to measure the safety response of the actual policy update. Flow-Lag uses this response to adapt the multiplier and calibrate the policy update, directly linking primal-dual correction to its endpoint effect. Across Safety-Gymnasium navigation and velocity-constrained Safe MuJoCo benchmarks, Flow-Lag achieves improved reward-safety performance over representative flow-based and conventional safe RL baselines.

Primary area:
reinforcement learning
Keywords:
Flow Matching, Reinforcment Learning, Primal-Dual

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_oxYnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies safe optimization of flow policies in reinforcement learning. The authors argue that the internal surrogate objectives of flow-matching policies are mismatched with terminal-action performance, and accordingly propose a sensitivity-guided primal-dual framework that uses endpoint behavioral feedback to improve safety optimization for flow policies. Experiments are conducted on the Safety-Gymnasium navigation and Safe MuJoCo velocity-constraint benchmarks, together with an analytically solvable CMDP, in comparison with FPO-Lag, PPO-Lag, CSPO, ALGD, Epi-Flow, and other baselines.

Critical strengths
  1. The paper formalizes the surrogate-endpoint mismatch in the primal-dual safe optimization of flow policies.
  2. The proposed method is accompanied by supporting theory, including the local contraction of the multiplier update and the convergence of the coupled dynamics to active KKT points, with the response margin inducing only a steady-state offset in the multiplier.
  3. The proposed mechanism is transferable to π_{RL} fine-tuning.
Critical weaknesses
  1. The problem setups of several compared algorithms focus on per-step cost constraints, which deviates from the expected discounted cost constraint of the safe RL problem studied in this paper (Eq. (2)). This compromises the fairness of the comparison.
  2. The paper does not measure, on the real benchmarks, the quantitative relation between the multiplier and the endpoint response during FPO-Lag training.
  3. No sensitivity analysis of the algorithm's hyperparameters is provided.
Questions for authors
  1. What are the additional computational and memory overheads of Flow-Lag relative to FPO-Lag?
  2. On the real benchmarks, is Q^C, computed with the critic held fixed, correlated or consistent with the variation of the actual episodic cost?
  3. Can the proposed method guarantee absolute constraint satisfaction and absolute safety of the agent?
  4. How can long-term saturation and persistent drift of the multiplier be avoided?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer oxYn [1/3]

Reply to Reviewer oxYn [1/3]

We sincerely thank you for your time and valuable suggestions. Your thoughtful feedback has helped us substantially strengthen our paper.


W1. Fairness of baselines

We thank the reviewer for this careful observation. We would like to clarify that except for Epi-Flow, the other compared methods all follow episode-level expected-cost constraints. To make this distinction explicit, we summarize their safety formulations below.

MethodSafety Constraint
Flow-Lag (Ours)Expected cumulative cost
FPO-LagExpected cumulative cost
PPO-LagExpected cumulative cost
CSPO (Adapted)Expected cumulative cost
ALGDExpected cumulative cost
Epi-FlowHard state-wise constraints

Our experiments already include multiple Lagrangian-based baselines and sensitivity mechanisms. We therefore include Epi-Flow as a representative non-Lagrangian flow-based safe RL method to provide a broader evaluation of Flow-Lag. Epi-Flow originally adopts an offline, hard state-constrained formulation. This distinction is inherent to the original method, and we intentionally preserve its problem formulation rather than modifying its underlying safety objective.

Importantly, all methods are evaluated in the same unmodified benchmark environments, using identical episodic reward and cost metrics. Thus, while their optimization objectives differ, the reported results provide a meaningful empirical comparison under a common evaluation protocol.


W2. Lack of the quantitative relation between the multiplier and the endpoint response during FPO-Lag training.

We thank the reviewer for this valuable suggestion. We have added a dedicated measurement in Fig. 19 of Appendix C.8 to quantitatively examine the multiplier-endpoint response relationship during FPO-Lag training.

Specifically, we measure the Lagrange multiplier λk\lambda_k, endpoint response SkS_k, and episodic cost across four Safe MuJoCo benchmarks with three random seeds, without modifying the original FPO-Lag optimization procedure.

As shown in Fig. 19, increasing the multiplier does not consistently produce a stronger negative endpoint response. In Walker2d and Ant, SkS_k fluctuates around zero despite continued multiplier escalation, while episodic costs frequently remain above the safety threshold. Similarly, Swimmer exhibits a nearly vanishing endpoint response even as the multiplier continues to increase.

These results provide direct empirical evidence that increasing the constraint penalty does not necessarily translate into effective endpoint cost reduction, which is precisely the failure mode that Flow-Lag is designed to address. Flow-Lag explicitly distinguishes whether this weak response can still be improved by adjusting the multiplier. If h^k≤−hmin⁡\hat h_k \leq -h_{\min}, the revised multiplier is corrected to drive SkS_k toward the desired negative response −ϵ-\epsilon. Otherwise, Flow-Lag avoids ineffective multiplier escalation and applies only a small multiplier probe. When an existing cost-decreasing direction is available (Sk<0S_k<0), it can further exploit that direction through bounded step calibration.


W3. No sensitivity analysis of the algorithm's hyperparameters is provided

We have conducted additional hyperparameter sensitivity experiments in Fig. 20 of Appendix C.10. We evaluate Flow-Lag on HalfCheetah and Walker2d, varying two key hyperparameters: the target endpoint response ϵ∈0.5ϵ0,ϵ0,2ϵ0\epsilon \in \\{0.5\epsilon_0,\epsilon_0,2\epsilon_0\\} and the maximum step-calibration scale αmax⁡∈2,4,8\alpha_{\max}\in\\{2,4,8\\}, where ϵ0=0.005\epsilon_0=0.005 denotes the default response target.

As shown in Fig. 20, the default response target achieves a favorable reward-safety trade-off in both environments. A weaker target 0.5ϵ00.5\epsilon_0 leads to insufficient safety correction on HalfCheetah, where the mean episodic cost exceeds the safety budget. Conversely, a stronger target 2ϵ02\epsilon_0 reduces reward in both environments without providing additional safety improvements.

For the step cap, αmax⁡=2\alpha_{\max}=2 and 44 achieve comparable performance, whereas increasing the cap to 88 reduces reward and increases episodic cost in both environments. These results suggest that Flow-Lag maintains favorable performance under moderate hyperparameter choices, while excessively weak response targets or overly large step caps can degrade performance.

We further investigate the influence of different update scales α∈1.0,1.5,2.0\alpha\in\\{1.0,1.5,2.0\\}. As shown in Fig. 18 (Appendix C.7), increasing the fixed update scale does not consistently improve reward or constraint satisfaction. The effects are non-monotonic and vary across environments. In particular, fixed scaling fails to achieve constraint satisfaction on HalfCheetah, Walker2d, and Swimmer, while its reward remains substantially below Flow-Lag on most of the stages.

AuthorsOpenReview ↗

Reply to Reviewer oxYn [2/3]

Q1.What are the additional computational and memory overheads of Flow-Lag relative to FPO-Lag?

We thank the reviewer for this insightful question. Flow-Lag introduces additional computation through endpoint sensitivity estimation and the auxiliary cost critic. However, the overhead remains limited for two reasons.

First, Flow-Lag does not require additional rounds of policy optimization beyond the underlying FPO training procedure. Second, endpoint sensitivity is computed through automatic differentiation on a fixed calibration batch, without additional environment interactions or rollouts.

We conducted matched wall-clock benchmarks across four Safe MuJoCo environments with five random seeds each (40 runs in total). Both methods use identical actor architectures, rollout settings, batch sizes, and optimizer configurations. The following table summarizes the measured training time per iteration.

EnvironmentFPO-Lag (s)Flow-Lag (s)OverheadMean
HalfCheetah5.005.18+3.36%+4.03%
Walker2d6.426.57+1.79%+2.59%
Ant9.019.24+2.26%+2.42%
Swimmer5.195.32+2.22%+2.69%
Overall——+3.08%+2.93%

We additionally measure process-level peak GPU memory and CPU resident set size (RSS) using three paired seeds per environment in the following table (Memory values are reported in MiB).

EnvironmentFPO GPUFlow GPUGPU OverheadFPO CPU RSSFlow CPU RSSCPU Overhead
HalfCheetah324332+2.47%1213.91205.8-0.66%
Walker2d324332+2.47%1204.51203.6-0.07%
Ant324334+3.09%1219.51214.7-0.37%
Swimmer308314+1.95%1199.71208.9+0.77%
Overall Mean——+2.50%———

Across the matched seed pairs, Flow-Lag incurs only 2.82% aggregate wall-clock overhead, with a 2.50% more GPU usage.


Q2. On the real benchmarks, is QCQ_C, computed with the critic held fixed, correlated or consistent with the variation of the actual episodic cost?

Yes. We conducted two additional validation experiments to examine the consistency of the learned cost estimators and whether fixed-critic predictions agree with actual episodic cost changes. The results are provided in the newly added Table 7 and Table 8 of Appendix C.11.

Q2.1 Consistency between QCQ_C and VCV_C.

We conducted additional experiments to evaluate the consistency between the two cost estimators in the following table (Table 7 of Appendix C.11 in the revised manuscript).

Since QC(s,a)Q_C(s,a) is action-conditioned whereas VC(s)V_C(s) is not, directly correlating their raw outputs would not constitute a meaningful comparison. We instead compare the two cost signals actually consumed by the algorithm: the action-relative cost advantage ACQ(s,a)=QC(s,a)−VC(s)A_C^Q(s,a)=Q_C(s,a)-V_C(s) induced by the endpoint critic, and the VCV_C-based GAE cost advantage used by the FPO actor update.

We conducted additional experiments to evaluate the agreement between the two learned cost estimators in the following table (Table 7 of Appendix C.11 in the revised manuscript), demonstrating consistency in their directional and ranking information.

EnvironmentSign AgreementSpearman
Analytical CMDP94.4%0.994
HalfCheetah81.3%0.744
Walker2d82.8%0.888

Q2.2 Consistency with realized episodic cost

More directly, we evaluate whether the cost changes predicted by the fixed critic QCQ_C agree with the actual cost changes following policy updates.

For each policy-update pair (πk,πk+1)(\pi_k,\pi_{k+1}), we compute the predicted cost difference ΔQˉkC\Delta\bar Q_k^C while holding QCQ_C fixed. The corresponding actual cost change ΔJ^C\Delta\widehat J_C is independently measured through environment rollouts. We evaluate policy-update pairs per environment across three training seeds, with 100 evaluation episodes per policy using common random seeds. As shown in the following table, the fixed-critic predictions achieve 86.7-91.9% sign agreement with realized episodic cost changes. Pearson and Spearman correlations are positive across all these environments.

EnvironmentSign AgreementPearsonSpearman
HalfCheetah91.9%0.8130.860
Walker2d86.7%0.7640.634
Ant87.9%0.6390.469
AuthorsOpenReview ↗

Reply to Reviewer oxYn [3/3]

Q3.Can the proposed method guarantee absolute constraint satisfaction and absolute safety of the agent?

Flow-Lag does not claim absolute or trajectory-wise safety guarantees. Our method follows the standard Constrained Markov Decision Process (CMDP) formulation defined in Eq. 2:

max⁡πJR(π)s.t.JC(π)≤d,\max_{\pi} J_R(\pi)\quad \text{s.t.} \quad J_C(\pi) \leq d,

where JC(π)=Eτ∼π[∑t=0T−1γtc(st,at)].J_C(\pi)=\mathbb{E}_{\tau\sim\pi}[\sum_{t=0}^{T-1}\gamma^t c(s_t,a_t)].

This expected cumulative-cost constraint is widely adopted in CMDP-based safe reinforcement learning, particularly in Lagrangian and primal-dual methods. All our compared baselines, except Epi-Flow, also follow expected cumulative-cost constraints.


Q4. How can long-term saturation and persistent drift of the multiplier be avoided?

Flow-Lag addresses multiplier-response saturation through the gain threshold hmin⁡h_{\min}. When the response gain is informative, the multiplier is corrected according to the observed endpoint response. When the gain becomes weak, Flow-Lag suppresses potentially unstable sensitivity-based corrections and applies only a small multiplier probe δλ\delta_\lambda. If a cost-decreasing direction exists, bounded step calibration (αk≤αmax⁡\alpha_k \leq \alpha_{\max}) further exploits it without relying solely on increased multiplier pressure. Theorem 1 establishes local stability under the stated regularity and informative-response assumptions.

We acknowledge that extreme cases may fall outside these conditions. For example, when the feasible policy set is empty or difficult to reach, policy updates may remain ineffective at reducing cost regardless of the multiplier. In such cases, sensitivity feedback cannot recover a cost-decreasing direction that the underlying policy optimizer fails to identify. This limitation is inherent to Lagrangian-based methods, rather than specific to Flow-Lag. The multiplier can only adjust the relative importance of the cost objective. If the policy optimizer cannot identify a feasible cost-reducing update, increasing the multiplier will not resolve the constraint violation.

Thank you for the author's carefully written response, which resolved my concerns. I have improved my score.

Reviewer_XueGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Lagrangian (primal-dual) safe RL with flow-based policies. It argues that flow-policy optimizers such as FPO update the policy through a surrogate loss on the internal flow rather than on the executed action. As a result, raising the Lagrange multiplier can change the flow substantially while barely reducing cost.

The proposed method, Flow-Lag, uses a learned cost critic QC(s,a)Q_C(s, a) to estimate how much each policy update lowers the predicted cost of the executed action (a quantity called SkS_k). It uses this estimate in two ways:

  1. It adjusts a second, "revised" multiplier that drives the policy update.
  2. When the multiplier stops having an effect, it scales up a cost-reducing update.

The theory consists of a one-step cost-decrease bound and local convergence results for the multiplier. Experiments cover four Safe MuJoCo velocity tasks, four Safety-Gymnasium navigation tasks, a small analytic problem, and fine-tuning with the πRL backbone. The paper compares against five baselines and includes a two-component ablation.

Critical strengths

Strength 1

The paper addresses a timely and underexplored problem: Lagrangian constraint handling for flow policies, whose action likelihood cannot be computed. Its central idea, evaluating the cost consequence of the optimizer's actual update, is concrete and useful. It goes beyond the cost feedback that ordinary primal-dual methods already use, because it separates two questions: whether more multiplier pressure still helps, and whether an existing cost-reducing direction can be exploited further. The mechanism also leaves the underlying flow-policy optimizer unchanged.

Strength 2

The evaluation goes beyond final reward and cost. Cumulative violation measures how quickly safety is corrected, the ablation separates the two components, and the πRL experiment gives some evidence that the method transfers to another backbone. The analytic CMDP has a known optimal solution and multiplier, which makes it a useful controlled test. The appendix is also open about how the baselines differ from their original versions.

Critical weaknesses

Weakness 1

The signals that drive Flow-Lag are not shown to be reliable.

First, SkS_k depends on an additional action-conditioned critic QC(s,a)Q_C(s,a), but the paper does not specify its training loss, targets, or data, even though the method uses its action gradient ∇aQC\nabla_a Q_C. The FPO actor update is driven by cost advantages from a separate network, VCV_C, and the consistency between these two cost estimates is not evaluated. Fig. 9 shows that SkS_k matches the derivative of the fixed-critic proxy. It does not show that SkS_k predicts the actual change in episodic cost after the update.

Second, the multiplier gain in Eq. 18 is a secant computed across consecutive training iterations. Between those iterations, the policy, sampled states, critic, advantages, and optimizer state all change. The estimated gain therefore need not measure the effect of the multiplier at a fixed training state: a negative or weak gain can reflect training drift or noise. Yet this estimate decides between multiplier adaptation and step extension, and the local theorems simply assume that it is informative and well behaved.

Both signals are central to the proposed contribution, so their reliability needs direct evidence.

Weakness 2

The comparisons do not yet establish where the improvements come from.

The closest baseline is not matched. FPO-Lag appears to differ from Flow-Lag in clipping, learning rate, and normalization on Safety-Gymnasium (Tables 3–4). Its actor width also differs: 256 in Appendix B.2, versus 64 for Flow-Lag in Table 5.

Other baselines are modified or not comparable. CSPO is a modified implementation with a reward-only warm start and a full-batch update, ALGD uses a reduced compute schedule, and Epi-Flow is an offline method with a different safety objective. These differences are disclosed, but they limit the broad superiority claim.

The step extension needs a control. Eq. 22 can extrapolate well beyond the optimizer's proposed update when SkS_k is only weakly negative, so some of the gains may simply come from larger steps. A matched FPO comparison against a tuned violation-based or PID multiplier, and against a simple step-scaling control, would directly test whether endpoint-response feedback adds value beyond ordinary tuning of the multiplier or step size.

Reporting is limited. Results are mostly training curves, with no per-task final metrics or uncertainty summaries. At minimum, a robust estimate of final reward and cost across seeds (e.g., median or trimmed mean, with confidence intervals) is needed so that outliers do not drive the conclusions. The headline cumulative-violation reductions each come from a single task.

Questions for authors

Question 1

How exactly is the auxiliary endpoint critic QC(s,a)Q_C(s,a) trained (loss, regression target, and data)? Since the actor update uses cost advantages estimated from VCV_C while Flow-Lag's response controller uses ∇aQC\nabla_a Q_C, did you evaluate consistency between these two cost estimates or the accuracy of the action gradients used by Flow-Lag?

Question 2

Does SkS_k predict the realized change in empirical episodic cost after applying the policy update, rather than only the finite-difference derivative of the fixed-critic proxy QˉC\bar Q_C? Please report sign agreement and, if possible, correlation with the subsequent observed cost change.

Additional questions

  • Were FPO-Lag and Flow-Lag matched exactly in clipping, actor learning rate, normalization, and other backbone settings on each benchmark? If not, please provide a matched FPO-Lag comparison. Please also clarify why the CSPO baseline uses a different training protocol.

  • How often is the α>1\alpha>1 calibration branch activated, what is the distribution of α\alpha, and how often does it hit αmax⁡\alpha_{\max}? Is the extrapolated policy re-evaluated for safety before being applied?

  • Proposition 1, the paper's only result on the joint convergence of the policy and multipliers, is stated and proved in Appendix A.5 but never referenced in the main text. Why is that ?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer XueG [1/3]

We sincerely thank the reviewer for the careful and thorough assessment of our work, especially for taking the time to examine our paper, including the Appendix in detail.


W1. Response to Weakness 1

W1.1 Implementation Details

We have expanded the Appendix with two detailed algorithms for FPO-Lag and πRL\pi_{\mathrm{RL}}-Lag (Algorithms 1 and 2 in Appendix D). These additions specify the critic training losses, regression targets, data, and actor optimization procedures.

W1.2 Consistency Between QCQ_C and VCV_C.

We conducted additional experiments to evaluate the consistency between the two cost estimators in the following table (Table 7 of Appendix C.11 in the revised manuscript).

Since QC(s,a)Q_C(s,a) is action-conditioned whereas VC(s)V_C(s) is not, directly correlating their raw outputs would not constitute a meaningful comparison. We instead compare the two cost signals actually consumed by the algorithm: the action-relative cost advantage ACQ(s,a)=QC(s,a)−VC(s)A_C^Q(s,a)=Q_C(s,a)-V_C(s) induced by the endpoint critic, and the VCV_C-based GAE cost advantage used by the FPO actor update.

We conducted additional experiments to evaluate the agreement between the two learned cost estimators in the following table (Table 7 of Appendix C.11 in the revised manuscript), demonstrating consistency in their directional and ranking information.

EnvironmentSign AgreementSpearman
Analytical CMDP94.4%0.994
HalfCheetah81.3%0.744
Walker2d82.8%0.888

W1.3 Endpoint Response and Actual Cost Changes.

To directly examine whether SkS_k reflects the actual cost change induced by policy updates, we conducted an additional validation on the Analytical CMDP using 1225 actual updates in Fig. 15 of Appendix C.6. The reference cost changes are computed from the analytical environment model, rather than the learned critic proxy. We observe 99.02% sign agreement, with Pearson and Spearman correlations of 0.921 and 0.925, respectively. The results exhibit a positive, approximately linear relationship between SkS_k and the subsequent cost change, supporting the predictive value of the local endpoint response in this controlled setting.

w1.4 Reliability of the Multiplier-Response Gain.

We further investigated the reliability of the secant estimate in Eq. 18 (Eq. 22 in revised manuscript) through additional controlled finite-difference experiments. Specifically, we compare the historical sensitivity estimate against a fixed-state reference, where the policy, critics, optimizer state, and training samples are held constant while varying only the multiplier. We additionally evaluate the same historical estimate against a reference obtained after further policy and critic updates, allowing us to examine the influence of training drift.

As shown in Fig. 16, the average directional/sign agreement reaches 87.4% under fixed-state evaluation and remains at 80.2% after policy and critic drift. Fig. 17 further demonstrates that the estimated sensitivity preserves meaningful rank consistency. These results confirm that policy and critic drift introduces estimation errors, as the reviewer correctly observed. Nevertheless, the historical sensitivity retains useful directional agreement and ranking information across most environments, supporting its practical informativeness during training.

We have revised the manuscript accordingly and included the results in Appendix C.6.


W2. Response to Weakness 2

W2.1 Matched FPO-Lag Comparison

On Safe MuJoCo, FPO-Lag and Flow-Lag share the same learning rate, clipping coefficient, and normalization settings. The main differences are the actor hidden width (256 vs 64) and optimization epochs (8 vs 4).

In our initial experiments, FPO-Lag struggled to sufficiently reduce the constraint cost through the standard Lagrangian flow-matching surrogate. We therefore used a larger actor and more optimization epochs to improve its representational capacity and optimization effectiveness. In contrast, Flow-Lag estimates the first-order endpoint-cost response along the actual policy update and uses it for step calibration. We adopt fewer optimization epochs to keep policy updates relatively local, where this first-order approximation is more reliable.

We agree that a matched comparison is necessary to isolate Flow-Lag's contribution. We therefore added additional experiments in which FPO-Lag adopts the same actor architecture and optimization epochs as Flow-Lag. As shown in Fig. 12 in Appendix C.3.2, the matched FPO-Lag configuration satisfies the constraints less consistently. This suggests that the surrogate-endpoint mismatch and limited optimization capacity make constraint satisfaction more difficult. In contrast, Flow-Lag demonstrates an overall better performance than both the original and matched FPO-Lag baselines.

AuthorsOpenReview ↗

Reply to Reviewer XueG [2/3]

W2.2 Justification of CSPO Baseline

We would like to clarify that our baseline is a CSPO-style adaptation to flow-policy optimization, as described in Appendix B.2.

Since the original CSPO is not designed for flow-based policy optimization, we adapt its sensitivity-guided constraint-handling mechanism to a common FPO backbone. This allows us to compare different sensitivity mechanisms under a consistent flow-policy optimization framework.

The reward-only warm start is introduced to establish a meaningful initial policy before applying sensitivity-based safety corrections, which can otherwise be unstable during the early stages of training. We additionally adopt full-batch updates to reduce noise in sensitivity estimation and improve optimization stability.

W2.3 Justification of ALGD Baseline

We appreciate the reviewer's concern regarding the reduced computational schedule of ALGD. To examine its influence, we conducted an additional comparison using the official implementation, which increases the optimization frequency to 10 updates per step in Fig. 10 of Appendix C.3.1. Based on our observation, while the increased computational budget improves reward on HalfCheetah and Walker2d, it does not substantially improve constraint satisfaction. Both configurations exhibit significant cost violations on these tasks, while their performance remains broadly similar on Ant and Swimmer.

ALGD first constructs a target-action distribution through counterfactual action evaluations and then distills it back into the policy. We identify a potential limitation is the counterfactual target distribution may provide insufficient guidance for policy optimization, affecting safety performance even with additional optimization epochs. To examine this issue, we conducted an additional analysis of ALGD's target-action weights in Fig. 11 of Appendix C.3.1. The weights exhibit relatively high normalized entropy (0.929--1.000) and effective sample size (0.721--0.999), indicating limited discrimination among candidate actions.

W2.4 Justification of Epi-Flow Baseline

Our experiments already include multiple Lagrangian-based baselines and sensitivity mechanisms. We therefore include Epi-Flow as a representative non-Lagrangian flow-based safe RL method to provide a broader evaluation of Flow-Lag. Epi-Flow originally adopts an offline, hard state-constrained formulation. This distinction is inherent to the original method, and we intentionally preserve its problem formulation rather than modifying its underlying safety objective.

Importantly, all methods are evaluated in the same unmodified benchmark environments, using identical episodic reward and cost metrics. Thus, while their optimization objectives differ, the reported results provide a meaningful empirical comparison under a common evaluation protocol.

W2.5 Fixed Step Scaling.

To examine whether the improvements simply arise from larger policy updates or conventional multiplier tuning, we conducted additional controlled comparisons.

We evaluate FPO-Lag with fixed update scales α∈\alpha \in {1.0, 1.5, 2.0}, while keeping the underlying optimization procedure unchanged.

The results in Fig. 18 (Eq. 22 in revised manuscript) in Appendix C.7 show that increasing α\alpha does not produce consistent improvements in reward or safety. The effects are non-monotonic and vary considerably across environments. In particular, fixed scaling fails to achieve constraint satisfaction on HalfCheetah, Walker2d, and Swimmer, while the reward remains substantially below Flow-Lag on HalfCheetah, Walker2d, and Ant.

W2.6 Comparison with FPO-PID baselines.

We additionally compare against multiple PID-based multiplier configurations, including FPO-PID with different gain scales (0.5×0.5\times, default, and 2×2\times), as well as FPO-PI, the results are shown in Fig. 14 in Appendix C.5. Although these adaptive multiplier methods can improve constraint satisfaction in certain tasks, they do not consistently generate the better performance than Flow-Lag.

Together with our existing ablation study (Fig. 8), where Response Only retains the response-aware multiplier correction with α=1\alpha=1, these results provide evidence that the improvements cannot be attributed solely to larger optimization steps or conventional multiplier tuning.

W2.7 Limited Reporting

We further summarized results in Table 9 of Appendix C.11. Please refer to the revised manuscript.


Q1. Cost Critic Training and Consistency

Please refer to W1.1 and W1.2 for more details.

AuthorsOpenReview ↗

Reply to Reviewer XueG [3/3]

Q2. Does SkS_k predict the realized episodic cost change?

Yes. We find evidence that SkS_k provides meaningful information about the actual episodic cost change induced by policy updates.

As shown in Fig. 15 of Appendix C.6, we directly compare the estimated endpoint sensitivity SkS_k against ΔJC=JC(πk+1)−JC(πk)\Delta J_C=J_C(\pi_{k+1})-J_C(\pi_k) on Analytical CMDP across five random seeds.

In our setting, ΔJC\Delta J_C is computed exactly from the environment model, independently of the learned cost critic and its fixed-critic proxy QˉC\bar Q_C. The results exhibit approximately 99% sign agreement, with both Pearson and Spearman correlations exceeding 0.9. The scatter plot further demonstrates a strong positive and approximately linear relationship between SkS_k and the actual cost change.

As complementary evidence, we also examine the reliability of the multiplier-response gain under policy and critic updates across nine benchmark environments. Fig. 16 and 17 in Appendix C.6 show that the average directional agreement is 87.4% under fixed-state evaluation and 80.2% after policy/critic drift. The corresponding average Spearman correlations are 0.680 and 0.584, respectively.


Q3. Justification of FPO-Lag and CSPO baselines.

Please refer to W2.1 and W2.2 for more details.


Q4. Activation and Safety of Step Calibration

We analyzed the actual step-calibration coefficients recorded during Flow-Lag training across Safe MuJoCo environments using three seeds per environment.

The statistics cover 939 policy updates per environment and 3,756 updates in total.

EnvironmentExtension RateAt αmax⁡\alpha_{\max}Mean α\alpha
HalfCheetah4.15%0.64%2.63
Walker2d0.32%0.00%1.21
Ant2.88%0.11%1.67
Swimmer0.00%0.00%--
Overall1.84%0.19%2.19

Step extension is activated in only 1.84% of all policy updates, meaning that 98.16% of updates retain the original optimizer step (α=1\alpha=1). Moreover, only 0.19% of updates reach the maximum extension scale αmax⁡=4\alpha_{\max}=4.

These results indicate that step calibration is activated selectively when the estimated endpoint response indicates a suitable cost-decreasing direction. Together with our fixed-step scaling experiments and Response Only ablation, these observations further support that the performance improvements cannot be attributed simply to larger update steps.

Regarding safety re-evaluation, we do not perform additional environment rollouts to evaluate the extrapolated policy before applying the update. Instead, calibration is only activated when h^k>−hmin⁡\hat h_k>-h_{\min}, QˉC(1)>d\bar Q_C(1)>d, and Sk<0S_k<0.


Q5. Proposition 1

We thank the reviewer for pointing out this omission. Proposition 1 is intended as a supplementary theoretical result rather than a primary contribution. Specifically, it assumes local exponential stability of the underlying primal-dual mean dynamics and establishes that the response-aware modification preserves local convergence to an active KKT policy under additional regularity conditions.

For learned nonlinear flow policies, establishing global convergence would require substantially stronger assumptions. We therefore deliberately avoid presenting conditional local convergence as a major theoretical contribution. Instead, the main text focuses on the endpoint-sensitivity mechanism and multiplier adaptation, while the more technical analysis of the coupled policy-multiplier dynamics is provided in Appendix A.5 for completeness.

We agree that Proposition 1 should be explicitly referenced in the main text, and we added a cross-reference in the revision following Theorem 1.

Reviewer_hffpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify a surrogate–endpoint mismatch when using flow-matching policies with a Lagrangian (primal-dual optimization) framework for constrained Markov decision processes (CMDPs).

The key observation is that increasing the Lagrange multiplier can significantly modify the internal flow dynamics (defined by the velocity field), but these changes may result in only small changes in the final action (endpoint). Consequently, the expected cost associated with final action may decrease only slightly, despite the larger Lagrange multiplier intended to enforce safety.

To address this issue, the authors propose a modification to the primal-dual optimization framework that accounts for this mismatch. They conduct experiments on Safety-Gymnasium and Safe MuJoCo benchmarks to demonstrate improvements in reward and safety performance.

Critical strengths
  1. Relevant problem: Authors identify a meaningful challenge in flow based RL policies. Increasing the Lagrange multiplier may not necessarily produce the expected change in the final action. Authors have provided, both theoretical and experimental results to indicate such a mismatch exist.
  2. The change they propose is specific and well described in the paper.
  3. Instead of relying only on constraint violations, the method evaluates how the proposed policy update affects predicted endpoint cost. This provides useful additional feedback for primal-dual optimization.
  4. Strong experimental results.
Critical weaknesses
  1. Limited explanation of the surrogate–endpoint mismatch. The authors should provide a more detailed explanation of this issue. Introducing the surrogate loss used in FPO in the preliminaries would significantly help readers understand the surrogate–endpoint mismatch discussed later in Section 4.1. Currently, the authors introduce flow matching in general but do not explain how the reward and cost advantages, ARA_R and ACA_C, are incorporated into the velocity-field optimization objective. As a result, readers must refer to the original FPO paper to understand the surrogate loss. It appears that the paper does not explicitly define this loss anywhere, despite its central role in motivating the proposed method.
  2. Recently most Flow based RL methods follow AWR (Advantage weighted regression) style weighed policy optimization loss functions[1]. The paper mainly focus on FPO instead, and author does not explore whether the identified issue exist in weighted regression methods. If it only applies to FPO, it should be discussed as a limitation.
  3. Limited theoretical justification for the modified multiplier update. The method targets an endpoint cost response Sk=−ϵS_k=-\epsilon, but it is unclear why reaching this target should lead to satisfaction of the original CMDP constraint JC(π)≤dJ_C(\pi)\leq d. Their Theorem-1 provides local stability of multiplier adaptation under assumptions, rather than policy feasibility or convergence.
  4. Computational overhead: As this author understand, the proposed method introduces additional computation to estimate endpoint sensitivity by differentiating through the flow dynamics, along with an additional cost critic. However, the paper does not report training-time or computational-overhead comparisons with FPO-Lag. Including these measurements would help assess whether the safety improvements justify the additional computational cost.
  5. Insufficient justification of surrogate–endpoint mismatch. "A flow policy generates an action by integrating several velocity steps. Some changes in intermediate velocities can cancel each other. Therefore, the internal flow can change significantly while the final action remains almost unchanged." This argument is correct but it does not prove that the problem actually occurs during FPO training. The authors should explain more clearly how the FPO loss and the Lagrange multiplier lead to the claimed mismatch.

[1] https://arxiv.org/pdf/2610.16275

Questions for authors
  1. Is surrogate–endpoint mismatch an inherent problem in flow-policy optimization, or is it a consequence of the specific FPO surrogate objective? Does this problem exist in other methods like FQL, AWR,
  2. Is there a computation overhead of the method? is it significant?
  3. Could established adaptive Lagrangian methods, such as PID-Lagrangian, mitigate the observed surrogate–endpoint mismatch by accounting for the history of constraint violations? Would comparing Flow-Lag against an FPO baseline with PID-based multiplier updates help demonstrate the additional benefits of the proposed endpoint-sensitivity mechanism?
  4. Constraint satisfaction: Why should targeting Sk=−ϵS_k=-\epsilon lead to better satisfaction of the original constraint JC(π)≤dJ_C(\pi)\leq d?
  5. Sensitivity estimation: How reliable is the estimated multiplier-response sensitivity when the policy and critics also change between iterations

This reviewer is willing to increase the score if the authors adequately address the main concerns raised.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer hffp [1/4]

We sincerely thank the reviewer for the careful evaluation of our manuscript and the thoughtful, constructive feedback. These comments can significantly help us to further strengthen the clarity and technical rigor of our paper. We provide point-by-point responses to the reviewer's concerns below.


W1. Clarification of the FPO Objective and Surrogate–Endpoint Mismatch

A1.1 Overall Clarification

We thank the reviewer for this valuable question and insightful observation. We agree that a more detailed introduction to the underlying flow-RL optimization objectives would help readers better understand our work. Accordingly, we have expanded the Preliminaries to introduce the optimization objectives of FPO and πRL\pi_{\mathrm{RL}}, clarifying how these methods optimize flow-based policies. More detailed optimization procedures are further provided in Algorithms 1 and 2 of Appendix D. We would also like to clarify that our work addresses primal–dual optimization for flow-based RL more generally, particularly methods relying on surrogate objectives, rather than FPO alone. We additionally integrate Flow-Lag with πRL\pi_{\mathrm{RL}}in Fig. 5. Rather than focusing on the specific surrogate objective of FPO, we provide a more general explanation of surrogate--endpoint mismatch that also applies to FPO and other surrogate-based flow-RL methods.

A1.2 General explanation of surrogate--endpoint mismatch.

To clarify the underlying mechanism, we begin with the original CMDP constraint. The expected discounted cost of the deployed endpoint policy is defined as JC(θ)=Eτ∼πθ[∑t=0∞γtc(st,at)]J_C(\theta)=\mathbb{E}_{\tau\sim \pi_{\theta}}[\sum_{t=0}^{\infty}\gamma^t c(s_t,a_t)].

Under the standard policy-gradient theorem, the gradient of this objective can be expressed as gCend≜∇θJC(θ)=E(s,a)∼dθγ[ACπθ(s,a)∇θlog⁡πθ(a∣s)],g_C^{\mathrm{end}}\triangleq \nabla_\theta J_C(\theta)=\mathbb{E}_{(s,a)\sim d_\theta^\gamma} [A_C^{\pi_\theta}(s,a)\nabla_\theta\log\pi_\theta(a\mid s)],

where dθγd_\theta^\gamma denotes the unnormalized discounted state-action occupancy measure.

Consider an ideal gradient-descent update that directly minimizes the endpoint cost: Δθend=−ηgCend.\Delta\theta_{\mathrm{end}}=-\eta g_C^{\mathrm{end}}.

However, many flow-based RL algorithms optimize surrogate objectives rather than directly evaluating the endpoint policy likelihood. Let LCsur(θ)\mathcal{L}_C^{\mathrm{sur}}(\theta) denote the cost-related surrogate objective, define gCsur≜∇θLCsur(θ),g_C^{\mathrm{sur}} \triangleq \nabla_\theta\mathcal{L}_C^{\mathrm{sur}}(\theta), and a local surrogate-gradient update takes the form Δθsur=−ηgCsur.\Delta\theta_{\mathrm{sur}} = -\eta g_C^{\mathrm{sur}}.

Applying the first-order expansion yields ΔJC=JC(θ+Δθsur)−JC(θ)≈∇θJC(θ)⊤Δθsur=−η(gCend)⊤gCsur.\Delta J_C=J_C(\theta+\Delta\theta_{\mathrm{sur}})-J_C(\theta)\approx\nabla_\theta J_C(\theta)^\top\Delta\theta_{\mathrm{sur}}=-\eta(g_C^{\mathrm{end}})^\top g_C^{\mathrm{sur}}.

This is where the surrogate-endpoint mismatch arises. The actual endpoint cost change depends on the alignment between the surrogate gradient and the true endpoint-cost gradient, while surrogate optimization does not generally guarantee (gCend)⊤gCsur>0(g_C^{\mathrm{end}})^\top g_C^{\mathrm{sur}}>0. This observation is closely related to the null-space argument in Section 4.1. For a KK-step flow with action dimension dad_a, the local linear mapping from intermediate velocity perturbations to the endpoint action perturbation is represented by a Jacobian of size (da×Kdad_a \times Kd_a). By the rank-nullity theorem, this Jacobian has a null space of dimension at least (K−1)da(K-1)d_a. Consequently, velocity perturbations can alter the intermediate flow while having no first-order effect on the endpoint action. Surrogate-driven updates may therefore contain components along these directions, resulting in changes to the internal flow that contribute little to the endpoint action.

We have added this derivation to Section 4.1 (highlighted in blue) to make the source of surrogate--endpoint mismatch and its implications for Lagrangian safety correction more explicit.

A1.3 Why the mismatch can specifically arise in FPO.

FPO constructs its PPO-style surrogate using the conditional flow-matching (CFM) loss. For a sampled action aa and state ss, the CFM loss is LCFM,θ(a;s)=1N∑i=1N∥vθ(xi,τi;s)−(a−zi)∥2,xi=τia+(1−τi)zi,{\mathcal L}_{\mathrm{CFM},\theta}(a;s)=\frac{1}{N}\sum_{i=1}^{N}\|v_\theta(x_i,\tau_i;s)-(a-z_i)\|^2, \quad x_i=\tau_i a+(1-\tau_i)z_i,

where ziz_i and τi\tau_i denote sampled noise and flow time, respectively.

Instead of the exact endpoint likelihood ratio, FPO uses a surrogate $ r_{\mathrm{FPO}}(\theta) = \exp({\mathcal L}_{\mathrm{CFM},\theta_{\mathrm{old}}}-{\mathcal L}_{\mathrm{CFM},\theta}).$

In FPO, increasing the multiplier therefore strengthens the cost-related surrogate update, but does not guarantee its alignment with the actual endpoint-cost gradient as analyzed above. Consequently, the induced velocity-field changes may have only a limited effect on endpoint safety, even under a stronger cost penalty.

AuthorsOpenReview ↗

Reply to Reviewer hffp [2/4]

W2. Advantage-Weighted Regression (AWR)

We thank the reviewer for carefully examining the related methods and for this valuable suggestion. We agree that AWR has become an important approach in flow-based RL. AWR-based methods construct a target action distribution directly and fit it back into the generative policy (velocity field), so they avoid the mismatch that arises from surrogation. We would also like to clarify that our Related Work and Experimental Results already include two closely related methods, ALGD and Epi-Flow, which similarly involve the construction or approximation of value-guided target action distributions during policy optimization. Epi-Flow learns an epigraph-guided target policy from offline data and distills it into a flow policy. ALGD further considers online constrained RL, constructing an augmented-Lagrangian-guided target action distribution for generative policy optimization. We have also considered AWR-based distributional optimization as an alternative to address the surrogate-endpoint mismatch. However, this alternative introduces two other important challenges that remain particularly relevant to safe flow-based RL.

(1) Target distribution estimation. AWR-based methods construct a target action distribution π(a∣s)\pi(a\mid s), which specifies the desired probability density of different actions under a given state. Constructing this distribution requires evaluating the reward and safety costs of alternative actions, typically through estimated action-value functions. These evaluations are counterfactual because they estimate the outcomes of actions that were not necessarily executed in the collected trajectories. Such estimates can be unreliable, especially for actions insufficiently covered by the training data, leading to inaccurate action preferences and a distorted target distribution. In constrained RL, this difficulty is further amplified by the continuously changing Lagrange multiplier λ\lambda, which repeatedly reshapes the target distribution and increases the influence of cost-value estimation errors.

(2) Generative distribution fitting. Even with a well-constructed target distribution, the flow policy must still approximate it through a separate generative fitting procedure. This introduces additional approximation and optimization errors, as discussed in the related work of [2].

Our experiments compare Flow-Lag against both ALGD and Epi-Flow in Fig. 3 and 6, where Flow-Lag achieves a stronger performance under the reported evaluation settings. Furthermore, we have conducted additional diagnostic experiments on ALGD to examine the influence of counterfactual action-value estimation in Fig. 11 of Appendix C.3.1. These results provide further evidence that inaccurate counterfactual evaluation is an important factor limiting AWR's performance in our tested settings.

We have revised (marked in blue) the Related Work and Experimental Results sections to highlight the ALGD and Epi-Flow in relation to the broader AWR-based distributional optimization family.


W3. Theoretical Justification of the Multiplier Update

The original constraint violation JC(π)−dJ_C(\pi)-d indicates whether additional safety correction is needed, whereas SkS_k measures whether the resulting policy update effectively reduces endpoint cost. Flow-Lag therefore targets Sk=−ϵS_k=-\epsilon to encourage a meaningful cost-decreasing response.

Under ideal conditions without surrogate-endpoint mismatch, standard Lagrangian primal-dual optimization in Eq. 3 converges to a KKT solution of the original CMDP problem in Eq. 2, satisfying JC(π)≤dJ_C(\pi)\leq d. Also in this setting, the response is consistently achieved (Sk+ϵ=0S_k+\epsilon=0). Consequently, the additional response correction in Eq. 23 vanishes, and the revised multiplier reduces to the standard Lagrange multiplier λ~k+1=λk+1\tilde{\lambda}_{k+1}=\lambda_{k+1}. Thus, Flow-Lag recovers standard primal-dual optimization under these ideal conditions, where constraint satisfaction follows from convergence to a feasible KKT solution.

Our theoretical analysis follows this setting. Lemma 1 establishes a conditional endpoint cost-decrease bound, while Theorem 1 proves local stability of the response-aware multiplier adaptation. Furthermore, Proposition 1 establishes conditional local convergence to the same feasible KKT policy.

Establishing global convergence for learning-based primal-dual RL remains challenging due to non-convex optimization and learned critics. Existing methods such as CSPO and ALGD, as well as our approach, can only provide local or assumption-dependent theoretical results rather than global convergence guarantees. Additionally, our experiments demonstrate effective safety correction and improved reward-safety performance in practice.

We have revised Section 4.3 to clarify this connection and explicitly reference Proposition 1 in the main text.

AuthorsOpenReview ↗

Reply to Reviewer hffp [3/4]

W4. Computational overhead

Flow-Lag introduces additional computation through endpoint sensitivity estimation and the auxiliary cost critic. However, the overhead remains limited for two reasons.

First, Flow-Lag does not require additional rounds of policy optimization beyond the underlying FPO training procedure. Second, endpoint sensitivity is computed through automatic differentiation on a fixed calibration batch, without additional environment interactions or rollouts.

We conducted matched wall-clock benchmarks across four Safe MuJoCo environments with five random seeds each (40 runs in total). Both methods use identical actor architectures, rollout settings, batch sizes, and optimizer configurations. The following table summarizes the measured training time per iteration.

EnvironmentFPO-Lag (s)Flow-Lag (s)OverheadMean
HalfCheetah5.005.18+3.36%+4.03%
Walker2d6.426.57+1.79%+2.59%
Ant9.019.24+2.26%+2.42%
Swimmer5.195.32+2.22%+2.69%
Overall——+3.08%+2.93%

We additionally measure process-level peak GPU memory and CPU resident set size (RSS) using three paired seeds per environment in the following table (Memory values are reported in MiB).

EnvironmentFPO GPUFlow GPUGPU OverheadFPO CPU RSSFlow CPU RSSCPU Overhead
HalfCheetah324332+2.47%1213.91205.8-0.66%
Walker2d324332+2.47%1204.51203.6-0.07%
Ant324334+3.09%1219.51214.7-0.37%
Swimmer308314+1.95%1199.71208.9+0.77%
Overall Mean——+2.50%———

Across the matched seed pairs, Flow-Lag incurs only 2.82% aggregate wall-clock overhead, with a 2.50% more GPU usage.

W5. Insufficient justification of surrogate–endpoint mismatch.

As discussed in W1, we have expanded Section 4.1 to explain surrogate--endpoint mismatch through gradient alignment and the null-space structure of flow policies. To provide direct evidence for whether null-space/misalgined internal gradients can occur during training, we conducted additional diagnostics using the analytically solvable CMDP environment in our paper. We retained the original FPO-Lag training configuration without modifying the optimization process.

Specifically, we measure two quantities: ρnull=∥(I−Jh†Jh)gCFPO∥∥gCFPO∥,κC=(gCend)⊤gCFPO∥gCend∥2,\rho_{\mathrm{null}}=\frac{\|(I-J_h^\dagger J_h)g_C^{\mathrm{FPO}}\|}{\|g_C^{\mathrm{FPO}}\|},\kappa_C=\frac{(g_C^{\mathrm{end}})^\top g_C^{\mathrm{FPO}}}{\|g_C^{\mathrm{end}}\|^2}, where JhJ_h is the Jacobian of the endpoint distribution statistics with respect to policy parameters. The first quantity measures the endpoint-null fraction of the FPO surrogate gradient, while the second measures its endpoint cost response relative to an exact endpoint-gradient step.

As shown in Fig. 13 of Appendix C.4, the endpoint-null fraction increases from 29.1% to 59.3% during FPO-Lag training. Meanwhile, the endpoint response gain decreases from its peak of 0.267 to 0.0099, representing approximately a 27-fold reduction. These results confirm that the FPO surrogate gradient develops substantial endpoint-null components and exhibits progressively weaker endpoint cost response during actual training. Please refer to our response to W1 for further discussion.


Q1. Applicability of Surrogate-Endpoint Mismatch

As discussed in W2, surrogate-endpoint mismatch is not an inherent property of all flow-based RL methods. It can arise when the policy is optimized through objectives that do not directly correspond to the actual endpoint cost, as in surrogate-based methods such as FPO and πRL\pi_{\mathrm{RL}}. We have provided a general gradient-alignment analysis in Section 4.1 to clarify this mechanism.

Other approaches, such as AWR-style methods and FQL, follow different optimization strategies and avoid the specific surrogate approximation. However, they introduce other challenges related to value estimation, target distribution construction, and policy distillation. As discussed in W2, our experiments included two related distributional optimization approaches, ALGD and Epi-Flow. We have also expanded the Related Work and Experimental Results sections to discuss their limitations in greater detail. Please refer to our response to W2 for further discussion.


Q2. Computational Overhead

As detailed in our response to W4, we conducted matched computational overhead benchmarks across four Safe MuJoCo environments. Please refer to W4 for the detailed measurements and explanation.

AuthorsOpenReview ↗

Reply to Reviewer hffp [4/4]

Q3. Comparison with PID-Lagrangian

We thank the reviewer for this valuable suggestion. We agree that PID-based Lagrangian adaptation is a meaningful baseline. We replace the standard multiplier update in FPO-Lag with a history-aware PID update: $ \lambda_k = \operatorname{clip} (K_P P_k + I_k + K_D D_k, 0, 100),$ where PkP_k, IkI_k, and DkD_k represent the proportional, integral, and derivative terms computed from episodic constraint violations. We use EMA smoothing for the proportional and derivative signals, with a derivative delay of 10 controller updates. We evaluate three PID gain configurations: 0.5×0.5\times, default, and 2×2\times, together with an FPO-PI variant that removes the derivative term while retaining the default proportional and integral gains. All variants retain the same underlying FPO optimizer, with only the multiplier controller modified.

The added results in Fig. 14 of Appendix C.5 show that PID-based adaptation can improve constraint regulation and reward performance over standard FPO-Lag in several tasks, demonstrating the benefits of history-aware multiplier updates. However, performance remains sensitive to PID gains, and improvements are not consistent across environments. Flow-Lag still achieves stronger performance in most of the stages of these MuJoCo tasks with less variability.

PID-Lagrangian can improve multiplier adaptation by incorporating historical constraint violations. However, this feedback alone is insufficient for flow policies, as it does not directly measure whether multiplier adjustments lead to effective endpoint cost reduction.


Q4. Constraint Satisfaction

Please refer to our response to W3 for details.


Q5. Reliability of Multiplier-Response Sensitivity

We agree that changes in the policy and critics between iterations may introduce noise or drift into the sensitivity estimate.

To evaluate the reliability of the historical sensitivity estimate and explicitly quantify this drift effect, we measured Directional Agreement and Spearman Correlation across all 9 benchmark environments. Specifically, we compared the estimated sensitivity under an ideal fixed setting where the actor and critic are frozen against an unfixed setting where the actor and critic are actively updating across training iterations to detect the impact of training noise directly.

To provide a comprehensive overview, we have included the full evaluation results in Appendix C.6, Fig. 16 and 17 of the revised manuscript. Specifically, Fig. 16 visualizes the Directional Agreement, while Fig. 17 illustrates the Spearman Rank Correlation across all nine environments. The figures clearly show that while intermediate network updates do introduce slight variances, the estimated sensitivity strongly preserves both its directional accuracy and relative magnitude trend compared to the ideal fixed-state reference.

Additionally, policy and critic changes introduce estimation errors, but the historical sensitivity and correlation still retain substantial directional agreement with the controlled multiplier response across the evaluated environments.


[2] Li, Z., Tang, S., & Azizan, N. (2026). Reverse flow matching: A unified framework for online reinforcement learning with diffusion and flow policies. ICML 2026.

Reviewer_Qe5fOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a sensitivity-aware primal-dual optimization for safe RL. The method is based on sensitivity analysis of the safety cost to the flow policy update. The Lagrange multiplier is updated when appropriate, otherwise the flow policy can be updated to further reduce cost violation. This paper offers a thorough analysis of the problem and provides a sensible solution in a systematic manner. The simulation experiments are convincing.

Critical strengths

The paper convincingly presents the potential lack of sensitivity that the cost action-value function associated with the safety constraint can have with respect to the flow policy. It then presents a method called Flow-Lag to update the policy based on a thorough sensitivity analysis so that the endpoint of the flow can improve the cost. The experiments are convincing as Flow-Lag accelerates the convergence to low costs for safety while enhancing the reward performance.

Critical weaknesses

No critical weaknesses were identified.

Questions for authors

It would be interesting to state how flow policies can improve the choice of an action a_t.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer Qe5f [1/1]

We sincerely thank the reviewer for the time to evaluate our work and for the valuable comments. Below, we first address the reviewer's question, then provide a brief summary of the key concerns raised by the other reviewers and the corresponding revisions and additional analyses in the Summary section.


Q1. How can flow policies improve the choice of an action ata_t?

Many policy-gradient RL algorithms rely on evaluating the policy likelihood πθ(at∣st)\pi_\theta(a_t \mid s_t) or its probability ratio. For flow-matching policies, actions are generated by integrating a neural ODE from an initial noise distribution. Although the resulting policy is stochastic, directly evaluating the likelihood of the final action is computationally expensive. Existing approaches mainly address this difficulty in two ways. First, methods such as πRL\pi_{\mathrm{RL}} [1] and ReinFlow [2] introduce stochasticity into the generation process, allowing tractable likelihood computation over the stochastic generation trajectory. Second, methods such as FPO [3] avoid explicit likelihood computation by constructing an ELBO-motivated surrogate using conditional flow-matching losses, enabling PPO-style policy optimization. These methods have proved effective in improving the choice of action ata_t.

Our work builds on these developments and focuses on a further challenge when combining flow-based RL with Lagrangian-constrained optimization. Flow-Lag addresses the surrogate-endpoint mismatch by explicitly measuring the predicted endpoint cost response of the actual policy update. Based on this feedback, we dynamically adapt the Lagrange multiplier and calibrate the update magnitude, making primal-dual optimization more responsive to the safety consequences of flow-policy updates.

[1]Chen, Kang, et al. "πRL\pi_{RL}: Online RL fine-tuning for flow-based vision-language-action models." arXiv preprint arXiv:2510.25889 8 (2025).

[2]Zhang, Tonghe, et al. "Reinflow: Fine-tuning flow matching policy with online reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 106282-106319.

[3]McAllister, David, et al. "Flow matching policy gradients." International Conference on Learning Representations. Vol. 2026. 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Summary of Revisions and Responses

We sincerely thank the Area Chair and all reviewers for their careful evaluation, constructive feedback, and valuable suggestions. The reviewers raised several important questions regarding the methodology, theoretical justification, experimental reliability, and baseline fairness of Flow-Lag.

In response, we have substantially revised the manuscript. Below, we summarize the major revisions and provide their locations in the revised manuscript for convenient reference.

1. Methodology Clarification

We clarify that Flow-Lag is a general primal-dual framework for flow-based RL, rather than a method restricted to FPO. In addition to FPO, we have demonstrated its applicability using πRL\pi_{RL} as an alternative flow-policy optimization backbone (Fig. 5).

To improve clarity and reproducibility, we have also expanded the methodological descriptions and added two detailed algorithms in Appendix D:

  • Algorithm 1: FPO-Lag optimization.
  • Algorithm 2: πRL\pi_{RL}-Lag optimization.

These additions provide further details on the actor and critic updates, training objectives, and overall optimization procedures.

2. Theoretical Clarification

We have expanded the theoretical discussion to address several important questions raised by the reviewers:

  • Surrogate–Endpoint Mismatch: We provide a more general explanation and derivation of the mismatch.
  • Justification of the Target Response Sk=−ϵS_k=-\epsilon: We explain the relationship between endpoint-response correction and satisfaction of the original CMDP constraint.

3. Additional Experiments and Empirical Validation

In response to the reviewers' suggestions, we conducted several groups of additional experiments to strengthen the empirical evaluation.

(1) Baseline Fairness and Justification

We clarified the implementations, safety formulations, and experimental settings of FPO-Lag, Epi-Flow, CSPO, and ALGD. We also conducted additional matched-baseline experiments and investigated the effects of different optimization configurations.

  • Appendix B.2: Baseline implementation details and fairness clarification.
  • Appendix C.3, Fig. 10–12: ALGD diagnostics and matched FPO-Lag comparisons.
  • Appendix C.5, Fig. 14: Additional PID-Lagrangian comparisons.

(2) Reliability of Endpoint-Response Estimation

We conducted additional experiments to examine the reliability of the proposed endpoint-response and multiplier-sensitivity estimates, including their consistency under fixed-state and moving-state evaluations.

  • Appendix C.6, Fig. 15–17: Analytical validation and sensitivity reliability across nine environments.
  • Appendix C.8, Fig. 19: Empirical multiplier–endpoint response measurements during FPO-Lag training.

(3) Step Calibration, Ablations, and Hyperparameter Sensitivity

We investigated whether the improvements of Flow-Lag could be attributed to larger policy updates or specific hyperparameter choices. We conducted additional fixed-step scaling controls, step-calibration activation analysis, and hyperparameter sensitivity experiments.

  • Fig. 8: Component ablation study.
  • Appendix C.7, Fig. 18: Fixed step-scaling comparisons.
  • Appendix C.9, Table 6: Step-calibration activation statistics.
  • Appendix C.10, Fig. 20: Hyperparameter sensitivity analysis.

(4) Computational Efficiency

We additionally evaluated the computational and memory overhead of Flow-Lag relative to FPO-Lag across four Safe MuJoCo environments.

  • Appendix C.2, Tables 4–5: Training-time and memory overhead comparisons.

(5) Consistency and Validation of QCQ_C, VCV_C, and JCJ_C

We conducted additional experiments to evaluate the consistency between the learned cost estimators and the relationship between their predictions and actual episodic cost changes.

  • Appendix C.6, Fig. 15: Validation of the relationship between SkS_k and actual episodic cost changes.
  • Appendix C.11, Table 7: Consistency between QCQ_C and VCV_C.
  • Appendix C.11, Table 8: Consistency between fixed-critic predictions and realized episodic cost changes.

Summary

In addition to the major concerns summarized above, the reviewers raised several other specific questions regarding implementation details, theoretical assumptions, safety guarantees, and practical limitations. We have addressed each of these questions individually in the detailed responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.