acceptodds
Under review as a conference paper at ICLR 2027

Constrained Flow Policy Updates: A Generalized Schrödinger Bridge View

Abstract

Online safe reinforcement learning (RL) seeks policies that maximize reward while satisfying safety constraints. Reward and safety can induce multimodal action distributions, challenging the prevailing primal-dual methods: Gaussian actors may collapse onto a single suboptimal mode, and optimization over the nonconvex Lagrangian landscape can be unstable. Diffusion and flow policies can represent such distributions, but recent work with a diffusion actor relies on estimating and matching the score of an augmented-Lagrangian target policy. Instead, we differentiate the augmented objective directly through the generation path of a flow policy, so no score needs to be estimated. Because a flow policy lacks a readily available action log-density for entropy regularization, we build on the density-free kinetic-energy regularizer of FLAC, a recent reward-only method, and propose Reparameterized Augmented-Lagrangian Flow Actor with Least Energy (RAFALE), an off-policy actor-critic method for safe RL. We formulate its update as a constrained one-ended generalized Schrödinger bridge and show that, for each source draw, this path-space problem is exactly an entropy-regularized problem in action space. At positive noise, its solution reweights the reward-only action distribution only where the estimated cost exceeds a threshold set by the Lagrange multiplier. As the noise vanishes, the optimal value converges to that of a least-energy map objective that the flow policy optimizes directly. Across seven Safety-Gymnasium tasks, RAFALE achieves competitive reward with mean final cost within budget on every task, whereas strong baselines trade one for the other; ablations support the necessity of both its augmented objective and its flow actor.

Primary area:
reinforcement learning
Keywords:
safe reinforcement learning, constrained reinforcement learning, flow policies, generative policies

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_q9VMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAFALE, an off-policy actor–critic method for online safe reinforcement learning. It combines FLAC’s flow actor and kinetic-energy regularization with an action-level augmented-Lagrangian objective based on reward and cost critics. The actor is updated by differentiating through a midpoint sampler, avoiding the target-score estimation used by ALGD. The paper provides a generalized Schrödinger-bridge interpretation of the frozen actor update, including a conditional action-space formulation, safety-dependent re-weighting, and a vanishing-noise map objective. Experiments on seven Safety-Gymnasium tasks report competitive rewards and mean final costs within budget.

Critical strengths
  1. A practical method with encouraging empirical results. RAFALE trains a constrained generative actor through direct pathwise differentiation without estimating a target score. Table 2 reports competitive reward–cost trade-offs, with mean final cost within budget on all seven tasks. The use of five training seeds and fixed endpoint checkpoints strengthens the empirical evaluation.

  2. An informative constraint-objective ablation. Table 3 compares the augmented objective against both the standard Lagrangian and a hinge penalty sharing its activation threshold. This helps distinguish thresholding from the penalty’s growth above the threshold. The reductions in peak evaluated cost on Hopper and Walker2d support the usefulness of the augmented objective.

Critical weaknesses
  1. The additional contribution over the closest prior work is not sufficiently established. Table 1 and Appendix A indicate that RAFALE inherits FLAC’s flow actor, kinetic regularization, and reparameterization-based update, while adopting ALGD’s action-level augmented-Lagrangian objective. The algorithm therefore appears to be a natural constrained extension of FLAC using an existing safety objective. Such a combination can be valuable, but the paper does not sufficiently explain the nontrivial technical obstacle involved or which theoretical insights go beyond instantiating the existing bridge framework with this terminal potential. This leaves the significance of the incremental contribution unclear.

  2. The experiments do not sufficiently establish the source and extent of the additional benefit. First, Table 4 shows similar Flow/GMM rewards on Hopper (1655 versus 1645) and Walker2d (3177 versus 3190), with all five final seeds feasible for both methods on both tasks. Second, the actor-family comparison also changes the regularizer: Flow uses kinetic regularization, whereas Gaussian/GMM actors use log-density regularization. As acknowledged in Appendix E.4, this prevents attributing the differences specifically to the policy representation, despite multimodal expressivity being a central motivation. Finally, Table 2 compares selected reward–cost operating points rather than comparable cost levels. For example, on Hopper, CAL and ALGD obtain rewards of 1664 and 1702 with costs of 12.2 and 15.7, while RAFALE obtains reward 1655 with cost 0.3, against a budget of 10. These results are encouraging, but do not establish whether the baselines would remain competitive after stronger safety tuning.

Questions for authors
  1. What specific technical difficulty arises when extending FLAC with ALGD’s augmented-Lagrangian terminal objective? Which theoretical result provides an insight that is not a direct instantiation of the existing framework?
  2. What evidence separates the contribution of the flow representation from that of kinetic regularization, particularly given the close Flow/GMM results on Hopper and Walker2d?
  3. Do you have results for CAL and ALGD at comparable feasible cost levels? Such evidence would help determine whether the advantage persists after accounting for the reward–cost trade-off. Minor presentation suggestion, not a reason for my recommendation: FLAC is an important reference, but its full name is not provided at its first occurrence in either the abstract or the main text. Please expand the acronym to help readers unfamiliar with this work.
5 comments
AuthorsOpenReview ↗

Response to Reviewer q9VM (1/3): contribution beyond combining FLAC with the augmented Lagrangian

W1 and Q1 (technical difficulty of the extension; which result is not a direct instantiation). We thank the reviewer for asking us to separate the adopted components from our own results. Forming the objective is indeed direct: FLAC's objective accepts any terminal potential (its Eq. (8)), so the augmented Lagrangian can be inserted as it is.

A direct instantiation, however, does not reveal how the constraint changes the generated actions. Instantiating FLAC's closed-form result (its Proposition 3.1) with the augmented Lagrangian yields a Boltzmann tilt of the action distribution as a whole, which has the form of ALGD's target. Two of our results go beyond it.

1. The constrained update for each source draw (Theorems 1 and 2). The tilt of FLAC's Proposition 3.1 is normalized once, over the whole distribution. Because a flow policy's source is fixed, we make the conditioning on the source draw explicit, following SoftGAC's fixed-base formulation (see our Appendix A), and normalize the tilt separately for each source draw before taking the action marginal.

Theorem 1 gives the exact action-space problem for each draw, and Theorem 2 compares its solutions for LAL_A and for −Qr-Q_r. Within each source draw, the constraint leaves the relative probabilities of actions below the threshold τ=h−λ/ρ\tau=h-\lambda/\rho unchanged and suppresses the other actions increasingly with their excess. Here, hh is the budget bb expressed in the critic's units. A globally normalized tilt does not express this property, because it does not track which action each source draw generates.

2. The actor that is trained (Theorem 3, Corollary 5, and Proposition 6 in the revised PDF). RAFALE's actor is noise-free. In this regime, FLAC provides a Wasserstein bound (its Eq. (7)) and notes that it gives no entropy guarantee. Theorem 3 shows that, as the noise vanishes, the optimal value converges to that of a map objective whose optimal maps send each source draw to a minimizer of LA(s,a)+α2∥a−x0∥2L_A(s,a)+\frac{\alpha}{2}\lVert a-x_0\rVert^2, and Corollary 5 shows that RAFALE's one-step midpoint loss evaluates this objective exactly.

Proposition 6 (Appendix B.3) shows that the selectivity survives the limit: a draw keeps its reward-only action if that action is below τ\tau, and otherwise it can move only to an action with a smaller excess.

Insight for the algorithm. This analysis describes the implemented update itself. The penalty behind the reweighting of Theorem 2 has gradient ρ[Qc(s,a)−τ]+∇aQc(s,a)\rho[Q_c(s,a)-\tau]_+\nabla_aQ_c(s,a), which is exactly the cost term of the actor gradient. The cost-gradient contribution of each generated action is therefore weighted in proportion to its own excess, whereas the standard Lagrangian gives every action the same weight λ\lambda and the hinge gives a constant weight above the threshold.

The paper's Table 3 isolates this difference with the flow actor and the regularizer fixed: the augmented objective has the lowest peak cost on all three tasks. The revised PDF states this relation in Section 4.2 ("Relation to FLAC") and adds Proposition 6; new text is blue.

AuthorsOpenReview ↗

Response to Reviewer q9VM (2/3): separating the flow representation from the kinetic regularizer

W2 and Q2 (flow representation versus kinetic regularization; close Flow/GMM results). We appreciate both observations and agree with them. First, Table 4 of the submission changes the actor and the regularizer together. Second, the flow and the GMM are close on Hopper and Walker2d. We therefore extended Table 4 to Ant and Swimmer, where the flow earns more reward than the GMM at lower cost (Table R1), and added the experiments below.

Comparable combinations. Each regularizer requires something that only some actors provide:

RegularizerGaussianGMMFlow
Entropy (needs the action density)paper's Table 4paper's Table 4no closed-form density¹
Kinetic (needs a map from source draw to action)new (Table R1)new (Table R2)²RAFALE

¹The one-step flow generates an action without providing its density, which is why FLAC replaces the entropy term by kinetic energy. ²A mixture draws its component independently of the source draw, so its kinetic objective is the weighted average of its components' objectives. On the multi-goal task, its eight components converge to the same map as the Gaussian actor.

Regularizer fixed, actor varied.

Table R1. Final reward / cost (runs within the budget of 10), five seeds each. Gaussian: X1=μθ(s)+σθ(s)⊙x0X_1=\mu_\theta(s)+\sigma_\theta(s)\odot x_0; residual: X1=x0+vθ(s,x0,0)X_1=x_0+v_\theta(s,x_0,0); flow: RAFALE's one-step midpoint map. In the columns marked †, the Hopper, Walker2d and Humanoid entries are from Table 4 of the submission, whereas the Ant and Swimmer entries are new. Bold blue / green: highest / second-highest reward among entries with mean cost within budget; red: mean cost above budget.

TaskGaussian + entropy†GMM + entropy†Gaussian + kineticResidual + kineticFlow + kinetic (RAFALE)
Hopper1408 / 7.6 (4/5)1645 / 1.5 (5/5)1658\color{#0046A8}{\mathbf{1658}} / 0.4 (5/5)1399 / 9.9 (4/5)1655\color{#38A600}{\mathbf{1655}} / 0.3 (5/5)
Walker2d3151 / 3.5 (4/5)3190\color{#0046A8}{\mathbf{3190}} / 1.4 (5/5)3124 / 2.0 (5/5)3155 / 7.6 (3/5)3177\color{#38A600}{\mathbf{3177}} / 1.3 (5/5)
Humanoid6510\color{#38A600}{\mathbf{6510}} / 8.1 (4/5)6503 / 8.5 (3/5)6351 / 8.4 (4/5)6504 / 12.3\color{#B22222}{12.3} (2/5)6513\color{#0046A8}{\mathbf{6513}} / 3.2 (5/5)
Ant3336 / 8.9 (3/5)3278 / 8.0 (4/5)3297 / 6.2 (4/5)3340\color{#38A600}{\mathbf{3340}} / 4.3 (5/5)3427\color{#0046A8}{\mathbf{3427}} / 4.4 (5/5)
Swimmer48 / 4.2 (5/5)56 / 7.2 (5/5)81\color{#38A600}{\mathbf{81}} / 4.1 (4/5)70 / 1.6 (5/5)93\color{#0046A8}{\mathbf{93}} / 5.4 (5/5)
Runs within budget20/2522/2522/2519/2525/25

With the kinetic regularizer fixed (last three columns), the flow is the only actor that keeps every run within budget on all five tasks. The regularizer alone does not produce this result: on Humanoid, the Gaussian actor's cost stays near 8 with either regularizer, whereas the flow lowers it to 3.2.

Separated safe modes. We test multimodal expressivity directly on a constrained version of FLAC's multi-goal task, with eight goals on a ring, two of which are unsafe (Figure 1 of the revised PDF). Under the budget, all actors use the same augmented Lagrangian and multiplier update.

Table R2. Constrained multi-goal task, five seeds. Coverage and budget criteria are defined in Appendix E.6 of the revised PDF. Bold blue / green: highest / second-highest reward.

Actor + regularizerNo budget: goals coveredNo budget: rewardb=0.02b=0.02: safe goals coveredb=0.02b=0.02: rewardb=0.02b=0.02: actions on unsafe goalsRuns within budget
Flow + kinetic (RAFALE)8/80.983\color{#0046A8}{\mathbf{0.983}}6/60.986\color{#0046A8}{\mathbf{0.986}}0.03%5/5
Gaussian + kinetic0/80.4604/60.5060%5/5
GMM + kinetic0/80.4604/60.5050%5/5
GMM + entropy8/80.976\color{#38A600}{\mathbf{0.976}}6/60.969\color{#38A600}{\mathbf{0.969}}1.1%4/5

Under the budget, the flow covers all six safe goals and places almost no actions on the unsafe ones. The Gaussian actor and the GMM with the same kinetic regularizer cover four; the entropy-regularized GMM covers six but keeps more actions on the unsafe goals and earns less reward.

Together, these experiments separate the two factors: with the regularizer fixed, the flow is the component that covers every safe mode and keeps every run within budget. Both are in the revised PDF (Figure 1, Table 4 and Appendix E.6).

AuthorsOpenReview ↗

Response to Reviewer q9VM (3/3): CAL and ALGD at feasible cost levels

Q3 (CAL and ALGD at comparable feasible cost levels). Thank you for suggesting this comparison. The paper's Table 2 reports every method at its final checkpoint, without selecting checkpoints. For this question, we made two additional comparisons, and RAFALE keeps its advantage in both.

Best feasible checkpoint of the baselines, all seven tasks. For CAL and ALGD, we selected after the fact the evaluated checkpoint with the highest mean reward among those whose mean cost is within budget (Table R7). This selection favors the baselines, because RAFALE is still evaluated at its final checkpoint. Even so, CAL and ALGD earn less reward than RAFALE's final policies on six of the seven tasks. The exception is Hopper, which the second comparison examines.

Table R7. Reward / cost of CAL and ALGD at their final checkpoint (as in the paper's Table 2) and at their best feasible checkpoint, next to RAFALE's final checkpoint. The best feasible checkpoint is selected after the fact: among the evaluated checkpoints whose mean cost is within budget, the one with the highest mean reward (five seeds; budget 10, and 25 on CarGoal). Bold blue / green: highest / second-highest reward among entries with mean cost within budget; red: mean cost above budget.

TaskCAL, finalCAL, best feasibleALGD, finalALGD, best feasibleRAFALE, final
Swimmer41 / 16.6\color{#B22222}{16.6}43 / 10.052 / 2.955\color{#38A600}{\mathbf{55}} / 3.093\color{#0046A8}{\mathbf{93}} / 5.4
HalfCheetah2650 / 15.5\color{#B22222}{15.5}2648 / 6.62817 / 22.4\color{#B22222}{22.4}2815\color{#38A600}{\mathbf{2815}} / 8.23025\color{#0046A8}{\mathbf{3025}} / 3.5
Hopper1664 / 12.2\color{#B22222}{12.2}1674\color{#38A600}{\mathbf{1674}} / 3.71702 / 15.7\color{#B22222}{15.7}1709\color{#0046A8}{\mathbf{1709}} / 3.91655 / 0.3
Walker2d2894 / 6.32922\color{#38A600}{\mathbf{2922}} / 7.92920 / 17.1\color{#B22222}{17.1}2915 / 6.63177\color{#0046A8}{\mathbf{3177}} / 1.3
Ant2815 / 11.1\color{#B22222}{11.1}2811\color{#38A600}{\mathbf{2811}} / 8.92218 / 16.8\color{#B22222}{16.8}1374 / 5.33427\color{#0046A8}{\mathbf{3427}} / 4.4
Humanoid5289 / 0.15380\color{#38A600}{\mathbf{5380}} / 0.04530 / 0.24756 / 0.16513\color{#0046A8}{\mathbf{6513}} / 3.2
CarGoal12.1 / 12.312.4 / 13.216.5 / 22.416.8\color{#38A600}{\mathbf{16.8}} / 24.518.3\color{#0046A8}{\mathbf{18.3}} / 20.7

Stronger safety tuning on Hopper, the task cited in the review. We retrained CAL and ALGD with the tighter training budget b′=5b'=5, while still judging feasibility at b=10b=10.

Table R6. Hopper, five seeds, 2M steps: CAL and ALGD retrained with training budget b′=5b'=5; feasibility is judged at b=10b=10 in every row. Rows marked Table 2 are from the paper. Bold blue / green: highest / second-highest reward among entries with mean cost within budget; red: mean cost above budget.

MethodTraining budgetRewardCostRuns within b=10b=10
RAFALE (Table 2)101655\color{#0046A8}{\mathbf{1655}}0.35/5
CAL (Table 2)10166412.2\color{#B22222}{12.2}1/5
CAL51655\color{#0046A8}{\mathbf{1655}}4.85/5
ALGD (Table 2)10170215.7\color{#B22222}{15.7}1/5
ALGD51377\color{#38A600}{\mathbf{1377}}7.23/5

With b′=5b'=5, every CAL run is within budget, and CAL's reward falls to RAFALE's level at a higher cost. The gaits explain why: on Hopper, the reward mainly reflects the average forward speed, whereas the cost counts the steps above the speed limit. The retrained CAL matches RAFALE's average speed, hence the equal reward, but it exceeds the speed limit more often. ALGD loses reward and still leaves runs above the budget.

After stronger safety tuning, neither baseline overtakes RAFALE on Hopper, and RAFALE reaches this reward at the original budget with the lowest cost. RAFALE's advantage thus persists at comparable feasible cost levels. Tables R7 and R6 are Tables 14 and 15 of the revised PDF.


Minor (FLAC's full name). We are grateful for this note. The revised PDF now writes "Field Least-Energy Actor-Critic (FLAC)" at its first occurrence in the abstract and in the main text.

Reviewer_14PQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reparameterized Augmented-Lagrangian Flow Actor with Least Energy (RAFALE), an off-policy reinforcement learning algorithm for constrained continuous control. Building upon FLAC, the method incorporates an augmented-Lagrangian objective into a kinetic-energy-regularized flow policy, enabling direct pathwise optimization of reward and safety critics without estimating action densities or target scores.

The paper further develops a generalized Schrödinger bridge interpretation of the constrained policy update. It establishes an equivalence between path-space regularization and a source-conditioned entropy-regularized action-space objective, characterizes how safety constraints reweight the resulting Gibbs distribution, and derives a vanishing-noise limit corresponding to the practical flow actor objective.

Experiments on seven Safety-Gymnasium tasks demonstrate competitive reward performance while maintaining mean final episodic costs within the prescribed budgets across all evaluated tasks.

Critical strengths
  • Meaningful extension of flow-based reinforcement learning to constrained optimization. The paper addresses an important problem in safe reinforcement learning, where expressive policies must optimize reward while satisfying safety constraints. Extending kinetic-energy-regularized flow policy optimization to this setting is technically meaningful, particularly because it enables direct optimization through the flow sampler without requiring explicit action log-densities or score estimation.

  • Coherent and well-developed theoretical framework. A major strength is the connection between the constrained flow update and the generalized Schrödinger bridge formulation. The action-space equivalence provides an interpretable relationship between path-space regularization, source-to-action displacement, and conditional entropy. The safety-tilting result further clarifies how the augmented-Lagrangian penalty changes the relative probabilities of actions according to their estimated costs. The vanishing-noise analysis establishes a clear connection between the stochastic bridge formulation and the deterministic actor objective used in practice.

  • Strong empirical performance in constrained continuous control. RAFALE achieves competitive rewards while satisfying the mean final cost budgets on all seven Safety-Gymnasium tasks. This is a notable result compared with the evaluated baselines, several of which achieve strong rewards at the expense of constraint violations or maintain feasibility with lower rewards. The results provide encouraging evidence of the effectiveness of the proposed framework.

  • Useful ablation studies. The paper compares augmented, standard, and hinge-based Lagrangian objectives, demonstrating the benefits of the augmented formulation for controlling cost violations. Comparisons with Gaussian and Gaussian-mixture actors also provide useful evidence regarding the effectiveness of the flow-based policy configuration.

  • Clear presentation and reproducibility. The paper is well organized, with a clear progression from the constrained optimization problem to the bridge formulation, theoretical analysis, and practical algorithm. The appendices provide detailed proofs, implementation settings, and additional experimental results.

Critical weaknesses
  • Methodological novelty is somewhat incremental relative to FLAC. The proposed algorithm largely retains the flow actor, kinetic-energy regularization, reparameterized policy update, and off-policy learning framework of FLAC, while incorporating an established augmented-Lagrangian constraint term. Although this is a reasonable and useful extension to safe RL, the algorithmic contribution is primarily the integration of existing components rather than a fundamentally new policy optimization mechanism. The authors could more clearly articulate which insights or design principles are specific to constrained flow optimization and cannot be directly obtained from the existing FLAC framework.

  • The theoretical guarantees do not directly establish constraint satisfaction during online learning. The theoretical analysis characterizes the constrained bridge and its optimal action distribution for fixed critics and Lagrange multipliers. However, the practical algorithm involves simultaneously evolving critics, actor parameters, and dual variables. Consequently, the safety-tilting result does not directly guarantee satisfaction of the original episodic constraint, nor does it ensure safe exploration during training. The paper acknowledges these limitations, but a clearer discussion of the relationship between the theoretical results and practical constraint satisfaction would further strengthen the contribution.

  • The specific benefits of flow-based multimodal representations are not fully isolated. One motivation for using flow policies is their ability to represent multimodal action distributions induced by competing reward and safety objectives. However, the experiments do not directly demonstrate such multimodality or evaluate mode coverage under safety constraints. Moreover, the Gaussian-mixture baseline achieves performance relatively close to RAFALE on several tasks. Since these actor families also use different regularizers, the comparisons do not fully isolate the contribution of the flow representation itself.

  • Safety evaluation primarily focuses on final-policy feasibility. Although RAFALE satisfies the mean final cost budgets across all seven tasks, some training trajectories exhibit substantial transient constraint violations. Additionally, final feasibility is not guaranteed for every individual seed. While the paper appropriately acknowledges the distinction between final-policy feasibility and safe exploration, a more comprehensive characterization of training-time constraint violations would provide a clearer picture of its practical safety properties.

Questions for authors

Please refer to the Weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 14PQ (1/3): insights specific to constrained flow optimization

W1 (insights specific to constrained flow optimization, beyond FLAC). We thank the reviewer for asking us to separate our insights from FLAC's. RAFALE does build on FLAC's flow actor and kinetic regularizer. The constrained setting, however, raises a question that FLAC does not face: which generated actions does a constraint change, and by how much? FLAC's results describe the action distribution as a whole, whereas this question concerns individual actions. Our analysis answers it in two steps.

1. An exact link between path space and action space, for each source draw. The kinetic regularizer acts on the generation path, whereas reward and cost are evaluated at the generated action. FLAC connects the two at the level of the whole action distribution, through bounds on its deviation from the reference (its Eqs. (6)–(7)) and through a tilt of that distribution (its Proposition 3.1).

Theorem 1 gives an exact equivalence for each source draw x0x_0: the path-space problem is equivalent to an action-space problem that balances the terminal potential and the displacement α2∥a−x0∥2\frac{\alpha}{2}\lVert a-x_0\rVert^2 against the entropy of the actions generated from x0x_0. Because a constraint acts on individual actions, this per-draw form is needed to state which actions it changes.

2. The effect of the constraint. Applying Theorem 1 to the reward-only potential −Qr-Q_r and to the augmented potential LAL_A, with the same source draw and the same regularizer, isolates the effect of the constraint (Theorem 2, and Proposition 6 in Appendix B.3 of the revised PDF). With the threshold τ=h−λ/ρ\tau=h-\lambda/\rho, where hh is the budget bb expressed in the critic's units, this effect has two properties:

  • Selective. Within each source draw, actions whose estimated cost is below τ\tau keep their relative probabilities, and an action above τ\tau is suppressed increasingly with its excess. At an optimum of the noise-free update that RAFALE trains, a draw keeps its reward-only action if that action is below τ\tau; otherwise, it can move only to an action with a smaller excess.
  • Proportionate. The penalty behind this reweighting has gradient ρ[Qc(s,a)−τ]+∇aQc(s,a)\rho[Q_c(s,a)-\tau]_+\nabla_aQ_c(s,a), which is exactly the cost term of the implemented actor gradient. The cost-gradient contribution of each generated action is thus weighted in proportion to its own excess. The standard Lagrangian, in contrast, gives every action the same weight λ\lambda, including actions below the threshold, and the hinge gives a constant weight above it.

In constrained flow optimization, the constraint thus acts on each generated action separately: its cost-gradient weight vanishes below the threshold and grows in proportion to the excess above it. The paper's Table 3 shows the consequence with the flow actor and the regularizer fixed: the augmented objective has a lower peak cost than the standard and hinge variants on all three tasks. The revised PDF states this relation in Section 4.2 ("Relation to FLAC") and adds Proposition 6; new text is blue.

AuthorsOpenReview ↗

Response to Reviewer 14PQ (2/3): what is guaranteed about the constraint during online learning

W2 and W4 (theory and constraint satisfaction during online learning; training-time violations). Thank you for raising these two points about safety during learning. We answer them together, because both concern the same question: what can be guaranteed about the cost while the policy is still being learned. Our answer has two parts.

1. The problem constrains the learned policy, not the intermediate ones. The CMDP constrains the expected episode cost of the learned policy, Eπ[C]≤b\mathbb{E}_\pi[C]\le b, and does not require every intermediate policy to satisfy the budget. As you note, final-policy feasibility is therefore distinct from safe exploration. In our locomotion tasks, the reward grows with forward speed, whereas the cost counts the steps above a speed limit. Learning a fast gait means approaching this limit, and policies cross it while the actor, the critics and the multiplier are still adapting.

The revised PDF now characterizes these violations: Table 13 (Appendix E.5) reports every method's peak evaluated cost during training next to its number of final policies within budget. RAFALE's peaks are high (for example, 174.6 on Hopper against a budget of 10) and occur while its multiplier is still zero in almost every run.

For the learned policy, the multiplier links the theory to the budget. Theorems 1–3 describe one actor update for a fixed multiplier: they state which generated actions the constraint changes, and by how much. The multiplier itself is updated from the measured cost of completed episodes (Eq. 44), and it keeps rising as long as this cost exceeds the budget bb.

Proposition 7 (Appendix D of the revised PDF) makes this link precise: in the long run, the measured cost can stay above budget on average only if the multiplier saturates at its cap, which never happened in RAFALE's locomotion runs.

The trained networks respond as the theory describes: in a frozen-critic experiment (Table 18 of the revised PDF), raising the multiplier from 0 to 1 and to 2.19 lowers the mean estimated cost of the generated actions from 0.99 to 0.63 and 0.53, in every run. By the end of training, 34 of RAFALE's 35 final policies are within budget, more than for any baseline.

2. A guarantee at every stage of learning is an open problem. Without prior knowledge, such as a known safe policy, or an assumed exact or conservative critic, guaranteeing the budget at every stage of learning, or the feasibility of every run's final policy, remains an open problem for deep off-policy safe RL, not only for RAFALE. Our theorems make no such claim.

In short, RAFALE addresses the constraint that the problem defines, the cost of the learned policy, and meets it more reliably than the baselines. Safety at every stage of learning lies outside this formulation and remains open.

AuthorsOpenReview ↗

Response to Reviewer 14PQ (3/3): isolating the multimodal representation

W3 (the benefit of the flow's multimodal representation is not isolated). We are grateful for this suggestion, which led us to test mode coverage under a constraint directly. We added two experiments.

Mode coverage under a safety constraint. To test mode coverage directly, we built a constrained version of FLAC's multi-goal task, with eight goals on a ring, two of which are unsafe. Without a budget, the task tests whether an actor covers all eight goals; with a budget (b=0.02b=0.02), whether it keeps the six safe goals and leaves the unsafe ones.

We compare four actors (the rows of Table R2): RAFALE's flow; a Gaussian actor, X1=μ+σ⊙x0X_1=\mu+\sigma\odot x_0, with the same source and kinetic regularizer; and an eight-component Gaussian mixture (GMM), once with the kinetic regularizer and once with SAC's entropy regularizer. Under the budget, all four use the same augmented Lagrangian and multiplier update. Figure 1 of the revised PDF plots actions sampled from the trained policies.

Table R2. Constrained multi-goal task, five seeds. An action is on a goal when it lies within distance 1 of it, and a goal is covered when at least 1/12 of the evaluation actions are on it, in all five runs. A run is within budget when the one-sided 95% upper confidence bound on its expected cost is at most bb. Bold blue / green: highest / second-highest reward.

Actor + regularizerNo budget: goals coveredNo budget: rewardb=0.02b=0.02: safe goals coveredb=0.02b=0.02: rewardb=0.02b=0.02: actions on unsafe goalsRuns within budget
Flow + kinetic (RAFALE)8/80.983\color{#0046A8}{\mathbf{0.983}}6/60.986\color{#0046A8}{\mathbf{0.986}}0.03%5/5
Gaussian + kinetic0/80.4604/60.5060%5/5
GMM + kinetic0/80.4604/60.5050%5/5
GMM + entropy8/80.976\color{#38A600}{\mathbf{0.976}}6/60.969\color{#38A600}{\mathbf{0.969}}1.1%4/5

Under the budget, the flow covers all six safe goals and places almost no actions on the unsafe ones. The Gaussian actor shares the flow's objective and differs only in its representation: its map can only shift and stretch the source into a rectangle, so it reaches four of the six safe goals at about half the reward.

The mixture with the kinetic regularizer does no better, because its eight components converge to the same map as the Gaussian actor. With the entropy term, the mixture also covers the six safe goals, but it keeps more actions on the unsafe goals and earns less reward.

The same regularizer on the benchmark. The GMM's reward is indeed close to RAFALE's on Hopper and Walker2d. On Ant and Swimmer, to which we extended Table 4 of the submission, the flow earns more reward than the GMM (3427 against 3278, and 93 against 56) at lower cost. To remove the difference in regularizers that you point out, we trained a Gaussian actor and a one-step residual actor, X1=x0+vθ(s,x0,0)X_1=x_0+v_\theta(s,x_0,0), with RAFALE's kinetic regularizer.

Table R1. Actors trained with RAFALE's objective and kinetic regularizer: final reward / cost (runs within the budget of 10), five seeds each. Gaussian: X1=μθ(s)+σθ(s)⊙x0X_1=\mu_\theta(s)+\sigma_\theta(s)\odot x_0. Bold blue / green: highest / second-highest reward among entries with mean cost within budget; red: mean cost above budget.

TaskGaussian + kineticResidual + kineticMidpoint flow + kinetic (RAFALE)
Hopper1658\color{#0046A8}{\mathbf{1658}} / 0.4 (5/5)1399 / 9.9 (4/5)1655\color{#38A600}{\mathbf{1655}} / 0.3 (5/5)
Walker2d3124 / 2.0 (5/5)3155\color{#38A600}{\mathbf{3155}} / 7.6 (3/5)3177\color{#0046A8}{\mathbf{3177}} / 1.3 (5/5)
Humanoid6351\color{#38A600}{\mathbf{6351}} / 8.4 (4/5)6504 / 12.3\color{#B22222}{12.3} (2/5)6513\color{#0046A8}{\mathbf{6513}} / 3.2 (5/5)
Ant3297 / 6.2 (4/5)3340\color{#38A600}{\mathbf{3340}} / 4.3 (5/5)3427\color{#0046A8}{\mathbf{3427}} / 4.4 (5/5)
Swimmer81\color{#38A600}{\mathbf{81}} / 4.1 (4/5)70 / 1.6 (5/5)93\color{#0046A8}{\mathbf{93}} / 5.4 (5/5)
Runs within budget22/2519/2525/25

With the regularizer fixed, only the flow keeps every run within budget on all five tasks, so this difference comes from the representation. Both experiments thus isolate the benefit of the flow: it covers every safe mode, and it satisfies the budget most reliably. Both are in the revised PDF (Figure 1, Table 4 and Appendix E.6).

Thank you for the detailed responses and additional experiments. The revisions meaningfully strengthen the empirical evidence and clarify the scope of the claims. I therefore maintain my original rating.

Reviewer_YYnBOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an off-policy safe-RL algorithm which differentiates an action-level augmented Lagrangian directly through a flow actor, avoiding the auxiliary noisy-score estimation. Building on FLAC's kinetic-energy regularization, it relates a constrained one-ended Schrödinger bridge to a source-conditioned Gibbs action distribution and derives a vanishing-noise, displacement-regularized map objective. A one-step midpoint actor optimizes this objective using two velocity-network evaluations. Across seven Safety-Gymnasium tasks, the method achieves competitive rewards and final mean evaluation costs within budget on all tasks.

Critical strengths
  1. A coherent and computationally simple constrained generative-policy update. RAFALE combines direct pathwise differentiation of the augmented objective with density-free kinetic regularization, removing the need for auxiliary score estimation and candidate-action importance sampling. The conditional bridge analysis explains the safety tilt and connects it to the implemented displacement-regularized objective, making the method more than an unexplained combination of losses.
  2. Strong final-policy feasibility with an informative mechanism ablation. The seven-task results and 34/35 feasible final seeds support a useful reward–cost trade-off. The threshold-matched hinge control particularly supports a benefit from the above-threshold gradient profile, rather than threshold activation alone, without establishing a universal causal mechanism or safe exploration.
Critical weaknesses
  1. The safety ablation does not isolate the proposed mechanism. With λ initialized to zero and multiplier updates delayed for 200k environment steps, the standard and hinge controls initially have zero cost-gradient weight, while the augmented penalty can already act above h. Table 3 therefore mixes gradient-shape benefits with earlier penalty activation. Post-warm-up comparisons and sensitivity to the dual schedule would clarify whether the peak-cost advantage persists.
  2. Robustness of budget satisfaction to tuning remains unclear. RAFALE achieves strong final feasibility, but uses task-specific multiplier caps, update gains, and nonzero multiplier-state warm-starts on two tasks (Table 7). These choices directly affect the reward–cost trade-off, yet their selection procedure and sensitivity are not established. This leaves uncertainty about how reliably the method would satisfy budgets on a new task without substantial retuning. A sensitivity analysis around the reported settings would strengthen the central empirical claim.
Questions for authors
  1. How were the task-specific multiplier caps, gains, and integral warm-starts selected, and what tuning budget was allocated to each method? How much do the main results change under modest perturbations or removal of the nonzero warm-starts?
  2. Given the native-regularizer difference in Table 4, what evidence distinguishes benefits of flow parameterization from kinetic regularization? A matched source-conditioned residual actor with the same displacement penalty would be a targeted way to test whether the midpoint construction adds value.
3 comments
AuthorsOpenReview ↗

Response to Reviewer YYnB (1/2): separating the augmented penalty from earlier activation

W1 (the Lagrangian ablation mixes the gradient profile with earlier penalty activation). We thank the reviewer for identifying this confound so precisely. In the submitted runs, the multiplier is not updated during the first 200k steps (the dual warm-up of the paper's Table 5). During this period, the standard and hinge variants have zero cost weight, whereas the augmented variant already acts above the budget.

We ran both proposed checks. First, we recomputed the peak cost and the integrated excess above budget of the submitted runs from the evaluations after the dual warm-up only. Second, we reran the three variants without the dual warm-up on five tasks, so that every multiplier can change from the 6,000th transition on.

Table R4. Peak evaluated cost / integrated excess above budget (budget 10), five seeds each: after the 200k-step dual warm-up of the submitted runs, and in new runs without it. Bold: lowest value of each metric in each row.

SettingAugmentedHingeStandard
Hopper, after 200k143.1 / 75.3532.6 / 150.1340.4 / 147.0
Walker2d, after 200k68.1 / 10.9190.8 / 26.6214.5 / 48.4
Hopper, no dual warm-up163.1 / 88.7617.3 / 144.4362.4 / 121.3
Walker2d, no dual warm-up76.2 / 15.863.5 / 17.799.4 / 22.5
HalfCheetah, no dual warm-up114.3 / 21.3867.5 / 54.8755.2 / 54.8
Ant, no dual warm-up124.6 / 47.4393.2 / 59.3481.8 / 90.9
Swimmer, no dual warm-up16.9 / 0.823.3 / 6.964.7 / 8.6

The advantage persists in both checks: the augmented variant has the lowest value in 13 of the 14 comparisons. An earlier penalty does not explain it: in the Hopper, HalfCheetah and Ant reruns, when the hinge and standard variants reach their peak cost, their multipliers are already several times larger than the augmented one.

To isolate the gradient profile from everything else, we also trained fresh actors against the frozen critics of a Hopper checkpoint, at six fixed multiplier values (Table 18 of the revised PDF). At every multiplier and in every run, the augmented weight lowered the upper tail of the generated actions' estimated cost relative to both other variants.

With the critics and the multiplier fixed, the weight is the only remaining difference: above the threshold τ=h−λ/ρ\tau=h-\lambda/\rho, where hh is the budget bb expressed in the critic's units, the augmented weight ρ[Qc−τ]+\rho[Q_c-\tau]_+ grows with an action's estimated cost, whereas the standard and hinge weights equal the constant λ\lambda. The gap therefore comes from the cost-dependent weight, not from an earlier penalty.

On Humanoid, every variant's cost stays near zero during the dual warm-up, so the peaks in the paper's Table 3 already occur after it.

Together, these checks support the augmented Lagrangian of the paper: it controls the cost better because its weight grows with each action's excess, and this holds with or without the dual warm-up. Table R4 is Table 16 of the revised PDF, and the frozen-critic comparison is its Table 18; new text is blue.

AuthorsOpenReview ↗

Response to Reviewer YYnB (2/2): selection and sensitivity of the dual settings; the matched residual actor

W2 and Q1 (sensitivity, selection and tuning of the dual settings). We appreciate these questions about the dual settings. In short, the main results do not hinge on them.

Sensitivity. The warm start is the value from which the multiplier starts at its first update, and only Swimmer and Walker2d use a nonzero one (the paper's Table 7). It is a different setting from the 200k-step dual warm-up of W1, which all tasks share. Table R5 removes the warm start and changes the multiplier's learning rate fourfold, with all other settings unchanged.

Table R5. Final reward, cost and runs within the budget of 10 (five seeds each) when the multiplier's warm start is removed or its learning rate is changed.

TaskDual settingRewardCostRuns within budget
Walker2dAs in the paper31771.35/5
Walker2dNo warm start31839.93/5
Walker2dNo warm start, learning rate ×431492.05/5
Walker2dLearning rate ×431471.05/5
SwimmerAs in the paper935.45/5
SwimmerNo warm start1085.44/5

Across these changes, the mean cost stays within budget, and the reward stays within 1% on Walker2d and rises on Swimmer. On Walker2d, the multiplier rises more slowly without the warm start and two runs end above budget; a fourfold learning rate keeps all five within budget without it. The other five tasks have no warm start to remove.

With both warm starts removed, RAFALE therefore still has more final policies within budget than any baseline (31 of 35, against 22 to 28 for the baselines).

Selection. All tasks start from shared defaults, and a task receives one extra term only if a development run ended above budget: a warm start (Swimmer, Walker2d), a larger learning rate (CarGoal) or a proportional gain (Humanoid). The task-specific caps were never reached in the locomotion runs. These settings change how fast the multiplier reacts, not the budget it targets: the multiplier keeps rising as long as the measured episode cost exceeds the budget.

Tuning. We tuned all baselines for our Safety-Gymnasium tasks, starting from their released implementations and reference configurations. The original studies used different environment versions or did not cover our full task suite, so we made task-specific adjustments where needed to obtain competitive performance, including for RCRL and RESPO (Appendix E.3).

The paper's Table 2 reports every method at its final checkpoint. As an additional comparison that favors the baselines, we selected for CAL and ALGD, after the fact, their highest-reward checkpoint within budget (Table 14 of the revised PDF). Even so, they earn less reward than RAFALE's final policies on six of the seven tasks.


Q2 (flow parameterization or kinetic regularization; a matched residual actor). Thank you for suggesting the matched residual actor, which we trained on five tasks. This actor, X1=x0+vθ(s,x0,0)X_1=x_0+v_\theta(s,x_0,0), keeps RAFALE's objective, displacement penalty and settings, so it differs from RAFALE only in the midpoint construction: it evaluates the velocity network once instead of twice. Table R1 reports it, together with a Gaussian actor trained with the same penalty.

Table R1. Actors trained with RAFALE's objective and kinetic regularizer: final reward / cost (runs within the budget of 10), five seeds each. Gaussian: X1=μθ(s)+σθ(s)⊙x0X_1=\mu_\theta(s)+\sigma_\theta(s)\odot x_0. Bold blue / green: highest / second-highest reward among entries with mean cost within budget; red: mean cost above budget.

TaskGaussian + kineticResidual + kineticMidpoint flow + kinetic (RAFALE)
Hopper1658\color{#0046A8}{\mathbf{1658}} / 0.4 (5/5)1399 / 9.9 (4/5)1655\color{#38A600}{\mathbf{1655}} / 0.3 (5/5)
Walker2d3124 / 2.0 (5/5)3155\color{#38A600}{\mathbf{3155}} / 7.6 (3/5)3177\color{#0046A8}{\mathbf{3177}} / 1.3 (5/5)
Humanoid6351\color{#38A600}{\mathbf{6351}} / 8.4 (4/5)6504 / 12.3\color{#B22222}{12.3} (2/5)6513\color{#0046A8}{\mathbf{6513}} / 3.2 (5/5)
Ant3297 / 6.2 (4/5)3340\color{#38A600}{\mathbf{3340}} / 4.3 (5/5)3427\color{#0046A8}{\mathbf{3427}} / 4.4 (5/5)
Swimmer81\color{#38A600}{\mathbf{81}} / 4.1 (4/5)70 / 1.6 (5/5)93\color{#0046A8}{\mathbf{93}} / 5.4 (5/5)
Runs within budget22/2519/2525/25

With the regularizer fixed, the residual actor leaves runs above budget on three tasks and earns less reward on all five, whereas the midpoint flow keeps every run within budget. The midpoint construction therefore matters beyond the kinetic regularizer. Tables R1 and R5 are Tables 4 and 8 of the revised PDF.

Thank you all for taking the time to run these specific experiments.

  1. The runs without the dual warm-up, especially the frozen-critic ablation, are pretty convincing. You've successfully isolated the cost-dependent weight as the true driver of safety performance, which resolves my concerns about early penalty activation.
  2. Seeing the method retain 31/35 safe seeds even after stripping out the nonzero warm-starts gives much more confidence in how well the algorithm will generalize to new tasks. Also appreciate the transparency around your default-first tuning strategy.
  3. The matched residual actor baseline was exactly the test this needed. Seeing the midpoint flow keep every run within budget while the residual architecture struggles on three tasks proves that your structural choice fundamentally matters beyond just the kinetic regularizer.

You've addressed my critiques with well-designed experiments. My initial Weak Accept rating already reflected a positive view of the core method, so I will be maintaining that score.

Reviewer_oSeYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an off-policy safe RL method with a flow policy. The actor is trained using augmented Lagrangian which backpropagates through the flow sampler. This paper uses a kinetic-energy regularizer instead of entropy same as FLAC. The authors show that this regularizer corresponds to an entropy-regularized problem in action space and becomes a displacement penalty in the noise-free limit, which is what the algorithm optimizes. The results show that RAFALE stays within the cost budget on all seven Safety-Gymnasium tasks with competitive reward.

Critical strengths
  1. The idea and storyline is clearly delivered, and the related work is thorough.
  2. The authors show necessary derivations to support the idea and theorem proposed. The assumptions are also clearly stated.
  3. The actor update is cheaper than previous approaches which require score estimation.
Critical weaknesses
  1. The statement of maximum entropy is not very clear to me. The entropy term has coefficient αϵ\alpha\epsilon, which is zero in the implemented noise-free version (i.e., no entropy term). Randomness only comes only from x0x_0, and the injectivity bound is weak when α\alpha is small. Could you please analyze the algorithm when ϵ>0\epsilon> 0 and show the generation when there is noise.
  2. The method is motivated directly by FLAC, but it seems that FLAC is not a clear baseline.
  3. It is not clear that the gains of the proposed approach are coming from the flow architecture or the regularizer. The motivation is multimodal policies, but the tasks may be unimodal. It would be better if authors can show the results on a toy constrained env with several safe modes, and plot the learned action distribution.
  4. Table 4 changes the actor and the regularizer together. A Gaussian actor with the same displacement penalty would show where the gain comes from.
  5. Could you please also report the peak cost during training against the baselines?
Questions for authors

Please see the comments on weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer oSeY (1/5): no entropy term at ε = 0: the kinetic regularizer in path space and in action space

W1 (no entropy term at ε=0\varepsilon=0; analysis and generation for ε>0\varepsilon>0). We are grateful for your close reading of the maximum-entropy statement. You are right that the implemented actor has no entropy term, and we do not claim that it maximizes entropy. Our claim concerns the effect of the kinetic regularizer on the generated actions.

Path space. From a source draw x0x_0, the generation process dXt=v dt+ε dWtdX_t=v\ dt+\sqrt{\varepsilon}\ dW_t produces the action a=X1a=X_1. The actor update penalizes the kinetic energy of this process:

min⁡v E[LA(s,X1)+α2∫01∥v∥2dt].\min_v\ \mathbb{E}\Big[L_A(s,X_1)+\frac{\alpha}{2}\int_0^1\lVert v\rVert^2dt\Big].

The second term is the kinetic regularizer.

Action space, ε>0\varepsilon>0. Let qq denote the distribution of the actions generated from one source draw x0x_0. For every ε>0\varepsilon>0, Theorem 1 shows that the problem above is equivalent, for each x0x_0, to

min⁡q Ea∼q[LA(s,a)+α2∥a−x0∥2]−αεH(q).\min_q\ \mathbb{E}_{a\sim q}\Big[L_A(s,a)+\frac{\alpha}{2}\lVert a-x_0\rVert^2\Big]-\alpha\varepsilon H(q).

In action space, the kinetic regularizer thus consists of two terms. The displacement term α2∥a−x0∥2\frac{\alpha}{2}\lVert a-x_0\rVert^2 keeps each action close to its own source draw, and the entropy term αεH(q)\alpha\varepsilon H(q) spreads the actions generated from that draw. The minimizer of this problem is the noisy generator of Theorem 2; the upper panels of Figure 3 in the revised PDF (Appendix B.6) show it for one source draw.

Action space, ε=0\varepsilon=0. The noise level multiplies the entropy term only. As ε→0\varepsilon\to0, the entropy term vanishes while the displacement term is unchanged. Each source draw then generates a single action T(x0)T(x_0), and the problem becomes (Theorem 3)

min⁡T Ex0[LA(s,T(x0))+α2∥T(x0)−x0∥2],\min_T\ \mathbb{E}_{x_0}\Big[L_A(s,T(x_0))+\frac{\alpha}{2}\lVert T(x_0)-x_0\rVert^2\Big],

which is exactly the actor loss of RAFALE (Corollary 5). Setting ε=0\varepsilon=0 thus removes the entropy term and keeps the displacement term: each source draw generates one action, and the policy remains random because the source draw is random. The next comment explains why we implement this limit and compares generation with and without noise.

AuthorsOpenReview ↗

Response to Reviewer oSeY (2/5): why ε = 0 is implemented; generation with and without noise

W1, continued (why ε=0\varepsilon=0 is implemented; generation with and without noise). This comment completes our answer to W1.

Why we implement ε=0\varepsilon=0. For ε>0\varepsilon>0, training has to handle the entropy term, and each of the two ways of doing so has a drawback:

  1. In action space, the entropy H(q)=−Ea∼q[log⁡q(a)]H(q)=-\mathbb{E}_{a\sim q}[\log q(a)] requires the log-density of the generated actions, which a flow sampler does not readily provide. This is the same obstacle that motivates the kinetic regularizer at the start of Section 4.
  2. In path space, no density is needed, but each action is generated by simulating the noisy process; the generator in the example below uses ten Euler–Maruyama steps, and training differentiates through all of them.

At ε=0\varepsilon=0, neither drawback arises: a single midpoint step, with two network evaluations, gives the displacement term without approximation. We therefore set ε=0\varepsilon=0 and regularize the actor with the displacement term alone.

Generation with and without noise (Figure 3 of the revised PDF). The example has two safe modes, m1m_1 and m2m_2, and one unsafe mode. At three noise levels, Figure 3 shows two kinds of results:

  • Analytic (blue): the solution given by Theorems 2 and 3, computed without training.
  • Trained (yellow dots): the actions of trained actors, namely a ten-step noisy generator trained on the path-space objective for ε>0\varepsilon>0, and RAFALE's actor for ε=0\varepsilon=0.

The upper panels follow one source draw, and the lower panels show the whole policy. The table gives the share of actions at m1m_1 / m2m_2 in these panels.

Panels of Figure 3Resultε=0.1\varepsilon=0.1ε=0.01\varepsilon=0.01ε=0\varepsilon=0 (implemented)
Upper: one source drawAnalytic73.0% / 25.5%100.0% / 0.0%100% / 0%
Lower: the whole policyAnalytic47.6% / 45.6%48.2% / 46.4%48.2% / 46.4%
Lower: the whole policyTrained (mean of five seeds)50.9% / 41.1%51.1% / 46.8%51.0% / 44.4%

One source draw (upper panels). The source draw lies midway between the safe modes. At ε=0.1\varepsilon=0.1, it generates actions in both modes. As the noise decreases, these actions concentrate on m1m_1, and at ε=0\varepsilon=0 the draw generates a single action: the map from source draw to action is deterministic. RAFALE's trained actor reproduces this action in all five runs (yellow dots on the star).

The whole policy (lower panels). In contrast, the policy barely changes with the noise level: at ε=0\varepsilon=0, the analytic solution still places 48.2% of the actions at m1m_1 and 46.4% at m2m_2; after training, the shares are 51.0% and 44.4%. Each source draw now generates a single action, but the displacement term keeps that action close to its own source draw, so different source draws are sent to different modes.

The noise therefore only spreads the actions of each source draw. The multimodality of the policy comes from the source draws, whose randomness alone covers both safe modes at ε=0\varepsilon=0.

Small α\alpha. We agree that the injectivity bound weakens as α\alpha decreases: at ε=0\varepsilon=0, the displacement term is the only regularizer, and its weight is α\alpha. This dependence on α\alpha, however, is not specific to ε=0\varepsilon=0. With noise, the entropy weight αε\alpha\varepsilon shrinks with α\alpha as well, so α\alpha sets the strength of the regularization at every noise level, as the temperature does in SAC.

Taken together, these results show that ε=0\varepsilon=0 suffices in practice. Without noise, the policy still covers both safe modes (Figure 3), and RAFALE reaches competitive reward with mean final cost within budget on all seven tasks (the paper's Table 2). The noise-free actor is also the cheaper one to train: each action takes one midpoint step, with no action density and no simulated noisy path. The revised PDF states this in Section 4.3 ("Benefits of the noise-free limit") and adds the example as Appendix B.6 (Figure 3); new text is blue.

AuthorsOpenReview ↗

Response to Reviewer oSeY (3/5): FLAC as a baseline; a task with several safe modes

W2 (FLAC as a baseline). We thank the reviewer for raising the comparison with FLAC. As published, FLAC maximizes reward only and cannot enforce a cost budget, so it cannot serve as a safe-RL baseline in its original form. Its closest constrained counterpart is already in the paper: the "Standard" variant of Table 3, which keeps FLAC's flow actor and kinetic regularizer, adds a standard Lagrangian, and differs from RAFALE only in the Lagrangian of the actor loss.

Compared with this variant, RAFALE has a lower peak cost on all three tasks; on Hopper and Walker2d, it also has a higher reward and a lower integrated excess above budget. On Humanoid, only RAFALE ends with all five final policies within budget.

This comparison shows what RAFALE adds to FLAC: with the same flow actor and kinetic regularizer, the augmented Lagrangian controls the cost better than the standard one. The revised PDF states this correspondence in Section 6.2.


W3 (a constrained task with several safe modes; learned action distributions). We appreciate this suggestion and built the test you describe, which has several safe modes by construction: a constrained version of FLAC's multi-goal task, with eight goals on a ring, two of which are unsafe. Without a budget, the task tests whether an actor covers all eight goals; with a budget (b=0.02b=0.02), whether it keeps the six safe goals and leaves the unsafe ones.

We compare four actors (the rows of Table R2): RAFALE's flow; a Gaussian actor with the same source and objective; and an eight-component Gaussian mixture (GMM), once with the same objective and once with SAC's entropy term. Under the budget, all four use the same augmented Lagrangian and multiplier update. Figure 1 of the revised PDF plots actions sampled from the trained policies.

Table R2. Constrained multi-goal task, five seeds. An action is on a goal when it lies within distance 1 of it, and a goal is covered when at least 1/12 of the evaluation actions are on it, in all five runs. A run is within budget when the one-sided 95% upper confidence bound on its expected cost is at most bb. Bold blue / green: highest / second-highest reward.

Actor + regularizerNo budget: goals coveredNo budget: rewardb=0.02b=0.02: safe goals coveredb=0.02b=0.02: rewardb=0.02b=0.02: actions on unsafe goalsRuns within budget
Flow + kinetic (RAFALE)8/80.983\color{#0046A8}{\mathbf{0.983}}6/60.986\color{#0046A8}{\mathbf{0.986}}0.03%5/5
Gaussian + kinetic0/80.4604/60.5060%5/5
GMM + kinetic0/80.4604/60.5050%5/5
GMM + entropy8/80.976\color{#38A600}{\mathbf{0.976}}6/60.969\color{#38A600}{\mathbf{0.969}}1.1%4/5

Under the budget, the flow covers all six safe goals and places almost no actions on the unsafe ones. The Gaussian actor shares the flow's objective and differs only in its representation: its map can only shift and stretch the source into a rectangle, so it reaches four of the six safe goals at about half the reward. The mixture does no better with the kinetic regularizer: its eight components converge to the same map as the Gaussian actor.

The 0% of these two actors on the unsafe goals reflects their narrow coverage rather than better safety, since their expected cost is about twice the flow's (0.018 against 0.009). With the entropy term, the mixture also covers the six safe goals, but it keeps more actions on the unsafe goals and earns less reward.

With the kinetic regularizer fixed, the flow thus covers all six safe goals where the Gaussian and GMM actors cover four; compared with the entropy-regularized GMM, RAFALE covers the same six goals with fewer actions on the unsafe ones. The task is now in the revised PDF (Figure 1, Section 6.2 and Appendix E.6).

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you very much for your careful reading and constructive suggestions. As the discussion period draws to a close, we summarize the main revisions below. The additions are marked in blue in the revised PDF.

1. Contribution beyond FLAC and the noise-free actor

  • What the constraint changes. FLAC’s results describe the overall action distribution through bounds and a globally normalized tilt. Our analysis gives an exact action-space problem for each source draw, with its conditional solution normalized separately as in SoftGAC’s fixed-base formulation (Theorems 1–2; Section 4.2). Within each draw, actions below the multiplier-dependent threshold in estimated cost retain their relative probabilities, while actions above it are increasingly downweighted as their excess grows. The corresponding cost-gradient weight is zero below the threshold and proportional to the excess above it.
  • Connection to the trained actor. Theorem 3 gives the limiting map objective, which RAFALE’s midpoint loss evaluates exactly (Corollary 5). Proposition 6 shows that an optimal noise-free map preserves a draw’s unique reward-only action when it lies below the threshold; if the action changes, its excess is strictly smaller than that of the reward-only action. We also explained why we train this noise-free objective and added a positive-noise comparison: the noise-free policy still covers both safe modes in the new example (Section 4.3; Appendix B.6).

2. Additional ablations and multimodal experiments

  • Actor and regularizer. On five tasks, we compared Gaussian and GMM actors with entropy regularization, as well as Gaussian, one-step residual, and flow actors with RAFALE’s kinetic regularizer (Table 4). With the objective and kinetic regularizer fixed, all 25 flow runs yield final policies within budget, against 22 for the Gaussian actor and 19 for the residual actor.
  • Multimodal action distributions. We added a constrained multi-goal task with six safe and two unsafe goals (Figure 1; Table 19). With the kinetic regularizer fixed, the flow covers all six safe goals, whereas Gaussian and GMM actors cover four. The entropy-regularized GMM also covers six, but places 1.1% of actions on unsafe goals, compared with 0.03% for RAFALE.
  • Augmented objective. We compared the three Lagrangian variants both after the dual warm-up and in new runs without it (Table 16). The augmented objective has the lowest peak cost or integrated excess in 13 of 14 comparisons. A separate experiment with frozen critics and a fixed multiplier also shows lower upper-tail estimated costs (Table 18).

3. Baseline comparisons and budget compliance

  • Comparisons at feasible cost levels. RAFALE’s final policies earn higher reward on six of seven tasks than CAL and ALGD at their best feasible checkpoints (Table 14). On Hopper, retraining the baselines with a tighter budget lets CAL match RAFALE’s reward at a higher cost, while ALGD earns less reward (Table 15).
  • Sensitivity. Removing the multiplier’s warm start or increasing its learning rate fourfold keeps the mean final cost within budget in every tested setting (Table 8). With both warm starts removed, 31 of RAFALE’s 35 final policies remain within budget, against 22 to 28 for the baselines.
  • Cost during training. Table 13 compares every method’s peak evaluated cost during training and its number of final policies within budget. RAFALE has higher peaks on some tasks but ends with the most final policies within budget (34/35). Proposition 7 (Appendix D) bounds the long-run average measured cost by the budget, provided the multiplier’s integral state eventually stops being clipped at its upper cap.

We are deeply grateful for the reviewers’ time and hope these additions address the concerns raised. We would be happy to clarify any remaining questions before the discussion closes.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.