- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RAFALE, an off-policy actor–critic method for online safe reinforcement learning. It combines FLAC’s flow actor and kinetic-energy regularization with an action-level augmented-Lagrangian objective based on reward and cost critics. The actor is updated by differentiating through a midpoint sampler, avoiding the target-score estimation used by ALGD. The paper provides a generalized Schrödinger-bridge interpretation of the frozen actor update, including a conditional action-space formulation, safety-dependent re-weighting, and a vanishing-noise map objective. Experiments on seven Safety-Gymnasium tasks report competitive rewards and mean final costs within budget.
Critical strengths
-
A practical method with encouraging empirical results. RAFALE trains a constrained generative actor through direct pathwise differentiation without estimating a target score. Table 2 reports competitive reward–cost trade-offs, with mean final cost within budget on all seven tasks. The use of five training seeds and fixed endpoint checkpoints strengthens the empirical evaluation.
-
An informative constraint-objective ablation. Table 3 compares the augmented objective against both the standard Lagrangian and a hinge penalty sharing its activation threshold. This helps distinguish thresholding from the penalty’s growth above the threshold. The reductions in peak evaluated cost on Hopper and Walker2d support the usefulness of the augmented objective.
Critical weaknesses
-
The additional contribution over the closest prior work is not sufficiently established. Table 1 and Appendix A indicate that RAFALE inherits FLAC’s flow actor, kinetic regularization, and reparameterization-based update, while adopting ALGD’s action-level augmented-Lagrangian objective. The algorithm therefore appears to be a natural constrained extension of FLAC using an existing safety objective. Such a combination can be valuable, but the paper does not sufficiently explain the nontrivial technical obstacle involved or which theoretical insights go beyond instantiating the existing bridge framework with this terminal potential. This leaves the significance of the incremental contribution unclear.
-
The experiments do not sufficiently establish the source and extent of the additional benefit. First, Table 4 shows similar Flow/GMM rewards on Hopper (1655 versus 1645) and Walker2d (3177 versus 3190), with all five final seeds feasible for both methods on both tasks. Second, the actor-family comparison also changes the regularizer: Flow uses kinetic regularization, whereas Gaussian/GMM actors use log-density regularization. As acknowledged in Appendix E.4, this prevents attributing the differences specifically to the policy representation, despite multimodal expressivity being a central motivation. Finally, Table 2 compares selected reward–cost operating points rather than comparable cost levels. For example, on Hopper, CAL and ALGD obtain rewards of 1664 and 1702 with costs of 12.2 and 15.7, while RAFALE obtains reward 1655 with cost 0.3, against a budget of 10. These results are encouraging, but do not establish whether the baselines would remain competitive after stronger safety tuning.
Questions for authors
- What specific technical difficulty arises when extending FLAC with ALGD’s augmented-Lagrangian terminal objective? Which theoretical result provides an insight that is not a direct instantiation of the existing framework?
- What evidence separates the contribution of the flow representation from that of kinetic regularization, particularly given the close Flow/GMM results on Hopper and Walker2d?
- Do you have results for CAL and ALGD at comparable feasible cost levels? Such evidence would help determine whether the advantage persists after accounting for the reward–cost trade-off. Minor presentation suggestion, not a reason for my recommendation: FLAC is an important reference, but its full name is not provided at its first occurrence in either the abstract or the main text. Please expand the acronym to help readers unfamiliar with this work.
5 comments
Response to Reviewer q9VM (2/3): separating the flow representation from the kinetic regularizer
W2 and Q2 (flow representation versus kinetic regularization; close Flow/GMM results). We appreciate both observations and agree with them. First, Table 4 of the submission changes the actor and the regularizer together. Second, the flow and the GMM are close on Hopper and Walker2d. We therefore extended Table 4 to Ant and Swimmer, where the flow earns more reward than the GMM at lower cost (Table R1), and added the experiments below.
Comparable combinations. Each regularizer requires something that only some actors provide:
| Regularizer | Gaussian | GMM | Flow |
|---|---|---|---|
| Entropy (needs the action density) | paper's Table 4 | paper's Table 4 | no closed-form density¹ |
| Kinetic (needs a map from source draw to action) | new (Table R1) | new (Table R2)² | RAFALE |
¹The one-step flow generates an action without providing its density, which is why FLAC replaces the entropy term by kinetic energy. ²A mixture draws its component independently of the source draw, so its kinetic objective is the weighted average of its components' objectives. On the multi-goal task, its eight components converge to the same map as the Gaussian actor.
Regularizer fixed, actor varied.
Table R1. Final reward / cost (runs within the budget of 10), five seeds each. Gaussian: ; residual: ; flow: RAFALE's one-step midpoint map. In the columns marked †, the Hopper, Walker2d and Humanoid entries are from Table 4 of the submission, whereas the Ant and Swimmer entries are new. Bold blue / green: highest / second-highest reward among entries with mean cost within budget; red: mean cost above budget.
| Task | Gaussian + entropy† | GMM + entropy† | Gaussian + kinetic | Residual + kinetic | Flow + kinetic (RAFALE) |
|---|---|---|---|---|---|
| Hopper | 1408 / 7.6 (4/5) | 1645 / 1.5 (5/5) | / 0.4 (5/5) | 1399 / 9.9 (4/5) | / 0.3 (5/5) |
| Walker2d | 3151 / 3.5 (4/5) | / 1.4 (5/5) | 3124 / 2.0 (5/5) | 3155 / 7.6 (3/5) | / 1.3 (5/5) |
| Humanoid | / 8.1 (4/5) | 6503 / 8.5 (3/5) | 6351 / 8.4 (4/5) | 6504 / (2/5) | / 3.2 (5/5) |
| Ant | 3336 / 8.9 (3/5) | 3278 / 8.0 (4/5) | 3297 / 6.2 (4/5) | / 4.3 (5/5) | / 4.4 (5/5) |
| Swimmer | 48 / 4.2 (5/5) | 56 / 7.2 (5/5) | / 4.1 (4/5) | 70 / 1.6 (5/5) | / 5.4 (5/5) |
| Runs within budget | 20/25 | 22/25 | 22/25 | 19/25 | 25/25 |
With the kinetic regularizer fixed (last three columns), the flow is the only actor that keeps every run within budget on all five tasks. The regularizer alone does not produce this result: on Humanoid, the Gaussian actor's cost stays near 8 with either regularizer, whereas the flow lowers it to 3.2.
Separated safe modes. We test multimodal expressivity directly on a constrained version of FLAC's multi-goal task, with eight goals on a ring, two of which are unsafe (Figure 1 of the revised PDF). Under the budget, all actors use the same augmented Lagrangian and multiplier update.
Table R2. Constrained multi-goal task, five seeds. Coverage and budget criteria are defined in Appendix E.6 of the revised PDF. Bold blue / green: highest / second-highest reward.
| Actor + regularizer | No budget: goals covered | No budget: reward | : safe goals covered | : reward | : actions on unsafe goals | Runs within budget |
|---|---|---|---|---|---|---|
| Flow + kinetic (RAFALE) | 8/8 | 6/6 | 0.03% | 5/5 | ||
| Gaussian + kinetic | 0/8 | 0.460 | 4/6 | 0.506 | 0% | 5/5 |
| GMM + kinetic | 0/8 | 0.460 | 4/6 | 0.505 | 0% | 5/5 |
| GMM + entropy | 8/8 | 6/6 | 1.1% | 4/5 |
Under the budget, the flow covers all six safe goals and places almost no actions on the unsafe ones. The Gaussian actor and the GMM with the same kinetic regularizer cover four; the entropy-regularized GMM covers six but keeps more actions on the unsafe goals and earns less reward.
Together, these experiments separate the two factors: with the regularizer fixed, the flow is the component that covers every safe mode and keeps every run within budget. Both are in the revised PDF (Figure 1, Table 4 and Appendix E.6).
Response to Reviewer q9VM (3/3): CAL and ALGD at feasible cost levels
Q3 (CAL and ALGD at comparable feasible cost levels). Thank you for suggesting this comparison. The paper's Table 2 reports every method at its final checkpoint, without selecting checkpoints. For this question, we made two additional comparisons, and RAFALE keeps its advantage in both.
Best feasible checkpoint of the baselines, all seven tasks. For CAL and ALGD, we selected after the fact the evaluated checkpoint with the highest mean reward among those whose mean cost is within budget (Table R7). This selection favors the baselines, because RAFALE is still evaluated at its final checkpoint. Even so, CAL and ALGD earn less reward than RAFALE's final policies on six of the seven tasks. The exception is Hopper, which the second comparison examines.
Table R7. Reward / cost of CAL and ALGD at their final checkpoint (as in the paper's Table 2) and at their best feasible checkpoint, next to RAFALE's final checkpoint. The best feasible checkpoint is selected after the fact: among the evaluated checkpoints whose mean cost is within budget, the one with the highest mean reward (five seeds; budget 10, and 25 on CarGoal). Bold blue / green: highest / second-highest reward among entries with mean cost within budget; red: mean cost above budget.
| Task | CAL, final | CAL, best feasible | ALGD, final | ALGD, best feasible | RAFALE, final |
|---|---|---|---|---|---|
| Swimmer | 41 / | 43 / 10.0 | 52 / 2.9 | / 3.0 | / 5.4 |
| HalfCheetah | 2650 / | 2648 / 6.6 | 2817 / | / 8.2 | / 3.5 |
| Hopper | 1664 / | / 3.7 | 1702 / | / 3.9 | 1655 / 0.3 |
| Walker2d | 2894 / 6.3 | / 7.9 | 2920 / | 2915 / 6.6 | / 1.3 |
| Ant | 2815 / | / 8.9 | 2218 / | 1374 / 5.3 | / 4.4 |
| Humanoid | 5289 / 0.1 | / 0.0 | 4530 / 0.2 | 4756 / 0.1 | / 3.2 |
| CarGoal | 12.1 / 12.3 | 12.4 / 13.2 | 16.5 / 22.4 | / 24.5 | / 20.7 |
Stronger safety tuning on Hopper, the task cited in the review. We retrained CAL and ALGD with the tighter training budget , while still judging feasibility at .
Table R6. Hopper, five seeds, 2M steps: CAL and ALGD retrained with training budget ; feasibility is judged at in every row. Rows marked Table 2 are from the paper. Bold blue / green: highest / second-highest reward among entries with mean cost within budget; red: mean cost above budget.
| Method | Training budget | Reward | Cost | Runs within |
|---|---|---|---|---|
| RAFALE (Table 2) | 10 | 0.3 | 5/5 | |
| CAL (Table 2) | 10 | 1664 | 1/5 | |
| CAL | 5 | 4.8 | 5/5 | |
| ALGD (Table 2) | 10 | 1702 | 1/5 | |
| ALGD | 5 | 7.2 | 3/5 |
With , every CAL run is within budget, and CAL's reward falls to RAFALE's level at a higher cost. The gaits explain why: on Hopper, the reward mainly reflects the average forward speed, whereas the cost counts the steps above the speed limit. The retrained CAL matches RAFALE's average speed, hence the equal reward, but it exceeds the speed limit more often. ALGD loses reward and still leaves runs above the budget.
After stronger safety tuning, neither baseline overtakes RAFALE on Hopper, and RAFALE reaches this reward at the original budget with the lowest cost. RAFALE's advantage thus persists at comparable feasible cost levels. Tables R7 and R6 are Tables 14 and 15 of the revised PDF.
Minor (FLAC's full name). We are grateful for this note. The revised PDF now writes "Field Least-Energy Actor-Critic (FLAC)" at its first occurrence in the abstract and in the main text.
Response to Reviewer q9VM (1/3): contribution beyond combining FLAC with the augmented Lagrangian
W1 and Q1 (technical difficulty of the extension; which result is not a direct instantiation). We thank the reviewer for asking us to separate the adopted components from our own results. Forming the objective is indeed direct: FLAC's objective accepts any terminal potential (its Eq. (8)), so the augmented Lagrangian can be inserted as it is.
A direct instantiation, however, does not reveal how the constraint changes the generated actions. Instantiating FLAC's closed-form result (its Proposition 3.1) with the augmented Lagrangian yields a Boltzmann tilt of the action distribution as a whole, which has the form of ALGD's target. Two of our results go beyond it.
1. The constrained update for each source draw (Theorems 1 and 2). The tilt of FLAC's Proposition 3.1 is normalized once, over the whole distribution. Because a flow policy's source is fixed, we make the conditioning on the source draw explicit, following SoftGAC's fixed-base formulation (see our Appendix A), and normalize the tilt separately for each source draw before taking the action marginal.
Theorem 1 gives the exact action-space problem for each draw, and Theorem 2 compares its solutions for LA and for −Qr. Within each source draw, the constraint leaves the relative probabilities of actions below the threshold τ=h−λ/ρ unchanged and suppresses the other actions increasingly with their excess. Here, h is the budget b expressed in the critic's units. A globally normalized tilt does not express this property, because it does not track which action each source draw generates.
2. The actor that is trained (Theorem 3, Corollary 5, and Proposition 6 in the revised PDF). RAFALE's actor is noise-free. In this regime, FLAC provides a Wasserstein bound (its Eq. (7)) and notes that it gives no entropy guarantee. Theorem 3 shows that, as the noise vanishes, the optimal value converges to that of a map objective whose optimal maps send each source draw to a minimizer of LA(s,a)+2α∥a−x0∥2, and Corollary 5 shows that RAFALE's one-step midpoint loss evaluates this objective exactly.
Proposition 6 (Appendix B.3) shows that the selectivity survives the limit: a draw keeps its reward-only action if that action is below τ, and otherwise it can move only to an action with a smaller excess.
Insight for the algorithm. This analysis describes the implemented update itself. The penalty behind the reweighting of Theorem 2 has gradient ρ[Qc(s,a)−τ]+∇aQc(s,a), which is exactly the cost term of the actor gradient. The cost-gradient contribution of each generated action is therefore weighted in proportion to its own excess, whereas the standard Lagrangian gives every action the same weight λ and the hinge gives a constant weight above the threshold.
The paper's Table 3 isolates this difference with the flow actor and the regularizer fixed: the augmented objective has the lowest peak cost on all three tasks. The revised PDF states this relation in Section 4.2 ("Relation to FLAC") and adds Proposition 6; new text is blue.