acceptodds
Under review as a conference paper at ICLR 2027

SAFTER: Safety-Aware Filtered Training for Efficient and Robust Reinforcement Learning with Closed-Form Backup CBFs

Abstract

Safe reinforcement learning under bounded actuation requires more than preventing an immediate constraint violation: a currently admissible action can still leave insufficient control authority for future recovery. Backup control barrier functions (bCBFs) encode this finite-horizon recoverability, but sampled backup prediction introduces many simultaneous current, future, terminal, and input-admissibility conditions. Repeatedly solving the resulting optimization inside RL training is undesirable, and an opaque or approximated projection Jacobian can also give the actor a mismatched view of how its actions are actually executed. We propose SAFTER (Safety-Aware Filtered Training for Efficient and Robust reinforcement learning), which converts sampled backup recoverability into a differentiable analytical training map. SAFTER augments the applied command with first-order dynamics, composes current-state, backup-horizon, terminal, and continuous-model input-admissibility conditions with a hierarchical Log-Sum-Exp certificate, and reduces the resulting CBF condition to a single affine half-space with an exact minimum-intervention projection. In the sample efficiency study, SAFTER reaches 90% training success after 4.06M transitions, versus 5.85M for Vanilla SAC and 6.16–6.26M for conventional CBF-filtered SAC, while closely matching the 4.13M transition count of bCBF–LSE–QP; this indicates predictive recoverability, rather than the closed-form realization alone, is primarily associated with the sample-efficiency gain. Under Low-, Medium-, and High-Shift geometries, SAFTER achieves 98.8%, 88.6%, and 61.4% success, improving over Vanilla SAC by up to 67.8 percentage points and over CBF–CF by up to 40.2 points. With the same correction reward, replacing the exact projection Jacobian by the identity reduces High-Shift success from 61.4% to 6.2%. These results show that backup recoverability improves learning efficiency, while the analytical, differentiable realization is important for learning policies that remain effective under geometric distribution shift.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Safe Reinforcement Learning, Control Barrier Functions, Backup Control Barrier Functions, Quadruped Navigation, Recoverability-Aware Safety Filtering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N9tFOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1(−1)
Read review
Paper summary

This paper proposes a bCBF safe filter for SAC actors. It employs LSE to reduce the large number of constraints generated by the backup policy rollout and uses state augmentation to allow a closed-form solution to the QP problem.

Critical strengths

This paper proposes a viable safe filtering mechanism for SAC actions, which is an important problem in RL applications. The results clearly show that bCBF outperforms current sate CBF filters.

Critical weaknesses

The paper applies a combination of existing methods, which is not a problem on its own, but the resultant combination does not solve previously unaddressed problems or lead to many new findings.

  1. Formulating the safety-filter design problem and reducing the bCBF certificate to a single constraint is not newly proposed. The framework in [Rabiee & Hoagg, 2025], which the authors cite, is applicable but omitted here as an action filter, which arguably has a similar effect as SAFTER, as solving a 3-variable QP can be very fast. (See question 1 for more detail).

  2. This work also adds state augmentation from [Rabiee & Hoagg, 2024], which, under practical control hold, results in input admissibility issues: the applied action can be different from the certified safe action. It is questionable whether the latency gain, if any, associated with augmentation is worth trading off guaranteed input admissibility.

  3. The empirical results in the work do not support the necessity of using a learned SAC to propose the nominal action. An important baseline here is to avoid the huge RL training cost and use a simple reaching for target policy as if no obstacle exists (a heuristic policy similar to [Rabiee & Hoagg, 2024, Rabiee & Hoagg, 2025]).

  4. Overall, the presentation of this paper significantly limits its informativeness. Clear motivations for the design choices are not well established. Important limitations, such as the clipped action differing from the certified action, are delayed to Appendix C.3. The paper also lacks implementation details, e.g., whether CBF-CF and CBF-QP use state augmentation. No source code is provided to support reproducibility.


The added results and implementations during rebuttal raises further concerns rather than resolving my original points. I therefore lower my rating from 2 to 1 and leave the final assessment to the AC.

Questions for authors

Can you compare against [Rabiee & Hoagg, 2025]? The body velocity can be modelled as a state, with the velocity command as a hard-bounded input. This should also include using a heuristic nominal policy and an SAC nominal policy.

11 comments
AuthorsOpenReview ↗

Baseline Comparison and Methodology Trade-Offs (Part 1/2)

We thank the reviewer for the detailed feedback. To address your questions directly, we implemented a hard-bounded soft-minimum bCBF-QP adaptation based on Rabiee & Hoagg (2025)—modeling body velocity as a state evolving toward a hard-bounded velocity command input umin≤u≤umaxu_{min} \leq u \leq u_{max} optimized via a 3-variable QP—and evaluated it alongside SAFTER under both an obstacle-blind heuristic P-controller and SAC training, in addition to conducting a 29,896-step clipping audit and 60 real-world Unitree Go2 hardware trials.

1. Comparison with Rabiee & Hoagg (2025) and Necessity of a Learned SAC Policy (Q1, W1, W3)

To test (i) how SAFTER compares against the hard-bounded soft-minimum bCBF-QP adaptation of Rabiee & Hoagg (2025) and (ii) whether an obstacle-blind goal-reaching heuristic can replace RL training, we evaluated both safety filters on the same obstacle navigation map paired with either an obstacle-blind P-controller (steering directly toward the goal) or an SAC policy trained with that filter (Table 8).

Table 8: Navigation performance with heuristic (P-controller) and learned (SAC) nominal controllers.

Safety FilterNominal ControllerSuccess (%) ↑Collision (%) ↓Timeout (%) ↓
SAFTER (Ours)Obstacle-blind P-controller1.00.099.0
SAFTER (Ours)Learned SAC policy84.016.00.0
Rabiee & Hoagg (2025) (bCBF-QP adaptation)Obstacle-blind P-controller6.635.458.0
Rabiee & Hoagg (2025) (bCBF-QP adaptation)Learned SAC adaptation0.00.0100.0

These controlled comparisons establish two findings:

  1. Filtering an obstacle-blind heuristic is insufficient in multi-obstacle clutter (1.0%–6.6% success): When an obstacle blocks the line of sight to the goal, an obstacle-blind P-controller repeatedly commands forward velocity toward the obstacle, and SAFTER's stopping-backup filter decelerates the robot to rest to preserve safety (0.0% collisions, 99.0% timeouts). With the SAFTER filter held fixed, replacing the P-controller with a learned SAC policy trained through the filter increases success from 1.0% to 84.0% and cuts timeouts from 99.0% to 0.0%, as the learned nominal policy steers around obstacles rather than stalling in filter deadlocks.
  2. Comparison against the hard-bounded bCBF-QP adaptation of Rabiee & Hoagg (2025): Under the same evaluation protocol, the hard-bounded bCBF-QP adaptation achieves 6.6% success with the heuristic P-controller (58.0% timeouts, 35.4% collisions) and 0.0% success (100.0% timeouts) under SAC adaptation, while incurring nearly two orders of magnitude higher runtime latency during batched updates.

2. State Augmentation Trade-Off: Computational Speedup vs. Input Admissibility (W1, W2)

  • Why a 3-variable QP becomes a bottleneck in batched RL training: Off-policy RL with 256 parallel environments and minibatch actor/target updates (batch size 1024) evaluates the safety filter across hundreds of states per update over millions of transitions. At batch size 500, the hard-bounded 3-variable QP adaptation of Rabiee & Hoagg (2025) requires 193.60 ms total (186.00 ms synthesis) per call, whereas SAFTER's augmented analytical implementation requires 2.71 ms total (0.10 ms synthesis)—a 71.4× total speedup and 1,860× synthesis speedup. In our formulation-matched comparison against bCBF–LSE–QP (same augmented half-space, differing only in numerical vs. analytical synthesis), SAFTER reduces total latency by 2.30×–2.80× (Table 9).
  • Quantifying the post-clipping input-admissibility trade-off: In SAFTER, physical bounds umin≤u≤umaxu_{min} \leq u \leq u_{max} are encoded inside haug(z)h_\mathrm{aug}(z) via Id+(z),Id−(z)≥0I_d^{+}(z), I_d^{-}(z) \geq 0, steering uu away from saturation before clipping, while Eq. (16) enforces hard bounds on every executed command. Auditing 29,896 control steps shows: (i) on 97.6% of steps, clipping did not modify the command (altering commands beyond 10−710^{-7} on only 2.4% of steps, max normalized change 0.19); and (ii) using endpoint-equivalent surrogate inputs, clipping induced a CBF residual violation (tolerance 10−310^{-3} s⁻¹) on only 171 steps (0.57% of all 29,896 steps) among 718 previously satisfied clipped residuals.
AuthorsOpenReview ↗

Contributions and Additional Implementation Details (Part 2/2)

3. Contributions, Controlled Findings, and Real-World Hardware Validation (W1)

Our contribution is integrating finite-horizon backup recoverability into an end-to-end differentiable policy-training map—introducing a forward-invariant anisotropic elliptical terminal recoverability set for scan-fixed LiDAR navigation (Prop. B.1) and deriving the exact finite-rate action Jacobian (Eq. 17) for SAC training. Experiments establish three findings (Table 9):

  1. Predictive recoverability drives RL sample efficiency: SAFTER reaches 90% training success in 4.06±0.08M transitions, reducing transitions by 30.6%–35.1% vs. Vanilla SAC (5.85M) and current-state CBFs (6.16M–6.26M), matching bCBF–LSE–QP (4.13M).
  2. Exact projection differentiation drives distribution-shift robustness: With identical forward filtering, replacing the exact projection Jacobian with identity drops High-Shift success from 61.4% to 6.2% (single checkpoint) and from 43.4% to 4.4% across five independent training seeds.
  3. Real-world Unitree Go2 hardware validation: Deployed zero-shot on a physical Unitree Go2 in two real-world obstacle layouts (30 trials/layout; 60 trials total), SAFTER achieved 73.3% and 76.7% success, outperforming the strongest baselines (63.3% for CBF–RL and 16.7% for CBF–CF).

Table 9: Summary of controlled RL ablations, runtime benchmarks, and real-world Unitree Go2 results.

Evaluation MetricReference / Baseline ResultSAFTER (Ours)
Transitions to 90% training SRVanilla SAC: 5.85M; Current-state CBFs: 6.16–6.26M4.06±0.08M (Matched QP: 4.13±0.43M)
High-Shift SR (same forward filter)Identity Jacobian: 6.2±4.4% (5-seed: 4.4±1.5%)Exact: 61.4±10.2% (5-seed: 43.4±12.9%)
Runtime (Batch = 1 / 100)Matched QP: 5.73 / 7.08 ms; Direct QP: 30.55 / 2504.79 ms2.50 / 2.53 ms (Synth.: 0.10 / 0.11 ms)
Runtime (Batch = 500, Total / Synth.)Rabiee & Hoagg (2025) bounded QP: 193.60 / 186.00 ms2.71 / 0.10 ms (71.4× / 1,860× faster)
Real-world Go2 (Layouts 1 & 2, 60 trials)Best baselines: 63.3% (CBF–RL) & 16.7% (CBF–CF)73.3% & 76.7% success

4. Presentation, Baseline Implementation Details, and Reproducibility (W4)

  • Main-text revisions: We expanded the design motivation in Section 4 and moved the finite-rate propagation, clipping equations (Eqs. 15–16), and guarantee-scope discussion from Appendix C.2–C.3 directly into Section 4 alongside our 29,896-step clipping audit.
  • Explicit baseline specifications (CBF–CF and CBF–QP): Both use the same first-order command-state augmentation (u˙=−Au+Au^\dot{u} = -Au + A \hat{u}), lifted HOCBF (ψi\psi_i), input barriers (Id+,Id−I_d^+, I_d^-), and finite-rate propagation/clipping (Eqs. 15–16) as SAFTER, omitting only the backup rollout and terminal set (CBF–CF uses LSE composition + closed-form projection following Rabiee & Hoagg, 2024; CBF–QP enforces separate QP constraints).
  • Reproducibility: Code, baseline configurations, and protocols are provided in our anonymous repository, accessible through https://anonymous.4open.science/r/SAFTER-ICLR2027/.

The added Table 8 raises further concerns. I believe the baseline comparison with [Rabiee & Hoagg, 2025] should model body velocity as a state, so that the continued motion after a zero-speed command is accounted for. (This is explicitly raised in my question.) However, please correct me if I am wrong, the released implementation seems to not have a velocity state and assumes instantaneous stopping. If the added Table 8 uses this implementation, the higher collision rate with the nominal controller, and 0% success with SAC controller do not establish stronger performance of the proposed method.

The added clipping results also confirm my concern that clipping can invalidate the CBF safety condition. Describing these violations as “only 0.57%” does not resolve the lack of a safety guarantee for the actually executed actions.

Reviewer_f4wEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAFTER adds a predictive backup safety filter to SAC for simulated quadruped navigation. It augments a reduced navigation model with command dynamics and evaluates current safety margins, sampled backup trajectories, a terminal recovery set, and command limits. A hierarchical log-sum-exp smooth minimum combines these margins into one scalar barrier condition. This gives a half-space constraint in a surrogate control input, which can be enforced with an analytical minimum-Euclidean correction. Training differentiates through this construction, the projection, the finite-rate command update, and clipping.

The theoretical result gives a conditional continuous-time invariance result and a terminal-set construction for the reduced model with a fixed scan. Experiments examine learning efficiency, filter runtime, and navigation under geometric distribution shift. They include a matched backup-filter QP baseline and a comparison between the exact projection Jacobian and an identity approximation. The results support the combined method, faster computation than the matched numerical solver, and improvements over current-state filters in the tested settings.

Critical strengths
  1. The paper combines backup recovery, command-state augmentation, smooth aggregation, and analytical differentiation in a clearly specified training procedure. The CBF-LSE-QP baseline uses the same certificate and forward optimization problem. Its similar learning performance supports the benefit of backup recovery, while the reported 2.30x and 2.80x runtime speedups show the added benefit of analytical computation. This comparison is more convincing than the much larger speedup against a different formulation.

  2. The experiments distinguish several effects that could otherwise be confused. Training curves use five seeds, and the exact-versus-identity Jacobian comparison keeps the forward safety map fixed. The results also show that penalizing correction is not always helpful. The authors state the limits of their continuous-time theorem instead of treating it as a guarantee for every simulated trajectory.

Critical weaknesses
  1. As Appendix D.2 acknowledges, different nominal actions can produce the same executed action but receive different correction penalties. Nominal-command history also enters the next observation. Thus Q function generally cannot represent the exact action value. Differentiating this critic through the projection can miss the gradient of the correction cost, especially in directions removed by the projection. The local projection Jacobian remains valid, but it is not an exact policy gradient for the full objective. A critic that also receives the nominal action, or an explicit correction-cost term, would address this issue more directly.

  2. The 25 seeds in Tables 3–4 are evaluation seeds, each with 20 episodes. They are not independent training runs, and their error bars do not capture variation in what the agent learns. This matters for the very large Jacobian-ablation effect. SAFTER also has 38.6% failures on High-Shift, while removing the correction penalty improves success and reduces failures. Results across trained policies and a breakdown of failure causes would make the robustness claims more convincing.

Questions for authors
  1. Appendix D.2 notes that different nominal actions can have the same executed action but different correction penalties or subsequent observation histories. How does the actor update account for this information when the critic receives only the executed action? Are there existing results with a nominal-action-aware critic or an explicitly differentiated correction penalty, and does the exact-Jacobian advantage persist? Please distinguish what is established analytically from what is currently an empirical approximation.

  2. Can the results in Tables 3–4 be reported across the five independently trained seeds, using the same evaluation seeds for each method? Please specify how the fixed checkpoints in the current tables were chosen and separate variation across training seeds from variation across evaluation episodes. If only one checkpoint was evaluated, please state that limitation explicitly. This is especially important for the large exact-versus-identity Jacobian effect.

  3. Can the High-Shift failures be separated into obstacle collisions, falls, and boundary-wall contacts, including the variant without the correction penalty? From existing results, how often are initial composite feasibility, post-clipping barrier satisfaction, or safety between updates violated, and how often is the certificate normal near zero? Please report any available command-tracking errors as well. This would help determine whether the observed failures arise from the certificate, implementation gaps, or dynamics outside the reduced model.

4 comments
AuthorsOpenReview ↗

Actor Critic Update (Part 1/3)

We thank the reviewer for the constructive feedback. Part 1/3 addresses the actor–critic update (W1, Q1), Part 2/3 addresses five-seed evaluations (W2, Q2), and Part 3/3 covers High-Shift failure diagnostics (W2, Q3).

1. Actor–Critic Update, Nominal-Action Ablation, and Analytical vs. Empirical Scope (W1, Q1)

  • Design rationale & analytical vs. empirical scope: SAFTER prioritizes learning a safe policy that achieves task objectives (goal reaching). Because only the executed command usafe=F(ξ,unom)u_\mathrm{safe} = F(\xi, u_\mathrm{nom}) physically drives state transitions and goal progress, Q(o,usafe)Q(o, u_\mathrm{safe}) evaluates whether the executed safe action improves task return, while the actor updates through FF. Analytically established: exactness of the surrogate half-space projection (Prop. 4.1), its local Jacobian JFJ_F (Eq. 17) for a fixed filter state, and continuous-time invariance of CaugC_\mathrm{aug} and CBC_B (Thm. 4.2, Prop. B.1). When the constraint is active, Jproj=I−(a⊤a)/∣∣a∣∣2J_\mathrm{proj} = I - (a^\top a)/||a||^2 removes sensitivity in the normal direction eliminated by the projection while preserving tangential directions along the safety boundary. Empirical approximation: rcorrr_\mathrm{corr} and unomu_\mathrm{nom} history in ot+1o_{t+1} enter indirectly via the TD target; conditioning Q(o,usafe)Q(o, u_\mathrm{safe}) only on usafeu_\mathrm{safe} aliases distinct unomu_\mathrm{nom} mapping to the same usafeu_\mathrm{safe} and omits the direct normal gradient of rcorrr_\mathrm{corr} and pathwise sensitivity of ot+1o_{t+1} to unomu_\mathrm{nom} (Appendix D.2).
  • New experiment—Nominal-action-aware critic vs. Executed-action critic: Following the reviewer's suggestion, we trained a nominal-action-aware variant Q(o,usafe,unom)Q(o, u_\mathrm{safe}, u_\mathrm{nom}) (where the critic takes both actions and the actor differentiates through both paths over the 20.48M-step budget) under the same 500-episode/map protocol:
Update Formulation (500 eps/map)Low-Shift SR/TR/FRMed-Shift SR/TR/FRHigh-Shift SR/TR/FROverall SR
SAFTER (Q(o,usafe)Q(o, u_\mathrm{safe}) only)98.8 / 0.0 / 1.288.6 / 0.0 / 11.461.4 / 0.0 / 38.682.93%
Nominal-action Q(o,usafe,unom)Q(o, u_\mathrm{safe}, u_\mathrm{nom})97.6 / 2.4 / 0.073.6 / 2.2 / 24.229.6 / 2.6 / 67.866.93%
  1. Advantage of Q(o,usafe)Q(o, u_\mathrm{safe}) (+15.0 and +31.8 points on Medium- and High-Shift): Because physical transitions and goal progress depend strictly on usafeu_\mathrm{safe}, focusing the critic on (o,usafe)(o, u_\mathrm{safe}) achieves 88.6% vs. 73.6% success on Medium-Shift and 61.4% vs. 29.6% on High-Shift (with 0.0% vs. 2.2%–2.6% timeouts), supporting our critic formulation.
  2. Persistence of the Exact-Jacobian advantage (Table 5): Across five training seeds, Exact Jacobian outperforms Identity both with the correction penalty enabled (43.4% vs. 4.4%, +39.0 points on High-Shift) and with the penalty removed (Penalty = Off, eliminating rcorrr_\mathrm{corr} dependence on unomu_\mathrm{nom}: 33.2% vs. 5.3%, +27.9 points on High-Shift; 63.6% vs. 13.5%, +50.1 points on Medium-Shift).
AuthorsOpenReview ↗

Five-Seed Evaluations (Part 2/3)

2. Multi-Seed Evaluations, Checkpoint Selection, and Variance Separation (W2, Q2)

  • Original checkpoint selection (Manuscript Tables 3–4): Manuscript Tables 3–4 evaluated a single fixed checkpoint per method (the final 20.48M-step checkpoint of one representative seed meeting the 90% training-map convergence threshold, chosen without unseen-map evaluation), so error bars reflect intra-policy evaluation SD across 25 evaluation seeds (500 episodes/map). We will state this explicitly in Section 5.
  • Five-seed baseline comparison (Table 6): Evaluating the final checkpoint from all five independent training runs per method across the same 25 evaluation seeds (mean ± SD across the five trained-policy means) shows that both realizations of our framework—bCBF–LSE–QP (Ours) (qpth projection) and SAFTER (Ours) (analytical projection)—achieve higher mean success than every external baseline across all three shifts (+25.2 to +54.7 points on Low-Shift, +8.6 to +46.3 points on Medium-Shift, and +11.1 to +57.2 points on High-Shift). Between our two realizations, bCBF–LSE–QP (Ours) attains higher mean High-Shift success (57.20% vs. 43.40%), whereas SAFTER (Ours) attains higher Medium-Shift success (75.44% vs. 66.72%), less than half the High-Shift training-seed SD (±12.90% vs. ±27.77%), and 2.3×–2.8× lower latency.
  • Correction penalty across five seeds (Table 5): Retaining the penalty (Exact / On) increases five-seed mean success on High-Shift (43.4% vs. 33.2%) and Medium-Shift (75.4% vs. 63.6%), lowers timeouts (1.2% vs. 5.0%), and cuts training-seed SD by more than half (±12.9% vs. ±29.5%).

Table 5: Ablation across five independently trained policies (mean ± SD across training-seed means).

JacobianPenaltyLow SR↑TR↓FR↓Med SR↑TR↓FR↓High SR↑TR↓FR↓
ExactOn99.2±0.90.1±0.30.6±0.875.4±13.10.0±0.024.6±13.143.4±12.91.2±2.655.4±12.0
IdentityOn71.8±15.90.2±0.328.0±15.918.8±13.20.0±0.181.2±13.24.4±1.50.1±0.195.5±1.4
ExactOff98.9±1.91.0±2.20.1±0.263.6±26.36.3±12.130.1±18.633.2±29.55.0±9.461.8±27.8
IdentityOff79.8±8.30.0±0.120.2±8.413.5±10.20.0±0.186.4±10.25.3±5.80.8±1.493.9±6.0

Table 6: Navigation performance across five independent training seeds (mean ± SD across training-seed means; 500 episodes/policy/map).

MethodLow SR↑TR↓FR↓Med SR↑TR↓FR↓High SR↑TR↓FR↓
Vanilla SAC [1]44.56±14.650.20±0.4555.24±14.9029.16±13.240.08±0.1170.76±13.2420.16±6.870.12±0.2779.72±7.02
SEA-Nav [2]74.0±19.00.0±0.026.0±19.040.0±28.70.0±0.060.0±28.70.0±0.00.0±0.0100.0±0.0
CBF–RL [3]59.3±6.217.2±4.223.4±4.537.7±16.929.2±14.133.1±9.029.2±7.821.1±2.549.7±6.9
CBF–CF [4]66.72±17.540.00±0.0033.28±17.5450.64±17.590.00±0.0049.36±17.5930.72±9.600.04±0.0969.24±9.63
CBF–LSE–QP [4]72.32±12.560.00±0.0027.68±12.5658.16±13.050.00±0.0041.84±13.0532.32±10.640.00±0.0067.68±10.64
bCBF–LSE–QP (Ours)99.20±0.620.08±0.180.72±0.7066.72±26.860.12±0.2733.16±26.9857.20±27.770.08±0.1142.72±27.85
SAFTER (Ours)99.24±0.850.12±0.270.64±0.8275.44±13.120.00±0.0024.56±13.1243.40±12.901.20±2.5755.40±11.98

[1] https://arxiv.org/abs/1812.05905

[2] https://arxiv.org/abs/2603.09460

[3] https://ieeexplore.ieee.org/abstract/document/11695808

[4] https://arxiv.org/abs/2406.16874

AuthorsOpenReview ↗

High-Shift Failure Diagonistics (Part 3/3)

3. High-Shift Failure Breakdown and Execution Diagnostics (W2, Q3)

To examine whether High-Shift failures arise from the certificate, finite-rate implementation gaps, or dynamics outside the reduced model, we report the five-seed failure breakdown (Table 7) and step-level diagnostics:

  • Obstacle collisions, falls, and boundary-wall contacts (Table 7): Across the five trained SAFTER (Ours, Exact/On) policies on High-Shift (55.40±11.98% total FR), 55.32±11.99% of episodes end in obstacle collisions, 0.08±0.11% end in falls, and 0.00% end in boundary-wall contacts (bCBF–LSE–QP (Ours) similarly records 42.64% collisions, 0.08% falls, and 0.00% wall contacts). While SEA-Nav and CBF–RL experience 7.5%–23.7% falls across maps (and CBF–RL incurs 17.2%–29.2% timeouts), all augmented CBF/bCBF variants record near-zero falls (0.00%–0.32%). For the penalty-off variant (Exact / Off), five-seed High-Shift failures are higher (61.8±27.8% vs. 55.4±12.0%, Table 5) with 0.00% wall contacts (full Exact / Off splits will be included in Appendix E).
  • Step-level certificate, clipping, and tracking diagnostics:
    1. Post-clipping barrier satisfaction: Over each 10 Hz interval (Δt=0.1\Delta_t = 0.1 s), command dynamics are integrated analytically via Λ=e−AΔt\Lambda = e^{-A \Delta_t} (Eq. 15) before clipping (Eq. 16). On High-Shift, post-clipping commands violate the composite barrier condition in 1.73% (Exact / On) and 1.13% (Exact / Off) of control steps; among clipping events, 34.0% and 32.1% have barrier residuals within ±0.1 s⁻¹ of zero. Separately, across 29,896 audited steps, clipping altered commands beyond 10−710^{-7} on 2.4% of steps (max normalized change 0.19) and made a previously satisfied endpoint-equivalent CBF residual negative on 171 of 718 clipped steps (0.57% of all steps) at tolerance 10−310^{-3} s⁻¹.
    2. Command-tracking error: On High-Shift, the mean Euclidean tracking discrepancy ∥vmeas−usafe∥2\left\lVert v_{\mathrm{meas}} - u_{\mathrm{safe}} \right\rVert_2 between the 10 Hz safe command usafeu_\mathrm{safe} and measured quadruped body velocity vmeasv_\mathrm{meas} is 0.546 (Exact / On) and 0.554 (Exact / Off).
    3. LSE sharpness & unlogged quantities: Sharpening LSE κ\kappa by 4×4\times across 82,831 states reduces the mean static LSE gap from 0.020 to 0.002 without improving High-Shift success for the fixed trained policy (61.0% at 1.0×1.0\times vs. 52.0% at 4.0×4.0\times; Table 4 in our response to Reviewer AScW). Separate counts for initial haug(z0)<0h_\mathrm{aug}(z_0) < 0, near-zero normal activation$\left\lVert a(z_t) \right\rVert_2^2$, and 50 Hz inter-step barrier values between 10 Hz scans were not logged separately.
  • Diagnostic takeaway: These results show that: (i) High-Shift failures are overwhelmingly obstacle collisions (55.32%) rather than falls (0.08%) or wall contacts (0.00%); (ii) Exact / On and Exact / Off exhibit comparable mean tracking discrepancies (0.546 vs. 0.554) and low per-step post-clipping violation rates (1.73% vs. 1.13%); and (iii) the measured low-level velocity tracking discrepancy (~0.55) and post-clipping boundary violations (1.13%–1.73% of 10 Hz steps) directly quantify the execution gaps between Theorem 4.2's continuous reduced-model assumptions and High-Shift navigation in tight unseen clutter.

Table 7: Failure-cause breakdown across five independent training seeds (% of episodes, mean ± SD; wall contacts are 0.00%).

MethodLow Coll.Low FallMed Coll.Med FallHigh Coll.High Fall
Vanilla SAC [1]54.44±14.780.80±1.3970.04±13.370.72±1.6179.16±6.670.56±1.25
SEA-Nav [2]2.3±1.123.7±19.344.7±34.615.3±22.992.5±10.77.5±10.7
CBF–RL [3]8.6±3.214.8±1.918.0±8.515.1±2.036.5±8.113.2±1.7
CBF–CF [4]33.28±17.540.00±0.0049.36±17.590.00±0.0069.24±9.630.00±0.00
CBF–LSE–QP [4]27.48±12.740.20±0.2441.64±13.220.20±0.3567.56±10.590.12±0.18
bCBF–LSE–QP (Ours)0.40±0.550.32±0.7233.08±27.000.08±0.1842.64±27.880.08±0.11
SAFTER (Ours)0.64±0.820.00±0.0024.56±13.120.00±0.0055.32±11.990.08±0.11

[1] https://arxiv.org/abs/1812.05905

[2] https://arxiv.org/abs/2603.09460

[3] https://ieeexplore.ieee.org/abstract/document/11695808

[4] https://arxiv.org/abs/2406.16874

I thank the author for the response. I will keep my original rating.

Reviewer_AScWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SAFTER, a safe reinforcement learning framework that uses Backup Control Barrier Functions (bCBFs) to ensure that an agent can recover safely from its current state using a predefined backup controller. Instead of only checking immediate safety, it predicts future trajectories under the backup controller to determine whether the agent can avoid constraint violations. To reduce computational cost, the authors combine multiple safety constraints into a single constraint using Log-Sum-Exp and derive a closed-form action projection, avoiding repeated quadratic programming. The safety filter is differentiable and integrated into Soft Actor-Critic (SAC), allowing the policy to learn how its actions are modified by the filter. At this stage paper become an action constrained rl, some-how authors want to enfoce the filtering on the SAC policy in a differentiable manner. Experiments on quadruped robot navigation show improved sample efficiency, faster safety filtering, and better generalization to unseen environments compared with standard SAC and conventional CBF baselines.

Critical strengths
  1. Efficient closed-form safety filtering: The method avoids iterative QP solvers by combining multiple safety constraints into a single constraint, achieving up to 12.24× faster filtering than direct bCBF–QP for batch size 1.

  2. Predictive safety integration: The use of backup CBFs considers future recoverability rather than only immediate safety, allowing the agent to avoid states from which safe recovery is no longer possible.

  3. Improved sample efficiency: SAFTER reaches 90% training success with approximately 30.6% fewer transitions than Vanilla SAC.

  4. Useful ablation studies: The authors investigate the effects of predictive recoverability, analytical projection, and exact Jacobian computation, providing evidence for their individual contributions.

  5. Clear mathematical formulation: The paper provides a well-structured derivation of the composite safety certificate, closed-form projection, and conditional safety guarantees.

Critical weaknesses
  1. Clarification of methodological novelty: The paper combines backup CBFs, LSE aggregation, and closed-form projection, which have been studied individually. It would be useful to clarify which parts of the proposed formulation are new compared with existing composite and backup CBF approaches.

  2. Evaluation with more complex backup controllers: The experiments use a backup controller that gradually reduces the robot's velocity to zero. Evaluating a more complex recovery strategy, such as steering around obstacles, would help demonstrate the general applicability of the approach.

  3. Impact of discrete-time execution on safety: The theoretical guarantees assume continuous-time control, while the high-level policy operates at 10 Hz. An experiment varying the control frequency could help demonstrate how discretization affects safety violations and recovery performance.

  4. Scalability with the number of constraints: The method evaluates 120 LiDAR rays over 20 future time steps, producing 2,400 geometric safety evaluations. Reporting runtime across different numbers of LiDAR rays and prediction steps would help demonstrate how well the method scales.

  5. Relation to action constrained RL: Once you have the backup filtering, the objective is to enforce the constraints on the SAC's action space in a differentiable manner. Essentially this becomes an action-constrained-rl problem. Authors can look at ACRL literature [1,2,3,4] to find alternative ways to enforce this constraints on the agent. Generally, learning SAC agent from pre-projection action yield better results[1]. Also note, "Differentiable Optimization Layer" methods mentioned in [1] is very similar to your proposed solution.

[1] https://arxiv.org/abs/2304.08743
[2] https://arxiv.org/abs/2502.10431
[3] https://ojs.aaai.org/index.php/AAAI/article/view/39758
[4] https://arxiv.org/abs/2503.12932

Questions for authors
  1. How does the computational cost of SAFTER scale with the number of safety constraints and the backup prediction horizon?

  2. How sensitive is the method to inaccuracies in the dynamics model used for backup trajectory prediction?

  3. Can the proposed framework support more complex backup controllers, such as collision-avoidance maneuvers instead of simply stopping?

  4. How does the conservativeness introduced by LSE aggregation affect the feasible action space and the learned policy's performance?

2 comments
AuthorsOpenReview ↗

Scalability, Sensitivity, and Alternative Backup Controller (Part 1/2)

We thank the reviewer for the positive assessment and constructive feedback. Below we address all four questions and five weaknesses with new scalability, model-sensitivity, maneuvering-backup, LSE, and real-world hardware experiments.

1. Computational Scalability with Constraints and Backup Horizon (Q1, W4)

Because all Nh×KN_h \times K backup margins and flow sensitivities are evaluated in parallel on the GPU via vectorized Gauss–Legendre quadrature and collapsed via LSE into a single affine half-space a(zt)u^+b(zt)≥0a(z_t) \hat{u} + b(z_t) \geq 0, the online synthesis step is an O(m)O(m) closed-form projection (~0.10 ms) once (a,b)(a, b) are evaluated, rather than a QP whose constraint count grows with Nh×KN_h \times K. As shown in Table 1 (300 CUDA-synchronized repetitions on an RTX 4090), increasing LiDAR rays from 60 to 960 and backup steps from 20 to 120 increases geometric checks by 96-fold (1,200 to 115,200 per call), while total latency rises by only 2.6% (2.497 to 2.562 ms). Extending horizon duration TT from 4 s to 12 s also has negligible impact.

Table 1: SAFTER filter-time scalability (mean ± SD, ms).

ConfigurationSetting20 Steps120 Steps
LiDAR Rays602.497±0.1072.554±0.190
1202.529±0.1862.565±0.219
9602.573±0.2152.562±0.197
Horizon4 s2.536±0.1932.562±0.206
8 s2.560±0.2042.590±0.232
12 s2.564±0.2112.573±0.192

2. Dynamics-Model Sensitivity and 10 Hz Discrete-Time Execution (Q2, W3)

  • Sensitivity to backup-model perturbations (Q2): Varying the assumed yaw-rate decay gain aωa_\omega in the filter across an 8-fold range (0.25×0.25\times to 2.00×2.00\times nominal) with the policy and environment fixed (Table 2) increases worst-case negative barrier residuals (from −17.5-17.5 to −34.1-34.1) and clipping frequency (5.0%5.0\% to 8.2%8.2\%), yet closed-loop success remains between 96.3% and 98.3% (collisions ≤3.7\leq 3.7%), showing strong empirical tolerance to prediction errors.
  • 10 Hz discrete-time execution & real-world validation (W3): At the 10 Hz high-level rate (Δt=0.1\Delta_t = 0.1 s, matching the 10 Hz LiDAR scan and 50 Hz locomotion controller), linear command dynamics u˙=−Au+Au^\dot{u} = -Au + A \hat{u} are integrated analytically via Λ=e−AΔt\Lambda = e^{-A \Delta_t} (Eq. 15). Because input bounds Id+(z),Id−(z)≥0I_d^+(z), I_d^-(z) \geq 0 are encoded inside haug(z)h_\mathrm{aug}(z), auditing 29,896 steps shows that post-projection clipping (Eq. 16) modifies commands on only 2.4% of steps and induces an endpoint-equivalent CBF residual violation on only 0.57% of audited steps (while overall post-clipping violations on High-Shift are 1.13%–1.73%). Furthermore, zero-shot deployment on a physical Unitree Go2 quadruped across two real-world cluttered layouts (30 trials/layout; 60 trials total) at 10 Hz / 50 Hz achieved 73.3% and 76.7% success, outperforming the best baselines (63.3% for CBF–RL [1] and 16.7% for CBF–CF [2]). We now summarize the finite-rate update, clipping audit, and guarantee scope directly in Section 4.

Table 2: Sensitivity to yaw-dynamics perturbations (aωa_\omega) in backup prediction on Low-Shift.

Gain ScalingSR (%) ↑Collision (%) ↓Timeout (%) ↓Worst ResidualClip (%)
0.25×98.31.70.0-34.18.2
0.50×96.33.70.0-32.36.6
0.75×97.32.70.0-21.25.8
1.00× (Nominal)98.31.70.0-17.55.0
1.50×98.02.00.0-14.12.8
2.00×96.33.00.7-23.81.7

3. Evaluation with an Alternative Maneuvering Backup Controller (Q3, W2)

SAFTER's variational flow sensitivity (Jb(τ,z0)J_b(\tau, z_0), Appendix A.3.1) supports time-varying recovery maneuvers beyond pure stopping. Replacing the zero-command backup with an exponentially decaying lateral-sidestep maneuver u^b(τ)=[0,0.3e−2τ,0]⊤\hat{u}_b(\tau) = [0, 0.3e^{-2 \tau}, 0]^\top under u˙=−A(u−u^b)\dot{u} = -A(u - \hat{u}_b) (which sidesteps during the transient and decays by >99.9% at T=4.0sT = 4.0 s) achieves 81.2%, 59.2%, and 31.6% success on Low-, Medium-, and High-Shift with 0.0% timeouts (Table 3). Because a fixed-direction unilateral sidestep always steers toward the same side regardless of obstacle asymmetry, it is more restrictive in narrow corridors than symmetric deceleration—motivating state-conditioned or learned steering backups within SAFTER.

Table 3: Navigation performance with a retrained lateral-sidestep backup vs. nominal stopping backup.

EnvironmentLateral-Sidestep SR / TR / FR (%)Nominal Stopping SR / TR / FR (%)
Low-Shift81.2 / 0.0 / 18.898.8 / 0.0 / 1.2
Medium-Shift59.2 / 0.0 / 40.888.6 / 0.0 / 11.4
High-Shift31.6 / 0.0 / 68.461.4 / 0.0 / 38.6

[1] https://ieeexplore.ieee.org/abstract/document/11695808

[2] https://arxiv.org/abs/2406.16874

AuthorsOpenReview ↗

LSE Conservativeness and Methodological Novelty (Part 2/2)

4. LSE Conservativeness, Feasible Action Space, and Performance (Q4)

We varied LSE sharpness κ\kappa from 0.5×0.5\times to 4.0×4.0\times across 82,831 fixed states, measuring the empirical static approximation gap (min⁡S−softminκ(S)\min S - softmin_\kappa(S)), the feasible action fraction (percentage of uniformly sampled actions satisfying the composite CBF condition), the intervention rate (percentage of states where the fixed trained policy's nominal action triggers projection, Ωt<0\Omega_t < 0), and High-Shift navigation success under the same fixed policy (Table 4). Increasing κ\kappa from 0.5×0.5\times to 4.0×4.0\times reduces the mean static gap by 29×29\times (0.058 to 0.002) and max gap by 8.8×8.8\times (0.219 to 0.025), while the feasible action fraction shifts by only 2.5 points (69.54% to 67.01%) and intervention rate shifts from 49.34% to 51.81%. Because changing κ\kappa alters both haug(z)h_\mathrm{aug}(z) and the LSE gradient weights wℓw_\ell (Eq. 9) entering the Lie derivatives (a(z),b(z))(a(z), b(z)), reducing the static gap does not monotonically expand the half-space. High-Shift success is highest at the nominal training setting (61.0% at 1.0×1.0\times), showing that reducing static LSE approximation error does not improve performance of the trained policy.

Table 4: LSE sharpness sensitivity across 82,831 states and High-Shift navigation under a fixed trained policy.

Sharpness ScalingGap (Mean / Max)Feasible (%)Intervention (%)High-Shift SR (%) ↑FR (%) ↓
0.5×0.058 / 0.21969.5449.3455.045.0
1.0× (Nominal)0.020 / 0.10768.2250.8861.039.0
2.0×0.007 / 0.05267.4251.4658.042.0
4.0×0.002 / 0.02567.0151.8152.048.0

5. Methodological Novelty and Relation to Action-Constrained RL [1–4] (W1, W5)

  • Novelty relative to prior CBF methods (W1): Rather than treating backup CBFs as standalone non-learning execution shields [5, 6] (Rabiee & Hoagg, 2024, 2025), our core contribution is reformulating finite-horizon backup recoverability and bounded-input admissibility into a closed-form, end-to-end differentiable action map for off-policy SAC reinforcement learning—supported by a new forward-invariant anisotropic elliptical terminal recoverability set CBzC_B^z for LiDAR navigation (Prop. B.1) and an explicit finite-rate physical-action Jacobian JFJ_F (Eq. 17) to achieve safe, sample-efficient training and zero-shot robustness across distribution-shifted environments. We realize this framework in two ways—via differentiable QP (bCBF–LSE–QP (Ours)) and via our exact closed-form analytical solution (SAFTER (Ours), Eq. 17)—showing that predictive recoverability drives the 30.6%–35.1% sample-efficiency gain, while exact projection differentiation is critical for distribution-shift robustness (61.4% vs. 6.2% single-checkpoint; 43.4% vs. 4.4% across 5 training seeds).
  • Connection to Action-Constrained RL [1–4] (W5): Enforcing our reduced state-dependent half-space a(zt)u^+b(zt)≥0a(z_t) \hat{u} + b(z_t) \geq 0 on SAC connects directly to ACRL (Kasaura et al., 2023 [1]; Brahmanage et al., 2025 [2, 3]; ARAM, 2025 [4]):
    1. OptLayer overhead [1, 2]: Our bCBF–LSE–QP realization corresponds to an OptLayer [1] solved via qpth; SAFTER replaces this QP with an analytical solution, cutting synthesis latency from 3.34 / 4.59 ms (qpth, batch 1 / 100) and 186.00 ms (hard-bounded bCBF-QP, batch 500) to 0.10 ms.
    2. Critic conditioning & zero-gradient remedies [1–4]: While [1] trains a pre-projection critic Q(o,unom)Q(o, u_\mathrm{nom}) and [2–4] address normal-gradient zeroing (Jprojat⊤=0J_\mathrm{proj} a_t^\top = 0 when active) via normalizing flows [2, 3] or acceptance–rejection [4], SAFTER trains an executed-action critic Q(o,usafe)Q(o, u_\mathrm{safe}) and propagates the tangential Jacobian JprojJ_\mathrm{proj} (outperforming a nominal-action-incorporated update: 88.6%/61.4% vs. 73.6%/29.6% on Medium-/High-Shift). We have added [1–4] to Section 2 and Appendix D.2.

[1] https://arxiv.org/abs/2304.08743

[2] https://arxiv.org/abs/2502.10431

[3] https://ojs.aaai.org/index.php/AAAI/article/view/39758

[4] https://arxiv.org/abs/2503.12932

[5] https://www.sciencedirect.com/science/article/pii/S0005109824004151

[6] https://arxiv.org/abs/2406.16874

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.