acceptodds
Under review as a conference paper at ICLR 2027

SAFE-HAC: A Modular Architecture for Safe Reinforcement Learning

Abstract

Safety information is heterogeneous across continuous-control tasks: some environments expose only a cost signal, others provide signed constraint margins or task-specific regulated outputs, and only some admit a usable conventional controller. Existing methods are typically specified around one such interface. We study safe policy learning under optional safety signals and optional controller assistance. We introduce SAFE-HAC (Soft Actor with Fallback Enhancement and Horizon-Aware Critics), a modular recurrent off-policy actor–critic whose availability mask activates any subset of three finite-horizon predictors: cumulative cost, worst future clearance, and regulated-output error. The last signal can represent distance, speed, attitude, or another task-defined objective; it is Lyapunov-inspired only when its zero set and descent condition have the corresponding control-theoretic meaning. Active predictions enter the actor loss. When a fallback controller is available, the same predictions assess both proposed and fallback actions and can reject a policy proposal in favor of an MPC, LQR, PID, or other conventional control action; without a fallback, learning remains well-defined and the policy executes directly. Because intervention changes the transition-generating action, critics train on the executed action while proposal–fallback pairs support intervention imitation. We formalize the error induced by proposal-labeled shielded replay. We evaluate both policy-only and controller-assisted modes on five continuous-control tasks against SAC-Lag, PPO-Lag, CPO, and TD3. Empirical evaluations demonstrate that SAFE-HAC achieves competitive task performance while reducing constraint violations and reaching the specified empirical safety criteria in fewer environment steps compared with standard constrained-RL baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, SafeRL, Data Based Control, Control theory

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_fa5pOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

A modular recurrent actor-critic architecture for safety-aware reinforcement learning is presented. The proposed method combines a standard actor-critic with optional finite-horizon predictors for cumulative cost, minimum future clearance, and regulation error. An availability mask determines which of these signals are used for a particular task. If a conventional fallback controller is available, the same predictors are additionally used to decide whether the learned policy or the fallback action should be executed.

Critical strengths
  • Safety information indeed differs between control applications, and a framework that can incorporate different types of available information is useful.

  • The authors are explicit about the limitations of their approach: the learned predictors are not safety certificates; satisfying their thresholds does not establish feasibility of the stated constraints, and that the screening mechanism does not guarantee invariance.

  • The authors provide an analysis of the Bellman-target discrepancy that may result when a transition generated by the fallback action is incorrectly assigned to the original policy action.

  • The modularity makes it possible to differently combine cost, margin, progress, and fallback-controller information.

  • The authors distinguish policy-only performance from controller-assisted performance instead of treating them equally.

Critical weaknesses
  • In my opinion, the paper lacks a clear line of thought connecting its different components. This starts with the abstract. It moves between heterogeneous safety information, modular predictors, fallback controllers, executed-action replay, intervention imitation, a theoretical discrepancy result, and experiments on several environments. The individual elements can be understood, but the central scientific question and the principal insight are difficult to extract.

  • Safety should be defined much more clearly and much earlier. The paper concerns "safe reinforcement learning," yet the proposed method does not guarantee safety. The authors correctly acknowledge that satisfying a prediction threshold does not establish the constraints in Eq. (3), and they even note that an inadmissible proposal may be executed if no admissible fallback is found. This distinction should not be left for the reader to reconstruct from different sections. The paper needs a concise statement early in the text specifying exactly what notion of safety it aims to improve and what it does, and does not, guarantee.

  • Do the authors consider the common interface to be a scientific contribution rather than primarily an engineering contribution? The availability mask switches cost, margin, progress, and fallback resources on or off. This is convenient from an implementation perspective. However, the paper should explain what scientific insight follows from putting these signals into one architecture. In the submitted version of the paper, the significance of the interface is not sufficiently clear.

  • The novelty of the intervention-aware learning rule is not stated sharply enough. The paper should explain much more explicitly what previous controller-assisted or shielded RL methods do instead, why that causes a problem, and which part of the proposed solution is genuinely new. The bound in Eq. (18) quantifies the discrepancy caused by the action mismatch, but the broader methodological novelty is unclear to me.

  • Overall, the paper combines many established ingredients: SAC-style actor-critic learning, recurrent state representations, finite-horizon prediction, constrained RL penalties, clearance information, Lyapunov-inspired regularization, fallback controllers, action screening, sequence replay, and imitation of interventions. The paper needs to explain why this particular combination is non-trivial. At the moment, several design decisions seem to be assembled rather than derived from one clear principle.

  • The related-work section contains relevant references on constrained optimization, Lyapunov and barrier methods, shielding, recovery policies, MPC-guided learning, recurrent RL, and finite-horizon constraint estimation. However, it reads mostly as a list of neighboring areas. The reader is not guided from a limitation of existing work to a precise gap and then to the proposed solution.

  • Several established elements are described at considerable length. For example, standard SAC machinery and recurrent replay conventions could largely be referenced. That space could instead be used to explain and analyze what is actually new: the motivation for the three-head interface, why the particular finite horizons are useful, the rationale behind the intervention rule, and the novelty of executed-action learning in comparison with the closest assisted-RL approaches.

  • A number of design choices are insufficiently motivated. Why should cumulative cost, expected minimum clearance, and regulation error be the three quantities combined within the same interface? Why is the relevant margin quantity an expected minimum rather than another risk measure? What principle determines the thresholds and relative penalties?

  • Some experimental choices make the overall contribution harder to interpret. The information available to SAFE-HAC varies between environments, training budgets differ across methods in some experiments, and the controller-assisted experiment does not hold controller assistance fixed across methods. The manuscript itself acknowledges that matched-supervision and matched-capacity ablations would be required to isolate the optimizer or architecture, and that the controller-assisted experiment does not identify the causal contribution of MPC.

Questions for authors
  1. What is the main scientific contribution of SAFE-HAC?

  2. What exactly does "safe" mean in SAFE-HAC? Is the objective empirical reduction of constraint violations, satisfaction of an expected-cost constraint, satisfaction of a chance constraint, or something else? The paper explicitly states that its learned surrogates do not establish Eq. (3).

  3. What scientific insight is obtained from the common availability interface?

  4. What precisely is novel about the intervention-aware learning rule? Which closest existing methods train on the proposed action even though a fallback action generated the observed transition? A direct comparison to those methods would be helpful.

  5. Is executed-action replay the central technical contribution? If so, I suggest reorganizing the paper around the action-label mismatch problem, because Proposition 1 provides a relatively clear motivation for this issue.

  6. Why were cumulative cost, minimum clearance, and regulation error selected as the three heads? Is there a theoretical or empirical reason why these quantities belong in one common framework?

  7. How sensitive are the results to the chosen prediction horizons?

  8. Which component is responsible for the empirical improvements? The limitations section itself states that auxiliary supervision, recurrence, and controller quality require separate ablations. Without these ablations, it is difficult to assess whether the contribution is the proposed learning rule or simply additional task-specific supervision and model capacity.

  9. Could the paper be reorganized around one central hypothesis?

  10. How should the controller-assisted results be interpreted when the baselines do not receive the same controller assistance?

Reviewer_2t2XOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces an algorithm for SAFE RL which utilizes multiple critics for task success, safety margin, cost minimization and reward maximization. It uses any information that is available in the environment, including safety backup actions and modifies the SAC algorithm to add additional critics and imitation-based learning in case of backup controllers to learn to be safe. The algorithm replaces traditional bootstrapping of the new critics with 3 fixed horizons (1, 4, 8) and aggregates the losses using weights for every horizon. There are 2 modes available at inference. The first mode uses policy action only, while the second mode uses the policy action and a backup action and the critics are used to decide which action to execute. Experiments with 4 safe RL benchmarks are provided.

Critical strengths

The modularity of this algorithm is quite interesting. Safe HAC can use whatever information is available as part of the environment and use it to its benefit.

Critical weaknesses

I will list down weaknesses in hopes that it motivates the authors to resubmit a refined version of this paper.

Method

  1. There is not much motivation about why bootstrapping is removed in favor of fixed horizon critics. It might be justified for the margin, but otherwise, the critic is blind to anything beyond horizon 8. Neither has the choice of 1, 4 and 8 been justified, nor has it been ablated or compared to bootstrapped critics.
  2. Using multiple critics is not particularly novel. This has been proposed multiple times in literature, including SAC Lag.
  3. The choice of POMDPs is also not justified. I would expect this to be a general safe RL algorithm, the only way the authors deal with the POMDP is a history encoder. Nothing else about the POMDP nature is ever mentioned.

Experiments

  1. The experimental protocol is not clear. I am not sure if the baselines were also provided similar POMDP history encoders or were they used as implemented in Omnisafe. I also don't understand why all algorithms use different step budgets.

  2. There is some AI written remnants in the paper. L304-309 and entire appendix J seem to be AI generated with placeholders to be replaced. In particular, appendix J seems like an instruction to the authors rather than readers. Using AI for polishing is fine, but leaving out details and using clear AI generated instructions in the paper is not right.

  3. The results do not establish that Safe-HAC is the best algorithm. The only true win is in Underwater drone, in the rest, it is quite weaker than untuned baselines. This raises doubt about the efficacy of the proposed technique.

  4. No ablations with step-size, step-counts in training, removing different critics are shown. Adding onto this, it is unclear if the baselines had access to the same information that safe HAC had.


Overall, the writing also is extremely unclear, making the paper very difficult to read.

Questions for authors

Please clarify experimental protocol properly, whether baselines had access to same data, they were also POMDP based and properly used state history encoders. Additionally, provide inter-seed variation and statistical significance results as outlined in appendix J.

Reviewer_atRVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a recurrent SAC agent whose actor is penalised by short-horizon cost, clearance and progress predictors, with an optional MPC that overrides the policy when these predictions deem its action unsafe. The model is validated across a variety of tasks and compared with classical baselines.

Critical strengths

The paper addresses a practically relevant problem: safety information and controller availability vary considerably across real tasks, and a single learning framework that can exploit whichever resources are present, with or without a backup controller, is an interesting topic that has gathered a lot of interest from the community overall.

Critical weaknesses

Overall, the main issues with the paper are as follows:

  • The paper is not at the standard required for publication. The methodology is conceptually simple, yet the notation and extensive caveats make it extremely difficult to follow. Several passages appear unfinished, and some of them are not easy to follow. For example, although these two are not exhaustive: (i) the evaluation protocol in Section 5 sounds like a first draft, and (ii) the appendices repeatedly state that configurations and aggregates “must be recorded” or “must be verified from the original logs”. The manuscript would need substantial revision before it can be properly assessed.

  • In terms of content, the contribution is thin. The paper presents a novel architecture (although it is a mix of known components) but the lack of ablation studies make it so that it is not clear what each component effectively does and how they affect the overall performance of the method. Considering that the whole paper's validation is based on experiments, this is not something that can be ignored. On top of that, there are several inconsistencies that worry me. One above all: the results for the underwater drone reported in Table 1, which are an evaluation across seeds, seem to be exactly identical to the results of the single seed in Figure 9 in the appendix. I think this requires an explanation.

Questions for authors

Can you please explain:

  • Why there is a lack of ablation studies to evaluate the various components of your controller;
  • The results in Table 1 seem identical to those in Figure 9 for the underwater drone. If this was a mistake, what are the correct numbers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.