acceptodds
Under review as a conference paper at ICLR 2027

Safety-Aware Value Alignment for Offline Safe Reinforcement Learning

Abstract

Offline safe reinforcement learning (OSRL) often extracts policies from learned value funcitons, but learned reward values can rank dataset-supported actions differently from the constrained objective. We introduce Safety-Aware Value Alignment (SAVA), which makes learned long-horizon cost information part of the Bellman target for reward value learning. Using cost values to identify high-cost regions, reshapes reward value learning when transitions enter these regions, allowing the resulting penalized Bellman targets to propagate to preceding state–action pairs and to reshape earlier action preferences before policy extraction. We theoretically establish sufficient conditions under which the penalized objective favors safe policies over unsafe ones and derive a bound on the reward gap to the optimal safe policy under value-estimation and policy-extraction errors. Empirically, controlled ablations show that the region-aware penalized reward value is the primary component responsible for constraint satisfaction. On the DSRL benchmark, SAVA achieves the highest normalized return on 27 of 32 tasks while satisfying the common normalized-cost safety criterion.

Primary area:
reinforcement learning
Keywords:
Deep Reinforcement Learning, Costrained Reinforcement Learning, Offline Safe Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6VHVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SAVA, an offline safe reinforcement learning method built on implicit Q-learning. A separately learned cost critic splits states into estimated low-cost and high-cost regions. Whenever a transition enters the high-cost region, the reward Bellman target is replaced by a fixed negative penalty. This penalty propagates backward, so actions leading toward hazards lose value before the policy is extracted. Extraction uses a conditional flow-matching actor with classifier-free guidance, plus a cost-based recovery rule for candidate actions in high-cost states. The paper has three contributions: (1) Injecting long-horizon cost directly into reward value learning through region-aware target replacement. (2) Two propositions: a sufficiently large penalty makes every safe policy preferable to every unsafe one, and the reward gap to the optimal safe policy is bounded by penalty, estimation, and extraction errors. (3) A data-dependent, quantile-based threshold for terminal-only value learning.

Critical strengths
  1. The paper is clearly structured and well written, presenting complex concepts in an accessible and easy-to-follow manner.
  2. The proposed method is conceptually sound and intuitive, offering a clean, well-motivated approach to incorporating safety constraints into offline value learning.
  3. The empirical results are convincing, demonstrating consistent and robust performance improvements over behavior cloning and five established offline safe RL baselines across benchmark suites.
Critical weaknesses
  1. The core mechanism—replacing the reward Bellman target with a fixed penalty when predicted cost exceeds a threshold—largely integrates established concepts: CPQ-style cost-conditioned backups, absorbing failure penalties, and in-sample IQL learning. Furthermore, the paper's ablations indicate that auxiliary components like multi-step propagation and guided flow-based extraction primarily influence reward rather than safety. Because 1-step backups and simpler AWR extractors retain safety on nearly all evaluated tasks, the contribution rests heavily on a heuristic penalty whose conceptual departure from prior masking schemes remains modest.

  2. There is a gap between the theoretical analysis in Section 4.2 and the proposed algorithm:

a. Proposition 4.1 relies on the strong assumption that the high-cost region perfectly separates safe and unsafe policies (ηπs=0\eta^{\pi_s} = 0 and ηπu≥l\eta^{\pi_u} \ge l), which rarely holds when partitioning is learned from finite, noisy offline data.

b. The propositions analyze per-step penalties on a true region, whereas the algorithm applies an absorbing penalty to an empirically thresholded region with penalty coefficients (CpenC_{\text{pen}}) well below the theoretical bound. Consequently, the propositions offer intuitive insights rather than formal safety guarantees for the practical policy.

  1. The algorithm relies critically on the accuracy of the learned cost critic V^c\hat{V}_c to partition the state space. In offline domains with limited coverage, estimation errors can lead to false penalties on viable behaviors or miss critical hazards. Additionally, while a shared quantile is evaluated, the reliance on data-dependent quantile thresholds (α\alpha) and task-specific calibration across environments suggests practical sensitivity that could hinder deployment without environment-specific tuning
Questions for authors
  1. Beyond empirical benchmarks, under what structural conditions (e.g., environment horizon, hazard topology, or offline dataset coverage) is replacing the Bellman target with a fixed negative penalty expected to outperform the zero-masking approach used in CPQ?

  2. The partition into low- and high-cost regions relies entirely on the learned value V^c\hat{V}_c. How robust is the upstream value propagation to value-estimation errors or boundary inaccuracies in V^c\hat{V}_c, and did the authors explore any explicit mechanisms (e.g., pessimistic cost ensembles or target smoothing) to ensure critic fidelity in sparse data regimes?

  3. Standard Lagrangian dual-variable methods are widely used in safe RL to adaptively balance rewards and costs. Could the authors clarify why a primal-dual offline baseline (e.g., Lagrangian-regularized IQL) was omitted from the empirical comparisons, and how SAVA’s fixed region-aware penalty compares in stability and performance against an adaptive multiplier?

Reviewer_iLiuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies offline safe RL, aiming to learn a policy from a fixed dataset that gets high reward while keeping cumulative cost within budget. The main idea of SAVA is to use cost values to identify states with high predicted future cost. If an action leads to such a state, its reward Bellman target is replaced with a fixed negative value. The effect then propagates backward through value updates, so earlier action choices also account for future risk. The implementation also uses multi-step backups, a flow-based actor, and recovery guidance. The authors analyze safety and reward loss under certain assumptions and evaluate the method on 32 DSRL tasks. SAVA meets the average-cost constraint on all tasks and achieves the highest return among safe methods on 27. The ablations also show that modifying the reward-value learning target is important for its safety performance.

Critical strengths
  1. I think the method has a clear motivation. An action can give high immediate reward but lead the agent to states with high future cost. SAVA directly modifies the reward Bellman target for these transitions and propagates the effect backward through value updates, so earlier decisions also account for future risk. This connects the cost information to the reward values used for action selection. The mechanism is straightforward, and it is easy to see why it could help. This is what I find valuable about the method itself.

  2. The results and ablations are convincing. SAVA meets the average-cost constraint on all 32 tasks and achieves the highest return among safe methods on 27, showing that it can control cost while retaining good task performance. More importantly, removing the penalized reward value while keeping the flow actor causes cost violations on all seven tasks tested. This makes it difficult to attribute the safety gains only to a more expressive actor. The ablation supports the central design and gives me more confidence that modifying the value-learning target plays an important role.

Critical weaknesses
  1. I think the connection between the theory and the actual algorithm needs to be clearer. The theoretical objective in the appendix replaces the per-step reward with −C in high-cost states, while the algorithm replaces the entire Bellman target with a fixed negative value when the successor state is classified as high-cost. These objectives are not directly equivalent, and I would like to understand how the theoretical results apply to the implemented update. Proposition 4.1 also assumes that all safe policies avoid the high-cost region, while all unsafe policies have sufficiently large occupancy in it. This is a strong assumption that does not directly follow from the learned cost values. I therefore see the theory mainly as an analysis of the penalization idea; its support for the practical algorithm’s safety is still limited.

  2. The difference from the closest methods needs more direct experimental support. CPQ already uses a cost critic to restrict reward Bellman backups, so incorporating cost into reward-value learning before policy extraction is not entirely new. The more specific difference in SAVA is replacing the whole target with a fixed negative value based on the successor state’s cost value. Removing the penalized reward value shows that this component matters within SAVA, but it does not establish that this particular replacement rule is better than existing backup modifications. A comparison that keeps the actor, recovery mechanism, and other training settings fixed while changing only the Bellman target would make this much clearer. This directly affects my assessment of the method’s novelty and contribution.

Questions for authors
  1. I am still unclear about the connection between the theory and the actual algorithm. The appendix replaces the per-step reward with −C in high-cost states, while the algorithm replaces the entire Bellman target with a fixed negative value when a transition enters a high-cost region. How are these objectives related? Do Propositions 4.1 and 4.2 directly apply to the implemented update, or mainly explain the penalization idea?

  2. Proposition 4.1 assumes that safe policies never enter the high-cost region, while unsafe policies have sufficiently large occupancy in it. Under what conditions does this hold for regions defined by learned cost values? Does the estimation error in the theory cover mistakes in this region classification, or only errors in the reward value?

  3. Do you have a comparison that keeps the actor, recovery mechanism, and other training settings fixed and changes only the reward Bellman target? I would particularly like to see SAVA’s fixed negative target compared with CPQ-style backup masking. Removing the penalized reward value shows that this component matters, but I still want to understand how much of the improvement comes from SAVA’s specific target design.

  4. How were the penalty coefficient and the quantile α in the adaptive-threshold experiments selected? Were they chosen using only offline data, or also using evaluation rewards and costs from the environment? If online evaluation was used, how many configurations were tried, and did the baselines receive a similar tuning budget? This affects my assessment of both the comparison and how the method would be used in practice.

Reviewer_oYLcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Safety-Aware Value Alignment (SAVA), an IQL-based method for offline safe reinforcement learning. SAVA incorporates learned long-horizon cost information directly into reward value learning. A separately learned cost value V ̂_c partitions states into estimated low-cost and high-cost regions using a threshold l. When a successor state, or an h-step bootstrap state, is classified as high-cost, the usual reward Bellman target is replaced by a fixed negative target P_pen. The resulting penalty is propagated backward through Bellman backups and is intended to lower the reward value of actions that lead toward costly regions. For policy extraction, SAVA uses a conditional flow actor with classifier-free guidance, together with candidate re-ranking based on a cost recovery residual in estimated high-cost states. The paper provides two propositions: Proposition 4.1 claims that a sufficiently large penalty makes safe policies preferable to unsafe policies under a penalized objective, while Proposition 4.2 bounds the reward gap to the optimal safe policy. Experiments on 32 DSRL tasks report satisfaction of the normalized cost criterion on all tasks and the highest reward on 27 tasks. Ablations attribute the main safety gains to the region-aware penalized reward value.

Critical strengths
  1. The central mechanism is simple and potentially useful. Replacing the reward target by a fixed penalty when predicted future cost is high is easy to implement on top of in-sample value learning. It is conceptually distinct from applying a reward-cost scalarization only during policy extraction or deployment-time filtering.
  2. The component ablation isolates the main claimed mechanism reasonably well. In Table 3, removing the penalized reward value violates the constraint on all seven reported tasks while retaining the same flow-based actor. This is useful evidence that the reported safety behavior does not arise solely from actor expressiveness.
  3. The empirical scope is broad, and the headline results are strong. The evaluation spans Safety-Gymnasium, Bullet-Safety-Gym, and MetaDrive, with stricter budgets than several prior protocols.
Critical weaknesses

The reported main-results protocol is internally inconsistent Sections 4.1 and 5.1 state that the standard benchmark setting uses l=κ, with alternative threshold instantiations evaluated only in Section 5.3. However, at the reported precision, all 18 Table 9 entries overlapping with Table 1 match the “terminals only” column of Table 1. For example: CarButton1: 0.15/0.960 CarGoal1: 0.42/0.57 AntVel: 0.92/0.88 AntRun: 0.62/0.53 BallCircle: 0.51/0.49 The “terminals only” setting uses the data-dependent quantile threshold in Eq. (19), and Table 8 lists task-specific α values ranging from 0.5 to 0.99. Thus, the reported Table 9 results appear to use the adaptive, task-specific threshold setting rather than the stated l=κ protocol. This issue is central because the main claim—safe on 32/32 tasks and highest safe reward on 27/32 tasks—depends on these results. The paper does not explain how the per-task α, C_pen, and other task-dependent choices were selected. Therefore, it is not possible to determine from the paper whether this selection used an independent validation procedure or may have been informed by evaluation outcomes. The shared-α experiment also suggests that this choice matters: with a single α=0.8, Table 2 reports safety on 17/18 tasks in the calibration subset rather than all 18. Table 9 reports only three seeds and no standard deviations or confidence intervals; this is especially concerning for results close to the safety boundary, such as normalized costs of 0.95 and 0.96.

  1. The theory does not establish guarantees for the practical algorithm The theoretical analysis appears substantially more idealized than the implemented method. First, Proposition 4.1 relies on Assumption (23): every safe policy has zero discounted occupancy of S_high, whereas every unsafe policy has occupancy at least l. This assumes a separation between safe and unsafe policies through the high-cost region. However, the practical high-cost region is estimated from V ̂_c, and the paper does not provide conditions under which this learned partition satisfies Assumption (23). More fundamentally, l is introduced as a cumulative CMDP cost budget, whereas η^π is a discounted state-occupancy quantity. These can only be directly connected under additional assumptions relating the cost function to membership in S_high. Such assumptions are not stated for the general CMDP formulation in the paper. Second, the objective analyzed in the proof is not the same as the practical Bellman target. The theory defines a per-step reward of -C inside a fixed, known high-cost region. In contrast, SAVA detects a high-cost successor or h-step bootstrap state using a learned V ̂_c, then replaces the entire reward target by P_pen. These constructions differ in the timing and accumulation of penalties, and the paper does not establish an equivalence or approximation relationship between them. Finally, Proposition 4.2 becomes limited under Assumption (23), because the high-cost occupancy term is zero for safe policies by construction. The remaining guarantee is an approximate-greedy error bound based on sup-norm value error and maximization over “admissible” actions. While the appendix mentions an admissible policy class, it does not formally connect this assumption to the actual in-sample flow policy, classifier-free guidance, and finite-candidate decoding used by SAVA. The paper explicitly states that the practical C_pen is distinct from the theoretical penalty coefficient. This is reasonable, but it also means that the paper currently does not show how its practical penalty choices relate to the sufficient condition in Eq. (13).

  2. Evaluation and baseline comparability need more detail The paper should clarify whether all baseline numbers were reproduced under identical budgets, random seeds, training budgets, and evaluation protocols, or whether some numbers were obtained from previous reports. The Bullet-Safety-Gym dataset visualization uses a cost threshold of 10 for describing dataset composition, while evaluation uses κ=5. This may be reasonable, but the distinction should be made explicit in the experimental protocol and baseline comparison description.

Questions for authors

Which threshold rule produced Table 9 and Figure 2? Please provide complete 32-task results under: the stated main protocol l=κ; a single shared quantile α; the task-specific quantile setting, if retained. How were α, C_pen, h, and reward scaling selected? Was there a separate validation procedure that did not use the same task evaluation outcomes reported in the paper? Please report mean and standard deviation, or confidence intervals, across more random seeds, particularly for tasks with costs near the normalized threshold C=1. Were baselines re-run using the same budgets, seeds, number of training steps, and evaluation episodes? If not, which reported baseline values were taken from prior work? Can Assumption (23) be replaced or justified by a condition on the learned V ̂_c and its estimation error? In particular, what connects the learned high-cost partition to the original CMDP constraint? How does the per-step -C penalized objective in the theoretical analysis relate formally to the full-target replacement used in Eqs. (6) and (15)? V ̂_c is learned in-sample from behavior data. Why should the partition induced by V ̂_c remain valid for the extracted policy, whose future state distribution and cost-to-go may differ from the behavior policy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.