- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes SAVA, an offline safe reinforcement learning method built on implicit Q-learning. A separately learned cost critic splits states into estimated low-cost and high-cost regions. Whenever a transition enters the high-cost region, the reward Bellman target is replaced by a fixed negative penalty. This penalty propagates backward, so actions leading toward hazards lose value before the policy is extracted. Extraction uses a conditional flow-matching actor with classifier-free guidance, plus a cost-based recovery rule for candidate actions in high-cost states. The paper has three contributions: (1) Injecting long-horizon cost directly into reward value learning through region-aware target replacement. (2) Two propositions: a sufficiently large penalty makes every safe policy preferable to every unsafe one, and the reward gap to the optimal safe policy is bounded by penalty, estimation, and extraction errors. (3) A data-dependent, quantile-based threshold for terminal-only value learning.
Critical strengths
- The paper is clearly structured and well written, presenting complex concepts in an accessible and easy-to-follow manner.
- The proposed method is conceptually sound and intuitive, offering a clean, well-motivated approach to incorporating safety constraints into offline value learning.
- The empirical results are convincing, demonstrating consistent and robust performance improvements over behavior cloning and five established offline safe RL baselines across benchmark suites.
Critical weaknesses
-
The core mechanism—replacing the reward Bellman target with a fixed penalty when predicted cost exceeds a threshold—largely integrates established concepts: CPQ-style cost-conditioned backups, absorbing failure penalties, and in-sample IQL learning. Furthermore, the paper's ablations indicate that auxiliary components like multi-step propagation and guided flow-based extraction primarily influence reward rather than safety. Because 1-step backups and simpler AWR extractors retain safety on nearly all evaluated tasks, the contribution rests heavily on a heuristic penalty whose conceptual departure from prior masking schemes remains modest.
-
There is a gap between the theoretical analysis in Section 4.2 and the proposed algorithm:
a. Proposition 4.1 relies on the strong assumption that the high-cost region perfectly separates safe and unsafe policies ( and ), which rarely holds when partitioning is learned from finite, noisy offline data.
b. The propositions analyze per-step penalties on a true region, whereas the algorithm applies an absorbing penalty to an empirically thresholded region with penalty coefficients () well below the theoretical bound. Consequently, the propositions offer intuitive insights rather than formal safety guarantees for the practical policy.
- The algorithm relies critically on the accuracy of the learned cost critic to partition the state space. In offline domains with limited coverage, estimation errors can lead to false penalties on viable behaviors or miss critical hazards. Additionally, while a shared quantile is evaluated, the reliance on data-dependent quantile thresholds () and task-specific calibration across environments suggests practical sensitivity that could hinder deployment without environment-specific tuning
Questions for authors
-
Beyond empirical benchmarks, under what structural conditions (e.g., environment horizon, hazard topology, or offline dataset coverage) is replacing the Bellman target with a fixed negative penalty expected to outperform the zero-masking approach used in CPQ?
-
The partition into low- and high-cost regions relies entirely on the learned value . How robust is the upstream value propagation to value-estimation errors or boundary inaccuracies in , and did the authors explore any explicit mechanisms (e.g., pessimistic cost ensembles or target smoothing) to ensure critic fidelity in sparse data regimes?
-
Standard Lagrangian dual-variable methods are widely used in safe RL to adaptively balance rewards and costs. Could the authors clarify why a primal-dual offline baseline (e.g., Lagrangian-regularized IQL) was omitted from the empirical comparisons, and how SAVA’s fixed region-aware penalty compares in stability and performance against an adaptive multiplier?