acceptodds
Under review as a conference paper at ICLR 2027

CASTLE: CAUSALLY ATTRIBUTED SOFT CONJUNC TIONS FOR SOUND AND INFORMATIVE SAFETY CERTIFI CATES

Abstract

A multi-predicate safety specification is almost always relaxed into a differen tiable conjunction, most often by the log-sum-exp conorm. The relaxation is then used as a penalty. Turning it into a certificate — a state predicate a runtime mon itor may act on — additionally requires choosing the value at which the conjunc tion is declared satisfied, and the standard choice, log K/β, is not the tightest admissible one. We make three contributions. (i) For the weighted conorm we derive the minimal admissible robustness threshold τω = log(1/ωk⋆)/β, pinned to the budget mass on the binding predicate; the resulting certificate is sound for every budget and carries no slack beyond the conorm’s own gap, whereas the uniform threshold of prior work adds log(Kωk⋆)/β of pure conservatism. (ii) We show that the quantity that should decide where budget goes is not bindingness but control authority, and introduce IARS, an interventional attribution that measures per-predicate action sensitivity conditional on the causal order. It is identically zero for agent-independent monitors and for restatements of an already-binding predicate. (iii) We prove that the resulting decomposition — a soft conjunction over attributable predicates conjoined with an exact nominal check on the rest — is sound, and we formalise the design principle that robustness margin should be spent only where the agent can act. Across three environments and three seeds, CASTLE raises the activation rate of a sound certificate from 0 .215 to 0 . 381 on a specification containing a non-attributable predicate — at zero certified-but-unsafe steps — while matching or improving the safety–performance frontier; removing the exact monitor reproduces 3253 certified-but-unsafe steps, while the product t-norm certifies 1463 such steps at its natural threshold and the Łukasiewicz cer tificate never activates at all.

Primary area:
reinforcement learning
Keywords:
Constrained Reinforcement Learning, Safe Reinforcement Learning, Causal Intervention, Log‑Sum‑Exp, Safety Certificate, Multi‑Predicate Constraints

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_S7UjOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CASTLE, a method for learning under multiple safety constraints and checking when those constraints are satisfied. It distinguishes constraints the agent can influence from conditions outside its control and redundant constraints. CASTLE combines a smooth safety check for controllable constraints with a direct check for uncontrollable conditions. The authors derive a threshold for the smooth check and evaluate the method in three simulated control environments.

Critical strengths

The paper makes a useful distinction between a safety penalty used for training and a certificate used to declare a situation safe. The weighted bound and its associated conditional soundness guarantee provide a mathematical basis for constructing such a certificate.

Critical weaknesses

The empirical conclusions are inconsistent with the reported results. Section 5.1 claims that CASTLE achieves the best safety–performance trade-off on SafeConf. However, Table 1 reports lower mean success and higher mean cost for CASTLE (0.89, 0.12) than PPO-Lagrangian (0.97, 0.09) and DENSSE-LSE (0.97, 0.11). These results do not support the claimed superiority. The safety-audit counts are also inconsistent: counting certified-but-unsafe steps over all predicates should yield at least as many as counting only controllable predicates, but the Product baseline reports the reverse, including 259 versus 518 on SafeNavC. These discrepancies undermine confidence in the central empirical conclusions.

Questions for authors

Could the authors reconcile the claimed safety–performance improvements with Table 1 and clarify the definitions and aggregation of CBU and CBU_all? Which numbers or conclusions require correction?

Reviewer_jNHeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Admittedly, the drafting and editing of the paper involved AI assistants. This is not a problem per se, but the resulting manuscript is extremely hard to parse, understand, and interpret. This is my rough understanding of it:

The paper studies an RL setting with safety constraints. Every constraint is associated with a margin (a number measuring how far the RL agent is from violating said constraint), and the goal is to maintain every margin (and hence the minimum one) nonnegative. Since the minimum has poor gradients, one can replace it with a smoothed version of it (a weighted log-sum-exp of the margins), and require such smoothed version to be above a threshold. The goal is to find the smallest threshold that guarantees that, whenever the smoothed version is above it, every margin is at least of a certain value. The authors show that this threshold depends on the weight of the constraint with the smallest margin and, building on prior (anonymous) work, they change the weights of the constraints, so that the constraints that the agent cannot affect are removed from the smoothed version and checked exactly. They empirically corroborate their findings in three small simulated tasks.

Critical strengths

None.

Critical weaknesses

i. Admittedly, the drafting and editing of the paper involved AI assistants. This is not a problem per se, but the resulting manuscript is extremely hard to parse, understand, and interpret. Moreover the text, tables and proofs are often in contradiction. For instance:

  • The abstract and introduction employs terms such as "bindingness", "control authority", "budget mass" and "natural threshold" before explaining them, and Table 1 never defines many of its metrics;
  • Eq. (1) uses a function εθ\varepsilon^\theta that is never defined, and it follows a citation that prints as "?";
  • Section 5.1 says that CASTLE "is best on SafeConf", but Table 1 contradicts this;
  • By the definitions in the caption of Table 1, I deduce that CBU_all is always greater than CBU. Table 1, however, reports the opposite in every row where the two are nonzero;
  • Three citations print as "?", and Fig. 5 refers to a Prop. 3 that does not exist.

ii. From my understanding, the algorithm proposed by the authors requires every residual as input (Algorithm 1, requirements), so it can immediately compute the minimum exactly. I therefore do not understand the purpose of the proposal. The smoothed version is useful during training, where it provides gradients. At deployment, however, one can check the minimum exactly. The paper does not compare with this immediate alternative, although the safety certificate is the paper's point.

Questions for authors

None.

Reviewer_TzK5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the soundness and informativeness of differentiable logical conjunction operators when used as safety certificates in multi-constraint safe reinforcement learning. To address the issue that the commonly used certificate threshold for the Log-Sum-Exp (LSE) operator is not the tightest admissible threshold, the authors propose CASTLE. The method derives the minimal admissible robustness threshold for the weighted LSE conorm and introduces IARS, a control-authority-based interventional attribution method. By considering action sensitivity and constraint redundancy, IARS identifies attributable safety predicates and combines them with exact monitoring of non-attributable predicates. Experiments are conducted across three environments with three random seeds. The results show that CASTLE improves the activation rate of a sound safety certificate from 0.215 to 0.381 for a safety specification containing a non-attributable predicate. Ablation experiments further show that removing the exact monitoring mechanism results in 3,253 certified-but-unsafe steps.

Critical strengths
  1. Valuable research problem: The paper addresses the different requirements of differentiable safety constraints when used as training penalties versus safety certificates. It highlights that safety certificates must not only guarantee soundness but also achieve a sufficiently high activation rate, making the research problem meaningful.

  2. Methodological novelty: The paper derives a safety certification threshold for the weighted LSE operator and incorporates action sensitivity and constraint redundancy into predicate attribution. This approach offers a degree of methodological novelty.

Critical weaknesses
  1. Insufficiently rigorous theoretical justification: The paper's soundness guarantees depend on the attribution mechanism correctly distinguishing controllable, uncontrollable, and redundant predicates. However, the proposed method primarily relies on action gradients and cosine similarity for classification, without proving that this procedure satisfies the logical implication conditions required by Theorem 2. Furthermore, the convergence claim in Proposition 1, stating that the certificate activation rate approaches its upper bound as β increases, is not sufficiently justified. These issues raise significant concerns about the rigor of the theoretical analysis.

  2. Inconsistencies between experimental claims and reported results: The paper claims that CASTLE achieves the best safety–performance trade-off on SafeConf. However, Table 1 reports a success rate of 0.89 and a cost of 0.12 for CASTLE, both worse than those of DENSSE-LSE, which does not support this claim. Additionally, the CBU values for the Product method in Table 1 are 518, 562, and 975, summing to 2,055. In contrast, the 1,463 certified-but-unsafe steps reported in the main text correspond to the sum of the CBUall column. These discrepancies raise concerns about the consistency between the experimental claims and the reported data.

  3. Insufficient experimental validation: The experiments are conducted only in custom-built simulation environments, and the main benefits of the attribution mechanism are concentrated in SafeConf, which was specifically designed to include redundant and uncontrollable predicates. Furthermore, the primary comparisons focus mainly on classical safe reinforcement learning methods and logical operators, without direct comparisons against recent methods for multi-constraint safe reinforcement learning. As a result, the experimental evidence is insufficient to establish the broader effectiveness and competitiveness of the proposed approach.

Questions for authors

Q1: How does IARS guarantee that the dropped predicates satisfy the logical implication conditions required by Theorem 2?

Q2: How exactly are CBU and CBUall calculated in Table 1? What is the relationship between the 1,463 certified-but-unsafe steps reported in the main text and the values in the table? Why is CBUall smaller than CBU?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.