acceptodds
Under review as a conference paper at ICLR 2027

Auditing a Learned Admission Gate for Offline-to-Online Safe RL: Distinct Certificates and Public Comparator Failures

Abstract

We audit one learned admission gate between BC-All proposals and BC-Safe fallbacks in offline-to-online safe reinforcement learning. We distinguish two certification targets. Fixed-time split-conformal accounting controls an immediate-cost proxy under deployment exchangeability and separately assumed fallback control. A weighted order-statistic rule instead bounds the marginal next-trajectory event \(\Pr(G_T>D,\mathcal Q\ne\varnothing)\le\alpha\) for a prespecified family of complete policies; it is not the high-confidence population-risk target of Learn then Test, and neither result protects qualification rollouts or repairs policy-induced shift. A synthetic study shows the intended mechanism and its pre-certificate exposure. On three public Safety-Gymnasium Point tasks, 54 completion/rejection-qualified BC-All/BC-Safe checkpoint pairs fail the frozen joint completion–violation criterion against a pre-treatment, deployable rate-matched mixer. A separate retrospective stress test permutes each gate trajectory's realized fallback mask at equal count and horizon; it is post-treatment and not a causal timing comparison. Gate violation is higher at all task-level point estimates under both comparisons, while completion often increases. Finally, an outcome-inspected diagnostic applying the trajectory rule to 600 retained gate trajectories per selected checkpoint yields an empty qualified set (0/54; rank 600; order statistics 134–810 for budget 10), under an unverified cross-campaign exchangeability premise. Thus the instantiated gate refuses trajectory certification and shows no joint public benefit; this does not generalize to a gate class or establish deployment safety.

Primary area:
reinforcement learning
Keywords:
offline-to-online reinforcement learning, safe reinforcement learning, admission gates, conformal prediction, policy switching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jXA9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper audits a learned admission gate used to switch between a BC-All proposal and a BC-Safe fallback in offline-to-online safe RL, clarifying what such a gate can and cannot certify under conformal-style assumptions. It distinguishes fixed-time per-decision proxy accounting (via split-conformal calibration of an immediate-cost surrogate) from a finite-horizon, trajectory-level next-unit bound over a prespecified family of complete policies (via a weighted order-statistic rule), emphasizing both targets’ limits relative to Learn-then-Test (LTT) and to policy-induced shift.

Critical strengths
  1. Clear and useful separation between two certification targets: fixed-time per-decision proxy control and finite-horizon, next-trajectory chance constraints for complete policies.
  2. Provides a much-needed, critical lens on admission gating in safe O2O RL, clarifying common confusion.
Critical weaknesses
  1. The paper is dense and notationally heavy; some readers may find the separation between targets and their assumptions difficult to track.
  2. Most theoretical ingredients (split-conformal calibration, union-bound weighting across candidates, convex support geometry) are adaptations of known tools; novelty is more in framing and audit practice than in new theory.
Questions for authors
  1. This paper looks like auto-research. And I do not find "AI USE STATEMENT" in the submission. Which parts of this paper were produced by AI?
  2. Can you elaborate on practical strategies to plausibly justify the complete-unit exchangeability assumption behind Proposition 3 in real deployments?
  3. How sensitive are your public-audit conclusions to the specific frozen joint criterion? Could you provide a sensitivity sweep or rationale for these thresholds?
Reviewer_uaCsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper appears to audit offline to online safe reinforcement learning gates of some kind. The contributions seem to be the development of this audit methodology and the findings of their analyses.

Critical strengths

None

Critical weaknesses
  • The framing and motivation for the this work are unclear. It is not well established in the abstract, introduction or related work what the goal of this paper is, why we need this work, and what problem or gap it fills. In the related work, it is unclear why these methods are being compared to, and what the novelty of this work is with respect to those previous works.
  • This paper has very poor presentation and is extremely hard to follow. The paper has no transitions and jumps into jargon-heavy technical details and context without ever prefacing the relevance to the project goal/problem or what the paragraph is introducing. For example, the methodology jumps into equations and formulations without ever describing the goal (i.e. the why about what is being introduced.) In the methods it is unclear why the propositions and proofs are needed, and what claims they are supposed to be supporting. Abbreviations are introduced but never defined (e.g., "LTT", "BC-ALL", "BC-Safe"). All of these factors make the paper nearly unreadable.
  • The methodology and experimental sections do not appear sound, in part related to the previous point because the presentation of these sections is obscured by the lack of readability. The experimental section does not present any experimental set up (i.e., what experiments are being conducted for what purpose to establish what claim.)
Questions for authors
  • What is the motivation for this work? What is the problem being solved?
  • Where is the novelty? How does this differ from previous works?
  • What is the overarching layout of the methodology or framework developed?
  • What is the experimental setup and what claims are being tested in the results?
Reviewer_3m9YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides a new algorithm that audits a learned admission gate that switches between a proposal policy and a conservative fallback policy in offline-to-online safe reinforcement learning. This method distinguishes fixed-time split-conformal accounting for an immediate-cost proxy from qualification based on complete-trajectory costs. The authors also discusses the limitations imposed by unsupported state-action pairs and the safety exposure incurred while collecting qualification trajectories.

The empirical evaluation includes synthetic experiments and three Safety-Gymnasium Point tasks, with the principal public analysis covering 54 selected BC-All/BC-Safe checkpoint pairs. The gate is compared against a deployable, rate-matched random mixer and a retrospective permutation of its realized fallback schedule. No task satisfies the prespecified joint completion-violation improvement criterion, and the retrospective trajectory-qualification procedure qualifies none of the evaluated gates.

The distinctions between certification targets and the attention to appropriate comparators are useful. However, I find the theoretical novelty limited, and the experiments do not establish sufficiently general or explanatory insights beyond the failure of this particular gate implementation.

Critical strengths
  1. The paper distinguishes immediate-cost control from trajectory-level safety, marginal guarantees from conditional or high-confidence population-risk guarantees, and post-qualification guarantees from safety during qualification. These distinctions help prevent overinterpretation of conformal guarantees.

  2. Comparing a gated policy only against its proposal policy can conflate the value of state-dependent selection with the benefit of executing a conservative fallback more often. The V4 rate-matched random mixer provides a useful control for evaluating whether the learned selection rule adds value.

  3. The authors retain unfavorable and null outcomes, acknowledge that V5 does not identify a causal effect of intervention timing, and use checkpoint pairs rather than individual episodes as the inferential units. These are positive aspects of the evaluation.

Critical weaknesses

W1. Limited theoretical novelty

Proposition 3 appears to follow directly from standard split-conformal rank guarantees and a weighted union bound over complete policies. The error allocation and rank-feasibility conditions are useful, but a substantial theoretical advance is unclear. Proposition 1 is direct monotonicity; Proposition 2 aggregates candidate and fallback risks; Propositions 6-7 apply standard convex geometry without yielding a gate-learning algorithm. Propositions 4-5 clarify important limitations through relatively elementary arguments.

The differences from LTT [1], CRC [2], and CPC [3] are appropriately acknowledged. However, a distinct certification target alone does not establish novelty; the paper should explain what new capability or understanding it enables.

W2. Limited general insight from one author-constructed gate

The gate combines established components rather than reproducing a representative published method. Its failure may therefore reflect implementation, training, or calibration choices rather than a broader limitation of safety gating. V4 establishes failure against the chosen mixer, but does not distinguish critic error, state-distribution shift, fallback inadequacy, or calibration-target mismatch. V5 cannot isolate timing causality, while V6 provides no clear component-level explanation.

Negative findings are valuable, but the paper does not sufficiently identify when and why this failure occurs or which other designs the lesson applies to. This limits its significance as an audit.

W3. Narrow evaluation and unclear positioning against existing offline RL

The principal evaluation covers only PointGoal1, PointButton1, and PointPush1. The 54 checkpoint pairs strengthen replication across seeds, not generality across dynamics, difficulty, or dataset quality. Exhaustive benchmark coverage is unnecessary, but broader representativeness is insufficiently justified; earlier experiments with uninformative completion outcomes do not resolve this concern.

Because the proposal is BC-All, the study also leaves unclear the gate's incremental value for stronger offline RL policies such as CQL or IQL, and its position relative to CAPS [4]. Representative comparisons or a general failure mechanism would strengthen the contribution; state-of-the-art performance is not required.

W4. Weak connection between theory and public evidence

Deployment exchangeability and fallback-risk bounds are not established publicly, and trajectory qualification is retrospective. The experiments therefore do not demonstrate useful operation of the theoretical procedure under realistic conditions.

The empty qualified set is also unsurprising given violation rates far above the target level; even policy-by-policy qualification fails. This supports reasonable refusal, but does not show an advantage over existing certification approaches. Although these limitations are acknowledged, the theory and experiments do not sufficiently reinforce one another.

W5. Incomplete performance reporting in the principal comparisons

Raw return and mean cost are reported in Section 6 and Appendix H, Table 7. However, the principal V4/V5 tables emphasize completion and violation differences without jointly presenting return, mean cost, and uncertainty. These metrics are complementary. Consistent reporting, including DSRL normalized scores where applicable, would clarify the performance-safety trade-off. This is a secondary concern that compounds W3.

References cited in Weaknesses

  • [1] Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., and Lei, L. "Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control." The Annals of Applied Statistics, 2025.
  • [2] Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., and Schuster, T. "Conformal Risk Control." ICLR, 2024.
  • [3] Prinster, D., Fannjiang, C., Park, J. W., Cho, K., Liu, A., Saria, S., and Stanton, S. "Conformal Policy Control." ICML, 2026.
  • [4] Chemingui, Y., Deshwal, A., Wei, H., Fern, A., and Doppa, J. "Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning." AAAI, 2025.
Questions for authors
  1. What is the substantive theoretical novelty of Proposition 3? Please identify what goes beyond standard conformal rank guarantees and weighted union bounds. What practical decision-making advantage does its certification target offer relative to the alternatives discussed?

  2. Why is this particular gate a sufficiently representative audit target? Which aspects reflect common published designs, and what evidence distinguishes implementation-specific failures from limitations that plausibly apply more broadly?

  3. Can the existing logs explain the V4 failures? For example, do analyses of risk-ranking quality, offline-to-deployment prediction errors, or downstream risk following fallback interventions help identify the dominant mechanism?

  4. Can the principal comparisons include the complete performance vector? Please report return, mean cost, completion, violation, and checkpoint-level uncertainty for the V4 arms. Also clarify whether any existing competitive-baseline results can be compared under the same protocol.

  5. Is the voting constraint redundant under the instantiated robust-cost condition? With the same threshold in Equations (4)-(6), K=3K=3, and κ=2\kappa=2,

    max⁡kQk≤μ+2σ≤μ+2σ.\max_k Q_k \leq \mu+\sqrt{2}\sigma \leq \mu+2\sigma.

    Passing the robust-cost constraint therefore appears to imply that every critic passes the corresponding voting threshold. Does this relationship hold in the implementation, and how should the identical no-vote decisions in V6 be interpreted?

1 comment

Thank you for the detailed clarifications. However, my main concerns remain unresolved. In particular, the implementation replaces the computed conformal threshold with a quantile targeting a 20% rejection rate, leaving a substantive gap between the theoretical specification and the evaluated gate. I therefore keep my recommendation of weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.