acceptodds
Desk rejected at ICLR 2027

DRESS: Auditing Reward-Preserving Safety-Cost Amplification in Offline RL

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Offline safe reinforcement learning is commonly evaluated under nominal dynamics and monitored through task return or binary constraint violations. Such evaluation can miss a critical deployment failure mode: a small environment shift may preserve task performance while substantially amplifying safety cost. We introduce DRESS (Directional Reward-preserving Environment Shift Screening), a post-training framework that converts this failure mode into a statistically controlled audit. DRESS retains perturbation direction to prevent cancellation, uses paired nominal and shifted rollouts, and tests cost increase jointly with reward noninferiority. A discovery-confirmation split, hierarchical effect intervals, exact training-seed tests, and family-wise correction separate candidate selection from reproducible evidence while respecting the hierarchical structure of RL evaluation. Experiments across multiple offline safe RL algorithms, tasks, and deployment shifts reveal a previously hidden delay-induced safety-cost amplification that is invisible to a saturated binary violation metric and persists on unseen policy checkpoints. Broader stress tests and nominal-placebo analyses further demonstrate that DRESS produces selective, well-calibrated conclusions, while its adaptive acquisition strategy improves screening efficiency under limited evaluation budgets.

Primary area:
reinforcement learning
Keywords:
offline safe RL, distribution shift, policy auditing, statistical evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_YBTaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

Summary

This paper introduces DRESS, a framework for auditing whether environment shifts increase the safety cost of offline RL policies while preserving task reward. The method combines paired rollouts, separate discovery and confirmation stages, hierarchical bootstrap intervals, and training-seed sign tests. Experiments evaluate BC-Safe, BCQ-Lag, and CPQ under action delay and observation/action noise. The main finding is that action delay increases the safety cost of BCQ-Lag on DroneRun without substantially reducing reward. Additional experiments examine task variations, unseen checkpoints, and screening efficiency.

Critical strengths

Overall, I think this work has the following strengths

Strengths

  1. The paper addresses a relevant problem in offline safe RL: maintaining task reward does not necessarily imply maintaining safety. The experiments illustrate how increased safety cost can be overlooked by reward monitoring and saturated binary violation metrics.

  2. The evaluation incorporates useful precautions, including paired rollouts, separate discovery and confirmation data, and replication on unseen checkpoints. The authors also distinguish between amplification in already-unsafe policies and newly induced safety-limit violations, which helps clarify the scope of the findings.

Critical weaknesses

After reading this work, I have several concerns that I would like the authors to address.

  1. Insufficient experimental evaluation. Table 1 presents results on only two held-out tasks, and the evaluated policies are limited to BC-Safe, BCQ-Lag, and CPQ. Other relevant methods, including COptiDICE [1], CDT [2], BEAR-Lag [3], and DRCORL [4], are absent from the evaluation. Applying DRESS to only these three policies is insufficient to establish its broader applicability. Furthermore, the main experiments cover only two tasks, with two additional tasks in the appendix. This narrow coverage makes it difficult to assess whether the findings generalize to other tasks and deployment conditions.

  2. Concerns about methodological soundness and clarity. There is a mismatch between the target effect and the confirmation test. The amplification criterion in Eq. (3) concerns expected cost and return, whereas the exact sign test in Eq. (11) concerns the probability of a positive checkpoint-level effect. These are different statistical targets: most training runs can show a positive change while the mean change is negative. Although the paper distinguishes mean estimation from directional confirmation, it remains unclear what error guarantee applies to the final mean-effect claim, since Holm correction is applied only to the sign-test p-values. The presentation also needs clarification. The authors introduce numerous quantities, such as Δm^θ\widehat{\Delta m}{\theta} and djim,θd{ji}^{m,\theta}, but their connections to the subsequent analysis are not always clear. More explicitly linking these quantities to the stages in Figure 1 and Algorithm 1 would help readers follow the overall procedure.

[1] Lee, Jongmin, et al. "Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation." arXiv preprint arXiv:2204.08957 (2022). [2] Liu, Zuxin, et al. "Constrained decision transformer for offline safe reinforcement learning." International conference on machine learning. PMLR, 2023. [3] Kumar, Aviral, et al. "Stabilizing off-policy q-learning via bootstrapping error reduction." Advances in neural information processing systems 32 (2019). [4] Zheng, Zhi, et al. "Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL." Advances in Neural Information Processing Systems 38 (2026): 159968-160002.

Questions for authors

I would appreciate it if the authors could respond to the following questions.

Questions

  1. Why were the baseline methods only limited to BC-Safe, BCQ-Lag, and CPQ? Would the findings persist with additional methods such as COptiDICE[1], CDT[2], DRCORL[4] or BEAR-Lag[3], particularly for policies that satisfy the nominal cost limit?

  2. Why was the main confirmation experiment designed with only three training seeds, given the resolution of the exact sign test? Can additional independent training runs establish the reported effect under the proposed confirmation criterion?

  3. Are the Holm-adjusted p-values in Table 9 based on bootstrap tests or exact training-seed tests? How should they be interpreted alongside the confirmation procedure in Section 4.7?

Reviewer_Z9aPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DRESS, a post-training framework for auditing offline safe RL policies under environment shifts. It combines statistical tools to identify perturbations that increase safety costs while preserving rewards. The framework is primarily evaluated on two tasks, AntRun and DroneRun, using three offline safe RL policies, with additional robustness and replication experiments.

Critical strengths

None

Critical weaknesses
  • The paper argues that reward-only monitoring overlooks safety failures, although safe RL already explicitly evaluates constraint costs. Its main finding concerns a policy whose nominal cost is 91.2 against a safety limit of 10. Increasing its cost to 111.2 does not change its safety classification, and no statistically supported nominally safe-to-unsafe transition is demonstrated.

  • The focus on reward-preserving cost amplification is insufficiently motivated, since safety violations remain consequential regardless of reward changes. The paper does not establish why this restricted criterion provides meaningful advantages over directly evaluating constraint satisfaction under environment shifts.

  • Although the English is mostly grammatically correct, the prose is not consistently coherent or natural. The introduction reads more like a collection of technical claims than a coherent argument. The writing is overloaded with abstract phrases such as “directional reward-preserving safety-cost amplification,” “interval-qualified,” and “exact directional confirmation.” While these terms are defined, their repeated use makes straightforward ideas unnecessarily difficult to understand. Sentences frequently appear stitched together rather than forming a smoothly developed scientific explanation.

Questions for authors

Please check weaknesses.

Reviewer_P2FvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an audit procedure for frozen offline safe-RL policies that asks a conjunctive question about deployment shifts: does the safety cost increase while task return stays within an agreed tolerance? The authors argue that return-only monitoring misses this regime and that a symmetric average over shift directions can cancel it, so candidates are kept directional. The procedure pairs nominal and shifted rollouts on the same episode seed, forms a hierarchical bootstrap with training runs as the outer cluster and episodes as the inner one, and separates a discovery phase on one range of episode seeds from a confirmation phase on a disjoint range. Confirmation is a one-sided exact sign test over per-checkpoint paired means, combined across the two conditions through an intersection-union maximum and adjusted with Holm over a prespecified family. Experiments audit three offline safe-RL policies on two tasks under action delay and two noise families, giving eighteen task-policy-shift claims, plus an outcome-blind five-checkpoint replication, a four-task severity expansion, and a nominal placebo.

Critical strengths
  1. The decision object is a real gap and the paper states it precisely. Separating "cost increases" from "cost increases while performance is preserved" turns a vague robustness question into a conjunction that can be tested, and keeping direction as part of candidate identity fixes a genuine defect of symmetric perturbation averaging that costs nothing to fix and is easy to miss. The taxonomy that follows from it, distinguishing a limit crossing from severity amplification in a policy that is already infeasible, is the part I would expect other evaluation work to adopt.

  2. The statistical construction is correct and well documented. Pairing on episode seed, clustering the bootstrap at the training run while resampling episodes within it, using an intersection-union maximum for the conjunctive alternative, and applying Holm to the prespecified family are each the right choice, and they are combined without hidden shortcuts. The paper also reports where its own machinery lands rather than only where it succeeds: the caption of the held-out table states that no row passes exact family confirmation, the appendix reports the full eighteen-claim family, and the placebo study quantifies false qualification at 2.17% over ten thousand repeats. That level of disclosure is what makes the paper useful as a template even where its empirical yield is thin.

Critical weaknesses
  1. The confirmation tier cannot fire at the scale on which it is run, so the two-level design reduces to a single confidence interval in practice. With three training runs the smallest attainable one-sided exact sign-test p-value is 1/8, which exceeds 0.05 even for a one-claim family, so no candidate in the main cohort could ever be confirmed regardless of effect size; the reported joint value of 0.125 is the best the design can produce, not a near miss. The replication uses five runs, where the floor is 1/32, but it is entered into an 18-claim Holm family whose smallest threshold is 0.05/18, so even a unanimous five of five would have failed; the observed four of five gives 0.1875. The single detected effect is therefore interval-qualified and nothing more, and the abstract's framing that the audit "reveals" an amplification that "persists on unseen checkpoints" is stronger than a p-value of 0.125 with no family-level rejection supports. Why this is consequential: the two-level design, with discovery separated from exact confirmation and multiplicity control, is contribution two of the paper. At three and five training runs on an eighteen-claim family, that machinery cannot reach its own second level, and the reader is left with a bootstrap interval plus considerable bookkeeping.

  2. The empirical payload is one cell, and it sits on a policy that already violates the constraint in every nominal episode. DroneRun BCQ-Lag has nominal cost 91.2 against a limit of 10 and a violation rate of 1.000; the detected amplification raises mean cost from 91.2 to 111.2, which is a 22% increase in a regime that is already infeasible by a factor of nine. The binary violation metric is constant at one there, so the observation that it fails to register the change is arithmetic rather than a subtle monitoring blind spot, and the alternative suggested, a continuous cost statistic, is what the paper's own CVaR and mean-cost columns already report. The abstract presents this as a previously hidden failure mode without conveying that the audited policy is one nobody would deploy and that the effect size is measured on an already-infeasible cost. Why this is consequential: the paper's applied claim is that this audit finds things standard evaluation misses. What it actually finds is that a saturated indicator cannot move, on a policy whose nominal violation rate is one; the more interesting candidate in the same table, the nominally lower-cost policy whose delay-induced cost rise comes with a collapse in return, is classified as joint degradation and set aside.

  3. The paper does not make its own contribution legible, and its key terms are defined neither in the abstract nor on first use. The abstract names seven devices in two sentences — retaining perturbation direction, paired rollouts, the joint test, a discovery-confirmation split, hierarchical effect intervals, exact training-seed tests, and family-wise correction — and "discovery-confirmation split" never appears again in that form anywhere in the paper, so a reader has to reconstruct what it denotes from the mechanics of the confirmation sections. "Power analysis" appears once, as a citation in related work, with no explanation of what it means for the design; more to the point, no power analysis is carried out anywhere, which is precisely the calculation that would have shown the exact tier cannot fire with three training runs. The gap the paper addresses is compressed into one sentence of the introduction, while the four failure modes that actually motivate the design are deferred to the design-objective subsection, so the reader must assemble problem, gap and mechanism across three separate places; at no point does the paper say which of the seven devices is load-bearing for the guarantee and which is optional engineering. Why this is consequential: in a methodology paper the contribution is the design, and the reader's ability to see which part of the design is the contribution is what the paper is offering. As written, the method sections read as a sequence of choices with justification attached rather than as a response to a stated problem, so a reader who wants to borrow the audit cannot tell what to take and what to leave.

Questions for authors
  1. Was it known before running that three training runs put the exact tier out of reach, and what would confirmation require? With three runs the floor is 0.125, with five it is 0.031, and an eighteen-claim Holm family needs 0.0028 for its smallest test. A short power calculation in the paper would make the design's requirements explicit, and would let readers judge how many runs the confirmatory mode actually needs.

  2. How should the headline be read given that the audited policy violates the cost limit in one hundred percent of nominal episodes? What does DRESS conclude on a nominally feasible policy, and is an amplification in an already-infeasible regime the deployment failure the framework is designed to catch, or a secondary category?

  3. The reward tolerance is five percent of each policy's own nominal return. For the policy with normalized return near 0.03, that band is a few units, and cells with a return change of −0.60 and an interval crossing zero are labelled reward-degrading. Is that label comparable across policies, or should the tolerance be defined on a task-level scale so that it does not classify statistical noise as degradation?

  4. The acquisition score multiplies a cost upper confidence bound by a normal-CDF term for reward feasibility. That product is not an estimate of the joint qualification event that the audit actually tests. Why this form rather than a criterion aligned with the decision rule, and how much of the 0.300 against 0.200 hit-rate difference over ten development problems is within noise?

  5. The eighteen claims draw on the same three training runs per policy and are therefore correlated. Holm is valid but conservative here, and the conservatism is what blocks confirmation. Would a focused family, or a hierarchical multiplicity correction, be more appropriate given the discovery-confirmation split?

  6. Will the audit code and the shift wrappers be released? The pairing, the split ranges, the freezing rules and the family definition all matter for reproduction, and the paper describes them precisely enough that code would make them usable by others.

  7. Which components are load-bearing for the guarantee and which are optional? If a practitioner adopted only the paired design and the conjunctive test, without the discovery split, the exact sign test or the acquisition heuristic, what would they lose in validity, and what is the minimal recipe the paper would stand behind?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.