- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
Summary
This paper introduces DRESS, a framework for auditing whether environment shifts increase the safety cost of offline RL policies while preserving task reward. The method combines paired rollouts, separate discovery and confirmation stages, hierarchical bootstrap intervals, and training-seed sign tests. Experiments evaluate BC-Safe, BCQ-Lag, and CPQ under action delay and observation/action noise. The main finding is that action delay increases the safety cost of BCQ-Lag on DroneRun without substantially reducing reward. Additional experiments examine task variations, unseen checkpoints, and screening efficiency.
Critical strengths
Overall, I think this work has the following strengths
Strengths
-
The paper addresses a relevant problem in offline safe RL: maintaining task reward does not necessarily imply maintaining safety. The experiments illustrate how increased safety cost can be overlooked by reward monitoring and saturated binary violation metrics.
-
The evaluation incorporates useful precautions, including paired rollouts, separate discovery and confirmation data, and replication on unseen checkpoints. The authors also distinguish between amplification in already-unsafe policies and newly induced safety-limit violations, which helps clarify the scope of the findings.
Critical weaknesses
After reading this work, I have several concerns that I would like the authors to address.
-
Insufficient experimental evaluation. Table 1 presents results on only two held-out tasks, and the evaluated policies are limited to BC-Safe, BCQ-Lag, and CPQ. Other relevant methods, including COptiDICE [1], CDT [2], BEAR-Lag [3], and DRCORL [4], are absent from the evaluation. Applying DRESS to only these three policies is insufficient to establish its broader applicability. Furthermore, the main experiments cover only two tasks, with two additional tasks in the appendix. This narrow coverage makes it difficult to assess whether the findings generalize to other tasks and deployment conditions.
-
Concerns about methodological soundness and clarity. There is a mismatch between the target effect and the confirmation test. The amplification criterion in Eq. (3) concerns expected cost and return, whereas the exact sign test in Eq. (11) concerns the probability of a positive checkpoint-level effect. These are different statistical targets: most training runs can show a positive change while the mean change is negative. Although the paper distinguishes mean estimation from directional confirmation, it remains unclear what error guarantee applies to the final mean-effect claim, since Holm correction is applied only to the sign-test p-values. The presentation also needs clarification. The authors introduce numerous quantities, such as and , but their connections to the subsequent analysis are not always clear. More explicitly linking these quantities to the stages in Figure 1 and Algorithm 1 would help readers follow the overall procedure.
[1] Lee, Jongmin, et al. "Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation." arXiv preprint arXiv:2204.08957 (2022). [2] Liu, Zuxin, et al. "Constrained decision transformer for offline safe reinforcement learning." International conference on machine learning. PMLR, 2023. [3] Kumar, Aviral, et al. "Stabilizing off-policy q-learning via bootstrapping error reduction." Advances in neural information processing systems 32 (2019). [4] Zheng, Zhi, et al. "Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL." Advances in Neural Information Processing Systems 38 (2026): 159968-160002.
Questions for authors
I would appreciate it if the authors could respond to the following questions.
Questions
-
Why were the baseline methods only limited to BC-Safe, BCQ-Lag, and CPQ? Would the findings persist with additional methods such as COptiDICE[1], CDT[2], DRCORL[4] or BEAR-Lag[3], particularly for policies that satisfy the nominal cost limit?
-
Why was the main confirmation experiment designed with only three training seeds, given the resolution of the exact sign test? Can additional independent training runs establish the reported effect under the proposed confirmation criterion?
-
Are the Holm-adjusted p-values in Table 9 based on bootstrap tests or exact training-seed tests? How should they be interpreted alongside the confirmation procedure in Section 4.7?