- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Certified Safety Curation for safe offline RL without dense cost supervision. It learns a state-only safety value from segment preferences, selects trajectories based on aggregated safety scores, and uses Learn-then-Test calibration to certify the unsafe fraction of the selected dataset before behavior cloning. Experiments on DSRL benchmarks demonstrate competitive safety performance against existing safe offline RL methods.
Critical strengths
-
The paper addresses a meaningful problem of safe offline RL under limited safety supervision and provides a statistical guarantee on the composition of selected training data.
-
The experiments cover multiple benchmarks and include extensive comparisons and ablations, showing that trajectory-level filtering can be effective for learning safer policies.
Critical weaknesses
-
The paper emphasizes the importance of trajectory-level safety, yet models safety using a state-only value function aggregated over time, potentially overlooking action-dependent or temporal safety information. More importantly, the preference pairs are constructed using ground-truth episodic costs to identify high- and low-cost trajectories (Appendix N), contradicting the motivating setting where such cost information is unavailable. The effectiveness of the method under genuinely cost-free preference sampling remains unclear.
-
The calibrated method and its uncertified V-filter achieve the same safety coverage (12/15 tasks), while only 4/15 tasks receive certificates under the default setting. Moreover, a simple classifier trained on 200 binary safety labels achieves comparable results (11/15), and a label-splitting approach provides similar certification rates. These results suggest that most benefits come from trajectory filtering rather than the proposed preference-learning and certification pipeline. Furthermore, the certificate only guarantees dataset composition, not the safety of the learned policy, limiting its practical significance.
Questions for authors
-
How does the method perform when preference pairs are sampled without relying on ground-truth costs? Have the authors considered state-action or sequence-based safety models to capture temporal dependencies?
-
Given the comparable performance of simple label-based filtering and the low certification coverage, what are the concrete advantages of the proposed pipeline under matched supervision budgets?
Thank you for the careful review.
Cost-free pairs. With pairs chosen uniformly, without cost (App. N), 12 of 15 DSRL tasks and all 5 Bullet tasks stay within budget. On DSRL certification falls from 0.161 to 0.089, the calibration draw used for the reported policies certifies 2 tasks rather than 5, and some rewards fall.
State-action and sequence models. Q(s,a) left four of five tasks unchanged and was less safe on Hopper, a result App. L calls suggestive. A sequence scorer would be covered by the guarantee, since any score fixed before the calibration draw is (Sec. 4.3), but we have not tested one. The state value, which does not condition on actions, still falls as violations approach on all nine analysis tasks (App. E).
Advantages at matched budgets. That trajectory filtering carries much of the safety is the paper's own submitted finding: its first contribution is that the readout scale, not the supervision, is the binding choice (Sec. 1). At matched labels, labels-only keeps 12 tasks to our 12, with no effort advantage for preferences, and a labels-only 100/100 split certifies about as often as ours (Sec. 6.4, App. M). The V-filter, the same score without the certificate, also keeps 12. What certification adds is a finite-sample guarantee or an explicit refusal; the refusal blocks reward-aware sub-selection on Hopper, where ungated it reaches mean cost 82 (Sec. 6.2).
Coverage and policy safety. Five of 15 tasks certify at α = 0.25 and eight of the nine analysis tasks at α = 0.40; one of the five, PointButton1, is a false certification, an event the guarantee allows with probability at most δ (Sec. 6.2). Elsewhere it refuses, as designed; Proposition 3 predicts the rate, so a deployer can estimate the labels a target rate needs (Sec. 6.3; the estimate runs high where certification is rare, App. J). As the abstract says, what is certified is the training set, not the policy. That is by design: offline, the data bounds what any method can achieve, and certifying it is a guarantee available before a policy is trained or run (Sec. 8). On the five DSRL tasks that certify, CDT trained on certified selections stays within budget in all 42 runs at α = 0.25, five of them on selections above α, at some cost in reward (near zero on PointButton1); at α = 0.40 it violates on three of four navigation tasks and one Swimmer selection, and on Bullet's CarRun one seed in three violates (Sec. 6.2, App. G, I).