acceptodds
Under review as a conference paper at ICLR 2027

Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning

Abstract

Safe offline reinforcement learning assumes a cost function on every transition. We ask what remains possible when safety can be judged only by comparing short clips and occasionally asking whether an episode exceeded its budget. Certified safety curation answers with a filter-then-clone pipeline: a state-only value trained from segment comparisons scores whole trajectories, Learn-then-Test calibration certifies a selection threshold under a distribution-free bound on the unsafe fraction of the selection, and behavior cloning follows. What is certified is the training set, not the policy, whose cost we report rather than bound. Where no threshold attains the target, the procedure refuses. We are not aware of prior work certifying the composition of a training set for offline RL or imitation. Oracle controls justify the design: reweighting individual transitions fails even with an exact value, so the value selects whole trajectories. The policies satisfy the cost budget on twelve of fifteen DSRL tasks, matching a clone of the ground-truth safe subset, which needs a label on every trajectory. Retrained on the certified selection, the strongest full-label method becomes safe where no setting of its own cost target rescues it. Refusal is predictable: the certificate's probability has a closed form in the purity of the pool's top quantile, which the calibration sample estimates and through which the scorer enters.

Primary area:
reinforcement learning
Keywords:
safe offline reinforcement learning, data curation, conformal risk control, preference-based learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RhhNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Certified Safety Curation for safe offline RL without dense cost supervision. It learns a state-only safety value from segment preferences, selects trajectories based on aggregated safety scores, and uses Learn-then-Test calibration to certify the unsafe fraction of the selected dataset before behavior cloning. Experiments on DSRL benchmarks demonstrate competitive safety performance against existing safe offline RL methods.

Critical strengths
  1. The paper addresses a meaningful problem of safe offline RL under limited safety supervision and provides a statistical guarantee on the composition of selected training data.

  2. The experiments cover multiple benchmarks and include extensive comparisons and ablations, showing that trajectory-level filtering can be effective for learning safer policies.

Critical weaknesses
  1. The paper emphasizes the importance of trajectory-level safety, yet models safety using a state-only value function aggregated over time, potentially overlooking action-dependent or temporal safety information. More importantly, the preference pairs are constructed using ground-truth episodic costs to identify high- and low-cost trajectories (Appendix N), contradicting the motivating setting where such cost information is unavailable. The effectiveness of the method under genuinely cost-free preference sampling remains unclear.

  2. The calibrated method and its uncertified V-filter achieve the same safety coverage (12/15 tasks), while only 4/15 tasks receive certificates under the default setting. Moreover, a simple classifier trained on 200 binary safety labels achieves comparable results (11/15), and a label-splitting approach provides similar certification rates. These results suggest that most benefits come from trajectory filtering rather than the proposed preference-learning and certification pipeline. Furthermore, the certificate only guarantees dataset composition, not the safety of the learned policy, limiting its practical significance.

Questions for authors
  1. How does the method perform when preference pairs are sampled without relying on ground-truth costs? Have the authors considered state-action or sequence-based safety models to capture temporal dependencies?

  2. Given the comparable performance of simple label-based filtering and the low certification coverage, what are the concrete advantages of the proposed pipeline under matched supervision budgets?

1 comment

Thank you for the careful review.

Cost-free pairs. With pairs chosen uniformly, without cost (App. N), 12 of 15 DSRL tasks and all 5 Bullet tasks stay within budget. On DSRL certification falls from 0.161 to 0.089, the calibration draw used for the reported policies certifies 2 tasks rather than 5, and some rewards fall.

State-action and sequence models. Q(s,a) left four of five tasks unchanged and was less safe on Hopper, a result App. L calls suggestive. A sequence scorer would be covered by the guarantee, since any score fixed before the calibration draw is (Sec. 4.3), but we have not tested one. The state value, which does not condition on actions, still falls as violations approach on all nine analysis tasks (App. E).

Advantages at matched budgets. That trajectory filtering carries much of the safety is the paper's own submitted finding: its first contribution is that the readout scale, not the supervision, is the binding choice (Sec. 1). At matched labels, labels-only keeps 12 tasks to our 12, with no effort advantage for preferences, and a labels-only 100/100 split certifies about as often as ours (Sec. 6.4, App. M). The V-filter, the same score without the certificate, also keeps 12. What certification adds is a finite-sample guarantee or an explicit refusal; the refusal blocks reward-aware sub-selection on Hopper, where ungated it reaches mean cost 82 (Sec. 6.2).

Coverage and policy safety. Five of 15 tasks certify at α = 0.25 and eight of the nine analysis tasks at α = 0.40; one of the five, PointButton1, is a false certification, an event the guarantee allows with probability at most δ (Sec. 6.2). Elsewhere it refuses, as designed; Proposition 3 predicts the rate, so a deployer can estimate the labels a target rate needs (Sec. 6.3; the estimate runs high where certification is rare, App. J). As the abstract says, what is certified is the training set, not the policy. That is by design: offline, the data bounds what any method can achieve, and certifying it is a guarantee available before a policy is trained or run (Sec. 8). On the five DSRL tasks that certify, CDT trained on certified selections stays within budget in all 42 runs at α = 0.25, five of them on selections above α, at some cost in reward (near zero on PointButton1); at α = 0.40 it violates on three of four navigation tasks and one Swimmer selection, and on Bullet's CarRun one seed in three violates (Sec. 6.2, App. G, I).

Reviewer_VPPbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether useful training data can be selected without per-step cost labels, using comparisons of short clips and a small number of judgments about whether whole trajectories exceeded the budget. The method learns a state-only safety scorer from segment preferences, scores whole trajectories, and uses 200 randomly sampled trajectory labels to calibrate the selection threshold through Learn-then-Test. The guarantee controls false certification of the selected data’s unsafe fraction; policy safety is evaluated separately. When certification fails, the method returns an explicitly uncertified conservative selection. Behavior cloning then learns from the selected data. The authors provide theoretical analysis, controls comparing transition-level weighting with trajectory-level selection, and DSRL experiments. The resulting policies meet the average-cost budget on 12 of 15 tasks, matching cloning from the fully labeled safe subset. Under the default setting, four tasks receive a certificate.

Critical strengths
  1. I think placing certification at the data-selection stage is valuable. The method gives a finite-sample guarantee on the selected data’s unsafe fraction and explicitly controls the probability of issuing a certificate when the selection does not meet the target. When the evidence is insufficient, it refuses to certify. The paper also clearly explains the gap between data certification and policy safety. This makes the scope of the guarantee easier to assess and treats the selected dataset as something that can be audited separately and passed to different learners.

  2. The experiments explaining why whole-trajectory selection matters are quite thorough. The authors provide oracle values to the transition-weighting methods and show that these methods can still fail even with accurate values. Filtering ground-truth-safe segments leads to policies within budget on only 2 of 15 tasks, compared with 12 of 15 when filtering whole safe trajectories. These controls make me more convinced that the granularity of data use matters, and improving scoring accuracy alone may not solve the problem. The proposed method also reaches 12 of 15 using weaker supervision, which further supports the approach on these tasks.

Critical weaknesses
  1. I am concerned that constructing the preference data uses additional ground-truth cost information. In the appendix, preference pairs are sampled from trajectories in the highest and lowest quartiles of true episodic cost, then labeled using segment costs. The initial sampling step is not directly available in the intended setting, where one can only compare short clips and occasionally judge whether a whole trajectory exceeded its budget. It may also make the training pairs easier to distinguish, affecting the scorer and subsequent selection. This does not invalidate the statistical certificate once the scorer is fixed, but it does affect whether the experiments establish that the method works in the claimed weak-supervision setting.

  2. I am not fully convinced by the additional practical benefit of preference learning and certification within the pipeline. Only 4 of 15 tasks receive a certificate under the default setting, while the 12-of-15 policy safety result includes the uncertified fallback. That result therefore supports the overall filtering pipeline more directly than certification itself. A simple filter trained on 200 budget labels already reaches 11 of 15, and splitting those labels between training and calibration gives a comparable certification rate. Since the proposed method additionally requires 1,000 segment comparisons, the current evidence does not clearly establish a substantial practical advantage over the simpler label-based route. This affects my assessment of the overall contribution.

Questions for authors
  1. Preference pairs are sampled from the highest and lowest quartiles of true episodic cost. But in the intended setting, the cost of every trajectory is not known. How would this sampling step work? Do you have results with randomly sampled pairs? I would like to see how well the filtering works without using this additional ground-truth cost information.

  2. How is the annotation budget counted in the comparison with labels-only filtering? The proposed method needs 1,000 segment comparisons in addition to 200 budget labels, while splitting the labels-only budget between training and calibration already gives fairly similar results. I would like to understand what the extra preferences provide, and when this route would be more cost-effective than directly labeling whether trajectories exceeded the budget.

  3. I am still unclear about the error rate after receiving a certificate. The paper uses δ/Pr(certify) to bound the conditional error rate, but Pr(certify) itself needs to be estimated. Is the uncertainty in that estimate accounted for? Is the returned quantity a statistically valid upper bound or a diagnostic estimate? Especially where certification is rare, could estimation error make the reported bound too optimistic?

  4. After certification, keeping the highest-return half or reweighting by return changes the training distribution. Which data does the final certificate cover: the original selected set or the data actually used for training? If it only covers the former, what formal guarantee remains after reward-aware curation, and which safety results are mainly empirical?

1 comment

Thank you for a precise review.

Q1, pair sampling. App. N chooses pairs uniformly, without cost. 12 of 15 DSRL tasks stay within budget, with the same three failures, and so do all 5 Bullet tasks. Certification is less frequent: the DSRL rate falls from 0.161 to 0.089, and the calibration draw used for the reported policies certifies 2 tasks rather than the 5 it certifies with the cost-chosen contrast pairs. Some rewards fall (our clone's HalfCheetah, 2104 to 1959), and false certification stays below δ.

W2 and Q2, budget and what preferences add. Counted in steps watched, ties included, the 1000 pairs add 60,000 steps to the labels' 200,000 on 1000-step tasks, and at matched labels the paper finds parity: labels-only keeps 12 tasks at n = 200 (11 in the submission), failing on partly different tasks (Sec. 6.4, App. L). So on these benchmarks preferences have no effort advantage; whether people find comparisons easier needs human labelers, which we do not test (Sec. 7). What the comparisons change is that the scorer consumes none of the 200 labels, so all stay for the audit, though a 100/100 split certifies about as often (App. L, M). This is the paper's third contribution as submitted, a measured account of what the supervision buys (Sec. 1). Against full-label safe offline RL the pipeline needs far fewer labels, of a weaker kind: at budget-matched cost limits CDT, CPQ and COptiDICE read a cost on every transition and keep 5, 4 and 2 of 15 tasks within budget (7 for CDT at its safest target; where CDT is safe, its reward is above ours), against our 12 (Sec. 6.2).

What certification adds is the guarantee or an explicit refusal. The refusal also gates reward-aware curation: it blocks it on Hopper, where ungated it reaches mean cost 82 (Sec. 6.2). The 12 safe tasks include the explicit uncertified fallback, and fewer tasks certify than are safe because the certificate counts any over-budget trajectory as unsafe, while policy safety asks only for mean cost within budget (Sec. 6.2, 6.3).

Q3, error after a certificate. It is a diagnostic estimate, not a bound, and the revision now says so: δ over the estimated certification rate is reliable where certification is common and optimistic where it is rare (Sec. 6.3, App. J). For a valid bound, App. J adds an independent audit of 20 to 100 extra labels. Over 27 task-seed cells it holds in at least 90.5 percent of certified draws against a 90 percent target, and it is informative, below 1 in the median draw, on 0, 6 and 10 of those cells at 20, 50 and 100 audit labels, the cells where certification is common.

Q4, reward-aware curation. The certificate covers the selection, not the subset kept from it. Sec. 4.4 now states what carries over: a fraction f of a selection at most α unsafe is at most min(1, α/f) unsafe, so at α = 0.25 the top half is covered only at 0.50. The safety of that variant is therefore empirical, as the paper says. After the top-half rule, unsafe fractions on certified seeds are 0.07 to 0.30 on DSRL and up to 0.39 on one seed of the Bullet task CarRun, above α twice: on PointButton1, whose certified selection was already above α, and on that CarRun seed.

Reviewer_PhFuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of curating a safe training subset for offline safe RL when per-transition cost labels are unavailable and only weak supervision—segment-level preferences plus a small budget of binary over-budget trajectory labels—can be obtained. It proposes a pipeline that learns a state-only safety value from segment preferences, selects whole trajectories at the trajectory scale, and calibrates the selection threshold via Learn-then-Test against the binary labels, yielding a statistical certificate on the selected training set's composition and explicitly refusing to certify when the pool cannot be purified.

Critical strengths

[S1] The experimental coverage is relatively broad, and the conclusions are persuasive.

Critical weaknesses

[W1] Missing Motivation / Cost Model: The paper’s motivating premise is that safety costs cannot be obtained directly during rollout, but must instead be acquired after rollout at substantial expense (in time or effort). However, the paper does not discuss any such real-world setting. In the DSRL environments used in the experiments, most costs can in fact be obtained at rollout time, and are therefore not the kind of expensive-to-acquire costs that the paper claims to consider. This makes me concerned about whether the paper’s motivation is meaningful. For the motivation to be sufficiently well grounded, the authors must discuss what kinds of environments actually satisfy the high-cost cost-acquisition regime considered here. In addition, they should justify that, in such high-cost cost-acquisition settings, obtaining preferences is substantially easier than obtaining binary 0/1 labels.

[W2] Incomplete Theory: The paper’s theoretical results (Proposition 1 & 3) only establish that “the selected data subset is sufficiently clean”; they do not establish that “the policy trained on this clean subset is necessarily safe.” One might further expect to see how large a “sufficiently clean subset” must be selected in order to obtain a safe policy (a PAC-style suboptimality/violation bound), and a comparison of this with the sample efficiency of methods such as COptiDICE. If nn is large, it would necessarily require a larger NN, so at the sample-efficiency level the method may be much less efficient than other approaches.

Questions for authors

See Weaknesses

3 comments

Thank you for the clear review.

W1, motivation. The premise, as submitted, is not that costs are expensive but that a trustworthy cost function may not be possible to write down (Sec. 1); the revision adds an example, how closely a robot may pass a person, where one can still compare clips and occasionally judge whether an episode went over budget (Sec. 1); the benchmarks do not test that setting. Where a cost function does exist, as in DSRL, it supplies both of our signals, letting us compare against safe offline RL on its benchmarks: at budget-matched limits CDT, CPQ and COptiDICE read a cost on every transition and keep 5, 4 and 2 of 15 tasks within budget (7 for CDT at its safest target; where CDT is safe, its reward is above ours), while our filter, from 1000 comparisons and 200 episode-level labels, keeps 12, five of them certified (Sec. 6.2, App. H). We do not claim preferences are cheaper than binary labels: a classifier on the same 200 labels matches our filter (Sec. 1, 6.4). What the comparisons buy is a scorer that consumes none of those labels, so all remain for the audit, though a 100/100 split certifies about as often (App. L, M). Noisy comparisons are tested: with up to 30 percent of comparisons flipped, false certification stays at or below δ in a coarse 100-draw test, reaching it on PointGoal1, because validity rests on the 200 budget labels being correct; precision falls, by up to 0.38, while the guarantee holds (App. L).

W2, theory. The paper gives no offline PAC bound on the learned policy: Proposition 1 certifies the data, and the policy is measured. The data is the intended object: offline, it bounds what any method can achieve, and certifying it is a guarantee available before a policy is trained or run, on a selection that can be handed to any offline learner (Sec. 6.2, 8). Clean data alone does not make a safe policy: Swimmer's deployed selection is 14.5 percent unsafe, yet its clone exceeds budget (Sec. 6.3). The paper does not derive the selection size a safe policy needs; Sec. 7 notes a too-small selection can underfit. Empirically, on the five DSRL tasks that certify, CDT trained on certified selections stays within budget in all 42 runs at α = 0.25, and CPL on four of those five tasks; at α = 0.40 CDT is safe on 16 of 24 selections across the eight tasks that certify there, and on Bullet's CarRun one CDT seed in three violates (Sec. 6.2, App. G, I). Where online rollouts are possible, App. K bounds the deployed policy's probability of exceeding budget. On sample efficiency against COptiDICE, the paper compares the kind of supervision each needs, not data volume (App. F).

Thanks for the detailed response and clarifications. I appreciate the added motivating example and the extra experiments, but my main concerns remain—and looking at the other reviews, it seems we share similar reservations. Specifically, the preference-learning component still doesn't show a clear advantage over simpler binary-label baselines at the same budget, and having guarantees only on the data composition without formal bounds or sample-complexity analysis for the learned policy itself feels incomplete for safe offline RL. Given these points, I will keep my original score of 2.

Thank you for the follow-up. We address both points in a comment to all reviewers. Briefly:

  • Labels-only is our method with one part changed. The scorer is trained on the 200 budget labels instead of segment comparisons. The selection, the certificate and the learner trained on the selection are unchanged (Sec. 6.4, App. L). The tie shows that the certified curation works with either kind of supervision. Training the scorer on comparisons keeps all 200 labels for the audit.
  • The guarantee is on the training data by design (abstract, Sec. 7). Policy-level bounds are rare in safe offline RL and are tied to a specific learner and to coverage assumptions. A certificate on the data holds for any learner trained on it. The policy's cost is measured on every task, and where rollouts are allowed App. K bounds the deployed policy's probability of exceeding the budget.

Comments on the paper

2 comments

Thank you for careful reviews. The shared question was how cost enters the preference data, so we reran the study choosing pairs uniformly, without cost (new App. N; the hyperparameters move to App. O); labels still come from cost, standing in for a human labeler. The pipeline keeps 12 of 15 DSRL tasks and all 5 Bullet tasks within budget, as before, though on DSRL it certifies less often (mean rate 0.161 to 0.089).

In checking this, we found that on 14 of the 20 tasks the pairs had been drawn by a different cost-based rule than the hyperparameter appendix states. We redrew them under the stated rule and recomputed every result; the other six tasks keep their original pairs, with the departures App. O lists. The pipeline's safe-task counts are unchanged. Five DSRL tasks now certify instead of four; on one, PointButton1, the certified selection turned out 28.7 percent unsafe, an event the guarantee allows with probability at most δ (Sec. 6.2). We also now evaluate every comparison and ablation arm on each seed's own episodes, where some had reused one seed's. Labels-only at n = 200 moves from 11 to 12 tasks within budget, so preferences now tie it at matched budget, and CPL moves from 4 to 2. Revised text is in blue, and changed tables and figures say so in their captions.

Two concerns recur across the reviews and the discussion. We clarify both.

Labels-only is the same method with one part changed. Our method has three steps:

  1. A learned state-only scorer scores whole trajectories.
  2. Learn-then-Test on 200 held-out budget labels certifies the selection threshold, or refuses.
  3. A policy is trained on the selection: by behavior cloning in the main pipeline, or by any offline learner, such as CDT (Sec. 6.2).

Labels-only changes only what the scorer is trained on: the 200 budget labels instead of segment comparisons. Everything else is the same (Sec. 6.4, App. L). The tie at 12 of 15 therefore shows that the curation and its certificate work with either kind of supervision. That is consistent with the paper's finding that the scale at which data is selected matters more than the source of supervision (Sec. 5). The paper's contribution is this certified curation, not a particular scorer.

The two scorers differ in how they use the 200 labels:

  • A scorer trained on comparisons uses none of them, so all 200 stay held out for the audit.
  • A scorer trained on the labels uses them up, so certifying requires a split. A 100/100 split certifies at a similar rate (App. L).

Both ask the supervisor the same question: did the episode exceed its budget? The safe offline RL baselines need a cost on every transition (App. F).

The guarantee is on the data, and the policy is measured. As the abstract states, Proposition 1 certifies the training set, not the policy. To our knowledge, certifying the composition of training data has not been done before in offline RL.

Policy-level bounds are rare in safe offline RL. Where they exist, they are tied to one learning algorithm and to assumptions about data coverage. A certificate on the data holds before any policy is trained, and for any learner given that data. For example, CDT retrained on certified selections stays within budget in all 42 runs (Sec. 6.2). The policy's cost is measured on every task. Where rollouts are allowed, App. K bounds the deployed policy's probability of exceeding the budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.