acceptodds
Under review as a conference paper at ICLR 2027

PSIL: Learning Safe Offline Policy from Preferred and Non-Preferred Demonstrations

Abstract

Offline imitation learning (IL) typically relies on expert (i.e., preferred) demonstrations and a large collection of supplementary trajectories for broader state-action coverage. However, existing approaches often struggle to effectively integrate the valuable signal from non-preferred demonstrations, which provide critical information about what the agent should avoid. We propose PSIL, Progressive Safe Imitation Learning, a self-training-based offline IL algorithm that can flexibly learn from preferred demonstrations, non-preferred demonstrations, or both. PSIL progressively propagates behavioral information from this small set of demonstrations to a large supplementary trajectory dataset by assigning preference-weights based on their similarity to preferred or non-preferred behaviors. These weights induce a relative ordering over trajectories and are used in weighted behavior cloning to emphasize preferred behavior while down-weighting non-preferred behavior. We evaluate PSIL in Learning from Preferred (LfP), Learning from Non-preferred (LfN), and Learning from Preferred and Non-preferred (LfP+N) settings on the DSRL benchmark. Across these varied settings, PSIL maintains a favorable balance between reward and safety trade-off. We support our empirical results with theoretical and empirical analysis of the trajectory-weighting mechanism.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Imitation Learning, Offline RL, Safe RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_uiGXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Progressive Safe Imitation Learning (PSIL), an offline imitation learning framework that leverages preferred demonstrations, non-preferred demonstrations, or both, together with a larger collection of unlabeled trajectories. PSIL trains a trajectory-level preference score function and progressively assigns decayed pseudo-weights to unlabeled trajectories based on their predicted similarity to preferred or non-preferred behaviors. The resulting weights are incorporated into an exponentially weighted behavior cloning objective.

The method is evaluated under three settings (LfP, LfN, and LfP+N) across ten DSRL benchmark tasks. The paper also provides a Lipschitz continuity analysis of the score function and studies the role of a regularization term in preventing degenerate constant-score solutions.

Critical strengths
  1. The paper addresses a meaningful problem in offline safe imitation learning: exploiting both desirable and undesirable demonstrations without requiring reward or cost annotations during policy training. The unified treatment of LfP, LfN, and LfP+N is practically appealing.

  2. PSIL combines trajectory-level preference propagation with weighted behavior cloning. The approach is relatively straightforward to implement and does not require explicit reward learning, cost modeling, or constrained policy optimization.

  3. The experiments cover ten environments from Safety-Gymnasium and Bullet-Gym, with comparisons against several established offline imitation learning and safe imitation learning methods. The paper also includes analyses of the regularization mechanism, score-function architecture, and several hyperparameters.

Critical weaknesses

W1. The empirical evaluation does not adequately isolate the benefit of progressive self-training.

The central algorithmic contribution is the iterative propagation of preference information from labeled demonstrations to unlabeled trajectories. However, the paper does not directly compare PSIL against a non-progressive alternative that trains the score function once using the original demonstrations and subsequently performs weighted behavior cloning.

Without this comparison, it is difficult to determine whether the reported improvements arise from progressive pseudo-label propagation or simply from using a learned trajectory preference score to reweight behavioral cloning.

Although the paper visualizes the progressive trajectory identification process, these results do not establish that iterative self-training is necessary for the final policy performance. More targeted ablations are needed to substantiate the main algorithmic contribution.

W2. Potential error propagation in progressive self-training raises concerns about policy safety.

A central concern with the proposed progressive self-training mechanism is that incorrectly assigned preference labels may propagate across subsequent iterations. According to Eq. (3), once a trajectory receives a nonzero preference weight, its assignment is never revised. Moreover, these pseudo-labeled trajectories are subsequently treated as supervision for the score function, potentially reinforcing incorrect predictions.

This issue is particularly important in the context of safe imitation learning. If an unsafe trajectory is incorrectly identified as preferred during an early iteration, it may not only influence subsequent pseudo-label assignments but also receive an increased weight in the behavioral cloning objective, potentially encouraging the learned policy to imitate unsafe behavior. Although Appendix E.2 acknowledges the possibility of trajectory misclassification, the paper does not investigate whether such errors accumulate during progressive self-training or how they affect the safety of the resulting policy.

Questions for authors

Q1: How much of PSIL's performance improvement is attributable specifically to progressive pseudo-label propagation? How does it compare with one-shot score-based weighted behavior cloning?

Q2. Given that pseudo-label assignments are irreversible in Eq. (3), how robust is PSIL to incorrect preference labels assigned during early iterations? In particular, can the authors clarify whether such errors accumulate or propagate through subsequent self-training iterations, and how they affect the safety of the learned policy?

2 comments
AuthorsOpenReview ↗

Response to the reviewer

Thank you for the review.

Q1: How much of PSIL's performance improvement is attributable specifically to progressive pseudo-label propagation? How does it compare with one-shot score-based weighted behavior cloning?

We have added our results in Appendix P & N. We kindly refer the reviewer to our common response titled:

  • Results on "DSRL threshold" and "Progressive Weight Assignment"
  • Results on "Policy Weighting Scheme"

Q2. Given that pseudo-label assignments are irreversible in Eq. (3), how robust is PSIL to incorrect preference labels assigned during early iterations? In particular, can the authors clarify whether such errors accumulate or propagate through subsequent self-training iterations, and how they affect the safety of the learned policy?

Since all pseudo-labels propagate from the initial demonstrations, we study this by injecting incorrect labels directly into them, which is the most severe case of an early error (added Appendix Q). In Noisy $\mathcal{D}_P$, the single preferred demonstration is replaced by a randomly sampled trajectory. In 20% Noisy $\mathcal{D}_N$, two of the ten non-preferred demonstrations are replaced by random trajectories. Following DSRL, each variation is evaluated at three cost thresholds κ\kappa (strict, medium, loose): {20,40,80} for Safety-Gymnasium and {10,20,40} for Bullet-Gym. Results in the LfP+N setting (normalized reward R and C/κC/\kappa at the DSRL thresholds; C/κ≤1C/\kappa \leq 1: satisfied; mean ± 95% CI over five seeds):

PointCircle2

DatasetRstrict (κ\kappa)medium (κ\kappa)loose (κ\kappa)
Noisy DP\mathcal{D}_P0.63±0.045.63±2.852.82±1.421.41±0.71
20% Noisy DN\mathcal{D}_N0.49±0.051.03±0.320.51±0.160.26±0.08
No noise0.48±0.050.56±0.310.28±0.160.14±0.08

DroneRun

DatasetRstrict (κ\kappa)medium (κ\kappa)loose (κ\kappa)
Noisy DP\mathcal{D}_P0.60±0.073.35±1.971.68±0.990.84±0.49
20% Noisy DN\mathcal{D}_N0.55±0.033.65±0.821.83±0.410.91±0.21
No noise0.54±0.023.63±1.091.82±0.550.91±0.27

Errors do propagate, and their effect depends on where the incorrect label lies.

Errors in $\mathcal{D}_P$ are the most severe. With a single preferred demonstration, every positive weight originates from it. On PointCircle2, a noisy DP\mathcal{D}_P leads the policy to imitate high-reward but unsafe behavior (strict C/κC/\kappa = 5.63 vs. 0.56). On DroneRun, results vary widely across seeds, depending on where the random trajectory falls.

Errors in $\mathcal{D}_N$ are tolerable. The correct non-preferred demonstrations still provide most of the negative supervision, and a random trajectory usually has intermediate reward and cost, so it only down-weights intermediate trajectories. Performance on DroneRun is nearly unchanged, and on PointCircle2 the medium and loose thresholds are still satisfied.

The same applies to errors made by the score function during propagation. In addition, the decay λk\lambda^k limits the influence of errors made in later rounds, since those trajectories receive smaller weights. However, when the score function cannot separate preferred from non-preferred trajectories, as in CarCircle2, early errors are not corrected and the policy violates every threshold (Appendix E, Limitation 1). Allowing previously assigned labels to be revised is an interesting direction for future work.

I thank the authors for their response, which addresses my questions and concerns. Overall, I will increase the score from 2 to 3.

Reviewer_gYqKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Progressive Safe Imitation Learning (PSIL), an offline imitation learning algorithm that learns from preferred demonstrations, non-preferred demonstrations, or both, together with an unlabeled trajectory dataset. PSIL progressively identifies unlabeled trajectories similar to preferred or non-preferred demonstrations using a regularized trajectory scoring model and assigns decayed preference weights through self-training. These weights are then incorporated into a weighted behavior cloning objective to encourage preferred behaviors while reducing the influence of non-preferred ones. Experiments on the DSRL benchmark demonstrate competitive reward–safety trade-offs across the LfP, LfN, and LfP+N settings.

Critical strengths

1. Unified framework for different supervision settings

A key strength of this paper is that PSIL can handle LfP, LfN, and LfP+N settings within a single framework. In particular, the method can utilize either positive or negative supervision without requiring both types of labeled demonstrations.

Critical weaknesses

1. Unclear safety-specific contribution and formulation

Although the paper introduces a CMDP formulation, safety is incorporated into the algorithm mainly through the construction of preferred and non-preferred demonstrations. The proposed objective does not explicitly consider the safety cost or constraint threshold. Furthermore, non-preferred demonstrations include both low-reward and high-cost trajectories, making it unclear how task performance and safety are distinguished during learning. The method therefore appears to be a general preference-guided offline IL approach. It is unclear what additional safety-specific problem the proposed formulation addresses. The primary evaluation using the F-beta score also does not directly measure whether the learned policy satisfies a safety constraint.

2. Insufficient comparison with closely related offline IL approaches

The proposed method appears to be applicable to general offline imitation learning with positive, negative, and unlabeled demonstrations, rather than being specific to safe imitation learning. From this perspective, comparisons with existing PNU-based offline IL methods are important. In particular, DualCOIL (Hoang et al., ICML 2026) considers a closely related setting with expert, undesirable, and unlabeled demonstrations, but is not discussed or compared. Moreover, the proposed framework shares similarities with discriminator-based data weighting and standard self-training approaches. Without comparisons with these methods, it is difficult to assess the contribution of the proposed progressive weighting procedure.

Questions for authors

1. Comparison with existing methods

Could the authors compare PSIL with DualCOIL (Hoang et al., ICML 2026), which addresses a closely related offline IL setting with positive, negative, and unlabeled demonstrations? In addition, classification-based weighting approaches, such as weighted BC using a classifier trained with existing PNU learning methods (e.g., Sakai et al., ICML 2017), seem relevant.

More importantly, a simple baseline that uses the same trajectory scoring model and weighted BC objective, but without progressive self-training, would help clarify the contribution of the proposed progressive weighting procedure.

2. Role of safety

What is the specific role of safety in the proposed algorithm beyond defining preferred and non-preferred demonstrations? In particular, how does the CMDP formulation and its cost threshold relate to the actual learning objective? Could the authors provide an evaluation of safety constraint satisfaction in addition to the F-beta score?

3. Progressive weighting

What motivates the use of hard pseudo-labels of ±1 for score learning while applying decayed weights for policy learning? Would directly using the learned scores or confidence-weighted pseudo-labels lead to similar performance?

4. Dataset construction

How are preferred and non-preferred demonstrations selected from DSRL? Are they randomly sampled from trajectories satisfying certain reward/cost criteria, and are the datasets resampled across random seeds? How robust is the performance to different selections of labeled demonstrations?

1 comment
AuthorsOpenReview ↗

Response to the reviewer

W/Q: The proposed objective does not explicitly consider the safety cost or constraint threshold. What is the specific role of safety in the proposed algorithm? In particular, how does the CMDP formulation and its cost threshold relate to the actual learning objective?

We thank the reviewer for this question. We agree with the reviewer's characterization: safety enters PSIL only through the preferred and non-preferred demonstrations, and the objective itself does not use costs or a constraint threshold. This is intentional. The CMDP formulation requires per-timestep cost information, which is often difficult to specify or obtain in practice. In contrast, examples of unsafe behavior (e.g., collisions, traffic violations, or user-reported toxic responses) are often available as a byproduct of deployment. Our goal is to study whether a policy can learn safe behavior from such demonstrations alone, without access to costs. This follows the setting of prior safe IL methods such as SafeDICE, UNIQ, SafeMIL and OSIL, which also encode safety only through labeled demonstrations.

The CMDP defines the preferred behavior π∗\pi^* that maximizes the reward subject to the cost constraint, and the demonstrations are labeled according to it. The cost threshold therefore enters only through which trajectories are labeled preferred. A consequence is that PSIL cannot target a different threshold as it learns a threshold-agnostic policy, which we will state as a limitation.

W: non-preferred demonstrations include both low-reward and high-cost trajectories, making it unclear how task performance and safety are distinguished during learning.

We agree that PSIL does not distinguish whether a trajectory is non-preferred because of low reward or high cost. However, this distinction is not needed to learn the preferred behavior. Under the CMDP objective, the preferred policy should avoid both low-reward behavior (fails the task), and high-cost behavior (violates the constraint). PSIL therefore down-weights both, and the policy is guided toward the remaining high-reward, low-cost behavior identified from DPD_P & DND_N.

W: The method therefore appears to be a general preference-guided offline IL approach.

We agree that PSIL is a general preference-guided offline IL method. We see this as a strength, since the same framework applies wherever preferred and non-preferred demonstrations are available. We focus on safety because non-preferred examples arise naturally there and imitating them is costly.

Q: safety constraint satisfaction

We now report constraint satisfaction at the original DSRL cost thresholds for all three settings (Appendix M). We kindly refer the reviewer to our common response titled: Results on "DSRL threshold" and "Progressive Weight Assignment"

Q: Why hard labels for score learning but decayed weights for policy learning?

They serve different purposes. The ±1\pm 1 labels drive propagation; each newly identified trajectory acts as a full-strength anchor that pulls its neighbors' scores above η\eta in the next round. The decayed weights ±λk\pm\lambda^k drive policy learning; they encode confidence, so trajectories identified earlier (closer to the demonstrations) have more influence.

Using equal weights for the policy hurts performance. The uniform scheme, which drops the decay magnitude, has about 4x higher strict cost on PointCircle2. See our common response titled Results on "Policy Weighting Scheme", uniform scheme.

Q: Would directly using the learned scores lead to similar performance?

No. There are two ways the learned scores can be used.

Using initial score function. This is trained only on D_P & D_N, so its scores are reliable only for trajectories close to them. For most of D_U, the scores are unreliable. In our experiments, this score-variant misses the strict threshold on PointCircle2 and every threshold on DroneRun. (see Appendix P)

Using Scores during propagation. Here the problem is the opposite; every identified trajectory during training ends up with a high score. Trajectories close to the demonstrations are identified first. Trajectories identified later are further away and often have higher cost. Therefore, using these scores as trajectory-weights would therefore treat a borderline trajectory almost the same as the safe high-return trajectory.

See our common response titled Results on "Policy Weighting Scheme" score scheme.

Comparison with existing methods: PNU, DualCoil

We have added the results in Appendix M.

Dataset construction

We thank the reviewer for pointing out. All experiments already resampled the datasets for each seed, but in the original submission Table 1 reported the dataset statistics for only one of these seeds. We have corrected the table to report the mean, std over all five seeds. No experiments were rerun; all results are unchanged. The full sampling procedure and updated statistics are in Appendix A. Changes are color coded.

Reviewer_E396OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PSIL, an offline imitation learning method that learns from a small set of preferred demonstrations and/or non-preferred demonstrations, together with a large unlabeled trajectory dataset. PSIL trains a trajectory score function to separate preferred from non-preferred demonstrations, and progressively assigns positive or negative weights to unlabeled trajectories that it scores as similar, with smaller weights for trajectories identified in later rounds. These weights are then used in a weighted behavior cloning objective. Empirically, on ten DSRL tasks in three settings, PSIL achieves a favorable reward-safety trade-off under a new F-beta score that combines normalized reward and normalized safety.

Critical strengths

PSIL handles preferred demonstrations and/or non-preferred demonstrations within a single simple procedure. This is practically useful because non-preferred trajectories can often be collected as a byproduct of deployment. In the experiments, the same procedure with the same hyperparameters is applied to all three settings on ten tasks, and it is competitive with the strong baselines in each setting.

Critical weaknesses
  1. The method lacks a clear technical connection between its trajectory-scoring procedure and the constrained policy learning objective. The proposed weights are based on similarity to demonstrations and the iteration at which each trajectory is identified, rather than being derived from the expected-cost constraint. The theoretical analysis does not explain why optimizing the resulting weighted BC objective should produce a policy satisfying that constraint. Furthermore, scores are assigned to trajectories or fixed-length sub-trajectories, whereas the policy is learned through per-state MSE minimization. When the conditional action distribution is multimodal, this objective can average distinct action modes and produce actions poorly supported by the demonstrations. Even when individual actions are supported by the data, combining behaviors from different trajectories may generate new rollouts whose cumulative costs are not captured by the original trajectory scores. It therefore remains unclear whether the proposed weighting scheme enables beneficial trajectory stitching while preserving safety. Consequently, the analysis provides limited justification for the central step from trajectory weighting to safe policy learning.

  2. The evaluation does not show whether PSIL satisfies the constraint defined in Section 3. The experiments do not give a cost threshold for each task or report how often it is met. Instead, the main results use the F-beta score introduced by the authors. This score combines normalized reward with a safety score that rescales cost linearly between the benchmark bounds (Eq. 6, Table 2). So it measures a reward-cost trade-off, not whether the constraint is met. For example, in the LfP+N setting, PSIL's cost on CarCircle2 is 205.36 (Table 5), which is above the unlabeled data average of 165.18 (Table 1). Yet it still receives substantial safety credit under this score. In the LfN setting, its costs on PointCircle2 (157.95) and CarCircle2 (236.77) are also above the unlabeled data averages (Tables 1 and 9). These comparisons do not prove that the constraint is violated, but they show that an explicit safety criterion is needed. A more direct test would report, for each task, the expected cost against a threshold, the fraction of seeds that meet it, and the reward among policies that meet it. Since the method itself does not link its weights to the constraint, the safety claim rests entirely on such a test.

Questions for authors
  1. The preferred dataset contains a single trajectory. How is it chosen, and how sensitive are the results to this choice or to using a few more preferred trajectories (PSIL vs baselines)?

  2. The score function evaluates 50-step sub-trajectories, and the resulting weights reflect the logged continuations rather than alternative continuations available from the same states. It is therefore unclear whether the resulting weighted BC encourages beneficial trajectory stitching, which value-based baselines such as IPL can do in principle. Moreover, each scoring window excludes outcomes beyond its 50-step horizon, although the constraint concerns cumulative cost over the whole episode. How sensitive are the results to the window length? Could the authors evaluate PSIL in an LfN setting where all complete unlabeled trajectories exceed a specified cost threshold, but their segments provide sufficient coverage to construct a safe policy? A controlled experiment of this kind, with comparisons against baselines, would help assess whether PSIL can recover safe behavior by combining segments from different trajectories.

  3. Appendix B describes the score network but not how a trajectory is turned into tokens. Is each token one state-action pair? How is the state-action pair mapped to the 64-dimensional embedding, and is a positional encoding used? Please add these implementation details for reproducibility.

  4. Could the authors isolate the benefit of progressive propagation by comparing against a score model trained only on the initial demonstrations and then frozen? More generally, freezing scores and weights after different numbers of propagation rounds, while keeping the policy-training budget fixed, would clarify whether additional rounds improve policy performance.

3 comments
AuthorsOpenReview ↗

Response to the reviewer

W: A more direct test would report, for each task, the expected cost against a threshold.

We have reported the results in the DSRL threshold format for all the LfP, LfN, LfP+N settings. We kindly refer the reviewer to our common response titled Results on "DSRL threshold" and "Progressive Weight Assignment" (Appendix M).

Q: Appendix B describes the score network but not how a trajectory is turned into tokens. Is each token one state-action pair? How is the state-action pair mapped to the 64-dimensional embedding, and is a positional encoding used? Please add these implementation details for reproducibility.

We thank the reviewer and we will add these details. Each token is one state-action pair, token is the concatenation [st;at][s_t; a_t], normalized using statistics of DU\mathcal{D}_U. A linear layer maps each token to a 64-dimensional embedding, and a positional embedding is added before the transformer blocks.

Q: Could the authors isolate the benefit of progressive propagation by comparing against a score model trained only on the initial demonstrations and then frozen? More generally, freezing scores and weights after different numbers of propagation rounds, while keeping the policy-training budget fixed, would clarify whether additional rounds improve policy performance.

We kindly refer the reviewer to our common response titled Results on "DSRL threshold" and "Progressive Weight Assignment" (also Appendix N). There we show that the safety improvement comes from propagating preference-weights to DU\mathcal{D}_U, not from the initial score function alone (K=1K=1), with the largest gain on PointCircle2 .

Q. The preferred dataset contains a single trajectory. How is it chosen and how sensitive are the results to this choice or to using a few more preferred trajectories (PSIL vs baselines)?

Selection: DPD_P contains a single high-reward, low-cost demonstration, defined as the highest-reward trajectory among the 5% lowest-cost trajectories. We have also added the selection procedure in the Appendix A for all the dataset. Changes are color coded.

Sensitivity to the choice: Because every positive weight in PSIL originates from DP, its choice matters, and we study this directly. To test the most adverse case, we replace the selected demonstration with a randomly sampled trajectory (Noisy DPD_P, Appendix Q). Following DSRL, each variation is evaluated at three cost thresholds (strict, medium, loose): {20,40,80} for Safety-Gymnasium and {10,20,40} for Bullet-Gym. Results are in the LfP+N setting: normalized reward R and C/κC/\kappa at the DSRL thresholds, where C/κ≤1C/\kappa \leq 1 means the constraint is satisfied. We report the mean ± 95% CI over five seeds.

TaskPreferred demoRstrict (κ\kappa)medium (κ\kappa)loose (κ\kappa)
PointCircle2Selected (ours)0.48 ± 0.050.56 ± 0.310.28 ± 0.160.14 ± 0.08
PointCircle2Random0.63 ± 0.045.63 ± 2.852.82 ± 1.421.41 ± 0.71
DroneRunSelected (ours)0.54 ± 0.023.63 ± 1.091.82 ± 0.550.91 ± 0.27
DroneRunRandom0.60 ± 0.073.35 ± 1.971.68 ± 0.990.84 ± 0.49

On PointCircle2, a random DPD_P makes PSIL imitate high-reward but unsafe behavior; on strict (\kappa) C/κC/\kappa rises from 0.56 to 5.63, and all three thresholds are violated. On DroneRun, the mean is similar, but the variance across seeds roughly doubles, depending on where the random trajectory falls. PSIL therefore relies on DPD_P being genuinely preferred. This is the minimal assumption of the LfP+N setting. We use a single preferred demonstration because it is a restrictive and practically relevant regime; preferred or expert-quality demonstrations are typically the scarcest. Adding more preferred demonstrations only provides additional sources of positive weight for propagation. We therefore expect this to help PSIL and to reduce its dependence on single demonstration. We are running the experiments for DP>1D_P > 1.

AuthorsOpenReview ↗

Question regarding trajectory stitching

We thank the reviewer for this question. We have added a discussion and a new experiment in Appendix S.

Stitching in PSIL: PSIL weights short sub-trajectories rather than whole episodes. Each episode is split into overlapping sub-trajectories of length H=50 with a stride of 1, and each sub-trajectory receives its own weight (Appendix C.1). Because of the discount γt\gamma^t in the weighted BC objective (Eq. 4), the importance of a state-action pair comes mainly from the sub-trajectories in which it appears early. Safe segments inside unsafe episodes therefore receive high importance, while segments that lead to unsafe behavior are down-weighted. Weighted BC then combines safe segments from many different episodes into one policy, so the default PSIL already performs a form of stitching. We provide a detailed explanation in Appendix S.

Value-based stitching: In the current proposed algorithm, we agree that the weight of a state-action pair depends only on what happens after it in its own sub-trajectory, even when the dataset contains a better path from a similar state. To address this, we study a value-based variant, PSIL (value), in Appendix S. It learns Q(s,a)Q(s,a) as a soft maximum over the weights of all sub-trajectories starting from (s,a)(s,a), using an extreme-value objective following XQL, and trains the policy with weighted BC using exp⁡(Q(s,a)/α)\exp(Q(s,a) / \alpha). The policy can then prefer actions that lead to good behavior anywhere in the dataset, not only in their own sub-trajectory. Results are in the LfP+N setting, with normalized reward R and C/κC/\kappa at the DSRL thresholds, where C/κ≤1C/\kappa \leq 1 means the constraint is satisfied.

Update on 18 Nov: Results on 5 seeds. PSIL (value) is currently evaluated on three seeds, and we are running the remaining two seeds. PSIL (traj) uses five seeds.

TaskMethodRstrict (κ\kappa)medium (κ\kappa)loose (κ\kappa)
PointCircle2PSIL (value)0.54 ± 0.010.26 ± 0.110.13 ± 0.050.06 ± 0.03
PointCircle2PSIL (traj) (default)0.48 ± 0.050.56 ± 0.310.28 ± 0.160.14 ± 0.08
DroneRunPSIL (value)0.59 ± 0.033.93 ± 1.621.97 ± 0.810.98 ± 0.41
DroneRunPSIL (traj) (default)0.54 ± 0.023.63 ± 1.091.82 ± 0.550.91 ± 0.27

PSIL (value) achieves slightly higher reward and lower cost on both tasks and satisfies the same thresholds as PSIL (traj). Value-based weighting therefore helps, but the gain is marginal. Because PSIL (value) requires training an additional value network, we keep trajectory weighting as the default. Neither variant passes value information across segments, and extending PSIL (value) with a Bellman-style target is an interesting direction for future work.

Window length We agree that each weight reflects only the next HH steps, while the constraint applies to the cost of the whole episode. We are running experiments with different window lengths HH and will share the results as soon as they are done.

AuthorsOpenReview ↗

Sensitivity to Number of Preferred Trajectories and Horizon Length

Q: Sensitivity to more preferred demonstrations

We vary ∣DP∣∈{1,3,5}| \mathcal{D}_P | \in \{1, 3, 5\} in the LfP+N setting, keeping ∣DN∣=10| \mathcal{D}_N | = 10 and ∣DU∣=500| \mathcal{D}_U | = 500. The additional demonstrations are chosen with the same criterion.

Task# preferred traj.Rstrict (κ\kappa)medium (κ\kappa)loose (κ\kappa)
PointCircle21 (ours)0.48 ± 0.050.56 ± 0.310.28 ± 0.160.14 ± 0.08
PointCircle230.45 ± 0.101.05 ± 0.910.52 ± 0.450.26 ± 0.23
PointCircle250.51 ± 0.021.21 ± 0.820.61 ± 0.410.30 ± 0.20
DroneRun1 (ours)0.54 ± 0.023.63 ± 1.091.82 ± 0.550.91 ± 0.27
DroneRun30.58 ± 0.011.11 ± 0.740.55 ± 0.370.28 ± 0.18
DroneRun50.60 ± 0.011.48 ± 0.840.74 ± 0.420.37 ± 0.21

On DroneRun, a few more preferred demonstrations help considerably. With ∣DP∣=3| \mathcal{D}_P | = 3, the strict C/κC/\kappa drops from 3.63 to 1.11, and PSIL now also satisfies the medium threshold. A single demonstration covers only a small part of the preferred behavior on this task, and more preferred demonstrations helps. On PointCircle2, a single demonstration is already enough. With 3 or 5 demonstrations PSIL narrowly misses the strict threshold, one likely reason is that the additional preferred trajectories have slightly higher cost. Overall, PSIL does not need many preferred demonstrations, and adding a few helps when one demonstration does not cover the preferred behavior well. We will add these results to the revised paper.

Q: Sensitivity to horizon length

PSIL assign preference-weights to sub-trajectories of length H, and we use H=50 in all experiments. We choose a longer horizon because it helps the score function group trajectories by behavior. A smaller segment from a safe episode can look very similar to one from an unsafe episode, so a short horizon may give high-cost and low-cost segments similar scores (Property 1). A longer horizon covers more of the behavior, which makes preferred and non-preferred trajectories easier to separate.

We vary H= {5, 20, 50} in the LfP+N setting with all other hyperparameters fixed.

TaskHRstrict (κ\kappa)medium (κ\kappa)loose (κ\kappa)
PointCircle250.52 ± 0.051.66 ± 0.820.83 ± 0.410.41 ± 0.21
PointCircle2200.49 ± 0.051.40 ± 0.570.70 ± 0.280.35 ± 0.14
PointCircle250 (ours)0.48 ± 0.050.56 ± 0.310.28 ± 0.160.14 ± 0.08
DroneRun50.57 ± 0.034.15 ± 1.572.07 ± 0.791.04 ± 0.39
DroneRun200.56 ± 0.033.04 ± 0.871.52 ± 0.430.76 ± 0.22
DroneRun50 (ours)0.54 ± 0.023.63 ± 1.091.82 ± 0.550.91 ± 0.27

On PointCircle2, the cost goes down as the horizon grows, and only H=50 meets the strict threshold. On DroneRun, H=5 misses even the loose threshold, while H= 20 and H=50 both meet it with similar cost. Short horizons therefore make it harder to separate safe and unsafe behavior. We will add these results to the revised paper.

Reviewer_Xz4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PSIL, an offline imitation learning method that learns from preferred demonstrations, non-preferred demonstrations, or both, together with unlabeled data. It trains a trajectory scoring model and progressively assigns positive or negative weights to unlabeled trajectories. These weights are then used for weighted behavior cloning.

Critical strengths
  1. PSIL handles preferred-only, non-preferred-only, and combined supervision within a single method.

  2. PSIL achieves the highest average reward–safety score in the combined setting across 10 DSRL tasks.

Critical weaknesses
  1. Several ablation and sensitivity figures report substantially different performance for the same default configuration. For example, on PointCircle2, Figure 3 reports an F0.5F_{0.5} score of approximately 0.53, consistent with Table 5, while Figures 9, 10, 13, and 22 show approximately 0.73 for the default PSIL configuration. Similar inconsistencies appear in Figure 6 and the DroneRun results in Figure 12. Different reward normalization may explain some differences, but not all of them. Moreover, in the DroneRun ablations that agree with the main tables, the regularizer's improvements have overlapping error bars. These inconsistencies make it difficult to determine whether the reported component improvements are reliable. The authors should clarify the experimental configurations and provide ablations consistent with the main results.

  2. The paper argues that progressively identifying and weighting preferred and non-preferred trajectories is important for learning effective policies. However, there is no direct comparison against simpler alternatives such as uniform weighting, binary weighting, or single-round identification. Furthermore, Figures 14–15 show that positive identification covers approximately 1% or less of the unlabeled data on PointGoal1 and CarGoal1, and remains below approximately 5% on several other tasks. The λ\lambda sensitivity experiment on DroneRun also shows similar performance for λ=0.1\lambda=0.1 and λ=0.99\lambda=0.99, suggesting that later-round weighting may contribute little. Although Lemma 2 rules out constant representations under the regularized objective, it does not guarantee meaningful identification of preferred trajectories. Therefore, it remains unclear whether the progressive mechanism provides benefits beyond simpler weighted behavior cloning.

  3. The proposed FβF_\beta metric combines normalized reward and cost but does not evaluate whether policies satisfy the actual safety constraints. Under the original DSRL benchmark's cost thresholds, PSIL violates safety limits on several tasks. In the non-preferred-only setting, it exceeds the middle cost thresholds on 8 of 10 tasks, compared with 7 for SafeDICE, 5 for OSIL, and 2 for UNIQ. This raises concerns about the claim that PSIL learns safe policies. Moreover, the combined-setting performance advantage is relatively small when compared with the original IPL baseline: approximately 0.03 in average FβF_\beta across different β\beta values. The paper also lacks comparisons with closely related methods such as PUBC and SafeMIL. Reporting constraint satisfaction and including stronger baseline comparisons would provide a clearer assessment of the practical improvements.

Questions for authors
  1. Why do the default PSIL results in several ablation figures differ substantially from the main experimental results? Were different settings or normalization methods used?

  2. Can the authors compare progressive weighting with uniform, binary, and single-round weighting to isolate its contribution?

  3. How much of the unlabeled dataset receives positive, negative, or zero weights during training, and how does this affect performance?

  4. Can the authors report safety constraint violations using the original DSRL benchmark thresholds?

  5. Why is the original IPL baseline not included in the combined-setting comparison, and how does PSIL differ from related methods such as PUBC and SafeMIL?

2 comments
AuthorsOpenReview ↗

Response to the reviewer questions

Can the authors compare progressive weighting with uniform, binary, and single-round weighting to isolate its contribution?

We have compared and added our results in Appendix P. We kindly refer the reviewer to our common response titled: Results on "Policy Weighting Scheme".

Can the authors report safety constraint violations using the original DSRL benchmark thresholds?

We have added these results in Appendix M. We kindly refer the reviewer to our common response titled: Results on "DSRL threshold" and "Progressive Weight Assignment".

Why do the default PSIL results in several ablation figures differ substantially from the main experimental results? Were different settings or normalization methods used?

We thank the reviewer for catching this discrepancy. The difference was caused by a normalization mismatch. In the main paper, all results are normalized using the task-specific min and max reward and cost values from the DSRL benchmark (Eq. 5-6, Table 2). However, the ablation figures for PointCircle2 in the Appendix were mistakenly normalized using the min and max values observed across the evaluated methods. The underlying experimental settings and the trained policies are the same, and only the normalization used for plotting differed. We have re-normalized all Appendix results (Fig 9, 10, 12, 13, 22) with the DSRL bounds, so they are now consistent with the main paper. Since the raw reward and cost values are unaffected, the conclusions of these ablations remain unchanged.

How much of the unlabeled dataset receives positive, negative, or zero weights during training, and how does this affect performance?

Figures 2(a), 14(a), and 15(a) (coverage plot) show, over training, the fraction of DU\mathcal{D}_U assigned positive weights (identified as preferred) and negative weights (identified as non-preferred). The remaining trajectories are zero weight. Figures 17–19 show the corresponding policy reward and cost over the same training steps (LfP+N).

How much of DU\mathcal{D}_U is weighted. Coverage rises quickly in the first rounds of propagation and then grows slowly. Negative coverage is typically higher than positive coverage, as similar preferred behavior is relatively less compared to the non-preferred behavior in DU\mathcal{D}_U. The remaining trajectories are at zero preference-weight.

How this affects performance. On most tasks, the policy incurs high cost early in training, when coverage is low and few trajectories are down-weighted. As coverage increases, cost drops and then remains relatively stable for the rest of training while reward is maintained (Figures 17–19). On AntRun, the early policy has both low reward and low cost because it has not yet learned the task. Positive coverage remains very low on this task, so the policy mostly imitates unidentified trajectories, which have high average cost in DU\mathcal{D}_U.. As a result, cost increases together with reward during training.

The trade-off between coverage and trajectory quality is controlled by η\eta. A lower η\eta increases coverage but admits more high-cost trajectories into the preferred set, which lowers policy performance (Appendix F.1, Figure 9).

Why is the original IPL baseline not included in the combined-setting comparison, and how does PSIL differ from related methods such as PUBC and SafeMIL?

We have already included IPL as a baseline in the LfP+N settings (Fig 3, and Appendix I). We have added PUBC as a baseline in LfP setting and the code for SafeMIL is not publicly available.

AuthorsOpenReview ↗

Response to the reviewer weakness

Weakness Raised:

the regularizer's improvements have overlapping error bars.

To give a clearer picture, we have added tables (See Appendix R) reporting normalized reward and C/κC/\kappa at the DSRL thresholds with and without the regularizer in all LfP+N, LfP and LfN settings. The regularizer has a large effect on PointCircle2. In LfP+N, it reduces the strict-threshold C/κC/\kappa from 5.91 to 0.56, and only the regularized variant satisfies any threshold. In LfP, it roughly halves the cost, and only the regularized variant meets the loose threshold. On DroneRun, and on PointCircle2 in LfN, cost differences are within the confidence intervals, although the regularized variant attains slightly higher mean reward and lower variance across seeds. The regularizer prevents score collapse in the LfP & LfN settings (Figs. 6, 11, 12; Lemma 2), substantially improves safety on PointCircle2, and has a limited effect on DroneRun.

sensitivity analysis on λ\lambda

We thank the reviewer for this question, which led us to revisit our analysis. Our original conclusion that PSIL is robust to λ\lambda (Appendix F.2) was not accurate. It was based on FβF\beta-scores, which normalize cost by Cmax⁡C_{\max}. Since Cmax⁡C_{\max} is much larger than the DSRL thresholds, FβF\beta compresses differences in cost, so the curves for different λ\lambda looked similar (see also Appendix N). We have kept the original analysis in Appendix F.2 so the reviewer can compare it with the updated one, and we will remove it in the final version.

In the updated analysis (Appendix O, Table 15), we evaluate constraint satisfaction at the DSRL thresholds. With λ=0.1\lambda = 0.1, weights decay to near zero after a few rounds, so later-identified trajectories are treated almost like unidentified ones. On PointCircle2, this lowers the reward (R=0.29R = 0.29), and only the loose threshold is satisfied. With λ=0.99\lambda = 0.99, every threshold is satisfied, and λ=0.99\lambda = 0.99 achieves higher reward (R=0.48R = 0.48 vs. 0.410.41). On DroneRun, all values of λ\lambda perform similarly because propagation saturates after a few rounds, consistent with Appendix N.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Results on "DSRL threshold" and "Progressive Weight Assignment"

Reviewer: @Xz4c, @E396, @gYqK

Safety constraint violations using the original DSRL benchmark thresholds?

We thank the reviewer for the suggestion. We now report constraint satisfaction at the original DSRL thresholds for all three settings (Appendix M). Following DSRL, each policy is evaluated at three cost thresholds κ\kappa (strict, medium, loose): {20,40,80} for Safety-Gymnasium and {10,20,40} for Bullet-Gym. We report the normalized cost C/κC / \kappa, where C/κ≤1C/\kappa \leq 1 means the constraint is satisfied.

We summarize the LfP+N results below; full results for all three settings are in Appendix M. A method succeeds on a task if it satisfies the constraint with non-trivial reward (C/κ≤1C/\kappa \le 1 and Rnorm>0.1R_{\text{norm}} > 0.1), which excludes trivially safe policies that do not perform the task. Safe reward is the normalized reward averaged over all ten tasks, with unsuccessful tasks counted as 0, so it is high only if a method is both safe and performs the task. Each entry reads #successes / safe reward; e.g., 1 / 0.06 means the method 'succeeds' on 1 of 10 tasks and its 'safe reward' is 0.06.

LfP+N (Appendix M.1, Table 11)

MethodStrict (\kappa)Medium (\kappa)Loose (\kappa)
SafeDICE1 / 0.063 / 0.165 / 0.32
UNIQ2 / 0.126 / 0.216 / 0.21
OSIL0 / 0.003 / 0.183 / 0.18
IPL1 / 0.032 / 0.073 / 0.10
DualCoil1 / 0.023 / 0.055 / 0.15
PNU2 / 0.165 / 0.317 / 0.42
PSIL4 / 0.245 / 0.288 / 0.44

In LfP+N, PSIL solves the most tasks at the strict and loose thresholds and has the highest safe reward at both. In LfP, PSIL is competitive with IPL, the strongest baseline (Table 12). In LfN, PSIL ties for the most successes at the strict threshold but lags at the medium and loose thresholds (Table 13), since without preferred demonstrations low-cost behavior is identified only after negative weights have propagated through most of DU\mathcal{D}_U (Appendix K). Overall, the threshold-based evaluation is consistent with our FβF\beta results: PSIL is competitive with the baselines and performs best when both preferred and non-preferred demonstrations are available.

We have also added the additional baselines: DualCoil, PNU and PUBC. SafeMIL code was not publicly available.



Reviewer: @Xz4c, @E396, @gYqK, @uiGX

Isolate the benefit of progressive propagation by comparing against a score model trained for different numbers of propagation rounds, while keeping the policy-training budget fixed.

We thank the reviewer for this suggestion. We have added Appendix N in whihch we freeze the score function (and hence the weights) after KK propagation rounds while keeping the policy-training budget fixed at 1M gradient steps. K=1K=1 corresponds to a score function trained only on the initial demonstrations DP,DN\mathcal{D}_P, \mathcal{D}_N and then frozen, i.e., no propagation. Following DSRL, each variation is evaluated at three cost thresholds κ\kappa (strict, medium, loose): {20,40,80} for Safety-Gymnasium and {10,20,40} for Bullet-Gym. Results in the LfP+N setting (normalized reward R and C/κC/\kappa at the DSRL thresholds; C/κ≤1C/\kappa \le 1: satisfied; mean ± 95% CI over five seeds):

PointCircle2

RoundsRstrict (\kappa)medium (\kappa)loose (\kappa)
K=1K=10.53±0.021.98±0.420.99±0.210.49±0.11
K=5K=50.51±0.061.53±0.560.76±0.280.38±0.14
Full (PSIL)0.48±0.050.56±0.310.28±0.160.14±0.08

DroneRun

RoundsRstrict (\kappa)medium (\kappa)loose (\kappa)
K=1K=10.56±0.024.03±1.212.01±0.601.01±0.30
K=5K=50.56±0.053.75±1.261.87±0.630.94±0.31
Full (PSIL)0.54±0.023.63±1.091.82±0.550.91±0.27

Additional rounds reduce cost with a small decrease in reward. On PointCircle2, strict C/κC/\kappa drops from 1.98 (K=1K=1) to 0.56 (full), and only full propagation satisfies all three thresholds. On DroneRun, propagation saturates early, and the loose threshold is satisfied from K=5K=5 onward. These results show that the safety improvement comes from propagating preference-weights to DUD_U, not just from the initial score function alone (K=1K=1), with the largest gain on PointCircle2. Results for K=20K=20 and the corresponding FβF\beta-scores are in Appendix N (Table 14, Figure 23); the FβF\beta curves largely overlap because FβF\beta normalizes cost by Cmax⁡C_{\max}, which is much larger than the DSRL thresholds.

AuthorsOpenReview ↗

Results on "Policy Weighting Scheme"

Reviewer: @Xz4c, @gYqK, @uiGX

Can the authors compare progressive weighting with uniform, binary, and single-round weighting to isolate its contribution?

Would directly using the learned scores or confidence-weighted pseudo-labels lead to similar performance?

However, the paper does not directly compare PSIL against a non-progressive alternative that trains the score function once using the original demonstrations and subsequently performs weighted behavior cloning. How does it compare with one-shot score-based weighted behavior cloning?

We thank the reviewers for this question. We have added Appendix P which compares trajectory weightings in the policy objective min⁡πEτ[ν(τ)∑tγt∣∣at−π(st)∣∣2]\min_\pi \mathbb{E}_\tau[\nu(\tau)\sum_t \gamma^t ||a_t - \pi(s_t)||^2] (LfP+N, same policy and 1M-step budget):

  • Single-round: $\nu(\tau) = \exp \left(w(\tau)/\alpha\right) $, score function trained once without propagation.
  • Score: $\nu(\tau) = \exp \left(s_\theta(\tau)/\alpha\right) $, i.e., one-shot score-based weighted BC. Score function trained once without propagation.
  • Binary: ν(τ)=1w(τ)>0\nu(\tau) = 1_{w(\tau)>0}, BC on identified preferred trajectories only.
  • Uniform: ν(τ)=exp⁡(sign(w(τ))/α)\nu(\tau) = \exp\left(\text{sign}(w(\tau))/\alpha\right), This keeps the sign of the w(τ)w(\tau) but discards its magnitude, so every trajectory identified as preferred (non-preferred) receives the same weight.
  • weight (PSIL default): with progressive and decayed weights.

Following DSRL, each variation is evaluated at three cost thresholds κ\kappa (strict, medium, loose): {20,40,80} for Safety-Gymnasium and {10,20,40} for Bullet-Gym. Normalized reward R and C/κC / \kappa at the DSRL thresholds (C/κ≤1C / \kappa \leq 1: satisfied; bootstrapped mean ± 95% CI over five seeds):

PointCircle2

WeightingRstrict (κ\kappa)medium (κ\kappa)loose (κ\kappa)
Single-round0.53±0.021.980.990.49
Score0.46±0.061.210.610.30
Binary0.00±0.308.474.242.12
Uniform0.51±0.062.181.090.55
weight (PSIL)0.48±0.050.560.280.14

DroneRun

WeightingRstrict (κ\kappa)medium (κ\kappa)loose (κ\kappa)
Single-round0.56±0.024.032.011.01
Score0.55±0.024.372.181.09
Binary0.52±0.032.671.340.67
Uniform0.58±0.014.052.021.01
weight (PSIL)0.54±0.023.631.820.91

The one-shot variants (single-round, score) learn weights only from the few demonstrations and incur higher cost; neither meets the strict threshold on PointCircle2 or any threshold on DroneRun. Binary discards most of the data and is unstable on PointCircle2 (near-zero reward), though it attains lower cost on DroneRun. Uniform ignores identification confidence and has about 4x higher strict cost on PointCircle2. PSIL is the only scheme that satisfies every threshold on PointCircle2, because weight propagation extends supervision to DU\mathcal{D}_U and the decayed magnitude λk\lambda^k gives higher weights to trajectories identified with higher confidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.