- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Progressive Safe Imitation Learning (PSIL), an offline imitation learning framework that leverages preferred demonstrations, non-preferred demonstrations, or both, together with a larger collection of unlabeled trajectories. PSIL trains a trajectory-level preference score function and progressively assigns decayed pseudo-weights to unlabeled trajectories based on their predicted similarity to preferred or non-preferred behaviors. The resulting weights are incorporated into an exponentially weighted behavior cloning objective.
The method is evaluated under three settings (LfP, LfN, and LfP+N) across ten DSRL benchmark tasks. The paper also provides a Lipschitz continuity analysis of the score function and studies the role of a regularization term in preventing degenerate constant-score solutions.
Critical strengths
-
The paper addresses a meaningful problem in offline safe imitation learning: exploiting both desirable and undesirable demonstrations without requiring reward or cost annotations during policy training. The unified treatment of LfP, LfN, and LfP+N is practically appealing.
-
PSIL combines trajectory-level preference propagation with weighted behavior cloning. The approach is relatively straightforward to implement and does not require explicit reward learning, cost modeling, or constrained policy optimization.
-
The experiments cover ten environments from Safety-Gymnasium and Bullet-Gym, with comparisons against several established offline imitation learning and safe imitation learning methods. The paper also includes analyses of the regularization mechanism, score-function architecture, and several hyperparameters.
Critical weaknesses
W1. The empirical evaluation does not adequately isolate the benefit of progressive self-training.
The central algorithmic contribution is the iterative propagation of preference information from labeled demonstrations to unlabeled trajectories. However, the paper does not directly compare PSIL against a non-progressive alternative that trains the score function once using the original demonstrations and subsequently performs weighted behavior cloning.
Without this comparison, it is difficult to determine whether the reported improvements arise from progressive pseudo-label propagation or simply from using a learned trajectory preference score to reweight behavioral cloning.
Although the paper visualizes the progressive trajectory identification process, these results do not establish that iterative self-training is necessary for the final policy performance. More targeted ablations are needed to substantiate the main algorithmic contribution.
W2. Potential error propagation in progressive self-training raises concerns about policy safety.
A central concern with the proposed progressive self-training mechanism is that incorrectly assigned preference labels may propagate across subsequent iterations. According to Eq. (3), once a trajectory receives a nonzero preference weight, its assignment is never revised. Moreover, these pseudo-labeled trajectories are subsequently treated as supervision for the score function, potentially reinforcing incorrect predictions.
This issue is particularly important in the context of safe imitation learning. If an unsafe trajectory is incorrectly identified as preferred during an early iteration, it may not only influence subsequent pseudo-label assignments but also receive an increased weight in the behavioral cloning objective, potentially encouraging the learned policy to imitate unsafe behavior. Although Appendix E.2 acknowledges the possibility of trajectory misclassification, the paper does not investigate whether such errors accumulate during progressive self-training or how they affect the safety of the resulting policy.
Questions for authors
Q1: How much of PSIL's performance improvement is attributable specifically to progressive pseudo-label propagation? How does it compare with one-shot score-based weighted behavior cloning?
Q2. Given that pseudo-label assignments are irreversible in Eq. (3), how robust is PSIL to incorrect preference labels assigned during early iterations? In particular, can the authors clarify whether such errors accumulate or propagate through subsequent self-training iterations, and how they affect the safety of the learned policy?
2 comments
I thank the authors for their response, which addresses my questions and concerns. Overall, I will increase the score from 2 to 3.
Response to the reviewer
Thank you for the review.
We have added our results in Appendix P & N. We kindly refer the reviewer to our common response titled:
Since all pseudo-labels propagate from the initial demonstrations, we study this by injecting incorrect labels directly into them, which is the most severe case of an early error (added Appendix Q). In
Noisy $\mathcal{D}_P$, the single preferred demonstration is replaced by a randomly sampled trajectory. In20% Noisy $\mathcal{D}_N$, two of the ten non-preferred demonstrations are replaced by random trajectories. Following DSRL, each variation is evaluated at three cost thresholds κ (strict, medium, loose): {20,40,80} for Safety-Gymnasium and {10,20,40} for Bullet-Gym. Results in the LfP+N setting (normalized reward R and C/κ at the DSRL thresholds; C/κ≤1: satisfied; mean ± 95% CI over five seeds):PointCircle2
DroneRun
Errors do propagate, and their effect depends on where the incorrect label lies.
Errors in $\mathcal{D}_P$ are the most severe.With a single preferred demonstration, every positive weight originates from it. On PointCircle2, a noisy DP leads the policy to imitate high-reward but unsafe behavior (strict C/κ = 5.63 vs. 0.56). On DroneRun, results vary widely across seeds, depending on where the random trajectory falls.Errors in $\mathcal{D}_N$ are tolerable.The correct non-preferred demonstrations still provide most of the negative supervision, and a random trajectory usually has intermediate reward and cost, so it only down-weights intermediate trajectories. Performance on DroneRun is nearly unchanged, and on PointCircle2 the medium and loose thresholds are still satisfied.The same applies to errors made by the score function during propagation. In addition, the decay λk limits the influence of errors made in later rounds, since those trajectories receive smaller weights. However, when the score function cannot separate preferred from non-preferred trajectories, as in CarCircle2, early errors are not corrected and the policy violates every threshold (Appendix E, Limitation 1). Allowing previously assigned labels to be revised is an interesting direction for future work.