- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether negative demonstrations (e.g., failed trajectories) can be used to identify optimal policies in offline reinforcement learning without expert demonstrations or explicit reward supervision. Within a finite Markov decision process framework with linear rewards, the authors establish some theorems. They characterize reward and policy identifiability through trajectory-feature null spaces, derive finite-sample policy-recovery guarantees and Fisher-information-based criteria for collecting informative failures.
Critical strengths
- The research question is interesting
Critical weaknesses
- Restrictive and insufficiently clarified theoretical assumptions
- Limited experimental realism and generalizability
- Limited comparison with existing methods
Questions for authors
Theoretical Novelty and Missing Assumptions
Definition 1 (Reward, policy, and avoidance identification) ... Theorem 1 (Bad labels do not identify what is best). ...
- Could the authors more clearly distinguish the assumptions from the conclusions of each theorem?
In particular, the identifiability results rely heavily on the assumed negative-rational exponential-family observation model, known reference distribution, and linear reward representation. To what extent do these theorems establish new properties of learning from negative demonstrations, rather than characterize direct consequences of the assumed data-generation mechanism?
-
Could the authors consolidate and explicitly state the assumptions required for each theoretical guarantee? The assumptions are distributed across the problem formulation, theorem-specific discussions, and appendices, making it difficult to assess their relative importance and practical restrictiveness.
-
Could the authors clarify which theoretical results represent novel contributions beyond established exponential-family identifiability, linear optimization, and Fisher-information analysis?
Bad-data, one trajectory is bad, a trajectory is labeled bad, bad label, ...
- Could the authors clarify how "bad" labels and negative trajectories are defined and obtained in practical settings? How would these different types of negative supervision affect reward and policy identifiability, particularly when the labeling mechanism is imperfect or unknown?
Experiments
These synthetic tests isolate mechanisms; they are not evidence about real failure logs.
- Limited experimental validation on realistic robotics and offline RL benchmarks. Could the authors extend the evaluation to established offline RL or robotic manipulation benchmarks? How would the proposed method perform in continuous state and action spaces?
The learner sees the feature map, the transition log and negative trajectories drawn under (1)
-
Could the authors provide more extensive evaluations under independently generated negative demonstrations, including unknown selection mechanisms, noisy failure labels, heterogeneous behavior policies, and misspecified reward features?
-
Gap between theoretical guarantees and practical offline policy recovery. The finite-sample policy-recovery analysis relies on several restrictive assumptions. However, they may be unknown or wrongly estimated in realistic environments. Could the authors investigate how simultaneous errors in reward estimation, reference-distribution estimation, feature representation, and transition dynamics affect policy recovery?
3 comments
Response to Reviewer qViW (2 of 3)
Q3
“Limited experimental validation on realistic robotics and offline RL benchmarks. Could the authors extend the evaluation to established offline RL or robotic manipulation benchmarks? How would the proposed method perform in continuous state and action spaces?”
Response. We have added D4RL locomotion and Adroit pen-cloned, with continuous actions and no known transition model supplied to the learner (Appendix F, “Continuous control from negative records”). IQL learns policies using rewards fitted from simulated return-selected negative records. Matched-feature logistic fitting and transition discrimination provide alternative reward estimators.
On Adroit, the reward-term feature basis gives a normalized score of [], compared with for matched-feature logistic fitting and for grouped discrimination. Smoother tilt features give . These results demonstrate that the approach can be implemented beyond tabular tasks, while also showing that representation matters and that tilt fitting is not consistently superior to alternative estimators.
The continuous-control experiments use favorable task-informed features and simulated selection. We report them as empirical extensions; the finite-MDP theory does not supply continuous-control guarantees. Natural-termination tests are described next.
Q4
“Could the authors provide more extensive evaluations under independently generated negative demonstrations, including unknown selection mechanisms, noisy failure labels, heterogeneous behavior policies, and misspecified reward features?”
Response. We have expanded the evaluation along each of these dimensions, while distinguishing independence in controlled experiments from natural failures in existing logs.
- Independent negatives: Appendix F, “Estimated references and combined model errors,” reports the separate-reference-log study and the joint-error experiment.
- Noisy and heterogeneous selection: we tested sign contamination, random contamination, mixtures of behavior policies, and source-dependent selection strengths, as summarized in Q2.
- Misspecified features: The factorial study in Appendix F removes the goal feature; this produces the largest average regret increase, as detailed in Q5.
- Natural failures: Appendix F and Table 6 report native termination records on Hopper/Walker2d medium-replay and Hopper medium-expert, with ten training seeds and 50 common evaluation starts per checkpoint.
In the natural-failure study, all ten failure-trained IQL arm–task comparisons had fewer falls than behavior cloning, with paired intervals excluding zero. Task-score rankings nevertheless vary across estimators and tasks (Q6). These results support avoidance and task-dependent control, rather than recovery of the underlying task reward.
The natural failures are extracted from fixed logs, not independently collected failure datasets; uncertainty is conditional on those logs. Separately collected real-world negative demonstrations remain untested. We state this explicitly rather than treating the natural-termination experiment as fulfilling that stronger requirement.
Response to Reviewer qViW (3 of 3)
Q5
“Could the authors investigate how simultaneous errors in reward estimation, reference-distribution estimation, feature representation, and transition dynamics affect policy recovery?”
Response. We have added a paired four-factor study on 50 fresh environments (Appendix F, “Estimated references and combined model errors”), varying reference estimation, selection misspecification, feature omission, and dynamics estimation jointly. Planning support is held fixed using the same 3,000-episode transition log; each setting uses 2,000 negatives and selection strength .
For selected ridge, the average increases in normalized global regret were:
| Error source | Mean increase [95% interval] |
|---|---|
| Omitted goal feature | 0.273 [0.255, 0.291] |
| Misspecified selection | 0.041 [0.026, 0.057] |
| Estimated reference | 0.018 [0.011, 0.025] |
| Estimated dynamics | 0.005 [0.002, 0.008] |
Neither prespecified interaction was resolved after Holm correction; this does not establish that the effects are generally additive. Most omitted-feature datasets passed the fit diagnostics despite high regret. The practical lesson is that representation error can dominate, and a good likelihood fit does not validate the inferred decision.
Corollary 1 separately bounds reward and dynamics errors under its stated assumptions. The same paragraph in Appendix F reports improved reference validation: with and 1,000 reference trajectories, held-out two-sample validation reduced empirical-normalizer regret from to on fresh environments. This identifies a useful procedural improvement without removing the need for a representative reference sample.
Q6
“Limited comparison with existing methods”
Response. We have added published-method comparisons and matched-information controls. Table 6 in Appendix F reports the natural-failure results reproduced below, using the same fixed logs; entries are D4RL normalized score / falls out of 500.
| Method | Hopper replay | Walker2d replay | Hopper medium-expert |
|---|---|---|---|
| Tilt–IQL | 43.5 / 11 | 68.0 / 34 | 93.8 / 132 |
| Label-only IQL | 50.9 / 117 | 65.1 / 23 | 106.8 / 42 |
| Episode-grouped discriminator–IQL | 76.9 / 15 | 55.6 / 41 | 89.7 / 199 |
| SafeDICE, stochastic | 41.0 / 454 | 31.1 / 408 | 48.7 / 491 |
| Behavior cloning | 26.0 / 454 | 16.7 / 472 | 54.9 / 499 |
These learners do not receive numerical task rewards. Tilt uses task-informed reward features; the discriminator uses raw transitions; label-only IQL uses a terminal-unit indicator. SafeDICE uses its native stochastic evaluation, while the other rows use deterministic evaluation. Consequently, the table compares the stated configurations, not methods under identical representation and evaluation choices.
The comparisons show useful failure-based control but no consistent advantage for tilt fitting over the alternative reward estimators. We also tested a modified UNIQ implementation, which has configuration and reproducibility limitations. We do not treat it as a clean evaluation of the unmodified published method.
Finally, Appendix F, “Information in selection frequencies,” isolates selection information with the same pipeline under a uniform logger: genuine counts give regret , versus after count permutation on 50 fresh environments. This addresses the concern that the original reward-informed logger could make behavior cloning sufficient, while keeping the conclusion specific to the tested selection model.
Thank you again for helping us clarify the contribution and strengthen the evaluation. We welcome any further questions.
Response to Reviewer qViW (1 of 3)
Thank you for your careful review. We address your concerns below and identify the corresponding changes in the revised paper. Manuscript revisions are highlighted in blue. Appendix F summarizes the experiments.
Response. Thank you for asking us to make the assumptions and contribution easier to assess. Table 1 now maps each main result to its assumptions and conclusion, with the appendix results covered separately in Appendix A.1.
Theorem 1 concerns correct suboptimality labels without a selection model. Theorem 2 assumes linear rewards and known-reference exponential selection; known selection strength is needed for reward scale. Theorem 3 uses population reward contrasts and known covered-policy occupancies. Theorem 4 additionally requires independent negatives, bounded features, compactness, uniform curvature, a true reward in the identifiable span, and known dynamics. Corollary 1 replaces known dynamics with estimates under explicit sampling assumptions.
We have also sharpened the novelty statement. Exponential-family identification and MLE analysis are established tools. Our central result characterizes when the reward ambiguity left by the specified negative channel still permits a common optimal covered policy. An invisible reward direction can be harmless if it changes competing policies' returns equally, or decisive if it can reverse their ordering. Theorem 3 gives the exact test over the population affine class. This separates ambiguity that requires new failure sources from uncertainty that more samples can reduce, and from transition coverage needed for execution. The result is conservative when bounded prior information further restricts the reward class.
The contribution is therefore a characterization under an explicit observation model, not an assumption-free guarantee for all bad data. We now make that scope explicit beside the results.
Response. We now distinguish suboptimality labels from selection records, because they carry different information. A correct label says that an observed choice is not optimal; by itself it need not rank the alternatives (Theorem 1). Under a specified selection mechanism, relative frequencies can additionally reveal reward contrasts after correcting for availability through the reference law (Theorem 2).
Appendix A gives a possible curation mechanism: one candidate is flagged from a reference batch with probability proportional to exp(−βR). The learner sees retained identities rather than numerical returns. A retained trajectory need not be globally suboptimal, and the finite-batch marginal generally differs from the limiting exponential tilt.
Appendix F, “Diagnosing misspecified selection,” presents the principal contamination and source-temperature results. We also tested sign errors, heterogeneous controllers, and alternative contamination models. For example, at temperature ratio 2 and 2,000 negatives, estimating the ratio reduced regret from 0.277 to 0.017 on 50 fresh environments; at ratio 2.7 with only 500 negatives, estimating it instead worsened regret by 0.127 relative to pooled fitting. Modeling heterogeneity can help, but is itself statistically demanding.
Unknown selection remains a substantive limitation. Fit diagnostics sometimes miss harmful contamination and sometimes reject harmless deviations. We therefore distinguish empirical robustness in these tested settings from reward or policy identification under an unrestricted unknown mechanism.