acceptodds
Under review as a conference paper at ICLR 2027

What Can Bad Data Identify? Negative Demonstrations in Offline Decision Making

Abstract

Offline logs often contain crashes, violations, rejected outputs, and abandoned episodes even when expert demonstrations and dense rewards are unavailable. Can these "bad" trajectories reveal what an agent should do, rather than only what it should avoid? Within a finite-MDP, linear-reward framework, we separate three requirements for using failure frequencies to guide control: an observation model that connects latent return to the selection of failures, variation that reveals reward contrasts between competing policies, and transition coverage for the policy ultimately selected. Labels without such an observation model generally identify only actions or trajectories to avoid. Under a negative-rational exponential-family selection model, we characterize reward ambiguity exactly by the null space of centered trajectory features and give an exact test for whether one policy is optimal throughout an affine reward-ambiguity set. The same geometry determines when failures add local Fisher information to an existing log, yields a finite-sample policy-recovery bound under exact-model assumptions, and supports a locally E-optimal design for collecting complementary failure sources. Under matched exponential channels, neither positive nor negative demonstrations are intrinsically more informative: the channel with greater feature variance along the contrasts needed for control has greater local Fisher information, with an exponential advantage possible in either direction. Extensions to negative-unlabeled mixtures and occupancy repulsion expose the additional assumptions behind common uses of bad data. Controlled synthetic tabular experiments isolate the predicted rank, alignment, and coverage effects and demonstrate offline hazard control using failures as the only reward supervision in that setting.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, inverse reinforcement learning, negative demonstrations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qViWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether negative demonstrations (e.g., failed trajectories) can be used to identify optimal policies in offline reinforcement learning without expert demonstrations or explicit reward supervision. Within a finite Markov decision process framework with linear rewards, the authors establish some theorems. They characterize reward and policy identifiability through trajectory-feature null spaces, derive finite-sample policy-recovery guarantees and Fisher-information-based criteria for collecting informative failures.

Critical strengths
  • The research question is interesting
Critical weaknesses
  • Restrictive and insufficiently clarified theoretical assumptions
  • Limited experimental realism and generalizability
  • Limited comparison with existing methods
Questions for authors

Theoretical Novelty and Missing Assumptions

Definition 1 (Reward, policy, and avoidance identification) ... Theorem 1 (Bad labels do not identify what is best). ...

  1. Could the authors more clearly distinguish the assumptions from the conclusions of each theorem?

In particular, the identifiability results rely heavily on the assumed negative-rational exponential-family observation model, known reference distribution, and linear reward representation. To what extent do these theorems establish new properties of learning from negative demonstrations, rather than characterize direct consequences of the assumed data-generation mechanism?

  1. Could the authors consolidate and explicitly state the assumptions required for each theoretical guarantee? The assumptions are distributed across the problem formulation, theorem-specific discussions, and appendices, making it difficult to assess their relative importance and practical restrictiveness.

  2. Could the authors clarify which theoretical results represent novel contributions beyond established exponential-family identifiability, linear optimization, and Fisher-information analysis?

Bad-data, one trajectory is bad, a trajectory is labeled bad, bad label, ...

  1. Could the authors clarify how "bad" labels and negative trajectories are defined and obtained in practical settings? How would these different types of negative supervision affect reward and policy identifiability, particularly when the labeling mechanism is imperfect or unknown?

Experiments

These synthetic tests isolate mechanisms; they are not evidence about real failure logs.

  1. Limited experimental validation on realistic robotics and offline RL benchmarks. Could the authors extend the evaluation to established offline RL or robotic manipulation benchmarks? How would the proposed method perform in continuous state and action spaces?

The learner sees the feature map, the transition log and negative trajectories drawn under (1)

  1. Could the authors provide more extensive evaluations under independently generated negative demonstrations, including unknown selection mechanisms, noisy failure labels, heterogeneous behavior policies, and misspecified reward features?

  2. Gap between theoretical guarantees and practical offline policy recovery. The finite-sample policy-recovery analysis relies on several restrictive assumptions. However, they may be unknown or wrongly estimated in realistic environments. Could the authors investigate how simultaneous errors in reward estimation, reference-distribution estimation, feature representation, and transition dynamics affect policy recovery?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qViW (1 of 3)

Thank you for your careful review. We address your concerns below and identify the corresponding changes in the revised paper. Manuscript revisions are highlighted in blue. Appendix F summarizes the experiments.

Q1

“Could the authors more clearly distinguish the assumptions from the conclusions of each theorem?”

“Could the authors consolidate and explicitly state the assumptions required for each theoretical guarantee?”

“Could the authors clarify which theoretical results represent novel contributions beyond established exponential-family identifiability, linear optimization, and Fisher-information analysis?”

Response. Thank you for asking us to make the assumptions and contribution easier to assess. Table 1 now maps each main result to its assumptions and conclusion, with the appendix results covered separately in Appendix A.1.

Theorem 1 concerns correct suboptimality labels without a selection model. Theorem 2 assumes linear rewards and known-reference exponential selection; known selection strength is needed for reward scale. Theorem 3 uses population reward contrasts and known covered-policy occupancies. Theorem 4 additionally requires independent negatives, bounded features, compactness, uniform curvature, a true reward in the identifiable span, and known dynamics. Corollary 1 replaces known dynamics with estimates under explicit sampling assumptions.

We have also sharpened the novelty statement. Exponential-family identification and MLE analysis are established tools. Our central result characterizes when the reward ambiguity left by the specified negative channel still permits a common optimal covered policy. An invisible reward direction can be harmless if it changes competing policies' returns equally, or decisive if it can reverse their ordering. Theorem 3 gives the exact test over the population affine class. This separates ambiguity that requires new failure sources from uncertainty that more samples can reduce, and from transition coverage needed for execution. The result is conservative when bounded prior information further restricts the reward class.

The contribution is therefore a characterization under an explicit observation model, not an assumption-free guarantee for all bad data. We now make that scope explicit beside the results.

Q2

“Could the authors clarify how "bad" labels and negative trajectories are defined and obtained in practical settings? How would these different types of negative supervision affect reward and policy identifiability, particularly when the labeling mechanism is imperfect or unknown?”

Response. We now distinguish suboptimality labels from selection records, because they carry different information. A correct label says that an observed choice is not optimal; by itself it need not rank the alternatives (Theorem 1). Under a specified selection mechanism, relative frequencies can additionally reveal reward contrasts after correcting for availability through the reference law (Theorem 2).

Appendix A gives a possible curation mechanism: one candidate is flagged from a reference batch with probability proportional to exp⁡(−βR)\exp(-\beta R). The learner sees retained identities rather than numerical returns. A retained trajectory need not be globally suboptimal, and the finite-batch marginal generally differs from the limiting exponential tilt.

Appendix F, “Diagnosing misspecified selection,” presents the principal contamination and source-temperature results. We also tested sign errors, heterogeneous controllers, and alternative contamination models. For example, at temperature ratio 2 and 2,000 negatives, estimating the ratio reduced regret from 0.2770.277 to 0.0170.017 on 50 fresh environments; at ratio 2.7 with only 500 negatives, estimating it instead worsened regret by 0.1270.127 relative to pooled fitting. Modeling heterogeneity can help, but is itself statistically demanding.

Unknown selection remains a substantive limitation. Fit diagnostics sometimes miss harmful contamination and sometimes reject harmless deviations. We therefore distinguish empirical robustness in these tested settings from reward or policy identification under an unrestricted unknown mechanism.

AuthorsOpenReview ↗

Response to Reviewer qViW (2 of 3)

Q3

“Limited experimental validation on realistic robotics and offline RL benchmarks. Could the authors extend the evaluation to established offline RL or robotic manipulation benchmarks? How would the proposed method perform in continuous state and action spaces?”

Response. We have added D4RL locomotion and Adroit pen-cloned, with continuous actions and no known transition model supplied to the learner (Appendix F, “Continuous control from negative records”). IQL learns policies using rewards fitted from simulated return-selected negative records. Matched-feature logistic fitting and transition discrimination provide alternative reward estimators.

On Adroit, the reward-term feature basis gives a normalized score of 72.772.7 [67.9,76.767.9,76.7], compared with 72.372.3 for matched-feature logistic fitting and 73.873.8 for grouped discrimination. Smoother tilt features give 67.867.8. These results demonstrate that the approach can be implemented beyond tabular tasks, while also showing that representation matters and that tilt fitting is not consistently superior to alternative estimators.

The continuous-control experiments use favorable task-informed features and simulated selection. We report them as empirical extensions; the finite-MDP theory does not supply continuous-control guarantees. Natural-termination tests are described next.

Q4

“Could the authors provide more extensive evaluations under independently generated negative demonstrations, including unknown selection mechanisms, noisy failure labels, heterogeneous behavior policies, and misspecified reward features?”

Response. We have expanded the evaluation along each of these dimensions, while distinguishing independence in controlled experiments from natural failures in existing logs.

  • Independent negatives: Appendix F, “Estimated references and combined model errors,” reports the separate-reference-log study and the joint-error experiment.
  • Noisy and heterogeneous selection: we tested sign contamination, random contamination, mixtures of behavior policies, and source-dependent selection strengths, as summarized in Q2.
  • Misspecified features: The factorial study in Appendix F removes the goal feature; this produces the largest average regret increase, as detailed in Q5.
  • Natural failures: Appendix F and Table 6 report native termination records on Hopper/Walker2d medium-replay and Hopper medium-expert, with ten training seeds and 50 common evaluation starts per checkpoint.

In the natural-failure study, all ten failure-trained IQL arm–task comparisons had fewer falls than behavior cloning, with paired intervals excluding zero. Task-score rankings nevertheless vary across estimators and tasks (Q6). These results support avoidance and task-dependent control, rather than recovery of the underlying task reward.

The natural failures are extracted from fixed logs, not independently collected failure datasets; uncertainty is conditional on those logs. Separately collected real-world negative demonstrations remain untested. We state this explicitly rather than treating the natural-termination experiment as fulfilling that stronger requirement.

AuthorsOpenReview ↗

Response to Reviewer qViW (3 of 3)

Q5

“Could the authors investigate how simultaneous errors in reward estimation, reference-distribution estimation, feature representation, and transition dynamics affect policy recovery?”

Response. We have added a paired four-factor study on 50 fresh environments (Appendix F, “Estimated references and combined model errors”), varying reference estimation, selection misspecification, feature omission, and dynamics estimation jointly. Planning support is held fixed using the same 3,000-episode transition log; each setting uses 2,000 negatives and selection strength β=4\beta=4.

For selected ridge, the average increases in normalized global regret were:

Error sourceMean increase [95% interval]
Omitted goal feature0.273 [0.255, 0.291]
Misspecified selection0.041 [0.026, 0.057]
Estimated reference0.018 [0.011, 0.025]
Estimated dynamics0.005 [0.002, 0.008]

Neither prespecified interaction was resolved after Holm correction; this does not establish that the effects are generally additive. Most omitted-feature datasets passed the fit diagnostics despite high regret. The practical lesson is that representation error can dominate, and a good likelihood fit does not validate the inferred decision.

Corollary 1 separately bounds reward and dynamics errors under its stated assumptions. The same paragraph in Appendix F reports improved reference validation: with β=4\beta=4 and 1,000 reference trajectories, held-out two-sample validation reduced empirical-normalizer regret from 0.2760.276 to 0.0330.033 on fresh environments. This identifies a useful procedural improvement without removing the need for a representative reference sample.

Q6

“Limited comparison with existing methods”

Response. We have added published-method comparisons and matched-information controls. Table 6 in Appendix F reports the natural-failure results reproduced below, using the same fixed logs; entries are D4RL normalized score / falls out of 500.

MethodHopper replayWalker2d replayHopper medium-expert
Tilt–IQL43.5 / 1168.0 / 3493.8 / 132
Label-only IQL50.9 / 11765.1 / 23106.8 / 42
Episode-grouped discriminator–IQL76.9 / 1555.6 / 4189.7 / 199
SafeDICE, stochastic41.0 / 45431.1 / 40848.7 / 491
Behavior cloning26.0 / 45416.7 / 47254.9 / 499

These learners do not receive numerical task rewards. Tilt uses task-informed reward features; the discriminator uses raw transitions; label-only IQL uses a terminal-unit indicator. SafeDICE uses its native stochastic evaluation, while the other rows use deterministic evaluation. Consequently, the table compares the stated configurations, not methods under identical representation and evaluation choices.

The comparisons show useful failure-based control but no consistent advantage for tilt fitting over the alternative reward estimators. We also tested a modified UNIQ implementation, which has configuration and reproducibility limitations. We do not treat it as a clean evaluation of the unmodified published method.

Finally, Appendix F, “Information in selection frequencies,” isolates selection information with the same pipeline under a uniform logger: genuine counts give regret 0.0190.019, versus 0.4810.481 after count permutation on 50 fresh environments. This addresses the concern that the original reward-informed logger could make behavior cloning sufficient, while keeping the conclusion specific to the tested selection model.

Thank you again for helping us clarify the contribution and strengthen the evaluation. We welcome any further questions.

Reviewer_NhnnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Authors consider offline RL in a setting where the dataset is selected to contain "negative trajectories".First, they show that when rewards are not observed, and only suboptimaltity is observed, learning is impossible in general. They they show that when the negative dataset is formed by retaining trajectories with a negative-reward-based Boltzmann distribution, then (i) the reward is identifiable up to directions orthogonal to the difference of any two trajectories (ii) the best covered policy can be recovered (under known dynamics, considering only reward estimation). They include empirical results in toy MDPs.

Critical strengths

None

Critical weaknesses

The main setting of "negative rational" selection rules is not standard in RL. Indeed, it is unclear why a rule that preferentially selects the worst outcomes would ever be implemented. The assumption also rules out adaptive logging procedures.

None of the theorems are surprising or interesting. Theorem 1 is obvious, Theorems 2 and 3 reflect basic facts about linear geometry/exponential families, and Theorem 4 is (for the most part) a standard guarantee for maximum likelihood.

The writing quality is poor. Terms like "accepted reward invariances","label-only supervision" are used without proper formal definitions.

Dynamics are assumed to be known in Theorem 4, so the error bound is missing the statistically most difficult part of offline learning. Assuming dynamics to be known is not justified in the paper.

There is existing literature on learning from poor coverage and low-quality data has been extensively addressed in [1] and its successors.

[1] Is Pessimism Provably Efficient for Offline RL? Jin et al 2021.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer Nhnn (1 of 2)

Thank you for your careful review. We address your concerns below and identify the corresponding changes in the revised paper. Manuscript revisions are highlighted in blue. Appendix F summarizes the experiments.

Q1

“The main setting of "negative rational" selection rules is not standard in RL. Indeed, it is unclear why a rule that preferentially selects the worst outcomes would ever be implemented. The assumption also rules out adaptive logging procedures.”

Response. Thank you for raising this concern. We have clarified that negative selection models which outcomes are retained for review, rather than a controller deliberately choosing poor actions. For example, a curator can inspect a candidate pool and preferentially flag low-return outcomes. The learner observes the retained trajectories, without observing numerical returns or the comparisons used by the curator.

Appendix A formalizes one such mechanism: a curator flags one of kk reference candidates with probability proportional to exp⁡(−βR)\exp(-\beta R). Its retained marginal approaches the exponential tilt as kk grows; finite-batch rejection generally differs from that limit. Thus, the construction gives a concrete interpretation of the assumption, while the guarantees remain conditional on the specified observation model. We do not assume that arbitrary crashes or terminations satisfy it.

Adaptive collection is compatible with a narrower protocol: first collect a candidate pool adaptively, then draw fresh negative selections conditional on that recorded pool. We evaluated five rounds of adaptive collection followed by fresh curation. Relative to a non-adaptive pool, the difference in regret beyond the true-reward planning reference was 0.0040.004 [−0.006,0.015-0.006,0.015] at 500 failures and 0.0000.000 [−0.003,0.002-0.003,0.002] at 2,000. This experiment used additional interim selections and does not justify pooling arbitrary adaptively selected failures. We now state that distinction explicitly.

Q2

“None of the theorems are surprising or interesting. Theorem 1 is obvious, Theorems 2 and 3 reflect basic facts about linear geometry/exponential families, and Theorem 4 is (for the most part) a standard guarantee for maximum likelihood.”

Response. Thank you for asking us to sharpen the contribution. The revised paper centers on when incomplete reward information is nevertheless sufficient for a covered decision. Theorem 3 answers this question exactly for the population affine reward class induced by the negative channel.

The distinction is useful even when the underlying geometry is elementary. If an unobserved reward direction changes two competing policies' returns equally, their ordering can still be determined. If that direction changes their return difference, the same negative-data distribution can support opposite decisions. Theorem 3 identifies precisely which case holds: every candidate–competitor feature contrast must be orthogonal to the invisible reward directions, and the identified component must favor the candidate. This is necessary and sufficient over the full affine class; bounded prior information can certify additional policies.

The resulting framework distinguishes three remedies: a better observation model for ambiguous label semantics, complementary failure sources for missing reward directions, and transition data for unsupported policies. Collecting more failures from the same source cannot resolve all three. We now present Theorem 1 as an impossibility construction and Theorems 2 and 4 as applications of established exponential-family and MLE tools; our claim is the decision-focused characterization and its collection implications.

Appendix F, “Information in selection frequencies,” isolates whether failure counts contain useful decision information. Under reward-independent uniform logging, genuine counts give mean regret 0.0190.019, compared with 0.4810.481 after permuting those counts in the same pipeline (paired difference −0.462-0.462 [−0.553,−0.373-0.553,-0.373], 50 fresh environments). This supports the value of the selection information under the model, without claiming a universal algorithmic advantage.

AuthorsOpenReview ↗

Response to Reviewer Nhnn (2 of 2)

Q3

“The writing quality is poor. Terms like "accepted reward invariances","label-only supervision" are used without proper formal definitions.”

Response. Thank you for pointing out these missing definitions. Section 3 now defines both terms before using them, and Appendix A.1 provides a glossary.

  • Accepted reward invariances: a subspace G\mathcal G of reward-weight changes declared task-irrelevant before observing data. Identification up to G\mathcal G means identifying the equivalence class θ+G\theta+\mathcal G; the default is G = {0}.
  • Label-only supervision: a set L\mathcal L of choices designated suboptimal, without frequencies, scores, rankings, or comparisons. It determines the version set Θ(L) = {θ : no element of L is optimal under θ}. A quantity is identified when it is constant over that set.

Table 1 also separates the assumptions and conclusions of the main results, so that label-only impossibility is not confused with identification under a specified selection mechanism.

Q4

“Dynamics are assumed to be known in Theorem 4, so the error bound is missing the statistically most difficult part of offline learning. Assuming dynamics to be known is not justified in the paper.”

Response. We agree that a known-dynamics analysis does not capture the full difficulty of offline learning. Theorem 4 isolates reward-estimation error; Corollary 1 now adds transition-estimation error, and a separate proposition addresses pessimistic planning under partial coverage.

Corollary 1 (Section 4.3; proof in Appendix B) uses a transition log independent of the negatives and explicit per-pair sampling assumptions. It bounds regret through reward error and transition error, showing where each data source enters. The all-pairs coverage requirement is strong: in our transition-log sweep, some covered pairs have only 1–3 visits and the resulting numerical bound is vacuous. We therefore do not present it as a practical certificate.

For partial coverage, Proposition 8 in Appendix A.6 combines a valid reward confidence set with transition confidence radii. The regret bound depends on their uncertainty along a comparator policy's occupancy, following the pessimism framework discussed below. This permits poor coverage away from that comparator while retaining the need for justified uncertainty sets.

Appendix F, “Estimated references and combined model errors,” reports the joint reference, selection, feature, and dynamics study. Its opening evaluation also reports continuous control without a known transition model supplied to the learner. We also varied the negative-sample and transition-log sizes independently. These experiments expose estimation effects rather than assuming them away; the continuous-control results are empirical extensions, not consequences of the finite-MDP guarantee.

Q5

“There is existing literature on learning from poor coverage and low-quality data has been extensively addressed in [1] and its successors.”

Response. Thank you for this connection. We now discuss Jin et al. (2021), Rashidinejad et al. (2021), and Uehara and Sun (2022), and explain how our reward-identification question complements pessimistic offline planning.

Those analyses use observed rewards to address uncertainty and poor coverage. Here, numerical rewards are unavailable: the additional question is whether negative selections determine the reward contrasts needed to compare covered policies. Pessimism can account for uncertainty, but cannot create reward information absent from the observations.

Proposition 8 in Appendix A.6 makes the connection explicit. Given a valid full-parameter reward confidence set and transition confidence radii, pessimistic planning bounds regret by reward and transition uncertainty along a comparator's occupancy. The reward set must retain uncertainty in unidentifiable directions. This is an application of established pessimism machinery, not a claim of a new general pessimism technique.

Our transition-log sweep further shows that a count-based pessimism penalty can change reward-attributable regret. We report this interaction and withdraw the earlier prediction that the two effects would be empirically separable.

Thank you again for helping us clarify the contribution and strengthen the evaluation. We welcome any further questions.

I thank the authors for their response to my review, which I have carefully read. Nevertheless, my concerns about the importance of the setting and the novelty of the technical results remain, and I will maintain my score.

Reviewer_zK5zOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies when negative demonstrations identify useful rewards or covered optimal policies. It separates selection semantics, reward-relevant feature variation, and transition coverage. Under a negative exponential-selection model, it characterizes reward ambiguity through centered-feature null spaces, derives an affine common-policy certificate, and develops statistical and data-collection consequences. Controlled tabular experiments examine these mechanisms and demonstrate reward-guided hazard control.

Critical strengths
  • The distinction between avoidance information, reward identification, and covered-policy identification is clear and useful. It explains why additional failures cannot resolve every obstacle to offline control.
  • The central null-space characterization and affine common-policy criterion (Theorems 2-3) provide precise, interpretable conditions. The finite-sample analysis explicitly states its stronger assumptions.
  • Experiments probe rank, policy relevance, source diversity, and misspecification. The transparent reporting of failed predictions, post-hoc experiments, loose bounds, and abstention is a strength.
  • In Table 1, regularized likelihood fitting improves within-support regret over occupancy repulsion at 500 and 2,000 negative draws, providing a useful sequential-control illustration.
Critical weaknesses
  1. Two auxiliary identification statements need sharper conditions. Proposition 2 equates Fisher-null directions with local nonidentification for an unspecified baseline model. This equivalence requires additional regularity: a Bernoulli model with probability sigmoid(theta^3) identifies theta despite zero Fisher information at theta=0. Theorem 5 also needs to distinguish parameters restricted to U from target functionals with unrestricted nuisance parameters. Its kernel-intersection condition establishes directional curvature, but does not generally identify the projected parameter. These issues appear repairable and do not invalidate the central null-space theorem.

  2. The connection to naturally collected failures remains insufficiently validated. The positive results rely on known or restricted reference distributions and specified selection semantics. The synthetic protocol supplies an empirical reference pool and known reward features; most experiments also use true dynamics. Existing misspecification tests are useful, but do not establish how to diagnose when an actual log supports the proposed interpretation. Exactly rank-deficient feature constructions also leave uncertainty about approximately low-rank settings.

  3. The decision guarantees are not yet practically informative. Appendix E.6 reports theoretical error radii including 63-404 against a maximum observed projected error of 2.61. The affine certificate accepts only 33.5% of instances despite 95% empirical epsilon-recovery. The more informative Wald alternative is asymptotic; a practically useful finite-sample calibration procedure is still missing. Moreover, the main hazard-control gains depend substantially on ridge regularization and are demonstrated at strong selection, beta=8.

Questions for authors
  1. Repair the auxiliary statements. Please add suitable regular-identification assumptions to Proposition 2 or restrict its claim to Fisher information. For Theorem 5, state whether U constrains the parameter; if nuisance directions remain unrestricted, clarify the stronger condition needed to identify all target functionals.

  2. Evaluate estimated reference distributions. Estimate q0 from a separate finite unlabeled log, vary its size alongside the negative-sample count, and compare with oracle q0. Report reward error, policy regret, and uncertainty calibration.

  3. Test approximate rank deficiency. Add small but nonzero feature singular values, varying their magnitude and the rank threshold. Report whether subspace estimation and policy certification remain reliable when there is no exact spectral cutoff.

  4. Provide a more informative certificate comparison. On the existing tabular tasks, compare the affine certificate with a bounded reward-version-set alternative. Report false-certification rates, abstention, and regret at matched confidence levels, distinguishing asymptotic Wald approximations from finite-sample guarantees.

  5. Connect model diagnostics to control failure. Using the existing nonlinear-selection and bottom-quantile experiments, test whether held-out likelihood or log-odds residuals detect misspecification before policy regret becomes large. Explain what a practitioner should check before trusting the inferred reward.

  6. Expand the sequential-control sensitivity analysis. Repeat E4b at moderate selection strengths and compare unregularized MLE, selected ridge, and occupancy repulsion with uncertainty intervals. This would clarify when the reward-inference benefit survives weaker selection and model-estimation error.

  7. Make evaluation and reproduction details explicit. Distinguish exact recovery from epsilon-recovery in the main text, show sensitivity to the regret threshold, and separate resampled negative draws from distinct trajectories. Ensure the accompanying artifact specifies the ridge grid, validation split, and optimizer settings, and includes seed-level outputs.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zK5z (1 of 4)

Thank you for your careful review. We address your concerns below and identify the corresponding changes in the revised paper. Manuscript revisions are highlighted in blue. Appendix F summarizes the experiments.

Q1

“Please add suitable regular-identification assumptions to Proposition 2 or restrict its claim to Fisher information. For Theorem 5, state whether U constrains the parameter; if nuisance directions remain unrestricted, clarify the stronger condition needed to identify all target functionals.”

Response. Thank you for identifying both issues. We have corrected the statements and included the counterexamples that explain why the stronger conditions matter.

Proposition 2 now states the Fisher-information identity ker⁡In=ker⁡I0∩N\ker I_n=\ker I_0\cap\mathcal N. Interpreting it as local identification additionally requires a regular-point condition on the baseline model. Your sigmoid⁡(θ3)\operatorname{sigmoid}(\theta^3) example is now in the text. The negative channel's global identification modulo N\mathcal N in Theorem 2 is unchanged.

Theorem 5 now separates two targets. For general source models the criteria concern first-order Fisher identification; for the stated fixed-reference negative channels they also characterize exact global identification. For a parameter restricted to θ̄ + U, the condition is U ∩ ker F(w) = {0}. For the projection of θ onto U with unrestricted nuisance directions, the condition is ker F(w) ⊆ U⊥, and the design requires the weighted information matrix ∑ⱼ wⱼFⱼ to dominate t times the projection onto U. The proof illustrates the distinction with Y ∼ N(θ₁ + θ₂, 1): curvature in the first coordinate does not identify θ₁ when θ₂ is free. The two versions coincide in E3 because every source-information matrix has range in U, so its numerical results are unchanged.

Q2

“Estimate q0 from a separate finite unlabeled log, vary its size alongside the negative-sample count, and compare with oracle q0. Report reward error, policy regret, and uncertainty calibration.”

Response. We have added the requested sweep using separate reference logs independent of the population negatives (Appendix F, “Estimated references and combined model errors”): reference sizes 250,1000,4000,16000,64000250,1000,4000,16000,64000, negative counts 100,500,2000100,500,2000, and β ∈ {4, 8}. We compare empirical and structured references with oracle q0q_0, reporting projected reward error, regret, and uncertainty coverage.

The original negative-only validation was sensitive to reference error. At β=8\beta=8 and 2,000 negatives, empirical-reference regret was 0.40/0.30/0.220.40/0.30/0.22 for reference sizes 250/1000/4000250/1000/4000, versus oracle 0.0230.023. Across the tested grid, reaching mean excess regret below 0.020.02 required 16,000–64,000 reference trajectories under that procedure. Reward error converged more slowly: at 64,000 reference trajectories, empirical and structured projected errors remained 0.0570.057 and 0.0400.040 above the oracle.

The same paragraph in Appendix F reports held-out two-sample validation on 50 fresh environments. At β=4\beta=4 and 1,000 reference trajectories, empirical-normalizer regret falls from 0.2760.276 to 0.0330.033. At this selection strength, empirical, structured, and case-control mean regrets are within 0.0060.006 of the oracle using the same validation rule. This shows that validation matters; it is not a universal reference-sample threshold.

For uncertainty, our separate-reference-log study finds naive 95% ellipsoid coverage of 0–22% at β=4\beta=4, whereas the empirical-normalizer sandwich covers 50/50 cases at each tested reference size, with wider covariance estimates. These checks concern unpenalized fits, show undercoverage in some other cells, and do not provide finite-sample calibration for selected ridge.

AuthorsOpenReview ↗

Response to Reviewer zK5z (2 of 4)

Q3

“Add small but nonzero feature singular values, varying their magnitude and the rank threshold. Report whether subspace estimation and policy certification remain reliable when there is no exact spectral cutoff.”

Response. We have added small nonzero singular values and rank-threshold sweeps (Appendix F, “Weak identification and finite-sample certification”), with s₅/s₁ ∈ {0, 10⁻⁴, 10⁻³, 10⁻², 10⁻¹}, negative counts 100,500,2000100,500,2000, and 50 fresh seeds per nominal policy geometry.

The tests show that retaining the correct rank does not make a plug-in certificate reliable. At s5/s1=10−3s_5/s_1=10^{-3} and 2,000 negatives, threshold 10−410^{-4} retained all eight directions, yet the plug-in heuristic falsely certified 3/50 instances in the nominal policy-irrelevant geometry and 45/50 in the policy-relevant geometry, at regret tolerance 0.020.02. Each issued 49 certificates.

Full-space split likelihood-ratio sets issued only 2/50 and 0/50 certificates, with no observed false certifications. These methods used separately fitted candidates; the comparison illustrates conservatism rather than a controlled dominance result. With only 50 instances, zero observed errors still permits a one-sided 95% upper error bound of 5.8%.

We now explain that weak directions require sampling uncertainty to be retained. Rank thresholding and regularization can stabilize estimation, but neither alone supplies a finite-sample decision guarantee.

Q4

“On the existing tabular tasks, compare the affine certificate with a bounded reward-version-set alternative. Report false-certification rates, abstention, and regret at matched confidence levels, distinguishing asymptotic Wald approximations from finite-sample guarantees.”

Response. We have added the requested comparisons and now separate bounded population geometry from finite-sample uncertainty. Appendix F presents the main comparison, and Corollary 4 in Appendix A.8 gives the finite-sample guarantee. The population-set and sample-size results are summarized below.

With exact population information, imposing ‖θ‖ ≤ 1 raises sound certification from 67/200 to 151/200 instances. That improvement comes from prior information. By contrast, plugging an estimated reward into the affine test ignores sampling error and can falsely certify.

For finite samples, Corollary 4 in Appendix A.8 constructs likelihood-ratio (LR) confidence sets. Under the correct likelihood, independent negatives, known reference, selection strength, features and dynamics, and a valid reward bound, coverage plus a sound decision check gives

Pr⁡(certify and normalized regret>ε)≤α.\Pr(\text{certify and normalized regret}>\varepsilon)\leq\alpha.

This is an unconditional error guarantee, not an error bound conditional on certification. On 1,000 fresh MDPs, with the same candidate policy, 500 negatives, α=0.05\alpha=0.05, ε=0.02\varepsilon=0.02, and ‖θ‖ ≤ 2, The table below reports certification, coverage, and regret conditional on certification or abstention; Appendix F summarizes the certification and coverage results:

SetCertified / abstainedReward-set coverageFalse certificationsMean regret: certified / abstained
Split LR92 / 908996 / 10000 / 10000.0001 / 0.0128
Cross-fit LR121 / 879997 / 10000 / 10000.0001 / 0.0132
Asymptotic Wald220 / 780939 / 10000 / 10000.0002 / 0.0148

Thus LR provides the finite-sample justification under the stated assumptions, but we observe no false-certification advantage over Wald in this experiment. Zero errors in 1,000 instances has a one-sided 95% upper bound of 0.30%.

At 8,000 negatives, split/cross-fit LR certify 379/412 of the same 1,000 instances, again with no observed false certifications (Appendix F). The global-curvature radius remains vacuous. We therefore present LR as a useful, sometimes conservative alternative—not a solution to misspecification or arbitrary weak-information settings.

AuthorsOpenReview ↗

Response to Reviewer zK5z (3 of 4)

Q5

“Using the existing nonlinear-selection and bottom-quantile experiments, test whether held-out likelihood or log-odds residuals detect misspecification before policy regret becomes large. Explain what a practitioner should check before trusting the inferred reward.”

Response. We have added calibrated diagnostic tests and a decision-sensitivity study (Appendix F, “Diagnosing misspecified selection”). Their main finding is that fit acceptance is insufficient to justify control, and rejection need not imply harm.

On 50 fresh hazard environments with β=4\beta=4 and 2,000 negatives, 30% random contamination increased regret from 0.0220.022 to 0.0630.063 (paired increase 0.0420.042 [0.015,0.0730.015,0.073]), but triggered only 1/50 alarms; 9/49 accepted datasets had regret above 0.050.05. Conversely, bottom-quantile selection triggered 50/50 alarms while changing regret by at most 0.0020.002.

The decision-sensitivity check compares plans under alternative contamination and temperature-mixture models. It flagged 6/13 harmful contaminated datasets versus 1/13 for fit diagnostics, but the difference was unresolved after Holm correction (p=0.063p=0.063), and 7/44 passing datasets remained harmful. It also missed harmful temperature mixtures.

Appendix H now recommends checking reference representativeness and model plausibility, using calibrated held-out diagnostics with power checks, and examining whether plausible alternative models change the chosen decision. A diagnostic pass is not a certificate. Formal certification additionally requires a calibrated reward set and a sound policy check under justified assumptions.

Q6

“Repeat E4b at moderate selection strengths and compare unregularized MLE, selected ridge, and occupancy repulsion with uncertainty intervals. This would clarify when the reward-inference benefit survives weaker selection and model-estimation error.”

Response. We have repeated E4b at β ∈ {1, 2, 4, 8} on the original 50 environments and 50 fresh ones, comparing unregularized MLE, selected ridge, and occupancy repulsion (Appendix F, “Information in selection frequencies”). The benefit survives moderate selection in some regimes, but depends materially on regularization.

At β=2\beta=2 and 500 negatives, the unpenalized constrained MLE reached the radius bound in all 50 fresh environments; grouped-fold ridge reduced regret by 0.2440.244 [0.199,0.2880.199,0.288]. At 2,000 negatives, selected ridge had lower regret than the tested repulsion configuration at every selection strength, with differences from −0.049-0.049 to −0.079-0.079; at 100 negatives, the comparison intervals did not exclude zero. Repulsion's prespecified likelihood selector chose its weakest tested strength, so this does not establish superiority over control-tuned repulsion.

We further evaluated penalty selection on fresh environments. The reward-free one-standard-error rule reduced regret from 0.0580.058 to 0.0290.029 at β=2\beta=2, 500 negatives, but worsened performance under uniform logging at β=8\beta=8. We report both outcomes rather than recommending heavier regularization universally.

The original logger also favors modal-action behavior cloning, which reaches zero regret. Appendix F therefore also reports reward-independent uniform logging: genuine failure counts yield regret 0.0190.019, versus 0.4810.481 after permuting counts in the same pipeline. This separates the value of selection information from the advantage already present in the original behavior log.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.