acceptodds
Under review as a conference paper at ICLR 2027

AUDITED PARTIAL IDENTIFICATION FOR OFF-POLICY EVALUATION UNDER NONIGNORABLE FEEDBACK

Abstract

When reward recording depends on the unseen reward itself, even unlimited logged data may not identify policy values. We study off-policy evaluation in this setting and ask which missing rewards are most informative to recover through targeted audits. Under observed transitions, target coverage, and a shared rectangular reward model, we characterize the sharp joint set of policy values and quantify how common uncertainty cancels in policy comparisons. At the population level, identifying a cell’s reward mean reduces comparison width by exactly the product of the policies’ absolute visitation difference and the cell’s reward-mean ambiguity, independently of the recovered mean. Minimizing the remaining width therefore reduces to top- audit selection for equal costs or a knapsack problem for unequal costs. For a fixed observation design, half this width lower-bounds the worst-case expected absolute estimation error. An independent design–confirmation procedure provides asymptotically valid outer confidence intervals, including when policy visitation differences are zero or near zero. Compared with separate marginal bounds, shared-law comparison yields 19.3-fold narrower population bounds in a controlled MDP and 4.4-fold narrower plug-in bounds in a semi-synthetic KuaiRec study. Two real follow-up analyses further demonstrate the importance of outcome stability and accounting for incomplete reward recovery.

Primary area:
reinforcement learning
Keywords:
off-policy evaluation, partial identification, audit design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5uPROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies off-policy evaluation in finite MDPs where reward recording depends on the unseen reward, so policy values are only partially identified. Under a bounded selection-odds model with parameter Γ\Gamma, sharp cellwise bounds are respondent expectiles. Assuming all policies share one rectangular reward law, a policy contrast’s sharp width equals the sum of occupancy contrast times reward ambiguity, and auditing a cell removes exactly its term regardless of the revealed outcome. Audit selection therefore reduces to top-KK or knapsack. A split-sample sign-envelope procedure gives outer confidence bands robust to near-zero occupancy contrasts, evaluated in a controlled MDP, semi-synthetic KuaiRec, and two field follow-up studies.

Critical strengths

Some strengths of the paper are as follows:

  1. The identification results are correct as far as I could check them. I verified the two-cell example, the binary expectile endpoints, and the selective-success counterexample showing that intersecting projections are not sharp. This augments the paper’s most reusable contribution that missing rewards can change a policy decision.

  2. The shared-law formulation makes clear why separately bounding each policy value can be unnecessarily conservative. The reward uncertainty in cells visited by both policies may cancel in their difference. The paper connects this observation to sequential occupancy measures, where the relevant weighting depends on discounted state–action visitation rather than simply on local action disagreement.

  3. The paper reports whole-set coverage rather than relying exclusively on pointwise coverage and includes Monte Carlo uncertainty in its simulation results. It also distinguishes identification-width improvements from sampling precision and discusses the limitations of the field-study analyses.

Critical weaknesses

The weaknesses are stated below:

  1. The theoretical contributions seem incremental. The principal sharp-width identity is fine, but under Assumption 2 it follows naturally from optimizing a linear functional over a rectangular set of admissible reward means. The additive audit gains and the top-(K) or knapsack allocation rules then follow directly from that identity.

  2. Proposition 3 applies a known expectile duality for odds-ratio sensitivity models. The authors themselves acknowledge that cancellation, and its advantage over separate subtraction, were already derived by Guerdan et al. (2024). What is the novelty then in this one?

  3. The paper’s main claims are not tested on real sequential data, and the field studies mainly show that the key assumptions are somewhat inconsistent. The YOP analysis shows that the audit-validity assumption (recovering the original outcome at the original time) fails. The Coppock recovery depends on outcome stability across a fourfold incentive change.

  4. KuaiRec is sort of semi-synthetic. The response mask is imposed, and the policies are constructed. So no real setting demonstrates nonignorable reward recording, recoverable audits, and a sequential policy comparison together.

Questions for authors

Here are the questions for the authors. If they are addressed, I am willing to raise my score.

  1. Given Assumption 2, Theorem 4 follows from optimizing a linear functional over a box; then what is new beyond Guerdan et al. (2024)?
  2. It seems that, in the controlled MDP, the reference policy is the behavior policy 0.55π+0.450.55 \pi+0.45 Unif? The cancellation ratio ranges from 11.5 × to 51.5 × across behavior mixtures, so how much of the 19.3×19.3 \times headline is overlap by construction?
  3. Is the sign-envelope needed? In Tables 13–14, the ordinary centered-endpoint sup-tt band attains 94.0–100% coverage, including at exact-zero occupancy pairs. Can you exhibit a DGP where ordinary endpoint inference materially undercovers and the sign-envelope corrects it?
  4. Is there any dataset with genuine reward-dependent recording, a sequential policy comparison, and recoverable outcomes (for example, delayed-conversion logs)?
Reviewer_p3VUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies off-policy evaluation under nonignorable missing rewards. It derives policy-value bounds, develops an audit-selection criterion, and proposes inference for policy comparisons.

Critical strengths

The central conceptual distinction between identification ambiguity and sampling uncertainty is useful and cleanly motivates the idea of targeted reward recovery.

Critical weaknesses

The results are not presented along a clear progression from the research question to the assumptions, then to the methods, and finally to the conclusions; the reader has to reconstruct the logic connecting them. Key notation and central concepts are introduced before they are adequately explained, so the reader encounters them without context. Also, the substantive advances are insufficiently distinguished from established tools and from what appear to be fairly direct geometric consequences; the paper should state precisely where the new contribution lies. Moreover, the benefits are demonstrated under a reward budget, but it remains unclear whether they persist under a realistic total audit cost, with activation and recovery costs included.

In summary, the paper is not well written. The authors should rewrite it totally otherwise I cannot read it, and I don't like to read it with help of AI.

Questions for authors
  1. Given rectangular reward-mean intervals, Theorem 4 appears to follow directly from linear optimization. Which part constitutes the substantive new contribution beyond existing expectile representations and uncertainty-cancellation results?

  2. Table 16 reports narrower intervals under full-support auditing at matched reward budgets. Under what activation and recovery-cost conditions does targeted support provide a meaningful advantage? Can this be demonstrated under a matched total-cost budget?

  3. Which audit sampling design in Algorithm 1 is covered by Corollary 10? Please explicitly connect its independence requirements to confirmation audits, particularly when missing rewards are recovered from the existing passive log.

Reviewer_ovzFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript studies OPE in a inifnite horizon MDP where transitions are observed but reward observability could depend on the unobserved reward, which causes MNAR. A sensitivity model is introduced to bound the variation in response odds per state and action. Further, the paper establishes the reward mean intervals using expectiles of the observed reward distribution. Values are obtained by applying the policy occupancy weights to a feasible reward distribution.

Critical strengths
  • Give a useful message that collecting more rewards in an already audited set improves prevision but doens't remove ambiguity in relevant cells. This is important for interpreting the proposed audit allocation.
  • The statistical inference is well taken.
  • The experiments are board enough with good controls.
Critical weaknesses
  1. The population results doesn't establish the chosen support can minimize finite sample CI length, minimize total acquisition cost, or minimize the probability of certifying a policy order. An additional experiment incorporate activation costs, heterogeneous recovery probabilities and confirmation minima under a total budget can demonstrate the advantage of the cost regime more directly.

  2. The KuaiRec experiment is informative but the missiness is simulated. The constructed comparison policies preserve the within-cell recommendation distribution so the shared reward distribution holds by construction. YOP and Coppock applications have similar issue. I would suggest to strenthen the real data applications with imperfect recovery and realistic costs.

  3. The manuscript is not well written, especially the algorithm and the evidence section are not well polished. The appendices provide a thoughtful acount of the boundaries of the proposed method, but it very hard for readers to determine which assumptions support which results.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.