acceptodds
Under review as a conference paper at ICLR 2027

Beyond Cost Sensitivity: Outcome-Aware Off-Policy Evaluation via Local Disclosure

Abstract

We study off-policy evaluation under local disclosure when policies attach fixed fees or subsidies to strategic actions. Ratio-only disclosure groups applicants with the same cost sensitivity , although their absolute stakes and default risks can differ. We characterize when this grouping retains enough information for evaluation. For zero-price targets, covered outcome averages by action and suffice. For a priced target, two worlds can agree under every zero-transfer experiment over a fixed library yet have default risks zero and one. We use randomized cardinal anchors to label outcome-relevant cells. A common shift of the displayed transfers preserves choices and matches the focal transfer to the target. Horvitz-Thompson moments then consistently estimate approval, risk, and value. One frozen log also supports uniform evaluation of a finite predeclared policy class after data-dependent selection. In the coarse independent experiment, direct labels improve the minimax mean squared error from to . Our budget analysis accounts for recruitment, follow-up, and gross subsidies. A complete priced-menu simulation evaluates eight policies from three logging menus and verifies transfer matching. Further experiments recover the rare-cell rates and test scalar-type insufficiency with public credit covariates.

Primary area:
learning theory
Keywords:
off-policy evaluation, local disclosure, strategic decision-making, identifiability, randomized experiments, minimax estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kvErOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies off-policy evaluation when agents strategically respond to disclosed policies and actions that can carry fixed fees or subsidies. Its main observation is that the scalar cost-sensitivity representation used by prior local-disclosure approaches can be sufficient when monetary transfers are absent, but can lose information needed for evaluating priced policies because agents with the same relative sensitivity may have different absolute stakes and outcome risks. The proposed remedy uses randomized monetary calibration to reveal outcome-relevant regions of a two-dimensional latent type and then matches the realized transfer to the target policy. The resulting logged moments can be reused to estimate approval, default risk, and value for a predeclared policy class. The paper also gives identification, consistency, uniform-evaluation, minimax-rate, and budget results, together with priced-menu simulations and a semi-synthetic credit experiment.

Critical strengths

The paper isolates a clear and consequential identification boundary. Showing that the usual one-dimensional sensitivity representation is adequate for zero-price targets but can be fundamentally insufficient for priced targets is more than an estimator refinement, it identifies when the existing information structure itself ceases to support evaluation. This gives the work a strong conceptual contribution.

The proposed solution is unusually complete from identification through policy selection. The paper does not stop at a non-identification result; it develops cardinal calibration, transfer matching, reusable moment estimation, finite-policy selection guarantees, budget accounting, and controlled experiments designed to verify the specific mechanisms. This coherence substantially strengthens the case for acceptance.

Critical weaknesses

The practical validity of the method depends on strong behavioral and experimental assumptions. In particular, identification requires that monetary calibration can be randomized in the proposed way and that outcomes for a selected action obey the relevant transfer-exclusion restriction. These assumptions seem central rather than technical, if transfers directly change downstream default behavior beyond the modeled channel, the identified cell moments may no longer transport to the target policy.

The empirical evidence establishes mechanism validity more convincingly than real-world applicability. The headline validation consists of a complete priced-menu simulation plus experiments using public credit covariates, rather than observed strategic behavior and outcomes under actual monetary menus. This is appropriate for verifying the theory, but leaves uncertainty about whether sufficiently informative and legally/operationally feasible calibration menus exist in realistic applications.

Questions for authors

How sensitive are the risk and policy-selection results to moderate violations of selected-action transfer exclusion? Could you provide a sensitivity analysis showing how much direct transfer dependence is needed before the preferred policy changes?

What observable conditions would let a practitioner determine, before running the study, that the available action library can generate sufficiently informative cardinal cells without requiring prohibitively large subsidies or fees?

Can the proposed identification strategy be validated with behavioral or outcome data in which agents actually face monetary menus, rather than simulated strategic types?

Reviewer_urtvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies off-policy evaluation with strategic applicants who respond to approval rules, fees, and subsidies. It shows that relative cost sensitivity may suffice for zero-transfer targets but can fail to identify default risk under priced policies, even when known exactly. To address this limitation, the paper introduces randomized transfer menus that reveal outcome-relevant latent-type cells and match focal transfers to target values. It develops consistent estimators and uniform guarantees for evaluating a finite prespecified policy class, together with rare-cell and budget analyses under a restricted observation model. The framework is validated through theoretical results, synthetic experiments evaluating eight policies from three logging menus against independent and pooled rollout baselines, and a semi-synthetic credit experiment using real covariates but simulated behavior and outcomes.

Critical strengths
  1. The target-matched cell construction provides a coherent connection between menu geometry, outcome identification, and reusable policy evaluation. The paper also makes its theoretical scope relatively explicit and provides consistency, finite-policy uniformity, rare-cell, and budget results rather than relying only on an algorithmic proposal.
Critical weaknesses
  1. Strong assumptions with unclear real-world applicability. The method requires common transfer shifts to be applied to all displayed options, including the displayed exit, and assumes that unchosen offers have no direct effect on outcomes under the selected action. In practice, an unshifted outside option or menu-context effects could violate these conditions and undermine the validity of the estimates.

  2. Validation relies largely on simulation, limiting evidence of generality. Behavior and default outcomes in the semi-synthetic experiment are still artificially generated, and the multi-policy experiment has substantial shared structure. The optimized three-policy target-span design is observation-equivalent to the reusable design in the reported setup, so the incremental advantage over optimized active collection remains unclear. Performance on more complex policy sets and real behavioral data therefore remains uncertain.

Questions for authors
  1. How would an unshifted outside option affect cell identification and policy evaluation? Could a small sensitivity experiment quantify the resulting bias?

  2. Since the optimized three-policy target span is observation-equivalent to the reusable design in the current experiments, in what regimes does the proposed construction provide a strict statistical or budget advantage over an optimized target-span collection design?

Reviewer_2951OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies off-policy evaluation with strategic agents when actions involve monetary transfers (such as fees or subsidies). The main observation is that cost sensitivity alone may not be sufficient to evaluate policies, as agents with the same cost sensitivity can respond differently to monetary transfers and have different outcomes. The authors propose randomized menus to identify hidden-type cells and estimate outcome moments that can be reused across multiple policies. They provide theoretical guarantees for identification and estimation, along with a minimax analysis and experiments.

Critical strengths

I think the problem is interesting, and the motivating example is clear. The idea of estimating only the relevant outcome moments instead of the full type distribution also makes sense to me. The theoretical results seem correct, and I did not find any major issues with the proofs.

Critical weaknesses

I am not fully convinced about the technical novelty of main results. For example, Theorem 1 seems to follow from standard convex geometry. The estimation results in Theorems 2 and 3 mainly use inverse propensity weighting and concentration bounds, given the assumptions on the menu coverage. The minimax result in Theorem 4 is interesting, but the improvement seems expected since one experiment directly observes cell membership while the other does not. I think the paper needs a better comparison with existing work on discrete choice identification and experimental design.

I also wonder when the proposed method would be better than directly collecting data using target-policy menus. In Appendix E, the authors show that an optimized target-policy experiment can be observationally equivalent to their design. This seems particularly relevant since reusing one log across multiple policies is one of the main motivations of the paper.

Finally, I find the assumptions rather restrictive. In particular, the common transfer shift also applies to the no-loan option, and the outcome exclusion assumption may be difficult to justify in practice. The paper only discusses lending, and the experiments use simulated outcomes. It would be helpful to see whether the framework works in other settings.

Questions for authors
  • Could the authors give a concrete example beyond lending where this framework would be useful and its assumptions are reasonable?

  • When does the proposed design offer a strict advantage over an optimally designed target-policy experiment?

  • Could the authors better distinguish their contributions from the existing literature on discrete choice identification and experimental design?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.