acceptodds
Under review as a conference paper at ICLR 2027

Safe Policy Selection for Resource-Constrained Treatment Allocation with Batch-Level Underperformance Control

Abstract

Resource-constrained treatment allocation assigns a limited intervention jointly across each deployment batch. Once a baseline policy is in use, updated data and models often yield several candidate policies, making deployment a policy-selection problem. Mean gain alone is insufficient when a candidate can improve average outcomes yet incur a material loss on some deployment batches. In this paper, we study selection from a fixed candidate set under a constraint on the probability of batch-level underperformance relative to the baseline. This event is counterfactual and must be inferred from finite randomized calibration logs. We construct conditionally valid one-sided lower bounds for paired batch gains, convert them into conservative loss flags, and calibrate their frequencies with simultaneous exact binomial upper bounds. Selection then maximizes estimated mean gain among the certified candidates and the baseline. We prove finite-sample control of the selected policy's underperformance risk and characterize certification probability and the mean gain retained by finite calibration. Experiments with learned policies and controlled repeated-log and repeated-calibration designs illustrate the two sources of uncertainty and the resulting finite-sample behavior.

Primary area:
causal reasoning
Keywords:
Causal Inference, Policy Selection, Causal Decision Making

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VYDYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies safe policy selection for resource constrained treatment allocation. Given a deployed baseline and a fixed set of candidate policies, the authors aim to maximize expected policy improvement while controlling the probability of material underperformance across future deployment batches. The proposed approach combines paired causal evaluation, conditionally valid lower confidence bounds, and simultaneous binomial risk calibration. The paper establishes finite sample safety guarantees and evaluates the framework through semi synthetic and controlled experiments.

Critical strengths

The paper addresses a meaningful deployment problem that is not fully captured by conventional expected value objectives. The distinction between uncertainty in within batch causal evaluation and variability across deployment batches is particularly useful. I also appreciate the clean formulation of the finite sample guarantee and the effort to validate different components of the framework through controlled experiments.

Critical weaknesses

See Qustions for authors.

Questions for authors
  1. Limited empirical evidence of the advantage of safety certification. Although the theoretical safety guarantee is valuable, the main end to end experiments do not clearly demonstrate when the proposed certification procedure is necessary. In Table 1, MeanOnly and NoisyCP also exhibit zero observed future violations across all settings. Moreover, the high risk candidates identified in the Criteo experiment all have negative mean gains, so MeanOnly does not select them. Could the authors include settings where a candidate has attractive expected gain but violates the batch level risk constraint? Such experiments would more directly demonstrate the benefit of certification over mean based selection and clarify the safety–utility trade off.
  2. Practical applicability of the randomized logging assumptions. The theoretical guarantee relies on conditionally independent treatment assignments within each calibration batch. However, the experiments use independent Bernoulli logging with treatment probability 0.5, while the deployment policies must satisfy a 20% treatment capacity and additional quota constraints. In practical resource constrained settings, collecting such independently randomized logs may itself violate the operational constraints. Could the authors clarify the intended logging setting and discuss whether the framework can accommodate feasible randomized allocations with dependent assignments, such as fixed budget or quota preserving randomization?
  3. Conservativeness and statistical efficiency of certification. The conservative lower bound flags can substantially limit certification power. For example, Figure 2 shows that a within tolerance batch is flagged in all repeated logs, and the candidate screening experiment excludes many candidates whose future risk intervals fall below the prescribed limit. It would be useful to characterize this conservativeness more systematically. How do certification probability and retained utility vary with batch size, the number of calibration batches, the candidate set size, and the lower bound confidence level? Could tighter conditional bounds improve certification without weakening the safety guarantee?
Reviewer_RzRdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies safe policy selection for resource-constrained treatment allocation, which is an important topic in causal inference and policy learning. It proposes a method that combines paired AIPW estimates with risk calibration to control batch-level underperformance relative to a baseline. For theoretical guarantee, the authors prove a finite-sample safety guarantee for the selected policy. Finally, they use synthetic and semi-synthetic experiments to study the safety and mean gain of their method.

Critical strengths
  1. This paper proposes a principled framework for controlling batch-level underperformance. The motivation is clear and easy to follow, since a policy with higher mean gain may still perform worse on some batches.

  2. This paper provides a finite-sample safety guarantee after policy selection. The same calibration data can be used for both certification and policy selection.

  3. The proposed paired AIPW estimator removes sampling noise from unchanged assignments. The authors also provide explicit conditional lower bounds, including the bound used in the experiments.

Critical weaknesses
  1. The novelty of this paper is limited. The proposed method mainly combines paired AIPW estimation with the existing Learn then Test framework (Angelopoulos et al., 2025) in a straightforward way. The main safety theoretical guarantee relies on standard stat/ML techniques such as concentration bounds and binomial calibration with Bonferroni correction. Therefore, I consider this a methodological extension with limited new contributions.

  2. The current experiments are not sufficient enough to show the main advantage of the method. For example, in table 1, MeanOnly and NoisyCP also have zero observed violations in all settings. In Criteo, all candidates classified as unsafe have negative mean gain. In the controlled experiment, the candidate with the highest mean gain also has zero risk throughout the scan. Therefore, the experiments do not really show the benefit of the method when a policy has high mean gain but is unsafe.

  3. The sample sizes used in the experiments are very large. Each calibration batch contains 500k or 2M individuals, and there are 160 batches. Figure 2 also shows 100% flagging for a condition where the true loss is within tolerance. The certification appears quite conservative, and it remains unclear whether the method can still provide useful certification when the batch size is much smaller.

Reference Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., & Lei, L. (2025). Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, 19(2), 1641-1662.

Questions for authors
  1. Please refer to my comments in Critical Weaknesses: can the authors include an experiment where the policy with the highest mean gain is unsafe?

  2. Can the authors show how the method performs with much smaller calibration batches?

  3. In practice, the logging policy may also need to treat a fixed number of individuals. How should we construct valid conditional lower bounds in this case (treatment assignments are dependent, and Assumption 1 can be violated)?

1 comment
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer RzRd,

Thank you for your constructive comments! We have revised the manuscript accordingly based on your feedback. As the discussion period draws to a close, we would appreciate your feedback on whether our responses have addressed your main concerns. If any questions remain, we would be happy to provide further clarification. Thank you again for your valuable time and effort in reviewing our paper!

Best regards,

Authors of paper39570

Reviewer_yC12OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

A baseline policy is deployed and the planner must choose among fixed candidates, with a cap on how often a batch may lose more than a small tolerance relative to the baseline. The bad event is counterfactual, so the paper uses randomised logs to build a conservative one-sided gain bound for each candidate and batch, flags a batch when the bound falls below the tolerance, calibrates the flag frequency with exact binomial bounds, and selects the certified candidate with the highest estimated mean gain, leaving the baseline always available. The guarantee is that the selected policy's true risk is below the target with high probability, at a price proved best possible, plus a second theorem on retained mean gain. Experiments are semi-synthetic and on a large real dataset.

Critical strengths

None

Critical weaknesses
  1. The failure the method exists to prevent never happens. The risk-blind baseline records zero violations in all 5120 future batches, and the noisy-flag variant reproduces its gains and standard deviations to five decimals, so across 80 repetitions certification changed no decision; the authors note that every high-risk candidate also has negative mean gain. The only visible effect is cost: retention falls to 91.4% and 98.9%, and 198 candidates are excluded although later measurement shows them safe, 124 of them with positive gain. The controlled scan cannot compensate, because there the best mean-gain candidate is always safe. The paper exhibits the price of the guarantee but never its benefit, so on its own evidence the method discards some good candidates for no measured gain.
  2. The mean-gain theorem is vacuous where it is used. Its error term is at least 0.86 in probability units while the reported gains are around 0.0005, so the guarantee holds for any policy; the constant is never printed, and the tighter risk bound the paper already derives is not used. That matters because the theorem is presented as one of the paper's two guarantees: at the reported scale it certifies nothing, which leaves the contribution's value resting entirely on a failure mode the experiments do not produce.
Questions for authors
  1. Can you show a candidate family in which the best mean-gain candidate is genuinely unsafe, so that certification must change the decision?
  2. What is the numerical error term of the mean-gain theorem at your settings, and can it be tightened?
1 comment
AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your review. As the discussion period draws to a close, we would appreciate your feedback on whether the additional experiments and clarifications in our response address your main concerns. If any questions remain, we would be happy to clarify them

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.