- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies safe policy selection for resource constrained treatment allocation. Given a deployed baseline and a fixed set of candidate policies, the authors aim to maximize expected policy improvement while controlling the probability of material underperformance across future deployment batches. The proposed approach combines paired causal evaluation, conditionally valid lower confidence bounds, and simultaneous binomial risk calibration. The paper establishes finite sample safety guarantees and evaluates the framework through semi synthetic and controlled experiments.
Critical strengths
The paper addresses a meaningful deployment problem that is not fully captured by conventional expected value objectives. The distinction between uncertainty in within batch causal evaluation and variability across deployment batches is particularly useful. I also appreciate the clean formulation of the finite sample guarantee and the effort to validate different components of the framework through controlled experiments.
Critical weaknesses
See Qustions for authors.
Questions for authors
- Limited empirical evidence of the advantage of safety certification. Although the theoretical safety guarantee is valuable, the main end to end experiments do not clearly demonstrate when the proposed certification procedure is necessary. In Table 1, MeanOnly and NoisyCP also exhibit zero observed future violations across all settings. Moreover, the high risk candidates identified in the Criteo experiment all have negative mean gains, so MeanOnly does not select them. Could the authors include settings where a candidate has attractive expected gain but violates the batch level risk constraint? Such experiments would more directly demonstrate the benefit of certification over mean based selection and clarify the safety–utility trade off.
- Practical applicability of the randomized logging assumptions. The theoretical guarantee relies on conditionally independent treatment assignments within each calibration batch. However, the experiments use independent Bernoulli logging with treatment probability 0.5, while the deployment policies must satisfy a 20% treatment capacity and additional quota constraints. In practical resource constrained settings, collecting such independently randomized logs may itself violate the operational constraints. Could the authors clarify the intended logging setting and discuss whether the framework can accommodate feasible randomized allocations with dependent assignments, such as fixed budget or quota preserving randomization?
- Conservativeness and statistical efficiency of certification. The conservative lower bound flags can substantially limit certification power. For example, Figure 2 shows that a within tolerance batch is flagged in all repeated logs, and the candidate screening experiment excludes many candidates whose future risk intervals fall below the prescribed limit. It would be useful to characterize this conservativeness more systematically. How do certification probability and retained utility vary with batch size, the number of calibration batches, the candidate set size, and the lower bound confidence level? Could tighter conditional bounds improve certification without weakening the safety guarantee?
Follow-up on our response
Dear Reviewer RzRd,
Thank you for your constructive comments! We have revised the manuscript accordingly based on your feedback. As the discussion period draws to a close, we would appreciate your feedback on whether our responses have addressed your main concerns. If any questions remain, we would be happy to provide further clarification. Thank you again for your valuable time and effort in reviewing our paper!
Best regards,
Authors of paper39570