- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies aggregation of scalar interval estimators when their uncertainty radii are observed but their bias bounds are known only through a partial order. It asks for the output minimizing the worst-case ratio to an oracle that knows the compatible bias bounds and chooses the smallest bias-plus-uncertainty bound. The resulting rule, OACIS, has a closed-form characterization and is unique whenever the minimax ratio is finite. The paper establishes sharp uniform bounds under a subadditivity condition on the radii, relates bounded inefficiency to Lipschitz stability, and derives statistical oracle inequalities with estimated scales and a plug-in confidence level.
Critical strengths
-
The exact minimax characterization makes the abstract problem tractable. It eliminates the unknown surrogate estimands and bias values while preserving the adversarial problem. Proposition 4.1 then expresses the epigraph through finitely many interval constraints, and Theorem 4.1 reduces feasibility to pairwise intersection on the real line. This gives an exact answer for each observed input, rather than only a uniform performance guarantee. The treatment of zero denominators and the separate uniqueness argument when the optimum ratio equals one make the characterization convincing, including at degenerate inputs.
-
The boundedness results explain what partial-order information can and cannot provide. Theorem 4.2 establish a sharp bound for non-total meet-semilattices under radius subadditivity, with explicit matching constructions. Corollary 4.1 shows that no uniform finite oracle ratio is possible for a non-total meet-semilattice. The negative result makes the additional assumption meaningful and prevents the partial-order extension from being presented as cost-free.
-
Stability is an additional reason to aggregate, and it supports a substantive statistical analysis. Theorem 4.3 goes beyond existence or optimality by bounding changes in the output in terms of changes in interval centers and radii. Remark 4.3 provides a useful contrast: bounded-inefficiency selection rules must sometimes jump between distinct midpoints, whereas continuous OACIS can remain stable.
Critical weaknesses
-
The experiments use the confidence rule beyond the assumptions of the MAE guarantee. Section 5 assumes strictly positive scales, a finite scale ratio, and, for Proposition 5.2, almost-sure bounds with . This is transparently specified, but its statistical status needs clarification. In particular, the last step of Equation (25) in Appendix G.2 (p. 28) uses for every candidate in the oracle minimum. Computing a finite ratio after omitting the zero-scale candidate does not preserve that step for the full family. The issue is substantive even when the nonzero scale is known exactly. Consider a two-element chain with target and both surrogate estimands equal to zero, both bias bounds zero, , interval for finite , and abstention interval . An extension of Equation (12) that retains abstention in its oracle minimum while using the finite ratio from the other candidate would have a zero right-hand side. However, for a realization , Equations (8)–(9) give the positive output In particular, and give . Since such realizations occur with positive probability, the MAE is positive. This is a counterexample to that unchanged extension, not to the stated positive-scale theorem.
-
The real-data bias order is plausible as a modeling choice, but its oracle interpretation remains underdeveloped. The distinction between actual biases and their bounds is important here. Section 3 requires ordered upper bounds, so nonmonotonicity of the actual biases would not invalidate the framework. Indeed, a monotone upper envelope can always be constructed, but it may be substantially larger than the actual bias of a favorable candidate, thereby weakening the oracle benchmark. The substantive question is therefore whether sufficiently informative ordered bias bounds exist in the proposed application. The empirical subadditivity reported in Section 7.2 concerns the constructed uncertainty radii and does not establish the usefulness of the assumed bias order. This limits the extent to which the real-data experiment validates the substantive bias-ordering assumption, without undermining the abstract minimax results. A sufficient condition separating the history-truncation error from the omitted-reward contribution, with appropriate uniformity over the other hyperparameter, would help justify the proposed two-dimensional bias order.
-
The real-data comparison lacks uncertainty information needed to assess the size and reliability of the advantage. Table 7.2 reports NMAE 0.754 for OACIS versus 0.843 for SLOPE++, a difference of 0.089, with marginal standard errors 0.074 and 0.078. Those marginal errors do not determine the standard error of the paired difference, because both methods are evaluated on the same 25 dates. The Pearson p-values concern correlation with the reference outcome, not a comparison between methods. The reported ordering of the point estimates is clear while the precision of that ordering is not.
Questions for authors
-
What statistical claim is intended for the B.5 calibration after abstention is excluded from ? In particular, is the desired oracle comparison restricted to non-abstaining candidates, or is the rule intended as a heuristic extension? A scope clarification can resolve the mismatch with Equation (12). A claimed guarantee for the full candidate family would need to address the zero-benchmark example above. A targeted comparison on the existing data with abstention removed would show how much this extension contributes to the reported performance.
-
Can the two-dimensional bias order in Appendix B be supported by a bound that separates residual history dependence from the reward discarded by the exposure indicator? The useful answer would explain why the bound remains informative while being monotone in both coordinates, rather than requiring monotonic actual biases. Such a result would strengthen the application substantially.
-
Can the OACIS–SLOPE++ comparison be accompanied by uncertainty for the paired MAE difference, together with user counts and the precision of the A/B reference outcomes? Please explain how the analysis treats repeated users and dependence across dates. This would establish whether Table 7.2 supports a reasonably precise advantage over the strongest baseline or should be interpreted mainly as an encouraging case study.
3 comments
Thank you for the clarification and revisions. I appreciate the additional theoretical analyses and experiments. The extension of the MAE guarantee to plug-in Wald intervals and the treatment of zero-width abstention substantially address my original concern. The additional experiments also strengthen the empirical evaluation. Some concerns remain regarding the practical tightness of the assumed bias bounds. Overall, I think the revisions have addressed most of my original concerns, and I am inclined to raise my assessment.
Thank you for reviewing our revisions and for your encouraging feedback! Your constructive comments have helped us strengthen both the theoretical analysis and the empirical evaluation.
Thank you for highlighting the limitation of Proposition 5.2 for zero-width abstention and for your detailed questions about Web OPE. Our General Response and Revision Summary (a separate post) presents the theoretical extensions, experimental revisions, and planned Web OPE analyses relevant to your questions. Please read it alongside the specific responses below.
Q1. Zero-width abstention and its ablation
Please refer to the general response, specifically Section I.2 for the theoretical treatment and Section III for the ablation study.
Q2. Joint bias bound for history truncation and discarded rewards
Please refer to Section I.2 in the general response.
Q3. Paired uncertainty, reference precision, and repeated users across dates
Please refer to Section III in the general response. We also clarify that there should be some dependency across dates due to the repeated users and the result should be treated as more of an encouraging result rather than statistically significant evidence.