acceptodds
Under review as a conference paper at ICLR 2027

Evidence Resolution: Certifying How Much Personalization Finite Evidence Can Support

Abstract

Flexible treatment-effect learners can propose action rules more finely than finite randomized evidence can defend. We formalize evidence resolution: the finest clinically worthwhile action granularity supportable by a representation and prespecified evidential criterion. A symmetric one-signal oracle yields the worst-case frontier ; instance-dependent theory separates exact localization from value-tolerant localization and yields a search-versus-multiplicity crossover against exhaustive certification. HonestER freezes representation choices upstream and independently certifies refinements with family-local backoff. A prospectively locked Criteo model for exact reference-best recovery plus certification failed as specified. The decomposition explains why: at the finest depth, selected regions certified in 94% of runs but matched the empirical reference-best bin only 40%, while the leading bins were only 0.35 standard errors apart. Post hoc, a zero-fit certification-resolution prediction from frozen reference geometry matches the observed at 8 of 11 sample sizes and misses the other three by one bit. This supports a central distinction: certifiable resolution, exact localization, and deployed value are different statistical objects.

Primary area:
causal reasoning
Keywords:
Causal inference, heterogeneous treatment effects, policy learning, evidence resolution, multiple testing, best-arm identification, subgroup discovery, uplift modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FqZkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper studies whether finite randomized data can support the granularity of recommendations produced by flexible treatment-effect learners. It introduces “evidence resolution” and proposes HonestER, which freezes representation choices before independently certifying refinements, with a backoff mechanism when evidence is insufficient. Theory and a Criteo experiment aim to distinguish certifying an action granularity, locating the best subgroup, and achieving treatment-policy value. While this distinction is interesting, the precise methodological contribution and the guarantees delivered by HonestER remain difficult to discern from the presentation.

Critical strengths

The paper starts from a very interesting idea: flexible data-adaptive learners of causal effects can support individualized recommendations at a granularity level that is much more detailed than what the actual data can support. This situation is subject of a current open debate in causal inference, with profound implications for personalized medicine, and has received attention and opinions from some of the most prominent statisticians such as F. Harrell and S. Senn (check [1] and [2]). It is thus a modern open problem with important methodological implications. I appreciate the attempt of this paper on touching such an important topic.

Critical weaknesses
  • While the paper touches an important topic in methodological causal inference, it does little to position it in such context. Prior relevant work objecting to the extrapolation of flexible data-adaptive learners of causal effects into recommendations beyond the abstraction level provided by the data, like [1] and [2], are completely omitted. In fact, the introduction of the paper does little to position and motivate the problem at hand in the prior scientific landscape. No clear motivation is given on why the problem is important, what its implications are, and what has been positioned before. The introduction section has no references, which are relegated to section 2, where comparisons with other frameworks are given. Yet, such comparisons are given in a very indirect way, so it is not clear to the reader how they sit in the focused topic of “evidence support”. This is because a one-line explanation of each is provided, but not cohesive line linking them to the specific topic of this paper is missing.
  • The writing seems rushed, unpolished and unfinished. There are big portions of the paper without a single bibliographic reference. Some paragraphs are unconnected, or it is hard to follow a cohesive chain-of-ideas linking them. For instance, the third paragraph in the introduction starts discussing benchmark results, but the objective, tasks, motivation and purpose have not been defined before.
  • Mathematical objects are used without being properly defined. Section 3 starts by defining CvC_v in terms of πv\pi_v and πpa(v)\pi_{pa(v)} but none of these objects are defined, therefore it is hard to understand what it is meant. In fact, many mathematical objects and tools used in section 3 are left implicit, undefined, uncommented or unmotivated. This particular section is written in a rushed way and there is no effort to be explicative or pedagogical for the reader. Even after reading it multiple times top-to-bottom, it is hard to evaluate the contribution, novelty, motivation and focus of such section.
  • The mathematical problem is not clearly defined. What is the target? Is it a policy/recommendation rule built from a heterogeneous treatment effect (HTE) estimated with flexible data-adaptive learner (super-Learner). Why are these objects and objectives not defined in section 3? What is the statistical certification/guarantee produced, and where is it exactly defined?
  • More worryingly, after reading the paper several times, it is still not clear to me what the exact methodological contribution of the proposed approach (HonestER). Algorithm 1 is not illustrative at all, and does not really clarify the methodological steps needed to produce the intended results. It is too coarse. I believe this is consequence of the problem not being clearly defined in term of concrete sub-problems and contributions to the literature.
  • In summary, I acknowledge that the seed idea motivating the problem is interesting and I recommend the authors to keep working on it. Yet, the current version of the paper is not really ready for submission. It needs more polishing, motivation, literature review and dedicated step-by-step presentation of objectives, methods, and contributions.

References

[1] Hoogland, J., IntHout, J., Belias, M., Rovers, M. M., Riley, R. D., Harrell, F. E., Jr., Moons, K. G. M., Debray, T. P. A., & Reitsma, J. B. (2021). A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint. Statistics in Medicine, 40(26), 5961–5981

[2] Senn, S. (2018). Statistical pitfalls of personalized medicine. Nature, 563(7733), 619–621.

Questions for authors
  • What is the precise statistical problem and guarantee? Please define the target, the role of the learner and resulting policy or recommendation rule, and what “evidence support” means mathematically. What exactly does HonestER certify, under which assumptions, and where is this guarantee formally stated?

  • What is HonestER’s methodological contribution, and how does it work? Please provide a step-by-step explanation of its inputs, intermediate quantities, and outputs, including definitions of all mathematical objects needed. How do these steps deliver the stated guarantee, and what is new relative to existing methods? Algorithm 1 is currently too coarse to resolve these questions.

  • How does the paper connect to the existing debate on unsupported personalization? Please explain how HonestER addresses concerns that flexible treatment-effect learners may produce recommendations at a finer granularity than the data support, including those discussed in [1] and [2]. What specific gap does it fill, and how do the frameworks reviewed in Section 2 relate to that gap?

Please also feel free to address any of the topics and comments raised in the list of weaknesses.

4 comments

Thank you for identifying where the submitted paper was hard to evaluate. We now introduce the treatment decision before the notation. The example after Algorithm 1 follows a change from treating the top half to treating the top quarter. HonestER is now presented as a reference procedure with a stated error guarantee, and the paper makes no claim that it outperforms other valid certifiers.

Q1. The statistical problem and the guarantee. Section 3.1 starts from randomized trial units (X,A,Y)(X,A,Y) with potential outcomes, consistency, no interference and a known assignment probability. A policy π\pi decides who is treated, and its value relative to treating nobody is Vκ(π)=E[π(X){τ(X)−κ}]V_\kappa(\pi)=\mathbb E[\pi(X)\{\tau(X)-\kappa\}], where the threshold κ\kappa is fixed before any certification outcome is read. Candidate rules come from nested regions Rb,1⊇Rb,2⊇⋯R_{b,1}\supseteq R_{b,2}\supseteq\cdots, and two kinds of claim are needed. Entering region Rb,ebR_{b,e_b} requires its mean effect to exceed κ\kappa (Equation 3). Moving from depth dd to d+1d+1 removes the shell Sb,d=Rb,d∖Rb,d+1S_{b,d}=R_{b,d}\setminus R_{b,d+1} and requires the shell's mean effect to be below κ\kappa, which is the condition for the refinement to increase value (Equation 4). Section 3.2 then separates three quantities that the submission ran together: the supported resolution of a fixed representation in the population, the resolution a procedure reports, and the minimax evidence resolution, which is an information limit.

Theorem 2 (Section 4.2) states the guarantee. Conditional on the training and validation data, each family runs a fixed sequence of tests and stops at its first non-rejection. If every true-null component test rejects with probability at most its allocated level, the probability of any false entry or exclusion certificate is at most α\alpha, and the same bound covers returning a policy whose required path contains a false claim. The guarantee concerns region averages along the selected path. It makes no claim about people outside the initial entry, benefit to every treated person, optimal value or recovery of the finest valid rule. A whole-population entry followed by supported exclusions also justifies comparison with treating everyone. The historical Welch–Normal tests are approximate and can be anticonservative in sparse binary cells; the exact error bound requires valid components. Appendix C gives a finite-sample alternative for bounded outcomes, with its assumptions.

Q2. The method and what is new. Algorithm 1 (Section 4.1) begins with the three independent samples. It then lists candidate construction on the training data, the choice of one path and entry per family on the validation data, the quantities frozen before certification, the entry test, the sequential shell tests, the stopping rule and the final choice among families. A worked example follows it. Appendix D gives the ECV planning equations, the tie rules and the family allocation. While checking Appendix D against the historical code, we found that the submission described the prior-variance estimate incorrectly. The revised equation matches the code that produced the results, and no original endpoint was recomputed with a changed planner.

The introduction now ends with the paper's four contributions: the resolution target with its exclusion claims, the ordered-gap analysis for a restricted passive Gaussian model with its value-tolerant variant, explicit entry and exclusion claims with error control under valid component tests, and the two randomized-data studies. The paper also says that the fixed-sequence argument and the union bound are standard. The experiments examine these distinctions and show their limits. ECV is a planning heuristic, and the completed comparisons do not confirm a method advantage.

Q3. Relation to earlier work. Thank you for the references. The introduction now discusses Senn (2018) and Hoogland et al. (2021), and explains the difference between conditional average effects and individual response. Section 2 acknowledges finite-sample treatment-choice guarantees (Manski), Bayesian credible subgroups with simultaneous posterior bounds (Schnell et al.), controlled subgroup selection, safe improvement and inference after selection. Our target concerns group-average action changes on a specified path, with frequentist error control conditional on the training and validation choices. Section 2 now also contrasts the aggregate value guarantees of safe and confidence-aware policy learning with these local claims: a rule can meet an aggregate guarantee while withholding treatment from a group that benefits. It also covers sorted-group analysis and rank-weighted average treatment effects, which summarize effects along a ranking, and the minimax theory of optimal subgroup selection. We do not claim to introduce valid subgroup selection or full-sample policy learning. The sample-reuse comparison in Section 3.5 is limited to the two constructions it defines.

Presentation and AI assistance. We take the criticism of the writing seriously. We rewrote the sentence quoted in the review. Section 3 now defines the data, policy value, entry, exclusion, validity and resolution targets before presenting the results. Appendix I.8 adds a notation table, Appendix A is typeset, and Appendix G keeps the frozen protocols with readable equations and wrapped code blocks. AI assistance was disclosed on the submission form, and the revised AI use statement describes its uses. We remain responsible for every claim, proof, citation and line of code, and for the final text.

I thank the authors for taking my comments seriously. The changes to the exposition do bring clarity. In particular, the opening example makes it much easier to see why evidence of benefit in one subgroup leaves open whether treatment should be withdrawn from everyone else. I also found it easier to follow what HonestER tests and when it stops.

Two concerns remain for me. First, although the procedure is clearer, I still find it difficult to see how much of the methodological contribution lies in the choice of claims to test and how much lies in the testing procedure itself. Second, the Hillstrom example illustrates why the distinction matters, but all three methods ultimately recommended emailing everyone. I would have liked to see a case where the methods recommend different rules and to understand what that difference means for policy value.

Taking everything together, I think the revision merits an increase of one rank in my score. These remaining concerns keep me from going any furhter.

Thank you for re-reading the revision. Your question about where the contribution lies helped us make that distinction more explicit.

Claims and procedure. The main methodological contribution is specifying what a treatment rule must establish at each step. To start treating a group, its average benefit must exceed the stated threshold. To narrow that rule, the average benefit in the group that would lose treatment must fall below the threshold. Within each family, HonestER tests the claims in order on data held out from planning and stops that family at its first failed test. The statistical tools are standard: Theorem 2 applies fixed-sequence testing within each family and a union bound across families. Theorem 1 addresses a separate question: how much data any selector needs to identify the best group in the balanced Gaussian model of Section 3.4.

Hillstrom illustrates why the choice of claims matters. Twenty-two distinct smaller groups passed a benefit test. Emailing only the passing group with the highest estimated benefit would have lost an estimated 34.1 net visit-equivalents per 1,000 eligible customers relative to emailing everyone. All three procedures recommended emailing everyone. Narrowing that recommendation required evidence that the customers left out benefited too little to justify the email cost. None of the proposed exclusions met that condition.

When the procedures chose different rules. On Criteo, we held the candidate groups, required claims, and component tests fixed. At n = 262,144, HonestER and single-path selection both certified a rule in 69 of 100 paired runs and chose different rules in 14 of them. In paired run 10, HonestER targeted 25.1% of the evaluation customers and single-path selection targeted 14.0%. Their estimated net values were 5.33 and 6.87 visit-equivalents per 1,000 eligible customers, respectively. Across these 14 runs, HonestER's rule had the higher estimated value in 10.

Most of the difference in average value came from how often each procedure certified any rule: Bonferroni reuse in 98 of 100 runs, HonestER in 86, and single-path selection in 71. Among runs in which each procedure certified a rule, its mean estimated value was about 6.6 per 1,000. Runs with no certified rule count as zero, giving overall means of 6.48, 5.68 and 4.69, respectively. Bonferroni reuse exceeded HonestER by 0.80 (paired 95% interval 0.39–1.21), and HonestER exceeded single-path selection by 0.99 (0.47–1.52). At n = 65,536, single-path selection certified a rule in 11 runs and HonestER in 2, and single-path selection had the highest mean value. These intervals hold the shared evaluation sample fixed. In these runs, the procedure mainly governed how often a trial supported a personalized rule; the claims governed what that rule means. Section 5.4 and Appendix I.6 report the comparison and its limits.

Reviewer_CghkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how finely a randomized trial can justify personalizing a treatment. For example, a model may rank patients by predicted benefit, and the question is whether the trial can certify that treating only the top half, the top quarter, or a much smaller group is worthwhile. The authors call the finest level the data can certify the "evidence resolution." Their theory separates three questions: whether a chosen subgroup can be certified, whether it is exactly the best subgroup, and how much value is lost by picking a near-tie. The method, HonestER, learns several candidate rankings, picks one candidate per ranking on validation data, and certifies it on untouched data. On a large randomized advertising dataset, a prespecified test of the theory failed. A post hoc power calculation then matches the observed resolution at most sample sizes, but it does not test the theory, and the method itself rests mainly on one small secondary comparison.

Critical strengths

The paper makes an interesting applied point that matters for anyone building treatment rules from trial data. Certifying that a fine subgroup benefits, identifying exactly which subgroup is best, and the value lost by choosing among near-ties are different statistical questions, and the paper shows on real outcomes that they come apart. In the Criteo analysis, the region selected at the finest level is almost always certified even though it is usually not the best region, because the leading regions cannot be told apart. Together with the value-tolerant analysis, this makes a clear case that "which subgroup is best" is often the wrong question to put to a trial. Applied subgroup work misses this point often, and I found it the most useful part of the paper.

The empirical reporting is unusually honest. The prospectively locked endpoint failed and is reported unchanged, later analyses are labeled post hoc, and a follow-up on a second dataset was stopped at its own prespecified gate rather than replaced with a friendlier one. That makes the empirical program easy to evaluate, and my first concern relies on the authors' own labeling.

Critical weaknesses

1. The real-data evidence does not test the theory, and the main agreement result is weaker than "8 of 11" suggests. The theory is about the extra cost of searching for a fine region on top of certifying it. Its one prospective test on real outcomes, which included that search cost, failed, and the authors say so openly. What remains is a post hoc prediction that plugs the best bin's estimated effect and noise into the standard power formula for a one-sided test. That formula should track certification rates whenever the normal approximation holds and the selected bin is the best one, so agreement confirms the power calculation and says little about the search costs the theory describes. The authors themselves note that the worst-case depth law is not a quantitative predictor for this dataset.

The count also overstates the agreement. At five of the eleven sample sizes, no depth reaches the 80% certification rate that defines the resolution, and the prediction says the same, so those agreements carry no information. At the six sample sizes where the resolution is above zero, the prediction matches at three (Figure 2(b)). The abstract and the experiments section should report agreement on those six, next to the cell-level calibration.

To be clear about the kind of concern: the main bounds I checked are correct, though standard in kind (two-point and Fano lower bounds, Bonferroni-type upper bounds). My concern is how the real-data evidence is framed.

2. The comparison behind the method is narrow, and the method itself has little direct evidence. Proposition 1 compares two ways of certifying a selected region: split the data, choosing on one part and certifying on the other, or certify every candidate at once with a Bonferroni correction. Given the paper's definitions of the two costs, the result follows in one line, so its content lies in which alternatives are compared. Bonferroni over all candidates is not the only way to reuse the selection data. Conditional and hybrid inference on the winner (Andrews, Kitagawa & McCloskey, 2024, Quarterly Journal of Economics) and data carving (Fithian, Sun & Taylor, 2014, arXiv:1410.2597) both certify a selected winner without paying the full multiplicity price, and the paper does not discuss them. I see this as a gap in novelty and positioning; the mathematics I checked is fine.

On the method side, the Criteo study does not run HonestER, as the authors note. The evidence for the portfolio is one secondary comparison on simulated cases, a gain of about 4% in value over a single-representation version, with no external baseline for the final method. The other variants the paper tried either failed their gates or showed no advantage.

A smaller point on presentation: the method section introduces entries, shells, families, expected certified value, and empirical-Bayes depth scores in two paragraphs, so the procedure cannot be reconstructed without the appendices, where much of the mathematics is typeset as plain text.

Questions for authors
  1. At each depth, the reference-best effect is the largest of many noisy bin estimates, so plugging it into the power formula should make the prediction optimistic at fine depths, which is where all three misses occur. Your Protocol A splits the reference pool, one part to pick the winner and the other to estimate its effect, for exactly this reason. Could you rerun the Criteo prediction with the same split? If the optimistic misses survive, I would accept the attribution to finite-sample localization. As it stands, the selection-conditioned curve removes the selection error and this bias at the same time, so I don't think it separates the two.

  2. Section 5.3 reports that the simplified method failed its locked noninferiority test and that the fixed-allocation procedure was retained afterward. Was the comparison of that procedure with the single-path version among the prespecified contrasts of the 900-case confirmation? If not, I would call it post hoc in the main text rather than secondary.

  3. Your exact-localization theorem gives a complexity computed from the gaps between the best region and its competitors, and the frozen reference geometry gives those gaps at each depth. Does that complexity at least rank the depths by how hard exact recovery turns out to be on Criteo, and roughly locate where recovery drops at the finest depth? That would be the first contact between the paper's main new theorem and real outcomes.

3 comments

Thank you for separating the useful statistical distinction from the evidence needed to support the method and the theory. We ran the analyses you asked for. The revised paper keeps the failed primary tests and labels every later analysis by its actual status (Table 1 and Appendix I).

Weakness 1 and Q1: the 8-of-11 agreement. We agree. At five of the eleven sample sizes both the observed and the predicted resolution are zero, so the informative agreement is 3 of 6 positive-resolution sizes. The abstract gives both counts and Section 5.1 reports the 8/11 total alongside them, and Section 5.1 notes that the eleven points are dependent and are not independent replications.

We reran the prediction with the split you proposed. Half A of the 2,796,306 reference units selects the winner at each depth, and half B estimates its effect and variance. The score model, the bin boundaries and the original 8,800 experiment rows are unchanged. The designated split gives agreement at 7 of 11 sizes overall and 2 of 6 positive-resolution sizes. Across 20 deterministic splits, positive-size agreement ranges from 2/6 to 6/6, with median 4/6. A joint bootstrap that resamples reference units and whole experiment-replicate curves gives a 2.5th–97.5th percentile range of 6 to 10 overall matches, conditional on the fitted model and the bin boundaries. Section 5.2 and Appendix I.1 report these results and keep them separate from a later run that generated fresh scaling rows.

The selection-conditioned curve fits better (10 of 11 on the designated split). As you pointed out, conditioning on the experiment's selected bin changes both the selection and the reference-selection bias. We therefore withdraw the attribution of its improvement to finite-sample localization. The post hoc power calculation remains a certification diagnostic and does not validate the search-cost theory.

Q3: do the reference gaps explain recovery difficulty? Partly. Appendix I.2 reports both diagnostics we have. A Gaussian approximation built from the archived reference means and variances, with no fitted constants, predicts observed recovery with cell RMSE 0.065. At the largest sample size it predicts recovery of 0.390 at depth 8, against 0.40 observed, but it misses depth 7, with 0.706 predicted against 0.93 observed. At depth 8 it omits one empty reference bin and keeps one bin with zero estimated variance. A separate support audit on fresh scaling rows treats such cells as unestimable, so depth 8 drops out. On the remaining seven depths, the balanced equal-variance proxy has rank correlation −0.178 with observed difficulty, and the support-adjusted gap proxy does no better than a simple count/support baseline (0.802 for both). The nearest-competitor term dominates at every depth. These data therefore give no distinctive real-data check of the multiplicity term or of the equal-variance theorem, and Section 5.2 says so.

Section 5.3 and Appendix I.3 add a controlled Gaussian calculation: 144 spectra in 72 sparse/crowded pairs, where the two spectra in a pair share the candidate count and the nearest gap. On 54 held-out pairs, the ordered-gap measure has mean absolute log-budget error 0.620, against 0.903 for both planned nearest-gap baselines and 0.738 for an inverse-squared-gap sum added post hoc. The planned baselines are constant within each pair by design, which makes the comparison easier for any predictor that uses the other gaps, and the paper says so. Across all spectra, the empirical-mean recovery budget lies between 0.88 and 3.23 times HselH_{\mathrm{sel}}, inside the theorem's constants. These budgets are for empirical-mean selection and are not a numerical solution of the minimax problem.

Weakness 2 and Q2: was the portfolio contrast prespecified? No. The FixedECV − SinglePath contrast was neither the locked primary nor a named secondary contrast, so it is post hoc, and Section 5.4 and Appendix I.4 now say so. Its mean is +0.001187, with pointwise paired 95% interval [0.000443, 0.001932]. The locked primary test, noninferiority of ValueFixed to FixedECV, failed: the one-sided lower bound was −0.000553 against the −0.0005 margin. The locked analysis also computed two descriptive contrasts, ValueFixed − SinglePath = +0.000932 [0.000213, 0.001651] and ValueFixed − adaptive ECV = +0.000252 [−0.000356, 0.000860]. Table 10 lists all four with their status. None of them replaces the failed primary test.

Proposition 1. We agree that it is elementary and that Bonferroni correction is not the best way to reuse selection data. Section 3.5 and Appendix B.7 now call it an algebraic comparison of defined component-budget proxies. Appendix B distinguishes marginal requirements from joint success and explains the extra occupancy assumptions needed to translate per-candidate counts to trial size. The comparison does not characterize all valid uses of selection data. Section 2 discusses inference on winners (Andrews, Kitagawa and McCloskey), data carving, selected-subgroup inference and SNPL. Appendix I.7 reports completed known-variance Gaussian comparisons, including conditional winner inference and data carving. These are special cases. They do not implement the full hybrid method or SNPL, and they show no uniform advantage for splitting.

Direct method evidence. Appendix I.6 adds matched comparisons on 6,000 synthetic cases and 300 full-pipeline Criteo runs, against single-path, Bonferroni, Holm and other reuse baselines. None of the 20 combined utility-and-safety checks passes. At the largest tested Criteo size, n=262,144n=262{,}144, Bonferroni and Holm reuse have higher mean value than HonestER. At n=65,536n=65{,}536 they are slightly lower, while SinglePath has the highest mean. On Hillstrom (Section 5.5), all three methods deploy the same treat-all rule. Their overlapping test records show 34 of 36 benefit claims passing and none of 26 exclusion claims. This illustrates the entry-versus-exclusion distinction and supplies no evidence of a method advantage. The original Criteo scaling study is still described as a depthwise certification study, which does not run the full procedure.

Presentation. Algorithm 1, the planning equations in Appendix D, the notation table in Appendix I.8 and the typeset Appendix A address the presentation concerns.

Reviewer_wBVTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how much personalization finite data can support. It separates finding the best subgroup, certifying its benefit, and achieving policy value.

Critical strengths

The paper clearly distinguishes failing to identify a unique winner from failing to certify benefit. This matters because several near-optimal choices may deliver similar practical value.

Critical weaknesses

The main positive resolution-prediction result is post hoc and lacks independent confirmation. The Criteo study also does not run the full HonestER pipeline, limiting validation of the complete method.

Questions for authors

How stable is the 8-of-11 agreement after accounting for uncertainty in the reference effects?

1 comment

Thank you for recognizing the distinction between finding a unique winner, certifying benefit and obtaining value. We agree that the post hoc agreement result needed an uncertainty analysis and that the original Criteo study did not test the full procedure.

How stable is the 8-of-11 agreement? Less stable than the submission suggested. Section 5.1 now separates the eight matches into five zero-resolution sizes and 3 of 6 positive-resolution sizes. In the new reference split, half A of the reference pool selects the winner and half B estimates its effect and variance. On the designated split, the prediction matches 7 of 11 sizes overall and 2 of 6 positive-resolution sizes. Across 20 deterministic splits, positive-size agreement ranges from 2/6 to 6/6, with median 4/6. A joint bootstrap over reference units and whole experiment-replicate curves gives a 2.5th–97.5th percentile range of 6 to 10 overall matches. Appendix I.1 states what stays fixed (the score model and the bins) and notes that the splits reuse observations. The agreement depends on how the reference is estimated, and it remains post hoc.

Full-method evidence. Section 5.4 and Appendix I.6 now report matched simulations and full-pipeline Criteo comparisons. None of the 20 combined utility-and-safety checks passes, so they do not confirm a method advantage. Section 5.5 adds a full-pipeline analysis of the Hillstrom e-mail trial. All methods treat everyone, and their audit policies are identical. A post hoc look at their test records finds 34 of 36 benefit claims passing and none of 26 exclusion claims. This is a further illustration of the paper's distinction, with no claim of independent confirmation or better value.

The main positive result is post hoc. We agree. The abstract, Table 1 and the Discussion keep the failed locked Criteo endpoint in view and separate the later analyses from confirmation. The descriptive separation of certification from winner identity is visible in the original experiment: at the finest depth the selected bin was certified in 94 of 100 runs but matched the empirical reference winner in only 40.

Comments on the paper

2 comments

Summary of the revision

We thank the three reviewers for their detailed reviews. We have uploaded a revised PDF. The revised definitions and algorithm are in Sections 3 and 4.

What changed

  1. Problem, method and guarantee (FqZk, Cghk). Section 3.1 defines the data, potential outcomes, treatment rule, benefit threshold κ\kappa and policy value before they are used, and states the two claims along a selected path: the entry region must benefit by more than κ\kappa, and each subsequently excluded shell must benefit by less. Entry alone makes no claim about people outside the entry region. Section 3.2 separates three meanings of resolution. Algorithm 1 (Section 4.1) lists every step and the data it may use, followed by a worked example, and Theorem 2 (Section 4.2) states the error event and its assumptions. Appendix A is typeset, and Appendix I.8 adds a notation table.
  2. Related work (FqZk, Cghk). The introduction discusses Senn (2018) and Hoogland et al. (2021). Section 2 covers finite-sample treatment choice, policy learning, sorted-group and rank-weighted effect summaries, controlled, optimal and Bayesian subgroup selection, safe policy improvement, winner inference, data carving and SNPL, and explains the different targets and assumptions.
  3. Claims narrowed (Cghk, wBVT). The 8-of-11 agreement is reported as five zero-resolution matches plus 3 of 6 positive-resolution sizes, and is labeled post hoc. We withdrew the attribution of the remaining error to finite-sample localization. The fixed-portfolio versus single-path contrast is relabeled post hoc, and Proposition 1 is described as an algebraic comparison of two budget proxies.
  4. New evidence, with its status. Table 1 lists each study and its status; Appendix I gives the details. It covers independent reference estimation on Criteo (I.1), reference gaps against observed recovery (I.2), a controlled check of the gap complexity (I.3), all 900-case contrasts (I.4), the full pipeline on the Hillstrom trial (I.5), matched comparisons on 6,000 synthetic cases and 300 full-pipeline Criteo runs (I.6), and Gaussian conditional-inference and data-carving comparisons (I.7).
  5. A correction. Checking Appendix D against the code showed that the submission described the prior-variance estimate in ECV incorrectly. Appendix D now matches the code that produced the results, and no original endpoint was recomputed.
  6. Framing and further work. The abstract and introduction now open with the difference between showing that a group benefits and justifying a restriction, and report the Criteo and Hillstrom results in numbers. The Discussion names three directions for further work.

What the paper now claims. It contributes a statistical formulation of how finely a trial supports changes of treatment action, an oracle analysis of localization and certification in a restricted Gaussian model, and an explicit procedure with its empirical record. The new comparisons do not establish a method advantage. None of the 20 combined utility-and-safety checks passes. At the largest tested Criteo size, n=262,144n=262{,}144, Bonferroni and Holm reuse have higher mean value than HonestER; at n=65,536n=65{,}536 they are slightly lower, while SinglePath has the highest mean. What the data do show is the distinction the paper is about. On Criteo, at the finest depth, the selected bin was certified in 94 of 100 runs but matched the empirical reference winner in only 40. On Hillstrom, 34 of 36 benefit-test records passed and none of 26 exclusion-test records did, and every method treats everyone.

Follow-up on the Hillstrom targeting decision (Section 5.5, Appendix I.5 and Table 12)

To show what the difference between finding a group that benefits and justifying a restriction costs in practice, we added a post hoc analysis on the original 12,800 audit customers, who were not used to form or test the groups. Among groups that passed the Holm comparator's benefit test, the group with the highest estimated average uplift contained 1,578 audit customers. At the study's assumed email cost, restricting email to that group would have lost an estimated 34.1 net visit-equivalents per 1,000 eligible customers relative to emailing everyone (pointwise 95% interval 22.3–45.9), about 72% of the value of emailing everyone. The loss is not specific to that group: across all 22 distinct passing groups, restricting email to one group lost 18% to 88% of that value (median 64%). Choosing the passing group with the highest estimated total net value on validation data led to emailing everyone.

A fixed mailing budget changes the comparison. At the original challenge's contact rate (10,000 of 64,000 customers, or 2,000 of the 12,800 audit customers), the same ranking gained 8.5 net visit-equivalents per 1,000 over random mailing at that volume (95% bootstrap interval 3.0–13.9). A ranking can show whom to email first, but it cannot say where mailing should stop; that is the question an exclusion claim answers. The three original methods still returned identical policies, and these contrasts reuse an existing trial, so they provide no independent confirmation of a method advantage.

Table 11 now also notes that six of the 26 exclusion records are empty shells created by tied score thresholds; the other 20 all have estimated effects of at least 0.0381.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.