- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies whether finite randomized data can support the granularity of recommendations produced by flexible treatment-effect learners. It introduces “evidence resolution” and proposes HonestER, which freezes representation choices before independently certifying refinements, with a backoff mechanism when evidence is insufficient. Theory and a Criteo experiment aim to distinguish certifying an action granularity, locating the best subgroup, and achieving treatment-policy value. While this distinction is interesting, the precise methodological contribution and the guarantees delivered by HonestER remain difficult to discern from the presentation.
Critical strengths
The paper starts from a very interesting idea: flexible data-adaptive learners of causal effects can support individualized recommendations at a granularity level that is much more detailed than what the actual data can support. This situation is subject of a current open debate in causal inference, with profound implications for personalized medicine, and has received attention and opinions from some of the most prominent statisticians such as F. Harrell and S. Senn (check [1] and [2]). It is thus a modern open problem with important methodological implications. I appreciate the attempt of this paper on touching such an important topic.
Critical weaknesses
- While the paper touches an important topic in methodological causal inference, it does little to position it in such context. Prior relevant work objecting to the extrapolation of flexible data-adaptive learners of causal effects into recommendations beyond the abstraction level provided by the data, like [1] and [2], are completely omitted. In fact, the introduction of the paper does little to position and motivate the problem at hand in the prior scientific landscape. No clear motivation is given on why the problem is important, what its implications are, and what has been positioned before. The introduction section has no references, which are relegated to section 2, where comparisons with other frameworks are given. Yet, such comparisons are given in a very indirect way, so it is not clear to the reader how they sit in the focused topic of “evidence support”. This is because a one-line explanation of each is provided, but not cohesive line linking them to the specific topic of this paper is missing.
- The writing seems rushed, unpolished and unfinished. There are big portions of the paper without a single bibliographic reference. Some paragraphs are unconnected, or it is hard to follow a cohesive chain-of-ideas linking them. For instance, the third paragraph in the introduction starts discussing benchmark results, but the objective, tasks, motivation and purpose have not been defined before.
- Mathematical objects are used without being properly defined. Section 3 starts by defining in terms of and but none of these objects are defined, therefore it is hard to understand what it is meant. In fact, many mathematical objects and tools used in section 3 are left implicit, undefined, uncommented or unmotivated. This particular section is written in a rushed way and there is no effort to be explicative or pedagogical for the reader. Even after reading it multiple times top-to-bottom, it is hard to evaluate the contribution, novelty, motivation and focus of such section.
- The mathematical problem is not clearly defined. What is the target? Is it a policy/recommendation rule built from a heterogeneous treatment effect (HTE) estimated with flexible data-adaptive learner (super-Learner). Why are these objects and objectives not defined in section 3? What is the statistical certification/guarantee produced, and where is it exactly defined?
- More worryingly, after reading the paper several times, it is still not clear to me what the exact methodological contribution of the proposed approach (HonestER). Algorithm 1 is not illustrative at all, and does not really clarify the methodological steps needed to produce the intended results. It is too coarse. I believe this is consequence of the problem not being clearly defined in term of concrete sub-problems and contributions to the literature.
- In summary, I acknowledge that the seed idea motivating the problem is interesting and I recommend the authors to keep working on it. Yet, the current version of the paper is not really ready for submission. It needs more polishing, motivation, literature review and dedicated step-by-step presentation of objectives, methods, and contributions.
References
[1] Hoogland, J., IntHout, J., Belias, M., Rovers, M. M., Riley, R. D., Harrell, F. E., Jr., Moons, K. G. M., Debray, T. P. A., & Reitsma, J. B. (2021). A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint. Statistics in Medicine, 40(26), 5961–5981
[2] Senn, S. (2018). Statistical pitfalls of personalized medicine. Nature, 563(7733), 619–621.
Questions for authors
-
What is the precise statistical problem and guarantee? Please define the target, the role of the learner and resulting policy or recommendation rule, and what “evidence support” means mathematically. What exactly does HonestER certify, under which assumptions, and where is this guarantee formally stated?
-
What is HonestER’s methodological contribution, and how does it work? Please provide a step-by-step explanation of its inputs, intermediate quantities, and outputs, including definitions of all mathematical objects needed. How do these steps deliver the stated guarantee, and what is new relative to existing methods? Algorithm 1 is currently too coarse to resolve these questions.
-
How does the paper connect to the existing debate on unsupported personalization? Please explain how HonestER addresses concerns that flexible treatment-effect learners may produce recommendations at a finer granularity than the data support, including those discussed in [1] and [2]. What specific gap does it fill, and how do the frameworks reviewed in Section 2 relate to that gap?
Please also feel free to address any of the topics and comments raised in the list of weaknesses.
4 comments
Q2. The method and what is new. Algorithm 1 (Section 4.1) begins with the three independent samples. It then lists candidate construction on the training data, the choice of one path and entry per family on the validation data, the quantities frozen before certification, the entry test, the sequential shell tests, the stopping rule and the final choice among families. A worked example follows it. Appendix D gives the ECV planning equations, the tie rules and the family allocation. While checking Appendix D against the historical code, we found that the submission described the prior-variance estimate incorrectly. The revised equation matches the code that produced the results, and no original endpoint was recomputed with a changed planner.
The introduction now ends with the paper's four contributions: the resolution target with its exclusion claims, the ordered-gap analysis for a restricted passive Gaussian model with its value-tolerant variant, explicit entry and exclusion claims with error control under valid component tests, and the two randomized-data studies. The paper also says that the fixed-sequence argument and the union bound are standard. The experiments examine these distinctions and show their limits. ECV is a planning heuristic, and the completed comparisons do not confirm a method advantage.
Q3. Relation to earlier work. Thank you for the references. The introduction now discusses Senn (2018) and Hoogland et al. (2021), and explains the difference between conditional average effects and individual response. Section 2 acknowledges finite-sample treatment-choice guarantees (Manski), Bayesian credible subgroups with simultaneous posterior bounds (Schnell et al.), controlled subgroup selection, safe improvement and inference after selection. Our target concerns group-average action changes on a specified path, with frequentist error control conditional on the training and validation choices. Section 2 now also contrasts the aggregate value guarantees of safe and confidence-aware policy learning with these local claims: a rule can meet an aggregate guarantee while withholding treatment from a group that benefits. It also covers sorted-group analysis and rank-weighted average treatment effects, which summarize effects along a ranking, and the minimax theory of optimal subgroup selection. We do not claim to introduce valid subgroup selection or full-sample policy learning. The sample-reuse comparison in Section 3.5 is limited to the two constructions it defines.
Presentation and AI assistance. We take the criticism of the writing seriously. We rewrote the sentence quoted in the review. Section 3 now defines the data, policy value, entry, exclusion, validity and resolution targets before presenting the results. Appendix I.8 adds a notation table, Appendix A is typeset, and Appendix G keeps the frozen protocols with readable equations and wrapped code blocks. AI assistance was disclosed on the submission form, and the revised AI use statement describes its uses. We remain responsible for every claim, proof, citation and line of code, and for the final text.
I thank the authors for taking my comments seriously. The changes to the exposition do bring clarity. In particular, the opening example makes it much easier to see why evidence of benefit in one subgroup leaves open whether treatment should be withdrawn from everyone else. I also found it easier to follow what HonestER tests and when it stops.
Two concerns remain for me. First, although the procedure is clearer, I still find it difficult to see how much of the methodological contribution lies in the choice of claims to test and how much lies in the testing procedure itself. Second, the Hillstrom example illustrates why the distinction matters, but all three methods ultimately recommended emailing everyone. I would have liked to see a case where the methods recommend different rules and to understand what that difference means for policy value.
Taking everything together, I think the revision merits an increase of one rank in my score. These remaining concerns keep me from going any furhter.
Thank you for re-reading the revision. Your question about where the contribution lies helped us make that distinction more explicit.
Claims and procedure. The main methodological contribution is specifying what a treatment rule must establish at each step. To start treating a group, its average benefit must exceed the stated threshold. To narrow that rule, the average benefit in the group that would lose treatment must fall below the threshold. Within each family, HonestER tests the claims in order on data held out from planning and stops that family at its first failed test. The statistical tools are standard: Theorem 2 applies fixed-sequence testing within each family and a union bound across families. Theorem 1 addresses a separate question: how much data any selector needs to identify the best group in the balanced Gaussian model of Section 3.4.
Hillstrom illustrates why the choice of claims matters. Twenty-two distinct smaller groups passed a benefit test. Emailing only the passing group with the highest estimated benefit would have lost an estimated 34.1 net visit-equivalents per 1,000 eligible customers relative to emailing everyone. All three procedures recommended emailing everyone. Narrowing that recommendation required evidence that the customers left out benefited too little to justify the email cost. None of the proposed exclusions met that condition.
When the procedures chose different rules. On Criteo, we held the candidate groups, required claims, and component tests fixed. At n = 262,144, HonestER and single-path selection both certified a rule in 69 of 100 paired runs and chose different rules in 14 of them. In paired run 10, HonestER targeted 25.1% of the evaluation customers and single-path selection targeted 14.0%. Their estimated net values were 5.33 and 6.87 visit-equivalents per 1,000 eligible customers, respectively. Across these 14 runs, HonestER's rule had the higher estimated value in 10.
Most of the difference in average value came from how often each procedure certified any rule: Bonferroni reuse in 98 of 100 runs, HonestER in 86, and single-path selection in 71. Among runs in which each procedure certified a rule, its mean estimated value was about 6.6 per 1,000. Runs with no certified rule count as zero, giving overall means of 6.48, 5.68 and 4.69, respectively. Bonferroni reuse exceeded HonestER by 0.80 (paired 95% interval 0.39–1.21), and HonestER exceeded single-path selection by 0.99 (0.47–1.52). At n = 65,536, single-path selection certified a rule in 11 runs and HonestER in 2, and single-path selection had the highest mean value. These intervals hold the shared evaluation sample fixed. In these runs, the procedure mainly governed how often a trial supported a personalized rule; the claims governed what that rule means. Section 5.4 and Appendix I.6 report the comparison and its limits.
Thank you for identifying where the submitted paper was hard to evaluate. We now introduce the treatment decision before the notation. The example after Algorithm 1 follows a change from treating the top half to treating the top quarter. HonestER is now presented as a reference procedure with a stated error guarantee, and the paper makes no claim that it outperforms other valid certifiers.
Q1. The statistical problem and the guarantee. Section 3.1 starts from randomized trial units (X,A,Y) with potential outcomes, consistency, no interference and a known assignment probability. A policy π decides who is treated, and its value relative to treating nobody is Vκ(π)=E[π(X){τ(X)−κ}], where the threshold κ is fixed before any certification outcome is read. Candidate rules come from nested regions Rb,1⊇Rb,2⊇⋯, and two kinds of claim are needed. Entering region Rb,eb requires its mean effect to exceed κ (Equation 3). Moving from depth d to d+1 removes the shell Sb,d=Rb,d∖Rb,d+1 and requires the shell's mean effect to be below κ, which is the condition for the refinement to increase value (Equation 4). Section 3.2 then separates three quantities that the submission ran together: the supported resolution of a fixed representation in the population, the resolution a procedure reports, and the minimax evidence resolution, which is an information limit.
Theorem 2 (Section 4.2) states the guarantee. Conditional on the training and validation data, each family runs a fixed sequence of tests and stops at its first non-rejection. If every true-null component test rejects with probability at most its allocated level, the probability of any false entry or exclusion certificate is at most α, and the same bound covers returning a policy whose required path contains a false claim. The guarantee concerns region averages along the selected path. It makes no claim about people outside the initial entry, benefit to every treated person, optimal value or recovery of the finest valid rule. A whole-population entry followed by supported exclusions also justifies comparison with treating everyone. The historical Welch–Normal tests are approximate and can be anticonservative in sparse binary cells; the exact error bound requires valid components. Appendix C gives a finite-sample alternative for bounded outcomes, with its assumptions.