acceptodds
Withdrawn from ICLR 2027

When Honest Evaluation Fails to Compose: Learner-Protected Targeting Curves

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating each learned treatment rule on fresh data does not ensure valid inference when the evaluations are combined. Earlier evaluation noise can influence later model selection and change the variance of later errors, invalidating a bootstrap that freezes the learned path. A two-candidate characterization identifies when this failure persists and when frozen inference remains valid. We construct asymptotically valid confidence bounds for the value of the realized rules, simultaneous over treatment budgets and version weights. Versionwise bands combine rectangularly even with overlapping evaluation samples. For disjoint evaluations with Gaussian limits, studentization gives radial bounds with a sharp common critical value at each budget. A matched four-route experiment separates the benefits of data reuse from those of calibration geometry: using all later cohorts can improve selected utility, while radial bounds improve precision for balanced mixtures. A matched selection–decision simulation shows a concentrated benefit of protection. At one predeclared cost, it leaves 99.6% of decisions unchanged and improves average utility by preventing harmful deployment in the remaining 0.4%; at a lower cost, it forgoes useful treatment. On Hillstrom, reuse increases the certified net value, while exact-budget targeting advantages remain uncertified. A finite-batch correction supports deployment with integer treatment quotas.

Primary area:
causal reasoning
Keywords:
causal inference, policy evaluation, sequential validation, treatment prioritization, simultaneous inference, Qini curves, post-selection inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MXxuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the failure of statistical inference when individually honest evaluations of treatment rules are combined through adaptive model selection. It characterizes conditions under which conventional frozen-path inference becomes invalid and proposes rectangular and radial confidence bounds that provide simultaneous protection over treatment budgets and model-version weights. Theoretical results establish coverage guarantees, while simulations and real-world applications examine the benefits and costs of the proposed methods.

Critical strengths
  1. Well-defined and meaningful problem. The paper identifies an important limitation of sequential honest evaluation: individually valid evaluations may lead to invalid inference when adaptively combined.

  2. Strong theoretical framework and careful validation. The paper establishes simultaneous confidence bounds through rectangular and radial constructions, with clearly stated assumptions. The experiments are thoughtfully designed to examine both statistical validity and deployment utility, including cases where additional protection is not beneficial.

Critical weaknesses
  1. Unclear practical significance. Although the paper establishes persistent undercoverage under adaptive evaluation, the observed deviations from nominal coverage are relatively modest in the reported examples. It remains unclear how frequently this issue becomes consequential in realistic model-selection pipelines.

  2. Limited practical advantages over simpler alternatives. The proposed protection can be conservative and does not consistently improve deployment utility. While the paper compares different evaluation strategies, the conditions under which the proposed methods should be preferred over a simpler independent holdout evaluation remain insufficiently characterized.

  3. Lack of an intuitive illustration of the core mechanism. The paper lacks a conceptual figure explaining how feedback between evaluation noise and model selection invalidates inference, and how the proposed methods address this issue. Such a figure would improve clarity and readability.

Questions for authors
  1. Practical significance: Could the authors clarify which characteristics of a sequential model-selection process determine the severity of undercoverage?

  2. Method selection: Based on the existing theoretical and empirical results, when should practitioners prefer rectangular or radial protection over a conventional final holdout evaluation, considering both coverage validity and deployment utility?

Reviewer_eXWsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a failure mode in sequential “honest” evaluation of learned treatment rules. Although each model version is evaluated on fresh data, earlier evaluation noise can affect later model selection, which in turn changes the distribution and variance of subsequent evaluation errors. The paper characterizes when frozen inference can fail and develops simultaneous confidence bounds over treatment budgets and version mixtures, including rectangular bounds for potentially overlapping evaluations and radial bounds for disjoint evaluations after studentization.

Critical strengths
  • The paper highlights that fresh-data evaluation at each stage does not automatically guarantee valid inference after adaptively combining multiple evaluations. The two-candidate analysis provides a clean characterization of when the resulting coverage failure persists.
  • The paper develops both rectangular and radial protection schemes with formal coverage guarantees, and the radial construction gives a sharper geometry under disjoint evaluations and Gaussian limits.
Critical weaknesses
  • The controlled examples demonstrate that frozen inference can undercover, but the magnitude of the distortion is relatively modest in the reported settings. For example, the unequal-noise simulation reports approximately 94.03% coverage versus 95.19% under an independent replay. It remains unclear how frequently such feedback produces practically consequential errors in realistic ML deployment pipelines.
  • In the Hillstrom experiment, all evaluated procedures ultimately select treat-all, while the exact-budget Qini lower bounds remain nonpositive. Thus, the real-data experiment does not clearly demonstrate a setting where the proposed protection materially changes the resulting policy or decision.
Questions for authors
  1. In what realistic sequential ML or policy-learning pipelines do the authors expect the identified feedback effect to produce materially important undercoverage? Could the authors provide a broader characterization of the regimes in which the failure becomes practically significant?

  2. Beyond the controlled matched-deployment simulation, can the authors provide or discuss a real-data setting in which the proposed protection would materially change the selected policy or prevent a consequential deployment error, rather than primarily changing the certified lower bound?

Reviewer_4UQJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work considers the setting where the training and evaluation data arrives in a sequence, and generates a sequence of trained models, and derives confidence bands for the performance (specifically, gains at population fraction qq) of a mixture of learned treatment rules (or models). In this scenario, a naive analysis that derives confidence bands for the mixture can fail if it does not consider the correlation between model evaluations, unlike the bands derived in this work which take the dependence into account. Firstly, a rectangular band is derived, which combines confidence intervals for the gain of each model to produce a confidence band for the gain of the overall mixture. Secondly, a lower confidence band is derived by standardising the gain estimates to remove the effect of scale. Experiments are conducted on a synthetic dataset and Hillstrom's randomized email data.

Critical strengths

This work analyses how to create confidence bands for evaluations of mixtures of learned treatment rules when the model evaluations are statistically dependent, which is an interesting aspect of the problem.

Critical weaknesses

The clarity of the manuscript is poor, making it difficult for the reader to extract even the core ideas presented in the paper. The goal and main contributions of the paper are not explicitly or formally stated, making it difficult to assess the overall contribution. The literature review is also brief and does not properly position the paper, making it hard to assess its novelty and contribution. For example, it does not answer which papers (if any) analyse the treatment effects in a setting where the data arrives in a sequence, and if confidence bands of the form derived in the paper have been derived in prior work.

There is no dedicated section defining and discussing the problem statement. However, most of the setup is done in Section 3.1, which suffers from missing definitions, ambiguous statements, and dense technical comments. The flaws in this section, enumerated below, subsequently make the remainder of the work hard to follow.

  • Section 3.1 starts with "We observe i.i.d. data (X,A,Y)(X,A,Y)...", yet none of the symbols X,AX,A or YY are defined or named. Indeed, the symbol AA does not appear in the text again.
  • Line 114 states "Order the sample into KK folds. Before opening fold IkI_k, fit a score sks_k using the history Hk−1H_{k-1}...". Here, 3 symbols and quantities Ik,sk,Hk−1I_k, s_k, H_{k-1} are simultaneously introduced in this short line, without further discussion or context. The definition of IkI_k is given only implicitly. The relationship between the history and the folds is also not made explicit.
  • When defining Gk(q)G_k(q), it is never explicitly named as the "gain", although this point becomes clear from later discussions.
  • Line 124 "we omit the index nn" : the symbol nn is never defined in the main text.
  • Line 118 defines Gk(q)=Eπsk,q(X,U)τ(X)∣Hk−1G_k(q) = E \\{\pi_{s_k,q}(X,U) \tau(X) | H_{k-1}\\} . However the symbol UU has not been defined and πsk,q\pi_{s_k, q} is only briefly defined in words, and its domain and codomain are not defined. It is also not clear which distribution defines the expectation.
  • Line 125 states "Write G^k(q)\hat{G}_k(q) for a gain estimated from data that were unopened when version kk was learned" which is ambiguous. Does "data that were unopened" mean all of the data after fold k−1k-1, or is it a subset of that data? Moreover, how is G^k(q)\hat{G}_k(q) estimated from the data, i.e. what is its specific form, or is the paper agnostic to the type of the estimator?

Similar issues with clarity and readability persist throughout the paper. Indeed, the concepts from the title itself do not appear again, i.e. "learner-protected" does not appear in the text again. The central idea of the paper and the first half of the title "honest evaluation(s) fail to compose" is never revisited in the text with this phrasing. Apart from the issues in the problem definition (Section 3.1), the following are other issues in the paper that affect the clarity and readability.

  • Line 174 defines the set of versions as SS of size dd. This is inconsistent with the notation setup earlier in section 3.1, which denotes K−1K-1 models (or scores) instead.
  • Assumptions for Theorem 3 (A1-A4) are referred to without explicitly defining them in the main text, or providing a reference to the appendix where they are defined. There is also no discussion on the assumptions, leaving several questions open, such as the stringency of the assumptions. Indeed, A1-A4 are not even named as assumptions when they are first used in Line 168 (which simply states "under A1-A4") further confusing the reader.
  • Algorithm 1 is presented without discussion, except a brief description on line 355 ("Algorithm 1 combines evaluation, selection and deployment").
  • Mathematical symbols and quantities are used directly with little or no definition, and almost never motivated. The following are some examples, that do not include the examples from Section 3.1 discussed earlier.
    • Line 95 : "the scores XX and −X-X" : these symbols and quantities have not been defined.
    • The symbol η^k\hat{\eta}_k is used in Algorithm 1 but is nowhere defined.
    • Line 137 : ∑kαk≤α\sum_k \alpha_k \leq \alpha : The variable α\alpha is used without definition.
    • Line 176 uses the symbols βq\beta_q which have not been defined yet (are they positive numbers? are they members of [0,1][0,1]?)
    • Line 79-82 : "ZZ is the first gain error, V+hV+h is the difference used to select the candidate,... The population difference hh...". VV is not explicitly defined here. Moreover, interpreting ZZ is difficult at this point, since "gain" is defined only in the next section, and the text also does not specify the kind of error referred to in "first gain error" (presumably the sample error).
    • Line 168 : The symbol se^k(q)\hat{\text{se}}_k(q) is assigned to the "conditional standard error" of G^k(q)\hat{G}_k(q) but it is not explicitly defined.
    • Line 148 states "The same result holds on a prespecified simplex face", but "simplex face" was not formally defined. Similarly, Theorem 3 defines Λ(S)\Lambda(S) as the "simplex supported on SS" which is not defined.
    • Line 175 introduces independent standard normal variables ZjZ_j but does not explain what they represent.
  • Technical terminology is introduced implicitly and used throughout the paper. The following are some examples.
    • The term "frozen" or "freeze" is used many times in the text without an explicit definition, e.g. line 13 ("freezes the learned path"), line 15 ("frozen inference"), despite being a central concept.
    • The concept of "protection" is used several times in the paper without proper definition, e.g. title ("learner-protected"), line 22 ("concentrated benefit of protection"), line 56 ("protection improves mean utility"), line 63 ("... tests protect evidence collected in stages"), line 102 ("...uses these constructions to protect subsequent choices").
    • The concept "utility" is used without definition, e.g. line 55 "utility gains or losses", line 232 ("maximises its lower utility"), line 232 ("Safe UU is the selected population utility"), line 237 ("safe utility"), line 250 ("the suffix route's utility advantage"), line 20 ("using all later cohorts can improve selected utility") etc.
    • The term "certification" is used several times without a formal definition, e.g., line 25 ("certified net value"), line 26 ("exact-budget targeting advantages remain uncertified").
    • The "value" of learned treatment rules is referred to at several points (line 16, 45, 217, 278). This presumably refers to the "gain" Gk(q)G_k(q), which is an inconsistent use of terminology.
    • Line 102 refers to "dividing each error by its conditional scale", but conditional scale is not formally defined.
    • line 123 states that "λk\lambda_k is the fraction of future recipients assigned to version kk", but "future recipients" is not formally defined (again, relating to the ambiguities in the problem definition).
    • Other examples of terminology that would benefit from a definition include "calibration geometry" (line 21), "selection-decision simulation" (line 22), "a matched deployment experiment" (line 54), "exact-budget targeting" (line 26), "paired trials" (line 233).
  • More generally, informal, technical, or ambiguous language hinders the readability of the paper at several points, some of which are enumerated below.
    • Line 81 states that "The 'frozen' analysis holds the selected models fixed when resampling and treats their pooled, studentized error as standard normal". While the failure of the "frozen" analysis (note the use of quotations by the authors) is the core motivating example for this work, it is not formally discussed anywhere in the main text. Moreover, in this statement, it is not clear what is being resampled, what it means for the models to be fixed, or how the models are pooled.
    • line 21 : "balanced mixtures" refers to mixtures of the model, but is ambiguous (how are they balanced?).
    • line 38 : "As new data arrive, the team updates the model... noise in the first evaluation helps choose the next model, and that model determines how variable the next evaluation will be." This paragraph informally talks about the failure mode of sequential evaluation, but at this point it has not yet been established if the data arrives in batches, what exactly is the sequence of evaluations (on which data?), how is the variability of the evaluations measured, and indeed why "that model" determines the "next evaluation".
    • Line 49 : "This removes the random scale that causes the failure above and leads to an exact calculation of the worst error over version weights". The "random scale" and "version weights" are not defined. The "failure above" is also ambiguous.
    • The paper interchangeably refers to the model "versions" (line 48) or "candidates" (line 37).
    • line 74 : "(the team) sends the better-looking one to a second, unopened cohort". "Cohort" has not been explicitly defined (also used in line 20, line 53), and the phrase "better-looking one" is ambiguous and informal, in particular, it is not clear what metric is being considered here.
    • Line 73 : "Consider two candidates with similar true gains at the same treatment budget", neither gains nor the treatment budget have been formally defined at this point in the paper.
    • line 52 : "Protection widens bounds and can prevent a useful rule from being certified" : It is not clear what protection is, how it widens the bounds, and what kind of certification is being referred to.
    • Line 137 : "allocate ∑kαk≤α\sum_k \alpha_k \leq \alpha on every path" : What does "path" mean here?
    • Theorem 2 : "the conditional band guarantees above" presumably refers to the coverage of the Gk(q)G_k(q) by [Lk(q),Uk(q)][L_k(q), U_k(q)] but is not formally stated as an assumption.
    • Line 11 in Algorithm 1 refers to Equation 81 in the appendix, hurting the self-containedness.
    • Line 194 : "The finite-grid error allocation can still be conservative; sharpness is for this radial family at one budget", What is the "finite-grid error allocation" and why is it conservative? Sharpness refers to which bound here?
    • Line 93 : "feedback matters when the selection noise is correlated with the earlier value error" - the selection noise has not been defined.

I also have the following concerns related to the main theoretical and empirical results :

  • lim infn\text{lim inf}_n appears in both Theorem 2 and Theorem 3, but none of the defined quantities appear to have a dependence on nn, indeed nn has not been explicitly defined anywhere.
  • The example presented in Proposition 1, which is used to motivate this work, features a simple setup limited to only two versions instead of a more general case with KK versions.
  • Theorem 3 only gives a lower confidence band for the mixture gains Gλ(q)G^{\lambda}(q) , instead of a both-sided band.
  • The construction of the lower bound defined by (6) is not interpreted, nor is it entirely clear how it was arrived at.
  • Table 1 does not include a baseline to compare the "protected" procedures for computing the confidence bands.
  • Table 1 should report the how tight the bounds are, e.g. the average width of the rectangular bounds. Clearly, very loose bounds will have very high coverage.
Questions for authors
  • Why is the total error for the two candidate example equal to Z+σ(V+h)WZ+\sigma(V+h)W? Why does the error from the first evaluation enter the analysis, since only the final/second evaluation matters and the first evaluation is only used for model selection?
  • Why is Proposition 1 significant to the rest of the work? Why is it favourable for the frozen analysis if the standardised error UhU_h is normally distributed, or approximately normally distributed (as in (2))?
  • Line 133 : "The same averaging argument holds for every mixture on that event, so choosing the weights afterwards cannot invalidate it." Firstly, "mixture" has not been formally defined, but I presume this refers to Gλ(q)G^{\lambda}(q). What does it mean here to choose the weights "afterwards" (after what?) and why should this affect (or not affect) the argument?
  • What is the advantage or disadvantage of the radial bound over the rectangular bound, and/or vice-versa?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.