The clarity of the manuscript is poor, making it difficult for the reader to extract even the core ideas presented in the paper. The goal and main contributions of the paper are not explicitly or formally stated, making it difficult to assess the overall contribution. The literature review is also brief and does not properly position the paper, making it hard to assess its novelty and contribution. For example, it does not answer which papers (if any) analyse the treatment effects in a setting where the data arrives in a sequence, and if confidence bands of the form derived in the paper have been derived in prior work.
There is no dedicated section defining and discussing the problem statement. However, most of the setup is done in Section 3.1, which suffers from missing definitions, ambiguous statements, and dense technical comments. The flaws in this section, enumerated below, subsequently make the remainder of the work hard to follow.
- Section 3.1 starts with "We observe i.i.d. data (X,A,Y)...", yet none of the symbols X,A or Y are defined or named. Indeed, the symbol A does not appear in the text again.
- Line 114 states "Order the sample into K folds. Before opening fold Ik, fit a score sk using the history Hk−1...". Here, 3 symbols and quantities Ik,sk,Hk−1 are simultaneously introduced in this short line, without further discussion or context. The definition of Ik is given only implicitly. The relationship between the history and the folds is also not made explicit.
- When defining Gk(q), it is never explicitly named as the "gain", although this point becomes clear from later discussions.
- Line 124 "we omit the index n" : the symbol n is never defined in the main text.
- Line 118 defines Gk(q)=Eπsk,q(X,U)τ(X)∣Hk−1 . However the symbol U has not been defined and πsk,q is only briefly defined in words, and its domain and codomain are not defined. It is also not clear which distribution defines the expectation.
- Line 125 states "Write G^k(q) for a gain estimated from data that were unopened when version k was learned" which is ambiguous. Does "data that were unopened" mean all of the data after fold k−1, or is it a subset of that data? Moreover, how is G^k(q) estimated from the data, i.e. what is its specific form, or is the paper agnostic to the type of the estimator?
Similar issues with clarity and readability persist throughout the paper. Indeed, the concepts from the title itself do not appear again, i.e. "learner-protected" does not appear in the text again. The central idea of the paper and the first half of the title "honest evaluation(s) fail to compose" is never revisited in the text with this phrasing. Apart from the issues in the problem definition (Section 3.1), the following are other issues in the paper that affect the clarity and readability.
- Line 174 defines the set of versions as S of size d. This is inconsistent with the notation setup earlier in section 3.1, which denotes K−1 models (or scores) instead.
- Assumptions for Theorem 3 (A1-A4) are referred to without explicitly defining them in the main text, or providing a reference to the appendix where they are defined. There is also no discussion on the assumptions, leaving several questions open, such as the stringency of the assumptions. Indeed, A1-A4 are not even named as assumptions when they are first used in Line 168 (which simply states "under A1-A4") further confusing the reader.
- Algorithm 1 is presented without discussion, except a brief description on line 355 ("Algorithm 1 combines evaluation, selection and deployment").
- Mathematical symbols and quantities are used directly with little or no definition, and almost never motivated. The following are some examples, that do not include the examples from Section 3.1 discussed earlier.
- Line 95 : "the scores X and −X" : these symbols and quantities have not been defined.
- The symbol η^k is used in Algorithm 1 but is nowhere defined.
- Line 137 : ∑kαk≤α : The variable α is used without definition.
- Line 176 uses the symbols βq which have not been defined yet (are they positive numbers? are they members of [0,1]?)
- Line 79-82 : "Z is the first gain error, V+h is the difference used to select the candidate,... The population difference h...". V is not explicitly defined here. Moreover, interpreting Z is difficult at this point, since "gain" is defined only in the next section, and the text also does not specify the kind of error referred to in "first gain error" (presumably the sample error).
- Line 168 : The symbol se^k(q) is assigned to the "conditional standard error" of G^k(q) but it is not explicitly defined.
- Line 148 states "The same result holds on a prespecified simplex face", but "simplex face" was not formally defined. Similarly, Theorem 3 defines Λ(S) as the "simplex supported on S" which is not defined.
- Line 175 introduces independent standard normal variables Zj but does not explain what they represent.
- Technical terminology is introduced implicitly and used throughout the paper. The following are some examples.
- The term "frozen" or "freeze" is used many times in the text without an explicit definition, e.g. line 13 ("freezes the learned path"), line 15 ("frozen inference"), despite being a central concept.
- The concept of "protection" is used several times in the paper without proper definition, e.g. title ("learner-protected"), line 22 ("concentrated benefit of protection"), line 56 ("protection improves mean utility"), line 63 ("... tests protect evidence collected in stages"), line 102 ("...uses these constructions to protect subsequent choices").
- The concept "utility" is used without definition, e.g. line 55 "utility gains or losses", line 232 ("maximises its lower utility"), line 232 ("Safe U is the selected population utility"), line 237 ("safe utility"), line 250 ("the suffix route's utility advantage"), line 20 ("using all later cohorts can improve selected utility") etc.
- The term "certification" is used several times without a formal definition, e.g., line 25 ("certified net value"), line 26 ("exact-budget targeting advantages remain uncertified").
- The "value" of learned treatment rules is referred to at several points (line 16, 45, 217, 278). This presumably refers to the "gain" Gk(q), which is an inconsistent use of terminology.
- Line 102 refers to "dividing each error by its conditional scale", but conditional scale is not formally defined.
- line 123 states that "λk is the fraction of future recipients assigned to version k", but "future recipients" is not formally defined (again, relating to the ambiguities in the problem definition).
- Other examples of terminology that would benefit from a definition include "calibration geometry" (line 21), "selection-decision simulation" (line 22), "a matched deployment experiment" (line 54), "exact-budget targeting" (line 26), "paired trials" (line 233).
- More generally, informal, technical, or ambiguous language hinders the readability of the paper at several points, some of which are enumerated below.
- Line 81 states that "The 'frozen' analysis holds the selected models fixed when resampling and treats their pooled, studentized error as standard normal". While the failure of the "frozen" analysis (note the use of quotations by the authors) is the core motivating example for this work, it is not formally discussed anywhere in the main text. Moreover, in this statement, it is not clear what is being resampled, what it means for the models to be fixed, or how the models are pooled.
- line 21 : "balanced mixtures" refers to mixtures of the model, but is ambiguous (how are they balanced?).
- line 38 : "As new data arrive, the team updates the model... noise in the first evaluation helps choose the next model, and that model determines how variable the next evaluation will be." This paragraph informally talks about the failure mode of sequential evaluation, but at this point it has not yet been established if the data arrives in batches, what exactly is the sequence of evaluations (on which data?), how is the variability of the evaluations measured, and indeed why "that model" determines the "next evaluation".
- Line 49 : "This removes the random scale that causes the failure above and leads to an exact calculation of the worst error over version weights". The "random scale" and "version weights" are not defined. The "failure above" is also ambiguous.
- The paper interchangeably refers to the model "versions" (line 48) or "candidates" (line 37).
- line 74 : "(the team) sends the better-looking one to a second, unopened cohort". "Cohort" has not been explicitly defined (also used in line 20, line 53), and the phrase "better-looking one" is ambiguous and informal, in particular, it is not clear what metric is being considered here.
- Line 73 : "Consider two candidates with similar true gains at the same treatment budget", neither gains nor the treatment budget have been formally defined at this point in the paper.
- line 52 : "Protection widens bounds and can prevent a useful rule from being certified" : It is not clear what protection is, how it widens the bounds, and what kind of certification is being referred to.
- Line 137 : "allocate ∑kαk≤α on every path" : What does "path" mean here?
- Theorem 2 : "the conditional band guarantees above" presumably refers to the coverage of the Gk(q) by [Lk(q),Uk(q)] but is not formally stated as an assumption.
- Line 11 in Algorithm 1 refers to Equation 81 in the appendix, hurting the self-containedness.
- Line 194 : "The finite-grid error allocation can still be conservative; sharpness is for this radial family at one budget", What is the "finite-grid error allocation" and why is it conservative? Sharpness refers to which bound here?
- Line 93 : "feedback matters when the selection noise is correlated with the earlier value error" - the selection noise has not been defined.
I also have the following concerns related to the main theoretical and empirical results :
- lim infn appears in both Theorem 2 and Theorem 3, but none of the defined quantities appear to have a dependence on n, indeed n has not been explicitly defined anywhere.
- The example presented in Proposition 1, which is used to motivate this work, features a simple setup limited to only two versions instead of a more general case with K versions.
- Theorem 3 only gives a lower confidence band for the mixture gains Gλ(q) , instead of a both-sided band.
- The construction of the lower bound defined by (6) is not interpreted, nor is it entirely clear how it was arrived at.
- Table 1 does not include a baseline to compare the "protected" procedures for computing the confidence bands.
- Table 1 should report the how tight the bounds are, e.g. the average width of the rectangular bounds. Clearly, very loose bounds will have very high coverage.