Since this section is long, I will give a summary and my overall opinion before it. My first, and most important criticism, is to do with writing of the paper, which is very very unclear. My other two critiques are more technical. The first is to do with evaluation, for which I point out that an accounting of the distribution of (ηH,ηA) studied is crucial to get pertinence. Finally, I talk a bit about the fact that the underlying hypothesis depends not only on the data (which is perhaps okay), but also on the pre-described cell paritioning of each region, which is a bit excessive.
My overall recommendation below, however, is mostly to do with the writing. This is so unclear and obfuscated that it is very hard, in my opinion, to usefully read this paper. The exposition is clearly well below the standards of any reasonable venue, and naturally, fixing this would require rewriting large parts of the paper, which cannot be accommodated into a conference review cycle. Thus, in my opinion, the paper is simply not ready, and needs more expository work before it's technical merits can even come to be considered.
On the writing. The writing of this paper is abysmal. Throughout, terminology is used without being defined, symbols appear out of nowhere, and language is needlessly dense and technical. One is left equal parts reeling, and trying to piece together scattered clues to figure out just what on earth the authors mean.
In fact, the writing is so poor that it is not even made clear that the main technical contribution is a new test for affineness. Instead, all this is dressed up amidst trivialities to do with Bonferroni corrections. I can hardly understand why an interesting contribution like that would be suppressed.
Of course, the main problem is not style, it is clarity. As a particularly striking example of the lack of clarity, even the the "problem setting" section is unclear! I have been reviewing ML conference papers since 2018, and reading for even longer. I have also written and reviewed statistics papers before. This is the first paper I have come across where unintroduced terminology and notation is used in the section whose typical purpose is to introduce terminology and notation!! How can this even happen?
All of this continues throughout the paper. The most egregious, perhaps, is the experiments section. Here, the baselines are not clearly specified, and their choice is not described. The actual details of experimental instances (how exactly the regression function μ is picked) is not described, nor are we told how (ηH,ηA) varied (or if they even are varied, rather than varying the thresholds). Further, trying to figure these things out in the appendix is even not easy. It took me a cool 20 minutes of scanning through pages upon pages before I found a definition of "NR" in a discussion in section E.9, and even this is stated inline. In any case, given the quality of the writing, I exercised my prerogative to not read the appendix to the largest extent my conscience would allow me.
Please please please note that it is NOT a readers job to intuit meaning and decipher clues. It is the authors' job to make sure that these things are clearly stated.
Let me give an example to justify this opinion, and why not just the problem setting section above.
-
152-161: The very first paragraph already leaves me lost.
- I don't understand what the pre-response design information is. Laws of the covariates? Something else?
- What are response-free design screens, and what are power screens?
- What are these prespecified certification rules, what are "requested coefficients"
- What does it mean for something to "freeze independently of audit responses and scale measurements?" While I am at it, I guess audit responses are the YR, but what are scale measurements?
- What are transductive partitions? What are stable ID maps?
It feels here that certain basic notations and terminology was just not set up. This immediately makes everything following confusing: now we are building upon this terminology that I don't know. So now I get the singular displeasure of trying to ferret out the meaning of all these terms from various context clues in the text. Why do this to your readers? Please be kinder to us.
Anyway, my understanding is as follows, and I will judge the correctness on the basis of this. If my understanding is wrong, well, not my fault that things were not explained well enough.
- there is a prespecified partition, as captured by r, and audit data is assigned to these, for some reason calling these slots. These slots may or may not be audited, depending on certain somewhat mysterious criteria---this is the so-called screen. All this filtering can be done without seeing YR, and this value is used only for slots which pass. "Releasing" is something that the auditor will do, whereby a linear fit over this region (on the basis only of the audit data, or some other linear fit given a priori?) will be made public. The problem is to decide what to release.
- Even this basic thing: that the auditor 'releases' slots, is not stated. Not to mention the various other terms used for this in the paper, again without telling the reader. How am I supposed to dream up what "gating" means, for instance?
-
162-168: I don't know what raw units are. It also seems needlessly complicated to set up things in terms of X and Z in this way, especially since this is basically never used in the main text...
-
Lines 169-183:
- I don't know what a completed discovery object is. SC,SI are used before they are defined. The term "pure-error bank" is used as if it has already been defined (cue me searching the previous text and getting nada). Yb,r+ and Yb,r− are never defined.
- What does it mean to "open after something freezes"? What does it mean to require independence and not just delayed opening? I have never come across "opening" in this context (also freezing might be overused).
- This is hardly complicated stuff, btw. If you tell someone what is going on here in straightforward terms, this paragraph is a triviality to someone with statistical training. Consider this explanation: our procedure extends to unknown noise variance given pure-error samples. We thus assume access to mC+mI samples drawn from the noise distribution. These may be realised by observing this many paired responses Y+,Y− with matching means. (If space is available, one should add a comment about this being standard in F-tests here). This results in a sample variance estimates SC,SI thus (3). These two variances are independent, which our analysis exploits, and used in different parts of the procedure: SC to certify slots, and SI to construct intervals. We let F0 denote the 'response-free' sigma-algebra over all aspects of the data and setting above, including covariates, router, slots et c., barring (Y,SC,SI).
On evaluation, and the comparison to NR. As mentioned above, in my opinion the key contribution is the affineness test of section 4.2. It is clear that this can be strung together with corrections into a procedure over the whole domain. The most natural question I had when turning to the experiments, thus, was how does this compare with using the standard F-test, aka NR in the paper.
While I mentioned the lack of clarity in the experimental section above, I have comments about the overall impression conveyed by this section, namely that NR is more conservative. I think that in order for me to accept this, the experiment design has to be particularly robust. The main issue is this: consider first N→∞, and imagine the performance of these two tests (the proposed C and NR) in the 2D space spanned by choices of (ηH,ηA) as considered in the paper. Naturally, sets where the acceptance (or "release" or "report" or whatever) rate goes to 1 in this space are boxes for C and quarter-discs for NR. But note that this immediately means that if NR is tuned to reject all cases wherein either one of (ηH,ηA) is larger than some threshold, it would leave a corner of the box uncovered (the disc would have to be tangent to the box, assuming even that the thresholds for the two are the same). In general, one could deviate from this somewhat, but nevertheless to get roughly similar size control over a certain region, NR would end up rejecting close to a corner point of a box where C accepts (I believe Prop 34 is essentially making this point). Of course, conversely, if the goal was to test if the residual energy relative to the affine fit were large, then the C test would have a mismatched acceptance region. In finite samples, this limiting behaviour manifests in two ways: both this high/low behaviour itself, and also in the shape of the rate curve as one explores various directions in the (ηH,ηA) space (so, how far below 1 you are may depend on (ηH,ηA) nonuniformly).
If we consider what we want in practice, neither of these hypotheses (box constraints on (ηH,ηA) or ellipsoidal constraints) quite captures things in general: it depends on the application. I think, thus, to make a convincing case, one would either identify situations in which this box type thresholds on these two deviations are intrinsically preferable (this would carve out at least a niche where you should ideally use C), or one would evaluate at (ηH,ηA) (or ideally on distributions over these) that favour both methods equally in some sense. Like, even if the goal was to sell C as a method over NR, I think a more convincing result would be something like showing that the loss of C relative to NR in situations that favour the latter is smaller than the loss of NR relative to C in situations which favour it. This would make C a good choice in absence of other information about the structure of how deviations from affineness arise. Functionally, this just requires a deeper scan over this parameter space, but with judicisouly chosen points. For instance, looking at table E.10, I found myself naturally wondering what happens at (0.28,0) and (0,0.28).
On the hypotheses. Given the context from the standard F-test, the use of a design-dependent hypothesis is okay. However, one thing that gives me pause is that the hypotheses in section 4.2 depend also on the cell partitions---as a result, the structure proposed ends up depending signficantly on an auxiliary detail. This might matter a fair deal, because in a sense the quality of the cells picked captures some sort of tradeoff between the two (ηH,ηA) terms---as an example, if I pick the "right" cells, then the residual may collapse, but ηH blow up, while if I pick two crude or misaligned cells, ηA would increase while ηH drops. Similarly, if I use more or fewer cells. I think I would find the results more convincing if either some notion of stability to the choice of this cell subdivision was developed. Is there a sense in which this can be given---if the cells are rich enough, et c.? Or alternatively, what happens if one averages over a few choices of these cells?