- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper considers the notion of checkpoints, which arise during sequential testing and indicate which model out of two is to be selected for deployment. Such a checkpoint is proposed using the principle of p+LLR, positive-ray LLR testing, and the natural notion of distance from it, the SINE. The final checkpoint estimator, termed NITA, is proved to correlate well with the SINE function for binary i.i.d sequential testing. Practically, since NITA is estimated from finitely many samples, a concentration inequality is provided that shows how well finite-time NITA approximates the true version. These results are extended in the appendix, and the extensions are also tested through experiments.
Critical strengths
The paper's strengths are
a) It attempts to resolve a problem that is fairly simple in nature, and almost motivates itself.
b) The setup of the result and its proof technique are clean.
c) Experiments are fairly comprehensive and attempt to explore outside the theoretical setup considered in the paper.
Critical weaknesses
a) Its writing is unclear. I wanted a mathematical definition of what a checkpoint is, and what it's trying to do, and this wasn't provided. The proof sketch is not informative, and
b) The theoretical main result is for a binary i.i.d. setup, with both hypotheses part of an affine noise family, which is quite restrictive: more general settings have only been dealt with empirically.
Questions for authors
a) The affine-noise theoretical setup is too mild, even with the transfer principle. There should be extensions to further substantiate the effectiveness of NITA.
b) In table 7, your results seem to be competing with . Is there any setup where is significantly outperformed, or any setup where your checkpoint is significantly outperformed? Alternately, are there guarantees that both will perform similarly?
c) The SINE-NITA antitonality applies for a large-enough comparison box. Precisely how large this is, is probably going to depend upon how large the noise is (alternately, how far the null and alternate hypothesis are). I saw a quantification of this in the appendix, but maybe some discussion should be brought to main body.
5 comments
To reviewer pUhj (2/5)
Where the submitted paper already states this, and how we improve further
While we appreciate your clarification request, most of the above information is already present in the submitted manuscript. Thus, we respectfully note that the statement "mathematical definition of what a checkpoint is, and what it's trying to do, and this wasn't provided" does not hold for the submitted text. We agree, however, that the formal definition was spread over Sec. 3.3 and App. A.3. Thus we gather the checkpoint definition in the revised Section 2. Please see the new paragraph "Learned evidence and checkpoints" highlighted in blue on pp. 3–4. Following reviewer HwfC's recommendation, we also add opening sentences to the Introduction on sequential tests and learned LLRs, and a related-work paragraph, all highlighted in blue.
The quotes below are unchanged from the submission and appear in black in the revised PDF. Page and line numbers refer to the revised PDF.
- What a checkpoint is. The abstract (p. 1, ll. 11–13) defines it in words, "one checkpoint, or a model state saved during training, must be selected to generate the LLR estimates used for threshold-based decisions." App. A.3 (p. 21, l. 1102) defines it formally, and p. 23, l. 1213 states the selection rule . The training protocol is in App.D.7 (p.76, ll.4085–4088, "use the same saved checkpoints … The best epoch is selected separately for each criterion") and also in Table 3 (p.77).
- What it is trying to do, and why the true LLR cannot be used. The Introduction states that "In practice, the true LLR is unknown and must be estimated from data" (p. 1, ll. 45–46). Also in the Introduction, we state that "as long as we insist on the exact-scale LLR as the estimand, no metric derived from operating curves can faithfully represent the LLR estimation error" (p.2, ll.63–64). Then we provide the central question, "What learning target permits principled checkpoint selection for calibrated sequential tests?" (l.66), and ll.68–71 give the answer right after: it's the positive ray (P+LLR).
- The validation metric and its guarantees. Page 2, ll.72–81 states that "computing SINE requires the true LLR, whereas computing NITA does not", "we prove an asymptotic antitone relation: lower SINE implies larger NITA," and "the estimation error of empirical NITA is bounded uniformly across candidate checkpoints", which gives the finite-sample guarantee for checkpoint selection. Section 3.1 (p. 5, ll. 230–232 and 249–253) states "The practical checkpoint selector must use only validation outcomes from the threshold sweep", "Unlike SINE, NITA requires no true LLR," and "under what conditions does a checkpoint with smaller oracle projective error (SINE) also have a larger observable operating area (NITA)?" And Theorem 3.8 (p. 8, l. 416) states that "the selected checkpoint is within of the best population grid-NITA."
- How it is used in practice. Fig.2d (p.6, ll.288–291) shows that during training CE and local Acc keep improving while SINE rises and NITA falls, so the SINE- and NITA-optimal checkpoints occur early. The resulting checkpoints selected by NITA or oracle SINE preserve the true-LLR trajectory shape (Fig. 2c), whereas CE and Acc select checkpoints with distorted estimated LLRs (Fig. 2e). Section 4 (p. 9, ll. 466–468) evaluates "NITA-based checkpoint selection for LLR estimators on practical streaming data." In the experiments, we compare NITA-selected checkpoints against every baseline on held-out test data (p. 10, ll. 506-515).
To reviewer pUhj (3/5)
Improved proof sketch
We revise the proof sketch of Theorem 3.6 (Section 3.2) so that it is organized around the three monotonicities in . Given that we now have an extra page, we also set it as an itemized list to reduce visual clutter. The following is an excerpt from the revised paper.
We prove by three monotonicities in (formal proof in Appendix C.1).
- (i) Geometry. By Assumption 3.4(3), the learned evidence trajectory can be written as the orthogonal sum , and the closest point of the ray is attained at . Hence the component orthogonal to the ray, whose norm defines SINE, is the cumulative nuisance alone. This gives Eq. (13), which shows that increases with .
- (ii) Sequential rate. The false-alarm exponent can be written as the positive root of . Using independence, the left-hand side factorizes into the moment generating functions (MGFs) of under and of under , (Lemma B.12). For a centered nondegenerate , for every and increases on , so increases with at every . Thus, the level is reached with a smaller : decreases in . With fixed, the effective slope decreases as well. In words, noise costs false-alarm protection per unit of delay.
- (iii) Frontiers. For , Eq. (12) gives with by (ii). Because the remainders are bounded, for all . Then the difference is positive for every , leading to Eq. (14). Integrating over a common box gives Eq. (15), and finally (i) transfers the ordering to SINE when .
Correction to potential misunderstandings
We respectfully clarify some potential misunderstandings we found in the review's summary of the paper.
- Checkpoints "arise during sequential testing" → checkpoints are saved during training, before any testing (Abstract, p.1, ll.11–13, Sec.1, p.1, ll.36–37, App.D.7, p.76, ll.4085–4088).
- A checkpoint selects "which model out of two" → during validation, a user chooses one checkpoint among the many saved checkpoints (Sec.2, pp.3-4, ll.160–167, Sec.3.3, p.8, ll.394–396, Thm.3.8, Eqs. (18)–(20), where is the number of candidates). The two hypotheses are distributions or labels, instead of two candidate checkpoints (Sec.2, p.3, ll.108–111).
- NITA is described as the "checkpoint estimator" → NITA is a validation metric (Sec.3.1, p.5, ll.230–232 and l.249, Def.3.3, Eq.(9)), and the selected checkpoint is the maximizer of it (Sec.3.3, p.8, ll.394–396, App.A.3, p.23, l.1213).
- "Both hypotheses part of an affine noise family" (weakness (b)) → the affine family is a model of the learned score, instead of the hypotheses. The hypotheses themselves are otherwise general: and are only assumed distinct and mutually absolutely continuous (Sec.3.2, p.5, ll.257–261).
Our main theorem is intentionally designed to handle binary i.i.d. data, while the extension theorems support broader settings
This is a clarification of weakness (b) and also addresses Question (a).
The extensions are covered by theorems in addition to experiments.
We respectfully disagree that "more general settings have only been dealt with empirically." Beyond the transfer principle (Prop. B.25, App. B.7), the submitted manuscript establishes explicit extensions, introduced in Sec. 3.4, as follows.
- Multiclass QCD: Thm. B.30.
- Markov QCD: Thm. B.37 and Cor. B.39.
- Binary, multiclass, and Markov SPRT: Props. B.47–B.48 (with Cor. B.49), Thm. B.57, and Thm. B.62, respectively.
Table 2 lists the effective slopes and App. C.4–C.7 contain the proofs. These are formal results with their own scope, as stated in the paper.
To reviewer pUhj (1/5)
We thank reviewer pUhj for the careful reading and for the suggestions that improved the manuscript.
Our quick responses are:
Your comments improved our paper's clarity and integrity, which we highly appreciate. Please see below for our detailed answers.
What a checkpoint is, and what this paper tries to do
This is a clarification of the raised weakness (a).
Our goal is to choose which saved version of a model, called a checkpoint, to deploy for sequential testing. Training the LLR estimator for Nckpt epochs yields the candidate set of checkpoints M=1,…,Nckpt. Checkpoint m is the parameter state wm∈Rdpar saved at the end of epoch m, where dpar is the parameter count. All the candidate checkpoints come from one training run. Each checkpoint produces estimated LLRs, or sequential evidence, and this evidence is what checkpoint selection evaluates. Applied to a sequential data record (Section 2), checkpoint m produces the evidence process gm=(gmt)t≥1, where gmt is its estimate of the one-step LLR λt based on the observations up to time t.
The deployed checkpoint's gm drives the SPRT or CUSUM. The challenge is that under real-world scenarios the true LLR is unavailable, so the best checkpoint must be selected without it. A checkpoint selector is a validation criterion A computed from held-out records together with the checkpoint it returns,
mA∈argmaxm∈MA(gm),A∈{NITA,,−CE,,Acc,,J},
and the oracle target is mSINE∈argminm∈MSINE(Λm,T,ΛT), where Λm,T is the cumulative trajectory of gm and ΛT is that of the true LLR.
By shifting the estimand from the exact-scale LLR to its positive ray (i.e., P+LLR), we obtain a validation metric, NITA, which can be computed without true LLRs. Within the scope and assumptions, we prove that NITA is inversely ordered (i.e., antitone) to the oracle error SINE. This enables a practitioner to select a checkpoint with larger NITA, which is expected to have smaller SINE. In experiments on 12 settings, NITA retains this antitone with SINE well beyond the regime covered by the theory. A practitioner can therefore monitor NITA during training and select the checkpoint with the largest NITA. This is the paper's main contribution, and we believe it is beneficial to the broader research community. Please also see our global comment for a summary of the contributions.