acceptodds
Under review as a conference paper at ICLR 2027

What to Learn for Sequential Log-likelihood Ratio Tests?

Abstract

In a learned log-likelihood ratio (LLR) test, one checkpoint, or a model state saved during training, must be selected to generate the LLR estimates used for threshold-based decisions. However, we find that conventional validation criteria can favor checkpoints whose estimated LLR trajectories poorly match the shape of the true LLR. Moreover, multiplying both the LLR and thresholds by the same positive constant leaves stopping times and decisions unchanged, thus test results alone cannot determine exact-scale LLR estimation error. To establish a proper validation metric for checkpoint selection, we therefore shift the estimand to the positive-projectivized LLR (P+LLR), the true LLR up to positive rescaling. With correspondingly rescaled decision thresholds, P+LLR gives exactly the same stopping decisions as the true LLR to preserve optimality guarantees of statistical tests. P+LLR lets us introduce a validation metric Normalized Integrated Tradeoff Area (NITA), which is provably ordered with an oracle P+LLR estimation error in a high-threshold detection task under an i.i.d. data stream with independent noise. We also establish a finite-sample checkpoint-selection guarantee for empirical NITA and provide a unified extension of the theory to broader non-i.i.d., multiclass data and classification tasks, accompanied by experiments on 12 diverse datasets.

Primary area:
learning on time series and dynamical systems
Keywords:
Sequential hypothesis testing, Density ratio estimation, Log-likelihood ratio estimation, Model selection, Quickest changepoint detection, Early classification of time series, Statistical learning theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_pUhjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the notion of checkpoints, which arise during sequential testing and indicate which model out of two is to be selected for deployment. Such a checkpoint is proposed using the principle of p+LLR, positive-ray LLR testing, and the natural notion of distance from it, the SINE. The final checkpoint estimator, termed NITA, is proved to correlate well with the SINE function for binary i.i.d sequential testing. Practically, since NITA is estimated from finitely many samples, a concentration inequality is provided that shows how well finite-time NITA approximates the true version. These results are extended in the appendix, and the extensions are also tested through experiments.

Critical strengths

The paper's strengths are

a) It attempts to resolve a problem that is fairly simple in nature, and almost motivates itself.

b) The setup of the result and its proof technique are clean.

c) Experiments are fairly comprehensive and attempt to explore outside the theoretical setup considered in the paper.

Critical weaknesses

a) Its writing is unclear. I wanted a mathematical definition of what a checkpoint is, and what it's trying to do, and this wasn't provided. The proof sketch is not informative, and

b) The theoretical main result is for a binary i.i.d. setup, with both hypotheses part of an affine noise family, which is quite restrictive: more general settings have only been dealt with empirically.

Questions for authors

a) The affine-noise theoretical setup is too mild, even with the transfer principle. There should be extensions to further substantiate the effectiveness of NITA.

b) In table 7, your results seem to be competing with J^\hat{J}. Is there any setup where J^\hat{J} is significantly outperformed, or any setup where your checkpoint is significantly outperformed? Alternately, are there guarantees that both will perform similarly?

c) The SINE-NITA antitonality applies for a large-enough comparison box. Precisely how large this is, is probably going to depend upon how large the noise is (alternately, how far the null and alternate hypothesis are). I saw a quantification of this in the appendix, but maybe some discussion should be brought to main body.

5 comments
AuthorsOpenReview ↗

To reviewer pUhj (1/5)

We thank reviewer pUhj for the careful reading and for the suggestions that improved the manuscript.

Our quick responses are:

  • The checkpoint definition was in the submission, and the revision makes it prominent. The submitted paper defines a checkpoint in words in the Abstract and formally in App. A.3 (p. 21, l. 1102 and p. 23, l. 1213). We now gather the definition in Section 2 (pp. 3–4), highlighted in blue. The proof sketch is also rewritten as an itemized list for visual accessibility (Sec. 3.2, p. 7, ll. 354–371). This answers your weakness (a).
  • The extensions beyond binary i.i.d. QCD are covered by theorems, not only by experiments. They were already in the submitted appendix (Thms. B.30, B.37, B.57, B.62 and Props. B.47–B.48 in the revised paper), and the revised Sec. 3.4 now says so in blue text (p. 8, ll. 430–431). This answers your weakness (b).
  • We add two propositions that extend the result beyond a fixed noise law. Under convex order (Prop. B.17) and under approximately common laws through a perturbation bound (Prop. B.19), the SINE–NITA antitone is preserved (Sec. 3.4, blue), and an audit of learned checkpoints supports this in practice (App. D.15). This answers your question (a).
  • A unanimous win over the baselines is not what the paper claims, yet NITA often outperforms J^\widehat{J}. Where its model is correctly specified, a baseline ties NITA, and it falls behind where its model is misspecified (Table 8). Table 7 (now with standard deviations) is J^\widehat{J}'s home setting, binary i.i.d. CUSUM, whereas J^\widehat J shows significantly positive delay overheads across the 12 settings (Fig. 4), and no baseline shows a significant advantage over NITA in any comparison we report (Fig.4, Tables 7-8). This answers your question (b).
  • The box-size requirement is now stated, and the box need not be located in practice. Your reading is correct, and the theoretical requirement x0>max⁡x1,B/Δx_0>\max\\{x_1,B/\Delta\\} is now stated after Cor. 3.7 (p. 8, blue). In practice the box need not be located, since full-box NITA is rank-aligned with high-threshold NITA at ρ~S=0.997\tilde\rho_S=0.997 (Fig. 12, Table 6, Fig. 13) and Theorem 3.8 (finite-sample selection guarantee) holds for any fixed box, including the full box. This answers your question (c).

Your comments improved our paper's clarity and integrity, which we highly appreciate. Please see below for our detailed answers.

What a checkpoint is, and what this paper tries to do

This is a clarification of the raised weakness (a).

Our goal is to choose which saved version of a model, called a checkpoint, to deploy for sequential testing. Training the LLR estimator for NckptN_{\mathrm{ckpt}} epochs yields the candidate set of checkpoints M=1,…,Nckpt\mathcal M=\\{1,\dots,N_{\mathrm{ckpt}}\\}. Checkpoint mm is the parameter state wm∈Rdpar\mathbf w_m\in\mathbb R^{d_{\mathrm{par}}} saved at the end of epoch mm, where dpard_{\mathrm{par}} is the parameter count. All the candidate checkpoints come from one training run. Each checkpoint produces estimated LLRs, or sequential evidence, and this evidence is what checkpoint selection evaluates. Applied to a sequential data record (Section 2), checkpoint mm produces the evidence process gm=(gmt)t≥1g_m=(g_m^t)_{t\ge1}, where gmtg_m^t is its estimate of the one-step LLR λt\lambda^t based on the observations up to time tt.

The deployed checkpoint's gmg_m drives the SPRT or CUSUM. The challenge is that under real-world scenarios the true LLR is unavailable, so the best checkpoint must be selected without it. A checkpoint selector is a validation criterion A\mathcal A computed from held-out records together with the checkpoint it returns,

m^A∈arg⁡max⁡m∈MA(gm),A∈{NITA^,,−CE,,Acc,,J^},\widehat m_{\mathcal A}\in\arg\max_{m\in\mathcal M}\mathcal A(g_m),\qquad \mathcal A\in\{\widehat{\mathrm{NITA}},\\,-\mathrm{CE},\\,\mathrm{Acc},\\,\widehat J\},

and the oracle target is mSINE∈arg⁡min⁡m∈MSINE(Λ^m,T,ΛT)m_{\mathrm{SINE}}\in\arg\min_{m\in\mathcal{M}}\mathrm{SINE}(\widehat{\boldsymbol{\Lambda}}_{m,T},\boldsymbol{\Lambda}_T), where Λ^m,T\widehat{\boldsymbol{\Lambda}}_{m,T} is the cumulative trajectory of gmg_m and ΛT\boldsymbol{\Lambda}_T is that of the true LLR.

By shifting the estimand from the exact-scale LLR to its positive ray (i.e., P+LLR), we obtain a validation metric, NITA, which can be computed without true LLRs. Within the scope and assumptions, we prove that NITA is inversely ordered (i.e., antitone) to the oracle error SINE. This enables a practitioner to select a checkpoint with larger NITA, which is expected to have smaller SINE. In experiments on 12 settings, NITA retains this antitone with SINE well beyond the regime covered by the theory. A practitioner can therefore monitor NITA during training and select the checkpoint with the largest NITA. This is the paper's main contribution, and we believe it is beneficial to the broader research community. Please also see our global comment for a summary of the contributions.

AuthorsOpenReview ↗

To reviewer pUhj (2/5)

Where the submitted paper already states this, and how we improve further

While we appreciate your clarification request, most of the above information is already present in the submitted manuscript. Thus, we respectfully note that the statement "mathematical definition of what a checkpoint is, and what it's trying to do, and this wasn't provided" does not hold for the submitted text. We agree, however, that the formal definition was spread over Sec. 3.3 and App. A.3. Thus we gather the checkpoint definition in the revised Section 2. Please see the new paragraph "Learned evidence and checkpoints" highlighted in blue on pp. 3–4. Following reviewer HwfC's recommendation, we also add opening sentences to the Introduction on sequential tests and learned LLRs, and a related-work paragraph, all highlighted in blue.

The quotes below are unchanged from the submission and appear in black in the revised PDF. Page and line numbers refer to the revised PDF.

  • What a checkpoint is. The abstract (p. 1, ll. 11–13) defines it in words, "one checkpoint, or a model state saved during training, must be selected to generate the LLR estimates used for threshold-based decisions." App. A.3 (p. 21, l. 1102) defines it formally, and p. 23, l. 1213 states the selection rule m^∈arg⁡max⁡m∈MNITA^m\widehat m\in\arg\max_{m\in\mathcal M}\widehat{\mathrm{NITA}}_m. The training protocol is in App.D.7 (p.76, ll.4085–4088, "use the same saved checkpoints … The best epoch is selected separately for each criterion") and also in Table 3 (p.77).
  • What it is trying to do, and why the true LLR cannot be used. The Introduction states that "In practice, the true LLR is unknown and must be estimated from data" (p. 1, ll. 45–46). Also in the Introduction, we state that "as long as we insist on the exact-scale LLR as the estimand, no metric derived from operating curves can faithfully represent the LLR estimation error" (p.2, ll.63–64). Then we provide the central question, "What learning target permits principled checkpoint selection for calibrated sequential tests?" (l.66), and ll.68–71 give the answer right after: it's the positive ray (P+LLR).
  • The validation metric and its guarantees. Page 2, ll.72–81 states that "computing SINE requires the true LLR, whereas computing NITA does not", "we prove an asymptotic antitone relation: lower SINE implies larger NITA," and "the estimation error of empirical NITA is bounded uniformly across candidate checkpoints", which gives the finite-sample guarantee for checkpoint selection. Section 3.1 (p. 5, ll. 230–232 and 249–253) states "The practical checkpoint selector must use only validation outcomes from the threshold sweep", "Unlike SINE, NITA requires no true LLR," and "under what conditions does a checkpoint with smaller oracle projective error (SINE) also have a larger observable operating area (NITA)?" And Theorem 3.8 (p. 8, l. 416) states that "the selected checkpoint is within 2εconc2\varepsilon_{\mathrm{conc}} of the best population grid-NITA."
  • How it is used in practice. Fig.2d (p.6, ll.288–291) shows that during training CE and local Acc keep improving while SINE rises and NITA falls, so the SINE- and NITA-optimal checkpoints occur early. The resulting checkpoints selected by NITA or oracle SINE preserve the true-LLR trajectory shape (Fig. 2c), whereas CE and Acc select checkpoints with distorted estimated LLRs (Fig. 2e). Section 4 (p. 9, ll. 466–468) evaluates "NITA-based checkpoint selection for LLR estimators on practical streaming data." In the experiments, we compare NITA-selected checkpoints against every baseline on held-out test data (p. 10, ll. 506-515).
AuthorsOpenReview ↗

To reviewer pUhj (3/5)

Improved proof sketch

We revise the proof sketch of Theorem 3.6 (Section 3.2) so that it is organized around the three monotonicities in σ\sigma. Given that we now have an extra page, we also set it as an itemized list to reduce visual clutter. The following is an excerpt from the revised paper.

We prove by three monotonicities in σ\sigma (formal proof in Appendix C.1).

  • (i) Geometry. By Assumption 3.4(3), the learned evidence trajectory can be written as the orthogonal sum αΛT+σηΣ,T\alpha\boldsymbol\Lambda_T+\sigma\boldsymbol\eta_{\Sigma,T}, and the closest point of the ray is attained at β=α\beta=\alpha. Hence the component orthogonal to the ray, whose norm defines SINE, is the cumulative nuisance σηΣ,T\sigma\boldsymbol\eta_{\Sigma,T} alone. This gives Eq. (13), which shows that SINE2\mathrm{SINE}^2 increases with σ\sigma.
  • (ii) Sequential rate. The false-alarm exponent κ(σ)\kappa(\sigma) can be written as the positive root of E0[eκgσ]=1\mathbb E_0[e^{\kappa g_\sigma}]=1. Using independence, the left-hand side factorizes into the moment generating functions (MGFs) of λ\lambda under P0P_0 and of ξ\xi under QQ, Fσ(κ):=E0[eκgσ]=Mλ(ακ),Mξ(σκ)F_\sigma(\kappa):=\mathbb E_0[e^{\kappa g_\sigma}]=M_\lambda(\alpha\kappa)\\,M_\xi(\sigma\kappa) (Lemma B.12). For a centered nondegenerate ξ\xi, Mξ(v)>1M_\xi(v)>1 for every v>0v>0 and MξM_\xi increases on v>0v>0, so Fσ(κ)F_\sigma(\kappa) increases with σ\sigma at every κ>0\kappa>0. Thus, the level Fσ(κ)=1F_\sigma(\kappa)=1 is reached with a smaller κ\kappa: κ(σ)\kappa(\sigma) decreases in σ\sigma. With μ1(σ)=αE1[λ]\mu_1(\sigma)=\alpha\mathbb E_1[\lambda] fixed, the effective slope E(σ)=κ(σ)μ1\mathcal E(\sigma)=\kappa(\sigma)\mu_1 decreases as well. In words, noise costs false-alarm protection per unit of delay.
  • (iii) Frontiers. For σ1<σ2\sigma_1<\sigma_2, Eq. (12) gives log⁡yσ1⋆(x)−log⁡yσ2⋆(x)=Δx+bσ1(x)−bσ2(x)\log y_{\sigma_1}^\star(x)-\log y_{\sigma_2}^\star(x)=\Delta x+b_{\sigma_1}(x)-b_{\sigma_2}(x) with Δ=E(σ1)−E(σ2)>0\Delta=\mathcal E(\sigma_1)-\mathcal E(\sigma_2)>0 by (ii). Because the remainders are bounded, ∣bσ1(x)−bσ2(x)∣≤B|b_{\sigma_1}(x)-b_{\sigma_2}(x)|\le B for all x≥x1x\ge x_1. Then the difference is positive for every x≥x0>max⁡x1,B/Δx\ge x_0>\max\\{x_1,B/\Delta\\}, leading to Eq. (14). Integrating over a common box gives Eq. (15), and finally (i) transfers the ordering to SINE when ηΣ,T≠0\boldsymbol{\eta}_{\Sigma,T}\neq0.

Correction to potential misunderstandings

We respectfully clarify some potential misunderstandings we found in the review's summary of the paper.

  • Checkpoints "arise during sequential testing" → checkpoints are saved during training, before any testing (Abstract, p.1, ll.11–13, Sec.1, p.1, ll.36–37, App.D.7, p.76, ll.4085–4088).
  • A checkpoint selects "which model out of two" → during validation, a user chooses one checkpoint among the many saved checkpoints (Sec.2, pp.3-4, ll.160–167, Sec.3.3, p.8, ll.394–396, Thm.3.8, Eqs. (18)–(20), where NckptN_{\mathrm{ckpt}} is the number of candidates). The two hypotheses are distributions or labels, instead of two candidate checkpoints (Sec.2, p.3, ll.108–111).
  • NITA is described as the "checkpoint estimator" → NITA is a validation metric (Sec.3.1, p.5, ll.230–232 and l.249, Def.3.3, Eq.(9)), and the selected checkpoint is the maximizer of it (Sec.3.3, p.8, ll.394–396, App.A.3, p.23, l.1213).
  • "Both hypotheses part of an affine noise family" (weakness (b)) → the affine family is a model of the learned score, instead of the hypotheses. The hypotheses themselves are otherwise general: P0P_0 and P1P_1 are only assumed distinct and mutually absolutely continuous (Sec.3.2, p.5, ll.257–261).

Our main theorem is intentionally designed to handle binary i.i.d. data, while the extension theorems support broader settings

This is a clarification of weakness (b) and also addresses Question (a).

The extensions are covered by theorems in addition to experiments.

We respectfully disagree that "more general settings have only been dealt with empirically." Beyond the transfer principle (Prop. B.25, App. B.7), the submitted manuscript establishes explicit extensions, introduced in Sec. 3.4, as follows.

  • Multiclass QCD: Thm. B.30.
  • Markov QCD: Thm. B.37 and Cor. B.39.
  • Binary, multiclass, and Markov SPRT: Props. B.47–B.48 (with Cor. B.49), Thm. B.57, and Thm. B.62, respectively.

Table 2 lists the effective slopes and App. C.4–C.7 contain the proofs. These are formal results with their own scope, as stated in the paper.

Reviewer_HwfCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies checkpoint selection for learned sequential likelihood-ratio tests. Under structured noise and high-threshold asymptotic, it establishes a connection between an oracle diagnostic of projective LLR error (SINE) and an observable validation metric (NITA), motivating NITA-based checkpoint selection. It further provides a finite-sample guarantee that selection using empirical NITA approximately maximises population grid-NITA. An extensive empirical study, including settings beyond the theoretical assumptions, reports improved held-out sequential performance relative to the evaluated baselines.

Critical strengths
  • The practical problem is useful and clearly motivated: good local classification scores need not select the best model for a stopped sequential decision.
  • The paper provides an approximation guarantee for the practical version of NITA computed from a finite validation set.
  • The presented empirical study is elaborate and solid: it includes settings beyond the theoretical assumptions, uses a reasonable set of baselines compared on the same set of checkpoints, and additionally evaluates a practically simpler formulation of NITA (presented in the form of an ablation study).
Critical weaknesses
  • The proven theoretical connection between NITA and the oracle diagnostics assumes the noise introduced by each checkpoint only differs in magnitude and its distribution is otherwise the same, which narrows this connection. (Note that the empirical evaluation setup does not enforce this assumption providing some evidence either of robustness to violations of this assumption or of its prevalence in practice.)
Questions for authors
  • Can you establish an ordering result or an informative error bound when checkpoint residual distributions vary?
  • Can the main text explicitly identify the new contribution relative to the already acknowledged (in F.6) area-based validation, hypervolume, projection geometry, and scale-invariant mismatched-CUSUM literature?

Minor: I think the introduction would benefit from a few opening notes explaining the general problem context (such as what sequential testing is, what is learned SPRT and so on) to make the paper better accessible.

4 comments
AuthorsOpenReview ↗

To reviewer HwfC (1/2)

We thank reviewer HwfC's sharp insight and constructive support.

(W1, Q1) Checkpoint residual distribution and its bound

This is a clarification of the raised weakness and the first bullet point of the Questions. Thank you for asking this important points.

Our quick responses to these comments are:

  • Yes, we can provide BOTH an ordering result across residual laws and an informative perturbation bound. We can derive sufficient conditions for preserving the SINE-NITA ordering beyond a common residual law, as we shown below.
  • Additional experiments support robustness to heterogeneous residual noise. Residual distributions can vary, yet the NITA-selected checkpoints remain close to the oracle SINE-selected checkpoints, which is important under real-world deployment.

While the original theory is mathematically rigorous under its scope, below we additionally allow the residual law to vary.

1. Ordering across different residual laws

Consider the binary i.i.d. centered independent-noise setting with the original moment and first-passage assumptions. After removing each checkpoint's positive signal scale (Prop. 2.1), write its score as g~j=λ+ηj\tilde g_j=\lambda+\eta_j, where λ=log⁡(dP1/dP0)\lambda=\log(dP_1/dP_0) is the true LLR and ηj\eta_j has a checkpoint-specific centered law, common to the two hypotheses and independent of λ\lambda under P0P_0. Define

\qquad K_j(u)=\log\mathbb Ee^{u\eta_j}.$$ Let $r_j>0$ be the unique positive root of $K_\lambda(r_j)+K_j(r_j)=0$. Its effective slope is $\mathcal E_j=Ir_j$, where $I=\mathbb E_1\lambda>0$ is the unchanged post-change drift. Suppose the residuals are in convex order, denoted by $\eta_1\le_{\mathrm{cx}}\eta_2$, indicating that the expectation of every integrable convex function is smaller or equal under $\eta_1$. This orders both residual variances and exponential moments, giving $$\mathrm{SINE}_1\le\mathrm{SINE}_2, \qquad K_1(u)\le K_2(u), \qquad r_1\ge r_2.$$ Here the SINE inequality uses the i.i.d. cumulative geometry (population orthogonality and i.i.d. increments give $\|\boldsymbol\eta_{j,\Sigma,T}\|^2_{\mathcal H_\Sigma}=\tfrac{T(T+1)}{2}\operatorname{Var}(\eta_j)$, Eq. (44) in Appendix B.4, which specializes Eq. (13)). Evaluating $K_\lambda+K_1$ at $r_2$ gives $K_\lambda(r_2)+K_1(r_2)\le K_\lambda(r_2)+K_2(r_2)=0.$ The function $K_\lambda+K_1$ is convex, vanishes at zero, and has a negative derivative there. It is therefore positive beyond its unique positive root $r_1$, so the inequality above implies $r_2\le r_1$. When $K_1(r_2)<K_2(r_2)$, we obtain $r_1>r_2$ and therefore **eventual NITA ordering on common, unsaturated high-threshold comparison boxes, by the same frontier argument as Theorem 3.6** (Proposition B.17 in Appendix B.4). Thus, a common residual law is not a strict necessary, as convex order supplies a sufficient condition across different laws. ### 2. An informative perturbation bound We now quantify how a change in residual law perturbs the positive Cramér root and the effective slope that governs the asymptotic NITA ordering. For this comparison, choose common-law reference residuals $\bar\eta_j=s_j\xi$, where $\mathbb E[\xi]=0$, $\operatorname{Var}(\xi)=1$, and $s_j^2=\operatorname{Var}(\eta_j)$. For centered residuals that are i.i.d. over time and independent of the common signal, variance matching gives each reference the same population cumulative SINE as its original checkpoint, using the same horizon and weighting. **The bound below controls the root differences between the original and reference models and yields a sufficient condition for preserving SINE–NITA ordering across checkpoints.** Let $\bar K_j$ and $\bar r_j$ denote the reference residual CGF and positive root, respectively. On an interval $[\underline u_j,\overline u_j]$ containing $r_j$ and $\bar r_j$, assume $$|K_j(u)-\bar K_j(u)|\le\varepsilon_j, \qquad (K_\lambda+\bar K_j)'(u)\ge m_j>0 .$$ The mean value theorem gives $$|r_j-\bar r_j|\le\frac{\varepsilon_j}{m_j}.$$ Consequently, the SINE-consistent reference ordering ($\bar r_1>\bar r_2\iff s_1<s_2\iff\mathrm{SINE}_1<\mathrm{SINE}_2$, by Theorem 3.6) survives whenever $$\bar r_1-\bar r_2> \frac{\varepsilon_1}{m_1}+\frac{\varepsilon_2}{m_2}$$ (Proposition B.19 in Appendix B.4). **The inequality above shows that even under different residual distributions, the checkpoint with lower SINE keeps the larger effective slope whenever this margin condition holds.**
AuthorsOpenReview ↗

To reviewer HwfC (2/2)

What happens for actual learned checkpoints?

We conduct additional experiments with ten binary i.i.d. QCD runs to compute scale-normalized residual noise distributions of checkpoint pairs at early, middle, and final training iterations, in addition to the NITA-selected checkpoint and the SINE-selected oracle checkpoint. We observe that the empirical residual shape can change, including the tail distributions. As a result, pairwise SINE-NITA inversions occur for a small fraction of checkpoint pairs (2.62.6–5.35.3\\%), more often among late training checkpoints.

However, NITA and oracle SINE select very similar checkpoints. Across the ten runs, NITA and oracle SINE select the same checkpoint in 5 out of 10 runs. The differences in NITA between the SINE-selected and NITA-selected checkpoints are as small as 7.7×10−57.7\times10^{-5} on average and 2.7×10−42.7\times10^{-4} at maximum, and the squared-SINE differences are 0.004360.00436 on average and 0.018380.01838 at maximum. These results complement Tables 1 and 6 and the stress tests presented in Appendix D.13.

On a separate note, Theorem 3.8 does not require a common residual law while ensuring practical regret bounds relative to the best population grid-NITA.

The uploaded revision states the two extensions in Section 3.4 (blue), with statements and proofs in Appendix B.4 (Propositions B.17 and B.19, Remark B.18). The additional experiment is in the new Appendix D.15.

(Q2) Contribution relative to the precedents

Thank you for this suggestion. Making the novelty over existing work explicit will greatly improve our paper's readability and impact. We add the following paragraph to the Introduction, in blue text.

Compared with prior work, our contribution directly connects the LLR estimation target to an observable validation metric. Although each ingredient of our theory, such as area-based validation (Wang et al., 2016; Achenchabe et al., 2021), hypervolume (Zitzler et al., 2007), scale-invariant projection (Roux et al., 2018), and threshold scaling invariance (Cooper & Meyn, 2024), has precedents (Appendix F.6), none of them asks what an operating-curve metric can identify about the true LLR itself.

On your "minor comment"

Actually we are very happy to see this suggestion to make the paper more accessible to the broader community. We have added opening context explaining sequential testing and SPRT/CUSUM at the very beginning of the Introduction.

Overall, your comments and suggestions greatly improve our manuscript. We authors are grateful for it.

Thank you very much for the additional theoretical results and experiments! They substantially address my concerns.

Thank you very much for the reply. In the same spirit as your suggestion to make the paper accessible to a broader community, our latest revision restates Theorem 3.6 and adds a notation glossary (Appendix G). The changes are listed in our comment "Revised PDF uploaded: summary of changes", and our replies to the AC also summarize and restate our theorem and mathematical setup with page and line pointers to the paper. We still have time to discuss further questions if you happen to come across any: the discussion closes soon, but we authors are here to answer.

Comments on the paper

10 comments
AuthorsOpenReview ↗

Global author comment

We sincerely thank the reviewers and Area Chairs for their extensive time and effort in reviewing our paper and coordinating ICLR 2027. We are encouraged to find that both reviewers HwfC and pUhj rate the soundness of the paper as good and raise no concern about the correctness of its results. Alongside our point-by-point reply to their questions, we briefly summarize the paper's core contributions, their scope, and their practical value.

Our central question is how to select a trained checkpoint for sequential decisions when true log-likelihood ratios (LLRs) are unavailable. HwfC highlights the practical value of this problem, while pUhj recognizes its natural motivation. We appreciate both. The key contributions are as follows.

  1. Decision-relevant estimation target. Multiplying the score and thresholds by the same positive constant preserves SPRT/CUSUM stopping times and decisions (Prop. 2.1, Thm. B.1). Consequently, operating-curve metrics cannot identify exact-scale LLR error (Cor. 2.2). In our novel framework, P+LLR incorporates this equivalence into the estimation target (Def. 3.1), and SINE measures the corresponding projective error (Def. 3.2).
  2. Theoretical connection to a practical validation criterion. NITA is the normalized area under the operating curve in a fixed comparison box (Def. 3.3). For binary i.i.d. QCD, Thm. 3.6 establishes an asymptotic antitone ordering between SINE and NITA within affine score families sharing a fixed independent-noise law. Cor. 3.7 gives a local quantitative relation. Formal extensions address multiclass, non-i.i.d. (order-N Markov process), and SPRT settings under their assumptions (Prop. B.25, Thms. B.30, B.37, B.57, B.62, Props. B.47–B.48, App. B.7–B.11). We thank reviewer pUhj for recognizing the clean setup and proof technique.
  3. Finite-sample selection guarantee. Thm. 3.8 bounds the population grid-NITA regret of empirical checkpoint selection relative to the best checkpoint. This guarantee does not require the affine-noise assumption or a shared residual law across checkpoints. HwfC identifies this finite-validation guarantee as a strength, which we appreciate.
  4. Evidence from actual learned checkpoints. NITA uses a labeled validation dataset without requiring true LLRs, densities, or knowledge of the true temporal dependence order. Across 12 settings, including real-world datasets, our experiments compare selectors on the same candidate checkpoints and evaluate held-out performance (Section 4, Table 1, Fig. 4, Table 6 for the full-box ablation, and two additional real datasets in App. D.10). We are glad to see both HwfC and pUhj recognize the comprehensive experiments beyond the main theoretical setup. HwfC additionally highlights the thorough baseline comparisons and the simplicity of the full-box NITA formulation.

Motivated by the important problem of deploying sequential tests under real-world scenarios, our experiments are designed around a highly practically relevant challenge, namely selecting learned evidence (i.e., estimated LLRs) under temporal dependence and real streaming conditions without access to oracle LLRs. We believe this combination offers a valuable contribution to ICLR and the broader research community, by providing a practical checkpoint selection method and a rigorous foundation for further theoretical work on learned sequential tests.

Please see our replies to the raised concerns and questions below. We appreciate both reviewers' valuable suggestions to improve the paper. The revised paper is now available, with new material highlighted in blue.

Area_Chair_hjvyOpenReview ↗

Comment to authors

Dear authors,

As you can see, there are only two reviews for your paper and we are having a very difficult time finding a third reviewer. To supplement this, I will also actively review your paper.

I am trying to understand your main theorem (and even as a theory guy), it is almost incomprehensible to me and I don't believe it is currently humanly readable. Can you explain in simple mathematical terms what the main theoretical result is about? I am having a hard time figuring out what the underlying statistical problem even is. The use of many notations, and somewhat arbitrary seeming assumptions make understanding the contribution very difficult, so I am directly reaching out to you.

Best, The AC

AuthorsOpenReview ↗

To Area Chair hjvy (1/2)

Dear AC, thank you for taking the time and substantive effort to review our paper directly. You are right that the statistical problem and the purpose of the assumptions should be much clearer. Please see our point-by-point response below.

The statistical problem

The challenge. To conduct log-likelihood ratio based sequential tests, like CUSUM or SPRT, we need to utilize the true log-likelihood ratio (LLR) as a score to enjoy its optimality. However, the true LLR is not available under real-world deployment. Therefore we need to train models to estimate the LLRs, and select one best model parameter set, called a checkpoint, for the final industrial product. This is a real problem that we currently face in our organization.

The challenge here is how to quantify the LLR approximation error without the learning target, the true LLR. Conventional metrics such as cross-entropy can fail under some setups (Fig.2). Thus we show that shifting the estimand to a positive ray of the true LLR (P+LLR) can provide a validation surrogate (NITA) for the otherwise oracle error (SINE).

Mathematical setup. Under the main theorem (Thm.3.6), we consider a binary quickest changepoint detection (QCD) on a stream X1,X2,…X_1,X_2,\dots, which is i.i.d. from P0P_0 before a changepoint ν\nu and i.i.d. from P1P_1 after it. The sequential test employed for the QCD task is CUSUM: a score gg (e.g., either the true or an estimated LLR) updates the CUSUM statistic as Wt=max⁡(0,Wt−1+g(Xt))W_t=\max(0,W_{t-1}+g(X_t)), and an alarm is raised when the statistic reaches a decision threshold hh: Wt≥hW_t\ge h.

To delineate a performance curve, hh must be swept (i.e., use various levels of hh) to evaluate the performance tradeoff. In QCD's case, the average run length to false alarm (ARLg(h)\mathrm{ARL}_g(h)) and the average detection delay (ADDg(h)\mathrm{ADD}_g(h)) are in tradeoff. The optimal score is λ=log⁡(dP1/dP0)\lambda=\log(dP_1/dP_0), which is unknown and must be estimated by training. The training produces MM candidate checkpoints with their estimated LLRs g1,…,gMg_1,\dots,g_M, and one must be chosen without access to the true LLR, λ\lambda. Thus, this is a model selection with an unobservable target.

SINE and NITA. Multiplying both the score and the threshold by the same constant c>0c>0 does not change the test, therefore the error can be measured as an angle between the cumulative trajectories of gg and λ\lambda, both represented as vectors. This angle error (the sine of the angle) is called SINE$(g)$, and to compute it we need the true LLR, λ\lambda (oracle). On the other hand, NITA$(g)$ is the normalized area under the ARL-versus-ADD curve of gg (larger is better) and it requires only validation data. Our main result connects SINE and NITA via noise magnitude σ\sigma to let us use NITA as a validation surrogate of SINE, as explained below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.