- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies how to monitor risk after early-stopping deployment: because early stopping hides the full output needed for verification, some risk violations cannot be detected without auditing. The authors propose an auditing protocol — randomly completing a subset of samples and delivering the full output to obtain feedback — and provide a detection-delay lower bound, an alarm method, and risk bounds. However, the paper does not do a good job of explaining what task this is, making it obscure and difficult to understand.
Critical strengths
-
The paper does not exceed the page limit;
-
The experiments are rich, and the supplementary material is substantial.
Critical weaknesses
-
Why is some inconsequential text in the paper italicized? For example, lines 202–205, 234–237, etc.;
-
A standard algorithm should have inputs and outputs, so why does the authors' algorithm only have inputs and no outputs?
-
The paper mentions AlexNet and ResNet in several places but does not cite them, which shows that the authors' academic writing is poor, since these are classic technical papers;
-
The overall logical structure of the paper is poor and difficult to read.
Questions for authors
See Weaknesses.
4 comments
The author has made certain revisions to the issues I raised, but some problems still remain.
(1) Algorithm output: First, the algorithm should have an output; this should not require the reviewer to remind the author. Second, I believe that in the algorithm optimization process (i.e., after line 5 of the algorithm), there should be an output, rather than merely writing an output at line 2.
(2) Citations of well-known works such as ResNet, AlexNet, and ImageNet should not require the reviewer's reminder before being cited.
(3) Among the paper's references, ArXiv accounts for approximately one-third of the total, which is unprofessional.
(4) The paper cites only two formally published papers from 2026 (excluding 2026 ArXiv preprints).
(5) Lines 840–854, 893–910, 946–959, and other parts of the paper are written in a very confusing manner, making the reader feel quite dizzy.
Response to Reviewer cRDg's follow-up
Thank you for the follow-up. In response to your review and comment, we offer the following clarifications.
(1) Algorithm output. Algorithm 1 is an online procedure over an unbounded stream (line 4: for subject ). Its outputs are produced inside the loop, once per subject: line 7 delivers the output for subject (the completed reference when the subject is audited, the proposed early output otherwise), line 8 updates the cumulative-risk bound, and line 9 sets the fallback flag when an alarm fires. The Output line at the head of the algorithm lists these quantities, following the Input/Output convention of the algorithmic package. A return statement after a non-terminating loop would never execute. The delivery step has been in the loop since the submission; the revision added the Output line that summarises these outputs.
(2) Citations. The method sees only whether an early output agrees with the full one, so no claim, result or proof depends on which classifier fills a stage of the cascade. The submission identified the networks by name and torchvision weights and cited the dataset, Imagenette; the revision adds the references to their original papers.
(3) "ArXiv accounts for approximately one-third." The paper belongs to the literature on anytime-valid sequential inference and risk control, where the work closest to ours appeared in 2025 and 2026. The bibliography has 44 entries; 36 are journal or conference papers. The eight preprints (18%) are the GSM8K and Qwen2.5-Math technical reports, the authoritative descriptions of the dataset and model used in Section 5, and six 2025–2026 preprints that are the most recent work on adaptive acquisition, certified self-consistency and anytime-valid risk control. Four of the published papers (Annals of Applied Statistics 2025, ICLR 2022, ICML 2026, ICML 2026) also list their arXiv identifiers, as is standard; they are not preprints. The ICLR reviewer guidelines say authors "are strongly encouraged to cite" related arXiv work they are aware of.
(4) "Only two formally published papers from 2026." The two are the ICML 2026 papers closest to this work; the four 2026 preprints are more recent work on the same questions. The ICLR guidelines treat work published on or after 17 July 2026 as contemporaneous and do not require comparison with it; the number of references from a calendar year is not a review criterion. The follow-up names no paper that should have been cited. Points (2)–(4) concern citation conventions. The reviewer guidelines count "suggesting peripheral citations" among the points that "add to everyone's workload without changing the final conference program".
(5) Lines 840–854, 893–910 and 946–959. These are three passages of Appendix A: the remark comparing Proposition 5 with Theorem 2, the proof of Corollary 1, and parts (ii)–(iii) of the proof of Proposition 6. Two of the three (the remark and Proposition 6) were added during the discussion to answer Reviewer A4Fa's technical questions Q1 and Q3. These passages are written for verification; the results they establish are stated or summarised in Section 4.3. The follow-up does not identify a step that is unclear, incorrect or missing. If one is named, we will address it.
On the scores. The review form defines Soundness 1 as "significant concerns about the paper's soundness that do not appear easy to address" and Informativeness 1 as "I learned little or nothing new from this paper". Across the review and this follow-up, nine points have been raised on seven topics: italic theorem statements, an algorithm's Output line, citations, the share of preprints, the number of 2026 references, the paper's structure and the readability of three proof passages. None questions the correctness of the problem formulation, a proposition, a theorem, a proof or an experiment. Our response asked for the specific soundness concern; the follow-up names none. The review's own summary describes the paper's contributions (an auditing protocol, a detection-delay lower bound, an alarm method and risk bounds) and lists the experiments as a strength. We ask that the Soundness and Informativeness scores be reconsidered against their definitions.
Thank you for reconsidering your assessment and raising the score. We appreciate the time you have given to the paper during the review and discussion. We remain available for any further questions before the discussion period closes.
Response to Reviewer cRDg
Thank you for your review. We answer the four points in order.
W1 — italicised text at lines 202–205 and 234–237. These passages are the statements of Theorem 1 and Theorem 2, two of the paper's formal results. The theorem environment (amsthm, plain style) sets theorem bodies in italics, as it does for the propositions. They are not incidental text.
W2 — algorithm without outputs. Algorithm 1 now states its outputs explicitly: per subject, the delivered output (the full-budget reference if audited, the proposed early output otherwise) and the deployed rule; the alarm time, if any, and the fallback flag; and the cumulative-risk bound. Algorithm 2 in the appendix likewise.
W3 — missing citations. The revision cites ResNet (He et al., 2016), AlexNet (Krizhevsky et al., 2012), ConvNeXt (Liu et al., 2022), ImageNet (Deng et al., 2009) and PyTorch (Paszke et al., 2019) at or next to their first use.
W4 — structure and readability; "does not do a good job of explaining what task this is". The introduction now includes a concrete classifier-cascade example. The cascade returns an early confident prediction or continues to the final network; an audit completes the cascade on a randomly selected early stop, which reveals whether the early output disagreed with the full output and, at the same time, replaces what the user receives. The task is to monitor, after deployment, whether the rate of such disagreement exceeds its target, when the early stops themselves hide the information needed to check it. Section 4 opens with a roadmap and a short explanation of the observation and selection notation, and the results that the main text previously cited from the appendix are now stated in the main text. The global response lists all changes.
On the scores. The review assigns Soundness 1 ("significant concerns about the paper's soundness that do not appear easy to address") and Informativeness 1, but it raises no question about the problem formulation, the propositions, the theorems, the proofs or the experiments; its specific examples concern typography and citations, together with general remarks on readability and the presentation of the algorithm. We ask the reviewer to state the specific soundness concern so that we can address it within the discussion period, and we would welcome a reassessment of the scores in light of the revision.