acceptodds
Under review as a conference paper at ICLR 2027

Auditing Early Stopping after Deployment: Sequential Inference with Selective Feedback

Abstract

Changes after deployment can invalidate rules that stop computation or measurement once intermediate results appear sufficient. Early stopping then hides the completed output needed to check its decisions. We study disagreement between early and completed outputs, and show that some violations are statistically indistinguishable from deployments meeting the risk target using the remaining observations alone. Randomised completion restores feedback on selected early stops; under our protocol, it also replaces their proposed outputs at additional cost. We develop sequential inference for this coupling of selective observation and intervention. An information lower bound shows how audit frequency limits detection speed when pre-audit observations reveal no change. A likelihood-ratio alarm matches its inverse audit-frequency dependence under uniform auditing over the analysed risk range, with grid, restart and overshoot penalties. Correcting for selective observation yields anytime-valid upper bounds on average disagreement risk since deployment began under changing distributions. Alarms separately control deployment-wide false-alarm probability when conditional risk always meets its target. Experiments across magnetic resonance imaging, language-model sampling and image classification distinguish observable change from excess disagreement and examine whether returning the reference improves task outcomes. In the primary classifier comparison, adaptive auditing achieves shorter detection delay and lower delivered disagreement at a common normal-operation cost target, with higher post-change expenditure.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
early stopping, sequential inference, uncertainty quantification, selective feedback, confidence sequences, deployment monitoring, adaptive auditing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cRDgOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies how to monitor risk after early-stopping deployment: because early stopping hides the full output needed for verification, some risk violations cannot be detected without auditing. The authors propose an auditing protocol — randomly completing a subset of samples and delivering the full output to obtain feedback — and provide a detection-delay lower bound, an alarm method, and risk bounds. However, the paper does not do a good job of explaining what task this is, making it obscure and difficult to understand.

Critical strengths
  1. The paper does not exceed the page limit;

  2. The experiments are rich, and the supplementary material is substantial.

Critical weaknesses
  1. Why is some inconsequential text in the paper italicized? For example, lines 202–205, 234–237, etc.;

  2. A standard algorithm should have inputs and outputs, so why does the authors' algorithm only have inputs and no outputs?

  3. The paper mentions AlexNet and ResNet in several places but does not cite them, which shows that the authors' academic writing is poor, since these are classic technical papers;

  4. The overall logical structure of the paper is poor and difficult to read.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cRDg

Thank you for your review. We answer the four points in order.

W1 — italicised text at lines 202–205 and 234–237. These passages are the statements of Theorem 1 and Theorem 2, two of the paper's formal results. The theorem environment (amsthm, plain style) sets theorem bodies in italics, as it does for the propositions. They are not incidental text.

W2 — algorithm without outputs. Algorithm 1 now states its outputs explicitly: per subject, the delivered output (the full-budget reference if audited, the proposed early output otherwise) and the deployed rule; the alarm time, if any, and the fallback flag; and the cumulative-risk bound. Algorithm 2 in the appendix likewise.

W3 — missing citations. The revision cites ResNet (He et al., 2016), AlexNet (Krizhevsky et al., 2012), ConvNeXt (Liu et al., 2022), ImageNet (Deng et al., 2009) and PyTorch (Paszke et al., 2019) at or next to their first use.

W4 — structure and readability; "does not do a good job of explaining what task this is". The introduction now includes a concrete classifier-cascade example. The cascade returns an early confident prediction or continues to the final network; an audit completes the cascade on a randomly selected early stop, which reveals whether the early output disagreed with the full output and, at the same time, replaces what the user receives. The task is to monitor, after deployment, whether the rate of such disagreement exceeds its target, when the early stops themselves hide the information needed to check it. Section 4 opens with a roadmap and a short explanation of the observation and selection notation, and the results that the main text previously cited from the appendix are now stated in the main text. The global response lists all changes.

On the scores. The review assigns Soundness 1 ("significant concerns about the paper's soundness that do not appear easy to address") and Informativeness 1, but it raises no question about the problem formulation, the propositions, the theorems, the proofs or the experiments; its specific examples concern typography and citations, together with general remarks on readability and the presentation of the algorithm. We ask the reviewer to state the specific soundness concern so that we can address it within the discussion period, and we would welcome a reassessment of the scores in light of the revision.

Reviewer_cRDgOpenReview ↗

The author has made certain revisions to the issues I raised, but some problems still remain.

(1) Algorithm output: First, the algorithm should have an output; this should not require the reviewer to remind the author. Second, I believe that in the algorithm optimization process (i.e., after line 5 of the algorithm), there should be an output, rather than merely writing an output at line 2.

(2) Citations of well-known works such as ResNet, AlexNet, and ImageNet should not require the reviewer's reminder before being cited.

(3) Among the paper's references, ArXiv accounts for approximately one-third of the total, which is unprofessional.

(4) The paper cites only two formally published papers from 2026 (excluding 2026 ArXiv preprints).

(5) Lines 840–854, 893–910, 946–959, and other parts of the paper are written in a very confusing manner, making the reader feel quite dizzy.

AuthorsOpenReview ↗

Response to Reviewer cRDg's follow-up

Thank you for the follow-up. In response to your review and comment, we offer the following clarifications.

(1) Algorithm output. Algorithm 1 is an online procedure over an unbounded stream (line 4: for subject i=1,2,…i = 1, 2, \dots). Its outputs are produced inside the loop, once per subject: line 7 delivers the output for subject ii (the completed reference when the subject is audited, the proposed early output otherwise), line 8 updates the cumulative-risk bound, and line 9 sets the fallback flag when an alarm fires. The Output line at the head of the algorithm lists these quantities, following the Input/Output convention of the algorithmic package. A return statement after a non-terminating loop would never execute. The delivery step has been in the loop since the submission; the revision added the Output line that summarises these outputs.

(2) Citations. The method sees only whether an early output agrees with the full one, so no claim, result or proof depends on which classifier fills a stage of the cascade. The submission identified the networks by name and torchvision weights and cited the dataset, Imagenette; the revision adds the references to their original papers.

(3) "ArXiv accounts for approximately one-third." The paper belongs to the literature on anytime-valid sequential inference and risk control, where the work closest to ours appeared in 2025 and 2026. The bibliography has 44 entries; 36 are journal or conference papers. The eight preprints (18%) are the GSM8K and Qwen2.5-Math technical reports, the authoritative descriptions of the dataset and model used in Section 5, and six 2025–2026 preprints that are the most recent work on adaptive acquisition, certified self-consistency and anytime-valid risk control. Four of the published papers (Annals of Applied Statistics 2025, ICLR 2022, ICML 2026, ICML 2026) also list their arXiv identifiers, as is standard; they are not preprints. The ICLR reviewer guidelines say authors "are strongly encouraged to cite" related arXiv work they are aware of.

(4) "Only two formally published papers from 2026." The two are the ICML 2026 papers closest to this work; the four 2026 preprints are more recent work on the same questions. The ICLR guidelines treat work published on or after 17 July 2026 as contemporaneous and do not require comparison with it; the number of references from a calendar year is not a review criterion. The follow-up names no paper that should have been cited. Points (2)–(4) concern citation conventions. The reviewer guidelines count "suggesting peripheral citations" among the points that "add to everyone's workload without changing the final conference program".

(5) Lines 840–854, 893–910 and 946–959. These are three passages of Appendix A: the remark comparing Proposition 5 with Theorem 2, the proof of Corollary 1, and parts (ii)–(iii) of the proof of Proposition 6. Two of the three (the remark and Proposition 6) were added during the discussion to answer Reviewer A4Fa's technical questions Q1 and Q3. These passages are written for verification; the results they establish are stated or summarised in Section 4.3. The follow-up does not identify a step that is unclear, incorrect or missing. If one is named, we will address it.

On the scores. The review form defines Soundness 1 as "significant concerns about the paper's soundness that do not appear easy to address" and Informativeness 1 as "I learned little or nothing new from this paper". Across the review and this follow-up, nine points have been raised on seven topics: italic theorem statements, an algorithm's Output line, citations, the share of preprints, the number of 2026 references, the paper's structure and the readability of three proof passages. None questions the correctness of the problem formulation, a proposition, a theorem, a proof or an experiment. Our response asked for the specific soundness concern; the follow-up names none. The review's own summary describes the paper's contributions (an auditing protocol, a detection-delay lower bound, an alarm method and risk bounds) and lists the experiments as a strength. We ask that the Soundness and Informativeness scores be reconsidered against their definitions.

Thank you for reconsidering your assessment and raising the score. We appreciate the time you have given to the paper during the review and discussion. We remain available for any further questions before the discussion period closes.

Reviewer_A4FaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of post-deployment risk monitoring and intervention for early-stopping systems (such as early-exit neural classifiers, dynamic MRI acquisition, and self-consistency LLM sampling). Because early stopping censors the full-budget reference output, post-deployment distribution shifts or system faults that increase disagreement risk above a target α\alpha can be undetectable from pre-stop observations alone. The authors formalize a randomized auditing protocol that completes selected early stops with some nonzero probability, simultaneously revealing the reference loss and replacing the delivered output with the reference. They establish an information-theoretic detection delay lower bound for "silent shifts", construct a restarted discrete-grid likelihood-ratio supermartingale alarm with a matching delay upper bound, and derive an AIPW empirical-Bernstein confidence sequence for cumulative disagreement risk. Experiments across vision, MRI, and LLM tasks evaluate detection delay, delivered disagreement, and compute cost.

Critical strengths
  • Well-motivated problem and clean impossibility result: The observation that early-stopping guarantees (e.g., via Learn-then-Test or conformal risk control) can silently fail post-deployment because early exits censor their own ground-truth reference is practically important and cleanly formalized in Prop. 1.

  • Principled coupling of monitoring and intervention: Modeling an audit not just as passive label acquisition, but as an active intervention that replaces the delivered output with the full-budget reference nicely captures the operational trade-off between compute expenditure, detection speed, and error.

  • Information-theoretic lower bound and martingale alarm analysis: Prop. 2 and Thm. 1 provide tight bounds on detection delay when shifts are invisible prior to completion. A restarted grid-mixture LR supermartingale achieves this rate up to grid discretization and restart penalties.

Critical weaknesses
  • Dense presentation and missing formal statements: The prose is highly compressed and occasionally impenetrable. The main text repeatedly cites formal results ("Proposition 3," "Proposition 4," "Theorem 3" on pages 3, 5, and 6) that are never stated in the body of the paper. While these results can be found in the appendix (which runs to ~50 pgs), this should not be regarded as a loophole for evading page limits.

  • Thm 2 is loose and unused: Thm. 2 rescales the AIPW increments Zi∈[1−1/pmin⁡,1/pmin⁡]Z_i \in [1 - 1/p_{\min}, 1/p_{\min}] by the global worst-case range ω=2/pmin⁡−1\omega = 2/p_{\min} - 1. When pmin⁡=0.05p_{\min} = 0.05, ω=39\omega = 39, inflating the confidence bound across the entire trajectory even when pi≫pmin⁡p_i \gg p_{\min} or Ai=0A_i = 0, rather than using predictable step-specific propensity scaling. Because Rˉtdep\bar{R}_t^{\text{dep}} averages over all history since t=1t=1, a shift occurring after a long period of normal operation is heavily diluted. Furthermore, Utdep,EBU_t^{\text{dep,EB}} appears not to be evaluated or plotted anywhere in Sect. 5. Also, the grid V⊂(0,1)\mathcal{V} \subset (0, 1) is left completely unspecified in the theorem statement. More generally, this theorem statement is quite messy and tough to parse.

  • Mismatch between pre-subject LR selection and prefix-adaptive auditing: In Sect. 4.2, the LR alarm requires drawing Bi∼Bernoulli(pˉi)B_i \sim \text{Bernoulli}(\bar{p}_i) independently of the current subject's prefix Hi\mathcal{H}_i, and only includes Bi=1B_i = 1 subjects in the LR product. This has two drawbacks: (i) the LR alarm cannot exploit prefix-adaptive audit probabilities (unlike the AIPW betting alarm in Sect. 4.3), and (ii) on natural completions, where Li=0L_i = 0 is observed with certainty at zero extra audit cost, the LR alarm discards (1−pˉi)(1 - \bar{p}_i) of these free observations just to maintain Li∣Fi−1,Bi=1∼Bernoulli(Ri(λi))L_i \mid \mathcal{F}_{i-1}, B_i=1 \sim \text{Bernoulli}(R_i(\lambda_i)).

  • Permanent latching fallback assumption: Alg. 1 permanently latches the system to 100% full completion (λi+1=λ(0)\lambda_{i+1} = \lambda^{(0)}) upon a single alarm. In long-running deployments with transient shifts or false alarm budget δ=0.05\delta = 0.05, permanent fallback without recalibration or recovery is operationally restrictive.

Questions for authors
  1. Role and tightness of Thm. 2: Is the empirical-Bernstein confidence sequence Utdep,EBU_t^{\text{dep,EB}} from Thm. 2 actually used in any of the Sect. 5 experiments, or are all experimental interventions driven solely by the LR and AIPW betting alarms? At pmin⁡=0.05p_{\min} = 0.05 (ω=39\omega = 39) and N=3000N = 3000, is Utdep,EBU_t^{\text{dep,EB}} ever non-vacuous (<1< 1), and how is the grid V\mathcal{V} chosen?

  2. Discarding natural completions in the LR alarm: On natural completions, Li(λi)=0L_i(\lambda_i) = 0 is known deterministically. Instead of subsampling natural completions with Bi∼Bernoulli(pˉi)B_i \sim \text{Bernoulli}(\bar{p}_i) and discarding 1−pˉi1 - \bar{p}_i of them, why not condition the LR test on early stops (Si=1S_i = 1) while tracking the observable stopping rate Pr⁡(Si=1∣Fi−1)\Pr(S_i = 1 \mid \mathcal{F}_{i-1}) directly? Would that also allow prefix-adaptive auditing inside the LR alarm?

  3. Delay bound for the AIPW betting alarm: In Sect. 5, the best-performing policy ("Smooth both") relies heavily on combining the AIPW betting alarm with the LR alarm, yet Thm. 1 only analyzes the standalone LR alarm under non-adaptive pˉi≥p\bar{p}_i \ge p. Can the authors provide a detection delay bound for the AIPW betting alarm (or clarify why it empirically outperforms LR alone by such a wide margin in Table 1)?

  4. Presentation and self-containedness: Please either state Props. 3–4 and Thm. 3 concisely in the main text or rephrase the text so readers are not expected to know the contents of unstated theorems, and consider revising the overly compressed prose in Sections 4.2–4.5 for readability.

3 comments
AuthorsOpenReview ↗

Response to Reviewer A4Fa part 1

Thank you for a detailed and accurate review. We have made additions and changes on each point below, including two new results, Propositions 5 and 6, in answer to Q1 and Q3.

W1 / Q4 — formal statements missing from the main text; compressed prose.

The revision uses the tenth page to state in Section 4 the results that the submission only paraphrased: Proposition 3 (escalation preserves the guarantees with p_min = p_floor, with the paid-audit cost identity E[B_i S_i C_i] = E[p̄_i S_i C_i]), Proposition 4 (audit economics: the audit-coin variance term (1 − p)b/p, when prediction reduces it, and the retained saving fraction 1 − p), Theorem 3 (closed-loop certification under i.i.d. subjects) and Corollary 1 (validity of the betting alarm under arbitrary drift). Section 4 now opens with a short roadmap and a prose guide to the observation and selection notation, and the densest passages of Sections 4.1, 4.2, 4.4 and 5 were reworded. The section order, result numbers and proofs are unchanged, and the full statements remain in Appendix A. Within the page limit we did not rewrite Sections 4.2–4.5 wholesale.

W2 / Q1 — Theorem 2 is loose and unused; dilution; grid unspecified.

The revision addresses all three points. None of the interventions in Section 5 uses U^{dep,EB}; they are driven by the LR and betting alarms, and the text now says so. The grid is now stated immediately after Theorem 2, 𝒱 = {0.9, 0.75, 2^{−1}, …, 2^{−12}} (the union over 𝒱 costs log|𝒱| in the bound), and the bound's values are reported in Section 4.3 rather than only in the appendix. The theorem statement itself is unchanged, because it lists the measurability conditions on λ_i, p_i, f_i and c_i that the proof uses; the paragraph after it now says what the bound reports and how it is used.

On its role: Theorem 2 is a reporting bound on the deployment-wide cumulative risk under arbitrary drift, not an alarm. The dilution you describe is a property of the quantity it bounds: a shift after a long normal period moves the cumulative average slowly, and the bound tracks that average. Localised or late shifts are the job of the restarted alarms (Theorem 1 and Corollary 1), which is why the paper has both. At N = 3000 and p_min = 0.05/0.1/0.2 the bound is 0.112/0.068/0.046 (medians over the five stationary MRI streams, population risk 0.017): non-vacuous, but loose, as you say.

On step-specific scaling: Proposition 3 already lets the betting processes use the predictable per-step range [1 − 1/p̄_i, 1/p̄_i] under scheduled selection. Theorem 2 uses the global range because, with prefix-dependent (H_i-measurable) p_i, a step-specific range cannot be taken outside the conditional expectation given F_{i−1}. For the upper bound that the target requires, however, the signed range can be avoided altogether. The inverse-propensity increments Z_i = A_i L_i / p_i are nonnegative, and a one-sided Chernoff argument that uses only this sign and the upper range 1/p_min gives new Proposition 5 (Appendix A): with probability at least 1 − δ, simultaneously for all t,

Rˉtdep≤min⁡x∈XxZˉt+log⁡(∣X∣/δ)/(pmin⁡t)1−e−x,\bar R_t^{\mathrm{dep}} \le \min_{x\in\mathcal{X}} \frac{x \bar Z_t + \log(|\mathcal{X}|/\delta)/(p_{\min} t)}{1-e^{-x}},

for any finite 𝒳 ⊂ (0, ∞), where Z̄_t is the running mean of the IPW increments, under arbitrary drift and prefix-dependent p_i ≥ p_min. The proof is short: for each x, convexity of z ↦ e^{−p_min·x·z} on [0, 1/p_min] gives the chord bound e^{−p_min·x·z} ≤ 1 − p_min(1 − e^{−x}) z, so the product over subjects of exp(−p_min·x·Z_i + p_min(1 − e^{−x}) R_i) is a nonnegative supermartingale, and Ville's inequality with a union bound over 𝒳 finishes it. Here p_min enters only through the penalty, and subjects with A_i = 0 contribute nothing. The proposition covers IPW increments only; with a predictor the increments can be negative, and Theorem 2 remains the applicable bound (prediction affects its variance term, not its validity).

In numbers (Appendix H.3; δ = 0.05, T = 3000, 2,000 streams per cell, p_min = 0.05/0.1/0.2): under stationary risk 0.017 the medians are 0.063/0.049/0.038 for the new bound against 0.105/0.070/0.047 for Theorem 2; under a step from 0.02 to 0.3 at subject 1500 they are 0.278/0.240/0.222 against 0.327/0.263/0.227. Whole-path miscoverage over t ≤ 3000 was at most 0.018 for the new bound and at most 0.0005 for Theorem 2. The gain depends on the propensity schedule: with prefix-adaptive propensities p_i = p_min + (0.5 − p_min) L̂_i, driven by a noisy pre-audit loss predictor, the realised rate rises to 0.086/0.132/0.224, Theorem 2 profits from the smaller realised variance while the one-sided penalty depends on p_min alone, and the new bound is below Theorem 2 under stationary risk but above it under the step. The revision reports both bounds and does not claim that the new one dominates; Appendix H.3 also reports the narrower IPW-range comparator, which can be tighter even at the floor.

AuthorsOpenReview ↗

Response to Reviewer A4Fa part 2

W3 / Q2 — the LR alarm cannot use prefix-adaptive rates and discards natural completions.

The LR sample does omit some available zero losses; the reasons are as follows.

  1. Why the coin is drawn before the subject. Conditioning the LR test on early stops would require the stopping probability q_i = Pr(S_i = 1 | F_{i−1}), which is unknown and changes under a shift; a plug-in estimate inside the likelihood ratio breaks the supermartingale property. The pre-subject coin makes the LR alarm model-free: selection is independent of the current subject given the past, so L_i | F_{i−1}, B_i = 1 ~ Bernoulli(R_i) holds exactly whatever the stop rule does. For the same reason, prefix-adaptive (H_i-measurable) rates cannot enter the LR alarm; rates adapted to past evidence (F_{i−1}-measurable) can, and Proposition 3 covers them.

  2. Natural completions are not discarded by the system. They enter the AIPW increments with p_i = 1, hence the betting alarm and the cumulative bound; this is the channel through which prefix-adaptive auditing is exploited, and "both" uses both channels. The LR alarm also includes them whenever B_i = 1.

  3. Their information for the LR null is small. If q were known and the test conditioned on S_i = 1, with alternatives r/q against α/q, the information per subject would be q·kl(r/q, α/q), which exceeds kl(r, α) by 2.9% at 160 pixels (q = 0.69, r = 0.077) and 3.7% at 128 pixels (q = 0.708, r = 0.156), with q the pool's early-stop rate. This holds only for a known stop rate; a valid joint test that estimates it could be useful, and we do not establish one. Section 4.2 now explains the choice.

W4 — permanent latching.

The latch is the conservative default of a safety protocol, not a necessity of the analysis. False alarms are budgeted at δ for the whole deployment (Theorem 1(i), Corollary 1), so a latch under H_0 is a probability-δ event, not a recurring one; in the budget-calibrated experiments, pre-change alarms occurred in 3 of 24 policy–budget cells, once each among the 100 shared prefixes (Table 11). After fallback every subject is fully observed, so recovery is a certification problem: if a new episode can be justified as stationary, Theorem 3 can certify fixed replayable candidates on fresh data, with summable budgets for repeated episodes; an alarm does not itself establish stationarity. Appendix A discusses this extension after Algorithm 2. Recovery is neither implemented nor evaluated, and we do not claim its validity under arbitrary drift; the experiments keep the latch because their question was detection and delivery after a persistent change.

Q3 — delay bound for the betting alarm; why "both" beats LR by a wide margin.

New Proposition 6 (Appendix A) covers a fixed-bet version of the alarm: with constant rate p and bets η_g = p(r_g − α)/(α(1 − pα)) on the G-point alternative grid of Theorem 1, each factor 1 + η_g(Z_i − α) equals the likelihood ratio of Bernoulli(pr_g) against Bernoulli(pα) at the coarsened observation B_i L_i ∈ {0, 1}. The alarm is therefore an LR test on B_i L_i, and Theorem 1(ii)'s argument goes through with a new information term: for a change at subject s, with s′ the first restart at or after s and w′ its weight, E[(τ − s′)^+] ≤ (log(G/(δw′)) + log(r_g/α)) / J_g, where J_g = kl(pr, pα) − kl(pr, pr_g) replaces p·I_g = p[kl(r, α) − kl(r, r_g)]. For r ≥ r_g, J_g ≤ p·I_g: the alarm cannot tell an unselected subject from a selected one without disagreement; the gap is the information in the selection indicator. At α = 0.05 and p = 0.05 the ratio J_g/(p·I_g) at r = r_g is 0.94 for r_g = 0.1 and 0.91 for r_g = 0.2. The adaptive-bet AIPW alarm keeps the validity guarantee of Corollary 1, but its delay is not covered by this proposition, even under constant-rate auditing.

On the margin: under uniform auditing at the 0.03 target, "both" detects 36/100 streams and LR 34/100 (Table 14; Table 15 of the submission); the margin in Table 1 appears under escalation, where the betting alarm's evidence raises the audit rate earlier (post-change cost 0.665 against 0.507 with LR alone). This is consistent with a contribution from evidence-driven allocation but does not isolate it; Section 5.1 keeps that qualification. A direct comparison is now in Appendix H.4: on the 128-pixel streams, at the rates calibrated for uniform AIPW, the fixed-bet alarm detects 13/31/68 of 100 streams at targets 0.015/0.03/0.06, against 13/34/73 for LR, 2/29/65 for the adaptive-bet IPW alarm (at its own rates) and 9/46/94 for the adaptive-bet AIPW alarm; paired delay differences against LR include zero at every target, while the AIPW alarm is faster at the 0.06 target by 168 subjects [128, 209]. Under uniform auditing, then, the gain over LR appears only with the AIPW predictor: adaptive bets without it do no better than fixed bets, though fixed bets with the predictor were not tested. Does this address Q3? Any remaining point on W2 or W3 is welcome.

Thank you again for your careful reading and detailed feedback, which have helped us improve the paper. As the rebuttal period ends tomorrow, we would be grateful for any further thoughts on our responses and revisions. We would be happy to discuss any remaining questions or concerns.

Reviewer_npcROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to monitor early termination after deployment, when early termination results in the loss of all data necessary for assessing disagreement risk. The authors propose randomized audits to recover this missing information and develop sequential monitoring methods with theoretical guarantees. Experiments investigate the trade-off between detection performance, the level of disagreement detected, and computational cost.

Critical strengths
  1. The paper addresses an important problem that offline calibration does not cover. Proposition 1 formalizes the missing feedback problem, and the proposed auditing protocol captures the fact that obtaining feedback also changes the output delivered to users.
  2. The evaluation provides useful evidence for the proposed approach. Table 1 shows improvements in detection delay and delivered disagreement at a similar cost before the change. The authors also report the additional cost after the change and examine cases where full completion does not improve task accuracy.
Critical weaknesses
  1. The method has limited detection power for moderate risk violations. In Table 14, the smooth policy with both alarms detects only 8 out of 100 streams for the 160 pixel classifier change at the primary budget, and 7 out of 100 at the highest tested budget. Although the appendix investigates this behavior, the results raise concerns about the practical effectiveness of monitoring smaller increases in risk.
  2. The benefit of adaptive auditing is not fully characterized in terms of computation cost. In Table 1, the improvements from smooth escalation come with substantially higher cost after the change. The theoretical analysis also does not quantify the improvement in detection delay from increasing the audit rate. A clearer comparison of detection performance at comparable computation costs would strengthen the practical contribution.
Questions for authors
  1. For moderate risk violations such as the 160 pixel classifier change, could the authors clarify what audit budget or deployment horizon would be needed for reliable detection? This would help understand the practical limits of the approach.
  2. Could the authors provide more insight into the cost efficiency of adaptive auditing? In particular, how much of the improvement comes from the allocation strategy itself rather than the additional computation used after the change?
4 comments
AuthorsOpenReview ↗

Response to Reviewer npcR

Thank you for the two practical questions, on moderate violations and on the computation that adaptive auditing spends.

W1 / Q1 — what audit budget or horizon does a moderate violation need?

The low power at 160 pixels (Table 14 of the submission, now Table 13) is a real limitation. We extended this condition (rule risk 0.0771, target 0.05) to 4,500 post-change subjects, 100 streams per setting, at audit rates 0.05/0.1/0.25/0.5 (Section 5.1, Appendix H.2):

  • Uniform auditing with both alarms detects 5/25/52/88 of 100 streams; restricted mean delays (truncated at 4,500) are 4,438/4,047/3,281/2,547.
  • Smooth escalation from the same floors detects 36/46/60/88. Its rate is capped at 0.5, so the last setting coincides.
  • Adding {0.055, 0.06, 0.07, 0.08, 0.09} to the LR alarm's alternative grid {0.1, 0.2, 0.4, 0.8, 0.95} raises these to 7/25/56/91 and 39/48/66/91.

So about nine streams in ten are detected within 4,500 subjects at rate 0.5, at a substantial price: pre-change extra computation is about 0.295 of full completion, against 0.030 at rate 0.05. No pre-change or control alarm occurred. These figures do not guarantee 90% power elsewhere, and the settings share the same 100 streams, so the 1,600 (policy, stream) pairs per condition are not independent replications.

Proposition 2 explains the price. It is an illustrative benchmark for a silent shift, not a lower bound for these observable changes; but a valid alarm cannot treat a visible change as evidence of excess disagreement, since within-target changes must leave it silent, so the evidence has to come from audited losses. At risk 0.0771, Proposition 2 prices that evidence at about 7,104/3,552/1,421/710 subjects for audit-probability caps 0.05/0.1/0.25/0.5 (90% power, false-alarm probability 0.05); a floor on adaptive auditing is not such a cap. Moderate violations thus stay hard at the lower budgets tested. If a particular budget, horizon or change magnitude would answer Q1 better, we can add it within the discussion period.

The windowed KS control answers a different question: it is a label-free detector of observable change and also fires on the within-target AlexNet replacement, so its greater sensitivity at 160 pixels does not come with control of false alarms for excess disagreement.

W2 / Q2 — better allocation, or simply more computation after the change?

We agree: the primary comparison matches the cost of normal operation, not post-change or total cost, and the lower post-change disagreement in Table 1 comes with greater post-change expenditure. Across the tested 128-pixel policies, mean post-change cost and disagreement lie close to a line of slope −0.228 (largest audited-policy residual 0.0011). This is an empirical pattern, not an identity: allocations can select different combinations of loss and remaining cost.

To separate the two, we added a comparison with constant random completion at matched total cost (Appendix H.1). For each audited cell we interpolate a completion rate (no alarms, no fallback) whose whole-stream mean cost matches, and evaluate it on the same 100 paired streams. For smooth escalation with both alarms at the primary 128-pixel target the costs are 0.549794 against 0.549737, and the audited policy delivers less disagreement: by 0.015 [0.011, 0.018] over the whole stream and 0.035 [0.027, 0.042] after the change, with whole-stream task error lower by 0.012 [0.009, 0.014]. Across all 96 matched cells the largest cost mismatch is 0.000374. The comparison is descriptive: the matching rate is fitted on the same streams and held fixed in the paired bootstrap, so the intervals omit that uncertainty. At 160 pixels the primary contrasts are within ±0.001, with intervals containing zero. Appendix H.1 reports all cells, including the control and the AlexNet replacement, and Figure 6 the post-change cost relationships.

Random completion has no alarm, so this comparison measures delivered outcomes, not detection; the detection comparison at a common pre-change rate is the W1 follow-up above (36/46/60/88 against 5/25/52/88). A separate interpolation estimates that uniform auditing would need 3.33 times the pre-change extra cost to reach smooth escalation's post-change disagreement of 0.097 (also descriptive).

Together, the results support a benefit from when completion is purchased at 128 pixels, while the original qualification stands: adaptive auditing spends more after the change. On the theory side, Theorem 1(ii) bounds the expected delay by a restart-and-grid constant divided by p·I_g (audit rate times information per audited subject), so it scales as 1/p; new Proposition 6 gives the analogous bound for a fixed-bet betting alarm; and Proposition 3 preserves validity under escalation with p = p_floor. None of these quantifies the gain from moving a fixed budget toward suspicious periods; the matched-cost comparison measures it descriptively, and we keep that limitation explicit.

Thank you again for your helpful feedback. As the rebuttal period ends tomorrow, we would appreciate any further thoughts on our response and additional experiments. We would be happy to address any remaining questions.

Thank you for the response. The additional experiments clarify the detection limits for moderate risk violations and provide useful evidence for the benefits of adaptive auditing at matched cost. My main concerns have been addressed, so I will keep my original score.

Thank you for your follow-up. We are glad that the additional experiments have addressed your main concerns, and we appreciate your constructive feedback throughout the discussion.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to all reviewers: summary of the revision

We thank the reviewers and the Area Chair for their time and their comments.

What the reviews recognise. Reviewer npcR finds the problem important and not covered by offline calibration, Proposition 1 a formalisation of the missing-feedback problem, the protocol faithful to the fact that obtaining feedback changes the delivered output, and the evaluation useful evidence, including our reporting of post-change cost and of cases where completion does not improve task accuracy. Reviewer A4Fa finds the problem well motivated, Proposition 1 a clean impossibility result, the coupling of monitoring and intervention principled, and Proposition 2 and Theorem 1 tight delay bounds that the restarted LR alarm matches up to grid and restart penalties. Reviewer cRDg notes that the experiments are rich and the supplementary material substantial. The questions the reviews raise concern the scope of the evidence, the role and tightness of one bound, and the presentation; we address each of them below.

The concerns and our answers. Reviewers npcR and A4Fa raise six points: the power of auditing against moderate violations, the accounting of post-change computation, the tightness and role of the cumulative bound (Theorem 2), a design choice in the LR alarm, the permanent latch, and the density of the prose. Each is answered in the individual responses, by a new analysis in the appendix or by a clarification in the text. In brief:

  • Moderate violations (npcR W1/Q1): a longer-horizon follow-up (Appendix H.2) measures detection against audit budget on the 160-pixel change. Reliable detection needs a high budget, and we explain why any alarm that is valid under the disagreement target must pay this price: a visible change is not by itself evidence of excess disagreement, because changes that stay within the target must leave the alarm silent; the evidence has to come from audited losses, and Proposition 2 prices it.
  • Cost accounting (npcR W2/Q2): a comparison at matched total cost (Appendix H.1) shows a gain from when completion is spent, 0.015 [0.011, 0.018] in whole-stream disagreement; we label it descriptive, and Table 1 continues to report the higher post-change expenditure.
  • Theorem 2 (A4Fa W2/Q1): the grid and the role of the bound are now stated next to the theorem, and new Proposition 5 gives a one-sided bound for IPW increments, compared with Theorem 2 in Appendix H.3.
  • Betting alarm (A4Fa Q3): new Proposition 6 bounds the delay of a fixed-bet betting alarm, and Appendix H.4 compares that alarm with the adaptive one used in the experiments.
  • LR design and latching (A4Fa W3/Q2, W4): the reasons for the pre-subject selection and for the latch are given in the response; recovery after fallback is discussed after Algorithm 2 as a possible extension.
  • Presentation (A4Fa W1/Q4; cRDg): Corollary 1, Propositions 3–4 and Theorem 3 are now stated in Section 4; the introduction has a concrete cascade example and Section 4 a roadmap and a guide to the notation; the presentation of the formal statements and of the algorithms was refined, and the references completed.

Reviewer cRDg raises no question about the soundness of the method. The four points concern the italic typesetting of the theorem statements, the presentation of the algorithm, the citations and the readability. The first is the standard amsthm style for formal statements; the others concern wording and the presentation of formal material, which the revision has improved as described above. We have asked the reviewer to state any specific soundness concern so that we can address it within the discussion period.

Conventions. Added passages and new statements are blue; sentence-level rewording is not marked. The title, abstract, section order, original results and the numbering of theorems, propositions and corollaries are unchanged. Tables 1–4 and 6–12 keep their numbers; later appendix tables and figures are renumbered (Table 14 of the submission is now Table 13). Page numbers below refer to the revised PDF.

Where the changes are.

  • Introduction, p. 1: cascade example. Sections 4–4.2, pp. 3–4: roadmap, notation guide, natural completions.
  • Section 4.3, pp. 5–6: grid and role of Theorem 2, MRI values (p. 5); Corollary 1 (p. 6). Sections 4.4–4.5, pp. 6–7: Propositions 3–4, Theorem 3, outputs of Algorithm 1.
  • Sections 5.1–5.2, pp. 8–9: follow-up detection results; matched-cost comparison. Limitations, p. 10: scope of Proposition 6.
  • Appendix A, pp. 14–21: restatements with proofs; Propositions 5–6; outputs of Algorithm 2 (p. 20); recovery (p. 21).
  • Appendix H, pp. 48–56: H.1 matched cost, H.2 moderate violations, H.3 bound comparison, H.4 fixed-bet alarm; Figure 6, Tables 28–36.
Area_Chair_E1K9OpenReview ↗

Regarding the paper scope

Dear authors,

Thanks for your submission. I quickly read the reviews and may have some clarifications question. Basically I may miss important detail. If you can clarify these in plain language, it's a great appreciation.

  • Changes after deployment can invalidate rules that stop computation or measurement once intermediate results appear sufficient.

I would like to ask

  • what does it mean by deployment in practice here? can you provide a concrete context a change after deployment ?
  • what are we focusing for the changes?
  • invalidate rules that stop computation or measurement once intermediate results appear sufficient. Can you explain what does it mean?
  • what are concrete applications that need this mechanism?
  • how/what the experiments want to justify?

Thank you for the questions. In short: early stopping saves cost by skipping the full computation, and after deployment that same skip removes the comparison needed to check that the stopping rule still meets its target. The paper recovers it by completing a random fraction of cases, with guarantees, and measures the cost. Throughout, the target is agreement between the early and the full output, not the correctness of either.

(1) What "deployment" means here, with one concrete change. In the paper's language-model system, a maths question-answering service can draw up to 64 solutions from the model and return the majority answer; all 64 is the full output. A stopping rule stops after a few solutions once they agree. Before deployment the rule is calibrated on fully sampled questions so that the early answer differs from the full output on at most 5% of them. Deployment is the routine use that follows: the rule runs on new questions, and for most of them the remaining solutions are never drawn. One change we study: a serving update caps output at 256 tokens. Long solutions are cut off before their final answer (57% of them), and the early answer now differs from the full output on 13% of questions.

(2) What changes do we study? Every change is judged by one quantity: how often the early output disagrees with the full output, against the calibrated target. The experiments illustrate three cases. Visible but within target: routing questions to a general-purpose model shifts the signals a monitor can see, yet disagreement stays at 4%. Visible and harmful: the token cap above. Harmful and silent: in our MRI and classifier experiments, a bug in the delivery step (the image from one measurement instead of the two the rule checked; a wrong label from a lookup table) leaves every signal available to a monitor unchanged, while most delivered outputs differ from the full output. Proposition 1 proves that no monitor restricted to the recorded data can fix this: some violations produce recorded data with exactly the distribution of a compliant deployment.

(3) "Invalidate rules that stop computation or measurement once intermediate results appear sufficient." The rule stops when the intermediate results look sufficient, on the calibrated promise that the full output would almost always agree. After such a change the results still look sufficient and the rule keeps stopping, but the early output now differs from the full output far more often. Nothing in the system shows it: the questions that stopped early never had their remaining solutions drawn, so the disagreement is never observed. Our remedy is to audit: for a random 5 to 10% of the questions the rule wants to stop, draw the remaining solutions anyway. Each audit reveals whether the two answers disagree, and the user gets the full answer. The audits yield three guarantees: a running upper bound on the disagreement rate since deployment, valid at every time under our audit protocol whatever the changes in data or rule; an alarm with false-alarm probability controlled over the whole deployment, which triggers full computation; and a limit on how fast any alarm can detect such a silent change at a given audit fraction, which our alarm matches up to constant factors.

(4) Applications. Any system that saves cost by stopping early and is judged against the full output. Three in the paper: adaptive MRI acquisition, a scan built from repeated measurements (repetitions, averaged to suppress noise) that can stop once the image is stable, with disagreement meaning the early image departs from the full image by more than a preset error tolerance; early termination of sampling or reasoning in language-model inference; and early-exit classifiers, which deliver a small network's label when it is confident. In all three the full output is expensive and the pipeline keeps changing; in the deployments we study, no other feedback arrives.

(5) What the experiments show. Change and harm are different targets: a policy that falls back to full computation whenever it sees a change spends up to 67% of the full budget on changes that were within target. Audits catch harm at small cost: in the three experiments where the change is invisible to the baseline monitor (the delivery bugs of (2)), alarm-driven auditing cuts the delivered disagreement over the whole run from 11 to 35% to 3 to 4% while still saving 16 to 36% of the full computation; a pre-registered check of nine real serving configurations raises no alarm on the eight within target and detects the one that is not. Directing audits to where evidence accumulates helps: at the same normal-operation cost it detects faster and delivers less disagreement than a fixed rate, though it spends more after the change. The paper also states its limits: a moderate violation (7.7% against the 5% target) needs many audits.

We are glad to clarify anything further before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.