acceptodds
Under review as a conference paper at ICLR 2027

Bootstrap-Zoom: Inference after Checkpoint Selection

Abstract

Validation data are often used both to select a checkpoint and to estimate its population risk. This reuse can invalidate ordinary confidence intervals, especially when many correlated checkpoints compete. Bootstrap-Zoom extends the zoom correction by replacing its known joint tail with a studentized multiplier-bootstrap tail estimated from the validation loss matrix used for selection. Because the tail, the variance scales, the observed gaps, and the selected checkpoint all depend on one sample, approximation guarantees for fixed regions do not by themselves justify the resulting confidence set. Conditional on a training path fixed before validation, we bound the probability of abstaining or covering from below, accounting for this dependence, finitely many bootstrap draws, and grid discretization. When population standard deviations are bounded away from zero and the checkpoint count and grid satisfy growth conditions, we derive coverage-deficit rates at both positive and zero variance floors. An exact endpoint search returns the same interval as testing every grid value, without assuming that the accepted set is connected. On seven combinations of training setting and loss, we separate the width saved by checkpoint dependence from the width saved by gap discounts. Selected-risk overcoverage can coexist with joint undercoverage, and a rare-loss counterexample shows that positive empirical variances alone do not justify Gaussian calibration.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
post-selection inference, checkpoint selection, multiplier bootstrap, uncertainty quantification, high-dimensional statistics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vNUHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Choosing the checkpoint with the lowest validation loss favors checkpoints that benefited from random noise. The Zoom algorithm (2025) accounts for this selection when constructing a confidence interval for the chosen checkpoint’s true loss. It reduces the influence of worse checkpoints by counting their errors only when they exceed certain thresholds, which needs a bound on the probability that any checkpoint’s error exceeds its threshold.

B-Zoom aims to replace this bound with a studentized multiplier-bootstrap estimate. Their algorithm cleverly maintains the empirical structure and dependence of the samples during bootstrap, and comes with asymptotic guarantees. The main contribution is establishing conditions under which this correction is justified despite using the same validation data both to select the checkpoint and to estimate its uncertainty.

Critical strengths
  1. The theory addresses a gap in applying the zoom correction. A bootstrap approximation for fixed regions does not justify a confidence region chosen from the same sample. The proof handles this dependence by comparing against a region fixed by training information. It explains when estimating the correction from the selection data is justified, rather than assuming that a standard bootstrap result is enough.
Critical weaknesses
  1. Unclear Writing One of the biggest weaknesses of the paper is the poor writing. The problem is not introduced and motivated sufficiently, and the paper delves into comparisons from the first paragraph. For example, Section 2.3 introduces the gap discounts, plausible-reference set, maximum radius, and final interval very quickly. However, the explanation and interpretation is missing, or left in the appendix (eg. Equation 1).

The paper does not explain the existing zoom method well, and assumes knowledge of it. This makes it hard to follow and assess what novel is being proposed in this paper.

  1. The asymptotic guarantee provides limited guidance for practical use. The coverage result is useful as an asymptotic validity result. However, it does not establish how large the validation sample must be before the intervals are reliable. The main theorem applies for sufficiently large samples and has an unspecified constant to the rate. The rare-loss example makes this limitation concrete: coverage is approximately 63% at nominal 90%, showing why asymptotic validity alone is not practically useful.
Questions for authors

Can you clearly separate the new algorithmic steps and proof arguments from those inherited from Zoom and existing bootstrap theory? What theoretical and empirical benefit does each new component provide?

Beyond asymptotic validity, what usable guidance does the theory provide for choosing the validation sample size and grid resolution? How should a practitioner assess whether the default procedure is reliable?

4 comments
AuthorsOpenReview ↗

Response to Reviewer vNUH (Part 1/2)

Thank you for your review. In response to your comments, we restructured the introduction and Section 2, and added finite-sample guidance in Section 4.5 and Appendix I.

Below, KK is the number of checkpoints, mm the number of validation labels, and coverage is at nominal 0.90. A configuration is a setting with a (K,m)(K, m) pair, and a replication is one validation sample.

W1. Motivation, Section 2.3 and the zoom method

  • Introduction. It now opens on the problem and the target before any comparison: 3000 checkpoints and 100 labels that must both pick a checkpoint and estimate its error, where the usual 90% interval covers the selected checkpoint's risk 64% of the time and B-Zoom 96% (Section 1).
  • The zoom method from first principles. Section 2.2 runs from selection bias through Bonferroni (radius 4.15 at K=3000K = 3000) and the simultaneous band (median radius 3.34 on the same checkpoints) to the zoom correction of Zrnic & Fithian (2025) which tests one candidate value at a time, discounts checkpoints by their gaps, searches over population winners and assumes a known joint tail.
  • Former Section 2.3 is now Sections 2.3–2.4. The test comes first, in five plain steps, and each formula is then explained in words. Under Equation (1), the discounts are traced to the joint confidence set of Zrnic & Fithian (2025), and with equal scales a discount is one third of the gap in standard errors. Appendix D.5 derives the general formula. A short "Proof idea" paragraph explains why B-Zoom covers.
  • Worked example. Figure 1 runs the five steps on three checkpoints, and Appendix C (Table 5) tabulates every step.

Q1. Inherited versus new, and the benefit of each new component

The band keeps the savings from correlation (7.77–25.98% of Bonferroni's mean width, Table 8) but has no gap discounts, and the zoom correction with a union-bound tail has the discounts but gives up the correlation. B-Zoom’s estimated tail keeps both: it is 14–29% narrower than Bonferroni on each setting, and the discounts save a further 2.95–12.36% beyond the band, more than dependence at m=5000m = 5000 on six of seven settings (Section 4.2). In our experiments, B-Zoom's width falls from 1.77 to 1.32 times the uncorrected width on the main setting as mm grows from 100 to 5000, while the band stays at 1.83 to 1.84 times it. Table 1 (Section 3), with the measured effect of each addition:

Building block (source)What B-Zoom addsWhat the 0.50% buys
Zoom test, discounts, inversion (Zrnic & Fithian, 2025)Tail, scales and discounts estimated from the selecting sampleBoth savings at once (Table 3)
Multiplier bootstrap for maxima (Chernozhukov et al., 2013; 2019; 2022)Coverage for the data-dependent rectangle when the same sample selects k^\hat k, at the leading rateTheorem 1, uniform over near ties
Search over population winners (Zrnic & Fithian, 2025, Section 5.4)References only in W(t)W(t), no separate level0.50% of mean width, with both covering 0.987 on 9000 replications. A separate level β=α/10\beta = \alpha/10 instead costs 1.60% (Section 4.4)
Monte Carlo rank (Besag & Clifford, 1989; Dufour, 2006)The bound holds for any finite number of draws B≥⌈1/α⌉−1B \ge \lceil 1/\alpha \rceil - 1, with no Monte Carlo termNo Monte Carlo error to budget for
Estimated scalesNo variance floor: dropping it lowers the coverage bound by at most 1/m1/mNo tuning constant. B-Zoom returned an interval in all 84,000 replications, while a factorized zoom baseline declines 1820 of 9000 with a constant loss column (Section 4.4)
Grid inversion (Zrnic & Fithian, 2025)An automatic grid rule that meets Theorem 1's grid condition (Proposition 2)The reported intervals come from a rule the theorem covers

New proof arguments (Section 3 and Appendix D.1). Regions chosen by the data lose coverage even for exact Gaussians: each of two independent standard Gaussians lies in [−z0.95,z0.95][-z_{0.95}, z_{0.95}] with probability 0.90, but the one larger in absolute value does so with probability only 0.81 (Section 1). The proof therefore compares the errors with a rectangle that training fixes (Lemma 5), extending the studentization step of Chernozhukov et al. (2019) from one common threshold to sides set by each checkpoint's scale and discount. An anti-concentration bound for Gaussian shells (Lemma 7) controls the coverage lost to the data-chosen rectangle. Under the maximum over W(t)W(t), the accepted set contains every value the joint confidence set allows (Theorem 11), so screening reserves no error level. Appendix D.1 lays this out before the details, with the coverage chain labeled lemma by lemma, a table of error terms (Table 6) and a dependency diagram (Figure 6).

AuthorsOpenReview ↗

Response to Reviewer vNUH (Part 2/2)

W2 and Q2. Practical guidance beyond asymptotic validity

Theorem 1 is asymptotic, so we measure (Section 4.5, Appendix I) where its Gaussian approximation holds at finite mm. They give checks that can be run before seeing the data, and a valid fallback when the checks fail.

Rare losses: count the expected rare events. On designs with two identical loss columns, which test the Gaussian approximation without selection, coverage depends mainly on the expected rare count mpmp. For 0–1 loss with rare errors this is the expected number of validation errors. Over the computed range, an approximate binomial calculation that includes grid rounding gives coverage of at least 0.88 once mp≥22mp \ge 22 and 0.89 once mp≥41mp \ge 41 (Appendix I.2, Table 27).

The 63% example that you raise is at p=0.01p = 0.01 and m=100m = 100, so mp=1mp = 1, inside the region this check flags. With probability (1−p)m(1-p)^m no rare loss appears, and then the exact-tail interval lies below the risk, so its coverage is capped at 1−(1−p)m=0.6341-(1-p)^m = 0.634. B-Zoom covers 0.626 (Table 28).

Other bounded losses: an empirical diagnostic. On the two-column designs, of the 70 configurations, all 15 with population Berry–Esseen ratio E∣L−μ∣3/(σ3m)E|L-\mu|^3/(\sigma^3\sqrt m) at most 0.15 cover at least 0.89 (Appendix I.4, Figure 11). The ratio is at most M/(σm)M/(\sigma\sqrt m), so a lower bound on σ\sigma fixed before the labels are drawn suffices. For 0–1 loss with error rate pp the check reads mp(1−p)≥45mp(1-p) \ge 45, and since p(1−p)p(1-p) increases for p<1/2p < 1/2, an underestimate of pp, as a training error rate usually is, errs toward the fallback. The check flags heavy tails: the capped Pareto law with a=1.1a = 1.1 has ratio 0.68 at m=5000m = 5000 and covers 0.798 there (Table 30).

Where the real-data settings fall (Section 4.5, Appendix I.4, Table 29, Figure 11). With the held-out pool as the population, the 64 real-data configurations with m≥1000m \ge 1000 or 0–1 loss have ratio at most 0.141 and cover 0.934 to 0.998. The other 20, the five cross-entropy settings at m=100m = 100, have median ratios 0.18 to 0.40 and cover 0.899 to 0.990, at least as much as the two-column configurations of similar ratio. On the error settings the rarer outcome's expected count is at least 25.9 in every replication, above the mp≥22mp \ge 22 rule.

Under selection, coverage is measured directly. B-Zoom covers at least 0.899 in all 84 real-data configurations (Section 4.1), and at least 0.916 on the near-tie design at KK up to 10,000 and mm down to 25 (Appendix E.3).

Choices that do not matter in practice. Proposition 2's grid rule meets Theorem 1's grid condition knowing only the loss bound MM, and 400, 600 and 4000 points agree in coverage to 0.002 in every tested configuration (Appendix G). Theorem 1 holds for any B≥⌈1/α⌉−1B \ge \lceil 1/\alpha\rceil - 1 with no Monte Carlo term, and BB from 500 to 5000 gives the same coverage (Appendix F.7, Table 21).

When a check fails. Hoeffding and empirical-Bernstein union-bound intervals hold at every mm for every loss law on [0,M][0, M], and both cover at least 0.999 in every configuration. At p=0.01p = 0.01 and m=100m = 100, the configuration the check flags, they are 5.8 and 5.4 times B-Zoom's width (Table 28). Once mp≥30mp \ge 30 the ratio is 2.1 to 6.3 (Section 4.5, Table 31). The choice is made before seeing the data, since a rule that switches after reading them inherits neither guarantee (Appendix I.1).

Every change described above is in the revised PDF. If any point still seems unresolved, we would be glad to discuss it further, and we hope you will take the revision into account in your final assessment and revise your score.

AuthorsOpenReview ↗

Follow-up for Reviewer vNUH

Dear Reviewer vNUH, Thank you again for your time and thoughtful review. We wanted to politely follow up on our rebuttal above. We would greatly appreciate it if you could let us know whether our responses and revisions resolve your concerns, or if there are any remaining questions we can clarify before the deadline. Best regards, The Authors

Thank you for your response and the experiments. I still believe that the paper is very hard to follow, and the jargon does not help clarify the issues I had raised earlier. I have decided to keep my score.

Reviewer_C5PMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem of generating confidence intervals for the population risk of the model when the model is selected out of many possible ones based on performance on some validation set and only the empirical performance on the validation set is available. Naturally, since the model is selected based on validation set performance, using the validation set performance as population estimate is overly optimistic. The paper proposed “’Bootstrap-Zoom (B-Zoom)”, a method based on the multiplier bootstrap to construct (asymptotically valid CI).

Critical strengths

Empirically, the proposed method seems to yield valid coverage and narrower CI than other methods

Critical weaknesses
  1. The paper would firstly benefit from examples showing why the problem under consideration is relevant and motivating the need for the proposal better. Use of test sets is prevalent in machine learning with the entire goal of providing an honest estimate of model risk when validation is used for hyperparameter-tuning/ early-stopping. Presumably, using this method could do away with the need for a test set but it is unclear if the constructed CI will be sharp enough for it to be useful for a practitioner.
  2. The paper also needs a background section discussing the gap discount idea and comparing the Zrnic and Fithian (2025) which is central to this paper. Background around multiple testing corrections and the “band” approach used in the experiments is also needed. This would help make clear the specific contributions of the paper.
  3. Section 2.3 needs a heavy rework in terms of providing intuition behind why the method works and what the key ideas are. Currently, I have little to no intuition behind why the method works.
  4. The experiment design is also poorly explained and limited in scope.

Overall, there may be useful technical novelty in the paper, but it does a poor job of explaining it and motivating its use.

Questions for authors

n/a

4 comments
AuthorsOpenReview ↗

Response to Reviewer C5PM (Part 1/2)

Thank you for your review. Your points on motivation, background and intuition led us to rewrite the opening of Section 1 and much of Section 2.

Below, KK is the number of checkpoints, mm the number of validation labels, and coverage is at nominal 0.90. A configuration is a setting with a (K,m)(K, m) pair, and a replication is one validation sample.

W1. Why the problem matters, and why not a test set

The use case now opens the paper (Section 1). A practitioner saves K=3000K = 3000 checkpoints of an image classifier and has m=100m = 100 labeled images to both pick a checkpoint and estimate its error. On CIFAR-10 runs whose risk reaches its minimum early in training, the usual 90% interval covers the selected checkpoint's risk only 64% of the time, and B-Zoom covers 96%. The same situation arises whenever the candidates are fixed before validation, such as the checkpoints of a completed run or models trained over prespecified seeds or hyperparameters. Labels can be this scarce, as in few-shot model selection (Perez et al., 2021). Picking the best of all checkpoints saved during a completed run is within the guarantee. Validation-based early stopping uses the labels during training and falls outside it (Section 2.1).

The test set. With mm labels in total, a held-out test set is sample splitting: select on some labels and report on the rest. We run five allocations (Section 4.3 and Appendix H) on all 84,000 real-data replications and score each choice by its selection regret, the excess population risk of the selected checkpoint over the best of the KK.

  • At every allocation from 20/80 to 80/20, splitting selects a worse checkpoint on average than B-Zoom in all 21 combinations of setting and mm, and the seed-paired 95% interval for the regret difference lies above zero in all 105 comparisons.
  • At matched mean width, sample splitting has 1.33 to 3.51 times B-Zoom's selection regret (Table 3).
  • A 50/50 split's interval is narrower than B-Zoom's in 16 of the 21 combinations, but its median regret ratio is 1.97, and on the two 0–1 error settings at m=100m = 100 its checkpoint's error rate is 1.4 to 1.6 percentage points higher.
  • B-Zoom covers 0.906 to 0.992 in all 21 combinations (Appendix H).

Using all mm labels for both selection and inference thus gives a better checkpoint, with an interval that remains asymptotically valid.

Width relative to the test-set interval. At m=5000m = 5000 on the main setting, B-Zoom's mean width, 0.062, is below the 0.067 of a 50/50 test-set interval, and its checkpoint is the full-sample winner. B-Zoom is the narrowest of the three corrected intervals in every row of Table 11. On the main setting its width falls from 1.77 to 1.32 times the uncorrected width as mm grows from 100 to 5000, while the band stays at 1.83 to 1.84 times it (Section 4.2, Table 3).

Width on the 0-1 error settings (Section 4.1, Table 12). On the two 0–1 error settings at m=5000m = 5000, B-Zoom's mean width is 3.3 and 3.5 percentage points, against 2.1 and 2.3 for the uncorrected interval, which covers only 0.869 and 0.886. At m=1000m = 1000 it is 8.2 and 8.7 points against 4.6 and 5.2, with uncorrected coverage 0.849 and 0.862. B-Zoom covers at least 0.984 in these four rows. A 50/50 test-set interval is itself 1.41 to 1.44 times the uncorrected width at m≥1000m \ge 1000 (Appendix E.1).

Coverage is well above nominal here because each corrected interval covers the selected risk through a confidence set for every risk (Appendix E.2).

W2. Background on the gap discount, Zrnic & Fithian (2025), multiple testing and the band

Section 2.2 now introduces the method in four steps.

  1. Selection bias. The smallest validation loss favors checkpoints whose error happens to be negative, so the uncorrected interval is too optimistic.
  2. Bonferroni covers every risk at once. At K=3000K = 3000 and α=0.10\alpha = 0.10 its radius is 4.15 standard errors.
  3. The simultaneous band sets the radius from the maximum of the correlated errors. For 3000 CIFAR-10 checkpoints at m=1000m = 1000 its median radius is 3.34, which Bonferroni reaches only with about 120 checkpoints (Section 1).
  4. The zoom correction of Zrnic & Fithian (2025) tests one candidate value tt of the selected risk at a time. A checkpoint far above tt could win only through a large error, so its error counts only beyond a gap discount. The correction needs a known bound on the joint tail of the errors, and a union bound supplies it at the cost of the correlation savings.

The subsection ends with what B-Zoom changes: it estimates that tail with a multiplier bootstrap on the same loss matrix, so it saves width through correlation, as the band does, and through gap discounts, as the zoom correction does. Dependence saves 7.77 to 25.98% of Bonferroni's mean width and the discounts a further 2.95 to 12.36% (Section 4.2, Table 8).

AuthorsOpenReview ↗

Response to Reviewer C5PM (Part 2/2)

Here we continue our response to some of the other points raised.

W3. Intuition for why the method works

Section 2.3 opens with what the bootstrap does: one Gaussian weight per validation example, shared by every checkpoint, so the simulated errors inherit the correlations between checkpoints. Section 2.4 states the test in five plain steps: (1) fix a candidate value tt, (2) discount each checkpoint whose validation loss lies above tt, in proportion to the gap, (3) find the plausible population winners, whose band interval reaches down to tt, (4) take as threshold the largest bootstrap radius under the discounts these winners imply, (5) accept tt if the selected checkpoint's loss lies within that many standard errors of tt.

Proof idea (Section 2.4). Take tt equal to the selected risk. The true risks lie in a joint confidence set unless some error exceeds both the radius and its discount, and that is the tail the bootstrap estimates. If the true winner has a nonconstant column and is not plausible, its error already exceeds the band radius, an event the tail counts. If it is plausible, step 4 includes its radius. If its validation losses are all equal, the selected checkpoint's radius is at least as large. So screening needs no separate error level.

Theorem 1 in plain words (Section 3). For a training path fixed before the labels are drawn, if every checkpoint's loss standard deviation is at least a fixed fraction of the loss bound, the chance that B-Zoom returns an interval missing the selected risk exceeds α\alpha by at most order m−1/4m^{-1/4}, up to factors of log⁡(K∨m)\log(K \vee m), for up to exp⁡(o(m1/5))\exp(o(m^{1/5})) checkpoints.

Worked example (Figure 1, Appendix C). On three checkpoints with m=100m = 100, the threshold stays between 1.95 and 1.97 while only checkpoint 1 is plausible. Above t=0.409t = 0.409 the noisy checkpoint 2 becomes plausible, and from about t=0.42t = 0.42 the threshold equals the band's 2.10. The interval [0.359,0.434][0.359, 0.434] has width 0.0756, against 0.0787 for the band and 0.0796 for Bonferroni.

W4. Experiment design and scope

Section 4 defines the settings and comparators, and Table 2 gives the protocol: 100 training seeds per setting, each storing all 3000 SGD iterates; a 25,000-example held-out pool, which makes the target exact; K∈30,150,600,3000K \in {30, 150, 600, 3000} and m∈100,1000,5000m \in {100, 1000, 5000}; 84,000 replications in all. Table 3 gives the results by setting. B-Zoom covers at least 0.899 in all 84 configurations, while the uncorrected interval falls to 0.529 (Section 4.1).

All real-data experiments use CIFAR image classifiers. Within that, the seven settings vary what governs coverage: the loss (cross-entropy and 0–1 error), label noise, an early risk minimum, fine-tuning, KK and mm. The near-tie design has independent loss columns, where correlation offers little saving and several risks nearly tie. There B-Zoom covers at least 0.916 at KK up to 10,000, and its mean width is 0.80 to 0.93 times Bonferroni's in every configuration (Appendix E.3). The rare-loss and heavy-tailed designs show where the Gaussian approximation holds, and Table 29 locates the seven real-data settings relative to them. (Section 4.5).

What is new. Table 1 pairs each building block with what B-Zoom adds, and Table 4 compares B-Zoom with related approaches. The zoom correction of Zrnic & Fithian (2025) assumes a known bound on the joint tail of the errors. Theorem 1 instead covers the case where the sample that selects the checkpoint also estimates that tail, the scales and the gap discounts, which to our knowledge has not been analyzed before. The labels pick the checkpoint and give an asymptotically valid interval for its risk. A zoom baseline that factorizes the covariance abstains in 1820 CIFAR-100 error replications in which B-Zoom returns an interval (Section 4.4). The maximum over plausible winners that the proof needs adds 0.50% of mean width over the single-reference zoom correction on the same draws. A two-step version, which first screens winners at a separate level β=α/10\beta = \alpha/10 and then tests at α−β\alpha - \beta, is wider by 1.60% pooled, about three times as much, with coverage within 0.003 (Section 4.4, Appendix F.3).

All four points are now addressed in the revised paper (Sections 1–4 and Appendices C, E and H). We hope this resolves your concerns and would be grateful if you considered the revision in your final assessment and revised your score.

AuthorsOpenReview ↗

Follow-up for Reviewer C5PM

Dear Reviewer C5PM, Thank you again for your time and thoughtful review. We wanted to politely follow up on our rebuttal above. We would greatly appreciate it if you could let us know whether our responses and revisions resolve your concerns, or if there are any remaining questions we can clarify before the deadline. Best regards, The Authors

Thank you for the further clarification. While I can see some promise in the topic of study, I am not yet persuaded that this is a large enough contribution. Other reviews also mention the quality of the writing, and the required changes are large enough that I'm not comfortable taking it for granted that the clarity will improve enough between now and an camera-ready version.

Reviewer_ZUe3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

The paper studies confidence intervals for the population risk of the checkpoint minimising validation loss when the same validation observations are used for selection and inference. It extends the zoom correction by estimating the dependence structure and heteroscedastic scales through a Gaussian multiplier bootstrap on the validation loss matrix. The main result proves an asymptotic abstain-or-cover guarantee with growing checkpoint count, same-sample estimated scales and discounts, finite bootstrap draws, and grid discretisation. Experiments on seven CIFAR-based arms report narrower intervals than Bonferroni, distinguish dependence savings from localisation savings, and demonstrate failure of Gaussian calibration under rare losses.

Critical strengths
  • The inference target is practically important and stated precisely. The paper distinguishes risk of the realised selected checkpoint from risk averaged over a model-selection procedure, and explicitly excludes validation-adaptive training paths.
  • The principal proof is careful and technically meaningful. A fixed-rectangle Gaussian approximation cannot simply be applied at a region chosen using the same errors. The training-measurable comparison rectangle, scale/covariance concentration, shell bound, and grid containment correctly address this dependence.
Critical weaknesses
  • The main theorem requires a refining grid with a deterministic spacing bound, while all main experiments use 400 grid points which, as the manuscript acknowledges, does not satisfy the asymptotic condition.
  • The paper clearly motivates its extension of original Zoom, but should distinguish its coverage theorem more explicitly from existing winner-inference and bootstrap-screening results that also use estimated gaps and dependence from the same sample (https://arxiv.org/html/2410.19212v3, https://arxiv.org/html/1312.7614v5)
Questions for authors
  1. Conditional on the loss matrix, Gaussian multiplier draws have exactly the Gaussian distribution specified by the corresponding empirical covariance, with matching normalisation and standardisation. Consequently, a plug-in Zoom comparator can use the same centred-loss matrix sampler without forming or factorising a K×K covariance matrix. The reported comparator instead factorises this matrix and is evaluated only up to K=600. Could you compare the methods using the same loss-matrix sampler, shared draws, grid, and matched constant-column handling at K=3000 and larger? What differences in coverage, interval width, abstention, runtime, and memory remain?

  2. Can you provide a practical grid rule satisfying the theorem and report its coverage, width, and runtime?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZUe3 (Part 1/2)

Thank you for the precise questions on the comparator and the grid rule.

Below, KK is the number of checkpoints, mm the number of validation examples, and coverage is at nominal 0.90. A configuration is one (K,m)(K, m) pair of a setting, and a replication is one validation sample. Your two questions are about the design of the new comparisons, which we ran in Section 4.4 and Appendices F and G.

Q1. Same loss-matrix sampler, shared draws, K=3000K = 3000 and larger

You are right that given the loss matrix, the multiplier draws are exactly Gaussian with the estimated covariance, so a plug-in zoom correction needs no K×KK \times K factorization. The paper now states this (Section 2.3, Appendix F.4), and swapping the two samplers changes pooled mean width by 0.001%. The comparator is the plug-in single-reference zoom correction (reference k^\hat k) on B-Zoom's own B=2000B = 2000 draws, scales with divisor mm, rank kα(B)=1800k_\alpha(B) = 1800, 400-point grid, and constant-column, abstention and tie rules (Appendix F.1).

The one remaining difference is the maximum over W(t)W(t). It costs 0.50% of mean width, raises the median inversion time from 0.58 s to 1.51 s (Appendix F.2), and moves coverage by at most 0.001 in any configuration of Table 15 and 0.014 on near ties (Appendix F.5).

K=3000K = 3000, 9000 replications; m∈{100,1000,5000}m \in \{100, 1000, 5000\}; Tables 15 and 16):

B-ZoomSingle reference
Coverage0.987 [0.985, 0.990]0.987
Mean width0.27420.2728
Abstentions, grid failures0, 00, 0
Median inversion time1.51 s0.58 s
Inversion memory, main setting (Table 20)69–188 MB62–75 MB

Sampling is shared (median 0.44 s; peak memory at most 1,009 MB). B-Zoom's interval contains the single-reference one. It covers in 4 replications where that interval misses, and the reverse never happens. Coverage is well above 0.90 here because B-Zoom covers the selected risk through a confidence set for all the risks (Section 4.1, Appendix E.2). We also empirically find the single-reference ablation's similar coverage.

Near ties, mm from 25 to 500 (Tables 19 and 20). Pooled coverage is 0.974 against 0.973 at K=10,000K = 10{,}000 and 0.961 against 0.957 at K=30,000K = 30{,}000, with width ratio 1.0020 to 1.0071 in every configuration and median inversion 3.13 s against 1.56 s and 10.22 s against 5.15 s. Both fall below nominal only at K=30,000K = 30{,}000, m=25m = 25: B-Zoom 0.876 [0.844, 0.904], single reference 0.862 [0.829, 0.891]. There the inversion adds 1,343 MB of memory against 595 MB, while factorizing the covariance at K=10,000K = 10{,}000 alone takes 260 s and 4,583 MB.

What the maximum over W(t)W(t) is needed. The maximum over W(t)W(t) is what containment (Theorem 11) needs. At t=μk^t = \mu_{\hat k}, a population winner with a nonconstant column outside W(t)W(t) already places the risk vector outside the joint confidence set, and one inside raises the threshold to its own radius. A winner with a constant column has the smallest scale, therefore k^\hat k gives a radius at least as large. With k^\hat k alone containment can fail when the winner is noisier than k^\hat k. In Appendix F.2, with a known Gaussian tail, the single-reference radius 1.949 rejects a value the joint confidence set allows, and the second reference raises it to 2.114, which accepts. The guarantee thus costs 0.50% of width, against 1.60% for a separate screening level (Section 4.4).

W1 and Q2. A practical grid rule that satisfies the theorem

Proposition 2 (Section 3, proof in Appendix D.10) gives the rule. With w=2c ψB(0) sk^/mw = 2c\,\psi_B(0)\,s_{\hat k}/\sqrt m, it places n=max⁡{n0, 1+⌈w/hauto(m,K)⌉}n = \max\{n_0,\ 1 + \lceil w/h_{\rm auto}(m,K)\rceil\} equally spaced points on [μ^k^±w/2][\hat\mu_{\hat k} \pm w/2], where hauto(m,k)=MΛk3/4/(Ckm3/4)h_{\rm auto}(m,k) = M\Lambda_k^{3/4}/(C_k m^{3/4}) and Λk=log⁡(k∨m)\Lambda_k = \log(k\vee m). The spacing is at most hautoh_{\rm auto} on every realization, and the grid term CKm hauto/aC_K\sqrt m\,h_{\rm auto}/a is at most (2/ξ)Λmax⁡3/4m−1/4(2/\xi)\Lambda_{\max}^{3/4}m^{-1/4}, which is o((Λmax⁡5/m)1/4)o\big((\Lambda_{\max}^5/m)^{1/4}\big), so condition (5) holds. The rule uses the bound MM but not the scale floor ξ\xi. With c=1.02c = 1.02 and n0=400n_0 = 400 it returns the 400-point grid in every real-data replication, so the reported intervals are those of this rule. Without n0n_0 it would ask for at most 42 points there (Appendix G.1).

On the same replications and draws, 400, 600 and 4000 points agree in coverage to 0.002 in every configuration, and pooled mean widths lie within 0.26% of the 4000-point width (Table 22). At K=3000K = 3000 it takes 1.2 to 4.0 s at 600 points, against 84 to 190 s for a 4000-point full sweep, and returns the full-sweep endpoints in all 5014 + 1240 replications checked (Appendix G.2, Table 23).

AuthorsOpenReview ↗

Response to Reviewer ZUe3 (Part 2/2)

Here we continue our response to some of the other points raised.

W2. Distinguishing the theorem from winner inference and bootstrap screening

Section 5 and Table 4 place B-Zoom in this literature. In the language of moment inequalities, B-Zoom's gap discounts recenter each moment from the same sample, and W(t)W(t) is a same-sample screen for a nuisance parameter, the identity of the population winner.

  • Two-step tests of Romano, Shaikh and Wolf (2014; RSW), Chernozhukov, Chetverikov and Kato (2019; CCK) and Bai, Santos and Shaikh (2022; BSS) studentize by same-sample scales and screen slack inequalities from the same sample. RSW and BSS recenter each moment by its own same-sample bound inside a bootstrap maximum, much as B-Zoom discounts each checkpoint. All three spend a separate level on the screen and test a fixed hypothesis, and CCK and BSS allow growing dimension. In CCK's one-step and two-step methods (Sections 4.2.1 and 4.2.2) the moments kept share one threshold, while each side of B-Zoom's rectangle has its own scale, discount and masked radius (Section 3).
  • Inference after ranking and on winners. Petrou-Zeniou and Shaikh (2026; arXiv 2410.19212) target the same unconditional coverage of a winner's mean, with gaps and covariance from the same sample and uniform validity. They bound the gaps at a separate level, for a fixed number of candidates. The hybrid intervals of Andrews, Kitagawa and McCloskey (2024; AKM) also spend a separate level, and their conditional intervals give coverage given k^\hat k for a fixed, finite set.
  • Moment selection (Andrews and Soares, 2010; Hansen's SPA test, 2005) avoids a separate level by screening at a threshold that grows with the sample size, with a fixed number of moments.
  • Inference for ranks (Mogstad, Romano, Shaikh and Wilhelm, 2024) gives confidence sets for which candidates could be best. B-Zoom bounds the risk of the candidate that won.

B-Zoom's discounts come instead from one joint confidence set at level α\alpha, and W(t)W(t) uses the band radius at that same α\alpha. Its screen drops a population winner with a nonconstant column only when the risk vector already lies outside this set, and a constant-column winner is covered through k^\hat k, so screening adds no failure event of its own, and there is no second level β\beta and no tuning sequence to choose (Section 2.4).

The measured value of spending no separate level (Section 4.4, Appendix F.3, Table 17). B-Zoom's two-step version follows RSW. It screens references with the band at level β=α/10\beta = \alpha/10, takes the worst case over them and reads the threshold at rank kα−β(B)k_{\alpha-\beta}(B), on the same 9000 replications and draws. In all 9000 its endpoints lie strictly outside B-Zoom's, so it contains B-Zoom's interval. The price is 1.60% of mean width pooled and 1.16 to 2.49% per configuration, about three times the cost of the maximum over W(t)W(t), with coverage within 0.003 in every configuration.

The AKM hybrid on real data (Section 4.4, Appendix E.4, Table 14). On 63,000 of B-Zoom's replications at K≤600K \le 600, our plug-in implementation of the hybrid, with a Ledoit–Wolf covariance estimate, conditions on k^\hat k inside a band at level β=α/10\beta = \alpha/10. With cross-entropy loss and m≥1000m \ge 1000 it is 11 to 21% narrower than B-Zoom and covers 0.902 to 0.914, against 0.951 to 0.989. At m=100m = 100 it covers 0.856 against 0.960. With 0–1 loss the winner's validation error ties another's in 4524 of 18,000 replications, where conditioning on k^\hat k leaves no usable interval, so it covers 0.694 against 0.982.

The guarantee. Theorem 1 bounds the probability that B-Zoom abstains or covers below by 1−α−Oξ((Λmax⁡5/m)1/4)1-\alpha-O_\xi\big((\Lambda_{\max}^5/m)^{1/4}\big), Λmax⁡=log⁡(Kmax⁡∨m)\Lambda_{\max} = \log(K_{\max}\vee m), uniformly over all conditional laws of the rows on [0,M]K[0,M]^K with min⁡kσk≥ξM\min_k\sigma_k \ge \xi M, exact and near ties included, and over training paths, whenever log⁡Kmax⁡=o(m1/5)\log K_{\max} = o(m^{1/5}). This is the rate of Chernozhukov, Chetverikov, Kato and Koike (2022, Theorems 2.1 and 2.2) over fixed rectangles with unrestricted correlations. Selecting k^\hat k and estimating the tail, scales and discounts from the same sample add only terms of smaller order (Table 6). The bound covers any finite B≥⌈1/α⌉−1B \ge \lceil 1/\alpha\rceil - 1 without a Monte Carlo term, and it accounts for abstention and the automatic grid. To our knowledge, Theorem 1 is the first coverage guarantee for the zoom correction when the sample that selects the checkpoint also estimates the tail, the scales and the gap discounts.

The revision answers both questions with the comparisons you specified, and we would be grateful if you took it into account in your final assessment and revise your score.

AuthorsOpenReview ↗

Follow-up for Reviewer ZUe3

Dear Reviewer ZUe3, Thank you again for your time and thoughtful review. We wanted to politely follow up on our rebuttal above. We would greatly appreciate it if you could let us know whether our responses and revisions resolve your concerns, or if there are any remaining questions we can clarify before the deadline. Best regards, The Authors

Dear authors,

Many thanks for revising the manuscript and conducting the additional evaluations and checks! Your response has addressed the weaknesses outlined in the initial review and answered my questions.

My remaining reservations pertain to the readability of the paper for humans. I think the presentation and accessibility of the paper can be greatly improved and would need to be improved to meet the "clear acceptance" threshold. I tried to put together a few non-exhaustive examples below to illustrate my point and be more concrete.

  1. Abstract

    • It opens with a general setting where the problem lives but then immediately jumps into the CIFAR-10 example as if the presented work is somehow specific to that dataset. In my opinion, the abstract would be easier to understand if, after the setting, it stated the problem being solved, e.g., "... ignores the selection. This leads to..."
    • It would benefit from highlighting the shortcomings of the existing methods and how the proposed method is superior before going into the details of its construction; these details and the advantages of the method are currently presented in a convoluted manner. Consider, for example, the sentence "The bootstraps perserves the correlation between ... which a union bound discards" - preserving the correlation is not a goal per se, the abstract should focus on how the resulting interval is better than corrections using the union bound, not why this is the case (or at least, state the former first).
    • the mention of interval abstantion is unclear - what should a practioner do in that case?
    • "with finitely many draws and finite grid" - at this point it's not clear which grid is meant here?
    • the mention of the experimental results should lead with a summary of observations while specific numbers - cited only as examples. The current version of the abstract makes some comparison with Bonferroni and a test-set interval but does not mention how the results fair against more specialised methods benchmarks.
  2. Contributions

    • the Method part:
      • It needs to state explicitly what the paper develops on top of existing literature and what is the merit of these new developments. The current version reads "We estimate the zoom correction's joint tail..." To me this describes a technique that the authors have used but not the end result, which is the method itself. The contribution should rather start with "We develop a novel method for ..."
      • The text of the contribution talks about the complexity of the dense implementation of the hybrid interval and B-Zoom but does not indicate what follows from those complexity bounds (i.e., why should anyone care about them). If a key advantage of the new method is reduced computational complexity in one way or another this needs to be stated explicitly. Good presentation leads with the conclusion rather than with facts that the reader then has to process on their own.
      • Saying that something was run at a specific parameter setting in the contribution outline is, in my opinion, out of place.
    • the Theory part:
      • needs more clarity (e.g., what does for up to exp(o(m^{1/5}))) checkpoints means? What is coverage at bootstrap rate means? The difference between K and K_{max} is completely unclear at this point.)
      • would benefit from being more focused, i.e., since the main problem is the coverage of the risk of the selected checkpoint, it's better to talk about how the method solves it directly rather than uniform coverage (otherwise the reader needs to connect the two)
    • the Experiments part:
      • needs to summarise the main observations from the experiments (e.g., "We conduct numerical experiments in which the method demonstrates ..."); it should not go into too many details as those, in my opinion, belong to dedicated sections
  3. Remaining text

    • Since the method builds on the zoom correction it would be better to introduce it fully first and then show where the difference between zoom and B-zoom is.
    • Expressions "B-Zoom inverts a five-step test" are quite technical, in my opinion, i.e., it would be better to explain what the inversion consists of. Somewhat relatedly, it would be good to define what grid is meant in assumption (iii) of the theorem and elsewhere. etc.

I understand that readability and presentation are subjective matters but I wouldn't be comfortable increasing the rating from 3 to 4 unless the paper, in my view, becomes more accessible for a human reader that does not have in-depth knowledge of the field.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the reviewers for engaging with our work. All three reviewers asked for a paper that is easier to follow, hence the revised PDF is rewritten with more background context. The main Theorem 1 is mathematically unchanged. Below, KK is the number of checkpoints, mm the number of validation labels, and coverage is at nominal 0.90. A configuration is a (K,m)(K, m) pair of a setting, and each has 1000 replications (validation samples).

  • Rewritten for a first reading. The introduction opens on the use case: 3000 checkpoints and 100 labels, where the usual 90% interval covers the selected checkpoint's risk 64% of the time and B-Zoom 96% (Section 1). Section 2.2 builds from selection bias through Bonferroni and the simultaneous band to the zoom correction and its gap discounts. Section 2.4 states the test in five plain steps with a short proof idea, and Figure 1 and Appendix C work it through on three checkpoints. Table 2 gives the protocol and Table 3 the results by setting. The proofs open with an overview (Appendix D.1), a table of error terms and a dependency diagram.
  • Positioned against the econometrics literature. Table 1 compares each building block with what B-Zoom adds. Section 5 and Table 4 compare it against two-step moment-inequality tests, inference on winners and after ranking, generalized moment selection, the SPA test and inference for ranks. B-Zoom screens plausible winners from the same joint confidence set at level α\alpha, with no separate screening level and no tuning sequence. Theorem 1's rate (Λmax⁡5/m)1/4(\Lambda_{\max}^5/m)^{1/4}, Λmax⁡=log⁡(Kmax⁡∨m)\Lambda_{\max} = \log(K_{\max}\vee m), matches that of the multiplier bootstrap over rectangles fixed in advance with unrestricted correlations (Chernozhukov et al., 2022), uniformly over loss laws with variances bounded away from zero.
  • Cost of avoiding a separate screening level. On 9000 real-data replications at K=3000K = 3000 with shared draws, B-Zoom and the single-reference zoom correction both cover 0.987. The maximum over plausible winners that the proof needs adds 0.50% of mean width. A two-step version that spends β=α/10\beta = \alpha/10 on the screen widens every one of these intervals, by 1.60% pooled, about three times as much, while coverage changes by at most 0.003 (Section 4.4, Appendix F.3).
  • An automatic grid rule satisfying Theorem 1. Proposition 2 gives an automatic rule that needs only the loss bound MM (proof in Appendix D.10). With its floor of 400 points it returns the grid used in every real-data replication. Grids of 600 or 4000 points change coverage by at most 0.002 in every tested configuration (Appendix G).
  • Using all labels beats a held-out split. At matched mean width, a held-out split has 1.33 to 3.51 times B-Zoom's selection regret, the excess population risk of the selected checkpoint over the best of the KK. At each of five allocations it selects worse on average in all 21 combinations of setting and mm (Section 4.3, Appendix H).
  • Coverage and width. B-Zoom covers at least 0.899 in all 84 real-data configurations and is 14 to 29% narrower than Bonferroni on each setting (Table 3). On the main setting its width falls from 1.77 to 1.32 times the uncorrected width as mm grows from 100 to 5000, while the band stays at 1.83 to 1.84 times it (Section 4.2).
  • When the approximation holds at finite mm. Under rare losses, an approximate calculation over the computed range puts coverage at 0.88 or more once the expected rare count mp≥22mp \ge 22 and 0.89 or more once mp≥41mp \ge 41 (Table 27). On two-column designs, every configuration with population Berry–Esseen ratio at most 0.15 covers at least 0.89. Hoeffding and empirical-Bernstein intervals chosen in advance cover at least 0.999 in every configuration (Section 4.5, Appendix I).
  • Three new real-data analyses. B-Zoom is 1.12 to 2.20 times as wide as the uncorrected interval, most where that interval undercovers (Section 4.1, Table 12). The 64 real-data configurations with Berry–Esseen ratio at most 0.15 cover at least 0.934 (Section 4.5). Our plug-in implementation of the hybrid interval of Andrews et al. (2024), run on 63,000 of the same replications, is narrower with cross-entropy loss at m≥1000m \ge 1000 but covers 0.856 with that loss at m=100m = 100 and 0.694 with 0–1 loss, against 0.960 and 0.982 for B-Zoom (Section 4.4, Appendix E.4).

Our answers to each reviewer's points are in that reviewer's thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.