- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
Choosing the checkpoint with the lowest validation loss favors checkpoints that benefited from random noise. The Zoom algorithm (2025) accounts for this selection when constructing a confidence interval for the chosen checkpoint’s true loss. It reduces the influence of worse checkpoints by counting their errors only when they exceed certain thresholds, which needs a bound on the probability that any checkpoint’s error exceeds its threshold.
B-Zoom aims to replace this bound with a studentized multiplier-bootstrap estimate. Their algorithm cleverly maintains the empirical structure and dependence of the samples during bootstrap, and comes with asymptotic guarantees. The main contribution is establishing conditions under which this correction is justified despite using the same validation data both to select the checkpoint and to estimate its uncertainty.
Critical strengths
- The theory addresses a gap in applying the zoom correction. A bootstrap approximation for fixed regions does not justify a confidence region chosen from the same sample. The proof handles this dependence by comparing against a region fixed by training information. It explains when estimating the correction from the selection data is justified, rather than assuming that a standard bootstrap result is enough.
Critical weaknesses
- Unclear Writing One of the biggest weaknesses of the paper is the poor writing. The problem is not introduced and motivated sufficiently, and the paper delves into comparisons from the first paragraph. For example, Section 2.3 introduces the gap discounts, plausible-reference set, maximum radius, and final interval very quickly. However, the explanation and interpretation is missing, or left in the appendix (eg. Equation 1).
The paper does not explain the existing zoom method well, and assumes knowledge of it. This makes it hard to follow and assess what novel is being proposed in this paper.
- The asymptotic guarantee provides limited guidance for practical use. The coverage result is useful as an asymptotic validity result. However, it does not establish how large the validation sample must be before the intervals are reliable. The main theorem applies for sufficiently large samples and has an unspecified constant to the rate. The rare-loss example makes this limitation concrete: coverage is approximately 63% at nominal 90%, showing why asymptotic validity alone is not practically useful.
Questions for authors
Can you clearly separate the new algorithmic steps and proof arguments from those inherited from Zoom and existing bootstrap theory? What theoretical and empirical benefit does each new component provide?
Beyond asymptotic validity, what usable guidance does the theory provide for choosing the validation sample size and grid resolution? How should a practitioner assess whether the default procedure is reliable?
4 comments
Response to Reviewer vNUH (Part 2/2)
W2 and Q2. Practical guidance beyond asymptotic validity
Theorem 1 is asymptotic, so we measure (Section 4.5, Appendix I) where its Gaussian approximation holds at finite . They give checks that can be run before seeing the data, and a valid fallback when the checks fail.
Rare losses: count the expected rare events. On designs with two identical loss columns, which test the Gaussian approximation without selection, coverage depends mainly on the expected rare count . For 0–1 loss with rare errors this is the expected number of validation errors. Over the computed range, an approximate binomial calculation that includes grid rounding gives coverage of at least 0.88 once and 0.89 once (Appendix I.2, Table 27).
The 63% example that you raise is at and , so , inside the region this check flags. With probability no rare loss appears, and then the exact-tail interval lies below the risk, so its coverage is capped at . B-Zoom covers 0.626 (Table 28).
Other bounded losses: an empirical diagnostic. On the two-column designs, of the 70 configurations, all 15 with population Berry–Esseen ratio at most 0.15 cover at least 0.89 (Appendix I.4, Figure 11). The ratio is at most , so a lower bound on fixed before the labels are drawn suffices. For 0–1 loss with error rate the check reads , and since increases for , an underestimate of , as a training error rate usually is, errs toward the fallback. The check flags heavy tails: the capped Pareto law with has ratio 0.68 at and covers 0.798 there (Table 30).
Where the real-data settings fall (Section 4.5, Appendix I.4, Table 29, Figure 11). With the held-out pool as the population, the 64 real-data configurations with or 0–1 loss have ratio at most 0.141 and cover 0.934 to 0.998. The other 20, the five cross-entropy settings at , have median ratios 0.18 to 0.40 and cover 0.899 to 0.990, at least as much as the two-column configurations of similar ratio. On the error settings the rarer outcome's expected count is at least 25.9 in every replication, above the rule.
Under selection, coverage is measured directly. B-Zoom covers at least 0.899 in all 84 real-data configurations (Section 4.1), and at least 0.916 on the near-tie design at up to 10,000 and down to 25 (Appendix E.3).
Choices that do not matter in practice. Proposition 2's grid rule meets Theorem 1's grid condition knowing only the loss bound , and 400, 600 and 4000 points agree in coverage to 0.002 in every tested configuration (Appendix G). Theorem 1 holds for any with no Monte Carlo term, and from 500 to 5000 gives the same coverage (Appendix F.7, Table 21).
When a check fails. Hoeffding and empirical-Bernstein union-bound intervals hold at every for every loss law on , and both cover at least 0.999 in every configuration. At and , the configuration the check flags, they are 5.8 and 5.4 times B-Zoom's width (Table 28). Once the ratio is 2.1 to 6.3 (Section 4.5, Table 31). The choice is made before seeing the data, since a rule that switches after reading them inherits neither guarantee (Appendix I.1).
Every change described above is in the revised PDF. If any point still seems unresolved, we would be glad to discuss it further, and we hope you will take the revision into account in your final assessment and revise your score.
Follow-up for Reviewer vNUH
Dear Reviewer vNUH, Thank you again for your time and thoughtful review. We wanted to politely follow up on our rebuttal above. We would greatly appreciate it if you could let us know whether our responses and revisions resolve your concerns, or if there are any remaining questions we can clarify before the deadline. Best regards, The Authors
Thank you for your response and the experiments. I still believe that the paper is very hard to follow, and the jargon does not help clarify the issues I had raised earlier. I have decided to keep my score.
Response to Reviewer vNUH (Part 1/2)
Thank you for your review. In response to your comments, we restructured the introduction and Section 2, and added finite-sample guidance in Section 4.5 and Appendix I.
Below, K is the number of checkpoints, m the number of validation labels, and coverage is at nominal 0.90. A configuration is a setting with a (K,m) pair, and a replication is one validation sample.
W1. Motivation, Section 2.3 and the zoom method
Q1. Inherited versus new, and the benefit of each new component
The band keeps the savings from correlation (7.77–25.98% of Bonferroni's mean width, Table 8) but has no gap discounts, and the zoom correction with a union-bound tail has the discounts but gives up the correlation. B-Zoom’s estimated tail keeps both: it is 14–29% narrower than Bonferroni on each setting, and the discounts save a further 2.95–12.36% beyond the band, more than dependence at m=5000 on six of seven settings (Section 4.2). In our experiments, B-Zoom's width falls from 1.77 to 1.32 times the uncorrected width on the main setting as m grows from 100 to 5000, while the band stays at 1.83 to 1.84 times it. Table 1 (Section 3), with the measured effect of each addition:
New proof arguments (Section 3 and Appendix D.1). Regions chosen by the data lose coverage even for exact Gaussians: each of two independent standard Gaussians lies in [−z0.95,z0.95] with probability 0.90, but the one larger in absolute value does so with probability only 0.81 (Section 1). The proof therefore compares the errors with a rectangle that training fixes (Lemma 5), extending the studentization step of Chernozhukov et al. (2019) from one common threshold to sides set by each checkpoint's scale and discount. An anti-concentration bound for Gaussian shells (Lemma 7) controls the coverage lost to the data-chosen rectangle. Under the maximum over W(t), the accepted set contains every value the joint confidence set allows (Theorem 11), so screening reserves no error level. Appendix D.1 lays this out before the details, with the coverage chain labeled lemma by lemma, a table of error terms (Table 6) and a dependency diagram (Figure 6).