- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies joint prediction for a batch of test observations when calibration data are available as individual samples. It forms calibration groups by bootstrap sampling and aggregates repeated calibrations through p-value or e-value merging. With a common score function, the merged decision reduces to a scalar threshold. The paper also analyzes bootstrap quantiles and conditional coverage for the single-test case, and evaluates the methods on image-classification tasks with perturbed labels.
Critical strengths
- The mismatch between individual calibration data and batch prediction is a relevant problem. Reducing a merged decision to a common score threshold is useful computationally and permits reuse of the underlying transductive inference procedure.
- The distinction between variability from the original calibration sample and additional Monte Carlo variability is informative. The single-test bootstrap analysis provides a useful starting point for understanding why repeated resampling can stabilize thresholds without supplying new calibration information.
Critical weaknesses
- The validity argument does not cover the central transductive procedure. Section 4 explicitly restricts its analysis to a single test point, and Proposition 4.2 corrects a quantile obtained by resampling existing scalar scores. Algorithms 1–2 instead resample groups of observations from a shared empirical distribution. Their group scores are not generally exchangeable with the score of a fresh test group, so Equation (2) cannot simply be treated as a valid p-value. Valid p-value or e-value merging does not repair invalid inputs. A guarantee addressing this distinction, with explicit dependence on group size, is essential to the paper’s main claim.
- Proposition 4.3 omits a centering error that can survive under its stated joint-growth condition. Remark G.2 acknowledges an error of order one over the number of bootstrap groups when replacing the exact Beta quantile by its Gaussian approximation. This error need not vanish after multiplication by the square root of the calibration size. For example, take both quantile levels to be one half, an even number N of bootstrap groups, calibration size N², and N folds. The relevant Beta(N/2 + 1, N/2) median differs from one half by approximately 1/(2N), leaving a limiting shift of one half. This contradicts the zero-centered limit and the claim that median aggregation removes the bias for every allowed scaling.
Questions for authors
- What finite-sample result justifies the fold-level p-values or e-values for groups of more than one observation? Please state the actual correction used in Algorithms 1–2 and how it depends on the original calibration size and the group size.
- Can Proposition 4.3 be corrected by centering at the exact Beta quantile or imposing an additional growth condition? Please address the median example above and specify which variance conclusions remain valid.
- At a fixed total number of sampled groups, how do multiple merged folds compare with one larger bootstrap calibration? This comparison is needed to distinguish an advantage of merging from the effect of additional Monte Carlo computation.