- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Bootstrapped Information Bottleneck (BIB), an uncertainty-aware framework for selecting the IB trade-off parameter β using bootstrap confidence ellipses rather than point estimates of and . BIB selects the most compressed model whose lower confidence bound on sufficiency satisfies a target threshold. Theoretically, the paper establishes asymptotic bootstrap validity for discrete plug-in MI estimators and several variational neural IB estimators (VIB, NIB, and CEB). Empirically, it demonstrates near-nominal 95% coverage on a high-dimensional synthetic GMM and shows on MNIST and ImageNet that BIB more reliably satisfies sufficiency constraints on held-out data and improves test accuracy compared with point-estimate β selection, with relatively small computational overhead.
Critical strengths
-
Point-estimate-based selection of β can lead to representations that appear to satisfy the sufficiency constraint but fail to do so under sampling variability. Framing β-selection as an uncertainty-aware statistical decision problem is conceptually clean and gives the work clear practical relevance.
-
The bootstrap-based uncertainty estimates are theoretically justified for the considered MI estimators and empirically show near-nominal coverage, while experiments on MNIST and ImageNet demonstrate improved constraint satisfaction and competitive downstream performance.
Critical weaknesses
-
The core BIB procedure largely combines established bootstrap-based uncertainty estimation, confidence regions, and existing MI/IB estimators to make a more conservative β selection. While the problem formulation is meaningful, the paper does not introduce a substantially new uncertainty-quantification or optimization technique, making the overall methodological contribution relatively incremental.
-
The theoretical coverage results appear to justify uncertainty regions for a fixed candidate β, whereas BIB selects β from multiple candidates using those same estimated regions. Marginal 95% coverage for each candidate does not automatically imply 95% coverage for the finally selected representation. Since reliable constraint satisfaction is the central motivation of the paper, this gap is consequential and weakens the strongest theoretical claim.
Questions for authors
- Clarification on the 95% guarantee: The theoretical results establish bootstrap coverage for a fixed candidate β, but BIB ultimately selects β from a grid based on the same confidence regions. Does the claimed 95% guarantee apply to the finally selected representation as well? If so, how is the data-dependent selection over multiple β values accounted for?
- Clarification on the probability being covered: In Remark 3.6, BIB treats the trained encoder/decoder as fixed and only bootstraps the validation set. Therefore, is the theoretical 95% guarantee conditional on the particular training set and trained model? If so, how should the statement that BIB provides guarantees for the learned representation on unseen data be interpreted when training-set randomness is also present?
- Clarification on the construction of the confidence ellipse: The theory establishes bootstrap consistency for the MI estimators, while the algorithm constructs the confidence region using the empirical covariance and the 95th percentile of Mahalanobis distances. What precise theoretical result connects the bootstrap consistency in Theorem 3.4 to the validity of this particular elliptical region, rather than a generic bootstrap confidence region?
- Clarification on the empirical robustness evaluation: The main text states that each of the 20 held-out test sets contains 400 samples, whereas Appendix I.2 states that the 20 subsets contain 100 samples each. Could you clarify which setting was actually used, and how the reported 95–100% robustness over only 20 test sets should be interpreted in relation to the paper's claimed 95% statistical guarantee?
2 comments
Response to Q4
We thank the reviewer for catching this inconsistency. We clarify that 400 samples per test set was the setting actually used. The mention of 100 samples in Appendix I.2 was a typo, and we will update the appendix in the revised version of the manuscript.
The 95% statistical guarantee states that the true sufficiency is above the desired threshold with 95% confidence. If the sufficiency constraint is truly satisfied, then when we draw an unseen dataset of size , we should expect the empirical sufficiency computed on that new sample to satisfy the constraint in at least 95% of random realizations. We evaluate this empirically across 20 independent, held-out test splits of size , where the sufficiency constraint is met on at least 19 of the test sets. It is evidence that supports that our theoretical claims hold in practice. We acknowledge that increasing the number of test sets would provide stronger evidence; however, MNIST only provides 10 000 images in the test set, limiting this increase.
References
[1] Davison and Hinkley, Bootstrap Methods and Their Application
We hope these results settle your concerns; we would be grateful if you would raise your rating.
All changes are implemented in the updated PDF. Changes are highlighted in blue. Comments and questions will be addressed in order.
Methodological contribution
We thank the reviewer for their perspective. While the non-parametric bootstrap is indeed a classical tool, our contribution lies in its novel application to solve a flaw in modern IB optimization:
β section and 95% coverage
The reviewer is entirely correct that the marginal 95% guarantees for each β do not imply a 95% guarantee for the selected optimal β. While marginal guarantees remain relevant and provide tight bounds when evaluating each model marginally, we agree that a joint guarantee is necessary to achieve guarantees when we ultimately select a model among multiple candidates using a single validation set.
To account for this, we have expanded our methodology to include joint coverage. BIB now provides confidence ellipses in two distinct cases: (I) marginal coverage for evaluating individual models; (II) joint coverage to explicitly handle the selection process over the grid. The BIB procedure remains the same; the only difference is how the width of the ellipses is designed and estimated. (I), constructed as previously, and provides tight marginal coverage. (II) computes the empirical quantile of the maximum Mahalanobis distance across all β values for each bootstrap sample. This provides strict joint coverage guarantees, ensuring that the final selected model retains a 95% confidence guarantee. (I) and (II) are described in Algorithms 1 and 2 in the revised draft. This approach is grounded in the theory of simultaneous bootstrap confidence bands ([1] Section 4.3.7). We added Corollary 3.6 to show that joint coverage across β values provides a guarantee for the selected model. We have re-run our selection experiments (Figure 1 and Table 2) using this joint coverage criterion.
The redone Table 2:
Response to Q1
With the new proposed algorithm described above and in Algorithm 2 in the revised PDF, BIB achieves 95% coverage guarantees for the selected model.
Response to Q2
The reviewer correctly identifies that the theoretical guarantee is indeed conditional on the particular trained model. Our claim regarding "unseen data" should be interpreted as follows: for this specific, fixed model, we are 95% confident that its true mutual information on the underlying population (i.e., unseen data) satisfies the sufficiency constraint. Furthermore, estimating training variance via the bootstrap would require re-training the architecture B times, significantly increasing the computational overhead. We will update Section 1 to make it explicit that our goal is the study of estimating uncertainty given a trained model.
Response to Q3
We thank the reviewer for the insightful theoretical question regarding the shape of the confidence regions. By Theorem 3.4 and the Continuous Mapping Theorem, any region that covers 95% of the empirical bootstrap distribution will asymptotically achieve valid 95% coverage for the true information bound E(I^).
Following the multivariate studentized bootstrap methodology outlined in [1], Section 5.8, the ellipses account for the fact that I^(X;T) and I^(Y;T) are correlated and approximately normal, yielding tighter confidence regions than e.g., rectangles with a Bonferroni assumption. In the robustness experiments in our paper, we confirm empirically that the ellipses achieve close to 95% coverage.