acceptodds
Under review as a conference paper at ICLR 2027

Bootstrapped Information Bottleneck (BIB)

Abstract

The Information Bottleneck (IB) balances prediction sufficiency and input compression. IB methods rely on data-dependent estimates of these two quantities and typically select a trade-off parameter from single-point estimates of these quantities without taking into account estimation uncertainty. We introduce the Bootstrapped Information Bottleneck (BIB), which provides it a distribution over prediction sufficiency vs. compression summarized with confidence ellipses. We provide the theoretical foundation for BIB by proving it asymptotic consistency of the bootstrap for mutual information estimators. BIB ensures that sufficiency constraints are met with high probability on unseen data and it improves held-out test accuracy by >7% compared to point estimate baselines on ImageNet. Finally, we show experimentally that the BIB data-dependent confidence regions are three orders of magnitude tighter than existing high-probability bounds on mutual information.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Information Bottleneck, Bootstrap, Mutual Information Estimation, Information Theory, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nV5nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Bootstrapped Information Bottleneck (BIB), an uncertainty-aware framework for selecting the IB trade-off parameter β using bootstrap confidence ellipses rather than point estimates of I(X;T)I(X;T) and I(Y;T)I(Y;T). BIB selects the most compressed model whose lower confidence bound on sufficiency satisfies a target threshold. Theoretically, the paper establishes asymptotic bootstrap validity for discrete plug-in MI estimators and several variational neural IB estimators (VIB, NIB, and CEB). Empirically, it demonstrates near-nominal 95% coverage on a high-dimensional synthetic GMM and shows on MNIST and ImageNet that BIB more reliably satisfies sufficiency constraints on held-out data and improves test accuracy compared with point-estimate β selection, with relatively small computational overhead.

Critical strengths
  1. Point-estimate-based selection of β can lead to representations that appear to satisfy the sufficiency constraint but fail to do so under sampling variability. Framing β-selection as an uncertainty-aware statistical decision problem is conceptually clean and gives the work clear practical relevance.

  2. The bootstrap-based uncertainty estimates are theoretically justified for the considered MI estimators and empirically show near-nominal coverage, while experiments on MNIST and ImageNet demonstrate improved constraint satisfaction and competitive downstream performance.

Critical weaknesses
  1. The core BIB procedure largely combines established bootstrap-based uncertainty estimation, confidence regions, and existing MI/IB estimators to make a more conservative β selection. While the problem formulation is meaningful, the paper does not introduce a substantially new uncertainty-quantification or optimization technique, making the overall methodological contribution relatively incremental.

  2. The theoretical coverage results appear to justify uncertainty regions for a fixed candidate β, whereas BIB selects β from multiple candidates using those same estimated regions. Marginal 95% coverage for each candidate does not automatically imply 95% coverage for the finally selected representation. Since reliable constraint satisfaction is the central motivation of the paper, this gap is consequential and weakens the strongest theoretical claim.

Questions for authors
  1. Clarification on the 95% guarantee: The theoretical results establish bootstrap coverage for a fixed candidate β, but BIB ultimately selects β from a grid based on the same confidence regions. Does the claimed 95% guarantee apply to the finally selected representation as well? If so, how is the data-dependent selection over multiple β values accounted for?
  2. Clarification on the probability being covered: In Remark 3.6, BIB treats the trained encoder/decoder as fixed and only bootstraps the validation set. Therefore, is the theoretical 95% guarantee conditional on the particular training set and trained model? If so, how should the statement that BIB provides guarantees for the learned representation on unseen data be interpreted when training-set randomness is also present?
  3. Clarification on the construction of the confidence ellipse: The theory establishes bootstrap consistency for the MI estimators, while the algorithm constructs the confidence region using the empirical covariance and the 95th percentile of Mahalanobis distances. What precise theoretical result connects the bootstrap consistency in Theorem 3.4 to the validity of this particular elliptical region, rather than a generic bootstrap confidence region?
  4. Clarification on the empirical robustness evaluation: The main text states that each of the 20 held-out test sets contains 400 samples, whereas Appendix I.2 states that the 20 subsets contain 100 samples each. Could you clarify which setting was actually used, and how the reported 95–100% robustness over only 20 test sets should be interpreted in relation to the paper's claimed 95% statistical guarantee?
2 comments

All changes are implemented in the updated PDF. Changes are highlighted in blue. Comments and questions will be addressed in order.

Methodological contribution

We thank the reviewer for their perspective. While the non-parametric bootstrap is indeed a classical tool, our contribution lies in its novel application to solve a flaw in modern IB optimization:

  1. Theoretical foundation: To the best of our knowledge, this is the first work that establishes formal theory on the application of the non-parametric bootstrap to MI estimation, both for classical discrete MI estimators and modern neural-net-based MI estimators.
  2. Solving the true IB objective: Most prior deep IB literature treats β\beta merely as a hyperparameter to tune for validation accuracy. BIB provides a statistically guaranteed mechanism to find the maximum possible compression among all candidate models that strictly satisfy the prediction sufficiency constraint (I(Y;T)≥αI(Y;T) \ge \alpha).
  3. We argue that BIB does not merely do a "more conservative β\beta selection". BIB selects a β\beta with guarantees on worst-case sufficiency and compression. Simply selecting a conservative/small β\beta does not guarantee high sufficiency, as illustrated in the new Table 3 in the revised PDF.

β\beta section and 95% coverage

The reviewer is entirely correct that the marginal 95% guarantees for each β\beta do not imply a 95% guarantee for the selected optimal β\beta. While marginal guarantees remain relevant and provide tight bounds when evaluating each model marginally, we agree that a joint guarantee is necessary to achieve guarantees when we ultimately select a model among multiple candidates using a single validation set.

To account for this, we have expanded our methodology to include joint coverage. BIB now provides confidence ellipses in two distinct cases: (I) marginal coverage for evaluating individual models; (II) joint coverage to explicitly handle the selection process over the grid. The BIB procedure remains the same; the only difference is how the width of the ellipses is designed and estimated. (I), constructed as previously, and provides tight marginal coverage. (II) computes the empirical quantile of the maximum Mahalanobis distance across all β\beta values for each bootstrap sample. This provides strict joint coverage guarantees, ensuring that the final selected model retains a 95% confidence guarantee. (I) and (II) are described in Algorithms 1 and 2 in the revised draft. This approach is grounded in the theory of simultaneous bootstrap confidence bands ([1] Section 4.3.7). We added Corollary 3.6 to show that joint coverage across β\beta values provides a guarantee for the selected model. We have re-run our selection experiments (Figure 1 and Table 2) using this joint coverage criterion.

The redone Table 2:

DatasetSelectionβ\betaAcc. (%)Rob. (%)I(X;T)I(X;T)I(Y;T)I(Y;T)
MNISTPoint0.072083.05%75%26.424 ± 2.2141.420 ± 0.120
MNISTBIB0.014889.70%95%29.619 ± 1.9851.603 ± 0.116
ImageNetPoint0.079457.50%70%47.345 ± 3.2103.712 ± 0.310
ImageNetBIB0.020065.20%100%52.610 ± 2.8504.150 ± 0.285

Response to Q1

With the new proposed algorithm described above and in Algorithm 2 in the revised PDF, BIB achieves 95% coverage guarantees for the selected model.

Response to Q2

The reviewer correctly identifies that the theoretical guarantee is indeed conditional on the particular trained model. Our claim regarding "unseen data" should be interpreted as follows: for this specific, fixed model, we are 95% confident that its true mutual information on the underlying population (i.e., unseen data) satisfies the sufficiency constraint. Furthermore, estimating training variance via the bootstrap would require re-training the architecture BB times, significantly increasing the computational overhead. We will update Section 1 to make it explicit that our goal is the study of estimating uncertainty given a trained model.

Response to Q3

We thank the reviewer for the insightful theoretical question regarding the shape of the confidence regions. By Theorem 3.4 and the Continuous Mapping Theorem, any region that covers 95% of the empirical bootstrap distribution will asymptotically achieve valid 95% coverage for the true information bound E(I^)\text{E} (\hat I).

Following the multivariate studentized bootstrap methodology outlined in [1], Section 5.8, the ellipses account for the fact that I^(X;T)\hat I(X;T) and I^(Y;T)\hat I(Y;T) are correlated and approximately normal, yielding tighter confidence regions than e.g., rectangles with a Bonferroni assumption. In the robustness experiments in our paper, we confirm empirically that the ellipses achieve close to 95% coverage.

Response to Q4

We thank the reviewer for catching this inconsistency. We clarify that 400 samples per test set was the setting actually used. The mention of 100 samples in Appendix I.2 was a typo, and we will update the appendix in the revised version of the manuscript.

The 95% statistical guarantee states that the true sufficiency I(Y;T)I(Y;T) is above the desired threshold with 95% confidence. If the sufficiency constraint is truly satisfied, then when we draw an unseen dataset of size N=400N=400, we should expect the empirical sufficiency computed on that new sample to satisfy the constraint in at least 95% of random realizations. We evaluate this empirically across 20 independent, held-out test splits of size N=400N=400, where the sufficiency constraint is met on at least 19 of the test sets. It is evidence that supports that our theoretical claims hold in practice. We acknowledge that increasing the number of test sets would provide stronger evidence; however, MNIST only provides 10 000 images in the test set, limiting this increase.

References

[1] Davison and Hinkley, Bootstrap Methods and Their Application

We hope these results settle your concerns; we would be grateful if you would raise your rating.

Reviewer_ZLdFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BG: Information Bottleneck (IB) trade-off the compression effective and model utility by choosing a compression strength. Existing methods choose the compression strength through point estimating. They train models with different compression strength, estimate the models' utility on validation set, and then select a suitable compression strength.

RP: However, point estimated compression strength may lead to over-compression on the data different with validation set.

Method: The authors propose Bootstrapped Information Bottleneck (BIB), it additional considers the uncertainty of the point estimation. Firstly, BIB repeatedly resamples the validation set to estimate the utility fluctuate, and capture that with a confidence ellipse. Secondly, the compression strength is considered feasible, when the lower boundary of confidence ellipse still satisfies the constraint.

Critical strengths
  1. The problem is practical. It is true that point estimates may be unreliable due to sampling uncertainty.
  2. The proposed method is simple: it only requires resampling the validation set after the models are trained, without changing the training process.
Critical weaknesses
  1. There is a multiple comparisons problem. The method uses bootstrap to build a 95% confidence ellipse for each candidate model, then select the compression strength based on these ellipses. However, the paper only proves that the confidence ellipse is valid for a fixed model, without showing whether the selected model can still satisfy the sufficiency constraint with 95% confidence. Since the selection can also introduce new bias.
  2. The paper attributes the failure of point-estimate selection to estimator variance, where the point estimates frequently overestimate. However, the validation estimate is roughly unbiased for a fixed model. I mean, the systematic overestimation may arise because when winner select the best-performing "winner" based on observed results, this winner's observed performance often systematically overestimates its true level (winner’s-curse effect in statistics). This may suggest the actual problem is selection bias rather than variance itself.
Questions for authors

Could authors compare BIB with “Statistically Valid Information Bottleneck via Multiple Hypothesis Testing” (ISIT 25)? It seems that try to solve same problem with this paper.

2 comments

All changes are implemented in the updated PDF. Changes are highlighted in blue. Comments and questions will be addressed in order.

Multiple comparisons problem

We thank the reviewer for bringing up the issue of selection bias when selecting the beta based on the BIB ellipses. The reviewer is entirely correct that the marginal 95% guarantees for each β\beta do not imply a 95% guarantee for the selected optimal β\beta. While marginal guarantees remain relevant and provide tight bounds when evaluating each model marginally, we agree that a joint guarantee is necessary to achieve guarantees when we ultimately select a model among multiple candidates using a single validation set.

To account for this, we have expanded our methodology to include joint coverage. BIB now provides confidence ellipses in two distinct cases: (I) marginal coverage for evaluating individual models; (II) joint coverage to explicitly handle the selection process over the grid. The BIB procedure remains the same; the only difference is how the width of the ellipses is designed and estimated. (I), constructed as previously, and provides tight marginal coverage. (II) computes the empirical quantile of the maximum Mahalanobis distance across all β\beta values for each bootstrap sample. This provides strict joint coverage guarantees, ensuring that the final selected model retains a 95% confidence guarantee. (I) and (II) are described in Algorithms 1 and 2 in the revised draft. This approach is grounded in the theory of simultaneous bootstrap confidence bands ([1] Section 4.3.7). We added Corollary 3.6 to show that joint coverage across β\beta values provides a guarantee for the selected model. We have re-run our selection experiments (Figure 1 and Table 2) using this joint coverage criterion.

We have redone Table 2:

DatasetSelectionβ\betaAcc. (%)Rob. (%)I(X;T)I(X;T)I(Y;T)I(Y;T)
MNISTPoint0.072083.05%75%26.424 ± 2.2141.420 ± 0.120
MNISTBIB0.014889.70%95%29.619 ± 1.9851.603 ± 0.116
ImageNetPoint0.079457.50%70%47.345 ± 3.2103.712 ± 0.310
ImageNetBIB0.020065.20%100%52.610 ± 2.8504.150 ± 0.285

Selection bias

We agree that whenever the "best" beta is selected, a selection bias may occur. This is addressed in the response above, where we replace the marginal coverage of the BIB with joint coverage, ensuring that the selected β\beta value retains a valid 95% confidence region.

Missing comparison with IB-MHT (ISIT 25)

We thank the reviewer for bringing our attention to this work, and we will cite it in our paper. While both papers share the goal of replacing heuristic hyperparameter tuning with statistical guarantees for the IB, our methodologies differ in two aspects. Firstly, IB-MHT is explicitly formulated for discrete X,Y, relying on plug-in histogram MI estimators, which do not scale well to high-dimensional deep learning settings. In contrast, BIB operates directly on continuous latent spaces in deep networks using variational MI bounds. Secondly, IB-MHT relies on concentration inequalities for its guarantees, while BIB relies on the data-driven non-parametric bootstrap. A comparison between the bootstrap and concentration inequalities (of [1]) is provided in Section 4.1 in our paper.

References

[1] Shamir et al., Learning and generalization with the information bottleneck

We hope these results settle your concerns; we would be grateful if you could raise your rating.

Dear Reviewer ZLdF,

Thanks again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

Reviewer_KmpuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BIB, bootstrapped information bottleneck, which uses bootstrapping to provide a distribution over the two measures of prediction sufficiency I(Y;T) and compression I(X;T). The paper provides theory on bootstrap consistency and run empirical experiments on MNIST and ImageNet features. The test results show better satisfaction of the information constraints and improved accuracy compared to the non-bootstrapped IB model selection.

Critical strengths
  1. The motivation of the paper is convincing, where information estimates are often noisy, resulting in unreliable IB model selection processes. As the paper points out, this can be particularly useful for low to medium datasets.

  2. The proposed approach is concise and compatible with several existing IB model selection method (i.e., vanilla MI estimation, VIB, NIB, CEB), with reasonable computational overhead. The evidence on MNIST and ImageNet-feature experiments also show promising results in terms of not only reducing information constraint violations, but also improving accuracy on the test set.

Critical weaknesses
  1. Table 2 reports only \beta, robustness, and accuracy, and does not report the compression I(X;T) achieved by each selection rule, even though minimizing I(X;T) subject to the constraints is the objective of IB. Since BIB selects a much smaller \beta, this is needed to tell whether BIB gives a better trade-off or simply compresses less.

  2. BIB tends to select smaller values of beta, which may be a confounding reason why there is generally better accuracy. I would like to see experiments where being more conservative in the non-bootstrap model selection resulting in poorer accuracy, this could strengthen the argument for using bootstraps.

  3. Experimental details seem inconsistent between the main text Section 4.2 and Appendix I.2, in terms of training and test samples and beta. Can the paper provide some clarification on this?

  4. Also, the Imagenet experiments scale seems over-claimed. As I understand, the paper doesn’t state the number of classes used and uses features extracted from a pre-trained resnet-50 with a validation set of 400 samples. So how well do the conclusions hold at larger scale?

Questions for authors
  1. How does BIB selection compare against methods that use standard metrics like validation accuracy or cross-entropy for selection?

  2. Given that the validation set is already used in determining the final model among multiple candidate models, does the 95% confidence guarantee hold for the final selected model? If so, how is the selection effect accounted for?

3 comments

All changes are implemented in the updated PDF. Changes are highlighted in blue. Comments and questions will be addressed in order.

Missing reports on compression in Table 2

We thank the reviewer for pointing out that Table 2 does not report I(X;T)I(X;T) for each selected model, and we agree that this should be reported, as compression is a key part of the objective. We have included an expanded version of Table 2, where I(Y;T) and I(X;T) are computed using the test sets (average and standard deviation). The updated Table 2 can be found in the revised PDF.

Although the BIB-selected model achieves a higher sufficiency at the cost of less compression, the IB objective is to select the model with the least compression among all models that satisfy the desired sufficiency constraint. Since the BIB-selected model satisfies the constraint with guarantees, we argue that the BIB does in fact lead to a better trade-off.

Beta selection and accuracy

The reviewer raises a very valid point: because BIB tends to select a smaller β\beta than the point-estimate approach, one might naturally ask if manually picking a more conservative β\beta achieves the same result. However, this is not necessarily the case, as a conservative β\beta value may result in worse accuracy if the model overfits. To demonstrate this, we added a new baseline to the MNIST experiment comparing the conservative β=0.001\beta = 0.001 to the point-selected and BIB-selected betas:

Selectionβ\betaAcc. (%)Rob. (%)I(X;T)I(X;T)I(Y;T)I(Y;T)
Point0.011483.25%90.0%28.708 ± 1.8061.712 ± 0.108
BIB0.002284.25%95.0%41.728 ± 1.6041.788 ± 0.122
Conservative β\beta0.000183.65%60.0%178.49 ± 6.561.502 ± 0.251

To strengthen the theory of the BIB, we added Corollary 3.6 to the revised manuscript which provides a 95% guarantee on the worst case compression and sufficiency for the BIB-selected model among the candidates. We have included this empirical comparison in the revised manuscript in Section 4.2.

Inconsistencies between main text and appendix

We thank the reviewer for their careful reading and for spotting these inconsistencies. To clarify: the experimental details in the main text (Section 4.2) are correct, while the values stated in Appendix I.2 are incorrect. This is fixed in the revised pdf.

Imagenet experiments and scalability

Regarding the ImageNet experiment, we used all the classes (c=1000c=1000). We also used extracted features as a backbone to the encoders in the VIB model. Regarding scale, BIB's utility remains strong as data and model complexity increase from MNIST to ImageNet. While the reviewer is correct that increasing NvalN_{val} reduces estimator variance, thereby narrowing the gap between BIB and point estimates (as addressed in the Limitations section of our paper), Figure 2 demonstrates that substantial estimator variance persist even at Nval=1000N_{val}=1000.

Response to Q1:

While standard metrics like validation accuracy only optimize for the highest possible sufficiency, BIB explicitly picks the largest compression among all models that (with high confidence) satisfy high sufficiency, thereby staying true to the fundamental IB objective. We have updated Section 1 to make this point clearer.

Response to Q2:

We thank the reviewer for bringing up the issue of selection bias when selecting the beta based on the BIB ellipses. The reviewer is entirely correct that the marginal 95% guarantees for each β\beta does not imply a 95% guarantee for the selected optimal β\beta. While marginal guarantees remain relevant and provide tight bounds when evaluating each model marginally, we agree that a joint guarantee is necessary to achieve guarantees when we ultimately select a model among multiple candidates using a single validation set.

To account for this, we have expanded our methodology to include joint coverage. BIB now provides confidences ellipses in two distinct cases: (I) marginal coverage for evaluating individual models, ; (II) joint coverage to explicitly handle the selection process over the grid. The BIB procedure remains the same, the only difference is how the width of the ellipses is designed and estimated. (I), constructed as previously, and provides tight marginal coverage. (II) computes the empirical quantile of the maximum Mahalanobis distance across all β\beta values consideren in the grid for each bootstrap sample. This provides strict joint coverage guarantees, ensuring that the final selected model retains a 95% confidence guarantee. ((I) and (II) are described in Algorithms 1 and 2 in the revised draft). This approach is grounded in the theory of simultaneous bootstrap confidence bands ([1] Section 4.3.7). We added Corollary 3.6 to show that joint coverage across β\beta values in the grid provides guarantee for the selected model. We have re-run our selection experiments (Figure 1 and Table 2) using this joint coverage criterion.

The redone Table 2

DatasetSelectionβ\betaAcc. (%)Rob. (%)I(X;T)I(X;T)I(Y;T)I(Y;T)
MNISTPoint0.072083.05%75%26.424 ± 2.2141.420 ± 0.120
MNISTBIB0.014889.70%95%29.619 ± 1.9851.603 ± 0.116
ImageNetPoint0.079457.50%70%47.345 ± 3.2103.712 ± 0.310
ImageNetBIB0.020065.20%100%52.610 ± 2.8504.150 ± 0.285

References

[1] Hall, The bootstrap and Edgeworth Expansion

We hope these results settle your concerns; we would be grateful if you would raise your rating.

Dear Reviewer Kmpu,

Thanks again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.