acceptodds
Under review as a conference paper at ICLR 2027

Declared Geometry: A Geometric Prior Bottleneck

Abstract

Representations for prediction should discard target-irrelevant input variation while preserving target-relevant information in a predictive geometry. Information-Bottleneck (IB) methods make this compression explicit but do not condition it on the target, a gap the Conditional Entropy Bottleneck (CEB) closes with a label-conditional KL criterion. Yet per-label translations leave the conditional KL unchanged, so collapsed and arbitrarily separated configurations coexist, and under a strict strong data-processing condition strong compression makes an input-independent encoder globally optimal for idealized CEB. We introduce the Geometric Prior Bottleneck (GPB), which declares the geometry in the label-conditional reference and ties the prediction head, making center collapse strictly KL-expensive and, whenever a feasible comparator has lower KL, excluding center-collapsed optima above an explicit compression threshold: OPB classifies from an equal-radius orthogonal frame through anchor energies; EPB regresses through a projection tied to an isometric label axis. We then analyze what the modified objective learns: strong regularization targets the conditional-expectation anchor rather than the input-independent collapse, and under the stated conditions the tied mean readouts remain Bayes-optimal at every compression strength. Across twelve dataset–backbone settings with IB and non-IB baselines, GPB is best or tied-best in eight and never significantly worse than CEB.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Information Bottleneck, Representation Learning, Conditional Entropy Bottleneck, Geometric Priors

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ShksOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a Geometric Prior Bottleneck (GPB), which constrains the means of label-conditional Gaussian priors and uses the same anchors for prediction. Its classification variant, OPB, uses equal-radius orthogonal anchors; its regression variant, EPB, places normalized targets on a label-dependent axis. The theoretical analysis studies geometric indeterminacy in unrestricted conditional references, conditions for information collapse, and the predictors selected by the constrained objective. Experiments cover twelve dataset–backbone settings, supplemented by geometry diagnostics and classification ablations.

Critical strengths

The paper provides a useful analysis of how conditional compression interacts with representation geometry. It distinguishes geometric indeterminacy of the conditional KL from information collapse of the full objective, and derives conditional guarantees for the predictors learned with constrained priors.

Critical weaknesses
  1. The paper does not clearly establish its methodological novelty beyond existing conditional and anchor-based bottlenecks. Conditional-reference-based prediction already appears in CEB’s Consistent Classifier, while Text-Anchored IB(https://arxiv.org/abs/2607.01657) also uses shared anchors for conditional regularization and classification. GPB differs in its Gaussian formulation, prescribed distance constraints, and theoretical analysis, but Table 14 does not adequately explain what these differences add beyond existing designs. The contribution needs to be established through these specific constraints and their analysis, rather than through the general combination of geometry, conditional KL, and a tied readout.

  2. The experiments support the constrained model’s usefulness more clearly than the proposed explanation of its gains. The GPB-G comparison does not clarify whether equal anchor norms and effective readout scale remain controlled when the anchor geometry is released. Without these controls, the energy classifier can also acquire different class-dependent biases, so the improvement does not isolate the effect of prescribed inter-class distances. These mechanism ablations are also limited to one classification setting. For regression, Appendix A.2.2 shows that the EPB mean penalty partly rescales the supervised squared-error loss, but corresponding controls separating mean matching, off-axis penalties, and covariance regularization are absent. The main regression results are mixed: EPB essentially matches CEB on Housing, falls below Base on STS-B, and falls below AdaCap on ZINC. The current evidence does not support a general mechanism claim across classification and regression.

Questions for authors
  1. Given CEB’s Consistent Classifier and Text-Anchored IB, what is the specific methodological contribution of GPB beyond these existing designs? Please explain how the distinction claimed in Table 14 accounts for their use of conditional references and shared prediction anchors. Does the CEB baseline in Table 1 use a free prediction head or the original conditional-reference-based classifier?

  2. Does GPB-G retain fixed, equal anchor norms and the same effective readout scale as the full model? If not, its energy scores also acquire different class-dependent biases. What reported evidence distinguishes the benefit of the prescribed distances from that of restricting anchor norms and classifier scale?

  3. Are there regression counterparts to the full-KL versus mean-matching controls in Appendix A.4.4? Given the EPB decomposition in Appendix A.2.2, what reported evidence establishes an additional benefit from the off-axis and covariance penalties beyond the supervised mean-matching term? If these effects have not been separated, please limit the mechanism conclusions accordingly.

3 comments

Answer for Q1

We agree that conditional references, shared anchors, and conditional-reference-based prediction are not individually new. We therefore removed the previous combination-novelty claim and the associated comparison table. The Introduction and Related Work now define the contribution more narrowly: GPB restricts the conditional reference to a fixed-covariance Gaussian family with prescribed relations among its means, ties prediction to that reference, and analyzes the representation selected by this restricted objective.

The original CEB baseline uses a separately learned prediction head. To isolate the role of conditional-reference-based prediction, we added CEB-C, which uses CEB's original Consistent Classifier, to the main comparison and the compression analyses in Secs. 3.2 and 3.7--3.8. We also added a direct comparison with CEB-C and Text-Anchored IB in the related-work appendix. The revised discussion distinguishes GPB's explicitly prescribed geometry from the geometry inherited from a frozen text encoder and does not suggest that other shared-anchor methods necessarily exhibit the limitations identified for unconstrained references.

Answer for Q2

We agree that the previous GPB-G comparison changed the pairwise geometry, anchor norms, and effective readout scale simultaneously and therefore could not isolate the effect of the prescribed inter-class relations. We replaced GPB-G with GPB-N, which releases the prescribed pairwise relations while retaining the anchor norm, readout scale, stochastic encoder, conditional KL, prior variance, and energy readout. The revised definition and discussion appear in Sec. 3.5 and the ablation appendix. We explicitly limit the resulting mechanism interpretation to the ImageNet-100 MLP setting in which this controlled comparison was performed.

Answer for Q3

We added two regression controls on the AgeDB MLP setting. EPB-noOff retains covariance regularization while removing only the off-axis mean penalty, whereas EPB-M retains full vector mean matching while removing the covariance term. The supervised component, stochastic encoder, tied readout, and model-selection procedure are held fixed so that the two additional components can be examined separately.

We also revised the main-results discussion to state the mixed regression outcomes directly. These controls support only a setting-specific interpretation on AgeDB MLP and are not used to explain every classification or regression result.

Table 1 shows strong classification performance, with GPB achieving the highest mean accuracy in six of the seven classification settings. Appendix A.5 clarifies how OPB's prescribed class geometry differs from the geometry derived from a frozen text encoder in Text-Anchored IB. The new ablations in Table 3 answer my questions about the ablations. With anchor norms and classifier scale controlled, OPB outperforms GPB-N by 0.69 percentage points on ImageNet-100 MLP (95% CI [0.32, 1.04]). On AgeDB MLP, removing the off-axis mean penalty reduces R² by approximately 0.04, while removing the covariance regularizer reduces it by approximately 0.02.

My remaining concern is the limited methodological advance over prior work. CEB's Consistent Classifier already uses the same class-conditional reference distribution for regularization and prediction. OPB builds on this principle by constraining the Gaussian references to have a fixed shared covariance and equal-norm, mutually orthogonal means. I recognize the value of the analysis of how these constraints affect the learned representations. However, while the overall classification results are strong, the controlled experiment directly isolating the contribution of the prescribed class geometry shows a modest gain in only one classification setting. The regression results in Table 1 also do not demonstrate consistent improvements over existing methods. Overall, I still find the methodological advance and the demonstrated benefits of the proposed geometric constraints too limited to warrant acceptance. I therefore maintain Rating 2, Confidence 3.

Answer for Comment

Thank you for clarifying your remaining concern. We agree that reusing a class-conditional reference for regularization and prediction is already established by CEB’s Consistent Classifier, and we do not claim this principle, or orthogonal class frames themselves, as new. Our methodological change is to constrain the conditional-reference family through prescribed geometry. Our theoretical contribution is to characterize how this restriction changes the representations and predictors selected by the resulting objective.

To address the concern that the controlled evidence was limited to one setting, we extended OPB--GPB-N comparison from ImageNet-100 (MLP) to all seven classification settings. The complete results added to the ablation appendix are:

SettingOPB −- GPB-N [95% CI]GPB-N mean ±\pm std
MNIST (MLP)+0.15+0.15 [+0.03+0.03, +0.30+0.30]98.39±0.1098.39\pm0.10
MNIST (CNN)−0.02-0.02 [−0.06-0.06, +0.03+0.03]99.34±0.0499.34\pm0.04
ImageNet-100 (MLP)+0.69+0.69 [+0.32+0.32, +1.04+1.04]83.71±0.4583.71\pm0.45
ImageNet-100 (CNN)+0.94+0.94 [+0.80+0.80, +1.15+1.15]85.14±0.2785.14\pm0.27
Cora (GNN)+1.15+1.15 [+0.44+0.44, +1.70+1.70]86.97±0.7086.97\pm0.70
IMDb (RNN)+1.18+1.18 [+0.52+0.52, +1.85+1.85]85.66±0.4085.66\pm0.40
AG News (RNN)+0.23+0.23 [+0.04+0.04, +0.43+0.43]92.64±0.1992.64\pm0.19

OPB improves mean accuracy in six of seven settings, with the corresponding paired 95% confidence intervals excluding zero. The remaining setting shows no statistically significant difference; no setting significantly favors GPB-N. The gains vary in magnitude, but the additional results show that the benefit of prescribing the pairwise geometry is not confined to ImageNet-100 MLP.

We also agree that the regression results do not establish consistent superiority. We have therefore revised the Introduction, Main Results, Conclusion, and Limitations to state explicitly that classification provides the main empirical evidence, while EPB is a continuous-label extension with an exact population characterization and setting-dependent empirical performance.

We hope these additions clarify that the contribution is not the shared-anchor classification principle itself, but the prescribed conditional-reference geometry, the characterization of the representation and predictor selected by the resulting objective, and the controlled evaluation of the geometric constraints.

Reviewer_CtkkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the Conditional Entropy Bottleneck (CEB) has two limitations. First, its conditional KL certificate is invariant for the translation of each label, thus folded and arbitrarily separated class means geometry share the same KL (Theorem 2.1(a), p. 3). Second, under the strict strong data processing condition η̄<1, once β ≥ η̄/(1−η̄), the idealized nested CEB objective is minimized by an input-independent encoder (Theorem 2.1(b)). As a remedy, it proposes the Geometric Prior Bottleneck (GPB). GPB restricts the label conditional Gaussian reference to declared anchors with a fixed covariance τ²I and associates the prediction head with these anchors: OPB uses an orthogonal polar coordinate system with equal radii and anchor energy fractions, while EPB uses equidistant label axes with associated projections (Equation 4-11, pp. 3-4).

Theoretically, Theorem 2.2 (p.4) lowers the KL bound for any centered folding model by C_col and excludes folding optima above the comparator-related β threshold. Theorem 2.3 (pp. 4-5) shows that strong β drives the posterior mean toward the conditional expectation anchor m̄(x). It also shows that, in the case of isotropic noise, uniform prior, and full expressivity, the bound mean reading preserves the Bayesian ordering (OPB) or equals the Bayesian regressor (EPB) at each β.

Empirically, Table 1 (p.6) covers 12 backbone settings for datasets, containing 10 seeds, 30 configuration budgets per method, and paired guided CIs (Tables 8-10, pp. 23-24). GPB is best or tied for best in the 8/12 settings and is significantly better in 57 out of 84 comparisons. This paper adds two-point control studies (Figure 1, page 7), MNIST PGD robustness (Figure 2), ImageNet-100 ablation (Tables 2 and 12), compressed scans (Figure 4), and information plane diagnostics (Section 3.8, App. A.4.7).

Critical strengths

For a methodological paper of this type, the empirical protocol is careful and transparent. All methods share a split, backbone, bottleneck d=256, ten seeds, and a balanced search of 30 configurations (Table 5, page 21). Results include preset pairwise percentile CIs with Holm-Bonferroni aggregation (pages 22-24). GPB's gains on ImageNet-100 CNN are very significant and quite large (+3.1 over CEB, +5.5 over NCBD) as well as Cora and IMDb. Two significant losses were publicly reported (STS-B vs. Base, ZINC vs. AdaCap). The ablations in Table 2 and Table 12 (pages 7, 26) are well designed. It is worth noting that NCM-S and NCM-M have separate noise injection, mean matching, and variance components of KL compared to GPB-M, which is beyond the scope of a single on/off ablation.

The scope of the theoretical statement is exceptionally clear and the proof is in the App. A.2 (pages 15-20) looks correct. Examples include the comment that restricting the reference family never tightens the certificate (p. 17), showing that counterexamples to H(Y| Corollary A.1 is a clean and useful result: it uses the commutative noise order-preserving argument to show that the OPB average path never requires a Bayesian loss. The control study (App. A.4.2) also matched the closed-form threshold and t∞ to four digits.

Critical weaknesses

The link between theory and key empirical results is weak, and some claims for “repair” are almost tautological. Theorem 2.2 follows almost directly from the prior average of fixed separations: for coincident centers, the KL of non-coincident anchors cannot vanish. Therefore, the OPB KL platform (Figs. 4b, 4d) and the “compression stop” narrative (Section 3.8) reiterate that GPB's KL has a lower bound δm rather than GPB compressing better. Theorem 2.1(b) is essentially the known SDPI/supercontractivity threshold of the trivial IB solution, rewritten by Eq. (19), and the literature is not positioned. The paper admits that its η̄<1 condition does not hold on any benchmark (p. 17). Table 1 does not report which β and a were selected. If the winning configuration is in a weakly compressed state, the anti-collapse theory cannot explain the accuracy gain.

The compression comparison with CEB uses β as the x-axis, although the paper itself states that the same β does not correspond to the same conditional information in the different methods (p.3). When a=12, there is an order of magnitude difference in the KL scale, so "GPB maintains accuracy with decreasing CEB" (Fig. 4a, Fig. 6) is affected by the effective regularization strength. Only Table 13 matches the estimated I(X;Z), and only at the two MNIST points with InfoNCE lower bounds. Several further results are still weak. Ablation uses ImageNet-100 MLP only on frozen features. The robustness study is only for MNIST, with 20 PGD steps, and no convergence check. The CEB Housing binding R²=0.491 of Table 3 is difficult to reconcile with the 0.7894 of Table 1 . Finally, NCM-T (81.71) is lower than Base (83.12), so part of the IB delta may reflect repair fixed scale energy readings.

Questions for authors

For each setting in Table 1, which (β, a/ρ) does GPB choose and which β does CEB choose? Do the gains in GPB persist when restricted to configurations where the realized KL or estimated I(X;Z|Y) is comparable to CEB? This will determine whether the anti-collapse mechanism (rather than binding energy readings or anchoring scale) explains the baseline improvement.

Can you plot the accuracy against estimates of residual information I(X;Z|Y) for CEB and OPB on ImageNet-100 and California housing (instead of against β)? This would indicate whether GPB achieves better compression accuracy bounds, or just a different β parameterization. This Pareto chart can greatly enhance Secs. 3.7-3.8.

How does Theorem 2.1(b) relate to existing results on IB phase transitions and simple solutions via powerful data processing or supercontraction coefficients? Considering you note that a limited dataset with a single label makes it impossible to establish it, is there evidence that any of the evaluated benchmarks satisfy η̄<1?

How to explain the CEB binding indicator R²=0.491 for housing in Table 3 versus 0.7894 in Table 1, and the lower NCM-T results than the benchmark? Would the base model with bound orthogonal energy readings but no KL, at matching scale a, reduce the partial gap attributed to the IB term?

4 comments

Answer for Q1

We added a table in Sec. 3.2 reporting the selected CEB and GPB hyperparameters for every setting, together with the selected and grid-wide realized KL information. We then added performance-versus-realized-KL and performance-versus-estimated-residual-information analyses for representative classification and regression settings: ImageNet-100 in Secs. 3.7--3.8 and California Housing in the corresponding appendix sections. These additions allow the operating points to be compared independently of the numerical value of the regularization multiplier. We accordingly revised the discussion to distinguish weak- and strong-compression regions and do not use these analyses to claim that the anti-collapse theory explains every benchmark improvement.

Answer for Q2

We added the requested performance-versus-estimated-residual-information plots for ImageNet-100 and California Housing, together with the corresponding realized-KL views. The ImageNet-100 analyses are presented in Secs. 3.7--3.8, the California Housing companion analyses are in the appendix. We also redrew the information-plane, certificate-plane, and Pareto curves using a common per-curve binning procedure to improve their readability.

Answer for Q3

We agree that the contraction threshold is an application of existing information-bottleneck, phase-transition, and strong-data-processing results to the idealized CEB parameterization, rather than an independent theoretical contribution. We therefore simplified and repositioned this result, added the relevant literature, and moved the detailed derivation and assumptions to the theory appendix.

We also state explicitly that the revised manuscript does not establish the required strict contraction condition for any evaluated benchmark. The controlled synthetic experiment examines objective selection only in a constructed setting where the relevant channel property is known; it is not evidence that the benchmark distributions satisfy the same condition. The theoretical result is therefore used as an idealized reference and motivation, not as a mechanism-level explanation of the benchmark gains. Whether realistic benchmarks satisfy the condition remains an open empirical question.

Answer for Q4

We clarified in Sec. 3.6 that the Housing result in the main comparison and the value in the posterior-geometry diagnostic come from different configurations and answer different questions: the former is validation-selected, whereas the latter uses a common diagnostic setting for inspecting learned geometry. The two values therefore should not be interpreted as results from the same trained configuration.

NCM-T is a component-isolation control that retains the anchor geometry and tied energy readout while removing the stochastic bottleneck and conditional KL; it is not proposed as an independent method expected to outperform the plain backbone. The relevant contrast therefore estimates the conditional-KL contribution within that specified architecture, rather than the complete GPB improvement over Base. To address the possible scale confound, we added a comparison across a shared anchor-scale grid in the ablation appendix.

Answer for the Additional Concerns in the Critical Weaknesses

We removed the previous collapse--repair--performance narrative and reorganized the theory around collapse-related motivation, prescribed reference geometry, and the conditional-anchor representation selected by the modified objective. The former anti-collapse theorem has been shortened to a consequence paragraph and is no longer used as evidence of better compression. The former CEB-collapse theorem has likewise been repositioned as an application of existing IB and strong-data-processing results, with its assumptions and scope limitations stated explicitly.

We also narrowed the empirical mechanism claims. The classification ablations are identified as evidence from the ImageNet-100 MLP setting, while the added regression controls provide a separate AgeDB MLP case rather than establishing a universal mechanism across datasets.

Thank you for your detailed revisions. I reviewed not only the rebuttal text but also the revised manuscript and the new experiments.

The revisions address the following issues: Table 2 now lists the selected β and a/ρ values ​​for CEB and GPB in each setting, along with the actual test KL value for each selected configuration and its position in the grid. This answers my first question: in 9 out of 12 settings, the actual KL value for the selected GPB configuration is lower than that for the selected CEB configuration, therefore, in these settings, GPB achieves the same or better performance at a smaller certification bound of I(X;Z|Y). The new performance versus actual KL values ​​and performance versus estimated residuals plots (Figures 4c, 5c, and 11) resolve my concerns regarding comparing different methods along the β axis on the ImageNet-100 (MLP) and California Housing datasets. My third question is addressed by repositioning the SDPI threshold to an application of existing research (Wu et al.; Wu and Fischer; Anantharam et al.) and clarifying that the strict contraction condition was not established for any benchmark. Differences in housing R² can be explained by different configurations. The CEB-C baseline is a useful supplement, and the results of GPB-N control and AgeDB regression ablation are also transparently reported, including the smaller GPB-N effect.

Remaining clarification requests:

  1. The text in Figure 11 and Appendix A.4.5. The text states that the EPB curve maintains R² above 0.70 even with residuals close to 0.01 nats, while CEB drops to 0.49 and below. In the rendered plot of Figure 11(b), the lowest CEB value I can see (by bin) is approximately 0.75, around 0.3 nats, and I cannot find a CEB value close to 0.49 in either panel. The EPB ρ=1 curve also drops to approximately 0.70. What points does the value of 0.49 refer to (e.g., ungrouped runs or points outside the plotted range), and how should the chart or text description be revised?

  2. The ranges plotted in Figures 4c and 5c. Figure 4a shows that the accuracy of CEB and OPB (a=1) drops to near-random levels, and Figure 4b shows that the KL value for CEB reaches approximately 1e-4. However, Figures 4c and 5c show that the accuracy only starts from approximately 65-70%, and the KL value or residual only starts from approximately 1 nat. Please indicate whether points with low accuracy or extremely low KL values ​​were excluded or pruned, report the number of runs within each interval, and add an unbinded scatter plot in the appendix if possible. Because there is a positive lower bound for the KL value of OPB, comparisons of matching KL values ​​should only be made if the curves overlap; the extent of overlap should be stated in the text. Furthermore, Figure 5c in the version I saw lacks a legend, even though CEB-C is discussed in the text.

  3. Residual Information Estimator. Class-conditional mixture models contain query points, therefore the KL value for each sample is limited by log N_y. Please report the N_y values ​​for ImageNet-100 and explain whether the upper limit of the x-axis in Figure 5c is close to this limit. For the Housing dataset, note that the KL values ​​estimated by the kernel mixture model are based on a reference value after label smoothing, and their bias depends on the bandwidth. I don't think this is an error, but the wording should reflect this.

  4. Conclusion Calibration. Section 3.8 states that the information plane estimation does not contain mechanistic conclusions, but it concludes by saying, "Therefore, the advantage lies in a better compression-accuracy tradeoff than the offset of the β-parameterization." Appendix A.4.5 draws the same conclusion from the β-axis in Figure 8. I suggest limiting this conclusion to the two settings examined. Similarly, the statement in Section 3.2 that CEB "cannot maintain accuracy under strong compression" is inferred from the location of the validation choice, while validation only indicates that it favors weak compression. This conclusion still mentions the collapse protection guarantee and exclusion above the compression threshold, and the statement that the gains in all twelve settings are "primarily attributable to geometric priors and IB terms" is based on ablation experiments in two MLP settings.

  5. Regression Results for CEB-C. In the regression analysis, CEB-C significantly outperformed GPB on the STS-B dataset, performed comparably to GPB on the ZINC and AgeDB (MLP) datasets, while GPB performed better on the Housing and AgeDB (CNN) datasets. A brief discussion of what these results might suggest for the role of tandem results in regression analysis would be helpful.

Overall, the revised version answers my main questions with complete experiments; the rest involves clarification and wording rather than the validity of the main results. Therefore, provided that points 1 and 2 need clarification, I will raise the score to 3 (barely acceptable).

Answer for Comment

Thank you for the detailed reading and for the precise clarification requests. We have revised the manuscript accordingly; the changes are summarized below. (Note that the appendix reorganization renumbered the figures: Fig. 11 in your numbering is now Fig. 9, and the new unbinned scatter is Fig. 10.)

1. Points excluded from the plotted range (Figs. 4(c), 5(c), and 9)

Yes, points were excluded, and this is now stated explicitly. In each information-axis panel, the configurations at random-level accuracy, which are exactly the configurations with the smallest realized KL (Fig. 4(c)) and the smallest estimated residuals (Figs. 5(c) and 9), are excluded from the plotted range, because including them would stretch both axes: the accuracy/R2R^2 axis down to chance level and the KL/residual axis leftward, flattening the informative part of the curves. The clarification is added in Sec. 3.7 (main text) and in the caption of Fig. 9. The value 0.490.49 refers to CEB's leftmost deep-collapse configurations on Housing, which fall outside the plotted range and are covered by the exclusion noted in the caption of Fig. 9. We also revised the accompanying EPB statement: the text now states that the EPB curves keep R2R^2 above 0.600.60 down to residuals near 0.010.01 nats, a threshold that holds for every EPB curve, including ρ=1\rho{=}1, whose minimum is ≈0.70\approx0.70 at the weakest-compression end.

2. Unbinned scatter plot in the appendix

We added the unbinned scatter version as Fig. 10 in the appendix: one point per configuration, with no binning and no trimming, for the four information-axis panels of Figs. 4(c), 5(c), and 9. The configurations underlying the binned curves, including those excluded from the plotted range, are directly visible there. Each binned point in the main panels is the mean over the configurations falling in that bin.

3. Overlap restriction for matched-KL comparisons

We agree that OPB's realized KL has a positive lower bound set by its anchor geometry, so matched-KL comparisons are meaningful only where the plotted curves overlap. The text now states this restriction explicitly: matched-KL and matched-residual comparisons are restricted to the overlap of the plotted ranges, where both methods have configurations above the random-accuracy floor; outside this overlap OPB's KL is bounded below by its anchor geometry.

4. Legend of Fig. 5(c)

The missing legend was a rendering choice to avoid the CEB-C curve overlapping the legend; the panel now carries the legend in the same position as the other panels, including CEB-C.

Thank you for your further revisions. I checked the new version against my November 16 comments.

Both requirements have been addressed. Section 3.7 and the Figure 9 caption now explain the excluded low-performing configurations. The 0.49 in Appendix A.4.5 refers to excluded CEB configurations on California Housing, which are visible in the unbinned Figure 10. The EPB threshold is now 0.60, the overlap limits for matching comparisons have been explained (Sections 3.7–3.8), and a legend has been added to Figure 5(c). Table 14 extends the control of GPB with GPB-N to all seven classification settings, which also benefits this paper.

On November 16th, I stated that I would raise the score to 3 points once points 1 and 2 were clarified. Now that they have been clarified, I need to explain why it remains at 2 points. My decision reflects a reassessment of the contribution. After reviewing the final revised draft, I reassessed the magnitude of the contribution and now place it slightly below the acceptable standard. I acknowledge that a rational reader might place this slightly above the acceptable level.

  1. Size of the controlled increment. As Appendix A.5 explains, OPB builds on CEB's Consistent Classifier. A key methodological addition is the prescribed geometry. Table 14 measures this: +0.15 to +1.18 points, with confidence intervals not containing zero in 6/7 of the settings, and an inconclusive difference on MNIST-CNN. According to reports, GPB-N (which leaves the pairwise anchor geometry unconstrained) has covered approximately 0.94 points of GPB's 1.63-point advantage over CEB on ImageNet-100 MLP and approximately 2.17 points of its 3.11-point advantage on ImageNet-100 CNN. GPB-N fixes the anchor norms while leaving their spacing free, so the comparison measures the benefit of enforcing equal separation. On the Cora and IMDb datasets, geometric features are the main advantage of GPB over CEB, but GPB does not significantly outperform the strongest baseline model (Base +0.4 [-0.1, +1.1]; NCBD +0.2 [-0.5, +0.7]). I believe this is a real but small improvement, shown with a 256-dimensional bottleneck on the tested datasets and backbones.

  2. Scope of Information Axis Conclusions. Figure 4(c) uses realized KL against each method’s own reference distribution. Equal upper bounds on I(X;Z|Y) need not imply equal residual information. Figures 5(c) and 9(b) use mixture-based estimates of residual information, which provide a more direct comparison. The ImageNet-100 mixture model includes each query in its own class reference, while the Housing estimate depends on the kernel bandwidth. This paper does not report the sensitivity of the matching residual comparison to these choices. Sections 3.8 and A.4.5 still conclude that the advantage lies in an improved trade-off beyond a change in β parameterization. I would qualify this claim to the two settings examined. Section 3.2 still interprets the locations of validation-selected configurations on the KL grids as evidence that CEB cannot maintain accuracy under strong compression.

  3. Theory and Benchmarking. Theorem 2.2, Corollary A.1, and Equation (15) are population-level results, as explicitly stated in the paper. The two-point controlled study illustrates the population results. Their assumptions have yet to be established on the benchmarks. Therefore, these results describe what the objective function selection is about, but do not predict when or by how much GPB improves benchmark performance. Statements in Section 3.5, such as “gain depends on probabilistic coupling,” make a broader attribution than the supporting ImageNet-100 MLP ablations establish. To differentiate the evidence: the improvement in overall accuracy (Table 1, achieving the best average in 6 of the 7 classification settings, but not all results significantly outperforming the strongest baseline) and the gain from geometric control (Table 14, results stable but small) are reliable. A broader probabilistic coupling attribution relies on ImageNet-100 MLP ablation experiments. In my assessment, the size and scope of the demonstrated contribution put the paper just below the acceptance threshold. I maintain a score of 2 (weak rejection).

Reviewer_d2VSOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose Geometric Prior Bottleneck (GPB) that learns features based on a preset label geometry. For classification task, each class label is assigned a fixed vector, with these vectors being mutual orthogonal to each other. The encoder is trained to match to a Gaussian distribution around the vectors. The resulting model outperforms other VIB, SVIB, etc information-bottleneck like objective for supervised learning.

Critical strengths
  • The theoretical derivation seems solid.
Critical weaknesses
  • The writing is very unclear. The second paragraph should start with a review on information bottleneck without directly jumping into CEB vs. VIB. The authors also didn't explain what's CEB and VIB.
  • In the abstract, there is also a mentioning of OPB without proper definition.
  • Empirical results are not super strong compared to other baseline.
Questions for authors

N/A

2 comments

Answer for the Critical Weaknesses

Thank you for identifying these issues concerning the organization and definitions in the manuscript, the insufficient introduction of OPB in the Abstract, and the strength of the empirical evidence. We have revised the manuscript to address each point and to calibrate our claims to the results.

1. Organization and Definitions of IB, VIB, and CEB

We appreciate the concern that the Introduction may appear to move directly into VIB and CEB without first reviewing the Information Bottleneck framework or explaining what the two methods are. We would like to clarify the intended progression of the Introduction. The second paragraph does not directly compare CEB with VIB. Instead, it reviews geometric and contrastive representation-learning approaches, including neural-collapse-inspired frames, equidistant or orthogonal constructions, and contrastive objectives. This discussion establishes one side of our motivation: these approaches explicitly organize target information in the representation geometry, but generally leave the compression of target-irrelevant input variation implicit. This setup is important because our method connects an explicit target geometry with information-bottleneck compression.

The third paragraph then reviews Information Bottleneck methods and introduces the relevant distinction between VIB and CEB. The Variational Information Bottleneck (VIB) and related IB methods make compression explicit by matching the encoder distribution to a marginal reference, thereby controlling the input information retained by the representation. The Conditional Entropy Bottleneck (CEB), in contrast, uses a label-conditional reference and penalizes the residual input information retained after the label is known. Thus, the paragraph first explains what VIB and CEB are and then contrasts their marginal and label-conditional compression mechanisms.

Although this was the intended structure, we recognize that the transition between the two strands of work and the explanations of the methods were not sufficiently prominent. We have therefore strengthened the transition from geometric and contrastive learning to IB methods and now give the concise explanations of VIB and CEB when they first appear. Sec. 2.1 further gives the formal CEB objective and its conditional-KL decomposition, explicitly stating that CEB replaces the marginal reference used by VIB with a label-conditional reference.

2. Proper Introduction of OPB in the Abstract

We agree that OPB was mentioned in the previous Abstract without being properly introduced. We have revised the Abstract to give its full name, Orthogonal Prior Bottleneck (OPB), and to briefly describe it as the classification instance of GPB based on an equal-radius orthogonal frame and anchor-energy prediction. For consistency, we also give the full name of the regression instance, Equidistant Prior Bottleneck (EPB), and briefly describe its projection tied to an isometric label axis. The priors and readouts of both instances are then formally defined in Sec. 2.2.

3. Empirical Strength

We agree that the empirical evidence should not be presented as uniformly strong across all tasks. We therefore revised the main-results analysis to separate classification and regression, whose results support different conclusions. On classification, OPB achieves the highest mean accuracy in six of the seven settings. The classification results therefore provide the main empirical evidence for the utility of the proposed construction. On regression, EPB does not show consistent superiority over existing methods: it achieves the highest mean in one of five settings, and its empirical benefits are setting-dependent. We revised the Abstract, Introduction, Main Results, Conclusion, and Limitations accordingly, positioning EPB as a continuous-label extension with an exact population characterization rather than as a consistently superior regression method.

We also expanded the empirical evidence beyond the aggregate benchmark table. The added experiments include a compression sweeps, item-wise ablations, information-plane diagnostics, the CEB-C baseline, comparisons at matched realized KL, estimated residual information, regression-side mechanism controls, and a norm- and scale-controlled OPB--GPB-N comparison across all seven classification settings. Together, these additions provide broader evidence for the proposed mechanism, enable more controlled comparisons with relevant baselines, and clarify both the settings in which the method is effective and the limitations of the current empirical evidence.

Additional Revisions for Clarity

Beyond these three direct changes, we also reorganized the theoretical narrative. The previous manuscript moved too quickly from collapse-related observations to a repair and then to performance claims. The revised manuscript instead separates three questions: the geometric ambiguity of the unconstrained conditional-KL certificate, the representation selected after restricting the reference geometry, and the finite-sample performance observed on benchmarks. The central theoretical contribution is now stated as an objective-level characterization: Theorem 2.2 identifies the conditional-expectation anchor selected by the geometry-constrained objective and characterizes the predictive structure of its tied readouts under the stated assumptions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.