- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a Geometric Prior Bottleneck (GPB), which constrains the means of label-conditional Gaussian priors and uses the same anchors for prediction. Its classification variant, OPB, uses equal-radius orthogonal anchors; its regression variant, EPB, places normalized targets on a label-dependent axis. The theoretical analysis studies geometric indeterminacy in unrestricted conditional references, conditions for information collapse, and the predictors selected by the constrained objective. Experiments cover twelve dataset–backbone settings, supplemented by geometry diagnostics and classification ablations.
Critical strengths
The paper provides a useful analysis of how conditional compression interacts with representation geometry. It distinguishes geometric indeterminacy of the conditional KL from information collapse of the full objective, and derives conditional guarantees for the predictors learned with constrained priors.
Critical weaknesses
-
The paper does not clearly establish its methodological novelty beyond existing conditional and anchor-based bottlenecks. Conditional-reference-based prediction already appears in CEB’s Consistent Classifier, while Text-Anchored IB(https://arxiv.org/abs/2607.01657) also uses shared anchors for conditional regularization and classification. GPB differs in its Gaussian formulation, prescribed distance constraints, and theoretical analysis, but Table 14 does not adequately explain what these differences add beyond existing designs. The contribution needs to be established through these specific constraints and their analysis, rather than through the general combination of geometry, conditional KL, and a tied readout.
-
The experiments support the constrained model’s usefulness more clearly than the proposed explanation of its gains. The GPB-G comparison does not clarify whether equal anchor norms and effective readout scale remain controlled when the anchor geometry is released. Without these controls, the energy classifier can also acquire different class-dependent biases, so the improvement does not isolate the effect of prescribed inter-class distances. These mechanism ablations are also limited to one classification setting. For regression, Appendix A.2.2 shows that the EPB mean penalty partly rescales the supervised squared-error loss, but corresponding controls separating mean matching, off-axis penalties, and covariance regularization are absent. The main regression results are mixed: EPB essentially matches CEB on Housing, falls below Base on STS-B, and falls below AdaCap on ZINC. The current evidence does not support a general mechanism claim across classification and regression.
Questions for authors
-
Given CEB’s Consistent Classifier and Text-Anchored IB, what is the specific methodological contribution of GPB beyond these existing designs? Please explain how the distinction claimed in Table 14 accounts for their use of conditional references and shared prediction anchors. Does the CEB baseline in Table 1 use a free prediction head or the original conditional-reference-based classifier?
-
Does GPB-G retain fixed, equal anchor norms and the same effective readout scale as the full model? If not, its energy scores also acquire different class-dependent biases. What reported evidence distinguishes the benefit of the prescribed distances from that of restricting anchor norms and classifier scale?
-
Are there regression counterparts to the full-KL versus mean-matching controls in Appendix A.4.4? Given the EPB decomposition in Appendix A.2.2, what reported evidence establishes an additional benefit from the off-axis and covariance penalties beyond the supervised mean-matching term? If these effects have not been separated, please limit the mechanism conclusions accordingly.
3 comments
Table 1 shows strong classification performance, with GPB achieving the highest mean accuracy in six of the seven classification settings. Appendix A.5 clarifies how OPB's prescribed class geometry differs from the geometry derived from a frozen text encoder in Text-Anchored IB. The new ablations in Table 3 answer my questions about the ablations. With anchor norms and classifier scale controlled, OPB outperforms GPB-N by 0.69 percentage points on ImageNet-100 MLP (95% CI [0.32, 1.04]). On AgeDB MLP, removing the off-axis mean penalty reduces R² by approximately 0.04, while removing the covariance regularizer reduces it by approximately 0.02.
My remaining concern is the limited methodological advance over prior work. CEB's Consistent Classifier already uses the same class-conditional reference distribution for regularization and prediction. OPB builds on this principle by constraining the Gaussian references to have a fixed shared covariance and equal-norm, mutually orthogonal means. I recognize the value of the analysis of how these constraints affect the learned representations. However, while the overall classification results are strong, the controlled experiment directly isolating the contribution of the prescribed class geometry shows a modest gain in only one classification setting. The regression results in Table 1 also do not demonstrate consistent improvements over existing methods. Overall, I still find the methodological advance and the demonstrated benefits of the proposed geometric constraints too limited to warrant acceptance. I therefore maintain Rating 2, Confidence 3.
Answer for Comment
Thank you for clarifying your remaining concern. We agree that reusing a class-conditional reference for regularization and prediction is already established by CEB’s Consistent Classifier, and we do not claim this principle, or orthogonal class frames themselves, as new. Our methodological change is to constrain the conditional-reference family through prescribed geometry. Our theoretical contribution is to characterize how this restriction changes the representations and predictors selected by the resulting objective.
To address the concern that the controlled evidence was limited to one setting, we extended OPB--GPB-N comparison from ImageNet-100 (MLP) to all seven classification settings. The complete results added to the ablation appendix are:
| Setting | OPB GPB-N [95% CI] | GPB-N mean std |
|---|---|---|
| MNIST (MLP) | [, ] | |
| MNIST (CNN) | [, ] | |
| ImageNet-100 (MLP) | [, ] | |
| ImageNet-100 (CNN) | [, ] | |
| Cora (GNN) | [, ] | |
| IMDb (RNN) | [, ] | |
| AG News (RNN) | [, ] |
OPB improves mean accuracy in six of seven settings, with the corresponding paired 95% confidence intervals excluding zero. The remaining setting shows no statistically significant difference; no setting significantly favors GPB-N. The gains vary in magnitude, but the additional results show that the benefit of prescribing the pairwise geometry is not confined to ImageNet-100 MLP.
We also agree that the regression results do not establish consistent superiority. We have therefore revised the Introduction, Main Results, Conclusion, and Limitations to state explicitly that classification provides the main empirical evidence, while EPB is a continuous-label extension with an exact population characterization and setting-dependent empirical performance.
We hope these additions clarify that the contribution is not the shared-anchor classification principle itself, but the prescribed conditional-reference geometry, the characterization of the representation and predictor selected by the resulting objective, and the controlled evaluation of the geometric constraints.
Answer for Q1
We agree that conditional references, shared anchors, and conditional-reference-based prediction are not individually new. We therefore removed the previous combination-novelty claim and the associated comparison table. The Introduction and Related Work now define the contribution more narrowly: GPB restricts the conditional reference to a fixed-covariance Gaussian family with prescribed relations among its means, ties prediction to that reference, and analyzes the representation selected by this restricted objective.
The original CEB baseline uses a separately learned prediction head. To isolate the role of conditional-reference-based prediction, we added CEB-C, which uses CEB's original Consistent Classifier, to the main comparison and the compression analyses in Secs. 3.2 and 3.7--3.8. We also added a direct comparison with CEB-C and Text-Anchored IB in the related-work appendix. The revised discussion distinguishes GPB's explicitly prescribed geometry from the geometry inherited from a frozen text encoder and does not suggest that other shared-anchor methods necessarily exhibit the limitations identified for unconstrained references.
Answer for Q2
We agree that the previous GPB-G comparison changed the pairwise geometry, anchor norms, and effective readout scale simultaneously and therefore could not isolate the effect of the prescribed inter-class relations. We replaced GPB-G with GPB-N, which releases the prescribed pairwise relations while retaining the anchor norm, readout scale, stochastic encoder, conditional KL, prior variance, and energy readout. The revised definition and discussion appear in Sec. 3.5 and the ablation appendix. We explicitly limit the resulting mechanism interpretation to the ImageNet-100 MLP setting in which this controlled comparison was performed.
Answer for Q3
We added two regression controls on the AgeDB MLP setting. EPB-noOff retains covariance regularization while removing only the off-axis mean penalty, whereas EPB-M retains full vector mean matching while removing the covariance term. The supervised component, stochastic encoder, tied readout, and model-selection procedure are held fixed so that the two additional components can be examined separately.
We also revised the main-results discussion to state the mixed regression outcomes directly. These controls support only a setting-specific interpretation on AgeDB MLP and are not used to explain every classification or regression result.