acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data

Abstract

Supervised contrastive learning and its balanced variants have demonstrated their effectiveness for representation learning under long-tailed distributions. However, although prior analyses suggest that their optimal representations may exhibit a regular simplex geometry, whether such a geometry is indeed exhibited remains insufficiently understood. In this work, we present a theoretical framework for these objectives that characterizes whether a regular simplex is formed at the optimum. By applying this to representative objectives, we show that these objectives fail to form a regular simplex at the optimum because they remain dependent on the number of samples per class. Motivated by this analysis, we introduce structural balancing that can be applied to contrastive objectives, and propose a fully balanced contrastive objective that provably forms a regular simplex geometry at its optimum. Both theoretical and empirical results demonstrate that the proposed method yields a more balanced representation space and improves performance on long-tailed recognition and transfer learning tasks. The code will be released.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation learning, Long-tailed recognition

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qPLwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the representation geometry of supervised contrastive learning under long-tailed data distributions. The authors develop a theoretical framework and show that the regular simplex is not a global minimizer of SCL or BCL under class imbalance, due to the remaining dependence on the number of samples per class.

To address this issue, the paper introduces structural balancing and proposes Fully Balanced Contrastive Learning (FBCL). The proposed method balances the aggregation across anchors and recovers the regular simplex as a global optimum. The paper further analyzes the effect of structural balancing on intra-class variance and shows that it makes the variance penalty approximately independent of the number of samples per class.

Experiments on ImageNet-LT and iNaturalist show improvements in long-tailed recognition, while experiments on multiple downstream datasets evaluate the transferability of the learned representations. The results also show that structural balancing can be incorporated into other long-tailed recognition methods and consistently improves their performance.

Critical strengths

1: The paper has a clear and coherent presentation, and I think the motivation and the main line of the paper are easy to follow. Through the theoretical framework developed in the paper, the authors identify the remaining dependence on the number of samples per class in SCL and BCL, which naturally motivates structural balancing and FBCL.

2: The idea is simple and easy to understand, while providing a novel and useful way to improve performance under class imbalance. By balancing the aggregation across classes, structural balancing removes the dependence on the number of samples per class in the lower bound and allows FBCL to recover the regular simplex as a global optimum. Empirically, FBCL improves over SCL and BCL on long-tailed recognition (Tables 1 and 2) and transfer learning (Table 3).

Critical weaknesses

As acknowledged by the authors, there is a gap between the theoretical setting and practical mini-batch training:

1: The main theoretical analysis assumes that each mini-batch contains all classes, while in practice a mini-batch generally contains only a subset of classes.

2: The dimensional condition required for the regular simplex result is not satisfied on iNaturalist.

Although FBCL still shows improvements in these practical settings, it remains unclear to what extent the regular simplex result can explain the empirical improvements observed in practice.

Questions for authors

1: As mentioned in the weakness, there is a gap between the theoretical and practical settings. Could the authors further clarify to what extent the theoretical regular simplex result explains the empirical improvements of FBCL in practical settings?

2: I read Appendix D.7, but I would still like the authors to further clarify the difference between structural balancing and conventional re-weighting, especially since they are equivalent in the full-batch setting. This distinction is important to the contribution of SB and would be helpful to clarify in the main paper.

2 comments

We appreciate the reviewer’s comments, which highlights the importance of connecting our theoretical results to practical improvements and clarifying the distinction between structural balancing and conventional re-weighting. We hope that the additional analyses and explanations help address these concerns.

Q1. Connecting the theoretical geometry with empirical improvements

Theoretical relevance: To the extent that class coverage increases, both geometry and accuracy improve, and FBCL remains effective even without full coverage

To examine how the theoretical full class setting relates to practice and empirical performances, we conducted a controlled experiment that increases the class coverage of batches while keeping the class subset and batch size fixed. We evaluate both classification accuracy and an NC2 metric [1], which measures the deviation of the class-center Gram matrix from the regular simplex Gram matrix.

We construct a 100 class subset of ImageNet-LT that approximately preserves the proportions of many-, medium-, and few-shot classes in the original dataset. We compare original sampling with controlled sampling, both with a batch size of 256. The controlled batch contains one image from every class, with the remaining 156 images drawn using the original sampling scheme. This increases average class coverage from 71.24% to 100.00%, satisfying the full class coverage condition of our analysis.

MethodOriginal sampling: NC2Original sampling: accuracy (%)Controlled sampling: NC2Controlled sampling: accuracy (%)
SCL1.08355.180.95956.26
BCL1.05655.460.94256.87
FBCL1.04855.790.93557.09

Under controlled sampling on the same 100 class subset, both NC2 and classification accuracy improve. The Pearson correlation between NC2 and classification accuracy across the six results (three methods and two sampling conditions) is -0.964, a negative correlation. This indicates that improved geometry is related to higher classification accuracy in these experiments.

Within each sampling condition, FBCL achieves larger improvements over SCL than BCL. Under original sampling, BCL reduces NC2 by 2.49% and improves accuracy by 0.28% relative to SCL, whereas FBCL achieves a 3.23% NC2 reduction and a 0.61% accuracy improvement. Under controlled sampling, the corresponding improvements are 1.77% and 0.61% for BCL, compared with 2.50% and 0.83% for FBCL. These results connect FBCL's larger geometric improvements with its larger gains in classification accuracy.

Regarding the dimensional assumption, understanding the geometry reached in practical training when this condition is not satisfied remains an important research direction under both balanced and imbalanced distributions, as discussed in Section 6. To explore this behavior in a simple setting, we extend the toy experiment in Figure 1 to four and five classes with two dimensional embedding space. Both settings violate the dimensional requirement for a regular simplex. The following table reports approximate angular gaps, in degrees, between neighboring class centers.

Number of classesBCL: angular gaps (°)FBCL: angular gaps (°)
4118, 117, 84, 4190, 90, 90, 90
5108, 100, 78, 73, 172, 72, 72, 72, 72

FBCL produces approximately uniform angular spacing in both cases, while BCL produces uneven spacing. These observations suggest that FBCL can encourage class centers to spread more uniformly within the available representation space even when the dimensional assumption is not satisfied.

We hypothesize that a similar tendency may contribute to the improvements on iNaturalist and may also arise on other datasets where this condition is violated. Moreover, Proposition 3 does not rely on the dimensional condition. That is, the class-level lower bound of FBCL remains independent of class cardinalities regardless of the representation dimension.

[1] Papyan, et al. "Prevalence of neural collapse during the terminal phase of deep learning training." PNAS, 2020.

Q2. Structural balancing(SB) and conventional re-weighting

Structural and conventional re-weighting behave differently in practical mini-batch training

We agree that SB and conventional re-weighting coincide in the full-batch setting when applied to the same loss, and therefore share the same geometric guarantee in that setting. However, they behave differently in practical mini-batch training. Specifically, they differ in the level at which class contributions are balanced. Conventional re-weighting balances class contributions in expectation across batches using dataset-level class frequencies, whereas SB balances class contributions within each batch by assigning equal outer weight to the mean anchor loss of each class present.

To examine this difference, we analyze the outer weights assigned to the same BCL objective. The total outer weight under conventional re-weighting ranges from approximately 0.6 to 1.6 times its mean across batches, whereas SB maintains a total weight of 1 in every batch. In the corresponding classification experiments, SB achieves 52.14 ± 0.15% accuracy, compared with 51.45 ± 0.45% for conventional re-weighting. The weight analysis is conducted on CIFAR-100-LT (IR=100) over three seeds. We will add this comparison in the revision.

Reviewer_HTaMOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the representation geometry of supervised contrastive learning under long-tailed distributions. The authors first establish a theoretical framework to analyze whether existing contrastive learning objectives can achieve a regular simplex geometry at the global optimum. Based on this framework, they show that both supervised contrastive learning (SCL) and balanced contrastive learning (BCL) fail to achieve this geometry under class imbalance. To address this issue, the authors introduce structural balancing and propose Fully Balanced Contrastive Learning (FBCL), which is theoretically guaranteed to achieve a regular simplex at the optimum under the stated assumptions. Experimental results on long-tailed recognition and transfer learning tasks demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper provides an interesting and meaningful theoretical analysis, revealing an important limitation of existing supervised contrastive learning methods under long-tailed distributions.
  2. The proposed FBCL method is simple and theoretically well-motivated, providing a principled solution to the identified representation geometry problem.
  3. The experimental results are convincing, showing consistent improvements on long-tailed recognition and transfer learning tasks, with additional analyses supporting the theoretical findings.
Critical weaknesses
  1. The theoretical analysis relies on some ideal assumptions. In particular, it assumes that each batch contains all classes and that the feature dimension is sufficiently large. These assumptions may not always hold in practical long-tailed learning, especially on datasets with many classes. It would be interesting to further investigate whether similar theoretical properties hold under more general settings.
  2. The performance improvements over existing methods are sometimes relatively modest. For example, FBCL improves the overall accuracy over BCL by 0.6% on ImageNet-LT and 0.9% on iNaturalist. Although these improvements are consistent, further analysis of the relationship between geometric improvements and classification accuracy would strengthen the paper.
  3. The experiments mainly use ResNet-50 as the backbone. Additional experiments with other network architectures, such as Vision Transformers, would help demonstrate the generality of the proposed method.
  4. The practical convergence behavior of FBCL could be further analyzed. Although the paper proves that FBCL achieves a regular simplex at the global optimum under its theoretical assumptions, it remains unclear how closely the learned representations approach this geometry during practical mini-batch training. More analysis of the representation geometry during training would be helpful.
  5. The differences between FBCL and Targeted Supervised Contrastive Learning (TSC) need further clarification. Both methods aim to learn a more uniform representation geometry under long-tailed distributions. In particular, the class-level lower bound of FBCL is closely related to the uniformity objective used in TSC, differing only by a constant under the same temperature and normalization settings. The authors should better explain the theoretical differences between the two methods. In addition, a fair comparison under the same training framework would help demonstrate the advantages of FBCL over the explicit target alignment strategy used in TSC.
Questions for authors

Please see the weakness.

4 comments

We appreciate the importance of the reviewer’s questions regarding the connection between theory and practice, the applicability to different architectures, and the comparison with TSC. We hope that the additional analyses and experiments below help address these concerns.

W1, W2, and W4. Further analysis of representation geometry in practical training

We agree that the assumptions may not hold in general and that further analysis of representation geometry in practical training would be helpful. As discussed in Section 6, our theory suggests a geometric tendency induced by the objective, which we further examine through additional experiments designed to approximate or satisfy the full class assumption. We address these comments in the order of W4, W2, and W1, first examining the representation geometry in practical training, then its relationship to classification accuracy, and finally the theoretical assumptions.

W4. Bringing practical training closer to the theoretical assumption is accompanied by improved representation geometry

In the original 1,000 class ImageNet-LT setting, a batch size of 256 does not allow every class to be represented in a batch. We therefore construct a 100 class subset that approximately preserves the proportions of the many, medium, and few class groups in the original dataset. With this subset, including all classes in a batch becomes possible without changing the batch size. We compare original sampling with controlled sampling that guarantees the presence of every class: each controlled batch contains one image from each of the 100 classes, with the remaining 156 images drawn using the original sampling.

To quantify how each setting satisfies the assumption, we use the same configurations as training and measure the average percentage of classes in each batch (represented by only anchors, prototypes are excluded). We refer to this percentage as class coverage. We report NC2 [1], which measures the deviation of the class center Gram matrix from the regular simplex Gram matrix. NC2 at random initialization is reported to examine the change in representation geometry after training.

SettingClass coverage (%)Random init. NC2SCL NC2BCL NC2FBCL NC2
1,000 classes, original sampling19.801.3561.2681.2641.259
100 classes, original sampling71.241.2231.0831.0561.048
100 classes, controlled sampling100.001.2230.9590.9420.935

Across these settings, class coverage increases from 19.80% to 71.24% and then to 100.00%, while NC2 decreases for all three methods. Compared with random initialization, all three methods achieve lower NC2 after training in each setting, which shows the overall change towards the simplex during practical training.

Relative to SCL, BCL and FBCL reduce NC2 by 0.32% and 0.71%, respectively, in the 1,000 class setting. The corresponding reductions are 2.49% and 3.23% in the 100 class setting with original sampling, and 1.77% and 2.50% with controlled sampling. FBCL shows a larger NC2 reduction relative to SCL than BCL in all three settings. Thus, in practice, bringing batches closer to the theoretic assumptions is accompanied by improved representation geometry.

[1] Papyan, et al. "Prevalence of neural collapse during the terminal phase of deep learning training." PNAS, 2020.

W2. Improved representation geometry is accompanied by improved classification accuracy

MethodOriginal sampling: accuracy (%)Controlled sampling: accuracy (%)
SCL55.1856.26
BCL55.4656.87
FBCL55.7957.09

Across the six combinations of methods and sampling conditions, NC2 and classification accuracy exhibit a negative Pearson correlation of -0.964. Within each sampling condition, lower NC2 is consistently associated with higher accuracy across all methods. FBCL, which has the lowest NC2 in both settings, shows a corresponding increase in accuracy from 55.79% to 57.09%, alongside the decrease in NC2 from 1.048 to 0.935. These results show a consistent relationship between improved representation geometry and classification performance.

W1. Representation geometry under insufficient feature dimension

Regarding the dimensional condition, understanding representation geometry when this condition is not satisfied remains an important direction for further analysis under both balanced and imbalanced distributions. To examine this behavior in a simple setting, we extend the toy experiment in Figure 1 to four and five classes with two dimensional embeddings. Neither setting satisfies the dimensional requirement for a regular simplex. The following table reports approximate angular gaps, in degrees, between neighboring class centers on the unit circle.

Number of classesBCL: angular gaps (°)FBCL: angular gaps (°)
4118, 117, 84, 4190, 90, 90, 90
5108, 100, 78, 73, 172, 72, 72, 72, 72

FBCL produces approximately equal angular spacing in both settings, while BCL produces uneven spacing. These toy experiments illustrate how FBCL can encourage a more uniform class arrangement within the available representation space. We therefore hypothesize that a similar tendency may contribute to the improvements on iNaturalist and may also arise on other datasets where the dimensional condition is not satisfied. Moreover, Proposition 3 does not rely on the dimensional condition. That is, the class-level lower bound of FBCL remains independent of class cardinalities regardless of the representation dimension.

W3. Evaluation with Vision Transformers

We appreciate the suggestion to evaluate additional architectures. We compare BCL and FBCL using ViT-T/4 and ViT-S/4 on CIFAR-100-LT with an imbalance ratio of 100. The following table reports classification accuracy (%).

BackboneMethodManyMediumFewAll
ViT-T/4BCL59.2940.6520.6241.16
FBCL59.6340.9920.9341.50
ViT-S/4BCL58.7940.0720.1940.66
FBCL59.2939.8820.5040.86

FBCL achieves higher mean overall accuracy than BCL with both ViT backbones, but the accuracies of ViT models are lower than our ResNet results. The additional experiments support the applicability of FBCL to these Transformer backbones, although the observed benefit is smaller in these settings.

Both backbones are trained for 200 epochs with a batch size of 128 using AdamW. For ViT-T/4, we use a learning rate of 3e−43e-4 and weight decay of 0.2; for ViT-S/4, we use a learning rate of 1e−41e-4 and weight decay of 0.10.1. Within each backbone, BCL and FBCL use the same training configuration.

W5. Theoretical and empirical comparison with TSC

We appreciate the reviewer for highlighting this connection to TSC, a competitive and closely related method. We address the theoretical and empirical differences between the two methods, beginning with the theoretical analysis. Since TSC builds on supervised contrastive learning, we analyze it within the framework developed in our paper: we first derive a lower bound FTSCF_\text{TSC} by aggregating the loss at the class level, and then show that the targeted regular simplex does not satisfy the necessary stationarity condition under class imbalance.

Theoretical result of TSC: The targeted regular simplex cannot minimize the TSC loss under class imbalance

We agree that, under the same temperature and normalization conditions, the class-level lower bound of FBCL coincides with the uniformity objective used for target generation in TSC up to a constant. In TSC, this uniformity objective is used to generate fixed targets, which subsequently guide representation learning through the TSC loss. In contrast, for FBCL, the same uniformity structure is derived directly as the class-level lower bound of the training loss itself, and the regular simplex is guaranteed to be a global minimizer under our theoretical assumptions. As shown below, the targeted regular simplex does not minimize the TSC loss under class imbalance. The corresponding derivation is provided in Appendix C.5 (Eqs. (123)–(127) in the revision).

Following the anchor-wise and class-wise arguments in Lemmas 3 and 4, respectively, we obtain the following bound. We use τ=1\tau=1 as in our theoretical analysis and denote TSC's assigned simplex targets by Zˉ∗\bar{Z}^{\ast}:

LTSC≥1B∑y∈[K]By[(1+λ)log⁡Γy+λ(1−zˉy⋅zˉy∗)]=:FTSC,\mathcal{L}_{\text{TSC}} \ge \frac{1}{B} \sum_{y\in[K]} B_y \left[ (1+\lambda)\log \Gamma_y + \lambda(1-\bar{z}_y \cdot \bar{z}_y^{\ast}) \right] =:F_{\text{TSC}},

where λ≥0\lambda \ge 0 is the weight of the contrastive term between each anchor and its assigned target in TSC, and Γy=Byexp⁡(∣zˉy∣2−1+1By2∑i∈By(zi⋅z~i−1))+e−1∑k∈[K]∖yBkexp⁡(zˉy⋅zˉk)+e−1∑k∈[K]exp⁡(zˉy⋅zˉk∗)\Gamma_y = B_y \exp \left( \\| \bar{z}_y \\|^2 -1 + \frac{1}{B_y^2} \sum_{i \in \mathcal{B}_y} (z_i \cdot \tilde{z}_i - 1 ) \right) + e^{-1} \sum_{k \in [K] \setminus \\{y\\} } B_k \exp(\bar{z}_y \cdot \bar{z}_k) +e^{-1} \sum_{k\in[K]} \exp(\bar{z}_y \cdot \bar{z}_k^{\ast}).

Here, z~i\tilde{z}_i denotes the augmented feature paired with ziz_i in TSC. The resulting bound retains the class averaged augmentation alignment, and equality holds when both the original and augmented features satisfy class collapse (C1).

We next examine whether the targeted regular simplex is the optimal geometry for the TSC loss, whether the features collapse to their targets Zˉ∗\bar{Z}^{\ast}. At this configuration, using d=−1/(K−1)d=-1/(K-1) as in Lemma 2, we have Γy∗=By+1+ed−1(B−By+K−1)\Gamma_y^{\ast} = B_y + 1 + e^{d-1} (B-B_y+K-1).

At Zˉ=Zˉ∗\bar Z=\bar Z^{\ast}, the target-dependent terms contribute only gradient components parallel to zˉy∗\bar z_y^{\ast}, since ∑k≠yzˉk∗=−zˉy∗\sum_{k\ne y}\bar z_k^{\ast}=-\bar z_y^{\ast} by the regular simplex property. Thus, these terms do not affect the necessary condition for regular simplex optimality under the unit-norm constraint. Applying the argument of Lemma 2 to the remaining class-center terms, we obtain, for every y∈[K]y\in[K],

ByBkΓy∗+BkByΓk∗=Constant, for all k∈[K]∖y.\frac{B_y B_k}{\Gamma_y^{\ast}} + \frac{B_k B_y}{\Gamma_k^{\ast}} = \mathrm{Constant}, \text{ for all } k\in [K] \setminus \\{y\\}.

Under the long-tailed assumption, we choose two classes a,ba,b with Ba>BbB_a > B_b and, since K≥3K \ge 3, y∉a,by \notin \\{a,b\\}. Then, we show that the difference between the results of substituting kk with k=ak=a and k=bk=b is not zero:

&\left(\frac{B_y B_a}{\Gamma_y^{\ast}} + \frac{B_a B_y}{\Gamma_a^{\ast}}\right) - \left(\frac{B_y B_b}{\Gamma_y^{\ast}} + \frac{B_b B_y}{\Gamma_b^{\ast}} \right) \\\\ &\qquad = B_y(B_a-B_b) \left[ \frac{1}{\Gamma_y^{\ast}} + \frac{1+e^{d-1}(B+K-1)}{\Gamma_a^{\ast} \Gamma_b^{\ast}} \right] > 0. \end{aligned}$$ The coefficients therefore cannot be constant, so the necessary condition for regular simplex optimality in Lemma 2 is not satisfied. Since $\mathcal{L}_{\text{TSC}} = F_{\text{TSC}}$ at class collapse, Lemma 1 transfers this non-optimality result to the original TSC loss. Thus, we show that the TSC loss does not attain its optimum at the targeted regular simplex under class imbalance.

Empirical result of TSC: FBCL shows better classification accuracy than TSC under both training frameworks

To complement the theoretical analysis, we compare TSC and FBCL on CIFAR-100-LT at imbalance ratios of 10, 50, and 100 under both training frameworks. Specifically, we evaluate FBCL within the TSC framework and TSC within the FBCL framework. The following table reports classification accuracy.

Training frameworkMethodIR = 10IR = 50IR = 100
TSCTSC54.6344.6941.03
FBCL56.1845.7641.92
FBCLTSC63.2355.3650.75
FBCL64.0056.0052.14

Within the TSC framework, FBCL improves accuracy over TSC by 1.55%, 1.07%, and 0.89% at imbalance ratios of 10, 50, and 100, respectively. Within the FBCL framework, the corresponding improvements are 0.77%, 0.64%, and 1.39%. FBCL achieves higher mean accuracy across all three imbalance ratios in both frameworks, including when it is evaluated within TSC's training protocol.

For the TSC framework, we follow its two-stage training protocol. In the first stage, the encoder is trained using only the contrastive objective. For FBCL, class-complement is therefore not applied in this framework. In the second stage, an LDAM classifier is trained on the learned representations.

Reviewer_zJgDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes the representation geometry induced by supervised contrastive objectives under class imbalance. It derives class-center lower bounds that are tight at within-class collapse and a necessary stationarity condition for a regular simplex. Applying this framework shows that unequal anchor-class weights prevent a simplex from minimizing SCL and BCL. Structural balancing averages anchor losses by class, yielding FBCL and a regular-simplex global optimum under the stated full-class and dimensional assumptions. An alignment-term decomposition explains how the weighting changes intra-class variance penalties. Experiments on long-tailed recognition, transfer and combinations with other methods report consistent, generally modest improvements.

Critical strengths
  1. The correction to the geometry argument for BCL is consequential. Balancing negative classes does not remove the class weights introduced by aggregating anchors. The tight class-center reduction and the simplex stationarity test make this distinction explicit, and the FBCL result uses a separate global-bound argument rather than treating stationarity as sufficient (pp.4–6; Appendix C). This sharpens our understanding of what existing contrastive objectives actually optimize.

  2. The practical modification is simple and supported by useful controls. The matched BCL-to-FBCL comparison improves ImageNet-LT accuracy from 56.03 to 56.64 and iNaturalist from 71.3 to 72.2, with repeated-run statistics (Appendix D.5). Transfer results and combinations with DBM and ProCo broaden the evidence. Importantly, Appendix D.7 does compare against inverse-prior weighting, effective-number weighting and class-balanced sampling.

Critical weaknesses
  1. The bridge from the theorem to the practical objective remains qualitative. The theorem assumes full anchor-class coverage, whereas Eq.(150) averages over only the classes present in a sampled batch. Even after within-batch balancing, expected class weight is E[1{y is present}/|Y_B|], which is generally frequency-dependent. Complement prototypes do not create anchors for absent classes. The paper acknowledges the issue, and the iNaturalist dimensional mismatch, but does not quantify how the practical geometry or gains depend on anchor coverage (pp.9,28–29).

  2. The evidence that improved geometry explains the accuracy gains is limited. Mean inter-class cosine similarity does not establish equiangularity, and the variance analysis isolates the alignment term rather than the complete objective (Appendices C.5 and D.6). Gains over matched BCL are useful but modest; transfer uncertainty is not reported in the main table. Comparisons against SCL also change class-complement usage, so they are less diagnostic than the matched BCL comparison.

Questions for authors
  1. Can you quantify anchor-class coverage and the effective class weighting of Eq.(150) under your sampler, and relate them to the observed geometry? A controlled coverage or batch-composition experiment would clarify how the full-class result informs practical training.
  2. Can you report a direct simplex diagnostic, such as distance of the normalized center Gram matrix from the simplex Gram matrix, alongside angle variability and class-mean norms? Please distinguish evidence of greater separation from evidence of equiangularity.
  3. Are prototype use and normalization held fixed in the PB/NB/SB ablations? Please also report uncertainty for the transfer differences and clarify the minor SimCLR standard-deviation discrepancy between Tables 1 and 6.
6 comments

We appreciate the reviewer’s constructive comments which help us clarify the connection between our analysis and empirical findings. We hope that our responses below help address the reviewer’s concerns. For a consistent discussion, we first address Q2 to introduce the geometry metric used in our response to Q1.

Q2 (W2). Direct evaluation of the regular simplex geometry

FBCL consistently achieves lower NC2 across datasets and imbalance ratios We additionally evaluate the deviation from regular simplex geometry using NC2 [1], which measures the distance between the class center Gram matrix and the regular simplex target, alongside angle variability and class mean norms. For this additional analysis, we enable class-complement for all three methods, including SCL, to hold prototype use fixed.

MethodCIFAR-100, IR=1: NC2CIFAR-100-LT, IR=10: NC2CIFAR-100-LT, IR=50: NC2CIFAR-100-LT, IR=100: NC2ImageNet-LT: NC2ImageNet-LT: Angle variabilityImageNet-LT: Class mean norm
SCL1.0241.0561.0751.0791.2680.1610.784
BCL1.0211.0391.0491.0491.2640.1550.826
FBCL1.0191.0371.0371.0361.2590.1510.830

FBCL achieves larger NC2 reductions than BCL, particularly under higher imbalance ratios FBCL achieves lower mean NC2 than both SCL and BCL across all settings. On CIFAR-100, BCL reduces NC2 relative to SCL by 0.29%, 1.61%, 2.42%, and 2.78% at imbalance ratios of 1, 10, 50, and 100, respectively, whereas FBCL achieves larger reductions of 0.49%, 1.80%, 3.53%, and 3.99%. The additional reduction achieved by FBCL over BCL is larger at higher imbalance ratios on CIFAR-100. On ImageNet-LT, BCL reduces NC2 by 0.32% relative to SCL, while FBCL reduces it by 0.71%. Also, the reduced angle variability further supports more uniform pairwise angles between class centers.

Although NC2 is zero for a regular simplex, the SCL baseline exhibits an NC2 value of 1.024 even on balanced CIFAR-100 (IR=1), indicating that deviations from the ideal geometry are also observed under balanced distributions in practical training. Nevertheless, FBCL consistently achieves lower NC2 than both SCL and BCL across datasets and imbalance ratios. Lower mean inter-class cosine similarity in Appendix D.6 indicates greater average separation between class centers, whereas lower angle variability provides evidence of greater equiangularity. NC2 further measures deviation from the regular simplex Gram matrix, capturing both relative class mean norms and pairwise geometry. Together, these results indicate that FBCL encourages improved representation geometry in practical training.

[1] Papyan, et al. "Prevalence of neural collapse during the terminal phase of deep learning training." PNAS, 2020.

Q1 (W1). Anchor-class coverage, effective class weighting, and their relationship to geometry with controlled batch experiments.

Higher anchor-class coverage and more uniform effective class weighting are associated with improved geometry We would first like to clarify that all methods use the same original sampler. To better understand the practical training, we quantify anchor-class coverage and the effective class weighting of FBCL in Eq. (150) (Eq. (155) in the revision). Anchor-coverage is the average fraction of classes represented by anchor samples, excluding class-complement prototypes. Effective class weighting refers to the expected outer coefficient assigned to each class's mean anchor loss. We summarize it by the ratio of the largest to the smallest mean effective weight across the many-, medium-, and few-shot groups; uniform weighting gives a reference value of 1.00.

For the controlled experiment, we construct a 100-class subset of ImageNet-LT by sampling classes while approximately preserving the proportions of many-, medium-, and few-shot classes in the original 1,000-class dataset. On this subset, we compare original and controlled sampling with a batch size of 256. Original sampling follows the original sampler, whereas controlled sampling selects one sample from each of the 100 classes and fills the remaining 156 samples using the original sampler.

We then compare the original 1,000-class setting and these two 100-class settings to examine how anchor-class coverage and effective class weighting relate to the observed geometry. The table combines the sampling statistics with NC2.

SettingAnchor-class coverage (%)Effective class weighting (FBCL)NC2: SCLNC2: BCLNC2: FBCL
1,000 classes, original sampling19.8014.021.2681.2641.259
100 classes, original sampling71.244.291.0831.0561.048
100 classes, controlled sampling100.001.000.9590.9420.935

FBCL achieves larger NC2 reductions than BCL under both non-theoretic and theoretic cases From the first to the third row, anchor-class coverage increases from 19.80% to 71.24% to 100.00%, while effective class weighting becomes more uniform, with the reported ratio decreasing from 14.02 to 4.29 to 1.00. Under controlled sampling, every class receives exactly the same outer weight in FBCL. Alongside these changes, NC2 decreases for all three methods: FBCL's mean NC2 decreases from 1.259 to 1.048 to 0.935 and remains the lowest in every setting. Thus, sampling conditions closer to the theorem's full anchor-class coverage and uniform effective class weighting are accompanied by improved geometry.

Relative to SCL, BCL and FBCL reduce NC2 by 2.49% and 3.23%, respectively, under original sampling, and by 1.77% and 2.50% under controlled sampling. These results connect batch composition to representation geometry while showing that FBCL improves geometry more effectively than the baselines in both settings, regardless of whether the theoretical full class condition is satisfied.

Q3 (W2). Ablation details, transfer uncertainty, and the standard deviation clarification

We agree with the reviewer and will clarify the use of class-complement in the original ablations in the revision, as detailed below.

Ablation configurationClass-complementImageNet-LT Acc.
PB (SCL)Not used55.78
PB + SBNot used55.92
PB + NB (BCL)Used56.03
NB + SBUsed55.35
PB + NB + SB (FBCL)Used56.64

The reason we omit class-complement for SCL is that we follow the ablation in BCL, where adding class-complement to SCL without negative class averaging did not improve accuracy, as discussed in Appendix D.2. We also apply the same choice to PB + SB. Class-complement use is therefore held fixed within both comparisons examining SB: PB versus PB + SB, and BCL versus FBCL. Comparisons that introduce NB also change class-complement use, which we will make explicit in the ablation description. Normalizations are held fixed across all PB/NB/SB ablations. As described in Section 5.2, contrastive features are L2-normalized. The reported intra-class variances are normalized by total representation variance, as specified in Appendix D.4.

Transfer uncertainty. The standard deviations over three runs for the Mean column of Table 3 are reported below.

MethodLinear evaluationFine-tuning
SCL68.60 ± 0.1383.50 ± 0.38
BCL68.70 ± 0.0883.80 ± 0.21
FBCL69.10 ± 0.0784.50 ± 0.19

SimCLR discrepancy. We thank the reviewer for pointing out this inconsistency. The correct standard deviation of SimCLR on ImageNet-LT is 0.20, and the entry in Table 1 should therefore be 55.20 ± 0.20, consistent with Table 6 (Table 8 in the revision). We will correct this error in Table 1 in the revision.

Title: Follow-up on Rebuttal Experiments and Revised Manuscript

Thank you for the clear responses and the carefully designed additional experiments.

Your updates address several of my questions well:

  1. Direct Simplex Diagnostics & Clarifications (Q2, Q3): Adding NC2, angle variability, and class-mean norms with class-complement held fixed (now Table 19 in Appendix D.6) provides a much cleaner test of equiangularity than mean inter-class cosine similarity alone. Clarifying class-complement usage across the PB/NB/SB ablations in Appendix D.2, adding the three-run transfer standard deviations in Table 9, and fixing the SimCLR entry in Table 1 also resolve my procedural questions.
  2. Anchor Coverage & Effective Weighting Analysis (Q1): The 100-class ImageNet-LT controlled-sampling experiment directly quantifies the bridge between your full-class theorem and stochastic mini-batch training. Showing that increasing anchor-class coverage (19.80%→71.24%→100.00%19.80\% \rightarrow 71.24\% \rightarrow 100.00\%) and equalizing effective outer weights (14.02→4.29→1.0014.02 \rightarrow 4.29 \rightarrow 1.00) steadily reduces NC2 (1.259→1.048→0.9351.259 \rightarrow 1.048 \rightarrow 0.935) makes the role of batch composition concrete, while confirming that 1,0001{,}000-class training operates in a regime with substantial residual anchor-frequency skew.

Could you follow up on the brief point for completeness?

  1. Classification Accuracy in the Controlled 100-Class Experiment: In the Q1 table, you report NC2 for SCL, BCL, and FBCL on the 100-class subset under both original sampling (71.24% coverage) and controlled sampling (100.00% coverage). Could you also share the corresponding Top-1 classification accuracies (overall and many/medium/few if available) for those two 100-class runs? Seeing how classification accuracy moves alongside the NC2 drop from 1.0481.048 to 0.9350.935—and how the BCL-to-FBCL accuracy gap behaves at 100%100\% anchor coverage—would help clarify how strongly improved simplex proximity translates into downstream accuracy.

Comments on the paper

1 comment

Dear Reviewers,

The authors have provided their rebuttals to the reviewers' comments. Please take a moment to review the responses and continue the discussion on any remaining concerns, unresolved issues, or points that may require further clarification.

Your further input and discussion are highly appreciated.

Regards,
AC for this paper

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.