- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the representation geometry of supervised contrastive learning under long-tailed data distributions. The authors develop a theoretical framework and show that the regular simplex is not a global minimizer of SCL or BCL under class imbalance, due to the remaining dependence on the number of samples per class.
To address this issue, the paper introduces structural balancing and proposes Fully Balanced Contrastive Learning (FBCL). The proposed method balances the aggregation across anchors and recovers the regular simplex as a global optimum. The paper further analyzes the effect of structural balancing on intra-class variance and shows that it makes the variance penalty approximately independent of the number of samples per class.
Experiments on ImageNet-LT and iNaturalist show improvements in long-tailed recognition, while experiments on multiple downstream datasets evaluate the transferability of the learned representations. The results also show that structural balancing can be incorporated into other long-tailed recognition methods and consistently improves their performance.
Critical strengths
1: The paper has a clear and coherent presentation, and I think the motivation and the main line of the paper are easy to follow. Through the theoretical framework developed in the paper, the authors identify the remaining dependence on the number of samples per class in SCL and BCL, which naturally motivates structural balancing and FBCL.
2: The idea is simple and easy to understand, while providing a novel and useful way to improve performance under class imbalance. By balancing the aggregation across classes, structural balancing removes the dependence on the number of samples per class in the lower bound and allows FBCL to recover the regular simplex as a global optimum. Empirically, FBCL improves over SCL and BCL on long-tailed recognition (Tables 1 and 2) and transfer learning (Table 3).
Critical weaknesses
As acknowledged by the authors, there is a gap between the theoretical setting and practical mini-batch training:
1: The main theoretical analysis assumes that each mini-batch contains all classes, while in practice a mini-batch generally contains only a subset of classes.
2: The dimensional condition required for the regular simplex result is not satisfied on iNaturalist.
Although FBCL still shows improvements in these practical settings, it remains unclear to what extent the regular simplex result can explain the empirical improvements observed in practice.
Questions for authors
1: As mentioned in the weakness, there is a gap between the theoretical and practical settings. Could the authors further clarify to what extent the theoretical regular simplex result explains the empirical improvements of FBCL in practical settings?
2: I read Appendix D.7, but I would still like the authors to further clarify the difference between structural balancing and conventional re-weighting, especially since they are equivalent in the full-batch setting. This distinction is important to the contribution of SB and would be helpful to clarify in the main paper.
2 comments
Q2. Structural balancing(SB) and conventional re-weighting
Structural and conventional re-weighting behave differently in practical mini-batch training
We agree that SB and conventional re-weighting coincide in the full-batch setting when applied to the same loss, and therefore share the same geometric guarantee in that setting. However, they behave differently in practical mini-batch training. Specifically, they differ in the level at which class contributions are balanced. Conventional re-weighting balances class contributions in expectation across batches using dataset-level class frequencies, whereas SB balances class contributions within each batch by assigning equal outer weight to the mean anchor loss of each class present.
To examine this difference, we analyze the outer weights assigned to the same BCL objective. The total outer weight under conventional re-weighting ranges from approximately 0.6 to 1.6 times its mean across batches, whereas SB maintains a total weight of 1 in every batch. In the corresponding classification experiments, SB achieves 52.14 ± 0.15% accuracy, compared with 51.45 ± 0.45% for conventional re-weighting. The weight analysis is conducted on CIFAR-100-LT (IR=100) over three seeds. We will add this comparison in the revision.
We appreciate the reviewer’s comments, which highlights the importance of connecting our theoretical results to practical improvements and clarifying the distinction between structural balancing and conventional re-weighting. We hope that the additional analyses and explanations help address these concerns.
Q1. Connecting the theoretical geometry with empirical improvements
Theoretical relevance: To the extent that class coverage increases, both geometry and accuracy improve, and FBCL remains effective even without full coverage
To examine how the theoretical full class setting relates to practice and empirical performances, we conducted a controlled experiment that increases the class coverage of batches while keeping the class subset and batch size fixed. We evaluate both classification accuracy and an NC2 metric [1], which measures the deviation of the class-center Gram matrix from the regular simplex Gram matrix.
We construct a 100 class subset of ImageNet-LT that approximately preserves the proportions of many-, medium-, and few-shot classes in the original dataset. We compare original sampling with controlled sampling, both with a batch size of 256. The controlled batch contains one image from every class, with the remaining 156 images drawn using the original sampling scheme. This increases average class coverage from 71.24% to 100.00%, satisfying the full class coverage condition of our analysis.
Under controlled sampling on the same 100 class subset, both NC2 and classification accuracy improve. The Pearson correlation between NC2 and classification accuracy across the six results (three methods and two sampling conditions) is -0.964, a negative correlation. This indicates that improved geometry is related to higher classification accuracy in these experiments.
Within each sampling condition, FBCL achieves larger improvements over SCL than BCL. Under original sampling, BCL reduces NC2 by 2.49% and improves accuracy by 0.28% relative to SCL, whereas FBCL achieves a 3.23% NC2 reduction and a 0.61% accuracy improvement. Under controlled sampling, the corresponding improvements are 1.77% and 0.61% for BCL, compared with 2.50% and 0.83% for FBCL. These results connect FBCL's larger geometric improvements with its larger gains in classification accuracy.
Regarding the dimensional assumption, understanding the geometry reached in practical training when this condition is not satisfied remains an important research direction under both balanced and imbalanced distributions, as discussed in Section 6. To explore this behavior in a simple setting, we extend the toy experiment in Figure 1 to four and five classes with two dimensional embedding space. Both settings violate the dimensional requirement for a regular simplex. The following table reports approximate angular gaps, in degrees, between neighboring class centers.
FBCL produces approximately uniform angular spacing in both cases, while BCL produces uneven spacing. These observations suggest that FBCL can encourage class centers to spread more uniformly within the available representation space even when the dimensional assumption is not satisfied.
We hypothesize that a similar tendency may contribute to the improvements on iNaturalist and may also arise on other datasets where this condition is violated. Moreover, Proposition 3 does not rely on the dimensional condition. That is, the class-level lower bound of FBCL remains independent of class cardinalities regardless of the representation dimension.
[1] Papyan, et al. "Prevalence of neural collapse during the terminal phase of deep learning training." PNAS, 2020.