acceptodds
Under review as a conference paper at ICLR 2027

OutnumberedTales: Prevalence-Aware Evaluation of Contrastive Learning under Extreme Binary Imbalance

Abstract

Extreme class imbalance is a defining challenge across countless real-world domains where detecting rare events is critical. Despite this reality, the standard playbook for evaluating contrastive learning models relies heavily on artificially balanced test sets and symmetric accuracy metrics. Evaluating models in this artificial vacuum fails to reflect actual deployment conditions and obscures whether specialized, minority-aware algorithms provide any genuine benefit. Capturing a model's true efficacy demands a multifaceted approach. By examining performance across a broad spectrum of methods and decision-aware metrics, we demonstrate that different analytical lenses tell fundamentally different stories about model performance, exposing critical realities that conventional balanced testing actively conceals. Revisiting the problem through these varied lenses, we identify two distinct failure modes in existing minority-aware objectives. Geometrically, prototype-based contrastive losses degrade when fixed class prototypes conflict with the learned feature geometry. We introduce OrthoProto, an orthogonality hinge that resolves this conflict and surpasses the strongest published baseline wherever the failure is present, with gains scaling with the severity of the degradation. Statistically, batch-local objectives lose their supervisory signal when the expected number of minority samples per batch falls below one. We introduce CompMin, which maintains a census of minority representations and stays effectively invariant to batch size in this starved regime, a robustness we predict in advance from prevalence and batch size alone and then confirm. To ensure future research captures this multifaceted analysis, we release OutnumberedTales: a reproduction framework that implements published imbalance-aware objectives behind a common interface, pairs them with a decision-aware evaluation suite, and is designed for extension to new losses, datasets, and metrics.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
extreme class imbalance, supervised contrastive learning, prevalence-aware evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pyc6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies supervised contrastive learning under extreme binary class imbalance and identifies two failure modes in existing minority-aware methods. First, SupProto's fixed prototypes can conflict with the representation geometry learned by the encoder. To address this, the authors propose OrthoProto, which adds an orthogonality hinge to SupProto. Second, SupMin's minority supervision remains confined to the batch, and its class-level term remains idle at most steps when the expected number of minority views per batch falls below one. To address this, the authors propose CompMin, which maintains a memory bank containing one slot per minority training example. The paper also introduces OutnumberedTales, a reproduction framework that implements published imbalance-aware objectives under a shared protocol and includes a decision-aware evaluation suite. Experiments on binary iNaturalist and MedMNIST tasks show that OrthoProto reduces geometric violations, while CompMin improves robustness in the small-batch regime.

Critical strengths

The paper provides a clear analysis of two failure modes (the geometric conflict in SupProto and the batch-local supervision limitation in SupMin) in existing minority-aware contrastive learning methods and proposes targeted solutions (OrthoProto and CompMin). The experiments are conducted under a shared protocol across multiple binary tasks (iNaturalist and MedMNIST) and minority prevalences (ranging from 0.5% to approximately 27%). The evaluation using both ranking and decision metrics offers useful insights into how different metrics can lead to different conclusions about model performance. The batch-size experiment provides convincing evidence for CompMin's robustness in the starved regime.

Critical weaknesses

I think the main weaknesses are the limited methodological novelty and effectiveness.

1: OrthoProto adds an orthogonality hinge to SupProto, but this is a relatively simple modification and does not consistently outperform SupMin (Tables 3 4).

2: CompMin improves robustness in the small-batch regime (batch size 32), but its use of a memory bank builds on established ideas like (MoCo, XBM, and GSupCon), and it provides limited improvement over SupMin at the standard batch size of 256 (Tables 3 4).

3: OutnumberedTales integrates existing imbalance-aware objectives under a shared protocol and pairs them with a decision-aware evaluation suite, but does not introduce fundamentally new methods or evaluation metrics.

Questions for authors

1: Given the concerns raised in the weaknesses, could the authors more clearly explain the main contributions and novel aspects of OrthoProto, CompMin, and OutnumberedTales compared with existing work?

2: In Table 4, OrthoProto still underperforms SupMin in several extreme imbalance settings. Why not directly use or improve SupMin instead?

3: CompMin is partly motivated by memory-intensive encoders that require small batch sizes, but all experiments use ResNet-50. Why did the authors not evaluate Vision Transformers (ViTs)? Could the authors also discuss whether CompMin's batch-size robustness would generalize to ViT-based encoders and more diverse datasets?

4: Tables 1 and 2 combine the evaluation of the proposed methods with the analysis of ranking and decision metrics. I find the results somewhat difficult to follow. Could the authors present these two analyses more clearly, possibly in separate tables?

2 comments
AuthorsOpenReview ↗

W1 to W3, Q1 and Q2: what is new, and why two repairs instead of one

We thank the reviewer for the feedback. Over the past week we ran the following experiments: two batch controls, an ImageNet-pretrained ViT-B/16 and ResNet-50 panel, and a ViT-Tiny/16 trained from scratch under the paper's protocol. This comment covers W1 to W3, Q1 and Q2, the next one Q3 and Q4. Numbers are AUROC / ER-AUROC / BAcc, ×100.

W1 to W3, Q1. We agree on what is borrowed. Neither loss is a new family, CompMin sits in the MoCo, XBM and GSupCon line, and no metric is ours. We disagree that nothing new follows.

  • First, the problem. Extreme binary imbalance had one systematic study (Mildenberger et al.), scored on iNaturalist by balanced accuracy on a balanced test set. To the best of our knowledge, we are the first to evaluate this specific problem under a protocol that genuinely reflects real deployment conditions. Our framework incorporates prevalence-invariant ranking metrics, evaluates the identical encoder across multiple prevalence levels, utilizes a bounded decision-analytic score, and introduces two distinct probes with varying label budgets. This comprehensive approach yields a fundamental result that no previous evaluation could capture. Specifically, it demonstrates that the apparent ranking of methods is inherently dependent on the specific evaluation lens being applied, a dynamic we detail further in our response to Q4 next.
  • Second, the diagnoses. The expected minority count per batch, pB=2Bπp_B = 2B\pi, predicts before training where SupMin's supervision thins (confirmed on two backbones, Q3). SupProto's failure is identified and measured directly: at the extreme rungs its backbone does not linearly separate the classes (cross-validated AUROC 0.77 to 0.94, against 0.99 to 1.00 for the other three methods), and the hinge restores separability.
  • Third, the comparison. Under this protocol five recent long-tailed contrastive methods land 3 to 10 AUROC points below SupMin at 99:1 (PaCo 12 to 22), and three of them reproduce plain SupCon's collapse at K=2. Extreme binary imbalance is not a small instance of long-tailed learning.

Q2. The revision does not present OrthoProto as a method that beats SupMin everywhere. No method does, and showing that is the point of the protocol. Under the label-matched probe (batch 256, mean over three seeds):

plantsinsectsmammals
SupMin 95:594.7 / 85.9 / 79.590.7 / 80.7 / 71.083.0 / 70.3 / 61.9
OrthoProto 95:595.3 / 88.3 / 81.691.5 / 81.4 / 72.784.6 / 71.2 / 66.5
SupMin 99:187.8 / 74.5 / 61.580.2 / 66.9 / 55.271.9 / 58.2 / 52.2
OrthoProto 99:185.3 / 72.9 / 63.682.7 / 68.7 / 58.269.4 / 58.4 / 52.2
SupMin 99.5:0.586.0 / 71.7 / 61.176.6 / 62.2 / 55.471.2 / 57.6 / 51.5
OrthoProto 99.5:0.581.7 / 68.7 / 59.978.3 / 62.8 / 53.866.3 / 56.1 / 51.8

In a direct comparison, OrthoProto achieves the highest AUROC in 5 of the 9 cells. This includes the insects category at every prevalence level and all three cells evaluated at the 95:5 ratio. SupMin leads in the remaining 4 cells covering plants and mammals at the 99:1 and 0.5 percent levels by a margin of 2.4 to 4.9 points. However, evaluating the exact same configuration using balanced accuracy reveals a different outcome where neither method wins. Among the four methods evaluated in our study, SupProto performs best across all nine cells and outperforms the others by 4 to 13 points at the extreme rungs. Across the 36 combinations of cells and metrics, OrthoProto leads in 16 columns, SupProto in 10, CompMin in 9, and SupMin in only 1. If we evaluated performance based on balanced accuracy alone, SupProto would appear universally superior and these nuanced dynamics would remain hidden. The revised manuscript explicitly acknowledges that no single method dominates across all conditions. The actual claim for OrthoProto is more specific and remains fully valid. It successfully improves upon SupProto in 22 to 25 of the 27 cell and seed pairs across every metric under the full-pool probe with a Wilcoxon p-value below 0.001.

On "why not directly use or improve SupMin instead”. We did actually improve SupMin by developing CompMin. SupMin suffers from statistical starvation when the expected minority count per batch drops below one rather than a geometric failure. CompMin solves this using a dedicated minority bank and significantly outperforms SupMin by an average of 2.4 AUROC and 4.8 ER-AUROC points in starved cells at a batch size of 32. Each repair specifically targets the unique failure of its base method. The orthogonality hinge requires a prototype pull to guide repelled minority samples, which SupMin lacks. Finally, when label budgets are strictly matched on the medical panel, OrthoProto clearly outperforms SupMin at extreme imbalances. On PneumoniaMNIST, OrthoProto achieves a 96.0 AUROC versus 92.9 for SupMin at the 1 percent level and 95.0 versus 90.1 at 0.5 percent.

AuthorsOpenReview ↗

Q3 and Q4: the batch ladder, a ViT-Tiny from scratch, and Tables 1 and 2 split by lens

Q3

Why did the submission not evaluate ViTs? Our core claim concerns the batch size rather than the specific backbone architecture. We updated our motivating text to clarify that limited hardware and high-resolution inputs frequently force the use of small batches in practice. The loss function is completely agnostic to the architecture. It strictly depends on the expected number of minority views per batch, which is defined as pB=2Bπp_B = 2B\pi. SupMin fails because its class term goes idle whenever this expected count falls below one. We demonstrated this by scaling a ResNet-50 from a batch size of 256 down to 32 across six extreme cells using three seeds and the full-pool probe. The paired AUROC changes and the performance gap between CompMin and SupMin at batch 32 are presented below (95% intervals):

cellp_B at bs32Δ SupMinΔ CompMinCompMin − SupMin, AUROCCompMin − SupMin, ER-AUROC
plants 99:10.64−1.6+0.7+1.5 [+0.5, +2.6]+5.8 [+4.0, +7.7]
insects 99:10.64−3.8+0.1+3.7 [+0.0, +7.3]+6.0 [−1.3, +13.4]
mammals 99:10.64−1.7−0.6+0.2 [−0.8, +1.2]+1.0 [+0.4, +1.6]
plants 99.5:0.50.32−2.4−0.6+2.0 [+0.8, +3.2]+5.2 [+2.1, +8.4]
insects 99.5:0.50.32−5.1+0.4+5.0 [+2.8, +7.3]+6.9 [+4.2, +9.6]
mammals 99.5:0.50.32−4.2−1.5+2.3 [+0.8, +3.7]+3.9 [+0.6, +7.3]

Across the six cells, CompMin leads SupMin by an average of 2.4 AUROC and 4.8 ER-AUROC at batch 32 while staying within a point of it at 256. That also highlights CompMin's practical case, where a batch of 32 on a less capable GPU stays within 1.5 AUROC points of a batch of 256 for the cost of one 128-d bank vector per minority training image.

Does CompMin's batch-size robustness generalise to ViT encoders? On a ViT trained from scratch the effect points the same way as on the ResNet-50, but it is smaller. We tested insects 99:1 in two settings.

With ImageNet-pretrained backbones (ViT-B/16 and ResNet-50), every method, plain SupCon included, reaches 96 to 98 AUROC at both batch sizes. Pretraining saturates the task, so it cannot show a difference between objectives, and every claim in the paper is scoped to training from scratch.

With a ViT-Tiny/16 trained from scratch (paper protocol, AdamW, three seeds, full-pool probe):

methodbatch 256batch 32change
SupMin84.2 / 71.1 / 76.882.6 / 69.2 / 75.2−1.6 / −1.9 / −1.6
CompMin83.9 / 70.3 / 75.683.1 / 71.0 / 75.3−0.8 / +0.7 / −0.3

Shrinking the batch from 256 to 32 costs SupMin 1.6 AUROC and CompMin 0.8. On the ResNet-50 the same change cost SupMin 3.8 and left CompMin unchanged (+0.1). At batch 32 CompMin is ahead of SupMin by 0.5 AUROC and 1.8 ER-AUROC on the three-seed mean and level on balanced accuracy. On one of the three seeds SupMin has the higher AUROC.

Q4. We agree with this suggestion and have updated the manuscript accordingly. The revision now splits Tables 1 and 2 by evaluation lens rather than by dataset. Table 1 is located in the ranking metrics subsection and exclusively reports the AUROC and ER-AUROC for all methods across the seven medical rungs. Table 2 is placed under the decision metrics subsection and focuses solely on the S_DCA at the clinical operating point alongside the S_DCA^cal after isotonic recalibration. The best trivial policy scores 70.4 on BreastMNIST and 50.8 on PneumoniaMNIST. The gap between the raw and recalibrated scores directly reflects the cost of miscalibration. A single column from each table for BreastMNIST at 20 percent illustrates this reasoning clearly:

methodAUROCER-AUROCS_DCAS_DCA^cal
Logit adj.67.154.369.978.6
BCL63.652.972.877.6
SupCon61.852.870.577.4
KCL59.252.170.576.3
PaCo61.553.570.777.5
SupMin59.151.470.775.8
SupProto67.356.570.878.4
CompMin64.554.473.277.2
OrthoProto77.061.972.781.6
trivial policy70.4

When evaluating through the ranking lens, SupProto outperforms BCL by four AUROC points. However, looking at these same methods through the decision lens reveals a completely different outcome. SupProto barely exceeds the trivial policy with a score of 70.8 compared to 70.4, whereas BCL clearly surpasses it with a 72.8. The ranking metrics completely mask this practical difference. Furthermore, five of the seven contrastive methods remain within half a point of the trivial policy. Among the two-stage contrastive methods, only CompMin and OrthoProto distinctly exceed this baseline, and every method loses between 4 and 9 points due to miscalibration. This fundamental disagreement between evaluation lenses serves as the central finding of our paper.

We would be grateful if the reviewer reconsidered the score in the light of these experiments.

Reviewer_32JGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper diagnoses that the two existing solutions for supervised contrastive learning under extreme binary imbalance, SupMin and SupProto, suffer from a statistical failure and a geometric failure, respectively. To address these failures, it proposes CompMin, which uses a memory bank to decouple the minority positive set from the batch, and OrthoProto, which adds an orthogonality hinge with respect to the learned majority direction. Furthermore, through an evaluation protocol that jointly employs ranking, low-prevalence, and cost-based decision metrics, along with OutnumberedTales, a reproducible framework, it shows that conclusions about which model is superior change depending on the evaluation perspective.

Critical strengths
  • The paper diagnoses the causes of failure as measurable quantities, and the fixes grounded in this diagnosis lead to large and consistent performance gains in the most challenging regime.

  • The paper demonstrates, through concrete examples, the claim that the evaluation perspective changes the conclusions, offering meaningful implications for practitioners.

  • The experimental design and reporting are transparent, and even unfavorable results are presented rather than concealed.

Critical weaknesses

W1. The mechanistic explanation based on the violation rate is inconsistent with the results. The paper claims that the violation rate explains the performance loss of SupProto, and that this loss in turn explains the magnitude of OrthoProto's recovery. However, even among cells with the same 100% violation rate, the loss varies from −1.1 to −5.6, so this relationship is evident only for insects. Moreover, even where recovery occurs, OrthoProto underperforms SupMin in the extreme mammals cell, which contradicts the claim that it surpasses the strongest baseline wherever the failure occurs. In addition, the violation rate is measured in the projection space, which the probe does not read, and its correlation with test AUROC is weak (iNaturalist R2\mathbb{R}^2 = 0.049), so the causal chain remains unverified.

W2. The main performance claims rely on an evaluation setting that is far from deployment conditions, and the scope and generality of the contributions are narrow.

The main iNaturalist results (Table 3) are based on a full-pool probe that uses minority labels excluded from pretraining. When the label budget is matched, OrthoProto's advantage disappears except for insects.

The experiments consist of only five small-scale binary tasks, and the iNaturalist tasks also appear to be very small relative to the full dataset. There is no discussion of whether the proposed diagnostics (pBp_{B}, violation rate) extend to multi-class long-tailed settings, nor any comparison with anomaly detection or one-class classification, which are natural competitors in a regime where the minority fraction drops to 0.5%. Therefore, the evidence is limited for these findings to serve as general knowledge in the imbalanced learning literature.

Questions for authors

Please refer to Critical Weaknesses

3 comments
AuthorsOpenReview ↗

W1: violation rate removed, methods re-evaluated as paired ablations over all 27 (cell, seed) pairs

We thank the reviewer for the careful reading. We agree with the central criticism of the violation rate and remove it in the revision. Over the past week we replaced it with a direct separability test and re-evaluated both proposed methods as paired ablations against their references.

W1. The violation-rate mechanism. We believe the points raised share one cause. The violation rate is a binary diagnostic of one failure, the prototype gate releasing the minority on the majority's side of the sphere, and the paper treated it as a continuous predictor of performance. Once the failure is present it saturates at 100%, so it cannot order the damage across cells (the reviewer's −1.1 to −5.6 example), it cannot explain where the repaired method overtakes SupMin, it was measured in the projection space the probe never reads, and it is not expected to predict AUROC for methods that never show the failure, which is what the iNaturalist R² of 0.049 reflects. The revision removes it together with Table 8, the right-hand column of Table 9 and its row in the diagnostics table.

Two measurements replace it:

  • First, a direct test of whether the two classes are separable in the features the probe reads. For every iNaturalist run we took the frozen backbone features of the training images and fitted a linear classifier with five-fold cross-validation. For SupMin, CompMin and OrthoProto the cross-validated AUROC is 0.99 to 1.00 in every cell. For SupProto it is 0.97 to 0.99 at 95:5 and 0.77 to 0.94 in the six extreme cells. Of the four methods, SupProto is the only one whose backbone fails to separate the classes, and that is the failure the hinge removes.
  • Second, the representation metrics of Mildenberger et al., which we keep only as a collapse check. On iNaturalist at 99:1, the five long-tailed methods we added for Reviewer QUkA show the collapse signature reported for plain SupCon: PaCo, ProCo and RBL fully (SAA 0 to 3%, GPU above 0.7), KCL and TSC partially, and the four binary-specific methods not at all (SAA 76 to 97%, GPU below 0.06). These metrics live in the projection space and cannot rank encoders that did not collapse; the revision states both limits.

How the methods are now evaluated. The abstract no longer claims that OrthoProto surpasses the strongest baseline wherever the failure is present. The reviewer's mammals example is the counterexample: under the label-matched probe OrthoProto is behind SupMin by 2.4 to 4.9 AUROC on plants and mammals at the extreme rungs and ahead only on insects. We now also report OrthoProto and CompMin as paired ablations against their references over all 27 (cell, seed) pairs on iNaturalist (three taxa, three prevalences, three seeds, batch 256), with a two-sided Wilcoxon signed-rank test; per-cell results stay in the appendix. Entries are wins out of 27 and the median paired difference:

pairprobeAUROCER-AUROCAUPRCBAcc
OrthoProto − SupProtofull-pool25/27, +1.2 (p < 0.001)24/27, +2.5 (p < 0.001)22/27, +4.5 (p < 0.001)25/27, +1.4 (p < 0.001)
OrthoProto − SupProtolabel-matched15/27, +0.8 (p = 0.39)20/27, +1.8 (p < 0.001)22/27, +4.8 (p < 0.001)1/27, −10.0 (p < 0.001)
CompMin − SupMinfull-pool6/27, −0.3 (p = 0.007)9/27, −0.6 (p = 0.24)13/27, 0.0 (p = 0.95)10/27, −0.2 (p = 0.16)
CompMin − SupMinlabel-matched16/27, +0.3 (p = 0.14)13/27, 0.0 (p = 0.43)13/27, −0.1 (p = 0.90)20/27, +1.5 (p = 0.049)

Under the full-pool probe the hinge improves SupProto in 22 to 25 of 27 pairs on every metric. Under the label-matched probe it improves ER-AUROC and AUPRC but not AUROC (15 of 27, p = 0.39), and on balanced accuracy SupProto is better in 26 of 27. CompMin and SupMin are within 0.6 points on every ranking metric at batch 256; the two significant differences are small and opposite in sign (−0.3 AUROC full-pool, +1.5 balanced accuracy label-matched). CompMin's claim is the small-batch one: at batch 32 it leads SupMin by 2.4 AUROC and 4.8 ER-AUROC on average over the six extreme cells. The revised text gives both results equal weight.

This is also the paper's framing point. Under the label-matched probe SupProto has, among the paper's four methods, the best balanced accuracy in all nine cells, yet the worst AUPRC in all six extreme cells and the worst ER-AUROC in five. OrthoProto leads 16 of the 36 (cell, metric) columns, SupProto 10, CompMin 9 and SupMin 1, while SupMin stays within two points of the leader in 20.

Which method is best is a property of the lens, and that observation, not a single winner, is what the evaluation framework is for.

AuthorsOpenReview ↗

W2: the two probes and what each measures, scope of the tasks, and a one-class anomaly baseline

W2a. The two probes. We believe this concern applies to Table 3 only. On the medical panel the probe is label-matched throughout: it is fitted on a balanced subset of the thinned pretraining split, so it uses no label beyond the pretraining budget. There, among the nine methods of the submission, OrthoProto has the best AUROC and ER-AUROC in five of the seven medical cells in the main text (Tables 1 and 2), including the two most extreme. At PneumoniaMNIST 1% (35 minority training images) it reaches 96.0 AUROC and 91.1 ER-AUROC against 94.1 and 82.9 for the best baseline, and at 0.5% (17 images) 95.0 and 88.8 against 91.6 and 80.4. The decision score agrees: 90.2 against 83.4 at 1% and 87.5 against 80.3 at 0.5%.

On iNaturalist the full pool does contain 4.7k to 7.3k minority labels per taxon that pretraining never saw. It is the published protocol of Mildenberger et al., so Table 3 is where a new method is compared with prior work. On it OrthoProto improves on SupProto in 25 of 27 (cell, seed) pairs on AUROC, 24 on ER-AUROC, 22 on AUPRC and 25 on balanced accuracy (p < 0.001). We added the label-matched probe (Table 4) because it is the stricter test, and every method loses 1 to 15 AUROC points on it. Under it OrthoProto still leads the most (cell, metric) columns, 16 of 36 against 10 for SupProto, 9 for CompMin and 1 for SupMin, and its gain over SupProto stays significant on ER-AUROC (20 of 27 pairs) and AUPRC (22 of 27), both p < 0.001. Where the failure is severe the repair is intact: on insects at 99:1 and 0.5% OrthoProto keeps +11.0 and +8.8 AUROC over SupProto. What the stricter probe removes is the lead over SupMin on plants and mammals at the extreme rungs (2.4 to 4.9 AUROC points) and the pooled AUROC and balanced accuracy gains over SupProto.

W2b. Task size and multi-class. The focus on binary classification is actually the core argument of our paper rather than a missing piece. Binary imbalance is not a small instance of the long-tailed problem: the simplex geometry of supervised contrastive learning degenerates to one antipodal pair at K=2 (Graf et al., 2021), and the long-tailed designs are built and analysed for K ≥ 3. The experiments added for Reviewer QUkA test this directly: five long-tailed methods, run under our protocol at 99:1, all stay below SupMin, CompMin and OrthoProto on all three iNaturalist tasks under the full-pool probe, and three of them reproduce plain SupCon's collapse. Machinery built for many tail classes does not transfer to two. Of the diagnostics, the starvation rule does extend, with 2Bπ_k expected views of class k per batch, and the violation rate is removed (see W1). On size, the tasks follow the construction of Mildenberger et al. across 19 (task, prevalence) cells, and minority counts as low as 17 are what a rare-event regime means. A larger cell would strengthen the study, and the revised limitations say so.

W2c. Anomaly detection. The request is fair with 47 to 72 positives at 0.5%, and we ran it. We score each test image by its distance to the majority training set on frozen ImageNet features (kNN with k = 5 and a Ledoit-Wolf Mahalanobis distance), using no minority training label. The majority sets at 1% and 0.5% differ by under 1%, so we report it once. AUROC / ER-AUROC / BAcc, with our full-pool results at 99:1 for comparison:

scorerplantsinsectsmammals
kNN to majority, ResNet-5070.1 / 58.2 / 62.254.8 / 49.6 / 56.080.8 / 64.8 / 72.6
kNN to majority, ViT-B/1671.0 / 55.1 / 64.846.2 / 48.6 / 53.575.3 / 55.1 / 68.7
Mahalanobis to majority, ResNet-5062.2 / 57.4 / 55.853.1 / 49.1 / 53.059.9 / 49.3 / 62.6
Mahalanobis to majority, ViT-B/1662.8 / 56.7 / 57.446.6 / 48.9 / 53.465.0 / 49.5 / 64.3
SupMin (from scratch)94.0 / 81.8 / 86.886.6 / 72.2 / 78.483.4 / 69.1 / 75.6
CompMin (from scratch)93.3 / 80.7 / 85.586.3 / 72.0 / 77.982.5 / 68.4 / 74.2
OrthoProto (from scratch)92.8 / 80.0 / 85.588.6 / 75.7 / 79.882.9 / 69.3 / 74.4

One-class detection is far below the trained methods on plants (62 to 71 AUROC vs 93 to 94), at chance on insects (46 to 55), and closest on mammals (80.8 against 82.5 to 83.4). This holds even though the comparison favours the detectors, which start from ImageNet-pretrained features while our methods train from scratch, the protocol of Mildenberger et al.

We would be grateful if the reviewer reconsidered the score in the light of these experiments.

Thank you for the detailed response. Most of the concerns have been resolved, but under the label-matched probe, the AUROC improvement of OrthoProto over SupProto is not statistically significant, and BAcc even drops substantially. In addition, the response does not address the lack of thinning in the validation split and the comparison of only two batch-size settings. Therefore, I will maintain my score, and I hope that these limitations will be explicitly stated in the limitations of the revised manuscript.

Reviewer_QUkAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates supervised contrastive learning under extreme binary class imbalance and argues that the balanced-test-set evaluation fails to capture crucial properties of the classification model. There are three claims of contributions: (1) they propose an evaluation combining prevalence-invariant ranking metrics (AUROC, ER-AUROC), prevalence-projected metrics (AUPRCπ,F1πAUPRC_{\pi}, F1_{\pi}), and a bounded cost-sensitive decision score (SDCAS_{\text{DCA}} and its recalibrated counterpart SDCAcalS_{DCA}^{cal}) implemented within a unified reproduction framework called OutnumberedTales; (2) they identify a geometric conflict in SupProto where the features of minority stay within 90∘90^\circ of the learned majority center, and propose OrthoProto, which adds a squared angular hinge against the EMA majority mean direction; and (3) and propose CompMin, which maintains a momentum-updated memory bank with one slot per minority training instance when expected number of minority views per batch is small (maybe smaller than 1).

Critical strengths

S1: A good evaluation metric is important to the study of real-case imbalanced learning. S2: The proposed OrthoProto and CompMin achieve improvements in specific settings.

Critical weaknesses

W1: First, for OrthoProto, substituting K=2K=2 into Yi et al.'s (2025) symmetric spherical-cap radius bound does not theoretically justify a one-sided 90∘90^\circ penalty around an empirical EMA majority center; furthermore, a 100%100\% hemisphere violation rate does not imply linear inseparability (e.g., two linearly separable compact clusters may form a 60∘60^\circ angle but have a large violation rate). W2: There are too many SOTA algorithms for the imbalanced learning task you should compare. Although they are designed for balanced-test evaluation, they may have outstanding performance in your proposed test.

Questions for authors

Can you compare the most recent SCL algorithms, for example, RBL (Gao et al., Feature Directions Matter: Long-Tailed Learning via Rotated Balanced Representation, ICML 2023) and ProCo (Du et al., Probabilistic Contrastive Learning for Long-Tailed Visual Recognition, TPAMI2024)?

5 comments
AuthorsOpenReview ↗

W1: K=2 substitution withdrawn, the 90° margin justified by a sweep and a separability check

We thank the reviewer for the constructive feedback.

We agree with W1 and have spent the past week running experiments to address both weaknesses. For W1 we ran a margin sweep, a symmetric control and a direct separability measurement. For W2 and Q1 we ran RBL and ProCo as requested, added TSC, and extended KCL and PaCo to iNaturalist. All experiments follow the protocol of the paper. Results are reported in this comment and the next as AUROC / ER-AUROC / BAcc, multiplied by 100, and will be included in the revised manuscript that we will upload today.

W1. The 90° margin. We acknowledge an error in our initial theoretical justification and have therefore withdrawn the K=2K=2 substitution. The results presented by Yi et al. are explicitly formulated for K≥3K \geq 3. Furthermore, their cap angle represents a radius condition at a symmetric optimum rather than a margin around an empirical center. However, the behavior of their model at K=2K=2 remains highly informative. In their one-versus-all setting, the head class contains ρa\rho a samples and the tail classes contain aa samples each.

If we assume all samples are located at their respective class centers within a full batch and set the temperature to 1, utilizing K=2K=2 eliminates the tail-to-tail term. This leaves the objective function as

f(x)=ρalog⁡((ρa−1)+e−1aex)+alog⁡((a−1)+e−1ρaex)f(x) = \rho a \log((\rho a - 1) + e^{-1} a e^x) + a \log((a - 1) + e^{-1} \rho a e^x)

where xx represents cos⁡θ\cos \theta. This function is strictly increasing in xx for every ratio ρ\rho. The minimum therefore occurs exactly at x=−1x = -1, meaning the full-batch optimum naturally consists of a pair of antipodal centers regardless of the class imbalance.

This idealized model predicts no imbalance-induced geometric failure at K=2K=2, which points to the finite batch and the prototype gate as the likely sources of the geometry we observe under SupProto. The hinge is motivated by this empirical observation rather than being mathematically derived from the idealized model. The 90-degree margin thus has a straightforward interpretation. Specifically, a cosine of zero to the majority center serves as the natural nearest-center decision boundary when the two centers are antipodal. We now justify this margin empirically using seed 2000 and the full-pool probe:

margininsects 99:1plants 99:1
60°81.1 / 64.2 / 72.592.6 / 77.4 / 85.1
75°88.6 / 77.3 / 80.193.1 / 79.8 / 85.3
90° (paper)89.1 / 75.9 / 80.493.6 / 81.9 / 86.7
105°85.6 / 73.0 / 75.892.5 / 78.6 / 84.4
120°83.3 / 69.0 / 74.890.9 / 78.1 / 82.0
180°83.2 / 67.4 / 76.186.1 / 71.9 / 78.6
90°, two-sided85.5 / 71.3 / 76.891.9 / 79.5 / 83.1
SupProto (no hinge)80.7 / 64.5 / 73.293.0 / 79.5 / 85.3

On insects there is a clear optimum at 75 to 90°. At 60° the hinge goes silent before the minority has reached orthogonality, and the result falls back to SupProto's level. Past 90° the hinge keeps pushing minority embeddings that are already separated, which costs 3.5 to 5.9 AUROC points. Plants shows the same ordering with smaller gaps: 1.0 below the 90° setting at 60°, and 1.1 to 7.5 below it past 90°. This is a single seed, so it cannot separate 75° from 90°: we claim the shape of the curve, not the exact optimum. The sweep also carries a confound that we now state in the appendix: for a general margin θm\theta_m, the penalty [max⁡(0,cos⁡θ−cos⁡θm)]2[\max(0, \cos \theta - \cos \theta_m)]^2 has tangential force 2(cos⁡θ−cos⁡θm)sin⁡θ2(\cos \theta - \cos \theta_m) \sin \theta, so a larger margin pushes harder as well as further.

We keep the one-sided form because the symmetric variant (the same hinge also applied to majority embeddings against an EMA minority centre) loses 2 to 5 points across the three metrics.

W1, separability. The reviewer’s 60° example is correct: a high violation rate does not imply linear inseparability. The violation rate is removed in the revision. We instead test directly whether the minority has merged into the majority: a linear classifier on the frozen backbone features of the training split, scored by five-fold cross-validated AUROC (1.00 means a hyperplane separates the classes). Averaged over the three seeds of each of the nine iNaturalist cells, this gives 0.99 to 1.00 for SupMin, CompMin and OrthoProto, and for SupProto 0.97 to 0.99 in the three 95:5 cells and 0.77 to 0.94 in the six extreme cells. SupProto at the extreme rungs is the one case where the backbone itself fails to separate the classes, and that is the failure the hinge repairs.

AuthorsOpenReview ↗

W2 and Q1: RBL, ProCo, KCL, TSC and PaCo under the paper's protocol

W2, Q1. Recent SCL methods. We evaluated RBL and ProCo as requested. We also included TSC by Li et al. from 2022 and extended the evaluation of KCL and PaCo from MedMNIST to the iNaturalist dataset. All of these methods were tested on the three iNaturalist tasks at the 99:1 imbalance ratio using the exact protocol established in the paper. This protocol involves training a ResNet-50 architecture from scratch for 350 epochs with a batch size of 256 and a temperature of 0.07 across three seeds while utilizing the same two probes. Furthermore, we evaluated ProCo and RBL on the BreastMNIST dataset.

The resulting metrics report AUROC, ER-AUROC, and balanced accuracy at the 99:1 ratio using the full-pool probe. These values represent the mean across three seeds:

methodplantsinsectsmammals
ProCo84.1 / 69.5 / 76.581.2 / 67.3 / 73.573.9 / 60.1 / 67.9
RBL85.6 / 70.9 / 76.881.3 / 70.3 / 73.373.4 / 58.7 / 66.8
KCL88.0 / 72.6 / 79.683.4 / 67.3 / 75.980.7 / 65.5 / 72.5
TSC88.6 / 73.3 / 80.982.8 / 67.8 / 74.979.4 / 63.6 / 72.5
PaCo72.0 / 60.7 / 66.074.9 / 65.1 / 68.966.2 / 56.0 / 61.9
SupMin94.0 / 81.8 / 86.886.6 / 72.2 / 78.483.4 / 69.1 / 75.6
SupProto92.7 / 79.0 / 84.881.0 / 64.5 / 73.181.7 / 67.4 / 74.0
CompMin93.3 / 80.7 / 85.586.3 / 72.0 / 77.982.5 / 68.4 / 74.2
OrthoProto92.8 / 80.0 / 85.588.6 / 75.7 / 79.882.9 / 69.3 / 74.4

None of the five reaches SupMin, CompMin or OrthoProto in any cell. On insects, where SupProto is damaged, ProCo, RBL, KCL and TSC pass SupProto on AUROC and all five pass it on ER-AUROC.

Under the label-matched probe, which uses only the minority labels the encoder saw, same cells and seeds:

methodplantsinsectsmammals
ProCo79.2 / 65.0 / 53.374.8 / 60.8 / 51.765.2 / 55.2 / 52.9
RBL83.8 / 69.8 / 69.277.8 / 68.0 / 68.166.9 / 56.8 / 56.4
KCL79.0 / 67.2 / 57.978.8 / 64.1 / 56.570.1 / 57.5 / 51.7
TSC78.9 / 67.2 / 57.878.3 / 65.3 / 58.069.9 / 56.7 / 51.2
PaCo70.3 / 59.4 / 64.073.1 / 63.1 / 65.762.2 / 53.7 / 58.7
SupMin87.8 / 74.5 / 61.580.2 / 66.9 / 55.271.9 / 58.2 / 52.2
SupProto87.8 / 71.0 / 77.671.7 / 57.3 / 64.374.2 / 59.5 / 66.0
CompMin89.3 / 76.7 / 65.780.8 / 66.1 / 55.773.3 / 59.9 / 53.0
OrthoProto85.3 / 72.9 / 63.682.7 / 68.7 / 58.269.4 / 58.4 / 52.2

On AUROC all five stay below SupMin and CompMin in every cell, KCL and TSC pass OrthoProto on mammals (70.1 and 69.9 vs 69.4), and all five pass SupProto on insects (73.1 to 78.8 vs 71.7). On ER-AUROC, RBL passes SupMin and CompMin on insects (68.0 vs 66.9 and 66.1) and all five pass SupProto there. Balanced accuracy orders them differently: RBL is above SupMin, CompMin and OrthoProto in all three cells, PaCo above SupMin and OrthoProto, and the best value belongs to SupProto on plants and mammals and to RBL on insects.

The representation diagnostics of Mildenberger et al. at 99:1 are consistent with the ranking results: PaCo, ProCo and RBL show plain SupCon's collapse signature (SAA 0 to 3%, GPU > 0.7, CAC 56 to 63%), KCL and TSC a partial one, the four binary methods none (SAA 76 to 97%, GPU ≤ 0.06). Machinery built for many tail classes degenerates with two.

ProCo and RBL on BreastMNIST (100 epochs, label-matched probe, three seeds), against the paper's four methods (five seeds, three at the 10% rung, which the submission did not report), AUROC / ER-AUROC / BAcc:

method10%15%20%native
ProCo61.0 / 53.3 / 56.675.2 / 56.3 / 57.877.4 / 66.0 / 72.988.7 / 74.6 / 80.9
RBL59.6 / 54.7 / 53.561.1 / 52.4 / 55.162.1 / 53.9 / 58.168.1 / 56.0 / 61.3
SupMin58.3 / 53.1 / 57.559.7 / 50.5 / 57.259.1 / 51.4 / 56.470.8 / 57.9 / 61.3
SupProto66.1 / 55.4 / 57.263.7 / 54.6 / 59.167.3 / 56.5 / 59.879.3 / 66.0 / 70.6
CompMin63.8 / 53.6 / 57.064.0 / 52.2 / 60.664.5 / 54.4 / 58.876.8 / 63.5 / 69.6
OrthoProto67.0 / 55.9 / 59.473.9 / 58.6 / 66.077.0 / 61.9 / 69.883.0 / 70.6 / 74.2

RBL tracks SupMin within 3 AUROC points at every rung. ProCo is best at native (88.7 vs 83.0) and matches OrthoProto on AUROC at 20% and 15%, with ProCo holding the higher ER-AUROC and balanced accuracy at 20% and OrthoProto at 15%. At 10% ProCo drops to 61.0 where OrthoProto holds 67.0. The revision adds both methods to the BreastMNIST table and no longer claims that OrthoProto leads all methods at breast 20%, where ProCo matches it.

Scope: the new methods have three seeds at 99:1 on iNaturalist and on BreastMNIST. They have not yet been run to three seeds at 99.5:0.5 or on PneumoniaMNIST.

We would be grateful if the reviewer reconsidered the score in the light of these experiments.

Your CompMin seems to be a special case of Barbano et. al. 2023 (when \epsilon = 0), and their paper is not referenced.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.