- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies supervised contrastive learning under extreme binary class imbalance and identifies two failure modes in existing minority-aware methods. First, SupProto's fixed prototypes can conflict with the representation geometry learned by the encoder. To address this, the authors propose OrthoProto, which adds an orthogonality hinge to SupProto. Second, SupMin's minority supervision remains confined to the batch, and its class-level term remains idle at most steps when the expected number of minority views per batch falls below one. To address this, the authors propose CompMin, which maintains a memory bank containing one slot per minority training example. The paper also introduces OutnumberedTales, a reproduction framework that implements published imbalance-aware objectives under a shared protocol and includes a decision-aware evaluation suite. Experiments on binary iNaturalist and MedMNIST tasks show that OrthoProto reduces geometric violations, while CompMin improves robustness in the small-batch regime.
Critical strengths
The paper provides a clear analysis of two failure modes (the geometric conflict in SupProto and the batch-local supervision limitation in SupMin) in existing minority-aware contrastive learning methods and proposes targeted solutions (OrthoProto and CompMin). The experiments are conducted under a shared protocol across multiple binary tasks (iNaturalist and MedMNIST) and minority prevalences (ranging from 0.5% to approximately 27%). The evaluation using both ranking and decision metrics offers useful insights into how different metrics can lead to different conclusions about model performance. The batch-size experiment provides convincing evidence for CompMin's robustness in the starved regime.
Critical weaknesses
I think the main weaknesses are the limited methodological novelty and effectiveness.
1: OrthoProto adds an orthogonality hinge to SupProto, but this is a relatively simple modification and does not consistently outperform SupMin (Tables 3 4).
2: CompMin improves robustness in the small-batch regime (batch size 32), but its use of a memory bank builds on established ideas like (MoCo, XBM, and GSupCon), and it provides limited improvement over SupMin at the standard batch size of 256 (Tables 3 4).
3: OutnumberedTales integrates existing imbalance-aware objectives under a shared protocol and pairs them with a decision-aware evaluation suite, but does not introduce fundamentally new methods or evaluation metrics.
Questions for authors
1: Given the concerns raised in the weaknesses, could the authors more clearly explain the main contributions and novel aspects of OrthoProto, CompMin, and OutnumberedTales compared with existing work?
2: In Table 4, OrthoProto still underperforms SupMin in several extreme imbalance settings. Why not directly use or improve SupMin instead?
3: CompMin is partly motivated by memory-intensive encoders that require small batch sizes, but all experiments use ResNet-50. Why did the authors not evaluate Vision Transformers (ViTs)? Could the authors also discuss whether CompMin's batch-size robustness would generalize to ViT-based encoders and more diverse datasets?
4: Tables 1 and 2 combine the evaluation of the proposed methods with the analysis of ranking and decision metrics. I find the results somewhat difficult to follow. Could the authors present these two analyses more clearly, possibly in separate tables?
2 comments
Q3 and Q4: the batch ladder, a ViT-Tiny from scratch, and Tables 1 and 2 split by lens
Q3
Why did the submission not evaluate ViTs? Our core claim concerns the batch size rather than the specific backbone architecture. We updated our motivating text to clarify that limited hardware and high-resolution inputs frequently force the use of small batches in practice. The loss function is completely agnostic to the architecture. It strictly depends on the expected number of minority views per batch, which is defined as . SupMin fails because its class term goes idle whenever this expected count falls below one. We demonstrated this by scaling a ResNet-50 from a batch size of 256 down to 32 across six extreme cells using three seeds and the full-pool probe. The paired AUROC changes and the performance gap between CompMin and SupMin at batch 32 are presented below (95% intervals):
| cell | p_B at bs32 | Δ SupMin | Δ CompMin | CompMin − SupMin, AUROC | CompMin − SupMin, ER-AUROC |
|---|---|---|---|---|---|
| plants 99:1 | 0.64 | −1.6 | +0.7 | +1.5 [+0.5, +2.6] | +5.8 [+4.0, +7.7] |
| insects 99:1 | 0.64 | −3.8 | +0.1 | +3.7 [+0.0, +7.3] | +6.0 [−1.3, +13.4] |
| mammals 99:1 | 0.64 | −1.7 | −0.6 | +0.2 [−0.8, +1.2] | +1.0 [+0.4, +1.6] |
| plants 99.5:0.5 | 0.32 | −2.4 | −0.6 | +2.0 [+0.8, +3.2] | +5.2 [+2.1, +8.4] |
| insects 99.5:0.5 | 0.32 | −5.1 | +0.4 | +5.0 [+2.8, +7.3] | +6.9 [+4.2, +9.6] |
| mammals 99.5:0.5 | 0.32 | −4.2 | −1.5 | +2.3 [+0.8, +3.7] | +3.9 [+0.6, +7.3] |
Across the six cells, CompMin leads SupMin by an average of 2.4 AUROC and 4.8 ER-AUROC at batch 32 while staying within a point of it at 256. That also highlights CompMin's practical case, where a batch of 32 on a less capable GPU stays within 1.5 AUROC points of a batch of 256 for the cost of one 128-d bank vector per minority training image.
Does CompMin's batch-size robustness generalise to ViT encoders? On a ViT trained from scratch the effect points the same way as on the ResNet-50, but it is smaller. We tested insects 99:1 in two settings.
With ImageNet-pretrained backbones (ViT-B/16 and ResNet-50), every method, plain SupCon included, reaches 96 to 98 AUROC at both batch sizes. Pretraining saturates the task, so it cannot show a difference between objectives, and every claim in the paper is scoped to training from scratch.
With a ViT-Tiny/16 trained from scratch (paper protocol, AdamW, three seeds, full-pool probe):
| method | batch 256 | batch 32 | change |
|---|---|---|---|
| SupMin | 84.2 / 71.1 / 76.8 | 82.6 / 69.2 / 75.2 | −1.6 / −1.9 / −1.6 |
| CompMin | 83.9 / 70.3 / 75.6 | 83.1 / 71.0 / 75.3 | −0.8 / +0.7 / −0.3 |
Shrinking the batch from 256 to 32 costs SupMin 1.6 AUROC and CompMin 0.8. On the ResNet-50 the same change cost SupMin 3.8 and left CompMin unchanged (+0.1). At batch 32 CompMin is ahead of SupMin by 0.5 AUROC and 1.8 ER-AUROC on the three-seed mean and level on balanced accuracy. On one of the three seeds SupMin has the higher AUROC.
Q4. We agree with this suggestion and have updated the manuscript accordingly. The revision now splits Tables 1 and 2 by evaluation lens rather than by dataset. Table 1 is located in the ranking metrics subsection and exclusively reports the AUROC and ER-AUROC for all methods across the seven medical rungs. Table 2 is placed under the decision metrics subsection and focuses solely on the S_DCA at the clinical operating point alongside the S_DCA^cal after isotonic recalibration. The best trivial policy scores 70.4 on BreastMNIST and 50.8 on PneumoniaMNIST. The gap between the raw and recalibrated scores directly reflects the cost of miscalibration. A single column from each table for BreastMNIST at 20 percent illustrates this reasoning clearly:
| method | AUROC | ER-AUROC | S_DCA | S_DCA^cal |
|---|---|---|---|---|
| Logit adj. | 67.1 | 54.3 | 69.9 | 78.6 |
| BCL | 63.6 | 52.9 | 72.8 | 77.6 |
| SupCon | 61.8 | 52.8 | 70.5 | 77.4 |
| KCL | 59.2 | 52.1 | 70.5 | 76.3 |
| PaCo | 61.5 | 53.5 | 70.7 | 77.5 |
| SupMin | 59.1 | 51.4 | 70.7 | 75.8 |
| SupProto | 67.3 | 56.5 | 70.8 | 78.4 |
| CompMin | 64.5 | 54.4 | 73.2 | 77.2 |
| OrthoProto | 77.0 | 61.9 | 72.7 | 81.6 |
| trivial policy | 70.4 |
When evaluating through the ranking lens, SupProto outperforms BCL by four AUROC points. However, looking at these same methods through the decision lens reveals a completely different outcome. SupProto barely exceeds the trivial policy with a score of 70.8 compared to 70.4, whereas BCL clearly surpasses it with a 72.8. The ranking metrics completely mask this practical difference. Furthermore, five of the seven contrastive methods remain within half a point of the trivial policy. Among the two-stage contrastive methods, only CompMin and OrthoProto distinctly exceed this baseline, and every method loses between 4 and 9 points due to miscalibration. This fundamental disagreement between evaluation lenses serves as the central finding of our paper.
We would be grateful if the reviewer reconsidered the score in the light of these experiments.
W1 to W3, Q1 and Q2: what is new, and why two repairs instead of one
We thank the reviewer for the feedback. Over the past week we ran the following experiments: two batch controls, an ImageNet-pretrained ViT-B/16 and ResNet-50 panel, and a ViT-Tiny/16 trained from scratch under the paper's protocol. This comment covers W1 to W3, Q1 and Q2, the next one Q3 and Q4. Numbers are AUROC / ER-AUROC / BAcc, ×100.
W1 to W3, Q1. We agree on what is borrowed. Neither loss is a new family, CompMin sits in the MoCo, XBM and GSupCon line, and no metric is ours. We disagree that nothing new follows.
Q2. The revision does not present OrthoProto as a method that beats SupMin everywhere. No method does, and showing that is the point of the protocol. Under the label-matched probe (batch 256, mean over three seeds):
In a direct comparison, OrthoProto achieves the highest AUROC in 5 of the 9 cells. This includes the insects category at every prevalence level and all three cells evaluated at the 95:5 ratio. SupMin leads in the remaining 4 cells covering plants and mammals at the 99:1 and 0.5 percent levels by a margin of 2.4 to 4.9 points. However, evaluating the exact same configuration using balanced accuracy reveals a different outcome where neither method wins. Among the four methods evaluated in our study, SupProto performs best across all nine cells and outperforms the others by 4 to 13 points at the extreme rungs. Across the 36 combinations of cells and metrics, OrthoProto leads in 16 columns, SupProto in 10, CompMin in 9, and SupMin in only 1. If we evaluated performance based on balanced accuracy alone, SupProto would appear universally superior and these nuanced dynamics would remain hidden. The revised manuscript explicitly acknowledges that no single method dominates across all conditions. The actual claim for OrthoProto is more specific and remains fully valid. It successfully improves upon SupProto in 22 to 25 of the 27 cell and seed pairs across every metric under the full-pool probe with a Wilcoxon p-value below 0.001.
On "why not directly use or improve SupMin instead”. We did actually improve SupMin by developing CompMin. SupMin suffers from statistical starvation when the expected minority count per batch drops below one rather than a geometric failure. CompMin solves this using a dedicated minority bank and significantly outperforms SupMin by an average of 2.4 AUROC and 4.8 ER-AUROC points in starved cells at a batch size of 32. Each repair specifically targets the unique failure of its base method. The orthogonality hinge requires a prototype pull to guide repelled minority samples, which SupMin lacks. Finally, when label budgets are strictly matched on the medical panel, OrthoProto clearly outperforms SupMin at extreme imbalances. On PneumoniaMNIST, OrthoProto achieves a 96.0 AUROC versus 92.9 for SupMin at the 1 percent level and 95.0 versus 90.1 at 0.5 percent.