- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This paper investigates how the preferred number of samples per class in supervised contrastive learning depends on training budget, objective formulation, and evaluation protocol. The authors first compare class-balanced batches with K=2 and K=4 under standard SupCon, observing that K=4 performs better at early training stages while K=2 becomes preferable after longer training in their single-view CIFAR-100 setting. They then examine SupCon's existing inside-log and outside-log objectives, showing that their difference corresponds to a within-positive KL term, and compare these formulations with an averaged per-target objective on identical K=4 batches. Experiments across multiple learning rates and training seeds demonstrate that changing the objective substantially improves K=4 performance relative to native SupCon, but often produces only small differences relative to K=2 at later training stages. Additional evaluations include encoder versus projection-head retrieval, frozen linear probing, class-disjoint retrieval on CUB and SOP, and experiments with stronger augmentations and multiple views.
Critical strengths
- The paper addresses a practically relevant question about the interaction between batch composition, training budget, and contrastive objective design. I appreciate the distinction between improving an objective under a fixed K and demonstrating that the resulting K is preferable to another batch composition. The matched comparisons and decomposition of the SupCon objectives provide a clear basis for interpreting these differences.
- The empirical investigation is extensive, with multiple training seeds, learning-rate comparisons, different training budgets, and evaluations using both projection-head and encoder representations. The additional class-disjoint retrieval experiments and frozen-feature linear probes help clarify the scope of the findings. The authors also explicitly report unsuccessful predictions, inconclusive comparisons, and limitations of their statistical analyses, which is valuable for interpreting the evidence.
Critical weaknesses
-
The technical novelty and significance are somewhat limited. The inside-log and outside-log formulations are already present in SupCon, and the averaged per-target objective is closely related to existing methods such as UniMoCo and SINCERE. The KL decomposition is useful for interpretation but follows directly from these established formulations. More importantly, the substantial improvements over native K=4 largely compensate for its performance gap relative to K=2. After learning-rate selection, the late differences from K=2 are only +0.08 and −0.40 Recall@1 points for the two alternative objectives. The paper therefore provides careful empirical observations, but relatively little new understanding that could predict or guide the optimal batch composition beyond the evaluated configurations.
-
The generality and reproducibility of the conclusions remain concerns. The most extensive controlled experiments use a relatively small convolutional encoder with single-view, weakly augmented CIFAR-100 training. Stronger augmentations and two-view training produce different K preferences, and the two-view experiments lack a same-objective comparison between K=2 and K=4 for the averaged per-target loss. In addition, several learning-rate-selected analyses rely on overlapping seed folds and repeatedly used evaluation data, while the complete historical training code, executed configurations, and raw experimental outputs are not provided. Although these limitations are acknowledged, they make it difficult to independently verify the broader conclusions or assess their robustness in more representative supervised contrastive learning settings.
Questions for authors
- The main K=4 versus K=2 reversal is established in a single-view setting, while the two-view experiments show substantially different behavior. How does the preference change when the same alternative objective is applied to both K values under standard multi-view SupCon training? In particular, the missing averaged per-target K=2 versus K=4 comparison would help isolate the effect of batch composition from the objective change.
- The early-to-late reversal is empirically consistent in the main configuration, but its underlying mechanism remains unclear. What primarily drives this transition: the number of positive samples, reduced class coverage, within-positive interactions, or optimization dynamics? A more explicit analysis of these factors would strengthen the contribution beyond the reported empirical comparisons.
- The alternative objectives substantially improve projection-head retrieval over native K=4, but their advantages over K=2 become small at longer budgets. How much of this improvement transfers to the backbone representations used for downstream tasks after learning-rate selection? The current linear-probe analysis uses only the reference-rate checkpoints, making it difficult to determine the practical benefit of the selected procedures.
- Several comparisons use overlapping learning-rate selection folds, previously examined evaluation data, and post-hoc analyses. How robust are the main conclusions to an independent evaluation protocol with fresh seeds and an untouched validation/test split? Please also clarify whether the complete executed training configurations and raw per-run results can be released, since the supplied companion currently supports only partial reproduction of the reported experiments.