acceptodds
Withdrawn from ICLR 2027

Multi-Source Multi-View (MSMV): A robust SSL framework for imbalanced data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

One of the most challenging problems in self-supervised learning (SSL) is to achieve robustness of the learned representations irrespective of whether the data is balanced or imbalanced. SSL methods address this problem through techniques like feature reweighting, regularization or by making use of out of distribution(OOD) data. However, contrastive self supervised learning (CSSL) is more sensitive to imbalance, as frequent samples produce disproportionately more contrastive pairs leading to biased representations. Our analysis on the long-tail behavior in SSL clearly shows that most existing methods are biased in terms of *gradient ratio* which could lead to degradation of representation quality as well as poor decision boundary separation. We believe that the core reason for such quality degradation comes from the traditional *multi-view* design on which most existing SSL frameworks are based. Building on these findings, we design a novel contrastive learning framework (MSMV) in which multiple sources are considered to promote balanced information flow across views rather than a single source as in multi-view learning. The proposed model is enhanced by introducing a new contrastive loss function that helps in better representation learning for tail classes. Additionally, we adopted a unique augmentation strategy to improve the overall performance. Extensive experiments achieve consistent improvements across multiple benchmarks, including a 5% increase in linear evaluation and an 8% increase in few-shot accuracy on Cifar100-LT with ResNet-18, highlighting its effectiveness in imbalanced scenarios. We also find consistent gains on other datasets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self Supervised learning, Contrastive Learning, Long-Tail Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_j99tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Multi-Source Multi-View (MSMV), a contrastive self-supervised learning framework for long-tailed data. MSMV constructs positive pairs by fusing complementary view configurations of two randomly paired images using online and momentum encoders. It additionally introduces a filtered contrastive loss (FCL) that modifies similarities outside a specified interval. The motivation is that cross-image interactions can alleviate frequency-dependent optimization bias and improve tail-class representations. Evaluation includes linear classification on CIFAR-10/100-LT and ImageNet long-tailed settings, gradient-ratio diagnostics, augmentation and loss ablations, and representation analyses. The reported results show improvements over the evaluated baselines, particularly on CIFAR-100-LT.

Critical strengths
  1. Promising empirical improvements on a relevant problem. Learning useful representations from unlabeled long-tailed data is practically important. On CIFAR-100-LT, MSMV+ achieves 60.18% overall accuracy and 61.50% rare-class accuracy, compared with 55.27% and 53.55% for FASSL. These reported improvements suggest that the proposed combination merits further investigation.

  2. The evaluation goes beyond aggregate accuracy. The paper includes head/medium/tail results, gradient diagnostics, augmentation comparisons, and ablations of FCL. In particular, the loss ablations provide useful evidence that FCL contributes to the performance of the combined framework.

Critical weaknesses
  1. The central mechanism connecting feature fusion to reduced long-tail bias is not established. Section 3 motivates the method through a shared-component estimation example, while Appendix .11 argues that fusion introduces an additional gradient pathway. However, neither establishes that this pathway transfers useful complementary semantics, expands tail-class semantic coverage, or reduces frequency-dependent bias. Standard contrastive learning already shares encoder parameters across samples, and its gradients depend on other examples through the loss. Consequently, the existence of an additional dependency does not establish the claimed distinction in learning behavior. Likewise, the shared-component example is not connected mathematically to the actual fusion objective. The gradient-ratio results are suggestive, but do not establish that a ratio near 0.5 is desirable or that deviations are the root cause of representation degradation. The current evidence remains compatible with alternative explanations such as feature augmentation, regularization, or changes in positive-pair difficulty. This substantially weakens the paper’s main conceptual contribution, although it does not negate the reported empirical gains.

  2. The experimental comparisons do not sufficiently establish the necessity of the proposed design or the breadth of its claimed advantage. Figure 5 and the FCL ablations are useful, but do not isolate the proposed mechanism from simpler alternatives. For example, the paper does not report a controlled comparison between its crossed fusion, (A_n + C_a) ↔ (A_a + C_n), and (A_n + C_n) ↔ (A_a + C_a), using the same four views. Nor does it compare against feature-noise controls or establish how much of the benefit can be obtained by adding FCL to the underlying baseline. These comparisons matter because they could change the interpretation of the contribution. Baseline coverage is also limited: the newest method in the main tables is FASSL (2023), ImageNet comparisons omit relevant long-tailed SSL methods such as TS and Geometric Harmonization, and the subsampling experiments compare only against MoCoV2. Thus, the large gains in the latter setting do not establish superiority over strong specialized alternatives. Additionally, proportional subsampling primarily tests reduced data availability rather than increased class imbalance, and the claimed 8% “few-shot” improvement lacks an identifiable few-shot evaluation protocol.

Questions for authors
  1. Mechanism clarification: What evidence specifically supports the claim that fusion transfers useful complementary semantics to tail classes, rather than acting as feature augmentation or regularization? Is there a result linking the additional gradient pathway to reduced frequency sensitivity, beyond the observed gradient ratios and downstream accuracy?

  2. Discriminating evidence: Do you have controlled results for (A_n + C_n) ↔ (A_a + C_a), baseline + FCL, or feature perturbations matched in scale and, where possible, cross-view correlation to the counterpart features? These are not requests for a broad future-work agenda: such evidence would materially affect whether I view the contribution as a long-tail-specific mechanism or an effective augmentation/filtering combination.

  3. Loss implementation: Equation (7) appears to set out-of-range similarities to zero before exponentiation, so those entries still contribute exp(0) = 1 to the denominator. Are these entries actually zeroed or excluded in the implementation, and does filtering apply to positive pairs as well? Please clarify the exact objective used for the reported experiments.

  4. Evaluation clarification: Does the reported 8% “few-shot” gain refer to the rare-class accuracy difference in Table 2, or to a separate few-shot protocol? Also, are comparisons with TS or Geometric Harmonization available under the same ImageNet training and evaluation settings? These clarifications would affect my assessment of the scope and strength of the empirical claims.

Reviewer_oYV7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multi-Source Multi-View (MSMV), a contrastive self-supervised learning framework for imbalanced data. The authors first analyze the gradient ratio between head and tail classes and find that gradient imbalance leads to degraded representation quality. To address this problem, MSMV uses multiple source images and their augmented views to learn both inter and intra relations through feature fusion. The authors also introduce a Filtered Contrastive Loss (FCL) to eliminate extreme similarities and improve representation learning. Experiments on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT show that MSMV improves linear evaluation accuracy, particularly for tail classes.

Critical strengths

The paper addresses an important problem in self-supervised learning, focusing on improving representation quality for long-tailed data. The paper also introduces several new designs, including multi-source multi-view learning, normalized and augmented views, and Filtered Contrastive Loss (FCL). The experiments provide a detailed evaluation, including gradient ratio, KNN, and margin analysis in Figure 4, comparisons of different view generation strategies in Figure 5, and linear evaluation results on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT in Tables 1–3.

Critical weaknesses

The paper has several weaknesses with its motivation and method design. 1: the gradient analysis does not clearly explain why a gradient ratio of 0.5 is ideal or whether gradient imbalance is the root cause of degraded representation quality. The claim that generating more views increases data imbalance also needs further explanation. 2: lacks a clear explanation of why the Multi-Source Multi-View design can reduce the impact of class frequency on representation quality and improve the gradient ratio. 3: the paper does not clearly explain the Filtered Contrastive Loss (FCL), why filtering extreme similarities improves representation quality, or why these thresholds are chosen.

Questions for authors

1: Why 0.5 is the ideal value for the gradient ratio? A gradient ratio < 0.5 indicates a larger gradient norm for tail classes, which may also be beneficial for learning tail representations. You describe gradient imbalance as the "root cause" of degraded representation quality in the Conclusion. Could it be that Long-Tailed Data affects both the gradient ratio and representation quality, instead of gradient imbalance directly causing the degradation?

2: Why does generating N views lead to more severe data imbalance? Since the number of views increases by the same factor for both head and tail classes, the original imbalance ratio remains unchanged. Could you explain more clearly why generating more views would make the imbalance worse?

3: How is the counterpart image selected in the Multi-Source Multi-View framework? Also could you explain how this design is related to class frequency and the gradient ratio?

4: I have 3 questions about FCL: (1) When the similarity outside the threshold range is set to 0, the corresponding negative pair still contributes 1 to the denominator. Could you explain how this filtering works? (2) Filtering extreme similarities seems to conflict with the idea of using hard negatives in contrastive learning so why filtering these similarities helps improve representation quality? (3) For ImageNet-LT, the upper threshold is set to 1.0. This means that high similarity pairs are not filtered out.

Minor 1 (Related Work): Please check the last statement in Section 4. Does the cited work (Lin et al. (2023)) also use sampling with extra domain data?

Minor 2 (Figures): (1) Figures 1 and 4 present highly overlapping information and could be combined to avoid redundancy. (2) Figure 5 (page 7) appears before Figure 4 (page 8), which makes the figure order confusing.

Reviewer_KztDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework, Multi-Source Multi-View (MSMV) for self-supervised representation learning on long-tailed data. The method combines features from normalized and augmented views of two images using an online encoder and a momentum encoder. It also introduces a Filtered Contrastive Loss (FCL) intended to filter out representation pairs with very high or very low similarity. The authors prove their approach useful through an analysis of gradient norm ratios between head and tail classes and report improved performance on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT.

Critical strengths
  1. The results on CIFAR100-LT are promising, particularly for rare classes. Proposed MSMV achieves overall accuracy of 60.18% compared with 55.27% for the strongest baseline listed in Table 2, while rare class accuracy improves from 53.55% to 61.50%. These gains suggest that the method could be useful for improving representation learning on underrepresented classes.
  2. The paper includes ablation analysis of the augmentation strategy and FCL, along with threshold sensitivity and representation analyses. These experiments help assess the contributions of individual components and provide a more detailed understanding of the learned representations beyond overall accuracy.
Critical weaknesses
  1. The analysis does not sufficiently support the paper’s central explanation for why the proposed method MSMV improves long-tailed learning. The authors argue that generating multiple views increases class imbalance, but generating the same number of views per image leaves the head-to-tail frequency ratio unchanged. Moreover, the gradient analysis shows an association rather than establishing that a ratio near 0.5 leads to better representations. These gaps weaken the claim that MSMV’s gains arise from correcting the proposed optimization imbalance.
  2. The definition of FCL is inconsistent with its stated purpose. It appears to set similarities outside the selected interval to zero before exponentiation in Equation 7. However, these entries then contribute exp(0) = 1 to the denominator, rather than being excluded from the loss. It is also unclear how positive pairs outside the interval are handled. Since FCL is a central contribution, this ambiguity prevents me from determining the exact training objective and assessing whether the reported results support the proposed filtering mechanism.
Questions for authors
  1. Could you clarify how FCL is implemented? Are similarities outside the threshold interval set to zero before exponentiation or removed from the softmax denominator? How are positive pairs outside the interval handled, and which implementation produced the reported results?
  2. Generating the same number of views per image preserves the head-to-tail frequency ratio. In what sense does the multi-view construction increase imbalance, as shown in Section 2?
  3. What evidence supports treating a head-to-tail gradient norm ratio near 0.5 as desirable for representation learning? Does the analysis establish that this ratio explains the performance gains?
  4. Could you clarify how MSMV differs from existing mixing-based contrastive methods, such as i-Mix, and whether you have evaluated them under the same long-tailed setting? Also, SimCLR+TS appears in the gradient analysis but is absent from the main performance tables. How does MSMV compare with this baseline under a matched training and evaluation protocol?
  5. Were the baseline results in Tables 1–3 reproduced in your experiments or taken from the original papers? If reproduced, how many training epochs were used, and what were the batch sizes and augmentation strategies? If taken from prior work, how did you ensure that the experimental settings were comparable?
Reviewer_V6DzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies contrastive self-supervised learning on long-tailed datasets. It introduces a diagnostic gradient ratio between head and tail classes on a balanced probe subset, and proposes the MSMV framework, which fuses features from two source images across normalized and augmented views using an online/momentum encoder pair, trained with a Filtered Contrastive Loss (FCL). Experiments on CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT show consistent linear evaluation gains, especially on rare classes.

Critical strengths
  1. Strong empirical performance and thorough diagnostic analyses. MSMV+(Sum) achieves notable gains over prior long-tailed SSL baselines, improving overall linear accuracy on CIFAR-100-LT from 55.27% (FASSL) to 60.18% and rare-class accuracy from 53.55% to 61.50% (Table 2), as well as reaching 52.90% on ImageNet-LT vs. 49.50% for MoCoV2 (Table 3). The appendix further supports these results with threshold sensitivity studies, alignment/uniformity curves, and per-class evaluations.
  2. Effective view design and clean gradient diagnostics. Figure 5 clearly demonstrates the benefit of pairing normalized and augmented views (NA vs. AA), and Appendix .10 carefully separates the balanced diagnostic probe set from network training without label leakage.
Critical weaknesses
  1. Exposition and implementation discrepancy in the Filtered Contrastive Loss (FCL). As written in Equation (7), setting similarities outside [λl,λh][\lambda_l, \lambda_h] to zero before exponentiation still contributes exp⁡(0/τ)=1\exp(0/\tau)=1 to the denominator rather than excluding those pairs. The linked code (Loss/loss.py) instead masks entries after exponentiation; however, when λh<1\lambda_h < 1 (e.g., λh=0.9\lambda_h=0.9 on CIFAR), diagonal self-similarities (Si,i=1S_{i,i}=1) are already zeroed by the mask while the denominator still subtracts exp⁡(1/τ)\exp(1/\tau), which likely causes the NAN entries reported in Tables 4–6 for λh<1\lambda_h < 1.
  2. Imprecise theoretical phrasing and lack of a momentum-matched baseline in Figure 5. In Section 2, the example showing view counts increasing from 5000/505000/50 to 10000/10010000/100 reflects a larger absolute gap rather than a changed relative ratio (100:1100:1), and the chain-rule notation in Appendix .11 (Equations (31)–(37)) does not strictly reflect the no_grad momentum encoder in Section 5. In addition, Figure 5 compares MSMV (which uses a momentum encoder) against SimCLR (which does not), so adding a momentum-matched baseline with the same NA views and FCL would more cleanly isolate the contribution of two-source fusion.
Questions for authors
  1. Could you clarify the exact FCL formulation used for Tables 1–3 to reconcile Equation (7) with Our_xent_loss, and confirm whether avoiding the duplicate subtraction of exp⁡(1/τ)\exp(1/\tau) when λh<1\lambda_h < 1 resolves the NAN cases in Tables 4–6?
  2. Under a momentum-encoder baseline using the same normalized/augmented (NA) views and FCL, how much of the accuracy gain in Figure 5 comes specifically from two-source fusion (⊛\circledast)?
Reviewer_zS3mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a contrastive self-supervised learning framework (MSMV) to improve robustness of representation learning on imbalanced data. The authors observe that most existing methods suffer from a biased gradient ratio, which can be harmful to representation learning. Based on this observation, MSMV extends the single-source paradigm to a multi-source setting, thereby enhancing balanced information flow across different views. In addition, the framework integrates a Filtered Contrastive Loss (FCL) to improve representation learning for tail classes. Experiments on CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT demonstrate improvements over existing SSL methods.

Critical strengths
  1. Long-tailed self-supervised learning is an important and challenging problem. The paper offers an insteresting perspective on the gradient imbalance issue.

  2. The proposed method demonstrates substantial performance improvements over existing methods on several long-tailed benchmarks, indicating its potential effectiveness in addressing data imbalance problem.

Critical weaknesses
  1. The design is not well motivated. The paper observes the gradient imbalance issue in existing methods. However, the motivation of the proposed multi-source multi-view framwork is not clear. The connection between the problem analysis and the methodological design needs to be better established.

  2. Insufficient theoretical justification. Although the authors make strong claims about the relationship between gradient balance and representation quality, the theoretical analysis that supporting this claim remains limited.

  3. Lack of Efficiency Analysis. The proposed method introduce additional view construction and similarity measure, which can lead to considerable computational and memory cost. The authors should provide a detailed efficiency analysis compared to existing methods.

  4. Poor paper organization. The paper is not well organized, making it difficult to follow the overall methodology and its motivation. Some figures and statements are unclear and hard to understand.

Questions for authors
  1. Please refer to weaknesses for major concerns.
  2. How are the lower and upper thresholds in FCL determined? How sensitive is the model performance to these thresholds?
  3. How does MSMV scale to large-scale datasets in terms of performance and efficiency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.