acceptodds
Withdrawn from ICLR 2027

GradMix: Improving Supervised–Self-Supervised Compatibility through Attribution-Guided Feature Coverage

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-supervised learning (SSL) has been widely applied to supervised models, either during pre-training or as an auxiliary objective, to improve representation feature coverage beyond the information directly associated with class labels. This is particularly relevant to open-set recognition (OSR), which requires models to discriminate among known classes while retaining information beyond class-discriminative labels to distinguish unknown samples. However, when applied as an auxiliary objective for OSR, SSL can lower known-class classification accuracy, leading to a generalization and discrimination dilemma. To understand this problem, we analyze the representation-learning characteristics of supervised and self-supervised learning and attribute the degradation to conflicting representation-learning signals. We then propose GradMix, a data augmentation strategy that enables the SSL objective to learn more generalizable representations, letting SSL learn a broader range of representations compatible with supervised learning, thereby expanding the solution space shared by both objectives. Experimental results on standard and domain-specific OSR benchmarks show that GradMix consistently improves over naive joint supervised–self-supervised training and achieves competitive performance against state-of-the-art OSR methods. Beyond OSR, evaluations on out-of-distribution detection, robustness to common corruptions, and linear probing of SSL representations further demonstrate its ability to improve both representation generalization and discriminative precision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XH5qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the potential conflict between supervised and self-supervised learning objectives when jointly optimizing a shared feature extractor, particularly in open-set recognition. The authors argue that auxiliary self-supervised learning can improve feature coverage and unknown-class detection while simultaneously weakening the discriminative representations required for known-class classification. To address this issue, the paper proposes GradMix, an attribution-guided data augmentation strategy. GradMix uses LayerCAM to identify image regions emphasized by the self-supervised objective and partially replaces these regions with content from other images in the minibatch.

Critical strengths
  1. The proposed method is intuitive and relatively easy to integrate. Using attribution maps to redirect SSL toward alternative visual evidence is a direct idea.
  2. The experimental results indicate improvements over conventional augmentation strategies. For instance, the ImageNet100 linear-probing results show that GradMix improves SimCLR top-1 accuracy from 48.1% to 54.8% and MoCo v1 from 40.31% to 47.75%.
  3. The paper attempts to investigate the underlying representation-learning mechanism.
Critical weaknesses
  1. Some claims about state-of-the-art performance are inconsistent with the reported results. Table 4 states that GradMix outperforms existing methods on four of the six OSR protocols. However, based on the reported numbers, GradMix achieves the highest AUROC on only three protocols: MNIST, CIFAR+10, and CIFAR+50.
  2. The benchmark and evaluation paradigm are quite old-school. Please consider more modern setting beyond the classification task, which is extensively studied several years ago. Moreover, the selected classification tasks in your main exp are also quite naive!
  3. The interaction between patch mixing and self-supervised positive-pair semantics is insufficiently analyzed.
  4. The paper is low-formatting. i. Figures are quite unclear; ii. the methodology part don't have a clear overview to present your holistic achitecture; iii. The presentation is verbose in introduction.
  5. Heavy computation cost: GradMix requires gradient-based attribution maps, potentially from multiple convolutional layers, and additional augmented-view processing.
Questions for authors

See weaknesses.

Reviewer_Zo2nOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the interaction between supervised and self-supervised contrastive learning, primarily in open-set recognition (OSR). It proposes GradMix, an augmentation strategy that uses SSL-based LayerCAM attribution maps to identify highly activated image regions and replaces a selected region with content from another image. The augmented SSL objective is combined with the original SSL and supervised objectives to encourage broader feature coverage. Experiments cover OSR, closed-set classification, out-of-distribution detection, corruption robustness, and SSL linear probing.

Critical strengths

A concrete augmentation strategy: Applying attribution-guided replacement only to the auxiliary SSL branch, while retaining the original views, provides a practical way to encourage exploration of additional visual evidence without explicitly forcing identical supervised and self-supervised representations.

Critical weaknesses

The motivation and problem formulation do not establish a sufficiently compelling and coherent contribution: The motivating issue is primarily a limitation introduced by choosing SSL as an auxiliary solution for OSR. Addressing such a limitation can be worthwhile, but the paper does not adequately establish why it represents a fundamental bottleneck of OSR or constitutes a sufficiently significant advance over alternative training strategies.

The abstract emphasizes a “generalization and discrimination dilemma,” whereas the introduction develops discrimination degradation more concretely than the meaning and measurement of generalization. The relationship between these concepts remains underdeveloped. The narrative also moves from an OSR-centered motivation to broader claims covering OOD detection, robustness, and linear probing without clearly explaining which common problem unifies these evaluations. Although the abstract mentions these additional evaluations, their inclusion does not establish a coherent scope.

Section 2.1 contains extensive descriptive discussion but lacks a sufficiently precise problem formulation. Equation (1) specifies a weighted training objective; it does not formally define the discrimination and generalization criteria, their trade-off, or an operational criterion for objective compatibility.

The empirical claims are overstated, and presentation problems substantially weaken the argument. Table 4 claims that GradMix achieves the best performance on four of six protocols, but the table supports only three: MNIST, CIFAR+10, and CIFAR+50. On TinyImageNet, GradMix achieves 81.92, below CMI’s 85.3. It also trails the best reported results on SVHN and CIFAR10. This discrepancy directly overstates the empirical evidence for the claimed advantage.

Several presentation issues further impede assessment:

  • Many references appear as bare author–year text embedded in sentences. Appropriate parenthetical citation commands, such as \citep, would improve readability.
  • Acronyms are unnecessarily reintroduced within the main text, including SSL around lines 43 and 96. Defining SSL separately in the abstract and main text is reasonable, but repeatedly defining it within the body is distracting.
  • Tables 1 and 2 appear on page 2, while their subsequent discussion is poorly coordinated with Table 3 and the order of references.
  • Equation (1), first invoked around line 98, should be introduced alongside its definition rather than deferred.

These presentation issues are individually editable, but correcting them would not resolve the substantive concerns about motivation, formulation, and supporting evidence.

Questions for authors

None. The principal concerns involve the contribution, formulation, and supporting evidence rather than ambiguities that a short clarification would resolve. Correcting the table claim and presentation would be necessary, but would not by itself change my assessment of the current submission.

Reviewer_tH8xOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes GradMix, a data augmentation strategy that enables the SSL objective to learn more generalizable representations along with a supervised objective, for improving the open set recognition performance. The method is tested against other baseline OSR methods, for out-of-distribution detection, robustness to common corruptions, and linear probing of SSL.

Critical strengths
  1. The paper is generally well written.
  2. The idea of enriching features through SSL beyond information related to class labels is interesting. The problem of competing objectives with naively integrating both the supervised and self-supervised loss is accurate.
  3. The idea of using attribution maps for self-supervision is also quite interesting.
Critical weaknesses
  1. The work tested their method with old benchmarks, such as MNIST, SVHN, CIFAR10, CIFAR+10, CIFAR+50 and missed some of the more recent benchmarks with fine-grained datasets, such as CUB, Aircraft, or Stanford cars.

  2. Although their method is compared with a lot of methods in Table 4, they missed the baseline established by Vaze et al 2022. The baseline simply well-optimizes the model with only supervised loss and good closed set training. Without it, it is hard to analyze the benefit of adding extra loss terms. Moreover, the improvements reported by the proposed method in Table 4 is not quite significant.

  3. Intuitively, I do not quite see how a self-supervised objective can improve unknown “semantics” separation, as the self-supervised objective is not directed towards extracting any semantic information.

Questions for authors
  1. Although a lot of OSR methods are discussed and compared with, the authors are encouraged to discuss the recent papers from the literature. For example, Kundu et al 2026, Boosting Open Set Recognition Performance through Modulated Representation Learning, shares a similar motivation of going beyond the class related information through instance-specific features.

  2. What happens to Figures 1 and 2 with beta = 0? The authors should report controlled ablation studies with fully supervised loss beta = 0, removing LSSLoriL^{ori}_{SSL}

  3. Authors should explain CKA in further details

  4. The authors should discuss the additional training compute while adding the extra loss terms and using LayerCam.

  5. What is the variance of performance across different trials for each dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.