acceptodds
Under review as a conference paper at ICLR 2027

On the Counterfactual Relation Certification of Self-Supervised Learning

Abstract

Self-supervised learning (SSL) learns representations by constructing supervision from unlabeled data. However, satisfying self-supervised objectives does not guarantee reliable representations, since the same training relations may be explained by semantic factors or by nuisance cues such as background. Consequently, an SSL model may achieve low training loss while relying on spurious dependencies that fail under distribution, task, or semantic shifts. We address this issue from a counterfactual perspective and characterize model reliability through two complementary principles: the learned dependency should be revised when essential semantic factors change, while remaining stable when semantic factors are preserved under nuisance variations. Since true counterfactual interventions are unavailable in SSL, we propose General SSL (GeSSL), which approximates these principles through observable training proxies. GeSSL models counterfactual necessity with a discriminative relation loss and counterfactual sufficiency through consistency at the sample, support-query set, and task-distribution levels. Theoretical analysis establishes bounded generalization error on novel tasks. Extensive experiments show that GeSSL consistently improves representative discriminative and generative SSL methods across multiple evaluation protocols.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xiAVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies unreliable relations in self supervised learning and proposes GeSSL based on counterfactual necessity and sufficiency.

GeSSL combines adaptive relation discrimination with consistency across samples, sets, and training batches.

Critical strengths

GeSSL improves a wide range of self supervised baselines across classification, transfer learning, few shot learning, and robustness evaluations.

The ablation and robustness results show consistent contributions from the proposed relation and consistency objectives.

Critical weaknesses

The algorithmic contribution is incremental because GeSSL mainly combines standard SSL consistency, support query bilevel optimization, and relation filtering under a counterfactual interpretation, without introducing a clearly new learning mechanism.

The theoretical analysis does not establish counterfactual reliability or generalization to new task distributions, since the main bound only considers iid episodes from the same distribution and downstream control requires an additional alignment assumption.

Questions for authors

Can the authors clarify whether the baseline and GeSSL use the same number of training samples, augmented views, and optimization updates?

9 comments
AuthorsOpenReview ↗

Response to W1: Part I

Response to W1

We thank the reviewer for this important comment. We agree that SSL consistency, relation discrimination, and support-query bi-level optimization are individually established techniques. However, we would like to emphasize that the central contribution of our work is not a particular regularization loss or optimization algorithm, but a general framework for reliable self-supervised relation learning, guided by two complementary counterfactual-inspired principles: necessity and sufficiency. GeSSL represents one concrete realization of this framework, rather than the only possible implementation.

To clarify the contribution, we distinguish between (i) the framework-level formulation, (ii) the specific learning mechanism implemented in GeSSL, and (iii) empirical evidence that the proposed principles can be realized through different optimization methods.

1. Framework-level contribution: Two complementary requirements for reliable relation learning.

As discussed in Section 2, conventional SSL objectives may be satisfied through spurious correlations even when the constructed self-supervised relations are valid. Our central question is therefore not simply how to improve discriminability or invariance, but how to characterize whether the learned relations depend on task-relevant semantic information.

We introduce two complementary behavioral principles:

  • Counterfactual necessity: Learned relations should be sensitive to changes in task-relevant semantic compatibility, even when nuisance factors remain similar.
  • Counterfactual sufficiency: Learned relations should remain stable when semantic information is preserved despite nuisance variations.

These principles provide a unified framework for evaluating and improving relation reliability. Importantly, the principles do not prescribe a specific loss function, relation boundary, or optimization algorithm. Different implementations can approximate the same behavioral requirements through different observable objectives.

Our contribution is to formulate these complementary requirements at the level of self-supervised relations and demonstrate how they can guide the design of reliable representation learning algorithms across different SSL paradigms.

2. Algorithmic contribution: Coupling adaptive relation discrimination with cross-view validation.

Within this framework, GeSSL provides a concrete learning mechanism rather than a straightforward addition of independent regularizers.

Specifically, the necessity proxy learns an anchor-specific compatibility boundary through the discriminative relation loss. The sufficiency proxies evaluate stability across augmented observations, support-query relation sets, and sampled mini-batch environments.

Crucially, these components are coupled through the bi-level optimization in Eqs. (4)–(5). The support-set objective updates the representation encoder using both the base SSL loss and the discriminative relation loss. The resulting encoder is then evaluated on separately augmented query relations.

Because the support-induced encoder update depends on the relation-boundary parameters, the outer-loop query loss also updates the boundary learner. Consequently, the learned relation boundary is optimized according to the effectiveness of its induced representation updates on query observations, rather than solely according to current representation distances.

This interaction distinguishes the complete GeSSL mechanism from independently combining similarity filtering and consistency regularization. Nevertheless, we do not claim that the underlying bi-level optimization technique itself is novel.

AuthorsOpenReview ↗

Response to W1: Part II

3. Existing evidence for the complementary contributions.

Our original ablations provide evidence that the proposed necessity and sufficiency proxies contribute independently and complementarily.

On ImageNet-100 (Table 25), SimCLR achieves 70.15% Top-1 accuracy. Introducing necessity alone improves accuracy to 71.24%, while set-level sufficiency alone achieves 71.58%. Combining necessity and set-level sufficiency further improves accuracy to 72.31%, and the complete GeSSL reaches 72.96%.

Similar improvements are observed with MAE, where accuracy increases from 76.56% to 78.45% with the full framework.

Furthermore, Figure 11 compares our bi-level implementation with joint single-stage optimization and independent mini-batch adaptation, supporting the effectiveness of the proposed coupled optimization strategy.

However, we agree that these ablations primarily validate our specific implementation and do not, by themselves, establish the broader applicability of the proposed principles. We therefore conduct an additional study that explicitly separates the framework from its particular realization.

4. New experiment: Principle–Implementation Decoupling Study.

To directly examine whether the proposed framework is more general than the specific GeSSL implementation, we introduce a new experiment in which the necessity and sufficiency principles are preserved while their underlying optimization mechanisms are replaced.

For counterfactual necessity, we consider two distinct implementations:

  • Necessity-A (Original GeSSL): Adaptive relation discrimination using an anchor-specific compatibility boundary.
  • Necessity-B (Alternative): Unsupervised mutual-neighbor relation ranking with a margin-based discriminative objective, without the adaptive boundary learner.

For counterfactual sufficiency, we similarly consider:

  • Sufficiency-A (Original GeSSL): Support-query bi-level optimization with multi-batch relation environments.
  • Sufficiency-B (Alternative): Cross-view relational distribution consistency, which directly aligns neighborhood relation distributions across augmented observations without bi-level optimization.

Both alternative implementations use unlabeled observations and do not require semantic annotations during SSL pre-training.

We evaluate the individual implementations and their combinations under matched training protocols, including the original GeSSL (A+A), two partially substituted variants (A+B and B+A), and a fully alternative implementation (B+B).

The results on controlled Waterbirds experiments are summarized below:

ConfigurationWorst-group Acc. (%)Semantic Sensitivity (%)Nuisance Invariance (%)
Baseline SSL47.867.860.4
Necessity-A only53.275.362.1
Necessity-B only52.373.961.6
Sufficiency-A only56.469.572.8
Sufficiency-B only55.168.771.3
A+A (Original GeSSL)66.778.274.6
A+B64.577.473.1
B+A63.875.874.1
B+B (Fully Alternative)62.374.872.2

The results reveal three important observations:

  1. Different implementations of the necessity and sufficiency proxies independently improve their corresponding reliability measures, suggesting that the two principles are not restricted to a particular loss function.
  2. Combining necessity and sufficiency consistently achieves more balanced semantic sensitivity and nuisance invariance than optimizing either requirement alone. This pattern holds across different combinations of implementation mechanisms.
  3. Importantly, the fully alternative implementation (B+B), which does not use GeSSL's original adaptive boundary or support-query bi-level optimization, still achieves substantial improvements over the baseline. This provides direct empirical evidence that the proposed framework can guide the construction of effective learning algorithms beyond its original GeSSL realization.

We further repeat this evaluation using both discriminative and generative SSL backbones, represented by SimCLR and MAE. The corresponding Top-1 improvements of the fully alternative implementation over their respective baselines are 1.9 and 1.4 percentage points, respectively.

These results support the portability of the proposed necessity-sufficiency framework across different optimization mechanisms and SSL paradigms.

AuthorsOpenReview ↗

Response to W1: Part III

5. Distinguishing the framework contribution from implementation-specific improvements.

To further isolate the benefit of GeSSL's specific optimization mechanism, we compare the original method with fixed-threshold similarity regularization, adaptive relation learning without bi-level optimization, and bi-level optimization without the discriminative relation loss.

Under matched training settings, Full GeSSL achieves a worst-group accuracy of 66.7%, compared with 50.9%, 54.6%, and 55.8% for these alternatives, respectively.

These comparisons support the effectiveness of the original coupled learning mechanism. Meanwhile, the Principle–Implementation Decoupling Study demonstrates that the broader reliability framework does not depend exclusively on that mechanism.

We therefore distinguish two complementary contributions:

  • Framework contribution: A counterfactual-inspired formulation of reliable self-supervised relation learning through semantic sensitivity and nuisance invariance, which can guide multiple algorithmic realizations.
  • Algorithmic contribution: GeSSL, a specific realization that couples adaptive relation discrimination with cross-view validation through bi-level optimization.

We emphasize that the framework provides behavioral design principles rather than an unconditional guarantee that every possible implementation will improve performance. Likewise, the observed improvements under controlled semantic and background variations do not imply formal causal identification.

In summary, we agree that the individual optimization ingredients of GeSSL are established. However, the contribution of our work extends beyond their combination. By explicitly distinguishing the necessity-sufficiency framework from its particular implementation and demonstrating consistent improvements across alternative realizations, we provide evidence that the proposed principles have broader methodological value for reliable self-supervised learning. We will revise the manuscript to make this framework-level contribution and its distinction from the specific GeSSL algorithm clearer.

Reviewer_Mc3bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a reliability problem in self-supervised learning (SSL), i.e., models may satisfy self-supervised relations through spurious/nuisance dependencies rather than semantically meaningful factors. First, it introduces two counterfactual-inspired principles: ounterfactual necessity and counterfactual sufficiency. Second, it proposes GeSSL to approximate them, with a generalization bound for the observable GeSSL objective and an additional assumption linking that objective to downstream risk. Finally, GeSSL is empirically applied to several discriminative and generative SSL baselines and evaluated on linear probing, semi-supervised learning, transfer, few-shot learning, ablations, and additional appendix experiments.

Critical strengths

S1. A unified relation-based perspective with practical plug-in generalizability.

The paper creatively frames both D-SSL and G-SSL paradigms under a single relation-based view and introduces counterfactual necessity and sufficiency as conceptual regularizers.This perspective yields an empirically effective, plug-and-play module. The paper evaluates GeSSL across many SSL baselines and protocols, including SimCLR, MoCo, BYOL, MAE, etc.. The consistent improvements demonstrate its broad applicability and practical value to the SSL community.

S2. Broad empirical evaluation.

The scope of the empirical validation is impressive. The authors evaluate GeSSL across a wide spectrum of downstream tasks, including linear probing, semi-supervised learning, object detection, video tracking, and few-shot learning. They also conduct ablation studies on model efficiency and augmentation reliance, which substantiates the method's efficacy and robustness.

Critical weaknesses

W1. The central counterfactual claims are not adequately substantiated.

The paper explicitly acknowledges that true interventions on semantic/nuisance factors are unavailable, and the theory only bounds the observable SSL objective. The downstream link relies on Assumption 4.2, which is strong and untested. This is consequential because the main novelty and framing depend on “counterfactual necessity/sufficiency,” but the evidence supports at most a proxy-based regularizer, not counterfactual certification.

W2. Methodological and reproducibility details are insufficiently specified.

No source code or dataset documentation is provided. Meanwhile, the main text does not clearly define the exact formulation of the discriminative relation loss, the relation-boundary learner, how positive/negative relations are constructed, and how the boundary is trained are not clear enough in the main text. There are also apparent inconsistencies/duplications in reported tables, e.g., two DINO+GeSSL rows in Table 1 with different numbers. This weakens confidence in the empirical claims and makes fair comparison difficult.

Questions for authors

Q1. Please provide the exact formulation of the discriminative relation loss and the relation-boundary learner, and specify how positive/negative relations are constructed. It would be better to include pseudocode or a precise loss equation in the main text.

Q2. In Table 1, there appear to be two DINO+GeSSL rows with different Top-1/Top-5 results. Please clarify which setting each corresponds to and whether all GeSSL comparisons use identical compute, batch size, epochs, and augmentation budgets as the baselines.

Q3. How does the proposed counterfactual necessity/sufficiency differ materially from existing invariance/equivariance regularization, hard-negative mining, or meta-learning-style support-query validation? A clearer positioning against these lines would help assess novelty.

7 comments
AuthorsOpenReview ↗

Response to W1: Part I

Response to W1

We thank the reviewer for this important concern. We agree that GeSSL does not establish exact counterfactual identification or provide formal guarantees for recovering latent semantic factors from unlabeled observations. However, we would like to clarify that our central contribution is not merely to introduce another regularizer, but to formulate two complementary counterfactual-inspired reliability principles and translate them into trainable mechanisms that explicitly target different failure modes of self-supervised relation learning. To substantiate this connection, we further provide controlled experiments that directly evaluate the behavioral properties associated with necessity and sufficiency.

1. Why are counterfactual necessity and sufficiency more than conceptual labels?

As emphasized in Section 2, the fundamental problem we address is not whether constructed self-supervised relations are valid, but whether the model learns them for the right reasons. Even valid relations can be satisfied through spurious dependencies on background, texture, or other nuisance attributes.

Our counterfactual perspective characterizes reliable relation learning through two complementary requirements:

  • Necessity (semantic sensitivity): When task-relevant semantic information changes while nuisance attributes remain unchanged, the learned relation should respond to this change.
  • Sufficiency (nuisance invariance): When semantic information remains unchanged despite variations in nuisance attributes, the learned relation should remain stable.

These principles directly guide the construction of GeSSL. The discriminative relation loss introduces an adaptive compatibility boundary to discourage indiscriminate relation preservation, whereas sample-, set-, and distribution-level consistency encourage stability across augmented views and relation environments. Crucially, support-query bi-level optimization couples these mechanisms by evaluating whether relation-induced updates remain effective on separately augmented observations.

Therefore, the counterfactual perspective provides a specific behavioral criterion for designing and evaluating the proposed optimization mechanisms, rather than simply assigning causal terminology to conventional losses.

2. New controlled experiments directly evaluating necessity and sufficiency.

To address the reviewer's concern that conventional accuracy and corruption benchmarks cannot establish the intended mechanism, we conduct additional controlled experiments on Waterbirds, where bird categories represent task-relevant semantics and backgrounds represent nuisance attributes.

We construct strongly correlated training environments (ρtrain=0.95\rho_{\mathrm{train}}=0.95) and evaluate the learned representations under balanced (ρtest=0.50\rho_{\mathrm{test}}=0.50) and reversed (ρtest=0.05\rho_{\mathrm{test}}=0.05) background-label correlations. Semantic and background annotations are used only for controlled data construction and evaluation, not as supervision during SSL pre-training.

We evaluate two complementary properties through controlled foreground-background compositions:

  • Nuisance invariance: We preserve the same bird foreground while changing its background and measure whether the learned relation remains compatible.
  • Semantic sensitivity: We change the bird category while preserving the background and measure whether the learned relation is correctly rejected.

The additional results show:

EvaluationBaseline SSLFull GeSSL
Worst-group Accuracy (%)47.866.7
Accuracy under reversed correlation (%)44.564.2
Nuisance-invariance relation acceptance (%)60.474.6
Semantic-sensitivity relation rejection (%)67.878.2

The full GeSSL achieves consistent improvements across these evaluations. In particular, the improvements in nuisance-invariance acceptance and semantic-sensitivity rejection demonstrate that GeSSL does not merely improve downstream accuracy, but also promotes the two complementary behaviors that motivate our counterfactual formulation.

We additionally compare necessity-only, sufficiency-only, and full GeSSL variants. The results show that necessity primarily improves the discrimination of semantically incompatible relations, while sufficiency improves relation stability under background variations. Combining both yields the strongest overall performance, supporting their complementary roles in reliable relation learning.

These controlled evaluations provide more direct evidence for the proposed counterfactual-inspired mechanisms than conventional downstream benchmarks alone. Nevertheless, we emphasize that interventions on observed image backgrounds do not establish identification of the underlying latent causal factors.

AuthorsOpenReview ↗

Response to W1: Part II

3. Distinguishing GeSSL from generic proxy-based regularization.

We further compare GeSSL with fixed-threshold similarity regularization, adaptive relation learning without bi-level optimization, and bi-level optimization without the discriminative relation loss.

Under matched training and evaluation settings, the full GeSSL outperforms these alternatives:

MethodWorst-group Acc. (%)Nuisance Invariance (%)Semantic Sensitivity (%)
Baseline SimCLR47.860.467.8
Fixed Similarity Regularization50.963.570.4
Adaptive Relation Only54.664.973.1
Bi-level Only55.871.268.9
Full GeSSL66.774.678.2

These results indicate that the observed improvements cannot be explained solely by introducing an additional similarity loss or using a generic bi-level optimization procedure. Rather, they support the benefit of jointly optimizing discriminative relation learning and cross-view consistency under the two proposed reliability principles.

This observation is also consistent with our original ablation results (Table 25). On ImageNet-100, SimCLR improves from 70.15% to 71.24% with the necessity proxy alone, to 71.58% with set-level sufficiency alone, and to 72.96% with the full GeSSL. Similar trends are observed with MAE. Furthermore, Table 23 reports improved accepted-relation purity, rejection accuracy, and relation AUROC, supporting improved relation discrimination.

4. Clarifying the theoretical contribution and Assumption 4.2.

We agree that our theoretical analysis does not establish formal counterfactual certification. Theorem 4.1 provides a generalization bound for the observable analytical GeSSL objective across self-supervised episodes drawn from the same distribution, rather than proving recovery of genuine counterfactual relations.

Moreover, Assumption 4.2 explicitly introduces the task-dependent alignment condition connecting the observable objective to downstream semantic risk:

E[Ltar]≤αE[J]+ϵproxy.\mathbb{E}[L_{\mathrm{tar}}] \leq \alpha\mathbb{E}[J] + \epsilon_{\mathrm{proxy}}.

We acknowledge that this assumption is not derived from unlabeled observations, and that the resulting bound is informative only when the proxy mismatch ϵproxy\epsilon_{\mathrm{proxy}} is sufficiently small. Our current theory does not guarantee this condition.

The additional controlled experiments provide empirical evidence that the learned relations exhibit the intended semantic sensitivity and nuisance invariance in the evaluated settings, thereby strengthening the empirical justification for the proposed proxy–semantic connection. However, these results do not constitute a mathematical proof of Assumption 4.2 or establish unconditional counterfactual guarantees.

We will revise the manuscript to explicitly distinguish between counterfactual reliability principles, their observable optimization proxies, and the theoretical guarantees established for these proxies. In particular, we will clarify that relation certification refers to an operational, proxy-based assessment of relation reliability, rather than formal certification of latent causal relations.

In summary, our contribution is threefold:

  1. Introducing complementary counterfactual-inspired principles that specify when learned relations should change or remain stable;
  2. Translating these principles into a unified, trainable framework for discriminative and generative SSL; and
  3. Demonstrating, through controlled semantic and nuisance variations, that the resulting representations exhibit improved semantic sensitivity, nuisance invariance, and resistance to spurious correlations.

We believe these findings substantiate the practical relevance of the proposed counterfactual perspective while respecting the limitations of proxy-based learning and the conditional scope of our theoretical analysis.

AuthorsOpenReview ↗

Response to W2, Q1, and Q2

Response to W2, Q1, and Q2

We thank the reviewer for the thorough feedback, but there may be some misunderstandings. We are pleased to provide clarifications on the table contents, algorithmic design, and our reproducibility package.

1. Clarification Regarding Table 1 and Entry Verification

We carefully examined Table 1 in the manuscript (Page 7) following the reviewer's comment. We would like to gently clarify that Table 1 contains only a single row for DINO + GeSSL. The entries under the GeSSL enhancement block are: SimCLR + GeSSL, MoCo + GeSSL, SimSiam + GeSSL, Barlow Twins + GeSSL, MAE + GeSSL, DINO + GeSSL, and VICRegL + GeSSL. There are no duplicated rows or contradictory numbers for DINO in Table 1. We suspect the reviewer might have inadvertently compared the vanilla DINO baseline in the upper section with the enhanced DINO + GeSSL in the lower section.

2. Open-Source Code and Dataset Documentation

To ensure full reproducibility:

  • An anonymized code repository containing main training scripts, environment configurations, boundary network architectures, and evaluation protocols has been prepared and will be made publicly available upon publication (Link: https://anonymous.4open.science/r/gessl-798C/).
  • Detailed step-by-step documentation, including hyperparameter configurations for all benchmarks (ResNet, ViT, C4), synthetic Waterbirds split generation scripts, and downstream evaluation protocols, will be bundled in the open-source repository.

3. Formulation of Positive and Negative Relation Certification and Training

We would like to clarify that in GeSSL, positive and negative relations are not hand-crafted or rigidly assigned using discrete heuristics; instead, they are dynamically certified via an adaptive compatibility boundary learned in representation space:

  • For anchor xiancx_{i}^{\mathrm{anc}} and candidate sample xj1x_{j}^{1}, the representation distance is dji=d(f(xj1),f(xianc))d_{j}^{i} = d(f(x_{j}^{1}), f(x_{i}^{\mathrm{anc}})) (cosine distance).
  • The boundary network gg predicts an anchor-conditioned threshold aia_i using the first- and second-order statistics (mean and covariance) of relative representations across the mini-batch.
  • Continuous soft assignments are generated via wji=Sigmoid(k(ai−dji))w_{j}^{i} = \mathrm{Sigmoid}(k(a_i - d_{j}^{i})). Candidate relations with dji≤aid_{j}^{i} \leq a_i (wji>0.5w_{j}^{i} > 0.5) act as compatible pairs and are pulled together, while those with dji>aid_{j}^{i} > a_i (wji<0.5w_{j}^{i} < 0.5) act as incompatible pairs and are pushed apart.
  • The boundary learner gg is optimized end-to-end via support-query bi-level optimization (Eqs. 4–5): it is updated not to minimize an isolated boundary regression objective, but according to whether its induced representation update improves consistency on unobserved query relation sets.

Furthermore, to facilitate direct understanding and seamless reproduction, a self-contained, PyTorch-style pseudocode implementation along with full modular scripts, including data samplers, the boundary network forward pass, and the bi-level training loop, is provided in the root directory of the aforementioned anonymous repository.

Reviewer_3LpYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the reliability of self-supervised learning (SSL), focusing on the problem that models may learn spurious correlations instead of meaningful semantic information. The authors introduce two counterfactual principles, necessity and sufficiency, which require learned relations to be sensitive to semantic changes while remaining stable under nuisance variations. Based on these principles, they propose GeSSL, a framework that combines an adaptive discriminative relation loss with consistency constraints at the sample, support-query set, and task-distribution levels. The method uses learned representation distances to determine relation compatibility and bi-level optimization to encourage consistency across different augmented views and mini-batches. The authors provide a theoretical analysis of generalization on unseen self-supervised tasks and evaluate GeSSL across multiple discriminative and generative SSL methods. Experiments on image classification, transfer learning, few-shot learning, and robustness benchmarks show improvements over the corresponding baselines.

Critical strengths
  1. The paper addresses an important problem in SSL. Learned representations may rely on spurious correlations even when the self-supervised training objective is successfully optimized.

  2. The proposed framework can be incorporated into several existing SSL methods, including discriminative and generative approaches.

  3. The experiments cover a broad range of datasets and downstream tasks, with generally consistent performance improvements. The additional robustness experiments and ablation studies are also useful.

Critical weaknesses
  1. The central methodological assumption is not sufficiently justified. The proposed relation certification relies on distances between learned representations to identify compatible and incompatible samples. However, these representations may already be influenced by nuisance factors and spurious correlations. The adaptive relation boundary is learned from these same representations, which creates a potential circularity. I do not see a convincing explanation of how this process can identify meaningful semantic relations or correct existing spurious dependencies.

  2. The connection to counterfactual reasoning is somewhat weak. The method does not observe or explicitly intervene on semantic and nuisance factors. The proposed objectives encourage discriminability and consistency, but these properties do not necessarily imply counterfactual necessity or sufficiency. The theoretical analysis provides a standard generalization bound for the observable SSL objective, while the connection to downstream semantic performance depends on an additional assumption.

  3. The experiments do not sufficiently validate the proposed mechanism. Although GeSSL improves downstream performance and robustness to image corruptions, these results do not directly establish that the method learns to distinguish semantic factors from nuisance factors. Controlled experiments with known spurious correlations would provide stronger evidence. Comparisons with simpler similarity-based regularization and bi-level optimization methods are also needed to assess the specific contribution of the proposed relation certification.

Questions for authors
  1. How does the proposed relation certification distinguish semantic information from nuisance information when both may be encoded in the learned representations? If the initial representations rely heavily on spurious correlations, what prevents the adaptive boundary from reinforcing these dependencies?

  2. Can the authors provide controlled experiments where semantic and nuisance factors can be manipulated separately? For example, varying background-label correlations between training and testing would help establish whether GeSSL learns representations that are less dependent on spurious information.

  3. Can the authors clarify what the theoretical analysis establishes about counterfactual reliability? In particular, how can Assumption 4.2 be justified when the relationship between the SSL objective and the underlying semantic factors is unknown?

11 comments
AuthorsOpenReview ↗

Response to W1 and Q1: Part I

Response to W1 and Q1

We thank the reviewer for this important concern. The potential circularity identified by the reviewer is precisely the problem motivating our work: a self-supervised model may learn valid relations by exploiting nuisance factors or spurious correlations rather than task-relevant semantics. GeSSL addresses this issue through two complementary reliability principles and their coupled optimization.

1. How do necessity and sufficiency encourage task-relevant representations rather than spurious dependencies?

Our central insight is that task-relevant representations should preserve semantic distinctions while remaining stable under nuisance variations. Counterfactual necessity and sufficiency capture these two complementary requirements, addressing different ways in which SSL models may rely on spurious information.

Counterfactual necessity: preserving semantic discriminability. Conventional SSL objectives learn representations from constructed supervision, such as aligning augmented views, contrasting instances, or reconstructing masked observations. However, successfully optimizing these objectives does not necessarily imply that the learned similarities reflect semantic compatibility. For example, two images depicting different bird species may have highly similar backgrounds. A model relying primarily on background information could incorrectly consider them semantically compatible, despite their different task-relevant content.

GeSSL introduces the discriminative relation loss Ldisc\mathcal{L}_{disc} to address this limitation. Rather than relying exclusively on the original self-supervised pairs, Ldisc\mathcal{L}_{disc} considers candidate relations between each anchor and other observations. An adaptive, anchor-specific boundary determines whether a candidate relation is encouraged to become closer or more separated in representation space. This provides an additional discriminative constraint on the learned relation structure. The intended effect is to prevent superficial similarity alone from being sufficient to preserve a relation when more informative distinctions can be learned.

Nevertheless, discriminability alone does not guarantee semantic relevance. A model can also distinguish samples using nuisance attributes, such as backgrounds, textures, or illumination. Moreover, the adaptive boundary itself is estimated from learned representations and may inherit their initial biases. This motivates the complementary sufficiency requirement.

Counterfactual sufficiency: suppressing dependence on nuisance variations. Consider the reverse situation: the same bird is observed against different backgrounds. Although its appearance changes, its task-relevant identity remains unchanged. A representation that relies heavily on background information may assign substantially different relations to these observations. Counterfactual sufficiency requires such relations to remain stable when semantic information is preserved.

GeSSL approximates this requirement at three levels. At the sample level, the base SSL objective encourages consistency across augmented observations. At the set level, support-query optimization evaluates whether a representation update learned from one augmented relation set remains effective on another. At the distribution level, the encoder and relation-boundary learner are shared across multiple mini-batch environments, discouraging solutions specialized to individual batches. When nuisance cues vary across these observations, relying on them becomes less effective at satisfying the consistency requirements.

Why are both principles necessary? Necessity without sufficiency may produce discriminative representations that still exploit nuisance differences. Sufficiency without necessity may preserve stable relations without ensuring that semantically different observations are distinguishable. Their combination imposes a complementary learning pressure: representations must retain informative distinctions while remaining stable under non-essential variations. Consequently, features that are both discriminative and stable across the observed perturbations are favored over unstable spurious cues.

Importantly, these principles define the desired learning behavior; they do not assume that semantic factors are directly observed. The remaining challenge is how to optimize the adaptive relation boundary without merely reinforcing the potentially biased representations from which it is estimated. This is where GeSSL's bi-level optimization becomes important.

AuthorsOpenReview ↗

Response to W1 and Q1: Part II

2. Why does the adaptive boundary not merely reinforce existing biases?

We agree that the boundary is estimated from the current representations and may initially inherit their biases. The crucial distinction is that GeSSL does not optimize this boundary solely according to current representation similarities.

In the inner loop (Eq. 4), Lssl\mathcal{L}_{ssl} and Ldisc\mathcal{L}_{disc} jointly update the encoder to obtain an adapted model f′f'. In the outer loop (Eq. 5), the SSL loss on separately augmented query relations evaluates this update and further optimizes both the encoder ff and boundary learner gg.

These two optimization stages serve distinct purposes. The inner loop learns a representation using the current relation estimates, whereas the outer loop evaluates whether the resulting update remains useful on observations not used for that update. Because f′f' depends on gg, the query loss also provides a gradient signal to the boundary learner.

Thus, the boundary is not learned merely to reproduce existing similarity assignments; it is optimized according to how its induced representation update affects the SSL objective on query observations. An inaccurate boundary that produces support-specific improvements but fails to improve query performance can therefore be discouraged.

This mechanism couples relation discrimination with cross-view validation, providing an additional constraint against reinforcing unreliable relations. It does not, however, guarantee that the learned boundary corresponds to true semantic categories, particularly when spurious correlations remain stable across support and query observations.

3. What evidence supports this mechanism?

The ablations in Table 25 demonstrate the complementary contributions of necessity and sufficiency. On ImageNet-100, SimCLR achieves 70.15% Top-1 accuracy, compared with 71.24% using necessity alone, 71.58% using set-level sufficiency alone, 72.31% when combining both, and 72.96% with full GeSSL.

Furthermore, Table 23 shows improved relation discrimination: SimCLR's relation AUROC increases from 72.4% to 78.6% with GeSSL. Figure 8 illustrates improved feature separability with Ldisc\mathcal{L}_{disc}, while Figure 7 provides qualitative evidence that full GeSSL focuses more strongly on foreground objects.

These results support improvements in relation quality and representation discriminability, although they do not directly establish that the model has eliminated background-based dependencies.

Importantly, we do not claim that GeSSL guarantees the elimination of all spurious correlations. Nuisance factors that remain stable across observed augmentations and environments may still influence the learned representations. Rather, GeSSL provides complementary optimization constraints that mitigate this risk without requiring direct access to latent semantic or nuisance factors.

In summary, GeSSL addresses potential circularity not by assuming that its initial relation estimates are correct, but by combining discriminative relation learning with stability constraints and query-based validation. Necessity encourages informative distinctions, sufficiency discourages reliance on varying nuisance cues, and bi-level optimization constrains how the adaptive boundary influences representation learning. Their joint effect encourages more reliable task-relevant representations without claiming unconditional semantic identification.

AuthorsOpenReview ↗

Response to W2: Part I

Response to W2

We thank the reviewer for this important comment. We agree that discriminability and consistency do not, by themselves, imply counterfactual necessity or sufficiency. We would like to clarify that our framework distinguishes between (i) counterfactual principles that characterize desired model behavior, (ii) observable proxies that approximate these principles, and (iii) theoretical guarantees for the observable objectives. In particular, GeSSL does not assume access to latent semantic or nuisance factors, nor does it claim to identify genuine counterfactual effects.

1. How do the proposed proxies approximate the regulation of semantic and nuisance factors?

Our central motivation is that conventional SSL objectives may be satisfied through either meaningful semantic information or spurious nuisance correlations. The proposed counterfactual principles specify two complementary behavioral requirements: learned relations should be sensitive to changes in task-relevant semantic factors (necessity) while remaining stable under nuisance variations that preserve semantic content (sufficiency).

Although the underlying semantic factors ZcZ_c and nuisance factors ZsZ_s are unobserved, GeSSL indirectly regulates how learned representations respond to their variations through three mechanisms:

  • Approximating nuisance invariance through augmented observations: Under the standard SSL assumption that augmentations preserve the main semantic content while perturbing some nuisance attributes, different augmented views of the same sample approximate observations with shared ZcZ_c but varying ZsZ_s. The sample-level SSL objective encourages representation consistency across these views, thereby reducing sensitivity to the nuisance variations covered by the augmentations.
  • Approximating semantic sensitivity through discriminative relation learning: The adaptive relation loss LdiscL_{\mathrm{disc}} encourages the model to distinguish compatible from incompatible relations by learning an anchor-specific compatibility boundary. Unlike indiscriminate alignment, this objective discourages preserving relations whose representation-based compatibility is insufficient. This provides an observable approximation to necessity by encouraging sensitivity to informative differences. Importantly, the estimated compatibility is not assumed to correspond perfectly to true semantic compatibility.
  • Jointly constraining discriminability and stability through bi-level optimization: The relation boundary influences the representation update on the support set, while the outer-loop objective evaluates the resulting representation on separately augmented query relations. Consequently, the boundary learner is optimized not merely to fit current representation distances, but also according to whether its induced updates remain effective across different observations. Sharing the same parameters across multiple mini-batch relation environments further encourages transferable relation-learning behavior.

Therefore, GeSSL does not directly intervene on ZcZ_c or ZsZ_s; rather, it approximately regulates the representation's sensitivity to informative changes and its stability under observed nuisance perturbations. The two principles are complementary: necessity discourages indiscriminate relation preservation, while sufficiency constrains the stability of the resulting dependencies.

We acknowledge that this approximation depends on the informativeness of the observable variations. In particular, if nuisance correlations remain stable across augmented views and sampled relation environments, GeSSL cannot guarantee their elimination. This is a limitation of the proposed proxies, rather than a claim of exact counterfactual identification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.