On the Counterfactual Relation Certification of Self-Supervised Learning

Self-supervised learning (SSL) learns representations by constructing supervision from unlabeled data. However, satisfying self-supervised objectives does not guarantee reliable representations, since the same training relations may be explained by semantic factors or by nuisance cues such as background. Consequently, an SSL model may achieve low training loss while relying on spurious dependencies that fail under distribution, task, or semantic shifts. We address this issue from a counterfactual perspective and characterize model reliability through two complementary principles: the learned dependency should be revised when essential semantic factors change, while remaining stable when semantic factors are preserved under nuisance variations. Since true counterfactual interventions are unavailable in SSL, we propose General SSL (GeSSL), which approximates these principles through observable training proxies. GeSSL models counterfactual necessity with a discriminative relation loss and counterfactual sufficiency through consistency at the sample, support-query set, and task-distribution levels. Theoretical analysis establishes bounded generalization error on novel tasks. Extensive experiments show that GeSSL consistently improves representative discriminative and generative SSL methods across multiple evaluation protocols.