- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies unreliable relations in self supervised learning and proposes GeSSL based on counterfactual necessity and sufficiency.
GeSSL combines adaptive relation discrimination with consistency across samples, sets, and training batches.
Critical strengths
GeSSL improves a wide range of self supervised baselines across classification, transfer learning, few shot learning, and robustness evaluations.
The ablation and robustness results show consistent contributions from the proposed relation and consistency objectives.
Critical weaknesses
The algorithmic contribution is incremental because GeSSL mainly combines standard SSL consistency, support query bilevel optimization, and relation filtering under a counterfactual interpretation, without introducing a clearly new learning mechanism.
The theoretical analysis does not establish counterfactual reliability or generalization to new task distributions, since the main bound only considers iid episodes from the same distribution and downstream control requires an additional alignment assumption.
Questions for authors
Can the authors clarify whether the baseline and GeSSL use the same number of training samples, augmented views, and optimization updates?
9 comments
Response to W1: Part II
3. Existing evidence for the complementary contributions.
Our original ablations provide evidence that the proposed necessity and sufficiency proxies contribute independently and complementarily.
On ImageNet-100 (Table 25), SimCLR achieves 70.15% Top-1 accuracy. Introducing necessity alone improves accuracy to 71.24%, while set-level sufficiency alone achieves 71.58%. Combining necessity and set-level sufficiency further improves accuracy to 72.31%, and the complete GeSSL reaches 72.96%.
Similar improvements are observed with MAE, where accuracy increases from 76.56% to 78.45% with the full framework.
Furthermore, Figure 11 compares our bi-level implementation with joint single-stage optimization and independent mini-batch adaptation, supporting the effectiveness of the proposed coupled optimization strategy.
However, we agree that these ablations primarily validate our specific implementation and do not, by themselves, establish the broader applicability of the proposed principles. We therefore conduct an additional study that explicitly separates the framework from its particular realization.
4. New experiment: Principle–Implementation Decoupling Study.
To directly examine whether the proposed framework is more general than the specific GeSSL implementation, we introduce a new experiment in which the necessity and sufficiency principles are preserved while their underlying optimization mechanisms are replaced.
For counterfactual necessity, we consider two distinct implementations:
- Necessity-A (Original GeSSL): Adaptive relation discrimination using an anchor-specific compatibility boundary.
- Necessity-B (Alternative): Unsupervised mutual-neighbor relation ranking with a margin-based discriminative objective, without the adaptive boundary learner.
For counterfactual sufficiency, we similarly consider:
- Sufficiency-A (Original GeSSL): Support-query bi-level optimization with multi-batch relation environments.
- Sufficiency-B (Alternative): Cross-view relational distribution consistency, which directly aligns neighborhood relation distributions across augmented observations without bi-level optimization.
Both alternative implementations use unlabeled observations and do not require semantic annotations during SSL pre-training.
We evaluate the individual implementations and their combinations under matched training protocols, including the original GeSSL (A+A), two partially substituted variants (A+B and B+A), and a fully alternative implementation (B+B).
The results on controlled Waterbirds experiments are summarized below:
| Configuration | Worst-group Acc. (%) | Semantic Sensitivity (%) | Nuisance Invariance (%) |
|---|---|---|---|
| Baseline SSL | 47.8 | 67.8 | 60.4 |
| Necessity-A only | 53.2 | 75.3 | 62.1 |
| Necessity-B only | 52.3 | 73.9 | 61.6 |
| Sufficiency-A only | 56.4 | 69.5 | 72.8 |
| Sufficiency-B only | 55.1 | 68.7 | 71.3 |
| A+A (Original GeSSL) | 66.7 | 78.2 | 74.6 |
| A+B | 64.5 | 77.4 | 73.1 |
| B+A | 63.8 | 75.8 | 74.1 |
| B+B (Fully Alternative) | 62.3 | 74.8 | 72.2 |
The results reveal three important observations:
- Different implementations of the necessity and sufficiency proxies independently improve their corresponding reliability measures, suggesting that the two principles are not restricted to a particular loss function.
- Combining necessity and sufficiency consistently achieves more balanced semantic sensitivity and nuisance invariance than optimizing either requirement alone. This pattern holds across different combinations of implementation mechanisms.
- Importantly, the fully alternative implementation (B+B), which does not use GeSSL's original adaptive boundary or support-query bi-level optimization, still achieves substantial improvements over the baseline. This provides direct empirical evidence that the proposed framework can guide the construction of effective learning algorithms beyond its original GeSSL realization.
We further repeat this evaluation using both discriminative and generative SSL backbones, represented by SimCLR and MAE. The corresponding Top-1 improvements of the fully alternative implementation over their respective baselines are 1.9 and 1.4 percentage points, respectively.
These results support the portability of the proposed necessity-sufficiency framework across different optimization mechanisms and SSL paradigms.
Response to W1: Part III
5. Distinguishing the framework contribution from implementation-specific improvements.
To further isolate the benefit of GeSSL's specific optimization mechanism, we compare the original method with fixed-threshold similarity regularization, adaptive relation learning without bi-level optimization, and bi-level optimization without the discriminative relation loss.
Under matched training settings, Full GeSSL achieves a worst-group accuracy of 66.7%, compared with 50.9%, 54.6%, and 55.8% for these alternatives, respectively.
These comparisons support the effectiveness of the original coupled learning mechanism. Meanwhile, the Principle–Implementation Decoupling Study demonstrates that the broader reliability framework does not depend exclusively on that mechanism.
We therefore distinguish two complementary contributions:
- Framework contribution: A counterfactual-inspired formulation of reliable self-supervised relation learning through semantic sensitivity and nuisance invariance, which can guide multiple algorithmic realizations.
- Algorithmic contribution: GeSSL, a specific realization that couples adaptive relation discrimination with cross-view validation through bi-level optimization.
We emphasize that the framework provides behavioral design principles rather than an unconditional guarantee that every possible implementation will improve performance. Likewise, the observed improvements under controlled semantic and background variations do not imply formal causal identification.
In summary, we agree that the individual optimization ingredients of GeSSL are established. However, the contribution of our work extends beyond their combination. By explicitly distinguishing the necessity-sufficiency framework from its particular implementation and demonstrating consistent improvements across alternative realizations, we provide evidence that the proposed principles have broader methodological value for reliable self-supervised learning. We will revise the manuscript to make this framework-level contribution and its distinction from the specific GeSSL algorithm clearer.
Response to W1: Part I
Response to W1
We thank the reviewer for this important comment. We agree that SSL consistency, relation discrimination, and support-query bi-level optimization are individually established techniques. However, we would like to emphasize that the central contribution of our work is not a particular regularization loss or optimization algorithm, but a general framework for reliable self-supervised relation learning, guided by two complementary counterfactual-inspired principles: necessity and sufficiency. GeSSL represents one concrete realization of this framework, rather than the only possible implementation.
To clarify the contribution, we distinguish between (i) the framework-level formulation, (ii) the specific learning mechanism implemented in GeSSL, and (iii) empirical evidence that the proposed principles can be realized through different optimization methods.
1. Framework-level contribution: Two complementary requirements for reliable relation learning.
As discussed in Section 2, conventional SSL objectives may be satisfied through spurious correlations even when the constructed self-supervised relations are valid. Our central question is therefore not simply how to improve discriminability or invariance, but how to characterize whether the learned relations depend on task-relevant semantic information.
We introduce two complementary behavioral principles:
These principles provide a unified framework for evaluating and improving relation reliability. Importantly, the principles do not prescribe a specific loss function, relation boundary, or optimization algorithm. Different implementations can approximate the same behavioral requirements through different observable objectives.
Our contribution is to formulate these complementary requirements at the level of self-supervised relations and demonstrate how they can guide the design of reliable representation learning algorithms across different SSL paradigms.
2. Algorithmic contribution: Coupling adaptive relation discrimination with cross-view validation.
Within this framework, GeSSL provides a concrete learning mechanism rather than a straightforward addition of independent regularizers.
Specifically, the necessity proxy learns an anchor-specific compatibility boundary through the discriminative relation loss. The sufficiency proxies evaluate stability across augmented observations, support-query relation sets, and sampled mini-batch environments.
Crucially, these components are coupled through the bi-level optimization in Eqs. (4)–(5). The support-set objective updates the representation encoder using both the base SSL loss and the discriminative relation loss. The resulting encoder is then evaluated on separately augmented query relations.
Because the support-induced encoder update depends on the relation-boundary parameters, the outer-loop query loss also updates the boundary learner. Consequently, the learned relation boundary is optimized according to the effectiveness of its induced representation updates on query observations, rather than solely according to current representation distances.
This interaction distinguishes the complete GeSSL mechanism from independently combining similarity filtering and consistency regularization. Nevertheless, we do not claim that the underlying bi-level optimization technique itself is novel.