acceptodds
Withdrawn from ICLR 2027

Can A Model Learn From Arbitrary Pairs?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive learning based on global embeddings has traditionally followed a simple rule (behind most supervised and self-supervised methods): semantically similar or identical samples should be pulled together, while distinct samples should be pushed apart. Despite its success, this learning principle does not use a pair of distinct classes as a positive. A snake and a cable can share an elongated outline, and a dog and a cow can share fur and a four-legged shape, but depicting that attribute is not trivial. This raises the question: can a model learn from arbitrary class pairs without explicit guidance or attribute annotations? The challenge is to pull a distinct pair together while preserving class discrimination. We address it with a discriminative subspace for every pair of classes, using a learning principle called SimLAP (Simple framework for Representation Learning from Arbitrary Pairs). SimLAP introduces a feature filter that activates and deactivates pair-specific feature dimensions. An arbitrary class pair conditions a shared feature gate for all samples. The gate weights coordinates so that the contrastive loss imposes similarity for that pair. The positive is another view of the partner class, which is a random sample in the batch, so it may be the same class or a distinct class. With one shared gate, this objective matches a SupCon trained with the same recipe on ImageNet (75.8 vs. 76.1 linear probe) and CIFAR-100 (76.3 vs. 72.1 kNN). The gate is what makes that training succeed: without it, distinct and arbitrary positives collapse. Arbitrary pairs can provide useful representation learning signals for cross-class features, but their value depends on the semantic structure of the dataset. SimLAP makes such pairs trainable, providing a robust framework to exploit them.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
SimLAP, Representation Learning, Supervised Learning, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Q3qVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes to learn visual representations by associating arbitrary images in a self-supervised fashion. The authors leverage label information to learn to mask image representations, before aligning unmasked portions of two images. The results show that the proposed model matches the supervised contrastive learning.

Critical strengths

The idea and intuition are well-motivated, novel and interesting. The contribution is well discussed relative to prior work. The experimental protocol is well-designed in general.

Critical weaknesses

The main limitation is the absence of clear utility: in general, the performance matches that of SupCon, which is much simpler to implement. The advantages of the method relative to SupCon must be clarified. Furthermore, the masking strategies are not sufficiently well characterized. It would help to see if the model extracts non-null subspaces when the classes differ. Otherwise, SimLAP collapses into SupCon, which could explain the performance match.

Some parts of the paper are hard to follow. For instance: Finding 1, or the concept of "sign filter" in Section 3.1. Furthermore, the objective of Section 6 is unclear, but raises one question: is the masking strategy applicable in diverse SSL methods ?

Questions for authors

Please see weaknesses.

2 comments

We thank the reviewer for the valuable comments.

W1: the performance matches that of SupCon.

We agree that SupCon is simpler, and we do not claim that SimLAP should replace it as a standalone objective on ImageNet. It is crucial to distinguish the main difference, that is, the learning scope of SupCon is identical pairs, while SimLAP is arbitrary pair. Comparing to SupCon is not to demonstrate that SimLAP is an alternative method. The core message, overlooked in the community, is that the semantically distinct pairs are important, not negligible in representation learning.

SimLAP provides a simple and effective way to utilize these pairs. As shown in Table 7, SimLAP prefers the data whose classes have more similarity. There are modest performance gains via joint loss (Table 9) and mid-training (Table 10). SimLAP is ahead in Image retrieval by 4.5 mAP on ROxford5k and 4.9 on RParis6k (Table 15, Medium).

W2: Are the subspaces non-null when the classes differ?

Yes, SimLAP does not collapse into SupCon, as shown in Figure 6 gate visualization. Our method prevent null subspaces.If the subspaces are null, all representations will be collapsed to one point, so the contrastive loss would be a constant. However, our experiment with distinct setting against this that loss decreases normally. Distinct pairs even achieves high performance (76.15, the second best) in Table 7. On CIFAR-100, 83 of 256 gate values exceed 0.5 on average over the pairs seen in training.

W3: SimLAP may collapse into SupCon.

SimLAP will not collapse into SupCon. We demonstrate experiments with the distinct pairs only in Table 7 and Table 11. SimLAP still works with distinct pairs, while SupCon fails to learn meaningful representations. A random partner shares the anchor's class about once in 1,000 draws on ImageNet and once in 100 on CIFAR-100. The learning signals almost come from distinct pairs.

W4: Finding 1, or the concept of "sign filter" in Section 3.1.

Sign Filter is a diagnostic and not part of SimLAP. Sign Filter selects the subspace according to the sign of two feature vectors, while Feature Filter used in SimLAP is learnable to find optimal subspaces. For two feature vectors from two classes, Sign Filter keeps the dimensions on which both have the same sign, drops the rest. It needs no training. Applied to a standard supervised encoder, it shows that a subspace in which two distinct classes are similar already exists (Figure 1b).

W5: the objective of Section 6 is unclear, but raises one question: is the masking strategy applicable in diverse SSL methods.

Section 6 asks whether SimLAP is useful as an addition to existing training, not as a replacement. It tests two uses: a joint loss with MoCo v3 (Table 9), and a mid-training stage for pretrained model MAE, DINOv2 and MoCo (Table 10).

SimLAP is not an alternative of any existing method. We are seeking to establish a new learning framework for semantically distinct samples. We first show it is feasible and learns competitive representations in Section 5. We admit that learning from arbitrary pairs is not normal way of representation learning and could be inefficient on ImageNet (Table 11). SimLAP should be applied into the datasets having lots of coarse classes, like CIFAR100 (Table 7). Therefore, we seek to the applications of SimLAP in ImageNet as an auxiliary loss. These experiments show that SimLAP is a complementary to existing methods.

The value of this paper is not an incremental method achieving better number in a benchmark. Instead, we provide a robust learning framework to utilize the distinct pairs, even ridiculous pairs like snake-lamp.

I thank the authors for the response, which address many concerns that I had.

My concern about the utility of the method is not entirely solved. In many evaluations, SimLAP is competitive with or outperformed by SupCon (Tables 2 and 4). SimLAP as a complementary loss provides only a 0.19% improvement compared to SimCon. Furthermore, experiments are conducted on a limited number of epochs (100 epochs). In table 9, the authors do not report the classical linear probe accuracy. In Table 10, the evaluation is non-standard and improvements are small. More convincing advantages are shown in the image retrieval tasks in appendix and in Cifar100. However, in general, the results do not demonstrate a general advantage.

Thus, could the authors further characterize when and why SimLAP is useful in some tasks or not others ? In the previous response, the authors argue that SimLAP is useful for datasets tasks composed of many coarse classes. However, experiments on one dataset (CIFAR100) do not suffice to support this claim.

Nonetheless, it is interesting that the authors show the emergence of a representational structure (Figure 5, 7). Have the authors looked into how the model performs (compared to SupCon) in predicting superordinate categories (animals, etc.) and/or if the models learnt by SimLAP are more aligned with inter-class human similarity judgements [1] ? I will increase my score if the authors demonstrate that SimLAP outperforms SupCon in superordinate category recognition with the models pretrained on cifar100 and Imagenet-1k. I will further increase my score if the authors demonstrate the generality of the gate mechanism through a direct and successful implementation in other SSL methods, such as BYOL or DINO (BYOLLap, DINOLap ...).

[1] Hebart, M. N., Contier, O., Teichmann, L., Rockter, A. H., Zheng, C. Y., Kidder, A., ... & Baker, C. I. (2023). THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife, 12, e82580.

Reviewer_njJvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a framework fro contrasive representation learning called SimLAP, which uses learned class-pair-conditioned feature "gates" to bring arbitrarily paired object classes (eg, snake and lamp) closer together in embedding subspaces, while preserving discriminativeness across different classes.

Critical strengths

N/A.

Critical weaknesses

Outdated experimental baselines and unclear relevance to contemporary visual representation learning

The paper motivates and evaluates its contributions against contrastive learning methods developed several years ago (2017-2023), like SimCLR, SupCon, and ResNet-50-based representations. While these are contextually useful to ground the problem, the field has advanced substantially, and it's unclear if the limitations the authors raise are still significant enough for contemporary vision foundation models:

(L54-60): "These methods improve representation learning by expanding the set of positive pairs with hierarchical structure, attribute annotations, or similarity graphs. Explicitly specifying pairwise relations becomes increasingly difficult to scale as the number of classes grows, because the number of possible class pairs is quadratic. We raise a fundamental question: can a model learn from arbitrary pairs of classes without this explicit relational guidance? This raises two related questions: first, can arbitrary class pairs be used as positives without destroying class discrimination? Second, once such pairs are trainable, when do they provide useful signals for representation learning?"

Missing closely related prior work and limited novelty

The paper omits several directly relevant works on learning conditional similarity, category-dependent embedding subspaces, and discovering similarity conditions without explicit attribute supervision:

1. Learning Type-Aware Embeddings for Fashion Compatibility (ECCV 2018). https://arxiv.org/abs/1803.09196
2. Learning Similarity Conditions Without Explicit Supervision (ICCV 2019). https://arxiv.org/abs/1908.08589
3. Why Do These Match? Explaining the Behavior of Image Similarity Models (ECCV 2020). https://arxiv.org/abs/1905.10797

The method in (1) already motivates this question, and introduces category-pair-conditioned masking for learning compatibility in different embedding subspaces: a mechanism closely related to SimLAP's pair-conditioned "feature gate".

In (2), the authors learn latent similarity conditions without requiring explicit annotations of the attributes responsible for a match, which according to L142-150 is the primary limitation of prior work that motivated the development of SimLAP. This seems contradictory to the submission's novelty claims.

In (3), the authors investigate the visual evidence behind pairwise similarity judgments for contrastively-trained models, and is relevant to interpreting if the learned subspaces capture meaningful shared visual attributes.

The authors should discuss (1-3) alongside Conditional Similarity Networks (Veit et al., 2017), clarify the distinctions between their proposed method and the existing conditional embedding approaches, and provide experimental comparisons where appropriate.

Video evaluation is ill-posed

For video object segmentation on DAVIS-2017 (Table 5), the authors use frozen ResNet-50 image features to propagate segmentation masks between consecutive frames via nearest-neighbor matching, without training a video-specific model or temporal reasoning component. It's unclear to me why they evaluate a video task using a static image encoder rather than comparing against contemporary video representation models, especially since their approach doesn't explicitly model temporal information, and underperforms SimCLR and SupCon on this benchmark.

References

Vasileva et al. (2018), ECCV. Learning Type-Aware Embeddings for Fashion Compatibility. https://arxiv.org/abs/1803.09196

Tan et al. (2019), ICCV. Learning Similarity Conditions Without Explicit Supervision. https://arxiv.org/abs/1908.08589

Plummer et al. (2020), ECCV. Why Do These Match? Explaining the Behavior of Image Similarity Models. https://arxiv.org/abs/1905.10797

Questions for authors
  1. Why isn’t there a baseline comparison with an off-the-shelf frontier model? For example, DINOv3 and SigLIP 2 provide strong pretrained representations with transferable visual and semantic features. The paper doesn't investigate if such models already capture the cross-category similarities that SimLAP is supposed to address, or whether SimLAP offers any advantages over these representations. Although the authors include experiments with pretrained DINOv2, MAE, and MoCo backbones, the reported improvements from SimLAP mid-training are modest (+0.2-0.6pp). These results are not enough to establish the practical impact of the approach. I would encourage the authors to evaluate contemporary off-the-shelf vision encoders, ideally including DINOv3 and SigLIP 2, on the same downstream tasks and to directly test if their representations already show the cross-class similarity properties that SimLAP is trying to learn. Why were DINOv3 and more recent vision-language embedding models omitted from the evaluation?

  2. What practical setting benefits from SimLAP's arbitrary-pair training instead of using an existing pretrained encoder?

  3. Do contemporary pretrained vision encoders already show the cross-category similarities that motivate this work? What specific failure of those representations does SimLAP resolve?

  4. What evidence supports interpreting the learned subspaces as shared visual properties, beyond their effect on similarity and downstream accuracy?

References

Siméoni et al. (2025). DINOv3. arXiv:2508.10104. https://arxiv.org/abs/2508.10104

Tschannen et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv:2502.14786. https://arxiv.org/abs/2502.14786

2 comments

We thank you for the detailed review and for the pointers to this line of work.

Q1: Baselines and relevance to contemporary encoders.

The recent released models, e.g., DINOv3 or SigLIP 2, differs in data and model scale. DINOv3 is trained on 1.7B images with a 7B-parameter teacher, and SigLIP 2 on the 10B images of WebLI. That comparison would not isolate the objective contribution. We appreciate the mentioned models achieving SOTA performance. However, the performance gain owes to scaling law, not the learning objectives. The learning principles were established in DINOv1, iBot, and CLIP. We study learning principles in the paper and propose a novel training objective, thus, SimCLR and SupCon are the relevant counterparts which differ in pair construction. We use these out-dated baselines to make sure the comparison is fair with the same architecture, data and cost.

Q2: Why not using an existing pretrained encoder.

Indeed, we demonstrate one application of SimLAP in mid-training to use pretrained encoder which has +0.2 to +0.6 points gain. Our work is a study of representation learning, and using pretrained encoders is the application of the learned features. The main benefit of SimLAP is not an incremental performance gain. Instead, it is a fundamental contribution of learning mechanism. What SimLAP resolves is the challenging of contrastive learning on semantically distinct pairs.

Q3: Pretrained vision encoders already show the cross-category similarities. The cross-category similarities What specific failure of those representations does SimLAP resolve?

We agree that strong encoders probably already contain cross-class similarity in subspaces. We do not claim that pretrained encoders lack such similarity, or that SimLAP should replace an existing pretrained encoder. The paper explores a different way of learning from arbitrary pairs. The experiments empirically show no conflict to the existing methods. We claim SimLAP as an supplemental method as shown in Section 6.

Q4: evidence supports interpreting the learned subspaces.

What we do observe is the structure at the level of classes: gates and label embeddings are more alike inside a WordNet super-synset than across super-synsets. The method of Plummer et al. is a potential tool for testing the attribute interpretation, and we will try it in the revision.

W1: Outdated experimental baselines.

The learning principles of contrastive learning are mainly established during 2019-2021. The subsequential works solve the problems in scaling. It is unfair and impractical to compare recent baselines pretrained on much more data. Dinov2/v3, SegLIP2's study scope is how to scale contrastive learning on billions data. MoCo, DINO, iBoT's study scope is how to perform contrastive learning. While, our paper's study scope is which sample pairs can be applied in contrastive learning. The answer is any pair, which is never proposed before us.

W2: Related work and novelty。

The mentioned papers are subsequent work of CSN which we have cited. Our novelty does not owe to feature filter which generates proper gates to mask some dimensions. Our work mainly discusses how to expand the scope of positive pairs to dissimilar pairs. Therefore, their contributions will not diminish our novelty. None of them considered using distinct pairs, like lamp-snake.

Following your suggestion, we placed the gating mechanisms of Vasileva et al. (2018) and Tan et al. (2019) to train models (CIFAR-100, ResNet-18, 100 epochs, kNN with k=10). We add gate method:

  • table: one free mask per class pair, as in the type-pair masks of Vasileva et al.
  • sce-pair: M shared masks, weighted by a branch that reads the two image embeddings and computed for each comparison, as in Tan et al.
  • sce-anchor: the same masks, weighted once from the anchor and its partner and reused for every candidate.

kNN top-1 (%) on CIFAR-100. Identical and distinct columns use seed 0. The arbitrary column is the mean ± std over three seeds, except asym (seed 0).

GateConditioned onOne gate for all candidatesIdenticalDistinctArbitrary
asym (re-run)labelsno11.017.666.72
sym, SimLAP (re-run)labelsyes63.4865.1965.39 ± 0.33
tablelabelsyes65.9564.1264.57 ± 0.47
sce-pair, M=5imagesno65.107.329.44 ± 0.78
sce-anchor, M=5imagesyes65.5328.9030.31 ± 0.93

Experimental comparison: • A free mask per class pair, in the style of Vasileva et al., matches our gate on arbitrary pairs (64.57 ± 0.47 vs. 65.39 ± 0.33). • The image-conditioned gate of Tan et al. trains on identical pairs (65.1–65.5%) but reaches 9.4% and 30.3% on arbitrary pairs in its two forms.

The experiments reveal the learning from arbitrary pairs is a robust mechanism which can be implemented in several ways. Although Vasileva et al. proposed a similar method, they did not expand this method to distinct pairs. Their method is still limited in constructing pairs via outfit.

W3: Video evaluation is ill-posed.

The DAVIS video evaluation is a standard benchmark to evaluate the quality of local features, which is applied in DINO. It is a fast alternative of object segmentation which relies on spacial features only. It assesses the invariance under posture change.

Reviewer_9TStOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new contrastive learning setup where a model learns from arbitrary class pairs without heuristics like self-consistency or attribute annotations. The proposed approach, SimLAP, applies a learned gating mechanism on the feature representations of an arbitrary pair, and minimizes the distance in the activated subspace. Experiments on ImageNet with ResNet-50 show the proposed approach has comparable performance with SupCon on classification, knn-probe transfer learning, and video object segmentation. The paper also carries out ablation to show training from arbitrary pair outperforms pairs with identical class labels.

Critical strengths
  1. Self-supervised representation learning is an important area and the paper attempts to propose a novel learning paradigm that removes the need for self-consistency or label/attribute information.
  2. The proposed approach is clear and easy to understand. The paper evaluates SimLAP under extensive experiment setups: classification, learner probe, transfer learning, class similarity graph, and video object segmentation. The paper also carries out ablation study to demonstrate effective representation and subspace learning.
Critical weaknesses
  1. My main concern is that it remains unclear why the proposed gating and subspace learning mechanism would work on examples from different class labels. Common representation learning methods are built upon heuristics that are easy to understand, e.g., invariance to data augmentation. In the proposed SimLAP, it is unclear why making the subspace of arbitrary class pairs similar would help learning. Also, the gating is happening at the final representation learning layer, but some class pairs may only share low level features and may not make sense to maximize similarity in the final representation layers.
  2. My second concern is on joint-training of the gating and representation learning. Gating functions in the form of sigmoid might be difficult to optimize due to joint-learning with the representation subspace. It is unclear how gating sparsity and training stability is managed in this setup. Furthermore, the gating in this approach is not a simple selector, but also modulates the magnitude of the activations, which might further modulate distance measures from the original ungated representation.
Questions for authors
  1. Why maximizing the similarity between different classes at the final representation layer is desirable? What if classes only share low level feature similarities?
  2. What is the sparsity level of gating activations and is training stable?
  3. How dataset dependent is SimLAP in terms of class relations? Training data in ImageNet naturally contain many visually related class labels. What if the training data is sampled in a way that maximizes visual feature distance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.