- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes to learn visual representations by associating arbitrary images in a self-supervised fashion. The authors leverage label information to learn to mask image representations, before aligning unmasked portions of two images. The results show that the proposed model matches the supervised contrastive learning.
Critical strengths
The idea and intuition are well-motivated, novel and interesting. The contribution is well discussed relative to prior work. The experimental protocol is well-designed in general.
Critical weaknesses
The main limitation is the absence of clear utility: in general, the performance matches that of SupCon, which is much simpler to implement. The advantages of the method relative to SupCon must be clarified. Furthermore, the masking strategies are not sufficiently well characterized. It would help to see if the model extracts non-null subspaces when the classes differ. Otherwise, SimLAP collapses into SupCon, which could explain the performance match.
Some parts of the paper are hard to follow. For instance: Finding 1, or the concept of "sign filter" in Section 3.1. Furthermore, the objective of Section 6 is unclear, but raises one question: is the masking strategy applicable in diverse SSL methods ?
Questions for authors
Please see weaknesses.
2 comments
I thank the authors for the response, which address many concerns that I had.
My concern about the utility of the method is not entirely solved. In many evaluations, SimLAP is competitive with or outperformed by SupCon (Tables 2 and 4). SimLAP as a complementary loss provides only a 0.19% improvement compared to SimCon. Furthermore, experiments are conducted on a limited number of epochs (100 epochs). In table 9, the authors do not report the classical linear probe accuracy. In Table 10, the evaluation is non-standard and improvements are small. More convincing advantages are shown in the image retrieval tasks in appendix and in Cifar100. However, in general, the results do not demonstrate a general advantage.
Thus, could the authors further characterize when and why SimLAP is useful in some tasks or not others ? In the previous response, the authors argue that SimLAP is useful for datasets tasks composed of many coarse classes. However, experiments on one dataset (CIFAR100) do not suffice to support this claim.
Nonetheless, it is interesting that the authors show the emergence of a representational structure (Figure 5, 7). Have the authors looked into how the model performs (compared to SupCon) in predicting superordinate categories (animals, etc.) and/or if the models learnt by SimLAP are more aligned with inter-class human similarity judgements [1] ? I will increase my score if the authors demonstrate that SimLAP outperforms SupCon in superordinate category recognition with the models pretrained on cifar100 and Imagenet-1k. I will further increase my score if the authors demonstrate the generality of the gate mechanism through a direct and successful implementation in other SSL methods, such as BYOL or DINO (BYOLLap, DINOLap ...).
[1] Hebart, M. N., Contier, O., Teichmann, L., Rockter, A. H., Zheng, C. Y., Kidder, A., ... & Baker, C. I. (2023). THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife, 12, e82580.
We thank the reviewer for the valuable comments.
W1: the performance matches that of SupCon.
We agree that SupCon is simpler, and we do not claim that SimLAP should replace it as a standalone objective on ImageNet. It is crucial to distinguish the main difference, that is, the learning scope of SupCon is identical pairs, while SimLAP is arbitrary pair. Comparing to SupCon is not to demonstrate that SimLAP is an alternative method. The core message, overlooked in the community, is that the semantically distinct pairs are important, not negligible in representation learning.
SimLAP provides a simple and effective way to utilize these pairs. As shown in Table 7, SimLAP prefers the data whose classes have more similarity. There are modest performance gains via joint loss (Table 9) and mid-training (Table 10). SimLAP is ahead in Image retrieval by 4.5 mAP on ROxford5k and 4.9 on RParis6k (Table 15, Medium).
W2: Are the subspaces non-null when the classes differ?
Yes, SimLAP does not collapse into SupCon, as shown in Figure 6 gate visualization. Our method prevent null subspaces.If the subspaces are null, all representations will be collapsed to one point, so the contrastive loss would be a constant. However, our experiment with distinct setting against this that loss decreases normally. Distinct pairs even achieves high performance (76.15, the second best) in Table 7. On CIFAR-100, 83 of 256 gate values exceed 0.5 on average over the pairs seen in training.
W3: SimLAP may collapse into SupCon.
SimLAP will not collapse into SupCon. We demonstrate experiments with the distinct pairs only in Table 7 and Table 11. SimLAP still works with distinct pairs, while SupCon fails to learn meaningful representations. A random partner shares the anchor's class about once in 1,000 draws on ImageNet and once in 100 on CIFAR-100. The learning signals almost come from distinct pairs.
W4: Finding 1, or the concept of "sign filter" in Section 3.1.
Sign Filter is a diagnostic and not part of SimLAP. Sign Filter selects the subspace according to the sign of two feature vectors, while Feature Filter used in SimLAP is learnable to find optimal subspaces. For two feature vectors from two classes, Sign Filter keeps the dimensions on which both have the same sign, drops the rest. It needs no training. Applied to a standard supervised encoder, it shows that a subspace in which two distinct classes are similar already exists (Figure 1b).
W5: the objective of Section 6 is unclear, but raises one question: is the masking strategy applicable in diverse SSL methods.
Section 6 asks whether SimLAP is useful as an addition to existing training, not as a replacement. It tests two uses: a joint loss with MoCo v3 (Table 9), and a mid-training stage for pretrained model MAE, DINOv2 and MoCo (Table 10).
SimLAP is not an alternative of any existing method. We are seeking to establish a new learning framework for semantically distinct samples. We first show it is feasible and learns competitive representations in Section 5. We admit that learning from arbitrary pairs is not normal way of representation learning and could be inefficient on ImageNet (Table 11). SimLAP should be applied into the datasets having lots of coarse classes, like CIFAR100 (Table 7). Therefore, we seek to the applications of SimLAP in ImageNet as an auxiliary loss. These experiments show that SimLAP is a complementary to existing methods.
The value of this paper is not an incremental method achieving better number in a benchmark. Instead, we provide a robust learning framework to utilize the distinct pairs, even ridiculous pairs like snake-lamp.