- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper contains an analysis of the InfoNCE loss function with a focus on exploring how varying (i) the way that positives and negatives are chosen and (ii) the sharpness of the positive vs. negative distribution impacts different aspects of downstream model performance. All experiments use either a dataset referred to as ‘digits’ or Fashion-MNIST. The paper is organized around explorations of (i) and (ii). To facilitate this, the paper introduces a parameter that controls how often the positive for a class is a variation of the same instance vs another instance in the class. The paper provides evidence that in the former case, the model learns features relevant to instance retrieval but in the latter case, representations more closely cluster in terms of class identity. The paper also introduces a way to vary the sharpness of the distribution through a parameter . The paper provides evidence that this sharpness plays a similar role to temperature and that sharp distributions can be detrimental if there is label noise for certain instances.
Critical strengths
- Though the experiments are limited, the direction is interesting: The reviewer believes that the idea of introducing parameters that can disentangle different sources of variation in InfoNCE is a good approach to better understanding this approach to self-supervised learning. The specific parameters that this paper introduces are reasonable and get at the issues without introducing to much additional complexity.
Critical weaknesses
- Clarity: The reviewer found this work very hard to read and understand. It is possible that some of this comes from the reviewer not being an expert in this area, but there are also basic practices that the paper could adopt to improve exposition and clarity. The value of ICLR papers does not come from the quality of their writing, but if the reader cannot understand the points the paper is making, that is a serious issue. As an example, the experimental set up should be very clearly stated. The purpose and motivation of various constructions should be made as clearly and simply as possible. Since the whole paper depends on Section 2, this should be expanded to ensure that the set-up is understood by the reader. It would be very helpful to also include a simple example. Another problem is the many points in the paper where vague statements are made and then never explained. For instance, on line 399 the paper says that “So “class structure was destroyed” can mean two things.” But what these two things are is never said. On line 307, the paper says “We reimplemented Eq. 1 from scratch instead of reusing our own trainer.” But then it never says how or why this was done. In general, the paper often reads as rough research notes. Finally, the paper often uses unnecessarily complex jargon like “granted-true-pairs diagnostic control”. There has to be a simpler way to say this (possible using more words).
- Broader or deeper analysis: As noted above, the paper is structured around two specific experiments. While the paper does summarize the results and (vaguely) states conclusions, no further analysis is provided. There are a range of ways that this could be increased. For example, connecting the experimental conclusions to actual design considerations that are of importance to practitioners (e.g., how can we use these results to build better algorithms?) or analysis into why varying the chosen parameters results in the observed behavior. In particular, it may be that at least one of these experiments is amendable to theoretical analysis. As it is, the impact of this work seems like it will be very limited.
- Impact: A result does not have to be surprising to be valuable. Confirming folk wisdom can be important. On the other hand, if a result largely confirms what is already expected, does not address an existing question, and does not deliver any additional insight, the value of the work can be questioned. To the reviewer, both of the results largely agree with community intuition. If you set the target to be another class element rather than a different view of the same instance, you will learn class features. Similarly, since sharpness of the distribution is closely related to temperature, it is not surprising that the effect of varying sharpness is very similar to the effect of varying temperature.
Questions for authors
- Are there ways in which the results in this paper can inform best practices in self-supervised training or new methods that use InfoNCE?
- What is the symbol after in (2)?
- What is the ‘digits’ dataset, is this MNIST?
- What does a ‘demand’ mean in this case?
- What is the symbol before in equation (2)?