- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper establishes a theoretical analysis on the optimal solutions of generative contrastive learning. The authors introduce a kernel called Conditional Generation Kernel based on the generation probability of positive pairs. Theoretically, they prove that the optimal solution of the spectral contrastive loss is the CGK. Empirically, they compare the performance of InfoNCE and Spectral Contrastive Loss on language data and tasks.
Critical strengths
- There exist various theoretical frameworks of traditional contrastive learning. However, the theoretical understanding of generative contrastive learning is still limited. This paper uses the main component of generative contrastive learning in their analysis. Specifically, they design the metric based on the property that the joint probability of positive pairs can be evaluated as they are generated by language models. Consequently, the theoretical analysis is non-trivial compared to the current works.
- The logic flow is clear and the writing is easy to follow. The theoretical proofs seem solid.
Critical weaknesses
- This paper mainly focuses on the theoretical understanding of generative contrastive learning. The empirical insights are relatively limited. It seems that the spectral contrastive loss in generative contrastive learning does not shows a clear improvement in language tasks. It would be better to add more discussions or theoretical analysis on the comparison of InfoNCE and spectral contrastive loss in different real-world scenarios.
- As the spectral contrastive loss is equivalent to the matrix decomposition objective, we can also obtain the optimal solutions of the spectral contrastive loss by the spectral graph theory. I think it would be better to add more theoretical analysis to discuss whether there exist some connections or differences of the optimal solutions calculated by different methods.
- The verification experiments in Section 5 are mainly conducted on specific models. It would be better to analyze both the downstream performance and the geometry in different models.
Questions for authors
See weaknesses.