acceptodds
Under review as a conference paper at ICLR 2027

Understanding Generative Contrastive Learning via the Conditional Generation Kernel

Abstract

Recent advances in unsupervised text embedding leverage the stochasticity of autoregressive generation by LLMs to generate semantically related positive pairs for contrastive learning. The resulting paradigm, which we term generative contrastive learning, has the unique property that the data-generating distribution is explicitly parametrized by the LLM, thus opening the door to a formal analysis of the contrastive objective. Building on this, we introduce the Conditional Generation Kernel (CGK), a positive semi-definite kernel that measures textual similarity through expected co-generation probabilities across a set of anchor documents. We show that the standard InfoNCE loss optimizes dot-product similarities toward the logarithm of the CGK, which exposes a fundamental representational bottleneck: because the logarithm of a kernel matrix is generally indefinite, the optimal target cannot be exactly realized as an inner product in any Euclidean space. Motivated by this bottleneck, we derive a contrastive objective whose global optimum is exactly the CGK and find that it aligns with the known spectral contrastive loss. We empirically confirm that the two objectives actually approach their theoretically derived optimal states, resulting in clearly differing geometries of the resulting embedding spaces. Finally, we evaluate the resulting models on downstream tasks, demonstrating that training with the spectral loss yields favorable properties for clustering and classification.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Contrastive Learning, Embeddings, Kernel

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ABPVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper establishes a theoretical analysis on the optimal solutions of generative contrastive learning. The authors introduce a kernel called Conditional Generation Kernel based on the generation probability of positive pairs. Theoretically, they prove that the optimal solution of the spectral contrastive loss is the CGK. Empirically, they compare the performance of InfoNCE and Spectral Contrastive Loss on language data and tasks.

Critical strengths
  1. There exist various theoretical frameworks of traditional contrastive learning. However, the theoretical understanding of generative contrastive learning is still limited. This paper uses the main component of generative contrastive learning in their analysis. Specifically, they design the metric based on the property that the joint probability of positive pairs can be evaluated as they are generated by language models. Consequently, the theoretical analysis is non-trivial compared to the current works.
  2. The logic flow is clear and the writing is easy to follow. The theoretical proofs seem solid.
Critical weaknesses
  1. This paper mainly focuses on the theoretical understanding of generative contrastive learning. The empirical insights are relatively limited. It seems that the spectral contrastive loss in generative contrastive learning does not shows a clear improvement in language tasks. It would be better to add more discussions or theoretical analysis on the comparison of InfoNCE and spectral contrastive loss in different real-world scenarios.
  2. As the spectral contrastive loss is equivalent to the matrix decomposition objective, we can also obtain the optimal solutions of the spectral contrastive loss by the spectral graph theory. I think it would be better to add more theoretical analysis to discuss whether there exist some connections or differences of the optimal solutions calculated by different methods.
  3. The verification experiments in Section 5 are mainly conducted on specific models. It would be better to analyze both the downstream performance and the geometry in different models.
Questions for authors

See weaknesses.

Reviewer_gfhVOpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce the “Conditional Generation Kernel” (CGK) for generative contrastive representation learning. They provide a theoretical analysis, connecting the CGK to known objects, such as the InfoNCE objective and the spectral contrastive loss of HaoChen et al (2021). They highlight that, compared to related kernels in prior work, CGK is “explicitly computable”. This allows them to evaluate whether optimal states are practically recovered, which they do on a corpus of 2000 one-sentence LLM-generated summaries of real scientific abstracts.

Critical strengths
  1. Developing a theoretical understanding of kernels in contrastive learning is relevant, and its practical usage to characterize trained models is an interesting gap.

  2. The mathematical exposition is quite clean, and the theoretical results are consistent with other literature.

Critical weaknesses
  1. The paper’s empirical work is not rigorous enough to interestingly complement the theoretical work, and the stated claims often don’t match the reported numbers. Some examples:
  • The general scope of the empirical work is very narrow, all analyses use a single model, one prompt and a single domain on around 2k generated samples.

  • The retrieval experiment in §2.2 does not have any baselines.

  • Downstream claims are derived from single-seed runs with very small margins (e.g., in Tables 4 and 5). Adding multi-seed runs would strengthen the claims.

  • Disagreeing claims like, that the abstract claims both objectives approach their theoretical optima, but this is contradicted in §5.3, as “all models lose their logarithmic alignment”.

  1. The paper’s theoretical contributions lack directionality, and, relatedly, the paper’s presentation, especially the abstract and introduction, makes the contributions hard to follow despite careful reading. Specific examples:
  • To my reading, the abstract and introduction fail to state the core question asked in this paper and are both very dense with undefined terms and loosely connected threads. I encourage the authors to provide clearer guidance on these theoretical results.

  • It seems to me that some of the theoretical results are restated from prior work. For instance, the optima of InfoNCE (Theorem 1) and the spectral loss (§4) have been derived in Johnson et al. 2023 and he CGK kernel seems to have already been introduced in the same form in that paper as well.

  • One instance of the lack of structure is that §3 derives the InfoNCE target (log K) is not realizable by dot products, and §4 introduces the spectral loss because its target (K) is realizable. The question that follows is whether this matters, i.e., does a model trained toward K capture the CGK’s similarity structure better than one trained toward log K? This should be central, yet, it is visible in §5.3, where, against the motivation, all objectives recover the CGK’s neighborhoods equally well (NDCG@50 of 0.31 to 0.34).

I encourage the authors to state the concrete question that a computable kernel answers and to design more rigorous experiments around it. They should also align the abstract with the actual findings and support the empirical claims with well-chosen baselines, and proper confidence estimates.

Questions for authors

Could the authors please clearly state the novelty with respect to the Johnson et al. 2023 paper? A clearer, more definitive distinction may positively affect my score.

Reviewer_S4ipOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes the Conditional Generation Kernel (CGK), which defines text similarity based on the text generation probabilities of a large model under different anchor document conditions. The authors claim that the theoretical optimal similarity of InfoNCE corresponds to the logarithmic form of the CGK, while the optimal objective of the Spectral Contrastive Loss directly corresponds to the CGK itself. Therefore, the authors attempted to introduce the Spectral Contrastive Loss during the training of text embeddings and, after making adaptive improvements, achieved overall performance comparable to that of InfoNCE.

Critical strengths
  1. The paper introduces CGK from existing research and demonstrates that it is both easy to estimate and can be estimated accurately in the context of LLMs.

  2. The theoretical analysis is clearly presented, the core mathematical arguments are sound, and there is a smooth transition from theory to downstream tasks.

  3. The improvement to the independent modulus prediction head for Spectral Loss is effective; experiments show that it significantly improves downstream performance.

Critical weaknesses
  1. On a theoretical level, the main advantage of CGK lies in the fact that the existence of LLMs allows its theory to be explicitly computed and verified; however, it cannot be proven that Spectral Contrastive Loss is a superior text representation learning objective compared to InfoNCE Loss. The authors’ belief appears to stem from existing research on visual representation learning [1]; however, other studies [2, 3] have demonstrated that conclusions from representation learning in the field of computer vision (CV) do not seem to transfer directly to the field of natural language processing (NLP), as there are inconsistencies in many of the assumptions underlying representation learning in CV and NLP—such as the assumption that the number of negative samples approaches infinity [3].

  2. Regarding the experiments, the training process for downstream tasks does not align with the theoretically assumed generative distribution. As described in Appendix B.5.3, multiple prompts were introduced during the actual training process, and the method for constructing anchor-positive sample pairs varied.

  3. In terms of contributions, the actual contributions of this paper may be limited to the theoretical analysis and empirical validation sections; the improved Spectral Loss has not consistently demonstrated an advantage over InfoNCE Loss in downstream tasks.

[1] Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. NeurIPS 2021

[2] SCD: Self-Contrastive Decorrelation of Sentence Embeddings. ACL 2022

[3] On The Inadequacy of Optimizing Alignment and Uniformity in Contrastive Learning of Sentence Representations. ICLR 2023

Questions for authors

Aside from the fact that LLMs make it easier to compute the exact conditional likelihood of generated samples compared to diffusion models, are there any other aspects of the method itself that are strongly related to text representation learning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.