- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Contrastive Policy Optimization (CPO), a framework for unsupervised foundation model alignment that addresses the potential limitations of existing intra-query optimization methods. Previous approaches primarily encourage consistency among multiple generations of the same query while overlooking potentially valuable cross-query information, which may lead to representation collapse and degenerate solutions. To address this issue, CPO models the joint generation distribution across a batch of diverse queries using a normalized density operator and introduces a von Neumann entropy-based objective. The proposed framework aims to jointly encourage intra-query consistency and inter-query diversity, with a spectral advantage formulation for policy optimization. The paper reports consistent improvements over existing unsupervised baselines across mathematical reasoning, general reasoning, crystal structure prediction, and antibody loop modeling. Overall, the motivation is reasonable, and the empirical results are promising. However, I have several concerns regarding the novelty of the proposed objective, the validity of certain theoretical claims, and whether the experimental results sufficiently establish the necessity and effectiveness of the proposed spectral formulation.
Critical strengths
- Clear and meaningful motivation. The paper identifies an important limitation of existing unsupervised policy optimization methods, which mainly focus on consistency among multiple generations of the same query while overlooking potentially useful cross-query information. This is a worthwhile research direction, particularly for tasks without reliable external verifiers, where intra-query self-consistency alone may reinforce incorrect consensus. Introducing cross-query constraints provides a reasonable complementary training signal.
- An interesting combination of intra-query consistency and inter-query diversity. The paper provides an interesting perspective on balancing generation consistency and representation diversity without relying on explicit ground-truth rewards. The proposed objective jointly considers these two aspects within a unified formulation, which is conceptually appealing.
- Broad experimental evaluation. The proposed method is evaluated across both language reasoning and 3D structure generation tasks, including mathematical reasoning, general reasoning, crystal structure prediction, and antibody loop modeling. The reported improvements over existing unsupervised approaches suggest that the framework may have applicability across different domains.
Critical weaknesses
-
Limited novelty. The core idea of encouraging intra-query consistency while promoting inter-query diversity closely resembles conventional contrastive learning. The novelty beyond applying existing contrastive objectives to policy optimization is unclear.
-
Necessity of von Neumann entropy. It is unclear why von Neumann entropy is necessary instead of simpler contrastive objectives such as InfoNCE. Could EMPO + InfoNCE achieve comparable improvements under the same training setup?
-
Consistency and diversity do not guarantee correctness. A model could generate different but consistently incorrect answers for different queries while still satisfying the proposed objective. How does CPO prevent reinforcing such incorrect consensus?
-
Dependence on initial policy quality. CPO may benefit models whose majority responses are correct but potentially reinforce errors when the majority responses are incorrect. Experiments across different initial model capabilities or majority-vote correctness levels would help clarify this issue.
-
Validity of the spectral advantage. The paper claims that the proposed spectral advantage yields an exact policy gradient. However, the connection between the gradient of the batch-level spectral entropy and the sample-level policy gradient is insufficiently justified. A rigorous proof is needed.
-
Influence of embedding representations. The spectral advantage may be dominated by query information rather than response quality. I suggest comparing Query-only, Answer-only, Reasoning-only, and Query+Response embeddings under a fixed policy, and measuring their correlations with ground-truth correctness.
-
Non-collapse guarantee. The kernel is defined as (K_{ij}=\exp(z_i^\top z_j/\tau)). Even when two embeddings are orthogonal, (K_{ij}=1), rather than zero. Therefore, the claimed relationship between embedding orthogonality and the global entropy optimum requires further justification.
Questions for authors
see aboved