acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Policy Optimization: Optimizing von Neumann Entropy Across Queries for Unsupervised Foundation Model Alignment

Abstract

Reinforcement learning (RL) post-training has emerged as a dominant paradigm for unlocking complex reasoning and continuous generative capabilities in foundation models. However, standard post-training methods (such as PPO, GRPO, and DPO) rely heavily on expensive supervised signals, including ground-truth reasoning trajectories, verified labels, or calibrated reward models. While recent unsupervised heuristics, such as Entropy Minimized Policy Optimization for language reasoning and Atomic Policy Optimization for 3D atomic generation, attempt to eliminate external supervision, they evaluate generations strictly in isolation within individual queries. Consequently, these isolated intra-query methods suffer from severe vulnerabilities: , where the policy converges to degenerate, uninformative generic templates or universal crystal packing motifs regardless of input conditions, and susceptibility to induced by ad-hoc cluster approximation errors. To address these foundational bottlenecks, we propose Contrastive Policy Optimization (CPO), a unified, theoretically grounded framework for fully unsupervised foundation model alignment. Instead of optimizing isolated Shannon entropy per prompt, CPO models the joint generation distribution across a batch of diverse queries as a global quantum state represented by a normalized density operator . By leveraging contrastive learning to optimize the , CPO simultaneously drives two synergistic objectives: (1) , forcing the policy to form decisive, high-confidence consensus modes on each specific query, and (2) , enforcing maximal semantic distinction and diversity across different queries. We derive an exact, analytical policy gradient and advantage formulation that eliminates the need for heuristic clustering, external reward models, or supervised coordinates. Extensive experiments across both continuous geometric generation and discrete language reasoning demonstrate the decisive superiority of CPO. Our theoretical and empirical results demonstrate that cross-query von Neumann entropy optimization provides a universal, mathematically rigorous foundation for unsupervised foundation model alignment.

Primary area:
reinforcement learning
Keywords:
Policy Optimization, 3D atomic generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZtpeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Contrastive Policy Optimization (CPO), a framework for unsupervised foundation model alignment that addresses the potential limitations of existing intra-query optimization methods. Previous approaches primarily encourage consistency among multiple generations of the same query while overlooking potentially valuable cross-query information, which may lead to representation collapse and degenerate solutions. To address this issue, CPO models the joint generation distribution across a batch of diverse queries using a normalized density operator and introduces a von Neumann entropy-based objective. The proposed framework aims to jointly encourage intra-query consistency and inter-query diversity, with a spectral advantage formulation for policy optimization. The paper reports consistent improvements over existing unsupervised baselines across mathematical reasoning, general reasoning, crystal structure prediction, and antibody loop modeling. Overall, the motivation is reasonable, and the empirical results are promising. However, I have several concerns regarding the novelty of the proposed objective, the validity of certain theoretical claims, and whether the experimental results sufficiently establish the necessity and effectiveness of the proposed spectral formulation.

Critical strengths
  1. Clear and meaningful motivation. The paper identifies an important limitation of existing unsupervised policy optimization methods, which mainly focus on consistency among multiple generations of the same query while overlooking potentially useful cross-query information. This is a worthwhile research direction, particularly for tasks without reliable external verifiers, where intra-query self-consistency alone may reinforce incorrect consensus. Introducing cross-query constraints provides a reasonable complementary training signal.
  2. An interesting combination of intra-query consistency and inter-query diversity. The paper provides an interesting perspective on balancing generation consistency and representation diversity without relying on explicit ground-truth rewards. The proposed objective jointly considers these two aspects within a unified formulation, which is conceptually appealing.
  3. Broad experimental evaluation. The proposed method is evaluated across both language reasoning and 3D structure generation tasks, including mathematical reasoning, general reasoning, crystal structure prediction, and antibody loop modeling. The reported improvements over existing unsupervised approaches suggest that the framework may have applicability across different domains.
Critical weaknesses
  1. Limited novelty. The core idea of encouraging intra-query consistency while promoting inter-query diversity closely resembles conventional contrastive learning. The novelty beyond applying existing contrastive objectives to policy optimization is unclear.

  2. Necessity of von Neumann entropy. It is unclear why von Neumann entropy is necessary instead of simpler contrastive objectives such as InfoNCE. Could EMPO + InfoNCE achieve comparable improvements under the same training setup?

  3. Consistency and diversity do not guarantee correctness. A model could generate different but consistently incorrect answers for different queries while still satisfying the proposed objective. How does CPO prevent reinforcing such incorrect consensus?

  4. Dependence on initial policy quality. CPO may benefit models whose majority responses are correct but potentially reinforce errors when the majority responses are incorrect. Experiments across different initial model capabilities or majority-vote correctness levels would help clarify this issue.

  5. Validity of the spectral advantage. The paper claims that the proposed spectral advantage yields an exact policy gradient. However, the connection between the gradient of the batch-level spectral entropy and the sample-level policy gradient is insufficiently justified. A rigorous proof is needed.

  6. Influence of embedding representations. The spectral advantage may be dominated by query information rather than response quality. I suggest comparing Query-only, Answer-only, Reasoning-only, and Query+Response embeddings under a fixed policy, and measuring their correlations with ground-truth correctness.

  7. Non-collapse guarantee. The kernel is defined as (K_{ij}=\exp(z_i^\top z_j/\tau)). Even when two embeddings are orthogonal, (K_{ij}=1), rather than zero. Therefore, the claimed relationship between embedding orthogonality and the global entropy optimum requires further justification.

Questions for authors

see aboved

Reviewer_njFQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • The authors propose a method for unsupervised post-training of foundation models (LLMs and 3D generative models)
  • My understanding of the general intuition is that we want: “answers to different questions to be different, and answers to similar questions to be similar”
  • Concretely, they propose to embed each question-response pair via an encoder, then compute pairwise similarities between all pairs (M×MM \times M global matrix, where M=B×GM = B \times G contains BB questions and GG samples per question).
  • Then, one can compute the von Neumann entropy (which to my knowledge is identical to the Vendi score [1], up to a log transform) for the global matrix and the per-question matrix. The objective is to maximize the global entropy while minimizing the per-question entropy (Eq 6)
  • Optimizing this objective via a policy-gradient like estimator. Authors report results which appear to out-perform even GRPO and SFT without any supervision in some tasks
Critical strengths

The empirical results suggest that a purely unsupervised method (CPO) based on the principle of minimizing similarity of within-question answers, and maximizing the diversity of across-question answers, can be used to train a model to out-perform supervised methods such as GRPO. If true, this is a very interesting result and encouraging for this style of post-training given the difficulty of obtaining ground-truth verifications for many tasks.

Critical weaknesses

I have major reservations about the paper regarding its presentation, clarity, and mathematical obfuscation.

Clarity To my understanding, the core idea of the paper is to construct an unsupervised post-training objective to “encourage answers to different questions to be different, and answers to the same question to be similar”. This is an interesting and easily conveyable idea. Yet, I find the paper’s actual writing to be incomprehensible. For instance, it’s unclear why the above needs to be called “intra-query state purity” (L30) and “inter-query state orthogonality” (L32). They can be explained in more intuitive terms.

Further, it’s unclear whether the core idea above needs to be introduced through terms such as “the cross-query quantum density matrix operator”? To my understanding, the whole idea is based on first computing a similarity matrix (between responses), then computing a notion of entropy / diversity between the responses via the similarity matrix using the entropy of the matrix’s eigenvalues (Eq 5). While the quantum information formulation may be mathematically valid, I do not see how it helps improve the understanding of the concept or method. I find it very distracting and obfuscates the core idea.

Previous Work Additionally, this idea of using the entropy of eigenvalues of similarity matrix has been extensively explored in works such as the “Vendi Score” [1], which also references von Neumann entropy and the quantum system state interpretation. While I understand the authors mainly aim to make use of Von Neumann entropy as part of their algorithm, the complete lack of acknowledgement and citation to prior work is alarming.

Results presentation Regarding comparisons, it seems like some alternative methods out-perform CPO in Table 5, yet CPO is the online line highlighted. This is somewhat mis-leading as Table 1 and 2 presents the best results in bold yet the authors move away from that convention in Table 5. Additionally, all results are presented without error bars / confidence intervals, which makes it difficult to assess the significance of the reported improvements.

Overall, I found myself arriving at a cursory understanding of the idea only by actively ignoring much of the writing but instead focusing on reconstructing the underlying intuition. As I cannot confidently follow much of the paper's presentation, I find it difficult to properly assess its scientific merits in its current form. I also believe the paper would require substantial rewriting to serve as an effective piece of scientific communication.


[1] Friedman, Dan, and Adji Bousso Dieng. "The vendi score: A diversity evaluation metric for machine learning." arXiv preprint arXiv:2210.02410 (2022).

Questions for authors

Q1) This method introduces two more designs: which encoder to use for embedding, and the temperature parameter of the kernel. How sensitive are these choices, and how would you recommend picking them in practice? I see Table 9 discusses the impact of temperature on the chemical tasks -- do the authors have thoughts on proper temperatures on the language tasks?

Q2) When do you expect this principle (same question, similar answers; different questions, different answers) to work versus not work in practice? Should this depend on the composition of the training set? For e.g. what if different questions have the same correct answer?

Q3) How did you generate Figure 2 and 8? Were these visualizations produced using chemical visualization software from actual generated atomic coordinates, or were they AI-generated or otherwise synthesized? Could the authors clarify the visualization procedure?

Q4) I am trying to understand how a purely unsupervised method (CPO) can outperform GRPO in, e.g., Table 3. Are they (i) trained on exactly the same questions and for the same number of updates & sampling budgets? (ii) How were GRPO's reward function, learning rate, KL regularization, and other hyperparameters chosen?

Reviewer_keaqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce Contrastive Policy Optimization (CPO), which they claim as a fully unsupervised foundation model alignment framework. Drawing on ideas from quantum mechanics and optimizing an objective based on the von Neumann entropy, the authors claim that CPO tackles the problem of cross-query representational collapse, forcing high per-query confidence and high inter-query diversity. The authors apply their method to a range of settings, ranging from math to flow matching for materials generation.

Critical strengths

The paper is very ambitious and attempts to tackle a number of different fields, and provides dense theoretical reasoning around their objective design. The problem framing is a core strength, where the authors are addressing an important problem of representational collapse. Furthermore, the intent behind the algorithm design makes sense, as the authors address this collapse issue by adding a loss term that penalizes the cross-query similarity. The design of the ablation is also a reasonable approach, where the authors deconstruct into different scenarios including intra-query, inter-query, and full CPO.

Critical weaknesses

There are a number of weaknesses that should be addressed:

  • The introduction and related work are incredibly dense, the authors should consider writing more concisely in these sections or rotating some of the related work to the appendix to avoid cluttering of the main text.
  • In equation 11, the integral presented is not correct, as the log likelihood exactly is supposed to be: log⁡pθ(x∣c)=log⁡N(x1∣0,I)+∫01∇vθ(xt,t∣c)dt\log p_{\theta}(x\mid c) = \log \mathcal{N}(x_1 \mid 0, I) + \int_0^1 \nabla v_{\theta}(x_t, t \mid c) dt This calls into question the soundness of the following derivations, both in the appendix and the main text
  • Figure 3 looks to be AI generated, or at least appears that way, a clearer higher quality figure would be good
  • There are multiple cases of claims being made without quantitative justification, such as claims around the presence/absence of interstitial defects

These issues focus on the factual soundness of the paper and should be addressed.

Questions for authors
  • Few details, if any, are provided for how the experimental comparison to EMPO and APO were performed, that should be provided in the main text or the appendix
  • In general, more experimental details would be appreciated to better contextualize the results presented in Tables 1 and 2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.