Contrastive Policy Optimization: Optimizing von Neumann Entropy Across Queries for Unsupervised Foundation Model Alignment

Reinforcement learning (RL) post-training has emerged as a dominant paradigm for unlocking complex reasoning and continuous generative capabilities in foundation models. However, standard post-training methods (such as PPO, GRPO, and DPO) rely heavily on expensive supervised signals, including ground-truth reasoning trajectories, verified labels, or calibrated reward models. While recent unsupervised heuristics, such as Entropy Minimized Policy Optimization for language reasoning and Atomic Policy Optimization for 3D atomic generation, attempt to eliminate external supervision, they evaluate generations strictly in isolation within individual queries. Consequently, these isolated intra-query methods suffer from severe vulnerabilities: $\text{cross-query representation collapse}$, where the policy converges to degenerate, uninformative generic templates or universal crystal packing motifs regardless of input conditions, and susceptibility to $\text{reward hacking}$ induced by ad-hoc cluster approximation errors. To address these foundational bottlenecks, we propose Contrastive Policy Optimization (CPO), a unified, theoretically grounded framework for fully unsupervised foundation model alignment. Instead of optimizing isolated Shannon entropy per prompt, CPO models the joint generation distribution across a batch of diverse queries as a global quantum state represented by a normalized density operator $\rho$. By leveraging contrastive learning to optimize the $\text{von Neumann entropy}$ $S(\rho )=-\mathrm{Tr}(\rho \ln \rho )$, CPO simultaneously drives two synergistic objectives: (1) $\text{intra-query state purity}$, forcing the policy to form decisive, high-confidence consensus modes on each specific query, and (2) $\text{inter-query state orthogonality}$, enforcing maximal semantic distinction and diversity across different queries. We derive an exact, analytical policy gradient and advantage formulation that eliminates the need for heuristic clustering, external reward models, or supervised coordinates. Extensive experiments across both continuous geometric generation and discrete language reasoning demonstrate the decisive superiority of CPO. Our theoretical and empirical results demonstrate that cross-query von Neumann entropy optimization provides a universal, mathematically rigorous foundation for unsupervised foundation model alignment.