acceptodds
Under review as a conference paper at ICLR 2027

Process Coherence Games for Code Alignment: Bootstrapping from Zero

Abstract

Large Language Models (LLMs) excel at code generation and complex multi-step reasoning, yet aligning them toward functional correctness remains a severe challenge. Standard alignment paradigms, such as Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), rely on syntactic matching against canonical reference traces, thereby penalizing valid but syntactically diverse reasoning paths. Conversely, outcome-based Reinforcement Learning (RL) suffers from extreme gradient variance and produces zero gradient signal in complex domains where the probability of an initial correct solution is near zero (). In this paper, we overcome these bottlenecks by formulating code generation alignment via *Pairwise Coherence*. We propose C2-GRPO and its probabilistic extension PC-GRPO, which bypass the need for exhaustive ground-truth oracles by optimizing for functional semantic consistency over dynamically generated unit tests. To solve the cold-start problem, we introduce two algorithmic techniques, *Soft Plausibility Weighting* and *Process Coherence*, which dynamically synthesize an implicit Process Reward Model (PRM) entirely through unsupervised group consensus, reducing the group size needed for a consensus signal on long-horizon tasks from exponential to linear under step-wise decomposition (via tree-based simulation or, for code generation, test-decomposed independent rollouts). We prove that standard GRPO's gradient signal vanishes as while PC-GRPO maintains advantage variance whenever the group's functional disagreement is non-degenerate (an explicit count-separation condition), and establish convergence and Rademacher generalization bounds for an idealized form of the coherence objective. Empirically, on Qwen2.5-Coder-7B-Instruct, PC-GRPO is the strongest trainable method across APPS Intro, APPS Interview, and LiveCodeBench release_v1. Even at 7B scale, APPS Interview retains a substantial cold-start regime; in these groups, PC-GRPO produces usable gradient variance in 91.0% of cases where binary rewards provide none. We further prove fundamental impossibility results for ungrounded consensus (the *Degenerate Consensus* collapse) and its partial converse (*grounded one-step improvement*), and integrate bounded Minimax Polynomial estimators (Mohri et al., 2026b) for finite-sample stability, and structurally bound reward hacking via the consensus formulation.

Primary area:
learning theory
Keywords:
code generation, GRPO, consensus-based rewards, process reward models, game theory, LLM alignment, reward hacking

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_aaoTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SFT/DPO relies on a single standard answer and thus mistakenly penalizes correct code that is semantically equivalent. In contrast, GRPO, which is a pure RL approach, gives zero rewards when the candidate success rate is zero at the beginning of training, resulting in vanishing gradients and inability to start from zero. The authors transformed alignment into a consensus game and proposed PC-GRPO, allowing the model to sample a batch of candidates, using the consistent pattern among the candidate groups as the reward, and then decomposing the answer-level consensus into step-level through soft solvability weighting, thus equivalent to an unsupervised process reward model. They capped the rewards for all wrong answers using Minimax polynomials and holdout method to ensure numerical stability. Theoretically, they proved that the variance of standard GRPO vanishes as the accuracy approaches zero, while PC-GRPO still maintains an effective gradient variance, with bounded rewards to prevent cheating, and revealed that pure self-games would lead to consensus collapse and require external signals for dependency. Experiments achieved the strongest trainable results on APPS and LiveCodeBench on Qwen2.5-Coder-7B, especially in sampling groups with approximately one-third of completely incorrect answers, where there were still 90% cases generating usable gradients, verifying the ability of cold-start learning from zero.

Critical strengths
  1. When there is no correct answer among a set of candidates, it is still possible to extract gradient signals from the consistent patterns among the candidates, solving the problem of traditional RL where the process immediately fails upon starting.
  2. The rewards are determined by the collective consensus and have an upper limit. The model cannot artificially inflate the rewards through fabrication or speculative behavior, thus avoiding reward hacking.
  3. Without the need for standard answers and manual step annotations, the process reward signals are automatically formed, and the required size of the candidate group is reduced from exponential to linear.
Critical weaknesses
  1. Theorem 4/5 of the paper was proved independently. Without continuous external ground-truth input, consensus will monotonically collapse and be irreversible, which is called "degenerate consensus". The ablation study also confirmed that pure ungrounded consensus is only 9.5%, while the complete PC-GRPO is 25.00%. That is to say, consensus can only help you get started in the cold start phase, but in the long term, it still needs to rely on real and correct samples for correction; otherwise, the more you practice, the more biased it becomes.
  2. The entire reward system is based on "being able to run unit tests and being identified by the test generator as candidates". The author himself stated in Appendix A that the real execution feedback of the complete SWE-bench is still a future work, and currently only syntactic-level command matching is used as a proxy. Moving to tasks without automatic execution feedback (pure reasoning, writing, open dialogue) will lose the discrimination signal.
  3. The inherent bias and cost of "majority consensus ≠ correctness": When the correct solution is a minority outlier, the consensus reward will favor the majority of incorrect solutions; and each group needs to sample 2K = 16 candidates, run tests, and make leave-one-out estimates, which results in a significantly higher training cost than single-sample methods; the experiments have only been verified on code problems with 7B + LoRA, and the effects on larger-scale and real engineering tasks are still preliminary.
Questions for authors
  1. How long is the "safe window" of consensus bootstrap? At what stage of training or when a certain number of real and correct samples have been accumulated, must the consensus reward be switched back to the grounding reward? Are there any online-monitorable collapse warning indicators?
  2. Theorem 4 is proved under the KL regularization and finite distribution assumptions; at what proportion of the ground-truth in the real environment will the method actually fail? What is the corresponding number of samples for this critical proportion in your experiments?
  3. Your Unit Test Generator and the trained model belong to the same LLM. Will they share the same kind of systematic blind spots, thereby misclassifying "common errors" as consensus? Have you tested the false positive/false negative rates of the test generator in the experiments?
  4. In domains without executable feedback (mathematical proofs, open reasoning, dialogue tasks other than code), what signal is used to replace the reward? Is there a plan to replace the unit test with another discriminator (such as a validator model)?
  5. When the correct solution is a rare minority (most candidates choose the common but incorrect formulation), will the consensus reward systematically penalize the correct solution? Are there any cases of such minority samples being wrongly suppressed in the APPS data?
  6. The cost of each group of 16 candidates plus LOO is what percentage of the throughput cost of GRPO in full training? With larger models (such as 30B+), the cold start problem has been alleviated, and how much marginal benefit can PC-GRPO retain?
Reviewer_hnthOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PC-GRPO combines seed-test progress with leave-one-out agreement on generated test inputs, using a bounded Minimax reward to train when binary rewards provide no variation. The paper analyzes consensus, grounding, and cold-start variance, and reports five-seed APPS/LiveCodeBench comparisons plus preliminary SWE-bench Lite results.

Critical strengths
  • The main comparisons match the 16-candidate generation budget and include Partial-Credit GRPO, with five training seeds (Table 1; Appendix I.1). These controls support gains beyond simply sampling more candidates.
  • The explicit analysis of degenerate consensus and conditional grounded improvement is useful: it distinguishes agreement from correctness and identifies when consensus can reinforce mistakes (§4.3).
Critical weaknesses
  • Table 6 measures nonzero advantage variance but omits Partial-Credit GRPO on the same cold-start groups. Since Theorem 3 guarantees signal existence rather than useful direction, the specific benefit over dense progress rewards in this regime remains incompletely isolated.
  • The SWE-bench process reward uses exact command-string agreement with uniform, ungrounded weights (Appendix I.10). Although Table 14 reports resolve gains, this implementation does not directly validate the full grounded semantic process-coherence mechanism. The agentic conclusions should remain correspondingly limited.
Questions for authors
  1. On the same cold-start groups as Table 6, how do PC-GRPO and Partial-Credit GRPO compare in signal yield and alignment with test-progress improvements?
  2. Among the controlled groups in Appendix I.12, how many satisfy the full valid-favoring margin of Theorem 6, beyond merely A(V) > 0?
Reviewer_VMGLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PC-GRPO, which uses agreement between sampled programs on generated tests as an additional reward for code RL. The method includes seed-test filtering, soft plausibility weighting, and a minimax estimator. The authors provide theoretical analyses and report improvements over GRPO baselines on APPS, LiveCodeBench.worth studying, and using agreement on generated tests to provide additional training signals is an interesting idea. The paper also shows improvements over several baselines.

Critical strengths

How to get training signal from the pass@k = 0 problems (termed as cold-start problem) worth studying, and using agreement on generated tests to provide additional training signals is an interesting idea. The paper also shows improvements over several baselines.

Critical weaknesses
  1. I am not convinced by the theoretical motivation and contributions of this paper.

From what I understand, the main idea is to reward programs based on how much they agree on generated tests. This seems relatively simple, but the paper introduces a lot of game-theoretic concepts and theorems. I find it difficult to understand why all of this is necessary, and how these theoretical results help explain the actual method in Algorithm 1.

Some concerns about the theorems as well:

  • Theorem 2 essentially shows that the log-probability reward is upper bounded by zero. However, this does not prevent incorrect programs from agreeing with each other and receiving high rewards. So it cannot "hack" it to infinity but it can still "hack" it.

More importantly, I don't think the paper sufficiently explains why agreement between programs should indicate correctness, especially when all sampled programs are incorrect. Having a nonzero reward variance does not necessarily mean that the reward provides a useful learning signal.

The coherence objective and minimax estimator are closely related to Mohri et al. (2026a,b). It is unclear to me what the main new theoretical contribution is beyond these works.

  1. The paper is unnecessarily complicated and difficult to follow.

I had a hard time following the method, especially Algorithm 1 and its connection to the theoretical analysis. The paper introduces many new concepts, but I don't see a clear justification for why they are needed.

I think the paper would be much more convincing with a simpler explanation of the method and experiments showing when agreement is actually useful. At the moment, I am not convinced that the theoretical analysis supports the main motivation or justifies the complexity of the proposed method.

  1. I also find the comparison with prior work insufficient. The paper tries to tackle the hard questions through: using generated tests, program agreement, and partial-credit rewards to train code models. 1. There're already some approaches doing this (e.g. can trace back to CodeT by Chen et al., ICLR 2023). It is unclear to me what the proposed method improves over these approaches. 2. There're also literature tacking the pass@k = 0 questions through sampling more, such as Reinforce-Ada or https://arxiv.org/pdf/2506.13923. These methods provide alternative ways to recover useful training signals without relying on agreement between incorrect solutions. I find it surprising that none of them is discussed or compared against.

In particular, I would like to see whether the additional complexity of PC-GRPO actually provides benefits over simpler reward constructions.

TL;DR: I'm quite disappointed with the presentation of the manuscript. The question the paper tackle is an interesting question and hard, which means that, if done right, the method that claims to solve the question should have huge impact. But for now I haven't seen it. Also, I'm questioning the complicated nature (also partly due to the presentation) of the method since there're simpler method in the literature to tackle the question "what if all samples are wrong?".

Questions for authors
  1. How exactly are the standard deviations in Tables 1 and 3 calculated? Could the authors provide the individual results for the five training seeds? Also the number of task and generation config etc?

  2. Why should agreement between incorrect programs provide a useful training signal? Can the authors show that the proposed reward actually correlates with correctness, especially in cold-start groups? How does this compare with simply using partial-credit rewards?

  3. What is the main theoretical contribution compared with Mohri et al. (2026a,b)? Which theoretical results directly apply to Algorithm 1 as implemented?

  4. What's the difference between GRPO and R1-style GRPO in Table 1, is it non-reasoning mode vs reasoning mode?

Reviewer_2AfTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Outcome-based code RL gives zero gradient exactly where it is needed most. When no sample in a group passes, all rewards are identical and the group-normalised advantage vanishes. This paper swaps the question "is this program correct?" for "does this program behave like its peers?". For each prompt the base model writes a semantics-preserving paraphrase, K=8K=8 samples are drawn from each version to form an orbit of ∣G∣=16|G|=16, a cached LLM-written script generates N=3N=3 random test inputs, and each candidate is rewarded by the log-likelihood of its outputs under the group's consensus distribution. Leave-one-out peer counts are read off a bounded Minimax polynomial table instead of log⁡(X/K)\log(X/K). Seed tests supply a progress score that hard-filters or softly weights the consensus, and that is what is meant to stop it collapsing onto agreed-wrong answers. Evaluation is Qwen2.5-Coder-7B with LoRA, 512 training examples, greedy pass@1 on APPS Interview, LiveCodeBench and APPS Introductory against five RL baselines at matched generation budget, plus end-to-end SWE-bench Lite. PC-GRPO reports 12.80 on APPS-I against 10.80 for vanilla GRPO, and 25.0/300 against 20.2 on SWE-bench Lite.

Critical strengths
  • The compute comparison is honest. Every RL method gets 16 samples per prompt per step, and the orbit is K=8×2K = 8 \times 2 rather than 16×216 \times 2, so the 2-point gain is not quietly bought with extra generations. The five baselines span genuinely different families, and Best-of-16 Public is labelled a non-trainable ceiling instead of being slipped in as a comparator.
  • The paper argues against itself with real content, not boilerplate. Appendix A lists nine idealisations with the actual deviation for each, and immediately after Theorem 13 the authors supply their own counterexample in which policy quality falls from π0(A)=0.1\pi_0(A)=0.1 to 2/832/83. Every mechanically recomputable quantity I checked holds: Welch t=3.27t = 3.27, the softmax weights, the Minimax grid argmaxes.
Critical weaknesses
  • Remove seed grounding and the method drops to 10.28, below vanilla GRPO's 10.80, so most of the gain comes from the weak ground truth in the seed tests rather than from consensus, which is the thing the paper claims to bypass. Everything else is one model family, one scale, LoRA only, 512 examples, 192 steps, and no pass@k curves for a paper whose motivating case is pass@k = 0. The SWE-bench run has plausibility weighting switched off, i.e. the ungrounded configuration Theorem 4 predicts should collapse, excused after the fact by trajectory fragmentation and a short budget.
  • Contribution 1 claims the exponential-to-linear result holds "for both tree-based and independent-rollout settings", while the paper's own proof of Theorem 19 says independent rollouts need K=Ω(η−t)K = \Omega(\eta^{-t}), "thereby restoring the exponential bottleneck", and Appendix A (v) repeats the restriction. Theorem 1 also states all three parts as Ω\Omega lower bounds, which implies no separation at all. The Θ\Theta orders that would establish one appear only inside the p.36 proof.
Questions for authors
  1. Please release the per-seed raw pass@1 counts behind Table 1(a) and Table 3. How do all 21 trainable-method entries come to lie exactly on the single-evaluation grid (APPS 0.2%0.2\%, LCB 0.25%0.25\%) when the same five-seed protocol in Table 7 and Table 8 produces off-grid values such as 10.28 and 12.08?
  2. For LiveCodeBench, "PC-GRPO ... 25.00 ± 0.86%" and "Partial-Credit GRPO ... 21.25 ± 0.93%" cannot be produced by five integer counts out of 400 under either the sample (n−1n-1) or population (nn) convention. The achievable neighbours of 0.860.86 are 0.850.85 and 0.870.87, and of 0.930.93 are 0.920.92 and 0.940.94. How were these two dispersion figures computed?
  3. Contribution 1 claims a reduction "from exponential to linear ... for both tree-based and independent-rollout settings (Theorem 1)", while the Theorem 19 proof concludes K=Ω(η−t)K = \Omega(\eta^{-t}), "thereby restoring the exponential bottleneck", for independent rollouts. Which statement do you intend to stand, and does the Abstract's claim survive the restriction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.