- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SFT/DPO relies on a single standard answer and thus mistakenly penalizes correct code that is semantically equivalent. In contrast, GRPO, which is a pure RL approach, gives zero rewards when the candidate success rate is zero at the beginning of training, resulting in vanishing gradients and inability to start from zero. The authors transformed alignment into a consensus game and proposed PC-GRPO, allowing the model to sample a batch of candidates, using the consistent pattern among the candidate groups as the reward, and then decomposing the answer-level consensus into step-level through soft solvability weighting, thus equivalent to an unsupervised process reward model. They capped the rewards for all wrong answers using Minimax polynomials and holdout method to ensure numerical stability. Theoretically, they proved that the variance of standard GRPO vanishes as the accuracy approaches zero, while PC-GRPO still maintains an effective gradient variance, with bounded rewards to prevent cheating, and revealed that pure self-games would lead to consensus collapse and require external signals for dependency. Experiments achieved the strongest trainable results on APPS and LiveCodeBench on Qwen2.5-Coder-7B, especially in sampling groups with approximately one-third of completely incorrect answers, where there were still 90% cases generating usable gradients, verifying the ability of cold-start learning from zero.
Critical strengths
- When there is no correct answer among a set of candidates, it is still possible to extract gradient signals from the consistent patterns among the candidates, solving the problem of traditional RL where the process immediately fails upon starting.
- The rewards are determined by the collective consensus and have an upper limit. The model cannot artificially inflate the rewards through fabrication or speculative behavior, thus avoiding reward hacking.
- Without the need for standard answers and manual step annotations, the process reward signals are automatically formed, and the required size of the candidate group is reduced from exponential to linear.
Critical weaknesses
- Theorem 4/5 of the paper was proved independently. Without continuous external ground-truth input, consensus will monotonically collapse and be irreversible, which is called "degenerate consensus". The ablation study also confirmed that pure ungrounded consensus is only 9.5%, while the complete PC-GRPO is 25.00%. That is to say, consensus can only help you get started in the cold start phase, but in the long term, it still needs to rely on real and correct samples for correction; otherwise, the more you practice, the more biased it becomes.
- The entire reward system is based on "being able to run unit tests and being identified by the test generator as candidates". The author himself stated in Appendix A that the real execution feedback of the complete SWE-bench is still a future work, and currently only syntactic-level command matching is used as a proxy. Moving to tasks without automatic execution feedback (pure reasoning, writing, open dialogue) will lose the discrimination signal.
- The inherent bias and cost of "majority consensus ≠ correctness": When the correct solution is a minority outlier, the consensus reward will favor the majority of incorrect solutions; and each group needs to sample 2K = 16 candidates, run tests, and make leave-one-out estimates, which results in a significantly higher training cost than single-sample methods; the experiments have only been verified on code problems with 7B + LoRA, and the effects on larger-scale and real engineering tasks are still preliminary.
Questions for authors
- How long is the "safe window" of consensus bootstrap? At what stage of training or when a certain number of real and correct samples have been accumulated, must the consensus reward be switched back to the grounding reward? Are there any online-monitorable collapse warning indicators?
- Theorem 4 is proved under the KL regularization and finite distribution assumptions; at what proportion of the ground-truth in the real environment will the method actually fail? What is the corresponding number of samples for this critical proportion in your experiments?
- Your Unit Test Generator and the trained model belong to the same LLM. Will they share the same kind of systematic blind spots, thereby misclassifying "common errors" as consensus? Have you tested the false positive/false negative rates of the test generator in the experiments?
- In domains without executable feedback (mathematical proofs, open reasoning, dialogue tasks other than code), what signal is used to replace the reward? Is there a plan to replace the unit test with another discriminator (such as a validator model)?
- When the correct solution is a rare minority (most candidates choose the common but incorrect formulation), will the consensus reward systematically penalize the correct solution? Are there any cases of such minority samples being wrongly suppressed in the APPS data?
- The cost of each group of 16 candidates plus LOO is what percentage of the throughput cost of GRPO in full training? With larger models (such as 30B+), the cold start problem has been alleviated, and how much marginal benefit can PC-GRPO retain?