- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes Continuous Co-Evolution (CCE), a framework for jointly training code and test generators using continuous execution feedback rather than binary pass/fail signals. The key motivation is that, in graded programming tasks, valid programs can differ substantially in solution quality, which binary feedback fails to capture. CCE employs a shared LLM for both code and test generation, optimized through alternating GRPO-style updates. The coder is rewarded based on its average performance on reference tests, while the tester receives a reliability-based pairwise reward that encourages generated tests to preserve the program rankings induced by reference tests. The authors also provide a theoretical connection showing that the proposed tester reward recovers the CURE reward under a binary execution model. The framework is evaluated on four tasks: Traveling Salesman Problem (TSP), Set Cover, Parallel Machine Scheduling (PMS), and Game 2048. Experimental results show that CCE generally outperforms coder-only training, CURE-style binary-feedback co-evolution, and direct-inference baselines, with particularly notable improvements on Set Cover, PMS, and Game 2048.
Critical strengths
-
Well-motivated extension of coder–tester co-evolution to graded programming tasks. The paper addresses a meaningful limitation of binary execution feedback, which fails to distinguish between valid programs with different solution qualities. The proposed reliability-based pairwise reward provides a principled way to train test generators using continuous execution scores while encouraging consistency with reference-induced program rankings. The theoretical connection to CURE under binary execution feedback further supports the formulation. This extension is particularly relevant to optimization and sequential decision-making tasks where solution quality cannot be adequately characterized by pass/fail outcomes.
-
Consistent empirical improvements across diverse graded programming tasks. The experiments demonstrate that CCE generally outperforms both coder-only training and CURE-style binary-feedback co-evolution across four tasks, covering combinatorial optimization and sequential decision-making. The improvements are particularly notable on Set Cover, PMS, and Game 2048, with gains in both mean performance and Best-of-N metrics. These results provide empirical evidence that preserving continuous execution feedback can improve the quality of generated programs beyond what is achieved with binary feedback alone.
Critical weaknesses
-
Insufficient ablation of coder and tester reward designs. The proposed CCE framework simultaneously modifies the reward mechanisms for both the coder and tester, replacing binary execution feedback with continuous quality scores and introducing a reliability-based pairwise reward for test generation. However, the current experiments do not sufficiently disentangle the individual contributions of these two modifications. Although comparisons with CURE and Coder-Only training demonstrate the overall effectiveness of the framework, they do not clarify whether the improvements primarily originate from the continuous coder reward, the proposed tester reward, or their interaction. A more comprehensive ablation study that independently varies the coder and tester reward designs (e.g., binary versus continuous coder rewards and binary versus ranking-based tester rewards) would help identify the source of the performance gains and better substantiate the necessity of each component.
-
Lack of analysis of coder–tester co-evolution dynamics. Although the central motivation of the proposed framework is the co-evolution of code and test generation capabilities, the experimental analysis primarily focuses on final-checkpoint performance without examining how these capabilities evolve throughout training. In particular, the paper does not provide learning curves for coder performance or tester quality metrics, such as generated-test validity, discriminative rate, and ranking agreement. The generated-test diagnostics in Appendix H are reported only for the final training iteration. Consequently, it remains unclear whether the coder and tester progressively improve together, whether improvements in tester quality coincide with gains in coder performance, or whether alternating updates introduce interference between the two roles. Reporting the training trajectories of both roles across optimization iterations would provide more direct empirical evidence for the claimed co-evolutionary behavior.
Questions for authors
-
Could the authors provide a more detailed ablation study that independently examines the contributions of the continuous coder reward and the reliability-based tester reward? Specifically, how does performance change when only one of the two reward mechanisms is modified while keeping the other unchanged? Such results would help clarify whether the observed improvements primarily stem from the coder reward, the tester reward, or their interaction.
-
Could the authors provide training curves illustrating how the capabilities of both the coder and tester evolve throughout training? In particular, tracking coder performance alongside tester quality metrics (e.g., generated-test validity, discriminative rate, and ranking agreement) would help determine whether both roles progressively improve and whether alternating updates lead to mutual improvements or interference. This evidence would be important for assessing the effectiveness of the claimed co-evolutionary mechanism.
3 comments
Thank you for your response. I have updated my rating accordingly.
Thank you for your follow-up and for updating your itemized rating! We sincerely appreciate your consideration of our additional experiments and analyses; we are glad that they have helped address your questions.
If possible, we would be immensely grateful if you might also consider adjusting your overall Rating and/or Confidence to reflect the current assessment.
Follow-Up on Author Response (Discussion Period Deadline: Nov. 18)
Dear Reviewer goqS,
Your comments on coder-tester training dynamics have been particularly helpful in improving the completeness of our experimental analysis. In response, we have added additional experimental results on complete training trajectories across all four tasks and the requested 2×2 reward ablation study.
As the author-reviewer discussion deadline approaches (Nov. 18), we would greatly appreciate it if you could consider our detailed rebuttal and additional results when finalizing your assessment.
Thank you again for your time and consideration.
Best regards,
Authors of CCE