- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new approach for evaluating and selecting unit tests in RLVR for code generation. The probelm it addresses is that increasing the number of unit test does not necessarily increase evaluation fidelity. Many unit tests are redundant and check siilar logic, resulting in candidate programs either passing or failing tests in bulk together. Using a 4-bit quantized 7B teacher model and a 1.5B student model, the authors empirically show that the verifier can act as the bottlneck instead of teacher's coding capability. To avoid wasting computational resources on running redundant unit tests with no clear gradient update signal, the paper proposes measuring effective constraints instead of simply counting how many unit test have been generated. Test outcomes are recorded in a binary table (signature matrix) and tests with identical pass/fail columns are grouped into same constraint cluster. The authors empirically show that selecting tests from distinct constraint clusters reduces zero-advantage issues with GRPO and improves coding performance.
Critical strengths
- The problem of advantage collapse in GRPO is an issue that is applicable across various fields not limited to coding AI. The paper addresses this issue well with empirical study
- The experiment details, those implemented, seems reasonable and well structured.
Critical weaknesses
- There's no baseline on how well this new idea performs compared to other algorithms / oracles. Having a training set that does not provide an adequate gradient signal is a long-lasting issue that was covered by various fields in computer science. Existing approaches has covered various ideas, such as curricular learning, adversarial sampling, etc, which are quite related to the paper. This paper could use some improvement in terms of baseline testing and how similar approaches has been made in the other fields of computer science to shed a better insight into the results.
- The scope of evaluation is feels a bit too small to grasp the impact of the work. For instance, the evaluation is limited to synthetic tasks and was not verified with real-world coding benchmark. The size of the models are bit too small compared to what's commonly used in coding practices today. The paper only used two models so it's difficult to differentiate whether this phenomena will extend to other models from different companies and teams. Also, Qwen 2.5 was released in 2024 so the problems this paper is addressing may no longer be relevant today.
Questions for authors
Other than critical weaknesses I have the following questions;
- What does the synthetic unit test cases look like? What are the all-pass, all-fail cases like and what are do the cases included in the training set look like? Can you share some characteristic examples?
- Does the algorithm work with other models of similar size / different size?
- How much of the zero-advantage issue are current SOTA models suffering?
7 comments
Examples of tests (Q1)
One task from the training set asks for a HealthProfileService class with the methods updateHealthProfile, classifyHealthProfile and isHealthy. The seventeen distinct programs that π0 wrote for it split its tests into the three kinds of clusters. A test that always passes calls classifyHealthProfile("HealthyProfile") and only checks that the result is not null, so every program passes it. A test that varies checks isHealthy("JustAboveBoundaryProfile") with assertTrue, and nine of the seventeen programs pass it. A test that always fails calls updateHealthProfile("StateTransitionProfile") and then expects classifyHealthProfile("StateTransitionProfile") to return "StateTransitionProfile". No program does this, and the task description does not ask for it either, because this expected value was set by the generator's rule for void methods (see the general comment). The redundant arm reads three tests of the first kind, so every program gets the same score on the tests. The diverse arm reads one test of each kind and can separate the nine programs that pass the second test from the eight that fail it. More generally, among the tests read by the diverse arm, most of the always passing tests use assertNotNull (58.6%) or assertDoesNotThrow (14.8%), and most of the always failing ones use assertEquals with an exact value (74.8%). The tests that vary are a mix of assertEquals (33.1%), assertNotNull (29.5%) and assertNull (18.7%).
Other models (Q2)
So far we have trained one other model, and the effect is larger there. Because training is the expensive part, we repeated only the diverse and redundant arms on Qwen3 1.7B, a model from the next Qwen generation, with one seed. Both arms start from the released model with thinking disabled and read the same three tests per task as before. The released model has a pass@1 of 4.1%, and after training the diverse arm reaches 9.5% and the redundant arm 6.3%, a paired difference of 3.2 points (95% interval [2.5, 3.9]). The gain of the diverse arm is about 2.5 times that of the redundant arm, while with π0 it was 1.2 to 1.4 times. Unlike with π0, the diverse arm also solves more tasks within twenty attempts, with a pass@20 of 13.4% against 9.9% (7.8% for the released model). As with π0, the gain comes from the 417 suites that a 32B model can pass, with 14.3 points there and 0.1 on the remaining 1,490.
We hope these answers address your concerns. If so, we would be grateful if you would consider raising your score, and we are happy to answer any further questions.
Reviewer h3qT: are you convinced, particularly about the baselines?
Thank you for the review, and in particular for the pointers to related work outside code generation, which we have incorporated into the revised paper.
Two statements in the summary no longer hold in the revised paper, both because of errors on our side. First, the comparison between teacher and student does not show that the verifier is the bottleneck. Once their programs compile, both models pass the full suite at almost the same rate (72.6% and 72.3%), so we removed this claim. Second, the diverse selection did not reduce the share of groups with zero advantage. In the training logs this share is 30.0% for the diverse arm and 29.8% for the redundant arm. The general comment explains these corrections and describes the corrected experiment, in which the diverse selection still beats the redundant one in every seed.
Baselines and related work (W1)
Section 2 now discusses curricula and difficulty filters (Foster et al., 2025, and Bae et al., 2026), dynamic sampling (Yu et al., 2025) and item analysis of LLM benchmarks (Kipnis et al., 2025). It also covers local independence and testlet models from psychometrics (Yen, 1993, and Bradlow et al., 1999) and query by committee from active learning (Seung et al., 1992). The work closest to adversarial sampling trains models to write tests that expose faulty code (Wang et al., 2025, and Prasad et al., 2025). Most of these methods decide which tasks or groups are used in training. Our selection keeps the task fixed and only changes which of its tests the reward reads, so these methods could be combined with any of our arms. For this kind of selection the natural baseline is a random draw of three tests from the same suite, which we added to the corrected experiment (answer (b) in the general comment). The random draw reaches a pass@1 of 10.3%, between the redundant selection (9.5%) and the diverse selection (11.0%), and the diverse selection is better in every seed.
Scope (W2)
We agree that the scope is limited. The training experiment uses one model family, one programming language and synthetic tasks, and Section 7 now reflects this. We use synthetic tasks because the selection needs suites with at least three clusters of at least three tests in every task, and our generator writes twenty tests per task for this reason. A public benchmark would need suites of this size, which we leave for future work. Model size is limited by compute, since the fifteen runs of the corrected experiment at 1.5B already took about 780 GPU hours. The measurement of Section 4 does not require training and covers three models, the base Qwen2.5 Coder 1.5B, the 7B teacher and the trained student. Among their programs that pass at least one test of a five test suite, 70.5%, 74.7% and 74.7% pass all five. As for the age of Qwen2.5, our answers to Q3 below and to Q2 in the next comment use more recent models.
Zero advantage in recent models (Q3)
It is still a common problem. He et al. (2026) report complete advantage collapse in 28% to 45% of the training batches for Qwen2.5 models from 0.5B to 14B on mathematical reasoning. Nie et al. (2026) find 69% of groups with zero advantage for Qwen3.5 9B on GSM8K with four samples per group. In our runs, which also use four samples per group, 30% of the groups are tied in the submitted runs and between 33% and 49% across the five arms of the corrected experiment.