acceptodds
Under review as a conference paper at ICLR 2027

Count the constraints, not the tests: verifier diversity governs reinforcement learning from execution feedback

Abstract

Reinforcement learning from verifiable rewards (RLVR) trains a code policy against a test suite, so what it can learn is bounded by the distinctions that suite draws. Practice summarises a suite by the number of tests it holds. If no sampled program separates two tests, they state the same constraint, and a reward computed from twenty of them still takes only two values. We measure the distinctions instead, grouping a suite's tests by the pass and fail column they produce over twenty programs from a fixed checkpoint, and counting the distinct columns, the effective constraint count. The gap is wide in practice: among generations that pass at least one test of a five test suite, three quarters pass all five, a share that barely moves across three different models. We then hold the tasks and the test count fixed and vary only whether the three tests the reward reads come from three clusters or from one, over tasks with held out and three seeds per arm. The pass@1 rises from to , with every diverse seed above every redundant seed, while pass@20 moves by points, so the effect is on reliability, not reach. The count needs only samples from a checkpoint that already exists, so a suite can be judged before compute is spent on it. Count the constraints, not the tests.

Primary area:
reinforcement learning
Keywords:
RLVR, GRPO, code generation, reward design, test redundancy

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_h3qTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new approach for evaluating and selecting unit tests in RLVR for code generation. The probelm it addresses is that increasing the number of unit test does not necessarily increase evaluation fidelity. Many unit tests are redundant and check siilar logic, resulting in candidate programs either passing or failing tests in bulk together. Using a 4-bit quantized 7B teacher model and a 1.5B student model, the authors empirically show that the verifier can act as the bottlneck instead of teacher's coding capability. To avoid wasting computational resources on running redundant unit tests with no clear gradient update signal, the paper proposes measuring effective constraints instead of simply counting how many unit test have been generated. Test outcomes are recorded in a binary table (signature matrix) and tests with identical pass/fail columns are grouped into same constraint cluster. The authors empirically show that selecting tests from distinct constraint clusters reduces zero-advantage issues with GRPO and improves coding performance.

Critical strengths
  • The problem of advantage collapse in GRPO is an issue that is applicable across various fields not limited to coding AI. The paper addresses this issue well with empirical study
  • The experiment details, those implemented, seems reasonable and well structured.
Critical weaknesses
  • There's no baseline on how well this new idea performs compared to other algorithms / oracles. Having a training set that does not provide an adequate gradient signal is a long-lasting issue that was covered by various fields in computer science. Existing approaches has covered various ideas, such as curricular learning, adversarial sampling, etc, which are quite related to the paper. This paper could use some improvement in terms of baseline testing and how similar approaches has been made in the other fields of computer science to shed a better insight into the results.
  • The scope of evaluation is feels a bit too small to grasp the impact of the work. For instance, the evaluation is limited to synthetic tasks and was not verified with real-world coding benchmark. The size of the models are bit too small compared to what's commonly used in coding practices today. The paper only used two models so it's difficult to differentiate whether this phenomena will extend to other models from different companies and teams. Also, Qwen 2.5 was released in 2024 so the problems this paper is addressing may no longer be relevant today.
Questions for authors

Other than critical weaknesses I have the following questions;

  • What does the synthetic unit test cases look like? What are the all-pass, all-fail cases like and what are do the cases included in the training set look like? Can you share some characteristic examples?
  • Does the algorithm work with other models of similar size / different size?
  • How much of the zero-advantage issue are current SOTA models suffering?
7 comments

Thank you for the review, and in particular for the pointers to related work outside code generation, which we have incorporated into the revised paper.

Two statements in the summary no longer hold in the revised paper, both because of errors on our side. First, the comparison between teacher and student does not show that the verifier is the bottleneck. Once their programs compile, both models pass the full suite at almost the same rate (72.6% and 72.3%), so we removed this claim. Second, the diverse selection did not reduce the share of groups with zero advantage. In the training logs this share is 30.0% for the diverse arm and 29.8% for the redundant arm. The general comment explains these corrections and describes the corrected experiment, in which the diverse selection still beats the redundant one in every seed.

Baselines and related work (W1)
Section 2 now discusses curricula and difficulty filters (Foster et al., 2025, and Bae et al., 2026), dynamic sampling (Yu et al., 2025) and item analysis of LLM benchmarks (Kipnis et al., 2025). It also covers local independence and testlet models from psychometrics (Yen, 1993, and Bradlow et al., 1999) and query by committee from active learning (Seung et al., 1992). The work closest to adversarial sampling trains models to write tests that expose faulty code (Wang et al., 2025, and Prasad et al., 2025). Most of these methods decide which tasks or groups are used in training. Our selection keeps the task fixed and only changes which of its tests the reward reads, so these methods could be combined with any of our arms. For this kind of selection the natural baseline is a random draw of three tests from the same suite, which we added to the corrected experiment (answer (b) in the general comment). The random draw reaches a pass@1 of 10.3%, between the redundant selection (9.5%) and the diverse selection (11.0%), and the diverse selection is better in every seed.

Scope (W2)
We agree that the scope is limited. The training experiment uses one model family, one programming language and synthetic tasks, and Section 7 now reflects this. We use synthetic tasks because the selection needs suites with at least three clusters of at least three tests in every task, and our generator writes twenty tests per task for this reason. A public benchmark would need suites of this size, which we leave for future work. Model size is limited by compute, since the fifteen runs of the corrected experiment at 1.5B already took about 780 GPU hours. The measurement of Section 4 does not require training and covers three models, the base Qwen2.5 Coder 1.5B, the 7B teacher and the trained student. Among their programs that pass at least one test of a five test suite, 70.5%, 74.7% and 74.7% pass all five. As for the age of Qwen2.5, our answers to Q3 below and to Q2 in the next comment use more recent models.

Zero advantage in recent models (Q3)
It is still a common problem. He et al. (2026) report complete advantage collapse in 28% to 45% of the training batches for Qwen2.5 models from 0.5B to 14B on mathematical reasoning. Nie et al. (2026) find 69% of groups with zero advantage for Qwen3.5 9B on GSM8K with four samples per group. In our runs, which also use four samples per group, 30% of the groups are tied in the submitted runs and between 33% and 49% across the five arms of the corrected experiment.

Examples of tests (Q1)
One task from the training set asks for a HealthProfileService class with the methods updateHealthProfile, classifyHealthProfile and isHealthy. The seventeen distinct programs that π0 wrote for it split its tests into the three kinds of clusters. A test that always passes calls classifyHealthProfile("HealthyProfile") and only checks that the result is not null, so every program passes it. A test that varies checks isHealthy("JustAboveBoundaryProfile") with assertTrue, and nine of the seventeen programs pass it. A test that always fails calls updateHealthProfile("StateTransitionProfile") and then expects classifyHealthProfile("StateTransitionProfile") to return "StateTransitionProfile". No program does this, and the task description does not ask for it either, because this expected value was set by the generator's rule for void methods (see the general comment). The redundant arm reads three tests of the first kind, so every program gets the same score on the tests. The diverse arm reads one test of each kind and can separate the nine programs that pass the second test from the eight that fail it. More generally, among the tests read by the diverse arm, most of the always passing tests use assertNotNull (58.6%) or assertDoesNotThrow (14.8%), and most of the always failing ones use assertEquals with an exact value (74.8%). The tests that vary are a mix of assertEquals (33.1%), assertNotNull (29.5%) and assertNull (18.7%).

Other models (Q2)
So far we have trained one other model, and the effect is larger there. Because training is the expensive part, we repeated only the diverse and redundant arms on Qwen3 1.7B, a model from the next Qwen generation, with one seed. Both arms start from the released model with thinking disabled and read the same three tests per task as before. The released model has a pass@1 of 4.1%, and after training the diverse arm reaches 9.5% and the redundant arm 6.3%, a paired difference of 3.2 points (95% interval [2.5, 3.9]). The gain of the diverse arm is about 2.5 times that of the redundant arm, while with π0 it was 1.2 to 1.4 times. Unlike with π0, the diverse arm also solves more tasks within twenty attempts, with a pass@20 of 13.4% against 9.9% (7.8% for the released model). As with π0, the gain comes from the 417 suites that a 32B model can pass, with 14.3 points there and 0.1 on the remaining 1,490.

We hope these answers address your concerns. If so, we would be grateful if you would consider raising your score, and we are happy to answer any further questions.

Reviewer h3qT: are you convinced, particularly about the baselines?

Reviewer_841fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how test-suite diversity affects reinforcement learning from execution feedback, grouping tests by their pass/fail signatures over programs sampled from a fixed checkpoint. A controlled experiment trains the same student on the same tasks with three tests drawn either from distinct clusters or from one cluster, improving held-out pass@1 from 7.1% to 10.0% across three training seeds, with a smaller change in pass@20.

Critical strengths

The paper addresses a useful question about the verifier itself. Counting tests can hide substantial behavioral redundancy, and the signature analysis provides a simple way to inspect what a suite distinguishes on samples from the current policy.

The main experiment controls several important factors: both arms share the training tasks, initial checkpoint, test count, optimization settings, and training duration. All three diverse runs outperform all three redundant runs on pass@1, and the task-paired confidence intervals exclude zero. This provides convincing evidence for the benefit of the proposed selection rule in the tested setting.

The analysis of the initial between-population comparison is valuable. The authors identify trivial all-void APIs and show that suite features predict population membership with an AUC of 0.86. Recognizing this confounding and moving to a within-task comparison strengthens the empirical contribution.

Critical weaknesses

I think the paper overstates what the effective constraint count determines. Distinct test columns need not produce distinct aggregate rewards: columns (1,0,0), (0,1,0), and (0,0,1) form three clusters but give every sampled program the same test-pass reward. The diverse selection rule also changes which conditions are tested and their pass frequencies, so the experiment does not isolate column count as the quantity governing reward resolution. A random three-test baseline and the resulting reward distributions would help clarify what cluster counting adds in practice.

The dataset validation protocol is internally inconsistent. Section 3.1 says a task is retained only when a teacher implementation compiles and passes every test, while Appendix C describes compilation against a signature-correct stub and explicitly says this does not establish that a program passes the suite. These procedures provide very different guarantees about the training reward and evaluation target. The authors should clarify which procedure was used for the controlled experiment and how inconsistent assertions were handled.

The conclusions about capability limits go beyond the results. In Table 2, the teacher’s 29.1% pass@1 accompanies a 40.0% compilation rate, while the student’s 66.6% accompanies 92.1%; conditional on compilation, their full-suite success rates are approximately 72.8% and 72.3%, so this comparison does not establish that the verifier is the binding bottleneck. Similarly, a small aggregate pass@20 difference does not show that both arms solve the same tasks or share an unchanged capability ceiling. The results support improved single-attempt reliability under the tested verifier, but the stronger capability interpretation remains unclear.

Questions for authors

How does the measured column count relate to the number and frequencies of distinct aggregate rewards on fresh rollouts? How does diverse selection compare with randomly choosing three tests from the same suites?

For the redesigned dataset used in the controlled experiment, was every retained full suite passed by a teacher implementation, or was validation limited to compilation against a stub? Please reconcile Section 3.1 with Appendix C.

What is the overlap between the tasks solved by the two arms within twenty attempts? After accounting for the compilation-rate difference in Table 2, what evidence supports the claimed capability bottleneck beyond improved reliability?

5 comments

Thank you for the review. You were right about the inconsistency between Section 3.1 and Appendix C and about the conditional rates in Table 2, and both are corrected in the revised paper.

Two corrections we must address. The arms did start from the same checkpoint, but it was a new LoRA adapter on the base model and not π0, the checkpoint that defines the clusters. The training prompt also differed between the arms in 82.8% of the tasks. We fixed both problems and reran the experiment, which now also includes the random baseline you asked for (see the general comment). With these corrections, pass@1 is 9.5% for the redundant arm and 11.0% for the diverse arm, compared to 7.1% and 10.0% in the submission, and the diverse arm is still better in every seed.

Column count and distinct rewards (W1, Q1).
We agree that the count only gives an upper bound, and Section 3.2 now says this. With three tests, one cluster allows at most two values of the test term and three clusters at most four. To see how this works in practice, we counted the distinct values on the π0 samples that define the clusters. Over the 7,628 training tasks, the diverse term takes one, two, three or four values in 8.8%, 45.2%, 37.5% and 8.5% of the tasks, and the 8.8% are your case of columns that offset each other. The redundant term takes one or two values in 25.0% and 75.0% of the tasks. We repeated the count on twenty new π0 samples for 1,000 of the tasks, of which 999 have a program that runs. There the diverse shares are 11.3%, 46.1%, 34.6% and 7.9%, against 7.8%, 44.2%, 39.1% and 8.8% on the original samples of the same tasks. The redundant term takes one or two values in 86.3% of them, and in the other tasks its cluster splits under the new samples. You are also right that the diverse rule changes which conditions are tested and how often they pass. This is why we added the matched redundant arm and the random arm (answers (a) and (b) in the general comment). The matched arm reads tests with pass rates close to those of the diverse tests and does no better than the redundant arm, and the diverse arm beats it by 1.4 to 1.8 points of pass@1. Against a random draw of three tests the diverse arm gains 0.4 to 0.9 points in every seed.

Overlap and capability (W3, Q3).
We agree that the results only support better reliability in a single attempt, and the revised paper no longer makes claims about capability limits. Within twenty attempts the two submitted arms solve mostly the same tasks. In seeds 42, 104171 and 123456789, 249, 257 and 251 of the 1,907 held out tasks are solved by both arms, 35, 28 and 36 only by the diverse arm, and 26, 15 and 16 only by the redundant arm. The corrected runs look similar. Over the three seeds together, 318 tasks are solved by both arms, 18 only by the diverse arm and 21 only by the redundant arm. Section 7 now claims only better single-attempt reliability under the tested verifier, and we removed the sentence that said the base model sets the ceiling. The teacher and student comparison moved to Appendix F (Table 4) with the conditional rates you computed. If we count the generations themselves, they are 72.6% and 72.3%, and the paper no longer says that the suite is the bottleneck. The matched redundant arm does reach clearly fewer tasks, with a pass@20 of 14.3%, and against it the counts are 292, 44 and 14.

Validation of the redesigned suites (W2, Q2).
The validation was limited to compiling each suite against a stub with the declared signatures, as Appendix C describes, and inconsistent assertions were not filtered. The check with a teacher implementation applies only to the original suites. The submitted Section 3.1 did not make this distinction, and the revised version says which check applies to which data (answer (c) in the general comment). It also reports that 52.0% of the populated redesigned suites have a test that no sampled program passes. Some of these tests come from a rule in the generator for void methods. When the teacher gives no observer, the generator calls a method of the class that returns a value and sets the expected value with a fixed rule, and this rule wrote 46.5% of the held out tests (revised Section 4). Even so, a 32B model with test feedback passes all twenty tests in 417 of the 1,907 held out suites, and programs sampled from the 1.5B models pass another 86. On those 417 suites the corrected runs reach a pass@1 of 47.5% for the diverse arm and 41.1% for the redundant arm, while on the other 1,490 every arm stays below 0.8% on average. All arms are evaluated on the same suites.

We hope this answers your questions. If it does, we would be grateful if you would consider raising your score, and we are glad to discuss anything else.

Reviewer 841f: are you convinced by the response, particularly about the conclusions drawn from the count?

Partly addressed. I’m still not convinced that the count itself explains the improvement. Matching individual test pass rates does not control which behaviors the tests cover or how their outcomes co-vary. The gain over random three-test selection is 0.4–0.9 percentage points across seeds. I’m comfortable with the narrower empirical claim about the selection rule, but I would avoid attributing the gain specifically to the count.

Reviewer_oYMfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that what RLVR on code can learn is bounded by how many independent constraints a test suite actually imposes, not by how many tests it contains. It counts constraints by grouping a suite's tests according to the set of pass and fail results they produce over programs sampled from a fixed checkpoint: tests whose results match cannot separate any of those programs, so they state the same constraint. The measurement finds that the original suites behave almost like a single constraint, and that a redesigned generator raises the median count. Holding the tasks and the number of training tests fixed, the paper compares suites whose tests come from distinct clusters with suites whose tests come from one cluster, and reports a pass@1 gain with pass@20 roughly flat, attributing it to partial credit reducing tied GRPO groups.

Critical strengths

1.The measurement is the right one and is cheap: computing indistinguishability against the policy's own samples rather than against mutants makes suite adequacy policy-relative and quotable before any training compute is spent, which is a genuinely useful diagnostic for practitioners.

2.The mechanism is measured rather than asserted: the share of tied groups and the within-group reward spread are reported, and the discussion is candid that the effect is small, in-distribution, single-family, and that nothing about the suite reaches the optimizer while compilation dominates the reward.

Critical weaknesses

1.The diverse arm is confounded with difficulty. Tests drawn from distinct clusters are, by construction, the ones some programs pass and others fail, while a redundant cluster tends to hold tests everyone passes or everyone fails; since no pre-training matching of the chosen tests' pass rate is reported, the gain may reflect harder tests rather than more independent axes. 2.The effect is established only on the minority of suites that already discriminate: tasks enter the experiment only if their suite shows several clusters with several tests each, which is precisely the population for which the paper's advice is least needed, and the recommendation to screen suites before committing compute is not tested outside it. 3.The teacher-versus-student headline is not a clean argument that the suite is the bottleneck, since the teacher is quantised and compiles far less often, and the comparison appears to be scored on suites the student was trained against rather than held out.

Questions for authors

1.Were the tasks in the teacher–student comparison held out from the student's training, and are the two reward arms matched on the selected tests' pass rate at the starting checkpoint? 2.Since pass@20 barely moves, how do the authors separate denser credit from simply obtaining more usable gradient steps — for instance by matching the number of non-tied groups, or by training the redundant arm longer? 3.Does the constraint count predict downstream gain out of sample, or is all the evidence a single binary contrast on a filtered population? A dose–response across suites of intermediate diversity would be far more convincing than one comparison.

3 comments

Thank you for the review. The difficulty confound and the question about usable gradient steps were both good points, and the analyses we ran to answer them changed the paper.

One correction to your summary. We attributed the gain to partial credit that reduces the number of tied GRPO groups, but the training logs do not support this. In the diverse arm 30.0% of the groups are tied and in the redundant arm 29.8%, so we removed this explanation from the paper. We also found two problems in the setup. The runs started from the base model instead of π0, the checkpoint that defines the clusters, and the prompts of the two arms differed in 82.8% of the tasks. We fixed both and reran all the arms, as described in the general comment. The diverse arm still beats both redundant arms in every seed, although the gain is about half of what we reported.

Difficulty confound (W1, Q1)

On average, the two arms are matched. The three tests read by the diverse arm pass 51.2% of the π0 programs and those read by the redundant arm pass 50.9%. Both arms also use the same mix of assertions, with assertEquals in 36.6% and 36.9% of the tests and assertNotNull in 28.9% of both. Task by task they are not matched, and we agree that the redundant cluster is more often constant. It always passes in 10.1% of the training tasks and always fails in 14.9%. To test whether this explains the gain, we added a matched redundant arm (answer (a) in the general comment). In each task it reads the cluster that varies under π0 and whose pass rate is closest to that of the diverse tests, 0.11 apart on average. This arm does no better than the original redundant arm. The diverse arm beats it by 1.4, 1.8 and 1.6 points of pass@1 in the three seeds, with paired 95% intervals of [1.0, 1.7], [1.4, 2.3] and [1.3, 2.1]. The difficulty of the selected tests therefore does not explain the gap.

Usable gradient steps (Q2)

We checked this in the training logs. In the submitted runs, 70.0% of the groups in the diverse arm and 70.2% in the redundant arm are not tied and carry a gradient, and all runs train for the same 11,442 steps. The two arms therefore got the same number of usable groups, so a longer redundant run is not needed to match them. The corrected runs give the same picture, with 57.7% and 58.1%. The difference between the arms is where the reward points the update, which the revised Section 6 measures on the π0 samples before any training. A program that passes the three diverse tests passes all twenty tests in 42.6% of the cases, while for the redundant tests this happens in 9.5%. The expected increase in the probability of passing the whole suite after one small GRPO step from π0 is 1.36 times larger with the diverse reward. Over the five arms, this quantity has a correlation of 0.994 with the final pass@1.

Dose response and the filtered population (Q3, W2)

The corrected experiment now has five arms instead of two. Two of them read one cluster (the redundant and the matched redundant arm), one reads two, the random arm reads 2.3 on average and the diverse arm reads three. Over these arms pass@1 grows by 0.76 points per cluster. On average most of the gain comes from the second cluster, and the third adds about half a point. The measure of Section 6, which is computed before training, ranks the arms in the same order as their final pass@1, except that the random and two-cluster arms swap places. We agree that the population is filtered. The experiment uses the 9,535 of 27,682 redesigned suites (34.4%) that have at least three clusters of at least three tests under π0, because every arm has to draw its three tests from the same task. We have not tested whether the count predicts gains outside this population, and Section 7 now says so. The count itself does not need training and can be computed for any suite.

Teacher and student (W3, Q1)

The 415 tasks of this comparison were held out. The training script removes every task whose identifier hashes below 0.1, and none of the 415 descriptions appears in the training data. We did find that in 43 of these tasks the test suite is identical to the suite of a training task with a reworded description, and the revised Appendix F reports this. Your other point is correct. Among the programs that compile, the teacher passes the full suite in 72.6% of the cases and the student in 72.3% (70.7% and 69.8% without those 43 tasks), so the comparison does not show that the suite is the bottleneck. The comparison is now in Appendix F with these rates, and we removed the bottleneck claim from the introduction and the contributions.

We hope this addresses your concerns. If it does, we would be grateful if you would consider raising your score, and we are happy to answer any further questions.

Reviewer oYMf: are you convinced, particularly by the authors' explanations about the confounding problem?

AuthorsOpenReview ↗

On the difficulty confound

Thank you again for the review. In W1 and Q1 you raised the concern that the gain of the diverse arm may reflect harder tests rather than more clusters. Since our first reply we have run two more analyses of this confound, one on the tests that each arm reads and one on the fifteen trained models. Both use run files that are already in the anonymous repository, and the revised PDF reports them in Appendix I.

The tests are equally hard in the arms that draw at random. Our first reply gave the mean pass rate under π0, 51.2% for the diverse arm and 50.9% for the redundant arm. The whole distribution matches too. In the redundant, two cluster, random and diverse arms, 24.7% to 25.4% of the tests that the reward reads are passed by fewer than 20% of the π0 programs, and 14.5% to 14.9% by none. These four arms differ in how many clusters the tests come from, not in how hard the tests are.

The outcome follows the number of clusters. We scored the trained models on the single tests of the held out tasks, with the pass rate of each test taken from a separate evaluation of π0.

ArmClusters readTests read that fewer than 20% of π0 programs passHeld out tests of that kind passed after training
Redundant125.4%19.2%
Matched redundant112.2%19.3%
Two clusters224.9%22.5%
Random2.324.7%24.9%
Diverse324.8%27.6%

π0 passes 5.1% of these tests. The fifteen runs are ordered by the number of clusters without overlap, at 18.7% to 20.0% for the six runs that read one cluster, 21.3% to 24.5% for two clusters, 24.7% to 24.9% for random and 25.9% to 30.2% for diverse. Paired by task, the diverse arm is 8.4 points above the redundant arm (95% interval [7.4, 9.5]). On the tests that at least 80% of the π0 programs pass, the five arms are within 1.6 points. This is also the dose response of your third question, at the level of single tests.

A change of difficulty at a fixed number of clusters leaves the outcome where it was. The matched redundant arm is the one arm whose tests differ in difficulty. It reads half as many hard tests as the redundant arm and no constant cluster, and it ends where the redundant arm ends, on pass@1 (9.4% against 9.5%) and on the hard held out tests (0.2 points apart, [−0.4, 0.8]).

The design therefore varies each factor with the other held fixed. Changing the difficulty of the tests at one cluster leaves the result where it was, and reading equally hard tests from more clusters improves it. The redundant arm does not lack hard tests. It reads as many as the diverse arm, three at a time from one cluster, where the second and third repeat the first.

Reviewer_tCRTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether test-suite diversity matters more than test count when training code models with execution feedback. The authors group tests according to their pass/fail behavior on programs sampled from an initial checkpoint and call the number of groups the effective constraint count.

The main experiment compares training with three tests from different groups against three tests from the same group, while keeping the tasks, starting checkpoint, and training settings fixed. Across three seeds, diverse selection improves held-out pass@1 from 7.1% to 10.0%, while pass@20 increases by approximately 0.7 percentage points. The authors also report fewer zero-advantage groups during training and propose this as an explanation for the improvement.

Critical strengths
  1. The main experiment is carefully controlled.

    Both arms use the same tasks, starting checkpoint, number of tests, and training settings, and they share one test per task. Every diverse run outperforms every redundant run, and the task-paired bootstrap intervals consistently exclude zero. These controls make the reported improvement worth taking seriously.

    I also appreciate the authors’ explanation of why comparing different task populations would be misleading. Moving to a within-task comparison is a sensible response to that problem.

  2. The paper examines an important part of the training setup.

    Test count is an appealing but incomplete description of a verifier. The paper gives a concrete way to inspect behavioral redundancy and connects that observation to downstream training.

    The most useful contribution is the combination of policy-based exact test grouping and the controlled three-test intervention. This builds on existing behavioral approaches to test selection, including the cited RobustTests work, while providing a focused experimental comparison.

Critical weaknesses
  1. The proposed count does not, by itself, establish reward informativeness or the claimed mechanism.

    Grouping identical columns gives a reasonable measure of observed redundancy. However, different columns do not necessarily represent independent constraints or create more reward contrast.

    For example, suppose three programs have pass vectors (1,0,0), (0,1,0), and (0,0,1). All three test columns differ, yet every program receives the same fraction-passed reward of 1/3. This limits the claim in Section 3.2 that cluster count determines reward resolution.

    Constant columns create a related concern about the experiment. Always-passed and always-failed tests count as clusters under the definition. A suite could therefore contain three clusters consisting of always-passed tests, always-failed tests, and one varying pattern. Diverse selection would include the varying signal, while redundant selection could choose a constant cluster. Some of the improvement could come from avoiding uninformative rewards rather than combining multiple informative constraints.

    I would find the interpretation more convincing with an analysis excluding constant clusters, comparisons matched on marginal test pass rates, and a random three-test baseline. The reduction in zero-advantage groups is useful supporting evidence, but separating the test, style, and compilation reward components would help establish the explanation.

  2. The guarantee that the generated suites are valid and satisfiable is unclear.

    Section 3.1 says a task is retained only when a teacher-generated implementation passes every test. Appendix C instead describes checking whether the suite compiles against a signature-correct stub and explicitly says this does not guarantee a passing implementation.

    I could not tell whether these procedures apply to different datasets or whether both are used for the main experiment. Since the generated suites also define evaluation correctness, this distinction matters to my assessment.

    The authors should explain which checks apply to the main dataset and what evidence establishes that its suites are jointly satisfiable and consistent with their task descriptions. The fact that twenty sampled programs fail a test would not, by itself, establish that the test is invalid.

Questions for authors
  1. Clarification: Does every suite in the main experiment have a verified implementation that passes all tests? How do the procedures in Section 3.1 and Appendix C fit together?

  2. Clarification and supporting analysis: How often does the redundant arm select an always-passed or always-failed cluster? Does the improvement persist when these clusters are excluded or marginal test pass rates are matched?

  3. Comparison affecting practical significance: Do you have results for uniformly random three-test selection? This would help determine how much the proposed selection strategy improves over a straightforward baseline.

  4. Measurement and mechanism: How stable are the partitions on fresh samples and later checkpoints? Can the reported reduction in zero-advantage groups be separated into contributions from test outcomes, compilation, and style?

5 comments

Thank you for the careful review. Your (1,0,0) example was right, and so was your worry about constant clusters.

Three points in your summary describe the paper correctly but not the runs behind it. The runs started from a new LoRA adapter on the base model, not from π0, the checkpoint that defines the clusters. The prompt differed between the two arms in 82.8% of training tasks. And the share of groups with zero advantage did not fall. The logs show 30.0% against 29.8%, so we withdraw that explanation. In the corrected experiment the diverse arm still beats both redundant arms in every seed, though at about half the submitted size: 1.4 points of pass@1 against 2.9.

How often does the redundant arm read a constant cluster? (W1, Q2)
In 25.0% of training tasks. Under π0 that cluster always passes in 10.1% of them and always fails in 14.9%. The diverse tests cannot all be constant, because a suite has at most two constant columns and they come from three clusters. Your hypothesis was reasonable, so we tested it. Pass rates had been matched only on average (51.2% diverse, 50.9% redundant), and the corrected experiment adds the matched redundant arm, answer (a) of the general comment. It reads only clusters that vary under π0, and in each task it takes the one whose pass rate is closest to that of the diverse tests. It does no better than the redundant arm (−0.33, +0.08 and −0.25 points), so constant clusters do not explain the gap. Against it, the diverse arm still gains 1.4, 1.8 and 1.6 points of pass@1, with paired 95% intervals of [1.0, 1.7], [1.4, 2.3] and [1.3, 2.1].

Do you have results for a random draw of three tests? (Q3)
Yes, answer (b) of the general comment. The random arm draws three of the twenty tests uniformly, and its pass@1 of 10.3% sits between the redundant arm at 9.5% and the diverse arm at 11.0%. The diverse arm beats it in every seed, by 0.4, 0.9 and 0.6 points, while the random arm in turn beats the redundant arm in every seed.

Does every suite in the main experiment have a verified implementation? (W2, Q1)
No. The submitted Section 3.1 implied otherwise, and that was wrong. Only the original suites of Section 4 were checked with a teacher implementation. The redesigned suites of the experiment were only compiled against a stub of the class, as Appendix C describes, and the revised Section 3.1 says which check applies to which data. It also reports that 52.0% of the populated redesigned suites hold a test that no sampled program passes. As you say, that alone does not make a test invalid. A 32B model with test feedback passes all twenty tests in 417 of the 1,907 held out suites, and sampled 1.5B programs pass 86 more (answer (c) of the general comment). Agreement with the description is weaker ground. Nothing in the pipeline checks it. When the teacher tests a void method without naming an observer, the generator sets the expected value by a fixed rule, and that rule wrote 46.5% of the held out tests (revised Section 4). Every arm is scored on the same suites, though, so an impossible test lowers pass@k for all of them on the same tasks.

Distinct columns need not give distinct rewards (W1).
Agreed. The revised Section 3.2 treats the number of clusters as a bound on the values the test term can take, at most two with one cluster and four with three. Cases like your example make up 8.8% of training tasks on the π0 samples. In the rest the diverse term takes two, three or four values (45.2%, 37.5% and 8.5%). The paper also now calls clusters distinct constraints, not independent ones.

How stable are the partitions, and where does the drop in ties come from? (Q4)
More stable in structure than in detail. With twenty new π0 samples on 1,000 training tasks, the exact partition recurs in only 18.4% of them, but 89.8% of the pairs of tests stay together or apart as before. The diverse tests still span three clusters in 77.6% of tasks, and the redundant tests stay in one cluster in 81.7%. We did not recompute partitions at later checkpoints. The selection is fixed on π0 before training, and Section 7 lists the dependence on the checkpoint as a limit. As for the ties, there is no drop to decompose. The corrected logs show 42.3% of groups tied for diverse and 41.9% for redundant. On the π0 samples, before training, the test term alone ties 20.4% of groups of four for diverse and 43.8% for redundant, compilation brings this to 19.2% and 28.0%, and style to 13.5% and 19.0%. So the reward separates more answers at the start, but not during training. Section 6 now looks instead at where the reward points the update (Mechanism, in the general comment).

If this settles your concerns, we hope you will reconsider your score. If anything is still unclear, we are happy to answer.

Reviewer tCRT: are you convinced by the authors' response, particularly as regards whether reward informativeness is sufficiently established?

AuthorsOpenReview ↗

Reward informativeness on the trained models

Thank you once again for your review. We would like to further clarify some details on your first point (W1) which stated that the count does not, by itself, establish reward informativeness or the mechanism. Section 6 and our general comment measured the mechanism on the samples of π0, before training. We have now tested it on the fifteen trained models, without new training or sampling, and the revised PDF adds this as Appendix I.

On the held out tasks we rebuilt the three tests that each arm would read, with the selection code and seed of the experiment. The clusters come from a separate evaluation of π0 with twenty samples per task, so no sample is scored on a cluster it helped define. Of the 1,907 held out tasks, 1,476 have at least three clusters of three or more tests and enter the analysis. None of the fifteen models was trained on these tasks, and in every arm the prompt lists the method signatures of the whole suite and no test. A model therefore cannot tell which tests its reward reads, and its pass rate on them measures what the reward taught it. The table gives the share of samples that pass each set of tests, as the mean over the three training seeds.

ModelOne cluster, matched rule (%)Three clusters, diverse rule (%)All twenty tests (%)
π042.36.82.8
Redundant71.815.67.5
Matched redundant74.615.67.5
Diverse74.718.78.9

The matched redundant model satisfies the kind of constraint its reward reads as often as the diverse model does. Paired by task and seed, the diverse model passes the matched tests 0.1 points more often (95% interval [−0.7, 1.0]), and the two stay within one point in each of four bins of the cluster's pass rate under π0. The diverse model pulls ahead when several constraints must hold together, by 3.1 points on three clusters ([2.6, 3.7]) and by 1.4 points on the whole suite ([1.1, 1.8]). The redundant arm, whose cluster is constant under π0 in 25.0% of training tasks, is already 2.9 points behind on one cluster ([2.2, 3.5]). The matched arm closes this gap and still loses the same 1.4 points on the suite. If its reward carried too little signal, the matched arm would also fall behind on the constraint it reads. It does not, and it falls behind as soon as other constraints must hold as well.

The precision of Section 6 also holds after training. A held out program that passes the three tests of its own arm passes all twenty in 47.7% of cases for the diverse model, 25.7% for random, 24.0% for two clusters, 12.4% for redundant and 10.1% for matched redundant, the same order as on π0 in Section 6. Clusters from another evaluation of π0 give the same conclusions.

Comments on the paper

6 comments
Area_Chair_91PrOpenReview ↗

From AC.

Dear Authors,

Given the deadline on 18 November, please submit your responses to the reviews to facilitate a productive discussion with the reviewers.

Thanks, AC

AuthorsOpenReview ↗

General comment (1/3)

Thank you for the reviews. The submitted experiment had two errors in its setup: its runs started from the base model instead of π0, the checkpoint that defines the clusters, and its two arms saw different prompts in most tasks. We fixed both and reran every arm under a cleaner protocol. We also changed the evaluation, which shared one sampling seed across each batch of 256 prompts and now gives each task its own (Appendix E). The effect holds at half the size, 1.4 points of pass@1 against 2.9 in the submission. The diverse arm still beats both redundant arms in every seed: the original one and a new matched redundant arm. The matched arm reads three tests from one cluster that varies under π0 and has the pass rate closest to that of the diverse arm.

A cluster is a set of tests that pass and fail together on twenty samples of π0 per task. Every arm trains on the same 7,628 tasks. The arms differ in which three tests of each task the reward reads.

Three questions recur across the reviews: (a) whether a redundant cluster can be constant, (b) how our selection compares with a random draw of tests, and (c) how the suites were validated. The revised Section 5 adds the matched redundant arm for (a) and, for (b), a random arm that draws three of the twenty tests uniformly. Sections 3.1 and 4 answer (c). The second comment gives each answer under Results, with the same letter.

A fifth arm reads two clusters. The table gives all five arms on the 1,907 held out tasks, with twenty samples per task. Each arm shows the mean over its three seeds, with their range in brackets.

ArmClusters readpass@1 (%)pass@20 (%)Compile (%)Mean test pass (%)
π0, start4.112.079.341.5
Redundant19.5 [9.4, 9.6]15.6 [15.4, 15.7]98.8 [98.7, 98.8]63.7 [63.7, 63.7]
Matched redundant19.4 [9.3, 9.5]14.3 [13.9, 14.5]98.7 [98.7, 98.8]63.9 [63.7, 64.1]
Two clusters210.5 [10.2, 10.7]15.9 [15.5, 16.4]98.7 [98.6, 98.8]65.2 [64.7, 65.8]
Random2.310.3 [10.3, 10.4]15.7 [15.4, 16.0]98.8 [98.7, 98.9]65.9 [65.9, 65.9]
Diverse311.0 [10.7, 11.3]16.0 [15.3, 16.7]98.8 [98.8, 98.9]67.2 [66.7, 68.1]

Against the matched redundant arm, the diverse arm gains 1.4, 1.8 and 1.6 points of pass@1 in the three seeds, and every paired 95% interval is above zero.

What was wrong

The submission says that every run continued from π0, whose pass@1 is 4.1% (introduction, Section 3.1, Appendix D). The runs instead trained a new LoRA adapter on the base model, whose pass@1 is 0.19%. π0 only defined the clusters.

The training prompt listed only the method signatures that the three selected tests call, so the prompts of the two arms differed in 82.8% of training tasks. The comparison thus changed the prompt as well as the reward. We reran seed 42 with the submitted setup and the full prompt in both arms. The gap fell from 3.4 to 2.2 points (95% interval [1.7, 2.6], Appendix G).

The evaluation gave one sampling seed to each batch of 256 prompts. Appendix E shows that this sharing leaves the submitted intervals valid.

The submission describes a check in which a teacher implementation passes every test (introduction, Section 3.1, Figure 1). That check applies only to the original suites. We checked the redesigned suites of the experiment only by compiling them against a stub.

We also withdraw three claims. The introduction, Contribution 4 and Section 6 no longer say that diverse tests lower the share of GRPO groups with zero advantage from 50.8% to 43.4%, since the logs show 30.0% against 29.8%. Section 7 no longer says that the base model sets the ceiling, which 841f found unsupported. The paper no longer concludes that the suite, and not the teacher, is the bottleneck, since the two pass at 72.6% and 72.3% once their programs compile (Appendix F).

What we did

Every arm now starts from π0 and trains a new LoRA adapter. Every arm also sees the evaluation prompt, which lists the method signatures of the whole suite. We checked this in all 7,628 training tasks. The diverse and redundant arms read the same three tests as in the submission. On average, the pass rate of the matched arm's cluster is 0.11 from that of the diverse arm. Each arm has three seeds. We also trained the diverse and redundant arms on Qwen3 1.7B, from its released weights, with one seed.

AuthorsOpenReview ↗

General comment (2/3)

Results

(a) Under π0, the redundant cluster is constant in 25.0% of training tasks. It always passes in 10.1% of training tasks and always fails in 14.9%. The matched arm avoids such clusters. It still does no better than the redundant arm: the differences are −0.33, +0.08 and −0.25 points. Constant clusters therefore do not explain the gap. Against the matched arm, the paired 95% intervals of the diverse arm are [1.0, 1.7], [1.4, 2.3] and [1.3, 2.1]. Against the redundant arm, its gains are 1.0, 1.9 and 1.4 points.

(b) The random arm reaches a pass@1 of 10.3%. The diverse arm beats it in every seed, by 0.4 to 0.9 points. Every interval is above zero. Over the five arms, pass@1 rises by 0.76 points per cluster that the reward reads. On average, most of the gain comes with the second cluster.

(c) Section 3.1 now reports that 52.0% of the populated redesigned suites contain a test that no sampled program passes. We evaluate every arm on the same suites, so such a test lowers pass@k for every arm on the same tasks. A 32B model with test feedback passes all twenty tests in 417 of the 1,907 held out suites. Samples from the 1.5B models pass 86 more. On the 417, pass@1 is 47.5% for the diverse arm and 41.1% for the redundant arm. On the other 1,490, every arm stays below 0.8%. Section 4 now describes how the generator writes tests of void methods, a rule behind 46.5% of the held out tests.

Against the redundant arm, the gain is in single-attempt reliability. pass@20 moves by only 0.4 points, and within twenty attempts 318 tasks are solved by both arms, 18 only by the diverse arm and 21 only by the redundant arm. The matched redundant arm reaches fewer tasks (pass@20 1.7 points lower), with counts of 292, 44 and 14.

Section 6 looks for the cause in the π0 samples behind the clusters. A program that passes the three diverse tests passes all twenty in 42.6% of cases, against 9.5% for the redundant tests, so full credit from one cluster goes mostly to programs that do not pass the whole suite. Before any training, a first order estimate of how a small update from π0 changes the chance of passing all twenty ranks the arms as their final pass@1 does, except that the random and two-cluster arms swap places.

On the second model, Qwen3 1.7B, the effect is larger. The diverse arm beats the redundant arm by 3.2 points of pass@1 (95% interval [2.5, 3.9]), and unlike with π0 it also raises pass@20, to 13.4% against 9.9%.

Revised PDF and code

The third comment lists every change with its place in the revised PDF. The anonymous repository linked in the paper was updated on 12 November. The corrected experiment and the other new runs are in 'runs/new-ablations/' (see its README), the submitted runs are in 'runs/diversity-experiment/', and the generator of the redesigned suites is in 'generation/'.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.