acceptodds
Under review as a conference paper at ICLR 2027

LUA: Functional Alignment of Large Code Models with Streaming Unit Tests

Abstract

Large code models (LCMs) have shown strong performance in solving practical software engineering tasks. However, the generated code may not fully align with the user's specification, motivating further alignment steps through reinforcement learning (RL) based finetuning. Recent RL-based finetuning methods for LCMs exploit code-specific properties, particularly the executable property, by using feedback from unit test executions. Yet, existing methods use a limited set of unit tests. In this paper, we investigate how to effectively leverage a large number of unit tests by rethinking curriculum learning as a unit test selection problem. Whilst prior work has applied curriculum learning to order training problems by difficulty, we utilize it to identify informative unit tests from a large continuously generated pool. To this end, we propose a lightweight test-difficulty learning layer that estimates the informativeness of individual unit tests, together with an adaptive decoding strategy that selects the tests according to the learned informativeness scores. In particular, we study the scaling behavior of unit-test feedback, the efficiency of test selection, and the need for adaptive selection as the informativeness changes during training. We find that simply executing more tests increases computation cost without necessarily improving performance. In contrast, our adaptive selector effectively tracks informative tests from a continuously generated pool, achieving better downstream performance at a lower computational cost. Experiments across diverse LCMs and multiple benchmarks demonstrate that our method effectively leverages large amount of unit tests consistently improves coding performance over standard reinforcement learning based finetuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, large code models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GNYbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LUA, a unit-test selection method for reinforcement learning fine-tuning of code models. It samples a candidate test pool, scores tests with a learned selector, and favors tests whose scores are near the median. Code generation and test selection are jointly optimized using execution rewards. Experiments examine its combination with existing training methods and test-pool augmentation.

Critical strengths

Selecting tests for reward computation is a useful direction that complements problem-level curriculum learning. The paper includes random-selection and test-augmentation comparisons, which help examine the contribution of test selection.

Critical weaknesses
  1. The central selection mechanism is insufficiently justified and specified. It is unclear why the median of learned selection probabilities should identify uncertain tests near the model’s capability boundary. The extensive policy formalization also leaves practical details unresolved, particularly how the selector is trained under the transformed sampling distribution and how additional tests are generated and validated.
  2. The empirical benefits are modest and not consistent across metrics. In particular, adaptive selection provides only a small additional gain over uniform sampling from the augmented test pool. Without repeated-run uncertainty or end-to-end cost measurements, it is difficult to judge whether the learned selector offers a reliable advantage over simpler alternatives.
Questions for authors
  1. How are additional tests and their expected outputs produced and validated? Are tests generated during training, or are candidates refreshed from a precomputed pool? Please also specify the pool sizes and number of selected tests.
  2. What is the exact optimization loss for the test selector, and how does it account for the adaptive sampling distribution? What evidence links the learned scores to test difficulty or uncertainty?
  3. Do the gains over random selection remain across training seeds and under matched compute budgets, including test scoring and generation costs? A comparison with the fixed single-test baseline under the same problem-sampling strategy would also help.
1 comment

We thank the reviewer for the careful evaluation and constructive feedback.

Q1. Question regarding the setup of unit tests

First, the datasets information we used are as follows

ProblemsTestsTests / problem (median, max)
Train split 24,2872,242,645101, 1,440

For each problem, the candidate pool U_small contains 16 tests sampled once from its full test set and refreshed every 5 steps.


Q2. Question regarding sampling budget

We appreciate the reviewer for this important point. Our method consists of the following three additional steps during the training.

(i) forward pass over the |U_small| tests per problem

(ii) linear head backward

For Qwen3-8B, the end-to-end training time was as follows,

StageRandom-KLUA
(i) Test scoring–100.8
(ii) Head update–0.4
Rollout generation152.1151.2
Log-prob (old + ref)286.5276.2
Actor update418.3403.5
End-to-end869.1943.8 (+8.6%)

resulting in additional +8.6% of overhead.

StepRandom-K (time)Random-K (score)LUA (time)LUA (score)
204.4 h30.444.9 h30.80
409.0 h31.3610.0 h31.64
6014.1 h30.1615.4 h30.30
8019.2 h32.7220.9 h32.82

Under matched wall-clock time, LUA remains comparable to Random-K, indicating that its 8.6% overhead does not negate its consistent per-step gains.


Q3. Which generator, expected-output source, and validity checks supply the tests? Please provide pool-growth records, candidate/subset sizes, training/evaluation split and deduplication details, and total costs including generation, encoding, rollouts, execution, and updates. Which evidence establishes streaming benefits beyond resampling a pregenerated pool?

The expected outputs are the ground-truth outputs provided by the dataset, used unmodified. We will revise the manuscript to make clear that the tests are selected from the existing suite rather than generated.

To evaluate the effect of varying the number of available unit tests, Table 1 reports model performance across different test subset sizes (1, 8, 16, and the full suite). Furthermore, our algorithm periodically re-draws U_small, so tests the head has never scored entering the candidate pool are handled without retraining. Table 4 demonstrates this effect.

Reviewer_sDMWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LUA, a method for selecting which unit tests to execute when training a code language model with reinforcement learning. For each programming problem, LUA samples a candidate set from a growing pool of unit tests and uses a linear head on the code model to score each problem–test pair. It then favors tests whose scores are close to the median of the candidate set. The motivation is that a test that some generated programs pass and others fail can provide a more informative reward than one that nearly all programs pass or fail. The fraction of selected tests passed by each program serves as its training reward. Experiments on LiveCodeBench v5, Codeforces, and HumanEval+ show that LUA improves the reported average over vanilla GRPO on three models. On Qwen3-4B, it also improves the average when combined with two existing problem-level selection methods.

Critical strengths
  1. The paper identifies a practical question in RL training for code generation. After selecting a training problem, one still has to decide which unit tests to use when computing the reward. This gives the authors a way to apply curriculum learning within each problem.
  2. I found the result in Table 1 interesting. At training step 60 on Qwen3-4B, using one fixed test per problem yields an average score of 36.32, compared with 34.56 when using the full test suite. Although broader test coverage is desirable when evaluating a finished program, this result suggests that executing more tests to compute the training reward does not necessarily improve downstream performance.
Critical weaknesses
  1. My main concern is that the proposed test selector provides only a small improvement over Random-K, which already obtains much of the reported performance by sampling tests uniformly. In Table 3, LUA improves the average over Random-K by 0.32 points on Qwen3-1.7B, 0.56 on Qwen3-4B, and 0.38 on Llama-3.2-3B-Instruct. Although the average improves on all three models, the gains are mixed across individual metrics. For example, on Qwen3-4B, LiveCodeBench pass@1 improves from 25.7 to 27.1, while pass@8 falls from 35.2 to 34.8. Table 4 shows a similar pattern on the augmented test pool, where replacing uniform sampling with adaptive selection raises the average from 37.02 to 37.28. Given how close vanilla GRPO, Random-K, and LUA are in these experiments, and the absence of variability estimates, I find it difficult to judge whether learning which tests to execute provides a reproducible benefit large enough to justify the additional scoring step.
Questions for authors
  1. Could the authors repeat the comparison between Random-K and LUA across several training seeds, using the same candidate pools, number of selected tests, and rollout budget for each paired run? It would be helpful to report the paired difference and its uncertainty, both for the average in Table 3 and for each benchmark. This would clarify whether the advantage over random selection is reproducible and where it comes from.
  2. Since Random-K does not require a model forward pass to score candidate tests, how do the two methods compare under the same end-to-end training time? In particular, does LUA reach a given downstream score faster, or achieve a higher score when Random-K is allowed to use the time that LUA spends scoring tests?
  3. Could the authors repeat the comparisons in Table 2 across multiple training runs with random seeds and report the mean and variance of the scores for each baseline, both with and without LUA? This would help determine whether the gains from adding LUA are consistent across runs or could be explained by training variability.
  4. Could the authors provide concrete examples or additional experimental evidence showing that LUA meaningfully improves downstream performance over both vanilla GRPO and Random-K? For example, it would be helpful to see a case where a test prioritized by LUA exposes an error that randomly selected tests miss, and where the trained model subsequently handles a similar case correctly on a specific problem.
1 comment

We thank the reviewer for the careful evaluation and constructive feedback.

Q1. Questions regarding multiple seeds and experiments

Due to computational resources, we re-ran additional experiments with two seeds. We used HumanEval+ pass@1 for comparison at step 60. We will include full experiments in the future version.

Original GRPOLUA
77.95 0.21278.05 0.495

Also, we ran an experiment on Qwen3-8B (larger model) and evaluated every 20 steps.

StepUniformLUA
2030.4430.80
4031.3631.64
6030.1630.30
8032.7232.82

On Qwen3-8B, LUA exceeds Random-K at every checkpoint, indicating that the method scales as well.


Q2. Since Random-K does not require a model forward pass to score candidate tests, how do the two methods compare under the same end-to-end training time? In particular, does LUA reach a given downstream score faster, or achieve a higher score when Random-K is allowed to use the time that LUA spends scoring tests?

We appreciate the reviewer for this important point. Our method consists of the following three additional steps during the training.

(i) forward pass over the |U_small| tests per problem

(ii) linear head backward

For Qwen3-8B, the end-to-end training time was as follows,

StageRandom-KLUA
(i) Test scoring–100.8
(ii) Head update–0.4
Rollout generation152.1151.2
Log-prob (old + ref)286.5276.2
Actor update418.3403.5
End-to-end869.1943.8 (+8.6%)

resulting in additional +8.6% of overhead.

StepRandom-K (time)Random-K (score)LUA (time)LUA (score)
204.4 h30.444.9 h30.80
409.0 h31.3610.0 h31.64
6014.1 h30.1615.4 h30.30
8019.2 h32.7220.9 h32.82
Under matched training time, LUA outperforms Random-K, indicating that its 8.6% overhead does not have a negative effect on its consistent per-step gains.

Q3. Regarding qualitative examples

Thank you for your suggestion. We will include examples in the revised versio

Reviewer_qFnNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

For RL on code with unit-test rewards, the paper proposes picking which unit tests to use for the reward from a big, continuously generated pool. It uniformly samples a small candidate pool, scores each test with a learned linear head on the LLM's last hidden state, then samples tests near the median score (a Gaussian kernel around the median) to target "uncertain" tests. Reward is the pass rate on the selected tests. A motivation experiment shows using more tests doesn't help (1 test does best at step 60). Experiments are on DeepCoder data with Qwen3-1.7B/4B and Llama-3.2-3B, evaluated on LCB v5, CodeForces and HumanEval+, and show ~0.4–0.7 point average gains over GRPO, AdaRFT and GAIN-RL.

Critical strengths
  1. Test-level curriculum (as opposed to problem-level) is a reasonable and fairly unexplored angle, and the observation that more unit tests doesn't automatically mean better RL (Table 1) is interesting and worth digging into.
  2. The method is cheap: a linear head on hidden states, and it plugs into existing problem-level curricula (Table 2 shows it stacked on AdaRFT and GAIN-RL).
Critical weaknesses
  1. The gains are tiny with no variance reported, and the paper's own numbers undercut them. Average improvement is +0.4 to +0.7 points, evaluated at a single checkpoint (step 60), with no seeds or CIs. In the same Qwen3-4B setup, using just 1 fixed test gets 36.32 (Table 1), which beats GRPO+LUA at 36.00 (Table 2). Random-K is within a few tenths of LUA across all three models. Fig 2b also shows the learned selection weights stay within roughly 0.99–1.02 of uniform, so the selector is basically uniform sampling. If the selector barely deviates from uniform, it's hard to credit it for any gain.
  2. Key parts of the method aren't specified. I couldn't find how w_unit is actually trained (what loss and gradient; the paper says plain reward maximization would favor easy tests, then switches to median decoding without saying how the head learns). I also couldn't find how the "continuously generated" unit tests are produced or how their expected outputs are checked for correctness, which matters a lot since wrong tests are wrong rewards. Fewer tests also means more false-positive rewards for wrong code, which isn't discussed. The MDP formalism with unit test index tokens in the action space is a lot of notation for what is basically a sampling heuristic.
Questions for authors
  • How is w_unit trained, exactly? What loss, and what signal does it get?
  • How are new unit tests generated during training, and how do you know their expected outputs are correct?
  • Can you report multiple seeds with CIs, and results at more than one checkpoint (or the final converged checkpoint)?
  • Given Fig 2b (weights ≈ uniform), what do you think explains the gain over Random-K?
  • Why does Table 4 (step 40) beat Table 2 (step 60) for the same model? Are the setups different?
  • How does LUA compare to just using 1 random test per problem, which Table 1 suggests is a strong baseline?
1 comment

We thank the reviewer for the careful evaluation and constructive feedback.


1. How is w_unit trained, exactly? What loss, and what signal does it get?

The LM generates G solutions for x, each executed on the drawn tests. The signal is the group-mean reward with the pass rate of the drawn tests. In particular, no extra labels or difficulty annotations are used.

Here, L(wunit)=−1B∑xR(x)∑k=1Klog⁡πw(ak∣x)\mathcal{L}(w_{\text{unit}}) = -\frac{1}{B}\sum_{x} R(x) \sum_{k=1}^{K} \log \pi_w(a_k \mid x) The decoding rule is treated as a non-differentiable sampling step: the gradient is taken through w, not through q.

Tests appearing in subsets that the current policy solves receive higher selection weights, reflecting policy-dependent test characteristics. The median-centered decoding is designed to favor tests with intermediate selection weights, while the selection distribution can evolve as the policy improves.


2. How are new unit tests generated during training, and how do you know their expected outputs are correct?

The expected outputs are the ground-truth outputs provided by the dataset, used unmodified. We have revised the manuscript to make clear of the dataset we used.


3. Question regarding experiments

Due to computational resources, we re-ran additional experiments with two seeds. We used HumanEval+ pass@1 for comparison at step 60. We will include full experiments in the future version.

Original GRPOLUA
77.95 0.21278.05 0.495

Also, we ran an experiment on Qwen3-8B (larger model) and evaluated every 20 steps.

StepUniformLUA
2030.4430.80
4031.3631.64
6030.1630.30
8032.7232.82

On Qwen3-8B, LUA exceeds Random-K at every checkpoint, indicating that the method scales as well.


4. Given Fig 2b (weights ≈ uniform), what do you think explains the gain over Random-K?

The learned selector distinguishes candidate tests, demonstrating the gain over uniform sampling. At the checkpoint of step 100, 7.6% of candidate tests receive near-zero probability, and the total-variation distance from uniform grows from 0.07 (step 20) to 0.14 (step 100). The suppressed tests are almost exclusively those whose scores lie far above the per-problem median, and they are spread evenly across difficulty groups (8.4% / 6.4% / 7.3% of hard / uncertain / easy tests).


5. Why does Table 4 (step 40) beat Table 2 (step 60) for the same model? Are the setups different?

Here, the setups are the same. Both tables evaluate checkpoints of the same training run, and the difference is real. Held-out performance peaks around step 40 and declines by step 60, then increases later.


6. How does LUA compare to just using 1 random test per problem, which Table 1 suggests is a strong baseline?

We agree that Table 1 suggests that using a single randomly selected test can be a strong baseline. LUA incurs additional verification and scoring costs compared with one unit test, as it executes 8 tests per rollout. We will clarify this trade-off in the revised manuscript.

To distinguish the benefit of adaptive test selection from that of using more tests, we compare LUA against Random-8, which uses the same number of selected tests. We will additionally include Random-1 in the main results table to make trade off more clear.

Comments on the paper

1 comment

We thank the reviewer for the careful evaluation and constructive feedback.

Q1. What is the actual selector objective? Please map one batch's candidates, eight rollouts, shared/independent actions, log probabilities, advantages, and losses; provide selector gradient norms and actual (p/q) distributions. How do weights track current pass rates or error-discrimination ability?

The LM generates G solutions for x, each executed on the drawn tests. The signal is the group-mean reward with the pass rate of the drawn tests. In particular, no extra labels or difficulty annotations are used.

Here, L(wunit)=−1B∑xR(x)∑k=1Klog⁡πw(ak∣x)\mathcal{L}(w_{\text{unit}}) = -\frac{1}{B}\sum_{x} R(x) \sum_{k=1}^{K} \log \pi_w(a_k \mid x) The decoding rule is treated as a non-differentiable sampling step: the gradient is taken through w, not through q.

Tests appearing in subsets that the current policy solves receive higher selection weights, reflecting policy-dependent test characteristics. The median-centered decoding is designed to favor tests with intermediate selection weights, while the selection distribution can evolve as the policy improves.


Q2. Question regarding sampling budget

We appreciate the reviewer for this important point. Our method consists of the following three additional steps during the training.

(i) forward pass over the |U_small| tests per problem (ii) linear head backward For Qwen3-8B, the end-to-end training time was as follows,

StageRandom-KLUA
(i) Test scoring–100.8
(ii) Head update–0.4
Rollout generation152.1151.2
Log-prob (old + ref)286.5276.2
Actor update418.3403.5
End-to-end869.1943.8 (+8.6%)
resulting in additional +8.6% of overhead.
StepRandom-K (time)Random-K (score)
---
204.4 h30.44
409.0 h31.36
6014.1 h30.16
8019.2 h32.72
Under matched training time, LUA outperforms Random-K, indicating that its 8.6% overhead does not have a negative effect on its consistent per-step gains.

Q3. Question regarding the setups (Which generator, expected-output source, and validity checks supply the tests)

The expected outputs are the ground-truth outputs provided by the dataset, used unmodified. We revised the manuscript to make clear of the dataset we used. Furthermore, for each problem, the candidate pool U_small contains 16 tests sampled once from its full test set and refreshed every 5 steps.


Q4. Regarding recent work such as CURE and UTRL

Thank you for your comment. We included additional discussions regarding methods CURE and UTRL in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.