acceptodds
Under review as a conference paper at ICLR 2027

Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation

Abstract

Reinforcement learning (RL) from unit-test feedback has become a standard post-training recipe for improving large language models (LLMs) on code generation. However, the *pass-all-tests* binary reward can be sparse, yielding no learning signal on challenging problems where none of the sampled solutions passes all tests. A common remedy is to use the *test-case pass rate* as a surrogate reward. In this work, we study pass-rate rewards in RL for code generation and report a consistent pattern across base models and algorithms: despite alleviating reward sparsity, pass-rate rewards do not reliably improve final performance over binary rewards in controlled experiments. To understand this discrepancy, we analyze reward density and the resulting gradient directions. We find denser rewards but weak local transfer to verified full-pass references when groups contain only partial-pass responses. Sample-level probes reveal positive and negative reference-transfer effects within the same group, while semantic analysis shows that higher pass rates can favor algorithmic errors over near-correct solutions. Together, these findings identify a mismatch between partial test-case success and guidance toward full solutions, motivating rewards that align intermediate feedback with full correctness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Code Generation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_y2ooOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an empirical analysis of test-case pass-rate rewards, which have been introduced to address the sparse learning signals of binary rewards in reinforcement learning (RL) for code-generating LLMs. The central finding is that, despite providing denser learning signals, pass-rate rewards do not consistently improve final performance over binary rewards. To support this claim, the authors compare the learning dynamics and final performance of binary and pass-rate rewards across multiple LLMs (DeepSeek-R1-Distill-Qwen-7B, Qwen3-4B, and Qwen2.5-7B-Instruct) and RL algorithms (GRPO and RLOO), using LiveCodeBench and LeetCode. The paper further examines reward density, confirming that pass-rate rewards indeed provide richer intermediate feedback. Through gradient transfer analysis, the authors observe that learning from partially correct solutions does not necessarily increase the likelihood of generating fully correct solutions. Additionally, semantic misranking analysis reveals cases where solutions with higher test-case pass rates contain fundamental algorithmic errors, while those with lower pass rates may require only minor modifications to become correct. Based on these findings, the paper suggests that effective reward design should consider whether intermediate feedback guides learning toward full correctness, rather than simply rewarding higher test-case pass rates.

Critical strengths

The paper presents an interesting empirical observation that denser learning signals from test-case pass-rate rewards do not necessarily translate into better final performance in code-generation RL. This observation is supported by comparisons across multiple models, RL algorithms, and benchmarks, suggesting that reward density alone may not be a reliable indicator of learning effectiveness.

Critical weaknesses

1. Insufficient consideration of test-suite characteristics

The paper investigates the limitations of test-case pass-rate rewards without sufficiently examining the characteristics of the underlying test suites. The number, difficulty distribution, redundancy, and behavioral coverage of test cases may substantially affect how well pass rates reflect progress toward a correct solution. Although the paper explores difficulty-based reweighting, it does not systematically analyze these factors. Consequently, it remains unclear whether the observed limitations arise from the pass-rate reward formulation itself or from the characteristics of the test suites used in the experiments.

2. Insufficient evidence supporting the proposed explanations

The paper provides empirical evidence that pass-rate rewards do not consistently outperform binary rewards, but the proposed explanations remain insufficiently validated. In particular, the gradient transfer analysis measures changes in the likelihood of a single verified reference solution, which may not adequately represent improvements in overall problem-solving capability. Similarly, the semantic misranking analysis identifies potentially misleading reward assignments but does not establish their causal relationship with the observed performance differences. As a result, the analyses provide suggestive observations rather than a convincing explanation of why pass-rate rewards fail to deliver consistent improvements.

Questions for authors
  • How do the number, difficulty distribution, redundancy, and behavioral coverage of test cases vary across problems? To what extent do these characteristics affect the relative performance of binary and pass-rate rewards?

  • What evidence supports the use of changes in the likelihood of a single verified reference solution as an indicator of progress toward full correctness, given that a programming problem may have multiple valid solutions?

  • Is there evidence that semantic misranking directly contributes to the observed performance differences between binary and pass-rate rewards, rather than merely being a phenomenon observed alongside them?

  • How sensitive are the relative performance differences between binary and pass-rate rewards to the number of training rollouts per problem (N)? In particular, would the conclusions remain the same with smaller rollout groups, where binary reward sparsity is expected to be more severe?

  • Given that pass@k is estimated from only 16 generated solutions per problem, how reliable are the reported performance differences, particularly for pass@16? Are the relatively small differences between binary and pass-rate rewards supported by confidence intervals or repeated evaluations with different random seeds?

Reviewer_fL1kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why replacing the sparse pass-all-tests binary reward with a denser test-case pass-rate reward in RL for code generation does not reliably improve final full-pass (pass@k) performance. Across experiments spanning three base LLMs, GRPO/RLOO/PPO, and two reward variants (difficulty-reweighted and two-stage) on LiveCodeBench and LeetCode, pass-rate reward speeds up early pass@1 learning in the primary DS-R1-7B + GRPO setting, but ends up trailing binary reward on average pass@16 across all four model–algorithm pairs in Table 1 (-1.7 to -2.3 percentage points).

To explain why, the authors run single-step gradient probes on rollout groups with no full-pass solutions (without-full), finding near-zero net log-probability transfer toward full-pass reference solutions (Figure 3a, Table 8) and both positive and negative per-sample effects on reference log-probability within groups (Figure 3b). Using LLM-as-judge and manual analyses, they show that algorithmically flawed or shortcut solutions can pass more tests than near-correct solutions with local errors, with flawed solutions receiving positive advantages while near-correct ones receive negative advantages in 225/392 tasks (57.4%) under the primary DeepSeek-R1 judge. Finally, in the appendix, they show that zeroing out negative advantages for non-overlong responses in without-full groups (Table 9) or using task-sampling curricula (Table 21) closes or narrows the pass@16 gap

Critical strengths
  1. Clean cross-model and cross-algorithm comparisons of binary vs. pass-rate rewards. Several recent code-RL papers use test-case pass rate to ease reward sparsity. The paper reports controlled GRPO comparisons across three base models, an additional RLOO comparison on DS-R1-7B, two reward variants in the primary GRPO setting, and an independent PPO comparison on LiveCodeBench and LeetCode. It clearly shows that pass-rate rewards mainly speed up early training under GRPO while yielding lower aggregate pass@8 and pass@16 in Table 1 and 94%–97% agreement in task-level solved/failed outcomes

  2. Intuitive semantic-misranking diagnosis backed by a simple clipping fix. The paper gives a clear picture of why raw pass rate can mislead group-relative updates: Figure 4 shows an invalid modular-division solution passing 84% of tests and receiving a positive advantage, while a near-correct solution with a local error receives a negative advantage (with 36.5% reward reversal and 57.4% advantage-sign conflict in Appendix E.1, and 28.8%–37.4% reversal across seven judge setups in Table 13). As supporting evidence, Appendix G.1 (Table 9) shows that simply clipping negative advantages to zero for non-overlong responses in without-full groups erases the pass@16 gap with binary reward on both benchmarks

Critical weaknesses
  1. The main empirical finding has prior precedent, and the gradient-probe evidence for the proposed mechanism has several gaps. The paper frames the comparison between binary and pass-rate rewards as largely unexplored (lines 43–45, 469–474). However, key parts of this observation have already been reported: DELTA-Code (Sun et al., 2025; arXiv:2509.21016v1) showed on synthetic tasks that per-test pass-rate rewards give early signal but quickly plateau, and VeRPO (Wang et al., 2026; arXiv:2601.03525v1) directly showed in multi-turn code generation that raw pass-rate can trail binary GRPO whereas a test-density-calibrated reward paired with a binary correctness anchor outperforms both. Section 3.6 and Appendix A evaluate one static percentile-based reweighting scheme, but do not report its numerical weights or per-benchmark results. Sensitivity to test-suite composition and reward calibration beyond the tested variants remains unclear.

    Because the headline phenomenon has precedent, the paper's contribution leans heavily on its gradient-probe analysis (Section 4.2, Appendix F), which has three limitations.

    • First, in Section 4.2 and Table 8, checking whether an update boosts the log-probability of one specific reference solution yfully_{\text{full}} per task is an indirect proxy for progress, since coding problems usually admit multiple valid algorithms and styles;
    • Second, the main 392-task σ\sigma-normalized GRPO probe in Figure 3a (Eq. 11) has no quantitative summary table, while the supplementary paired probe in Table 8 (Appendix F.3, lines 992–998) reports a separate analysis using an unnormalized mean-baseline update (ai=ri−rˉa_i = r_i - \bar{r} on 168 tasks) and cannot fill that gap;
    • Third, promoting the top partial response alone (Table 7) gives opposite transfer directions on late-solved tasks (96.2% positive, where references come from later successful rollouts of the same trajectory) versus never-solved tasks (81.0% negative, where references come from external QwQ/DeepSeek-R1 models). Pooling 100 late-solved and 68 never-solved tasks in Table 8 leaves unclear how much of the weak net transfer comes from within-group advantage cancellation versus differences in task difficulty or reference source
  2. Small pass@k differences lack run-to-run uncertainty, and key experimental settings are omitted from the main comparison. The final gaps between binary and pass-rate rewards in Tables 1 and 3 are small (0.0–3.1 percentage points), yet no multi-seed uncertainty is reported (and Section 3.4 notes that PPO uses a single training run per condition), making it hard to tell which differences are above noise. In addition, the main experiments (Section 3.1, Table 1, Appendix B) do not report the maximum response length or the untrained Base models' test-set pass@k scores, whereas the supplementary runs (Appendices G and J) use a 4,096-token cap where filtering untrained Base 16K training responses at 4,096 tokens substantially cuts task coverage (Table 23)

Questions for authors
  1. How do you position your findings and proposed mechanism relative to VeRPO (Wang et al., 2026; arXiv:2601.03525v1) and DELTA-Code (Sun et al., 2025; arXiv:2509.21016v1)? Do existing analyses shed light on sensitivity to test-suite composition or reward calibration? What numerical weights wkw_k were used for the Reweighted variant, and what per-benchmark pass@k scores were obtained for both variants in Section 3.6?

  2. What are the summary statistics (mean, median, and fraction of tasks with positive transfer) for the 392-task standard-deviation-normalized GRPO probe in Figure 3a (Eq. 11)? For Table 8, can you report the existing results separately for late-solved tasks (with references from later successful rollouts of the same trajectory) and never-solved tasks (with external references), and, if available, whether the same pattern holds under standard-deviation-normalized GRPO advantages?

  3. Can you provide multi-seed uncertainty (if available), the training and evaluation response-length limits used for Table 1, and the untrained Base test-set pass@k scores for the three models?

Reviewer_MN87OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Pass-rate reward, the fraction of unit tests a generated program passes, is widely used as a dense surrogate in RLVR for code, on the reasoning that binary all-tests-pass reward gives a zero group-relative advantage when no rollout in a group fully passes. This paper asks whether that substitution actually helps, and reports that it does not. No new method is proposed. The first layer is a controlled comparison on LeetCodeDataset plus LiveCodeBench (3,521 train / 403 test) across DeepSeek-R1-Distill-Qwen-7B, Qwen3-4B and Qwen2.5-7B-Instruct with GRPO, RLOO and PPO, trained strictly on-policy (batch = mini-batch, so the importance ratio is identically 1 and clipping is inert), with only the reward function swapped among binary, raw pass rate, a difficulty-reweighted pass rate, and a two-stage schedule. Two diagnostic layers follow: a reference-transfer gradient probe, and an LLM-judge labelling of incorrect responses as helpful or harmful. Headline: pass@1 is essentially tied while binary pulls ahead as kk grows (DS-R1-7B+GRPO pass@16 57.6 vs 55.6), with 94% task-level agreement on the test set.

Critical strengths
  • The controlled layer is genuinely controlled. Setting batch = mini-batch forces ρi,t≡1\rho_{i,t}\equiv 1, which makes clipping inert and shrinks GRPO vs RLOO to the single question of whether you divide by the group standard deviation; scaffold, prompt, temperature, lr and step count are then held fixed and only the reward changes. That discipline is rare in LLM-RL papers, and it is why the negative result is worth taking seriously as a direction.

  • The bookkeeping across 28 tables holds up to rounding. The per-cohort solved counts in Table 25 exactly rebuild Table 26's new/lost/net = 180/137/+43. What breaks is the step from numbers to claims, not the experiments, so a revision would not have to re-run the training grid.

Critical weaknesses
  • The main claim is not statistically established. One run per cell, no error bars, no significance test anywhere, and the differences carrying the conclusion run 0.1 to 3.1 points. With n=k=16n=k=16 the pass@kk estimator degenerates to I[c≥1]\mathbb{I}[c\ge 1], so pass@16 is one Bernoulli draw per problem and SE ≈2.5\approx 2.5 points over 403 problems. The paper's own paired data settle it: McNemar on the 15 binary-only vs 8 pass-rate-only discordant problems in Table 4 gives p≈0.21p\approx 0.21.
  • The headline binary row contradicts the tables that decompose it, and the gap is big enough to move the claimed effect.
    • Table 1's 48.6% on LiveCodeBench uniquely implies 85/175 solved. Table 5's percentages uniquely imply 82, i.e. 46.9%, which takes Δ(PR−Bin)\Delta(\text{PR}-\text{Bin}) from −2.3-2.3 to −0.6-0.6.
    • Table 4's binary total is 216+15=231216+15=231 against Table 1's 232.
    • Tables 4 and 5 are mutually consistent and the pass-rate side agrees at 224 across all three, so "different run or split" does not explain it.
  • Section 4, the claimed main contribution, is contradicted by its own appendix. The main text says yfully_{\text{full}} is sampled from DeepSeek-R1; the appendix says the late-solved cohort, which is the bulk of the sample, uses the trained model's own first successful rollout. The two cohorts have opposite signs (+0.01280+0.01280 vs −0.01793-0.01793), reference provenance is therefore perfectly collinear with cohort, and "transfer is weak" cannot be told apart from "the probe measures self-similarity." The mixed-sign finding (344/397) is an algebraic consequence of ∑iAi=0\sum_i A_i=0 and comes with no binary control. The one clean result, negative-advantage clipping recovering binary-level pass@16 with the reward untouched, sits in Appendix G.1 and is absent from the abstract and conclusion.
Questions for authors
  1. Table 1 vs Tables 4/5, binary row, DS-R1-7B+GRPO. Table 1 reports binary pass@16 of 48.6% (LiveCodeBench) and 64.5% (LeetCode), which at n=k=16n=k=16 uniquely imply 85 and 147 solved problems (232 total). Table 5's per-benchmark percentages uniquely imply 82 and 149 (231 total), matching Table 4's 216+15=231216+15=231. Which set is correct? The pass-rate row agrees across all three tables at 224, so a generic "different evaluation run" explanation does not apply.
  2. If the correct LiveCodeBench binary pass@16 is 46.9% rather than 48.6%, Δ(PR−Bin)\Delta(\text{PR}-\text{Bin}) on that benchmark becomes −0.6-0.6 instead of −2.3-2.3. Does the Section 3.2 claim about "underperforming the binary baseline on pass@kk for larger values of kk" survive that correction?
  3. Applying McNemar to your own Table 4 discordant pairs (15 vs 8) gives p≈0.21p\approx 0.21. On what basis is "binary outperforms pass-rate" asserted rather than "no significant difference was detected"? Please state the test and effect-size estimate you rely on, or qualify the claim.
  4. Confirm which cohort of the transfer probe uses which reference source. If the late-solved cohort uses the trained model's own successful rollout while the never-solved cohort uses QwQ/DeepSeek-R1 solutions, how do you separate the cohort effect from the in-distribution/out-of-distribution effect, given that the two cohort means have opposite signs (+0.01280+0.01280 vs −0.01793-0.01793)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.