acceptodds
Under review as a conference paper at ICLR 2027

Recovering Verifiable Process Credit in Reinforcement Learning for Code Generation

Abstract

Group-based reinforcement learning with verifiable rewards (RLVR) and its recent advances train a code model by examining the outcomes of sampled rollout programs against a suite of test cases. It thus suffers from the sparse-reward issue: a reward is credited at the level of the whole program. In contrast to this coarse-grained perspective, we view a program as a composition of syntax-tree units and find that even a failing program often contains units that already make tests pass, and that its failure can be traced to a few specific units. Based on the findings, we propose ASTrace, an abstract syntax tree (AST) reward attribution method that assigns process credit to program units from code execution. ASTrace parses each sampled program into units, namely its functions, methods, and top-level blocks, and credits them through two components: (i) *progress credit*, which, for all-fail rollouts, credits each unit with its marginal gain in passing tests; and (ii) *failure localization*, which, in partial-fail rollouts, penalizes the unit of a failing rollout where a missed test raises its error. We compare ASTrace against GRPO and its most recent variants across six code-generation benchmarks and two backbones. ASTrace consistently improves pass@1 over GRPO on both common coding benchmarks (e.g. 2.2% on HumanEval with Qwen3-4B) and more challenging coding contests (e.g. 2.6% on OJBench with Qwen3-4B). Our analysis shows that ASTrace turns near-miss programs into solutions and gains the most where failing programs are partially correct.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning with verifiable rewards

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_13umOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

To address the sparse reward problem of RLVR in code generation tasks, this paper proposes ASTrace, which decomposes programs into AST units and performs fine-grained credit assignment via two mechanisms: Progress Credit and Failure Localization. The paper reports pass@1 evaluation results across six benchmarks using two model backbones, demonstrating convincing empirical performance.

Critical strengths

ASTrace divides credit assignment into two components: progress credit for groups in which all rollouts fail, and failure localization for groups in which some rollouts succeed. The former estimates the marginal testing benefit of restoring code units in a particular order, while the latter leverages execution traces to pinpoint failing units. The two mechanisms are designed to address different rollout-group scenarios, resulting in a clear and intuitive framework. Moreover, the credit signals are derived directly from executable programs and test outcomes, without relying on an additionally trained reward model or reference solutions. The experiments cover multiple code generation benchmarks, supplemented by competitive programming evaluations on OJBench and LCB-Pro. The results show that ASTrac improves the average pass@1 across all six benchmarks on both model backbones.

Critical weaknesses
  1. Progress Credit restores code units sequentially in a random order and uses the marginal gains from a single permutation to estimate credit. This single-sample estimate may have high variance, particularly when complex dependencies exist between code units.
  2. Failure Localization identifies code units based on the traceback or execution trace of failed tests and assigns a fixed negative credit. However, the location where an exception is reported during execution does not necessarily correspond to the root cause of the error. How does Failure Localization avoid incorrect attribution in cases involving collaborative errors across multiple code units or downstream exceptions caused by upstream errors?
  3. The method is evaluated on only Qwen3 and Qwen2.5 models. It would be beneficial to include newer models, such as Qwen3.5, to further validate the effectiveness and generalizability of ASTrace.
Questions for authors

Please refer to the weaknesses above.

4 comments

We thank Reviewer 13um for the positive assessment, in particular for finding the two-route framework "clear and intuitive" and the empirical results "convincing". We measured the fidelity of both credit signals directly and address each question below.

[W1. Variance of the single-permutation estimate]

  • Thanks for the insightful question. The single-order estimate remains reliable under unit dependencies, for two reasons.
  • (1) Dependencies affect only how credit is split, never its direction or total. The estimate is unbiased over orders, and its credits sum exactly to the rollout's net test gain for every order (efficiency, Eq. (5)). To check the split, we took the 56 all-fail ClassEval programs of the Table 3 probe (k ≤ 4 units), executed all 2^k unit coalitions to obtain exact Shapley values, and compared them with the estimate from every one of the k! orders.
Single order vs. exact creditShare of unit-order pairs
Opposite sign to the exact credit0%
Positive credit to a unit whose exact credit is 00%
Zero credit to a unit with positive exact credit (its share goes to a jointly needed unit)47%
  • The 47% is the dependency case: when several units are jointly needed for a test, a single order gives their shared credit to whichever unit completes the coalition. In 22 of the 56 programs every order gives identical credit, and the most-credited unit matches the exact one in 90.6% of programs with net progress.
  • (2) Averaging orders further reduces the split error, at proportional execution cost. With the error measured as the summed absolute error over units divided by the rollout's net test gain, it falls as follows, and we add the number of orders as an option in the revision.
Orders averaged1248
Relative error0.700.430.290.21

[W2. Reported location vs. root cause]

  • We appreciate this careful question. Failure localization handles both cases the reviewer raises, for two reasons.
  • (1) The localized unit is the fault in most failures. We ran a new swap analysis on the ClassEval probe. For each of the 108 failing rollouts with a passing sibling, we replaced units with the sibling's in three ways: the localized unit alone, all units except the localized one, and the localized unit together with other units. The swap that repairs the program shows where the fault lies.
Repairing swapWhere the fault liesShare of 108 failures
Localized unit aloneLocalized unit63%
Localized unit with other unitsSeveral units including the localized one (collaborative error)19%
All units except the localized oneA direct caller or callee (localized unit shows only the symptom)17%
NoneUnresolved1%
  • In collaborative errors, the localized unit is itself part of the fault, so the −1 still lands on a faulty unit; with these, the localized unit is part of the fault in 82% of failures. Downstream exceptions caused by an upstream error always originate one call away and come from only 5 problems. Extending the credit to the localized unit's direct callers and callees would cover them, and we discuss this extension in the revision.
  • (2) A wrong location has a bounded effect. Localization credit applies only to failing rollouts in partial-fail groups, which GRPO already penalizes with an advantage of at most −0.2. The added term is at most 0.1 in magnitude, so a wrong location only shifts blame within a program that is already penalized and cannot turn its update positive.

[W3. Newer backbones]

  • We trained GRPO and ASTrace from Qwen3.5-4B-Base, a newer model generation than our two backbones. Given the rebuttal timeline, we ran the first training stage (200 steps, about 240 GPU-hours) with the data and hyperparameters of the main experiments, and evaluated with 8 samples per problem.
Qwen3.5-4B-Base, pass@1 (%)LCB v6ClassEvalHumanEvalHumanEval+Avg.KodCode val
Before RL21.4315.1267.8463.4941.9731.8
GRPO42.0029.6294.7490.0964.1165.0
ASTrace42.5032.1294.9791.5465.2867.4
Δ vs. GRPO+0.50+2.50+0.23+1.45+1.17+2.4
  • ASTrace improves over GRPO on all four benchmarks and on the held-out KodCode validation split, with the largest gain on ClassEval (+2.50). The benefit therefore carries over to a newer architecture and to training from a base model.

We would be glad to discuss any remaining questions.

Dear Reviewer 13um,

Thank you again for your positive assessment and helpful questions. In our response, we:

  1. compared the single-order estimate against exact Shapley values.
  2. ran a swap analysis on 108 failing rollouts.
  3. trained GRPO and ASTrace from Qwen3.5-4B-Base, a newer base model.

We would like to know whether these results address your questions. We are happy to discuss anything further.

Best regards, The Authors

Thank you for your detailed responses and clarifications! I think this is a good paper, and will keep my rating of 3 (Weak Accept).

AuthorsOpenReview ↗

Thanks to Reviewer 13um

Thank you for your prompt reply. We are glad that we have addressed your concerns. Again, we sincerely appreciate your time and support.

Best wishes,

Authors

Reviewer_evDTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the sparse reward in GRPO based RLVR training. They view a program as a composition of syntax-tree units, and find that even a failing solution can contain some useful units. Therefore, they propose ASTrace, based on AST reward attribution method. It pases the sampleed program into units, including the functions, methods, and top-level blocks. The empirical experiments on diverse benchmarks show the improvements than other baselines.

Critical strengths
  1. The figure is clean and the paper is well-written, easy to follow.
  2. I think the idea is kind of novel and interesting, which utilizes the units in the fine-grained level for credit assignment.
  3. They conducted the experiments across diverse benchmarks, which is comprehenive.
Critical weaknesses
  1. I am wondering what is the CodeScaler's model size used in your experiments? I did not find the information in the paper.
  2. I also noticed in table 1, Qwen3-4B with CodeScaler can even observe the performance drops? can you explain that? Since in CodeScaler, it performs quite well on both 8B and 14B scales.
  3. In addition, the experiments are conducted on instruct models, did you do the experiments on base models? maybe it will have better improvements.
Questions for authors

See weaknesses.

3 comments

We thank Reviewer evDT for finding the idea "novel and interesting", the paper "well-written, easy to follow", and the experiments "comprehensive". We answer the three questions below, with a new analysis of the CodeScaler baseline and new training from a base model.

[W1. CodeScaler model size]

  • We use the released CodeScaler-4B reward model (LARK-Lab/CodeScaler-4B), frozen during training. Its score is the entire reward, with no test execution, and, following the official implementation, a response without exactly one parsable code block receives zero reward. We add these details to Appendix A.

[W2. CodeScaler below the base model on Qwen3-4B]

  • The CodeScaler paper trains Qwen3-8B/14B-Base policies with its own recipe. Our comparison instead isolates the reward signal: we plug the CodeScaler reward model into the protocol shared by all methods (same backbone, data, steps and evaluation), so a learned reward is compared directly with execution-based rewards.
  • Under this protocol, most of the drop on Qwen3-4B comes from an output-format failure. On HumanEval and HumanEval+, 10.4% of CodeScaler samples open the code block twice, which the answer extractor shared by all methods cannot parse. No GRPO or ASTrace sample on any benchmark shows this. The format drifts during training (0% of such samples at step 200, 9.9% at step 400 on HumanEval), because a learned reward never runs the code and so never penalizes output that cannot be executed. After we remove the repeated fence and keep every other verdict unchanged, CodeScaler matches the base model on average and still falls below GRPO and ASTrace on every benchmark:
Qwen3-4B, pass@1 (%)HumanEvalHumanEval+BCB-FullBCB-HardLCB v6ClassEvalAvg.
Base model89.1084.6039.7919.4330.7926.1248.31
CodeScaler (Table 1)80.0176.6837.4313.9633.9028.4645.07
CodeScaler, repeated fence removed89.6386.2037.4313.9634.8328.4648.42
GRPO91.0686.2640.0320.2738.8131.2951.29
ASTrace93.2788.4140.1722.8940.6732.9253.05
  • The drop is therefore a format failure rather than lost coding ability, and the conclusion of Table 1 is unchanged.
  • It also illustrates why ASTrace derives all of its credit from execution: a reward that runs the program cannot be satisfied by output the executor rejects.
  • We keep the strict numbers in Table 1, since the protocol is identical for all methods, and add the re-graded row with this explanation to the appendix.

[W3. Base models]

  • We start from instruct checkpoints, as recent work on RL for code does [1, 2, 3], because this keeps the comparison focused on the reward. From a base model, early gains are dominated by learning the output format: AceCoder reports a gain of over 25% on HumanEval+ within 80 steps from Qwen2.5-Coder-7B-Base, against 0.7 points on average from the instruct model [2], and format requirements are known to hinder exploration in base models [4].
  • To test this directly, we trained GRPO and ASTrace from Qwen3.5-4B-Base. Given the rebuttal timeline, we ran the first training stage (200 steps, about 240 GPU-hours) with the data and hyperparameters of the main experiments, and evaluated with 8 samples per problem.
Qwen3.5-4B-Base, pass@1 (%)LCB v6ClassEvalHumanEvalHumanEval+Avg.KodCode val
Before RL21.4315.1267.8463.4941.9731.8
GRPO42.0029.6294.7490.0964.1165.0
ASTrace42.5032.1294.9791.5465.2867.4
Δ vs. GRPO+0.50+2.50+0.23+1.45+1.17+2.4
  • Both methods gain more than 20 points on average within 200 steps, so most of the early gain from a base model is shared by both rewards, which supports our choice of instruct checkpoints for comparing rewards.
  • On top of this shared gain, ASTrace improves over GRPO on all four benchmarks and on the held-out KodCode validation split.

We hope these answers address the reviewer's questions, and we would be glad to discuss any remaining concerns.

[1] Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generation.

[2] AceCoder: Acing Coder RL via Automated Test-Case Synthesis.

[3] Execution-Grounded Credit Assignment for GRPO in Code Generation.

[4] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild.

Thanks for your response, and i think most of my concerns have been addressed. I would like to raise my score to 3.

Dear Reviewer evDT,

We sincerely thank you for confirming that our rebuttal has adequately addressed the concerns. We greatly appreciate the constructive feedback throughout this process, which has helped improve the quality of our work.

Best, Authors

Reviewer_ss7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the sparse-reward problem of group-based RLVR for code, where GRPO gives all-zero advantages to groups in which every sampled program fails. It argues that a failing program is a composition of AST units (functions, methods, top-level blocks) and often contains units that already pass tests. ASTrace adds unit-level credit on top of the unchanged binary GRPO reward via two routes: in all-fail groups, a progress credit estimates each unit's Shapley contribution to passing a few "signal" tests by restoring disabled units in one sampled order (Eqs. 4–5); in partial-fail groups, failure localization assigns −1 to the unit where a missed test first raises an error, taken from the traceback or a call tracer at no extra execution (§4.3). Credits are split over unit tokens, batch-standardized, clipped, and added to the GRPO advantage with λ=0.1 (Eq. 7). On Qwen3-4B and Qwen2.5-Coder-7B over six benchmarks, ASTrace raises mean pass@1 over GRPO on all twelve benchmark–backbone pairs and is best among trained methods on 11/12 (Table 1), with gains extending to OJBench and to C++ LCB-Pro despite Python-only training, at 1.16× GRPO wall-clock.

Critical strengths
  1. The motivation is empirically grounded rather than assumed: 80.5% of all-fail ClassEval programs contain a unit that passes signal tests, and swapping one localized unit for a passing sibling's repairs 58.6% of failing programs (§1, Table 13), which directly motivates unit-level credit.
  2. The progress credit is methodologically clean: restoring disabled units in a uniformly random order is an unbiased single-permutation Shapley estimate, and the credits sum exactly to the rollout's own net test gain, preventing double counting when units are jointly needed (§4.2).
  3. Failure localization is essentially free and well validated: it reuses the terminal run's traceback/call trace, and the localized unit repairs 58.3% of failures on swap vs 13.5% for an unblamed unit (Table 3), a strong causal-style check that the signal lands in the right place.
Critical weaknesses
  1. The reported SDs are across three decoding seeds of one checkpoint, and the paper concedes this in C.2. Several headline deltas are small (Coder-7B mean +0.88; HumanEval +0.10, BCB-Full +0.14 for Qwen3-4B in Table 1), so they are not distinguishable from training-run noise.
  2. HumanEval/HumanEval+ share 164 problems and BCB-Hard ⊂ BCB-Full, which the paper notes; on HumanEval the gap is ~2–3 correct problems.
  3. Progress-only scores below GRPO (37.36 vs 38.86 on LCB, 30.75 vs 32.00 ClassEval, Table 2a), and the benefit is explained by a specific interaction with localization's batch mean — but that synergy is shown for only one seed on two benchmarks.
Questions for authors

See the Weaknesses

4 comments

We thank Reviewer ss7x for the careful reading and for finding our motivation "empirically grounded" and the progress credit "methodologically clean". We trained two new seeds, ran paired per-problem tests, and re-evaluated the ablation under the full protocol, and address each concern below.

[W1. Training-run noise]

  • Training each method once and averaging over several samples per problem is the standard protocol in RL for code and reasoning [1, 2, 3, 4].
  • To measure training-seed variance directly, we trained GRPO and ASTrace on Qwen3-4B with two new seeds. Given the rebuttal timeline, we ran the first training stage (200 steps, about 145 GPU-hours for the four new runs) and compared against the step-200 checkpoints of the original runs, with identical data, hyperparameters, hardware and evaluation within each seed.
Δ pass@1 (ASTrace − GRPO)LCB v6ClassEvalHumanEvalHumanEval+BCB-FullBCB-HardAvg.
seed 0+2.93+1.88+1.68+1.75+0.09+2.16+1.75
seed 1 (new)+1.50+2.37+1.07+1.68−0.14+1.96+1.41
seed 2 (new)+2.69+1.62+1.23+1.84+0.21+2.62+1.70
mean+2.37+1.96+1.33+1.76+0.05+2.25+1.62
  • ASTrace improves over GRPO on five of the six benchmarks in every seed, with an average gain between +1.41 and +1.75, so the improvement is not training-run noise. BCB-Full stays within ±0.21 in all seeds, in line with the +0.14 of Table 1.
  • On Qwen2.5-Coder-7B, the headroom over GRPO is smaller, as expected from an instruct model that already starts strong; we use instruct checkpoints to isolate the comparison between rewards. Within this setting the gain is consistent across coding tasks: ASTrace improves over GRPO on all six benchmarks in Table 1 and on OJBench (Appendix D).

[W2. Overlapping benchmarks and problem counts]

  • We keep both members of each pair, as prior work does [1, 4], because each pair measures something different: HumanEval+ adds stricter tests to the same problems, and BCB-Hard isolates the hardest BCB tasks.
  • To rule out double counting, we ran paired per-problem tests (pass@1 over all 24 samples per problem, problem-level bootstrap CI). Pooled over the four benchmarks with disjoint problems (HumanEval, BCB-Full, LCB v6, ClassEval; 1,579 problems, equal weight per benchmark), ASTrace improves over GRPO by +1.46 [+0.24, +2.66] (p = 0.017), so the gain holds when no problem is counted twice. Per benchmark:
Qwen3-4B# problemsΔ pass@1 [95% CI]problems improved / regressed
HumanEval164+2.21 [+0.15, +4.55]19 / 9
HumanEval+164+2.16 [+0.36, +4.22]18 / 5
BCB-Hard148+2.62 [+0.96, +4.39]33 / 12
LCB v6175+1.86 [+0.21, +3.62]40 / 23
ClassEval100+1.62 [−2.25, +5.33]23 / 14
BCB-Full1140+0.14 [−0.74, +1.02]217 / 239
  • The CI excludes zero on four of the six benchmarks, and BCB-Hard remains significant after Holm correction over all six (p = 0.019).
  • On the problem count, pass@1 counts samples rather than problems. On HumanEval, +2.21 corresponds to 87 more correct samples out of 3,936. The number of problems solved at least once barely moves (159 → 160), while the number solved in all 24 samples rises from 134 to 142. The gain comes from making near-miss problems reliable, which a count of solved problems does not show. The same holds on every benchmark with a clear gain: 51 to 74 of the extra correct samples come from problems GRPO solves only partially, and more of these problems improve than regress (e.g., 27 vs. 9 on BCB-Hard).

(continued in the next comment)

[W3. Robustness of the ablation]

  • We evaluated the Progress-only and Localization-only checkpoints of Table 2a under the full protocol of Table 1, with three decoding seeds on all six benchmarks (mean ± SD).
Qwen3-4B pass@1LCB v6ClassEvalHumanEvalHumanEval+BCB-FullBCB-HardAvg.
GRPO (Table 1)38.81±0.1531.29±0.6491.06±0.0986.26±0.2240.03±0.1420.27±0.3751.29
Progress-only32.45±4.2630.92±0.1490.04±0.9385.17±0.9339.59±0.4819.14±0.3949.55
Localization-only39.31±0.4031.78±0.1991.46±1.2286.38±1.5340.15±0.1819.82±0.3951.48
ASTrace40.67±0.1132.92±0.5693.27±0.2388.41±0.2040.17±0.1122.89±0.5353.05
  • The ordering of Table 2a (Progress-only < Localization-only < ASTrace) holds on all six benchmarks. Progress credit alone falls below GRPO (−1.74 on average), yet adding it to localization yields a further +1.57, and ASTrace is best on every benchmark.
  • This follows from Eq. (7), which subtracts the batch-mean credit. Without localization, nearly all credits are positive, so a useful unit with below-average credit receives a negative advantage, and in all-fail groups, where the GRPO advantage is zero, this is its only signal. With localization, blamed units (−1 each) outnumber progress rollouts (at most +6 each) by nearly 9 to 1 during training (Table 3), so the batch mean is negative and every positive progress credit stays positive. We add this table to the revision.

We would be glad to discuss any remaining questions.

[1] Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generation.

[2] CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models.

[3] DAPO: An Open-Source LLM Reinforcement Learning System at Scale.

[4] AceCoder: Acing Coder RL via Automated Test-Case Synthesis.

Dear Reviewer ss7x,

Thank you again for your careful review and the time you spent on our paper. In our response, we:

  1. trained GRPO and ASTrace with two more seeds on Qwen3-4B.
  2. ran paired per-problem tests on the four benchmarks that share no problems.
  3. evaluated the ablation checkpoints under the full six-benchmark, three-seed protocol.

We would like to know whether these results address your concerns. We are happy to answer any further questions before the discussion period ends.

Best regards, The Authors

Dear Reviewer ss7x,

As the author–reviewer discussion period ends soon, we would greatly appreciate it if you could let us know whether our additional experiments and analyses have addressed your concerns. We would also be happy to provide any further clarification if needed.

Thank you again for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and constructive comments. We are glad that the reviewers found the motivation "empirically grounded" and the progress credit "methodologically clean" (ss7x), the idea "novel and interesting" and the paper "well-written, easy to follow" (evDT), and the two-route framework "clear and intuitive" with "convincing empirical performance" (13um). We summarize the new experiments and analyses below; full results are in the individual responses.

[More training seeds and new backbone model]

  • Robustness across training seeds. (ss7x) We trained GRPO and ASTrace on Qwen3-4B with two additional seeds. ASTrace improves over GRPO on five of the six benchmarks in every seed.
  • Newer backbone, trained from a base model. (evDT, 13um) We trained GRPO and ASTrace from Qwen3.5-4B-Base, a newer model generation and a base rather than instruct checkpoint. ASTrace improves over GRPO on all benchmarks and on the held-out validation split.

[New analyses]

  • Statistical tests and ablation. (ss7x) Paired per-problem tests confirm the gain on benchmarks with disjoint problems, and the ablation of Table 2a keeps its ordering under the full six-benchmark, three-seed protocol.
  • CodeScaler baseline. (evDT) We report the reward model size and show that the drop on Qwen3-4B comes from an output-format failure; the conclusion of Table 1 is unchanged.
  • Fidelity of the credit signals. (13um) Comparing against exact Shapley values, the single-order estimate never reverses the sign of a unit's credit. A swap analysis shows that the localized unit is part of the fault in most failures, and the remaining cases lie one call away.

[Revisions]

  • We add the new tables and analyses above to the paper, together with the CodeScaler-4B configuration, the multi-order option for progress credit, and the caller/callee extension of localization.

We hope our responses address the reviewers' concerns, and we would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.