acceptodds
Under review as a conference paper at ICLR 2027

Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair

Abstract

Code-agent RL often operates under weak online execution feedback: rollout-time signals are reliable and executable, but capture only necessary conditions rather than the target semantic predicate. Using agentic compile-fix as the setting, we study signal reshaping for GRPO with offline reference-conditioned reward construction. Our central claim is that GRPO's within-group comparison is meaningful only after three kinds of signals are reshaped: outcome rewards recover semantic ranking, process signals localize intra-trajectory credit, and rollouts from the same prompt remain execution-comparable. We operationalize these conditions by reshaping signals around GRPO: compile-and-semantic layered rewards reshape trajectory ranking, step-level process scores placed outside group reward normalization act as token loss weights to reshape intra-trajectory update strength, and failure-cause-aware rollout governance preserves within-group comparability. Experiments show a clear end-to-end gain: full signal-reshaped GRPO improves strict compile-and-semantic accuracy from the base model's zero-shot to . Controlled comparisons further explain the source of this gain: binary rewards remove the compile-only middle tier and degrade trajectory control; on top of layered rewards, process-score weighting further improves accuracy from to and reduces average evaluation steps from to . As a boundary comparison, token-level privileged self-distillation underperforms step-level process weighting, showing that local distribution matching cannot replace action-step credit.

Primary area:
reinforcement learning
Keywords:
reinforcement learning for code agents, GRPO, reward shaping, weak execution feedback, process-level credit assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5GJ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies GRPO training for agentic code repair under weak execution feedback. It proposes a signal-reshaping framework combining layered semantic rewards, step-level process-score weighting, and failure-aware rollout handling. On an internal C/C++ compile-fix benchmark, the full method improves strict repair accuracy from 38.5% to 53.5%.

Critical strengths

The paper identifies an important mismatch in agentic RL: compilation is a useful but incomplete reward, and the decomposition into outcome ranking, intra-trajectory credit, and execution comparability provides a coherent framework for addressing this issue.

The empirical validation is relatively strong. Beyond the training reward model, Tables 6 and 7 use an independent LLM judge and blind human evaluation on the same 400 instances, both showing a statistically significant improvement of GRPO-full over GRPO-early.

Critical weaknesses

The layered reward relies on a questionable and insufficiently justified ordering assumption. It assigns every compiling but semantically incorrect patch a higher reward than every non-compiling patch (0.5 vs. 0), even though a nearly correct repair with a minor compilation error may be much closer to the target than a compilable patch that substantially changes or even reverses the intended behavior. Moreover, the specific choice of 0/0.5/1 is heuristic, and no sensitivity analysis is provided for alternative intermediate rewards.

The paper improves how semantic feedback is incorporated into GRPO, but does not substantially address the more fundamental problem of how reliable semantic supervision should be obtained in weak-feedback code repair. I do not consider the use of an LLM-based semantic judge itself to be a weakness; the paper provides reasonable human and independent-judge validation of Kimi-K2.5. My concern is instead about the scope of the claimed contribution. Recent work in code generation and automated program repair has already explored richer supervision beyond sparse compile/test outcomes. For example, BoostAPR learns sequence- and line-level reward models from execution-grounded repair outcomes [1], CODERL+ explicitly introduces execution-semantic signals into RL rather than relying only on binary correctness [2], and recent agentic program-repair systems combine static analysis, test execution, and LLM-based judging as complementary validation signals [3]. LLM-based semantic assessment of generated patches is itself also becoming an established tool in software-engineering evaluation [4]. Against this background, the central difficulty is not simply recognizing that compilation is semantically incomplete and replacing it with a stronger LLM-derived signal. The harder question is how to obtain semantic supervision that is grounded enough to go beyond the limitations of the evaluator itself. In this paper, semantic correctness is ultimately supplied by a strong offline LLM judge that is conditioned on a ground-truth repair, and the step-level process signal is produced by the same class of model-based supervision. The proposed signal reshaping therefore provides a better optimization interface for already-available semantic supervision, but it does not reduce the fundamental dependence on such supervision. This distinction is particularly important for the paper's weak-feedback framing: if a sufficiently strong GT-conditioned semantic oracle is assumed to be available during training, then a substantial part of the original semantic-feedback problem has already been outsourced to that oracle. I would therefore like the paper to more clearly distinguish its contribution to credit assignment and reward utilization from the broader problem of acquiring reliable semantic feedback, which, in my view, remains largely unresolved.

[1] Li, Yuanhao, Hongbo Wang, Xiaotang Shang, Xunzhu Tang, Yiming Cao, and Xuhong Chen. “BOOSTAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models.” In Proceedings of the 43rd International Conference on Machine Learning, 70897–70918. Proceedings of Machine Learning Research 306. PMLR, 2026.

[2] Jiang, Xue, Yihong Dong, Mengyang Liu, Hongyi Deng, Tian Wang, Yongding Tao, Zhi Jin, Wenpin Jiao, and Ge Li. “CODERL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment.” In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3608–3622. Association for Computational Linguistics, 2026. ACL Anthology

[3] Maddila, Chandra, Adam Tait, Claire Chang, Daniel Cheng, Nauman Ahmad, Vijayaraghavan Murali, Marshall Roch, et al. “Agentic Program Repair from Test Failures at Scale: A Neuro-Symbolic Approach with Static Analysis and Test Execution Feedback.” IEEE Transactions on Software Engineering 52, no. 8 (2026): 2446–2462. https://doi.org/10.1109/TSE.2026.3696849

[4] Zhou, Xin, Kisub Kim, Ting Zhang, Martin Weyssow, Luís F. Gomes, Guang Yang, and David Lo. “An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks.” arXiv preprint arXiv:2505.20854, 2025.

Questions for authors
  1. Reward ordering. The layered reward assumes that every compiling but semantically incorrect patch should receive a higher reward than every non-compiling patch (0.5 vs. 0). What justifies this ordering? For example, a nearly correct patch with a minor compilation error may be closer to the target than a compilable patch that substantially changes the intended behavior. Have you tested alternative orderings or intermediate reward values?

  2. Dependence on a strong semantic oracle. The method relies on a GT-conditioned Kimi-K2.5 judge to provide both semantic outcome rewards and process-level supervision. To what extent is the proposed framework still applicable when such a strong semantic evaluator or ground-truth repair is unavailable? More generally, is the main contribution intended to address weak feedback itself, or to improve optimization given access to a strong offline semantic oracle?

  3. Sensitivity to judge quality. The human audit shows that the semantic judge is reasonably accurate but still imperfect. Have you evaluated how training performance changes when the semantic judge is replaced by another model or when controlled noise is introduced into its labels? This would help clarify whether the reported gains depend on the particular quality of Kimi-K2.5.

2 comments

Response to Reviewer 5GJ1

Q1. Why give higher rewards to patches that compile but are incorrect, and why choose 0.5?

The reward reflects the checks a patch has passed. Passing compilation is necessary for semantic success and a successful repair, so it earns partial credit. Full credit requires passing both compilation and semantic checks.

The ablation results also show that adding the intermediate reward improves strict accuracy by 0.5pp, compilation success by 3.5pp, and single-edit success by 7.4pp. This shows that an intermediate reward helps the model make progress on repairs. We tested an intermediate reward of 0.5. We have not tested other values or reversed the reward order.

Q2. Does the method solve the weak-feedback problem or make better use of existing semantic supervision?

We thank the reviewer for pointing out the distinction between where supervision comes from and how it is used. The task originally provides only weak compiler feedback. We introduce semantic supervision during training through reference repairs and a judge. We then use outcome rewards to compare trajectories and process scores to adjust updates at each step, using this added supervision for both trajectory ranking and fine-grained learning. During rollout, the agent still receives only compilation feedback, and reference repairs are not needed at deployment. Our method therefore adds semantic supervision to the original weak feedback and uses it to guide both trajectory selection and step-level learning.

Q3. What happens if the training judge is replaced or label noise is added?

We changed the judge during evaluation: GLM-5.2 and blind human evaluation found gains of 5.75pp and 6.25pp for Full over Early, respectively. The improvement holds when the evaluator changes. We have not run experiments that replace the judge during training or add controlled noise to the labels.

Thank you for the clarifications. However, my main concerns remain largely unresolved.

For Q1, the reported ablation supports the usefulness of an intermediate reward, but does not justify the specific reward ordering or value. For Q2, the response clarifies the distinction between training and deployment, but does not establish how the framework would perform without strong offline semantic supervision. For Q3, changing the evaluation judge does not address sensitivity to the judge used during training, which was the focus of my question. Furthermore, while the response clarifies that the method uses strong offline semantic supervision to improve optimization under weak rollout feedback, it does not sufficiently establish the methodological novelty beyond existing approaches that leverage richer semantic and process-level signals for code repair (which I mentioned in Critical Weaknesses of the original review).

I appreciate the authors' transparency in acknowledging these limitations. However, without additional experiments directly addressing these concerns, I do not find sufficient evidence to revise my assessment. I therefore maintain my original rating of Reject.

Reviewer_akzWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a signal reshaping framework for GRPO in agentic code repair tasks under weak-feedback environments. To address the limitations of GRPO over long trajectories and weak feedback, the authors introduce signal reshaping at three levels: layered rewards, step-level process scores, and failure-cause-aware rollout governance. Experiments conducted on the Qwen3-Coder-30B-A3B-Instruct model demonstrate that the full signal-reshaping framework improves strict compile-and-semantic accuracy from 0.385 to 0.535.

Critical strengths
  1. The framework design is modular, with clear responsibilities assigned to each of the three components. By modifying where and how signals enter the GRPO training pipeline,the proposed framework lowers the complexity of iintegration with existing RL frameworks, offering potential for transferability to other settings.
  2. The paper provides a detailed data preprocessing pipeline. Minimal Sufficient Repairs removes unrelated refactoring and stylistic changes mixed into human commits. This data-cleaning step is essential for ensuring the reliability of offline semantic judgment.
Critical weaknesses
  1. The framework relies heavily on LLM-based scoring, requiring Kimi-K2.5 to evaluate trajectories step by step. This introduces external dependencies, potential reliability concerns, and substantial computational costs.
  2. Model and domain generalization require further validation. The experiments are limited to a single base model and a C/C++ compile-fix benchmark. Further experimental evidence is needed to establish whether the proposed method generalizes to other models and benchmarks.
  3. The main experimental results are presented in Figures 2 and 3 rather than in a summary table, lacking clarity and direct readability.
Questions for authors
  1. The paper does not ablate the individual components of rollout governance. Could the authors provide partial ablation results, such as retaining only the last step versus applying a full mask?
  2. Could the authors provide generalization experiments using additional models or benchmarks?
1 comment

Response to Reviewer akzW

Q1. Can you compare keeping only the last step with masking the whole trajectory?

These rules handle different problems and cannot simply replace each other. Whether the model should receive a penalty depends on whether it caused the failure.

When the sandbox or infrastructure fails, the model should not be penalized, so we mask the whole trajectory. When the model’s generation degenerates, we keep the last step so the model receives a penalty. We train on ordinary repair failures as usual. Keeping the last step for every failure would penalize the model for environment failures; masking every failed trajectory would also remove penalties for the model’s own errors.

Applying either rule to all failures would mix up cases that need different treatment. Such a comparison can show what happens when failures are handled incorrectly, but it does not tell us which rule should replace the other.

Q2. Can you add other models or benchmarks?

Our experiments ask whether changing how we use rewards and process scores helps under the same conditions. All experimental settings use the same base checkpoint, data, sandbox, and budget, so the differences can be traced to these changes. Adding other models or benchmarks is beyond what this paper needs to demonstrate.

How reliable is scoring, how much does it cost, and can you show the results in a table?

Both an independent judge and blind human evaluation confirm Full’s improvement on the same 400 problems:

EvaluationFull minus Early
Training judge+5.00pp
Independent GLM-5.2 judge+5.75pp (p=0.018)
Blind human evaluation+6.25pp (p=0.008)

About 89% of training trajectories receive process weighting. Over the same range of training iterations, rollout time increases from 969s to 1177s per iteration, an increase of about 208s, or 21%.

Reviewer_xmMEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses weak-feedback agentic code repair, where a policy trained with GRPO receives execution signals (compilation success/failure) that are reliable but capture only necessary conditions for semantic correctness. The authors identify three ways this breaks GRPO's within-group ranking: binary rewards conflate compile-only shortcuts with genuine fixes; sparse outcome rewards fail to credit individual steps within long trajectories; and infrastructure failures mix with genuine policy failures, corrupting group comparisons. To address these, the paper proposes three signal-reshaping interventions that wrap around the unchanged GRPO objective: (1) layered {0, 0.5, 1} outcome rewards separating non-executable, compile-only, and semantically correct repairs; (2) step-level process scores emitted by a judge and used as token loss weights outside group normalization; and (3) failure-cause-aware rollout governance routing unlearnable exceptions to masking, last-step retention, or normal termination. Experiments on Qwen3-Coder-30B with a 1110-sample training set show that full signal-reshaped GRPO improves strict compile-and-semantic accuracy from 0.385 to 0.535. Controlled comparisons attribute roughly half the gain to layered rewards and the other half to process-score weighting, with governance enabling stable training rather than directly improving accuracy.

Critical strengths
  • The problem formulation is precise. Identifying three distinct failure modes in GRPO's within-group comparison—outcome semantics, diffuse process credit, within-group comparability—and building a separate intervention for each is more rigorous than the usual "add reward shaping and see what happens."

  • The ablation structure is unusually clean. GRPO-binary, GRPO-early, and GRPO-full share a training fork point and diverge only in reward form and process scores. This is much stronger evidence for attribution than most RL papers provide, where ablations typically differ in multiple dimensions simultaneously.

  • Corroboration via an independent judge strengthens the results. Using GLM-5.2 as a second measurement channel—separate from the Kimi-K2.5 judge used in reward construction—and obtaining +5.75pp (p=0.018) removes a significant concern about the model learning to exploit the specific judge's preferences rather than genuinely improving repair quality.

Critical weaknesses
  • The empirical scope is narrow in ways that matter for the claims. Everything rests on a single proprietary dataset from one internal CI build system and one base model family. The paper acknowledges this, but the theoretical framing is general enough that readers will want to know how much of the +15pp gain is specific to this data distribution versus genuinely addressing structural problems with GRPO under weak feedback.

  • The semantic judge's reliability sits at 0.88–0.93 depending on error type, meaning roughly 7–12% of reward labels are incorrect. The paper audits this with human annotation, which is good practice, but there is no analysis of how noisy reward labels interact with process score weighting. If the judge systematically mislabels a particular error category, process scores could amplify those errors rather than correct them.

  • The process score judge is conditioned on full trajectory context at 2048 tokens. Longer trajectories—the paper reports up to 50 steps—could exceed this window, and there is no analysis of what happens at the tail of the length distribution. If the judge truncates context, process scores for early steps in long trajectories may be unreliable.

Questions for authors
  1. How does the semantic judge's behavior evolve over training? Is there evidence that the policy learns to produce outputs satisfying the Kimi-K2.5 judge without actually fixing the underlying error? The blind human audit validates the judge's reliability on static samples but does not directly test for judge exploitation during RL.

  2. The process score judge operates on trajectories up to 2048 tokens. What fraction of training trajectories exceed this window, and how does the method handle scoring when the full context does not fit? Is there degradation in process score quality for longer trajectories?

  3. Would the signal-reshaping framework transfer to agentic tasks with different weak-feedback structures—for example, test-suite execution where some tests are fast and cheap (like compile) and others are slow or expensive (like integration tests)? The three conditions seem general, but compile-fix may have structural properties that make the specific interventions easier to design.

  4. The group size is fixed at K=8. For the layered reward with three distinct values, the within-group ranking signal depends on having a mix of outcomes in each group. How often do all K rollouts land in the same reward tier, and does this vary significantly across training stages? A group where all rollouts compile but none pass semantic judgment provides no ranking signal under the layered scheme.

2 comments

Response to Reviewer xmME

Q1. Is the model just learning to satisfy the training judge?

Full still improves over Early when evaluated by an independent judge and by humans in a blind evaluation. On the same 400 problems, Full scores higher than Early by 5.75pp under GLM-5.2 (p=0.018) and by 6.25pp under human evaluation (p=0.008), consistent with the 5.00pp gain under the training judge, Kimi-K2.5. Both evaluations support the gain measured by the training judge.

Q2. How does the 2048-token limit affect long trajectories?

The 2048-token limit applies to the judge’s output, not its input context. This setting does not cut off input trajectories longer than 2048 tokens. We will label it as “maximum output length” in the paper.

Q3. Can this apply to cheap tests followed by expensive integration tests?

Cheap tests can quickly rule out candidates that fail and provide a gradient signal through an intermediate reward level. Expensive integration tests then determine final success. This fits our problem setting and method directly.

Q4. How often do all K=8 trajectories receive the same reward?

We counted these groups in the training logs. Over iterations 240 through 399, each iteration contains eight groups of eight trajectories:

Training settingGroups with equal rewards / total groupsPercentage
Binary604/128047.19%
Early567/128044.30%
Full577/128045.08%

In Full, groups where every trajectory compiles but none passes the semantic check (R=0.5) account for 10.23% of all groups (131/1280). Groups where all rewards are R=1 account for 33.28%, making them the most common type of group with equal rewards. The other 54.92% have different rewards, which allow GRPO to rank trajectories within each group. When all rewards are equal, GRPO advantages are zero, and process weighting does not change this.

Can errors in outcome and process scores make each other worse?

We use outcome rewards to compare trajectories and process scores to compare steps within a trajectory, reducing how much the two signals affect each other. Adding both scores into a single reward would let process scores affect the ranking across trajectories. We use the relatively more accurate outcome reward to set the sign of the advantage. The judge’s ranking of steps then determines how much to adjust the update at each step. Process scores do not change the sign of the advantage, so scoring errors cannot reverse the update direction at that step.

Reviewer_xmMEOpenReview ↗

Reply to authors

My concerns have been resolved, therefore I keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.