acceptodds
Withdrawn from ICLR 2027

Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) for code agents often uses executable tests to provide binary rewards. With these rewards, Group Relative Policy Optimization (GRPO) assigns identical advantages to test-passing trajectories within each rollout group, overlooking differences in implementation quality and adherence to task requirements. This leaves the policy without a learning signal that favors clean, targeted implementations over those containing unnecessary or out-of-scope changes. We introduce GAGAR, a framework for quality-aware credit redistribution in code agent RL. Built on dynamic sampling that retains groups containing both passing and failing trajectories, GAGAR places all trajectories from each group in a shared workspace, where an SFT-trained agentic grader jointly inspects them and ranks the test-passing candidates. Based on this ranking, we downweight lower-ranked trajectories and proportionally rescale the advantages of all test-passing trajectories to restore their original sum. This sum-preserving redistribution retains the relative weights established by quality-based downweighting while shifting credit toward higher-quality implementations. We evaluate GAGAR at industrial scale using pre-RL SFT checkpoints of MiMo-V2.6-Flash (310B total parameters) and MiMo-V2.6-Pro (1.02T total parameters). Controlled code-only Flash experiments show improved code agent performance, reduced trajectory-length growth, and more stable training. We further apply GAGAR in large-scale mixed-task RL with both Flash and Pro. Our results support combining test-based verification with groupwise agentic grading to improve the quality and stability of code agent RL.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Code Agents, Agentic Judge, Groupwise Evaluation, Advantage Redistribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_JPGrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In code-agent RL with GRPO, all test-passing trajectories in a rollout group receive identical positive advantages, ignoring differences in implementation quality. The paper proposes GAGAR (Groupwise Agentic Grading for Advantage Redistribution), in which an interactive agentic grader jointly inspects the trajectories, patches, repository, and test outputs of each group and ranks the test-passing candidates along five criteria: suitability of the solution approach, implementation precision, minimality of changes, avoidance of side effects, and consistency with codebase conventions. A sum-preserving scaling factor λ\lambda (Eq. 2) then reallocates the positive credit removed by quality-based downweighting back to the passing trajectories in proportion to their quality, conserving total positive advantage (Eq. 3–4) while leaving failed-trajectory advantages unchanged. The method is validated at industrial scale on pre-RL SFT checkpoints of MiMo-V2.6-Flash (310B total, 15B active) and MiMo-V2.6-Pro (1.02T total, 42B active).

Critical strengths
  1. The redistribution is grounded in an explicit conservation law. Equations 2–4 clearly establish total-positive-advantage preservation and zero-sum change within the passing subset, and the equivalent quality-mean form (Eq. 4) makes the credit-shifting interpretation transparent. The downweight-only ablation (Section 4.4, Fig. 4) directly substantiates the central claim that downweighting without redistribution breaks the balance between positive and negative advantages: policy entropy rises from 0.358 to 0.69k versus 0.513 to 0.539, the policy-gradient loss turns consistently negative, and DeepSWE pass rates become unstable (56.5% → 48.8% → 56.2%) while rollout length balloons.
  2. The blind quality evaluation is methodologically careful. The 30-task DeepSWE study uses graders from a different model family (Claude Opus 5, distinct from the online grader), anonymized identifiers, and randomized presentation order, with joint groupwise review. It provides evidence beyond pass rates — rubric-weighted quality 4.03 vs. 3.70, 69.8% average win rate, and ranked first in 65.0% of groups at the final shared checkpoint.
  3. Genuine industrial-scale validation with efficiency co-benefits. The mixed-task RL runs span 310B and 1.02T models, and the paper reports not only performance but also interaction savings (at step 28 on DeepSWE: mean turns −15.6%, 132.3 → 111.6; mean tokens −9.9%, 191.9k → 172.9k), alongside more stable training dynamics.
Critical weaknesses
  1. The industrial-scale results cannot be attributed to GAGAR. Table 1 compares Flash/Pro mixed-RL checkpoints only against external frontier models (K3, GPT-5.6 Sol, Claude Opus 5); there is no "without GAGAR" row produced under the same mixed-task RL recipe. Meanwhile, the only controlled binary-reward baseline is stopped at step 28 due to its own degradation, so neither experiment isolates GAGAR's contribution at scale. The causal claim that GAGAR drives 71.9 / 62.7 is currently unsupported.
  2. Grader validity and same-family self-preference are unaddressed. The same pre-RL SFT checkpoint of MiMo-V2.6-Pro serves as the online grader for both the Flash and Pro experiments, and it is SFT-trained on Opus 5 labels. The paper reports neither the ranking agreement (e.g., Kendall's τ / Spearman) between the distilled grader and Opus 5 (or humans), nor any robustness check with a grader from a different model family. A grader that shares ancestry with the policy it scores may favor its own family's coding style, which would bias both training and the internal quality measurements.
  3. Evaluation breadth and rigor are limited. (a) Only DeepSWE v1.1 and SWE-bench Pro are used; the standard SWE-bench Verified benchmark is absent. (b) All controlled results appear to come from a single training seed (three samples per task at eval is not a seed study), so the stability claims and the 12.1 pp gap at step 28 lack variance estimates. (c) The tier-based discount factors fif_i are hand-crafted with no sensitivity analysis, even though the entire redistribution magnitude depends on them. (d) No empirical comparison is provided against closely related process-reward / credit methods discussed in related work, notably ReCode and PAPO.
Questions for authors
  1. Does the same-family / same-checkpoint grader introduce self-preference? Please report ranking agreement between the online grader and Claude Opus 5 (on the 30 audited tasks, or a fresh set), and rerun the controlled study (or at least the grading/quality evaluation) with a grader from a different model family.
  2. Can you add a "without GAGAR" same-scale mixed-task RL row to Table 1, and report GAGAR's gains under matched FLOPs / matched optimizer steps (rather than comparing at step 28, where the baseline is already collapsing)?
  3. How does GAGAR perform relative to ReCode- and PAPO-style process rewards under the same setup? Can you also (a) add SWE-bench Verified, (b) report multiple training seeds with variance, and (c) provide a sensitivity sweep over the tier discount factors / redistribution strength?
  4. After asynchronous overlap with rollout collection (Section 3.4), what fraction of end-to-end wall-clock training time does grading actually occupy? The per-group figures (~2000 s for Opus 5, ~600 s for the SFT grader) are given, but not the net overhead after partial-rollout scheduling.
Reviewer_EfYVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GAGAR, a method for incorporating implementation quality into RL for code agents. An agentic grader compares trajectories for the same task and ranks the test-passing implementations based on their approach, precision, minimality, side effects, and consistency with the codebase. These rankings redistribute positive advantages toward better implementations while preserving the total positive advantage and leaving failed-trajectory advantages unchanged. Experiments on DeepSWE and SWE-bench Pro report improvements in task success, implementation quality, and training stability, alongside applications in large-scale mixed-task training.

Critical strengths

(1) The paper addresses an important limitation of binary rewards. Passing tests does not establish that a patch is focused, maintainable, or free of unnecessary changes. Introducing quality distinctions among successful implementations is relevant to practical code-agent training. (2) The redistribution mechanism is simple and explicitly defined. (3) The controlled experiments show useful improvements beyond task success. (4) The method is demonstrated in substantial training settings. Its integration into both code-only and large-scale mixed-task training provides evidence of practical applicability.

Critical weaknesses

(1) The comparisons do not establish the advantage of the proposed redistribution rule over reasonable alternatives. The main ablation compares against quality downweighting without restoring positive credit or re-centering advantages. This shows that the chosen downweighting scheme can be unstable, but does not establish that sum-preserving redistribution is preferable to centered quality-reward shaping or other normalized quality adjustments. This is the main gap because the redistribution rule is a central contribution.

(2) The largest reported gain depends partly on baseline deterioration. The DeepSWE baseline reaches 58.5% at step 20 before falling to approximately 50.2% at step 28, where GAGAR achieves 62.2%. This is useful evidence for improved stability, but the large checkpoint-specific gap should not be interpreted as a consistent improvement of that magnitude. Stronger stability controls would help establish whether the proposed mechanism is needed to obtain this benefit.

(3) The evidence for implementation quality is limited. The quality analysis covers 30 DeepSWE tasks and relies on an automated evaluator. Using a separate evaluator is a strength, but the results do not directly establish reduced developer review effort or improved maintainability. Those broader practical claims require more evidence.

(4) The end-to-end computational trade-off is unclear. The online grader takes approximately 600 seconds per group. Asynchronous execution can reduce scheduling delays, but it does not eliminate the computation. Comparisons at equal training steps therefore do not establish whether GAGAR offers better performance under the same total compute budget.

Questions for authors

(1) How does sum-preserving redistribution compare with centered quality-reward shaping or re-centered quality-weighted advantages, using the same grader and training budget? This comparison would materially affect my assessment of the central contribution.

(2) Does the binary-reward baseline consistently deteriorate across independent runs? How sensitive is this behavior to length controls and optimization settings, and does GAGAR retain its advantage against a stable baseline?

(3) How uncertain are the quality improvements measured on the 30-task sample? Have the grader’s preferences been checked against human judgments or another independent assessment of patch quality?

(4) What fraction of retained groups contain multiple passing candidates with different quality factors? Since redistribution has no effect with a single passing candidate or identical factors, this would help explain when the method contributes.

(5) Are confirmed-hack detection and reward correction applied identically to the binary-reward baseline and GAGAR? If not, how much of the observed improvement comes from this correction rather than quality-based redistribution?

(6) How does GAGAR relate to MURPHY (Ekbote et al.) which assigns retrospective credit across feedback-conditioned code refinements? Please clarify the distinction between redistributing credit among successful trajectories and assigning credit to earlier attempts that enable later success.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.