- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In code-agent RL with GRPO, all test-passing trajectories in a rollout group receive identical positive advantages, ignoring differences in implementation quality. The paper proposes GAGAR (Groupwise Agentic Grading for Advantage Redistribution), in which an interactive agentic grader jointly inspects the trajectories, patches, repository, and test outputs of each group and ranks the test-passing candidates along five criteria: suitability of the solution approach, implementation precision, minimality of changes, avoidance of side effects, and consistency with codebase conventions. A sum-preserving scaling factor (Eq. 2) then reallocates the positive credit removed by quality-based downweighting back to the passing trajectories in proportion to their quality, conserving total positive advantage (Eq. 3–4) while leaving failed-trajectory advantages unchanged. The method is validated at industrial scale on pre-RL SFT checkpoints of MiMo-V2.6-Flash (310B total, 15B active) and MiMo-V2.6-Pro (1.02T total, 42B active).
Critical strengths
- The redistribution is grounded in an explicit conservation law. Equations 2–4 clearly establish total-positive-advantage preservation and zero-sum change within the passing subset, and the equivalent quality-mean form (Eq. 4) makes the credit-shifting interpretation transparent. The downweight-only ablation (Section 4.4, Fig. 4) directly substantiates the central claim that downweighting without redistribution breaks the balance between positive and negative advantages: policy entropy rises from 0.358 to 0.69k versus 0.513 to 0.539, the policy-gradient loss turns consistently negative, and DeepSWE pass rates become unstable (56.5% → 48.8% → 56.2%) while rollout length balloons.
- The blind quality evaluation is methodologically careful. The 30-task DeepSWE study uses graders from a different model family (Claude Opus 5, distinct from the online grader), anonymized identifiers, and randomized presentation order, with joint groupwise review. It provides evidence beyond pass rates — rubric-weighted quality 4.03 vs. 3.70, 69.8% average win rate, and ranked first in 65.0% of groups at the final shared checkpoint.
- Genuine industrial-scale validation with efficiency co-benefits. The mixed-task RL runs span 310B and 1.02T models, and the paper reports not only performance but also interaction savings (at step 28 on DeepSWE: mean turns −15.6%, 132.3 → 111.6; mean tokens −9.9%, 191.9k → 172.9k), alongside more stable training dynamics.
Critical weaknesses
- The industrial-scale results cannot be attributed to GAGAR. Table 1 compares Flash/Pro mixed-RL checkpoints only against external frontier models (K3, GPT-5.6 Sol, Claude Opus 5); there is no "without GAGAR" row produced under the same mixed-task RL recipe. Meanwhile, the only controlled binary-reward baseline is stopped at step 28 due to its own degradation, so neither experiment isolates GAGAR's contribution at scale. The causal claim that GAGAR drives 71.9 / 62.7 is currently unsupported.
- Grader validity and same-family self-preference are unaddressed. The same pre-RL SFT checkpoint of MiMo-V2.6-Pro serves as the online grader for both the Flash and Pro experiments, and it is SFT-trained on Opus 5 labels. The paper reports neither the ranking agreement (e.g., Kendall's τ / Spearman) between the distilled grader and Opus 5 (or humans), nor any robustness check with a grader from a different model family. A grader that shares ancestry with the policy it scores may favor its own family's coding style, which would bias both training and the internal quality measurements.
- Evaluation breadth and rigor are limited. (a) Only DeepSWE v1.1 and SWE-bench Pro are used; the standard SWE-bench Verified benchmark is absent. (b) All controlled results appear to come from a single training seed (three samples per task at eval is not a seed study), so the stability claims and the 12.1 pp gap at step 28 lack variance estimates. (c) The tier-based discount factors are hand-crafted with no sensitivity analysis, even though the entire redistribution magnitude depends on them. (d) No empirical comparison is provided against closely related process-reward / credit methods discussed in related work, notably ReCode and PAPO.
Questions for authors
- Does the same-family / same-checkpoint grader introduce self-preference? Please report ranking agreement between the online grader and Claude Opus 5 (on the 30 audited tasks, or a fresh set), and rerun the controlled study (or at least the grading/quality evaluation) with a grader from a different model family.
- Can you add a "without GAGAR" same-scale mixed-task RL row to Table 1, and report GAGAR's gains under matched FLOPs / matched optimizer steps (rather than comparing at step 28, where the baseline is already collapsing)?
- How does GAGAR perform relative to ReCode- and PAPO-style process rewards under the same setup? Can you also (a) add SWE-bench Verified, (b) report multiple training seeds with variance, and (c) provide a sensitivity sweep over the tier discount factors / redistribution strength?
- After asynchronous overlap with rollout collection (Section 3.4), what fraction of end-to-end wall-clock training time does grading actually occupy? The per-group figures (~2000 s for Opus 5, ~600 s for the SFT grader) are given, but not the net overhead after partial-rollout scheduling.