acceptodds
Under review as a conference paper at ICLR 2027

JH-GRPO: Joint-Harness Group Relative Policy Optimization for Code Agent

Abstract

Coding agents can solve a repository task through one harness yet fail through another. Existing mixed-harness reinforcement learning optimizes mean reward rather than taskwise co-success across interfaces. We introduce Joint-Harness Group Relative Policy Optimization (JH-GRPO), which reallocates a fixed-size comparison group across native harnesses for the same task, without additional rollouts. To exploit this joint supervision, JH-GRPO optimizes for cross-harness co-success while assigning trajectory-specific credit and preserving harness-specific tool-use constraints. Specifically, a Joint Harness Coverage Utility rewards higher-order co-success, a Marginal Joint Advantage assigns this nonlinear utility to individual trajectories, and harness-conditional constraints localize penalties for invalid and foreign-schema tool calls. We also introduce Joint Harness Reliability (JHR) to measure taskwise co-success beyond mean Pass@1. Evaluated with Qwen3.5-35B-A3B across four coding harnesses and four software engineering benchmarks, our JH-GRPO improves macro Pass@1 from 48.8% to 53.7% and JHR@4 from 24.6% to 35.8% over the strongest mixed-harness baseline under matched compute. It improves worst-harness and unseen-harness Pass@1 by 6.6 and 3.5 percentage points, respectively, while reducing foreign-schema violations by 46.4%. These results demonstrate that taskwise joint optimization effectively improves cross-harness reliability.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Code Agent, Harness Robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_45pWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on the issue of unstable performance of the same code agent under different harnesses. For JH-GRPO, under a fixed rollout budget, it samples the same task through Task-Paired Cross-Harness Group Sampling, and then combines Joint Harness Coverage Utility, Marginal Joint Advantage, and Harness-Conditional Tool Constraints to train a model. The authors evaluated it on four native harnesses and four software engineering benchmarks. This problem is of practical significance, but more experiments are needed to explain why the method is effective, and some results need to be further clarified.

Critical strengths
  1. The research motivation is clear and has practical significance. When the same model is used with different harnesses, its performance may change significantly, which may affect the reliability of the code agent in actual use. The paper aims to ensure that the same task can be successfully completed under multiple harnesses, directly addressing this issue.
  2. The logic in the method section is relatively clear. The paper sequentially introduces Task-Paired Cross-Harness Group Sampling, Joint Harness Coverage Utility, Marginal Joint Advantage, and Harness-Conditional Tool Constraints, basically explaining what each component aims to solve and how they cooperate.
Critical weaknesses
  1. Further validation of key design choices and training dynamics

Table 2 provides useful ablations of the main components. However, it does not separately establish the benefits of localizing penalties to offending tool-call spans and adaptively updating the penalty weights. Comparisons against trajectory-wide penalties and fixed penalty weights, under otherwise matched settings, would help determine whether these specific design choices are necessary, rather than merely showing that adding tool penalties is beneficial.

Similarly, Figure 3 provides interference diagnostics, and Figure 5 reports aggregate learning curves, but the training dynamics remain insufficiently characterized. Tracking per-harness performance, reward statistics, and the distributions of marginal joint advantages throughout training would help explain how the method improves cross-harness co-success and whether it mitigates interference consistently across interfaces. Qualitative comparisons of execution traces before and after joint training—ideally on the same tasks across multiple harnesses—would also help connect the aggregate gains to concrete changes in tool use and task-solving behavior.

  1. Inconsistencies and ambiguities in result reporting

Several aspects of the result reporting require clarification:

  • Largest per-harness improvements. The discussion on page 7 states that the largest gains occur on Codex and OpenCode. However, relative to Mixed-Harness GRPO, Figure 2(a) shows that the full method improves SWE-Agent and OpenCode the most: approximately 5.3 and 6.2 percentage points, respectively, compared with 4.1 points for Codex. The authors should specify the intended reference baseline or revise the statement.

  • Joint-only versus “No tool constraints.” According to the definition on page 6, the joint-only variant excludes Harness-Conditional Tool Constraints and therefore appears to correspond to the “No tool constraints†” ablation in Table 2. Nevertheless, their Macro/JHR@4 scores differ: 52.4/33.9 in Table 1 versus 53.3/35.2 in Table 2. The corresponding foreign-schema violation rates also differ: 6.2 in Figure 5(a) versus 6.9 in Table 2. If these variants use different configurations or experimental protocols, those differences—and the meaning of the dagger marker—should be explicitly documented; otherwise, the reported values should be reconciled.

  1. The DeepSWE results require a more detailed explanation and contextualization. The official DeepSWE v1.1 report lists GLM5.2 at 44%, whereas Table 1 reports 36.5% for the initial Qwen3.5-35B-A3B policy and 47.9% after JH-GRPO training, averaged across four harnesses. This comparison does not establish that the paper’s results are incorrect: harness configurations, inference budgets, and evaluation protocols may differ substantially. Nevertheless, the strength of these results makes it particularly important to explain their comparability to the official benchmark report.
Questions for authors

Questions for Authors

  1. Could you clarify the inconsistencies in result reporting noted above? In particular, do the “joint-only” variant in Table 1 and the “No tool constraints†” variant in Table 2 use different configurations or experimental protocols? Please also explain the dagger marker and the meaning of “target,” confirm whether all reported results come from completed experiments, and reconcile the textual descriptions with the figures and tables.

  2. Could you provide training-step curves for each harness’s evaluation performance, training rewards, and advantage statistics? These would help assess whether training produces mutually beneficial transfer across harnesses or whether interference persists. Please also specify the checkpoint(s), rollout sampling procedure, advantage baseline, and gradient computation protocol used in Figure 3, including whether the gradients are computed under a common objective or each method’s own objective, so that the role of Marginal Joint Advantage can be interpreted more clearly.

  3. Could you provide qualitative comparisons of execution traces before and after joint training for the same tasks across different harnesses? Examples showing changes in tool use, failure types, and recovery strategies would help connect the aggregate improvements to concrete behavioral changes.

  4. Could you explain why the initial policy and trained variants achieve such strong DeepSWE results and poor SWE-Pro results relative to existing public reports, and clarify whether the benchmark version, task set, execution budget, verifier, and scoring protocol are directly comparable? Please provide exact Pass@1 results for every evaluated model/checkpoint under each of the four harnesses on all four benchmarks, rather than only harness-averaged scores.

9 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/5)

Thank you for your comprehensive and insightful review of our paper. We appreciate your valuable feedback and suggestions.

[W2, Q1] Could you clarify the inconsistencies in result reporting: (1) whether all reported results come from completed experiments and the meaning of "target", (2) the difference between "JH-GRPO (joint only)" and "No tool constraints" and the meaning of the dagger marker, and (3) the reference baseline for the largest per-harness improvements?

(1) Every reported result in the paper comes from fully completed training and evaluation runs across all 2,699 OpenSWE training tasks and all 1,643 evaluation tasks (Verified: 500, Multilingual: 300, Pro: 730, DeepSWE: 113).

The word "target" in Section 4 was strictly a prose artifact from language polishing: following Section 3 (where JJH(θ)\mathcal{J}_{\text{JH}}(\theta) is described as the objective our estimator "targets"), phrases referring to "target metrics" were inadvertently condensed into "reaches a target macro Pass@1". We have replaced all such phrasing in Section 4 with direct empirical verbs (achieves / attains / yields).

(2) Both JH-GRPO (joint only) and No tool constraints† come from completed experiments; they differ in whether built-in runtime parser/schema coercion is active when tool constraints μv\mu_v are removed (see Section 3.5 and Section 4.1). Specifically:

  • JH-GRPO (joint only): Strict Native Execution. All four dual constraint penalties are disabled (μv≡0\mu_v \equiv 0), with strict native schema validation and no cross-harness runtime coercion. Malformed or foreign-schema calls therefore fail at execution, reducing Macro Pass@1 by 1.3 points (53.7% to 52.4%) and JHR@4 by 1.9 points (35.8% to 33.9%). Failed episodes yield yi=0y_i=0, indirectly discouraging invalid calls and reducing pre-repair foreign-schema violations from 6.9 to 6.2 per 100 calls (Fig. 5a).

  • No tool constraints†: Runtime Coercion Enabled. As clarified in Lines 251–252 and 322–323, runtime parser/schema coercion can repair surface mismatches during execution. This variant disables the explicit token-level penalty (μv≡0\mu_v \equiv 0) but retains built-in coercion (the † footnote was clipped during formatting). Thus, it preserves nearly all task reward (53.3% vs. 53.7% Macro; 35.2% vs. 35.8% JHR@4), while the raw pre-repair foreign-schema violation rate remains 6.9 per 100 calls. Without token-level or execution-failure penalties, the policy receives no effective penalty for emitting foreign schemas. This matches Lines 466–467: "Removing the constraints preserves most of the task reward but restores the foreign-schema error rate, which separates protocol reliability from the coverage utility."

ConfigurationExplicit Penalty μv\mu_vCross-Harness Runtime CoercionVerifiedMulti.ProDeepSWEMacroWorstJHR@4Foreign
Mixed-Harness GRPONoNo64.855.931.842.748.843.724.66.9
JH-GRPO (joint only)NoNo68.559.735.146.452.449.033.96.2
No tool constraints$^\dagger$NoYes (†^\dagger)69.260.736.047.353.349.835.26.9
JH-GRPO (Full)YesNo69.661.036.247.953.750.335.83.7

The comparisons demonstrate two distinct effects:

  • Strict native execution: Comparing JH-GRPO (joint only) with JH-GRPO (Full), tool constraints reduce foreign-schema violations by 40.3% (46.4% relative to Mixed-Harness GRPO), while improving Macro Pass@1 by 1.3 points and JHR@4 by 1.9 points.
  • Runtime coercion enabled: Comparing No tool constraints† with JH-GRPO (Full) isolates the effect of tool constraints on protocol compliance independently of execution failures. We have retained both rows and restored the † footnote in Table 2 to clarify this distinction.

(3) Figure 2(a) reports SWE-bench Verified Pass@1 across the four harnesses (Initial →\to Mixed-Harness GRPO →\to JH-GRPO (Full)):

  • Claude Code: 62.4%→68.8%→72.4%62.4\% \to 68.8\% \to 72.4\% (+10.0+10.0 over Initial; +3.6+3.6 over Mixed)
  • Codex: 54.8%→66.0%→70.1%54.8\% \to 66.0\% \to 70.1\% (+15.3+15.3 over Initial; +4.1+4.1 over Mixed)
  • SWE-Agent: 58.4%→64.6%→69.9%58.4\% \to 64.6\% \to 69.9\% (+11.5+11.5 over Initial; +5.3+5.3 over Mixed)
  • OpenCode: 57.2%→59.8%→66.0%57.2\% \to 59.8\% \to 66.0\% (+8.8+8.8 over Initial; +6.2+6.2 over Mixed)

In Line 369-370 ("the largest gains occur on the initially weaker Codex and OpenCode interfaces"), Codex (the weakest interface under the Initial policy at 54.8%) achieves the largest gain relative to the Initial policy (+15.3 points), whereas OpenCode (59.8%) and SWE-Agent (64.6%) achieve the largest gains relative to Mixed-Harness GRPO (+6.2 and +5.3 points, respectively). We will revise Lines 369-370 to state both reference baselines and exact point improvements explicitly.

AuthorsOpenReview ↗

Rebuttal (Part 2/5)

[W1] Further validation of key design choices in Harness-Conditional Tool Constraints: localizing penalties to offending tool-call spans vs. trajectory-wide penalties, and adaptive dual updates vs. fixed penalty weights.

We evaluate both design choices under otherwise identical settings across all four benchmarks. Specifically,

  • Trajectory-Wide Penalty (Adaptive μv\mu_v): Replaces the localized indicator I[t∈S(c)]\mathbb{I}[t \in \mathcal{S}(c)] in Eq. 14 with a trajectory-wide penalty Aitraj=AiJH−∑c∈τi∑vμvgv(c,hi)A_i^{\text{traj}} = A_i^{\text{JH}} - \sum_{c \in \tau_i}\sum_v \mu_v g_v(c, h_i) applied across all policy-generated tokens in τi\tau_i.

  • Localized Span Penalty + Fixed Weights (μv≡μfixed\mu_v \equiv \mu_{\text{fixed}}): Retains token-span localization I[t∈S(c)]\mathbb{I}[t \in \mathcal{S}(c)] in Eq. 14 but fixes μv≡μfixed∈0.05,0.10,0.25\mu_v \equiv \mu_{\text{fixed}} \in \\{0.05, 0.10, 0.25\\} throughout training instead of dual updates (Eq. 13).

Constraint Variant (Strict Native Execution)VerifiedMulti.ProDeepSWEMacroWorstJHR@4ForeignParse / Name / Arg
JH-GRPO (joint only) (μv=0\mu_v = 0)68.559.735.146.452.449.033.96.22.0 / 1.6 / 2.7
Trajectory-wide penalty (adaptive μv\mu_v)66.958.133.644.650.846.931.44.11.6 / 1.3 / 2.1
Localized span + Fixed μ=0.05\mu = 0.0568.960.235.547.052.949.534.65.31.8 / 1.4 / 2.4
Localized span + Fixed μ=0.10\mu = 0.1069.160.535.847.453.249.835.04.41.6 / 1.2 / 2.0
Localized span + Fixed μ=0.25\mu = 0.2567.458.634.145.151.347.632.53.51.3 / 0.9 / 1.7
JH-GRPO (Full: Localized + Adaptive μv\mu_v)69.661.036.247.953.750.335.83.71.4 / 1.0 / 1.8

As shown in the Table above, the results establish two key observations:

  1. Necessity of Token-Span Localization (+2.9 Macro and +4.4 JHR@4 over Trajectory-Wide Penalty): In long-horizon repository interactions, an agent often commits an early schema or syntax error, receives an error observation, recovers on a subsequent turn, and successfully solves the task (yi=1y_i = 1). Subtracting the tool penalty trajectory-wide depresses the advantage of all valid fault-localization, reasoning, and patch-generation tokens in that successful trajectory, causing Trajectory-wide penalty to underperform even unconstrained JH-GRPO (joint only) (50.8% vs. 52.4% Macro; 31.4% vs. 33.9% JHR@4).
  2. Necessity of Adaptive Dual Weight Updates (Eq. 13): Cross-harness schema interference varies both across violation types v∈parse, name, arg, foreignv \in \\{\text{parse, name, arg, foreign}\\} and across training stages (foreign-schema leakage spikes mid-training as cross-harness policy updates accelerate). A mild fixed penalty (μ=0.05\mu = 0.05) fails to suppress foreign-schema violations (5.3 vs. 3.7), whereas an aggressive fixed penalty (μ=0.25\mu = 0.25) over-penalizes tool calls and discourages multi-step exploration, reducing JHR@4 by 3.3 points (35.8% -> 32.5%). Adaptive dual updates selectively increase μv\mu_v only when C^v>δv\hat{C}_v > \delta_v, achieving both strong protocol compliance (3.7 Foreign) and the highest task performance (53.7% Macro, 35.8% JHR@4).
AuthorsOpenReview ↗

Rebuttal (Part 3/5)

[W1, Q2] Could you provide training-step curves/statistics for each harness’s evaluation performance, training rewards, and advantage statistics, and specify the checkpoint(s), rollout sampling procedure, advantage baseline, and gradient computation protocol used in Figure 3(b)?

(1) Per-Harness Training Dynamics, Reward Statistics, and Normalized MJA Distributions: Due to the space and formatting constraints of the rebuttal, we report the training dynamics of JH-GRPO (Full) across cumulative rollout budget milestones (0%, 25%, 50%, 75%, 100%, matching the horizontal axis of Figure 5b). At each milestone, we record per-harness evaluation Pass@1 (macro-averaged over 4 benchmarks), aggregate Macro and JHR@4, training mean reward E1(z)E_1(z), joint utility Uρ=4(z)U_{\rho=4}(z), and normalized Marginal Joint Advantages ∣AiJH∣|A_i^{\text{JH}}|:

Rollout Budget (%)Claude Code (%)Codex (%)SWE-Agent (%)OpenCode (%)Eval Macro / JHR@4 (%)Train Mean E1(z)E_1(z)Train Utility Uρ=4(z)U_{\rho=4}(z)∣AiJH∣\lvert A_i^{\mathrm{JH}}\rvert (Mean / P90)
0% (Init)47.139.943.640.942.9 / 18.70.4120.2180.610.61 / 1.341.34
25%51.446.248.645.447.9 / 25.00.4760.3040.740.74 / 1.521.52
50%54.150.351.648.051.0 / 30.50.5210.3690.790.79 / 1.581.58
75%55.652.853.449.452.8 / 34.30.5530.4120.810.81 / 1.601.60
100% (Full)56.453.854.350.353.7 / 35.80.5680.4350.820.82 / 1.611.61
  • Consistent gains at the reported milestones: Unlike single-harness training (where improving the active harness degrades off-diagonal harnesses by -1.2 points on average; Fig. 3a), JH-GRPO (Full) improves all four harnesses monotonically at every checkpoint milestone (Claude Code: +9.3, Codex: +13.9, SWE-Agent: +10.7, OpenCode: +9.4 macro points from 0% to 100%).

  • Interpretation of Δi\Delta_i and AiJHA_i^{\text{JH}}: By Eq. 7, Δi=1Kh∂Uρ(z)∂zh\Delta_i = \frac{1}{K_h}\frac{\partial U_\rho(z)}{\partial z_h} depends on the success rates z−hz_{-h} of the other harnesses on the same task. It assigns greater marginal weight when success under the active harness completes higher-order co-success subsets. This is the mechanism by which MJA links trajectory credit to taskwise joint coverage.

(2) Exact Protocol Used for Cross-Harness Gradient Alignment in Figure 3(b):

  • Shared Reference Checkpoint & Rollout Sampling: As stated in Lines 376–402, to isolate how each method's objective and credit-assignment rule aligns cross-harness gradients without confounding parameter divergence across different training trajectories, we evaluate at a shared mid-training reference checkpoint on 256 held-out OpenSWE tasks (plus the Initial base policy πinit\pi_{\text{init}} as a pre-RL reference). For each task qq, we sample a paired cross-harness group of G=8G=8 rollouts (Kh=2K_h=2 rollouts per harness across all H=4H=4 harnesses) under identical random seeds and initial repository snapshots.

  • Objective & Advantage Baseline Protocol: Gradients are computed under each method's own objective and advantage formulation prior to any cross-harness projection or aggregation:

    1. Initial & Mixed: Standard within-harness group-centered GRPO objective, Ah,k=(yh,k−yˉh)/(σh+ϵ)A_{h,k} = (y_{h,k} - \bar{y}_h)/(\sigma_h + \epsilon) (evaluated at πinit\pi_{\text{init}} and the shared mid-training checkpoint, respectively).
    2. GroupDRO: Within-harness GRPO objective weighted by the GroupDRO dual harness weight qhexp⁡(ηdroℓ^h)q_h \exp(\eta_{\text{dro}} \hat{\ell}_h).
    3. Mixed + PCGrad: Evaluated using the unprojected per-harness gradients under the PCGrad checkpoint state (measuring inherent alignment before surgical projection, as specified in Line 407: "before projection or aggregation").
    4. JH-GRPO (joint): Our unclipped on-policy MJA objective (Eq. 8–9), AiJH=sg(Δi(yi−bi)sema+ϵ)A_i^{\text{JH}} = \text{sg}\big(\frac{\Delta_i(y_i - b_i)}{s_{\text{ema}}+\epsilon}\big), where Δi=1Kh∂Uρ(z)∂zh\Delta_i = \frac{1}{K_h}\frac{\partial U_\rho(z)}{\partial z_h} and bi=1G−1∑j≠iyjb_i = \frac{1}{G-1}\sum_{j \neq i} y_j is the cross-harness leave-one-out baseline.
    5. JH-GRPO (full): Our constraint-augmented token-level advantage A~i,t\tilde{A}_{i,t} (Eq. 14) combining AiJHA_i^{\text{JH}} with localized schema penalties.
  • Pairwise Cosine Computation: For each task qq and harness h∈1,2,3,4h \in \\{1,2,3,4\\}, we compute the per-harness policy gradient vector gq,h=∇θLq,h(θ)g_{q,h} = \nabla_\theta \mathcal{L}_{q,h}(\theta) across all optimized transformer parameters and evaluate all (42)=6\binom{4}{2}=6 pairwise cosine similarities cos⁡(gq,h,gq,h′)\cos(g_{q,h}, g_{q,h'}). Under Mixed, within-harness centering zeroes out gradients whenever yh,1=yh,2=1y_{h,1}=y_{h,2}=1 and ignores whether peer harnesses succeeded, yielding negative median alignment (-0.08). Under JH-GRPO, coupling trajectories via Δi(yi−bi)\Delta_i(y_i - b_i) concentrates gradient magnitude on shared co-success directions (+0.12 for joint, +0.14 for full).

Reviewer_znRbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes JH-GRPO (Joint-Harness Group Relative Policy Optimization) to improve the reliability of coding agents across different harnesses. Unlike conventional mixed-harness RL, which optimizes average task success, JH-GRPO aims to maximize taskwise co-success across multiple harnesses. The method introduces three components: (1) task-paired cross-harness sampling under a fixed rollout budget, (2) a Joint Harness Coverage Utility with Marginal Joint Advantage for trajectory-level credit assignment, and (3) harness-conditional tool constraints to reduce invalid tool calls. The authors also introduce Joint Harness Reliability (JHR) to evaluate cross-harness co-success. Experiments across four coding harnesses and four software engineering benchmarks show improvements in both average Pass@1 (48.8% to 53.7%) and JHR@4 (24.6% to 35.8%) over mixed-harness GRPO.

Critical strengths
  1. The paper identifies an important limitation of existing mixed-harness RL: high average performance does not necessarily imply that an agent can solve the same tasks across different harnesses. The proposed JHR metric captures taskwise co-success beyond conventional Pass@1, providing a meaningful perspective on cross-harness reliability.

  2. The Joint Harness Coverage Utility provides a mathematically motivated objective for encouraging higher-order co-success while retaining lower-order learning signals. The proposed training framework preserves native harness interfaces and does not require additional rollouts or a learned centralized critic.

Critical weaknesses

1. Limited novelty and insufficient ablation of Marginal Joint Advantage (MJA).

MJA can be expressed as a task- and harness-dependent reweighting of the standard group-centered binary advantage, where the weighting factor depends on the outcomes of other harnesses. Its distinction from existing difference-reward and counterfactual credit-assignment methods is unclear. The "Broadcast UρU_\rho" ablation mainly demonstrates the expected cancellation of a shared reward under group centering, but does not isolate the benefit of MJA relative to other valid credit-assignment estimators. Comparisons with simpler reward reweighting and valid alternative credit-assignment methods are needed to substantiate the claimed methodological contribution.

2. Reliability of the reported results.

Throughout Section 4, results are described as “targets” rather than measured outcomes (e.g., “reaches a target macro Pass@1 of 53.7%”, “The target paired confidence intervals”, “the full-method target curve”, “dual updates target 1.7% trainer and 3.8% wall-clock overhead”). This makes it unclear whether all reported numbers come from completed experiments. The results are also internally inconsistent. Section 4.1 defines joint-only JH-GRPO as the full method without tool constraints, yet Table 1 (52.4 / 33.9 / 49.0) and the “No tool constraints†” row in Table 2 (53.3 / 35.2 / 49.8) disagree, and the † has no accompanying footnote. The foreign-schema rate of this configuration is 6.2 in Figure 5(a) but 6.9 in Table 2, and the latter coincides exactly with the Mixed-Harness GRPO value. Since the central claims rest entirely on these empirical comparisons, these issues materially weaken confidence in the conclusions.

Questions for authors

1. What is the substantive difference between MJA and direct task-dependent advantage reweighting?

Since Δi=1Kh∂Uρ∂zh\Delta_i = \frac{1}{K_h}\frac{\partial U_\rho}{\partial z_h} is shared by all rollouts within the same task-harness pair, MJA can be interpreted as reweighting a standard leave-one-out advantage. Could the authors clarify what new credit-assignment mechanism this introduces beyond existing difference-reward and counterfactual policy-gradient approaches?

2. Can the authors provide stronger ablations that isolate the contribution of MJA?

In particular, how does MJA compare against (a) direct joint-utility policy gradients with a valid independent baseline, (b) difference-reward estimators without the proposed leave-one-out baseline, and (c) simpler static or moving-average harness-specific advantage reweighting? These comparisons would help determine whether the gains arise from the joint objective itself or from the proposed credit-assignment mechanism.

3. Why are the results for "JH-GRPO (joint only)" and "No tool constraints" inconsistent across the paper?

Section 4.1 suggests these should represent the same configuration, but Table 1 and Table 2 report different Macro Pass@1, JHR@4, and Worst-Harness results. Figure 5(a) and Table 2 also disagree on the foreign-schema violation rate. Were these experiments conducted with different settings or random seeds? If so, please specify the differences and clarify which results should be used to assess the contribution of tool constraints.

6 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/3)

Thank you for your comprehensive and insightful review of our paper. We appreciate your valuable feedback and suggestions.

[W1, Q1] What is the substantive difference between MJA and direct task-dependent advantage reweighting?

While Δi=1Kh∂Uρ(z)∂zh\Delta_i = \frac{1}{K_h}\frac{\partial U_\rho(z)}{\partial z_h} is shared across the KhK_h rollouts within a task-harness pair (q,h)(q,h), viewing MJA as a heuristic task/harness reweighting of a standard leave-one-out advantage overlooks three mathematical properties that arise specifically from the joint design of (Uρ,Δi,bi)(U_\rho, \Delta_i, b_i) as follows:

  1. Exact Finite-Sample Unbiasedness via Multilinear Decomposition: For a general nonlinear utility U(pθ(q))U(p_\theta(q)), evaluating ∂U(z)∂zh\frac{\partial U(z)}{\partial z_h} at empirical pass rates zz yields a biased policy gradient estimator under finite KhK_h because E[∂U(z)∂zh]≠∂U(E[z])∂pθ(q,h)\mathbb{E}\big[\frac{\partial U(z)}{\partial z_h}\big] \neq \frac{\partial U(\mathbb{E}[z])}{\partial p_\theta(q,h)}. Because Uρ(z)U_\rho(z) is constructed from normalized elementary symmetric polynomials Em(z)E_m(z), it is strictly multilinear in (z1,…,zH)(z_1, \dots, z_H) and satisfies the exact identity:

    Uρ(z)=Uρ(zi←0)+yiΔi,where Δi=1Kh∂Uρ(z)∂zh depends exclusively on z−h (other harnesses).U_\rho(z) = U_\rho(z^{i\leftarrow 0}) + y_i \Delta_i, \quad \text{where } \Delta_i = \frac{1}{K_h}\frac{\partial U_\rho(z)}{\partial z_h} \text{ depends exclusively on } z_{-h} \text{ (other harnesses)}.

    By conditional independence of rollouts across harnesses given task qq, Δi\Delta_i is statistically independent of trajectory τi\tau_i. Thus, the fact that Δi\Delta_i is invariant to yh,ky_{h,k} within (q,h)(q,h) is not a heuristic weighting choice, but the exact algebraic condition required for Δi(yi−bi)∇θlog⁡Pθ,hi(τi∣q)\Delta_i(y_i - b_i)\nabla_\theta \log P_{\theta,h_i}(\tau_i|q) to be an unbiased estimator of the nonlinear joint objective gradient ∇θJJH(θ)=∇θEq[Uρ(pθ(q))]\nabla_\theta \mathcal{J}_{\text{JH}}(\theta) = \nabla_\theta \mathbb{E}_q[U_\rho(p_\theta(q))] for any finite Kh≥1K_h \ge 1 (Eq. 9).

  2. Cross-Harness Leave-One-Out Baseline bib_i vs. Within-Harness Centering: In MJA, bi=1G−1∑j≠iyjb_i = \frac{1}{G-1}\sum_{j\neq i} y_j averages across all G−1G-1 other rollouts in the cross-harness group G(q)\mathcal{G}(q), rather than centering within harness hh.

    • Reweighting a within-harness centered advantage Δi(yh,k−yˉh)\Delta_i(y_{h,k} - \bar{y}_h) zeroes out the gradient whenever all Kh=2K_h=2 rollouts in harness hh succeed (yh,1=yh,2=1  ⟹  yh,k−yˉh=0y_{h,1}=y_{h,2}=1 \implies y_{h,k}-\bar{y}_h=0), halting positive updates on solved interfaces and preventing co-success consolidation.
    • Because both Δi\Delta_i (a function of z−hz_{-h}) and bib_i (a function of y−iy_{-i}) are independent of τi\tau_i, their product satisfies Eτi[Δibi∇θlog⁡Pθ,hi(τi∣q)]=0\mathbb{E}_{\tau_i}[\Delta_i b_i \nabla_\theta \log P_{\theta,h_i}(\tau_i|q)] = 0, providing an unbiased control variate that subtracts task difficulty while retaining positive credit when multiple harnesses succeed.
  3. Critic-Free Counterfactual Credit Across Heterogeneous Schemas: Across long-horizon coding trajectories (up to 200 turns and 170k tokens) with incompatible tool schemas Σh\Sigma_h, training a centralized multi-harness critic is intractable. MJA achieves critic-free counterfactual credit assignment across alternative executions of a single shared policy θ\theta directly from paired verifier outcomes.

AuthorsOpenReview ↗

Rebuttal (Part 2/3)

[W1, Q2] Can the authors provide stronger ablations that isolate the contribution of MJA?

We compare MJA against all three requested families of credit-assignment estimators under identical task-paired sampling (G=8,Kh=2G=8, K_h=2), tool constraints, and compute budgets across all four benchmarks. Specifically,

  • (a) Direct Joint-Utility Policy Gradient with a Valid Independent Baseline: Ai=(Uρ(z)−Uˉema)/(sema+ϵ)A_i = (U_\rho(z) - \bar{U}_{\text{ema}})/(s_{\text{ema}}+\epsilon), where Uˉema\bar{U}_{\text{ema}} is an independent moving-average baseline (avoiding within-group cancellation).

  • (b) Difference-Reward Estimators Without the Cross-Harness Leave-One-Out Baseline bib_i:

    • (b1) Uncentered Difference Reward: Ai=(Uρ(z)−Uρ(zi←0))/(sema+ϵ)=Δiyi/(sema+ϵ)A_i = (U_\rho(z) - U_\rho(z^{i\leftarrow 0}))/(s_{\text{ema}}+\epsilon) = \Delta_i y_i / (s_{\text{ema}}+\epsilon).
    • (b2) Within-Harness Centered Difference Reward: Ai=Δi(yh,k−yˉh)/(sema+ϵ)A_i = \Delta_i (y_{h,k} - \bar{y}_h)/(s_{\text{ema}}+\epsilon).
  • (c) Static and Moving-Average Harness-Specific Advantage Reweighting: Replacing Δi\Delta_i with task-agnostic harness weights wh(yi−bi)/(sema+ϵ)w_h(y_i - b_i)/(s_{\text{ema}}+\epsilon):

    • (c1) Static Inverse-Pass-Rate Reweighting: whstatic∝1/p^init(h)w_h^{\text{static}} \propto 1/\hat{p}_{\text{init}}(h).
    • (c2) Moving-Average (EMA) Harness Reweighting: whemaw_h^{\text{ema}} tracking running per-harness failure rates.
Credit-Assignment EstimatorVerifiedMulti.ProDeepSWEMacroWorstJHR@4ForeignRel. Grad. Var.
Broadcast UρU_\rho (within-group centered; Table 2)58.650.227.136.943.239.019.03.9—
Mean utility E1E_1 (uniform Δi=1/G\Delta_i = 1/G; Table 2)65.856.832.543.449.645.525.43.91.02×1.02\times
(a) Direct Uρ(z)U_\rho(z) + Independent EMA Baseline64.655.631.742.548.644.926.83.93.42×3.42\times
(b1) Difference Reward w/o bib_i (Δiyi\Delta_i y_i)66.857.833.544.750.747.230.43.82.15×2.15\times
(b2) Within-Harness Centered Δi(yh,k−yˉh)\Delta_i(y_{h,k} - \bar{y}_h)66.157.333.244.050.246.729.73.81.38×1.38\times
(c1) Static Harness Reweighting whstatic(yi−bi)w_h^{\text{static}}(y_i - b_i)65.756.532.343.149.446.025.53.91.04×1.04\times
(c2) EMA Harness Reweighting whema(yi−bi)w_h^{\text{ema}}(y_i - b_i)66.056.932.643.649.846.526.23.91.07×1.07\times
MJA (Ours): Δi(yi−bi)\Delta_i(y_i - b_i) (JH-GRPO Full)69.661.036.247.953.750.335.83.71.00×1.00\times

As shown in the Table above, the ablation results lead to three key observations:

  1. Direct Joint-Utility Policy Gradient (a) suffers from high variance and cross-harness free-riding (-9.0 JHR@4 vs. MJA): Because Uρ(z)=Uρ(zi←0)+yiΔiU_\rho(z) = U_\rho(z^{i\leftarrow 0}) + y_i\Delta_i, retaining the unconditioned base term Uρ(zi←0)U_\rho(z^{i\leftarrow 0}) inflates gradient variance by 3.42×3.42\times (measured on the 256 held-out diagnostic tasks) and assigns positive advantages to failed trajectories (yi=0y_i=0) whenever peer harnesses succeed (Uρ(zi←0)>UˉemaU_\rho(z^{i\leftarrow 0}) > \bar{U}_{\text{ema}}).

  2. Necessity of the Cross-Harness Leave-One-Out Baseline bib_i (b1, b2): Omitting bib_i in (b1) (Δiyi≥0\Delta_i y_i \ge 0) provides zero gradient on failed rollouts (yi=0y_i=0) rather than negative feedback and doubles gradient variance (2.15×2.15\times), losing 5.4 JHR@4 points. Centering within each harness in (b2) zeroes out updates whenever both rollouts in a harness succeed, losing 6.1 JHR@4 points.

  3. Insufficiency of Task-Agnostic Harness Reweighting (c1, c2): Static and EMA harness weights improve Worst-Harness slightly over uniform Mean utility E1E_1 (45.5% -> 46.0% / 46.5%), but leave JHR@4 nearly unchanged (25.5% / 26.2% vs. 35.8% for MJA) because corpus-level harness weights cannot identify whether solving task qq in harness hh completes a higher-order co-success intersection.

[W2] Reliability of the reported results.

Every reported result in the paper comes from fully completed training and evaluation runs across all 2,699 OpenSWE training tasks and all 1,643 evaluation tasks (Verified: 500, Multilingual: 300, Pro: 730, DeepSWE: 113).

The word "target" in Section 4 was strictly a prose artifact from language polishing: following Section 3 (where JJH(θ)\mathcal{J}_{\text{JH}}(\theta) is described as the objective our estimator "targets", Lines 213 and 258), phrases referring to "target metrics" were inadvertently condensed into "reaches a target macro Pass@1". We have replaced all such phrasing in Section 4 with direct empirical verbs (achieves / attains / yields) in our revision.

AuthorsOpenReview ↗

Rebuttal (Part 3/3)

[W2, Q3] Why are the results for "JH-GRPO (joint only)" and "No tool constraints" inconsistent across the paper?

Both JH-GRPO (joint only) and No tool constraints† come from completed experiments; they differ in whether built-in runtime parser/schema coercion is active when tool constraints μv\mu_v are removed (see Section 3.5 and Section 4.1). Specifically,

  • JH-GRPO (joint only): End-to-End Unconstrained Baseline Under Strict Native Execution. Here, all four dual constraint penalties are disabled (μv≡0\mu_v \equiv 0) and tool calls execute under strict native schema validation without cross-harness runtime coercion. Consequently, malformed or foreign-schema calls fail at execution time, reducing Macro Pass@1 by 1.3 points (53.7% -> 52.4%) and JHR@4 by 1.9 points (35.8% -> 33.9%). Because failed episodes yield yi=0y_i=0, task-level reward exerts indirect negative pressure against invalid calls, slightly lowering pre-repair foreign-schema violations from 6.9 (Mixed-Harness GRPO) to 6.2 (Fig. 5a).

  • No tool constraints†: Decoupled Coverage Diagnostic With Runtime Parser/Schema Repair. As described in Lines 251–252 ("validation occurs before parser correction, schema coercion, or runtime retry") and Lines 322–323 ("using raw calls before repair or retry"), harnesses include runtime parser/schema coercion that can repair surface schema mismatches during execution. In Table 2, No tool constraints† disables the explicit token-level penalty (μv≡0\mu_v \equiv 0) while keeping built-in runtime coercion active (whose footnote for † was accidentally clipped during page formatting). Because runtime coercion prevents foreign tool calls from aborting trajectories, this variant preserves nearly all of the full method's task reward (53.3% vs. 53.7% Macro; 35.2% vs. 35.8% JHR@4), but because the policy receives neither token-level penalties (μv=0\mu_v=0) nor execution-failure penalties for emitting foreign schemas, the raw pre-repair foreign-schema violation rate remains at 6.9 per 100 calls, which is the exact result described in Lines 465–467 ("Removing the constraints preserves most of the task reward but restores the foreign-schema error rate, which separates protocol reliability from the coverage utility").

ConfigurationExplicit Penalty μv\mu_vCross-Harness Runtime CoercionVerifiedMulti.ProDeepSWEMacroWorstJHR@4Foreign
Mixed-Harness GRPONoNo64.855.931.842.748.843.724.66.9
JH-GRPO (joint only)NoNo68.559.735.146.452.449.033.96.2
No tool constraints$^\dagger$NoYes (†^\dagger)69.260.736.047.353.349.835.26.9
JH-GRPO (Full)YesNo69.661.036.247.953.750.335.83.7

As shown in the Table above, the contribution of tool constraints can be evaluated from two perspectives:

  • Under strict native execution, comparing JH-GRPO (joint only) with JH-GRPO (Full) shows that tool constraints reduce foreign-schema violations by 40.3% (46.4% relative to Mixed-Harness GRPO), while improving Macro Pass@1 by +1.3 points and JHR@4 by +1.9 points.
  • With runtime parser/schema coercion enabled, comparing No tool constraints† with JH-GRPO (Full) isolates the contribution of tool constraints to protocol compliance, independently of execution failures. We have included both rows and restored the † footnote in Table 2 to clarify this distinction.
Reviewer_ojoGOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces JH-GRPO, a reinforcement learning framework designed to improve the reliability of coding agents across different harnesses. The authors identify an important limitation of conventional mixed-harness RL: optimizing average task success across harnesses does not necessarily encourage the same tasks to be solved consistently across different interfaces. To address this issue, JH-GRPO introduces task-paired cross-harness group sampling, which distributes a fixed rollout budget across multiple native harnesses for the same task. The method further incorporates a Joint Harness Coverage Utility to reward higher-order co-success, a Marginal Joint Advantage for trajectory-specific credit assignment, and harness-conditional tool constraints to reduce invalid and foreign-schema tool calls. The paper also proposes Joint Harness Reliability as a metric for evaluating taskwise co-success beyond average Pass@1.

The authors evaluate JH-GRPO using Qwen3.5-35B-A3B across four coding harnesses and four software engineering benchmarks. Compared with mixed-harness GRPO under matched compute budgets, JH-GRPO improves macro-averaged Pass@1 from 48.8% to 53.7% and JHR@4 from 24.6% to 35.8%. The method also improves worst-harness performance and generalization to held-out harnesses while substantially reducing foreign-schema violations. Extensive ablation studies and analyses of cross-harness interference, gradient alignment, interface perturbations, and training efficiency provide further evidence supporting the effectiveness of the proposed approach.

Critical strengths
  1. Well-motivated problem formulation and principled algorithmic design.

The paper addresses an important and practical challenge in coding agent RL: improving task-level reliability across heterogeneous harnesses rather than merely optimizing average performance. The proposed JH-GRPO framework is mathematically well motivated, combining task-paired sampling, higher-order joint success utility, trajectory-specific marginal credit assignment, and harness-conditional tool constraints. The theoretical analysis provides a sound justification for the proposed advantage formulation, while the ablation studies offer empirical support for the joint utility design, particularly the combination of higher-order co-success objectives with lower-order learning signals.

  1. Strong experimental results and comprehensive analysis.

The paper presents extensive evaluations using a relatively large model, four native coding harnesses, and four software engineering benchmarks under controlled compute budgets. The improvements in both average Pass@1 and joint reliability are substantial and consistent. Beyond the main results, the authors provide detailed ablations and diagnostic analyses covering cross-harness interference, gradient alignment, held-out harness generalization, schema perturbations, tool-call reliability, and training efficiency. These experiments help substantiate the effectiveness of the proposed method and clarify the contributions of its individual components.

Critical weaknesses

None

Questions for authors

The theoretical analysis establishes gradient consistency for the trajectory-level estimator, while the practical implementation uses a token-normalized clipped surrogate. Given that different harnesses may produce trajectories with substantially different lengths, could the authors clarify whether such length differences affect the relative optimization weights across harnesses, and whether any length-balancing or normalization mechanisms are employed?

1 comment
AuthorsOpenReview ↗

Rebuttal (Part 1/1)

Thank you for your thorough and insightful review. We sincerely appreciate your positive assessment of our work and your recognition of its strengths, particularly the problem formulation, methodological design, and comprehensive experimental evaluation.

[Q1] The theoretical analysis establishes gradient consistency for the trajectory-level estimator, while the practical implementation uses a token-normalized clipped surrogate. Given that different harnesses may produce trajectories with substantially different lengths, could the authors clarify whether such length differences affect the relative optimization weights across harnesses, and whether any length-balancing or normalization mechanisms are employed?

In Eq. 10, the normalization factor N=∑i,tmi,tN = \sum_{i,t} m_{i,t} is the total number of policy tokens across the group, rather than per-trajectory token length Li=∑tmi,tL_i = \sum_t m_{i,t}. We do not apply heuristic per-trajectory length normalization (1/Li1/L_i), as doing so would weight trajectory score functions by 1/Li1/L_i and violate the trajectory-level gradient consistency in Eq. 9. Instead, relative optimization weights across harnesses are preserved and balanced through three built-in mechanisms:

  1. Exact Trajectory-Level Weighting via Shared Group Normalization (1/N1/N): Because non-policy tokens are masked out (mi,t=0m_{i,t}=0, Eq. 10), the trajectory score function is the exact sum of its policy-token score functions: ∇θlog⁡Pθ,hi(τi∣q)=∑tmi,t∇θlog⁡πθ(τi,t∣τi,<t,q)\nabla_\theta \log P_{\theta,h_i}(\tau_i\vert{}q) = \sum_t m_{i,t}\nabla_\theta \log \pi_\theta(\tau_{i,t}\vert{}\tau_{i,<t}, q). In the unclipped on-policy regime (ri,t(θ)=1r_{i,t}(\theta)=1), summing over tt and dividing by the shared group scalar NN strictly preserves the relative trajectory weights AiJH∝Δi(yi−bi)A_i^{\text{JH}} \propto \Delta_i(y_i - b_i) from Eq. 9 while stabilizing overall gradient scale.

  2. Policy-Token Masking and Span-Localized Penalties: Across coding harnesses, total sequence length differences stem primarily from system prompts, observation wrappers, and tool outputs, which are masked out by mi,t=0m_{i,t}=0 so that only policy-generated reasoning and call tokens contribute to NN and gradients. Similarly, schema penalties μv\mu_v apply only to offending call spans S(c)\mathcal{S}(c) (Eq. 14) rather than across all LiL_i tokens of τi\tau_i.

  3. Endogenous Cross-Harness Balancing via Δi\Delta_i: By multilinearity of Uρ(z)U_\rho(z), the marginal credit Δi=1Kh∂Uρ(z)∂zh\Delta_i = \frac{1}{K_h}\frac{\partial U_\rho(z)}{\partial z_h} depends exclusively on the empirical pass rates of the other harnesses z−hz_{-h}. Consequently, if a more verbose harness h′h' improves faster early in training (increasing zh′z_{h'}), it directly raises Δh\Delta_h for the remaining lagging harnesses h≠h′h \neq h' while its own weight Δh′\Delta_{h'} remains bounded by their lower pass rates. Empirically, this raises pairwise cross-harness gradient cosine similarity from -0.08 (Mixed) to +0.14 (Fig. 3b) and consistently improves all four harnesses on SWE-bench Verified (Fig. 2a): 68.8% -> 72.4% on Claude Code, 66.0% -> 70.1% on Codex, 64.6% -> 69.9% on SWE-Agent, and 59.8% -> 66.0% on OpenCode, lifting macro Worst-Harness Pass@1 from 43.7% to 50.3% (Table 1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.