- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper focuses on the issue of unstable performance of the same code agent under different harnesses. For JH-GRPO, under a fixed rollout budget, it samples the same task through Task-Paired Cross-Harness Group Sampling, and then combines Joint Harness Coverage Utility, Marginal Joint Advantage, and Harness-Conditional Tool Constraints to train a model. The authors evaluated it on four native harnesses and four software engineering benchmarks. This problem is of practical significance, but more experiments are needed to explain why the method is effective, and some results need to be further clarified.
Critical strengths
- The research motivation is clear and has practical significance. When the same model is used with different harnesses, its performance may change significantly, which may affect the reliability of the code agent in actual use. The paper aims to ensure that the same task can be successfully completed under multiple harnesses, directly addressing this issue.
- The logic in the method section is relatively clear. The paper sequentially introduces Task-Paired Cross-Harness Group Sampling, Joint Harness Coverage Utility, Marginal Joint Advantage, and Harness-Conditional Tool Constraints, basically explaining what each component aims to solve and how they cooperate.
Critical weaknesses
- Further validation of key design choices and training dynamics
Table 2 provides useful ablations of the main components. However, it does not separately establish the benefits of localizing penalties to offending tool-call spans and adaptively updating the penalty weights. Comparisons against trajectory-wide penalties and fixed penalty weights, under otherwise matched settings, would help determine whether these specific design choices are necessary, rather than merely showing that adding tool penalties is beneficial.
Similarly, Figure 3 provides interference diagnostics, and Figure 5 reports aggregate learning curves, but the training dynamics remain insufficiently characterized. Tracking per-harness performance, reward statistics, and the distributions of marginal joint advantages throughout training would help explain how the method improves cross-harness co-success and whether it mitigates interference consistently across interfaces. Qualitative comparisons of execution traces before and after joint training—ideally on the same tasks across multiple harnesses—would also help connect the aggregate gains to concrete changes in tool use and task-solving behavior.
- Inconsistencies and ambiguities in result reporting
Several aspects of the result reporting require clarification:
-
Largest per-harness improvements. The discussion on page 7 states that the largest gains occur on Codex and OpenCode. However, relative to Mixed-Harness GRPO, Figure 2(a) shows that the full method improves SWE-Agent and OpenCode the most: approximately 5.3 and 6.2 percentage points, respectively, compared with 4.1 points for Codex. The authors should specify the intended reference baseline or revise the statement.
-
Joint-only versus “No tool constraints.” According to the definition on page 6, the joint-only variant excludes Harness-Conditional Tool Constraints and therefore appears to correspond to the “No tool constraints†” ablation in Table 2. Nevertheless, their Macro/JHR@4 scores differ: 52.4/33.9 in Table 1 versus 53.3/35.2 in Table 2. The corresponding foreign-schema violation rates also differ: 6.2 in Figure 5(a) versus 6.9 in Table 2. If these variants use different configurations or experimental protocols, those differences—and the meaning of the dagger marker—should be explicitly documented; otherwise, the reported values should be reconciled.
- The DeepSWE results require a more detailed explanation and contextualization. The official DeepSWE v1.1 report lists GLM5.2 at 44%, whereas Table 1 reports 36.5% for the initial Qwen3.5-35B-A3B policy and 47.9% after JH-GRPO training, averaged across four harnesses. This comparison does not establish that the paper’s results are incorrect: harness configurations, inference budgets, and evaluation protocols may differ substantially. Nevertheless, the strength of these results makes it particularly important to explain their comparability to the official benchmark report.
Questions for authors
Questions for Authors
-
Could you clarify the inconsistencies in result reporting noted above? In particular, do the “joint-only” variant in Table 1 and the “No tool constraints†” variant in Table 2 use different configurations or experimental protocols? Please also explain the dagger marker and the meaning of “target,” confirm whether all reported results come from completed experiments, and reconcile the textual descriptions with the figures and tables.
-
Could you provide training-step curves for each harness’s evaluation performance, training rewards, and advantage statistics? These would help assess whether training produces mutually beneficial transfer across harnesses or whether interference persists. Please also specify the checkpoint(s), rollout sampling procedure, advantage baseline, and gradient computation protocol used in Figure 3, including whether the gradients are computed under a common objective or each method’s own objective, so that the role of Marginal Joint Advantage can be interpreted more clearly.
-
Could you provide qualitative comparisons of execution traces before and after joint training for the same tasks across different harnesses? Examples showing changes in tool use, failure types, and recovery strategies would help connect the aggregate improvements to concrete behavioral changes.
-
Could you explain why the initial policy and trained variants achieve such strong DeepSWE results and poor SWE-Pro results relative to existing public reports, and clarify whether the benchmark version, task set, execution budget, verifier, and scoring protocol are directly comparable? Please provide exact Pass@1 results for every evaluated model/checkpoint under each of the four harnesses on all four benchmarks, rather than only harness-averaged scores.
9 comments
Rebuttal (Part 2/5)
[W1] Further validation of key design choices in Harness-Conditional Tool Constraints: localizing penalties to offending tool-call spans vs. trajectory-wide penalties, and adaptive dual updates vs. fixed penalty weights.
We evaluate both design choices under otherwise identical settings across all four benchmarks. Specifically,
-
Trajectory-Wide Penalty (Adaptive ): Replaces the localized indicator in Eq. 14 with a trajectory-wide penalty applied across all policy-generated tokens in .
-
Localized Span Penalty + Fixed Weights (): Retains token-span localization in Eq. 14 but fixes throughout training instead of dual updates (Eq. 13).
| Constraint Variant (Strict Native Execution) | Verified | Multi. | Pro | DeepSWE | Macro | Worst | JHR@4 | Foreign | Parse / Name / Arg |
|---|---|---|---|---|---|---|---|---|---|
JH-GRPO (joint only) () | 68.5 | 59.7 | 35.1 | 46.4 | 52.4 | 49.0 | 33.9 | 6.2 | 2.0 / 1.6 / 2.7 |
| Trajectory-wide penalty (adaptive ) | 66.9 | 58.1 | 33.6 | 44.6 | 50.8 | 46.9 | 31.4 | 4.1 | 1.6 / 1.3 / 2.1 |
| Localized span + Fixed | 68.9 | 60.2 | 35.5 | 47.0 | 52.9 | 49.5 | 34.6 | 5.3 | 1.8 / 1.4 / 2.4 |
| Localized span + Fixed | 69.1 | 60.5 | 35.8 | 47.4 | 53.2 | 49.8 | 35.0 | 4.4 | 1.6 / 1.2 / 2.0 |
| Localized span + Fixed | 67.4 | 58.6 | 34.1 | 45.1 | 51.3 | 47.6 | 32.5 | 3.5 | 1.3 / 0.9 / 1.7 |
| JH-GRPO (Full: Localized + Adaptive ) | 69.6 | 61.0 | 36.2 | 47.9 | 53.7 | 50.3 | 35.8 | 3.7 | 1.4 / 1.0 / 1.8 |
As shown in the Table above, the results establish two key observations:
- Necessity of Token-Span Localization (
+2.9Macro and+4.4JHR@4 over Trajectory-Wide Penalty): In long-horizon repository interactions, an agent often commits an early schema or syntax error, receives an error observation, recovers on a subsequent turn, and successfully solves the task (). Subtracting the tool penalty trajectory-wide depresses the advantage of all valid fault-localization, reasoning, and patch-generation tokens in that successful trajectory, causingTrajectory-wide penaltyto underperform even unconstrainedJH-GRPO (joint only)(50.8%vs.52.4%Macro;31.4%vs.33.9%JHR@4). - Necessity of Adaptive Dual Weight Updates (Eq. 13): Cross-harness schema interference varies both across violation types and across training stages (foreign-schema leakage spikes mid-training as cross-harness policy updates accelerate). A mild fixed penalty () fails to suppress foreign-schema violations (
5.3vs.3.7), whereas an aggressive fixed penalty () over-penalizes tool calls and discourages multi-step exploration, reducing JHR@4 by3.3points (35.8% -> 32.5%). Adaptive dual updates selectively increase only when , achieving both strong protocol compliance (3.7Foreign) and the highest task performance (53.7%Macro,35.8%JHR@4).
Rebuttal (Part 3/5)
[W1, Q2] Could you provide training-step curves/statistics for each harness’s evaluation performance, training rewards, and advantage statistics, and specify the checkpoint(s), rollout sampling procedure, advantage baseline, and gradient computation protocol used in Figure 3(b)?
(1) Per-Harness Training Dynamics, Reward Statistics, and Normalized MJA Distributions:
Due to the space and formatting constraints of the rebuttal, we report the training dynamics of JH-GRPO (Full) across cumulative rollout budget milestones (0%, 25%, 50%, 75%, 100%, matching the horizontal axis of Figure 5b). At each milestone, we record per-harness evaluation Pass@1 (macro-averaged over 4 benchmarks), aggregate Macro and JHR@4, training mean reward , joint utility , and normalized Marginal Joint Advantages :
| Rollout Budget (%) | Claude Code (%) | Codex (%) | SWE-Agent (%) | OpenCode (%) | Eval Macro / JHR@4 (%) | Train Mean | Train Utility | (Mean / P90) |
|---|---|---|---|---|---|---|---|---|
| 0% (Init) | 47.1 | 39.9 | 43.6 | 40.9 | 42.9 / 18.7 | 0.412 | 0.218 | / |
| 25% | 51.4 | 46.2 | 48.6 | 45.4 | 47.9 / 25.0 | 0.476 | 0.304 | / |
| 50% | 54.1 | 50.3 | 51.6 | 48.0 | 51.0 / 30.5 | 0.521 | 0.369 | / |
| 75% | 55.6 | 52.8 | 53.4 | 49.4 | 52.8 / 34.3 | 0.553 | 0.412 | / |
| 100% (Full) | 56.4 | 53.8 | 54.3 | 50.3 | 53.7 / 35.8 | 0.568 | 0.435 | / |
-
Consistent gains at the reported milestones: Unlike single-harness training (where improving the active harness degrades off-diagonal harnesses by
-1.2points on average; Fig. 3a),JH-GRPO (Full)improves all four harnesses monotonically at every checkpoint milestone (Claude Code: +9.3,Codex: +13.9,SWE-Agent: +10.7,OpenCode: +9.4macro points from0%to100%). -
Interpretation of and : By Eq. 7, depends on the success rates of the other harnesses on the same task. It assigns greater marginal weight when success under the active harness completes higher-order co-success subsets. This is the mechanism by which MJA links trajectory credit to taskwise joint coverage.
(2) Exact Protocol Used for Cross-Harness Gradient Alignment in Figure 3(b):
-
Shared Reference Checkpoint & Rollout Sampling: As stated in Lines 376–402, to isolate how each method's objective and credit-assignment rule aligns cross-harness gradients without confounding parameter divergence across different training trajectories, we evaluate at a shared mid-training reference checkpoint on 256 held-out OpenSWE tasks (plus the
Initialbase policy as a pre-RL reference). For each task , we sample a paired cross-harness group of rollouts ( rollouts per harness across all harnesses) under identical random seeds and initial repository snapshots. -
Objective & Advantage Baseline Protocol: Gradients are computed under each method's own objective and advantage formulation prior to any cross-harness projection or aggregation:
Initial&Mixed: Standard within-harness group-centered GRPO objective, (evaluated at and the shared mid-training checkpoint, respectively).GroupDRO: Within-harness GRPO objective weighted by the GroupDRO dual harness weight .Mixed + PCGrad: Evaluated using the unprojected per-harness gradients under the PCGrad checkpoint state (measuring inherent alignment before surgical projection, as specified in Line 407: "before projection or aggregation").JH-GRPO (joint): Our unclipped on-policy MJA objective (Eq. 8–9), , where and is the cross-harness leave-one-out baseline.JH-GRPO (full): Our constraint-augmented token-level advantage (Eq. 14) combining with localized schema penalties.
-
Pairwise Cosine Computation: For each task and harness , we compute the per-harness policy gradient vector across all optimized transformer parameters and evaluate all pairwise cosine similarities . Under
Mixed, within-harness centering zeroes out gradients whenever and ignores whether peer harnesses succeeded, yielding negative median alignment (-0.08). UnderJH-GRPO, coupling trajectories via concentrates gradient magnitude on shared co-success directions (+0.12forjoint,+0.14forfull).
Rebuttal (Part 1/5)
Thank you for your comprehensive and insightful review of our paper. We appreciate your valuable feedback and suggestions.
(1) Every reported result in the paper comes from fully completed training and evaluation runs across all 2,699 OpenSWE training tasks and all 1,643 evaluation tasks (
Verified: 500,Multilingual: 300,Pro: 730,DeepSWE: 113).The word
"target"in Section 4 was strictly a prose artifact from language polishing: following Section 3 (where JJH(θ) is described as the objective our estimator "targets"), phrases referring to "target metrics" were inadvertently condensed into "reaches a target macro Pass@1". We have replaced all such phrasing in Section 4 with direct empirical verbs (achieves / attains / yields).(2) Both
JH-GRPO (joint only)andNo tool constraints†come from completed experiments; they differ in whether built-in runtime parser/schema coercion is active when tool constraints μv are removed (see Section 3.5 and Section 4.1). Specifically:JH-GRPO (joint only): Strict Native Execution. All four dual constraint penalties are disabled (μv≡0), with strict native schema validation and no cross-harness runtime coercion. Malformed or foreign-schema calls therefore fail at execution, reducing Macro Pass@1 by1.3points (53.7%to52.4%) and JHR@4 by1.9points (35.8%to33.9%). Failed episodes yield yi=0, indirectly discouraging invalid calls and reducing pre-repair foreign-schema violations from6.9to6.2per 100 calls (Fig. 5a).No tool constraints†: Runtime Coercion Enabled. As clarified in Lines 251–252 and 322–323, runtime parser/schema coercion can repair surface mismatches during execution. This variant disables the explicit token-level penalty (μv≡0) but retains built-in coercion (the†footnote was clipped during formatting). Thus, it preserves nearly all task reward (53.3%vs.53.7%Macro;35.2%vs.35.8%JHR@4), while the raw pre-repair foreign-schema violation rate remains6.9per 100 calls. Without token-level or execution-failure penalties, the policy receives no effective penalty for emitting foreign schemas. This matches Lines 466–467: "Removing the constraints preserves most of the task reward but restores the foreign-schema error rate, which separates protocol reliability from the coverage utility."The comparisons demonstrate two distinct effects:
JH-GRPO (joint only)withJH-GRPO (Full), tool constraints reduce foreign-schema violations by 40.3% (46.4% relative to Mixed-Harness GRPO), while improving Macro Pass@1 by 1.3 points and JHR@4 by 1.9 points.No tool constraints†withJH-GRPO (Full)isolates the effect of tool constraints on protocol compliance independently of execution failures. We have retained both rows and restored the†footnote in Table 2 to clarify this distinction.(3) Figure 2(a) reports SWE-bench Verified Pass@1 across the four harnesses (
Initial→Mixed-Harness GRPO→JH-GRPO (Full)):In Line 369-370 ("the largest gains occur on the initially weaker Codex and OpenCode interfaces"), Codex (the weakest interface under the Initial policy at
54.8%) achieves the largest gain relative to the Initial policy (+15.3points), whereas OpenCode (59.8%) and SWE-Agent (64.6%) achieve the largest gains relative to Mixed-Harness GRPO (+6.2and+5.3points, respectively). We will revise Lines 369-370 to state both reference baselines and exact point improvements explicitly.