acceptodds
Under review as a conference paper at ICLR 2027

Harness Scaling for Generalizable Agentic Reinforcement Learning

Abstract

Agent harnesses have become a central component of modern agentic systems, shaping how models use tools, manage context, and coordinate long-horizon interaction. Realizing the benefits of these harnesses depends on how effectively the underlying model can make use of them, motivating reinforcement learning directly through the harness. However, existing approaches typically train through a single fixed harness, which can lead to specialization and limited transfer across harness configurations. To address this, we introduce Harness Scaling, which extends agentic RL from a single harness to a structured space of diverse harness configurations. Concretely, we construct a factorized harness space, select a compact training subset with balanced factorial coverage, and organize cross-harness training to combine global coverage with controlled variation for the same task. Across multiple benchmarks, model scales, and RL algorithms, Harness Scaling consistently improves generalization to unseen white-box configurations. Moreover, models trained through structured white-box variation transfer effectively to unseen external black-box harnesses, outperforming models trained through a single complex black-box harness. These results establish harness diversity as a meaningful scaling dimension for agentic RL and highlight the importance of how this diversity is organized for cross-harness generalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Harness, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uqduOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies harness generalization in RL post-training, or more specifically how RL through a given harness affects transfer to other execution configurations. It proposes Harness Scaling, which varies the harness while keeping the underlying tasks and environments fixed. The authors define five components: system prompt, tool interface, observation, context management, and workflow. From there, they instantiate three variants of each, which results in 243 distinct configurations. They select 18 training configurations with balanced component variants and pairwise combinations.

The experiments use Qwen3-4B-Instruct and Qwen3-30B-A3B-Instruct with PPO and GRPO, evaluated on ClawGym-Bench, JobBench-Easy, and PinchBench. Using the various harnesses during training, the paper reports improvements on held-out harness combinations as well as transfer to Claude Code and OpenClaw (which demonstrates in-domain and out of domain generalization).

The contribution is an empirical training methodology showing that harness variation makes the models more robust to harness variation during deployment.

Critical strengths
  1. The paper shows useful practical results, and the evidence goes beyond the constructed harness space which is what one would wish to see. Improvements hold across two model sizes and both PPO and GRPO, and transfer extends to Claude Code and OpenClaw. The results also control that the performance on the base harness does not degrade, which is crucial for practitioners who would consider using this method.

  2. The controls are well chosen. They really help to make the empirical contribution stronger than the intuitive premise. The paper compares random selections, marginally balanced selection, random assignment, alternative fixed harnesses, and repeated exposure to the same task and harness. The proposed paradigm consistently increases the performance in ClawGym, and this scales with KK.

Critical weaknesses
  1. It looks like most of the reported Claude Code gain can also be achieved by cherry picking a better single training harness. In the 4B/GRPO experiment, Harness Scaling scores 37.22, compared with 28.57 for the single harness used in the main comparison. However, Table 12 shows that training on another single harness, h1h_1, already reaches 36.11. Thus most of the apparent 8.65 point improvement can also be obtained simply by choosing a different training harness, without training across multiple harnesses. The authors should discuss this result prominently - why is the quest a composition of harnesses and not the best harness there could be?

  2. The proposition that the compact design (18 configurations) vs full (243) is better isn't fully explained. While structured vs random selection is useful, the 18 vs 243 comparison changes the nearest neighbour hamming distance (2 for the former, 1 for the latter). Coverage changes and as a result, within task variation changes as well. Given that updates and rollouts are matched but actual token consumption and execution cost are not reported, these results require further investigation to draw a precise conclusion.

Questions for authors
  1. Related to Weakness #1: the paper needs a stronger fixed-harness baseline. Does the advantage over h1h_1 also hold on OpenClaw? This would help assess whether the smaller external-transfer advantage is robust; an exhaustive search over all fixed harnesses is not necessary.

  2. Does nearest-neighbor assignment produce distance-1 pairs in the full-space experiment and distance-2 pairs in the structured-18 experiment? If so, please discuss whether this difference could contribute to the advantage of the structured set. Would a 243 configurations run with distance-2 pairs be a feasible new control?

  3. A concrete composed prompt and the corresponding execution rules would clarify the implementation. I'm curious how the system-prompt and workflow variants are composed, while keeping non-target components fixed in the system prompt. I can imagine that these interactions between factors are compatible with a factorial design; still, for reproducibility concerns I think this information should be shared with the readers.

2 comments
AuthorsOpenReview ↗

Reply to Q1, Q2, and Q3

We sincerely thank the reviewer for the thoughtful and constructive feedback, particularly for recognizing the practical value of cross-harness transfer and our controlled comparisons.

Q1. Generalization Beyond a Single Harness

As the reviewer noted, a well-chosen single harness can achieve competitive performance on Claude Code (36.11 vs. 37.22). The key question is whether such a harness also transfers reliably to other execution configurations.

To examine this, we evaluate the Table 13 single-harness baselines on OpenClaw.

Training HarnessClaude CodeOpenClaw
Single hbaseh_{\mathrm{base}}28.5732.02
Single h1h_136.1130.41
Single h2h_233.1132.58
Single h3h_332.7233.72
Harness Scaling37.2236.07

Qwen3-4B + GRPO, three-benchmark average.

The strongest single-harness configuration differs across targets: h1h_1 performs best on Claude Code (36.11), but achieves only 30.41 on OpenClaw, where h3h_3 performs best (33.72). Harness Scaling outperforms the strongest evaluated single-harness baseline on both targets, by 1.11 and 2.35 points, respectively.

These results highlight the value of training across diverse harness configurations: improved cross-harness generalization without relying on the selection of a particular training harness for each deployment target.

Q2. Assignment Distance, Coverage, and Training Budget Controls

We appreciate the reviewer highlighting potential confounding factors in the Structured-18 versus Full-243 comparison. We provide additional analyses of assignment distance, coverage, and training cost.

(1) Assignment Distance. Nearest-neighbor pairs have distance 2 in Structured-18 and 1 in Full-243. We therefore add a distance-2 Full-243 control.

Training SetAssignment DistanceWhite-BoxClaude Code
Structured-18252.7845.37
Full-243147.9441.52
Full-243247.2842.11

Qwen3-4B + GRPO, ClawGym-Bench, 300 updates. Full-243 includes the white-box evaluation configurations.

With matched assignment distance, Structured-18 retains higher performance on both White-Box and Claude Code evaluations, supporting the effectiveness of structured harness selection.

(2) Coverage Analysis. We compare the mean Hamming distance from each held-out configuration to its nearest training configuration.

Training SetMean Nearest DistanceIn-Space
Random-18 (seed 42)1.4447.90
Random-18 (seed 68)1.6746.90
Random-18 (seed 98)1.7248.02
Marginal-Balanced-181.5049.53
Structured-181.5652.78

Qwen3-4B + GRPO, ClawGym-Bench. All methods are evaluated on the same 18 held-out configurations.

Structured-18 outperforms Random-18 (seed 42) and Marginal-Balanced-18 despite greater nearest-training distance, supporting structured selection beyond marginal balance and proximity.

(3) Training Budget. At 300 updates, Structured-18 uses 1.4× policy tokens and 1.2× GPU hours relative to Single-Harness RL, compared with 1.2× and 1.1× for Full-243. We further extend Full-243 to approximately match Structured-18's cumulative policy tokens.

Training SetUpdatesIn-SpaceClaude Code
Full-24330035.2133.85
Full-243 (Token-Matched)35037.0235.11
Structured-1830040.8637.22

Qwen3-4B + GRPO, three-benchmark average. Token-matched checkpoints approximately match cumulative policy-token consumption.

With approximately matched policy tokens, Structured-18 leads Full-243 by 3.84 points In-Space and 2.11 on Claude Code.

Q3. Harness Composition and Reproducibility

We clarify how the harness components are composed in our implementation.

System Prompt (S) and Workflow (W) operate through different mechanisms. S determines the static system-message template, while W controls runtime message injections during agent execution. The Tool Interface (T) independently supplies tool guidance and schemas through a designated placeholder in the system-message template. Specifically, W1 requests an explicit plan before execution and then injects a continuation instruction after the plan is generated. These workflow instructions are added as user-role messages and do not modify the system prompt.

For example, S1T0O0C0W1 combines a product-oriented system-message template (S1) with plan-first message injections (W1), while retaining the base tool interface (T0), raw observations (O0), and full-context management (C0). Changing W1 to W0 leaves the S1 system message unchanged, while changing S1 to S0 leaves the workflow mechanism unchanged.

We have added a concrete harness composition example in Appendix A.2 (Table 6 and Figure 5), illustrating how the system-message template, tool guidance, and runtime message injections are combined, together with the corresponding component substitution rules.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback. We have carefully addressed the concerns raised in your review and provided additional experiments and clarifications in our rebuttal.

If there are any remaining questions or points that would benefit from further clarification, we would be very happy to continue the discussion and provide additional details. We would also be grateful if you could consider revisiting your initial rating in light of our responses.

Thank you for your time and consideration.

Reviewer_1TDhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Harness Scaling: instead of running agentic RL through one fixed harness, it factorizes the harness into five components (system prompt, tool interface, observation, context management, workflow) with three realizations each, selects 18 of the 243 configurations with a strength 2 orthogonal array, and assigns each task an anchor plus its nearest neighbors in Hamming distance during training. On Qwen3 4B and Qwen3 30B A3B with GRPO and PPO, the method improves over single harness RL on 18 held out configurations and transfers to Claude Code and OpenClaw better than SFT or RL run directly through OpenClaw.

Critical strengths

(1) The framing is clear and the construction is principled. Turning harness choice into a factorial design problem, inducing a Hamming graph over configurations, and using an orthogonal array with the Rao bound argument for N = 18 is a clean way to replace ad hoc harness choices with something a reader can verify and reuse.

(2) The empirical evidence is broad and consistent. Gains over single harness RL hold across two backbones, two RL algorithms, three benchmarks, the in space held out set, and two external harnesses, and the per seed deltas in Table 13 are stable. The result that a policy trained only on white box variation beats black box SFT and black box RL on an unseen real harness, while losing essentially nothing on the base harness (Table 4), is the most practically useful finding.

(3) The ablations are more careful than typical. The Repeat 2 control separates cross harness variation from repeated task exposure, Random 18 is run with three seeds, Marginal Balanced 18 separates pairwise balance from marginal balance, random assignment isolates the locality aware assignment, and Table 12 checks that the advantage does not depend on the base harness being a weak training harness.

(4) The behavioral analysis offers a plausible account of where the gains come from (state tracking, workflow compliance, error recovery), and the paper is well written with enough detail in Tables 5 to 8 and Appendix C.1 (including how compaction segments enter the loss) to be reproducible.

Critical weaknesses

(1) The framing is clear and the construction is principled. Turning harness choice into a factorial design problem, inducing a Hamming graph over configurations, and using an orthogonal array with the Rao bound argument for N = 18 is a clean way to replace ad hoc harness choices with something a reader can verify and reuse.

(2) The empirical evidence is broad and consistent. Gains over single harness RL hold across two backbones, two RL algorithms, three benchmarks, the in space held out set, and two external harnesses, and the per seed deltas in Table 13 are stable. The result that a policy trained only on white box variation beats black box SFT and black box RL on an unseen real harness, while losing essentially nothing on the base harness (Table 4), is the most practically useful finding.

(3) The ablations are more careful than typical. The Repeat 2 control separates cross harness variation from repeated task exposure, Random 18 is run with three seeds, Marginal Balanced 18 separates pairwise balance from marginal balance, random assignment isolates the locality aware assignment, and Table 12 checks that the advantage does not depend on the base harness being a weak training harness.

(4) The behavioral analysis offers a plausible account of where the gains come from (state tracking, workflow compliance, error recovery), and the paper is well written with enough detail in Tables 5 to 8 and Appendix C.1 (including how compaction segments enter the loss) to be reproducible.

Questions and weaknesses

(1) How far are the held out configurations from the training set? Comparing Tables 6 and 8, 7 of the 18 held out configurations are at Hamming distance 1 from a training configuration (for example S0T2O1C1W2 vs S0T2O1C1W0, S2T1O1C2W0 vs S2T1O0C2W0, S0T1O0C2W2 vs S0T1O0C0W2, S2T2O0C0W1 vs S2T2O0C2W1) and the remaining 11 are at distance 2; none is farther than two components from training, and both harnesses used in Figure 4c/d are at distance 1. The in space test is therefore recombination of 15 realizations the policy has already seen rather than generalization to unseen harness components. Please report the nearest training distance for each held out configuration and stratify results by it, and consider a leave one realization out experiment (train with two realizations per component, test on the third), which would be a much stronger in space test.

(2) The scaling framing is not well supported by Figure 3a/b, and the N ablation is confounded. Performance peaks at N = 18 and drops at N = 243 under a fixed 300 step budget, so the evidence is that structured diversity helps, not that scaling harness count helps. With K = 2 nearest neighbor assignment, the distance of the second harness from the anchor also changes with N: it is 5 for the three harness set (any two configurations with first order coverage differ in every component), 2 within the orthogonal array (no two rows in Table 6 differ in a single component), and 1 in the full space. Set size, amount of local variation, and samples per configuration all move together. A related concern for Random 18: the orthogonal array spreads rows evenly, so held out configurations are on average closer to a training row than for a random set. Please report, for each training set, the mean distance from held out configurations to the nearest training row and check whether this coverage statistic alone predicts the ordering in Table 10. If it does, the advantage comes from coverage rather than pairwise balance.

(3) Variance is reported for one setting only (Table 13). Tables 1, 2, 3, 9, 10, 11 and 12 are point estimates from three evaluation runs with no standard deviation, and benchmark sizes are not stated, so several smaller gaps are hard to judge: PPO on Qwen3 4B gives 47.74 vs 47.82 on PinchBench under Claude Code, and JobBench Easy differences are 1 to 5 points on a scale where the base model scores 7. More importantly, Table 12 shows single harness RL on h1 reaching 36.11 on Claude Code against 37.22 for Harness Scaling, so against a well chosen single training harness the headline out of space advantage shrinks to about one point, while the spread among single harness runs (28.57 to 36.11) is much larger than the gap between the best single harness and Harness Scaling. Please add seed standard deviations at least to Tables 2 and 12, report tasks per benchmark, and discuss whether the black box transfer result is mainly about avoiding a bad training harness.

(4) The black box baselines need more explanation. Black box RL with 300 online updates underperforms black box SFT on a fixed trajectory set at both scales (33.38 vs 35.08 and 36.28 vs 38.08) and barely moves JobBench Easy above the untrained base, which suggests the run may be undertrained or poorly tuned rather than a strong baseline. Please show its training curve and the OpenClaw specific choices (how a native OpenClaw or Claude Code system prompt fits within the 4K or 8K prompt limit, how tool call formats were adapted). Table 9 also omits the black box baselines on OpenClaw, which is their own training harness. Whether Harness Scaling matches black box RL on the very harness it was trained through is the natural complement to Table 2.

(5) Budget matching is defined by updates and rollouts, but the variants change how many tokens each rollout contributes. Variant tool interfaces add distractor tools, variant observations add metadata, and the compaction variants split one rollout into several segments that all enter the policy loss. Harness Scaling likely trains on more tokens and longer contexts per update than single harness RL, and the full 243 setting may differ again. Please report total policy tokens and GPU hours per method, and clarify whether token level mean reduction gives rollouts under compaction variants a larger share of each update.

(6) The realizations were generated by Opus 4.7 from transformation specifications, but it is unclear whether each component has three fixed texts shared by all tasks or is regenerated per task, and whether solvability and difficulty were checked. Several Variant II realizations are simply harder versions of the base (opaque tool names with distractors, metadata heavy observations, aggressive compaction), which invites the reading that Harness Scaling is robustness augmentation against nuisance perturbations, consistent with Table 12 where a single harder harness already closes most of the gap. Relatedly, the behavioral analysis rests entirely on GPT 5.4 scoring with the harness specification visible to the judge and no human agreement check, and the claim in Section 5.2 that rewards are more closely aligned across harnesses is not quantified; a correlation or mean absolute reward difference between Harness A and B would make it concrete.

Questions for authors

Please refer to weaknesses section.

8 comments
AuthorsOpenReview ↗

Reply to Q1 and Q2

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of our structured design, extensive evaluation, and careful ablations. We address each concern below.

Q1. Held-Out Distance and Generalization

As defined in Section 3.1.2, in-space evaluation measures generalization to unseen combinations of previously observed component realizations. We provide two additional analyses to clarify its scope.

(1) Distance-Stratified Generalization. Based on Tables 7 and 9, we compute the minimum Hamming distance from each held-out configuration to the training set:

  • Distance 1 (8 configurations): S0T2O1C1W2, S2T1O1C2W0, S0T1O0C2W2, S2T2O0C0W1, S0T1O2C0W1, S1T0O0C2W1, S0T2O2C2W0, S1T0O1C1W0.
  • Distance 2 (10 configurations): S0T0O0C1W2, S0T0O1C0W1, S1T0O2C2W2, S1T1O0C0W0, S1T2O1C0W2, S1T2O2C1W1, S2T0O2C0W0, S2T1O1C1W2, S2T1O2C2W1, S2T2O0C1W0.

The mean nearest-training distance is 1.56. We further stratify evaluation results by distance.

Distance# ConfigurationsSingleScalingΔ
1837.9254.07+16.15
21036.8651.75+14.89
Overall1837.3352.78+15.45

Qwen3-4B + GRPO, ClawGym-Bench.

Harness Scaling achieves substantial improvements at both distances, showing that its benefits extend beyond configurations immediately adjacent to the training set.

(2) Generalization to Unseen Component Realizations. We further conduct Leave-One-Realization-Out experiments. For each target realization (T1, T2, C1, or C2), we train a separate model excluding that realization entirely, then evaluate on the same six held-out configurations containing it.

Test RealizationSingleFull ScalingLeave-One-Out Scaling
T137.6552.1347.86
T233.8552.3945.11
C137.0451.7947.24
C236.6751.9444.90

Qwen3-4B + GRPO, ClawGym-Bench. Each row evaluates six configurations containing the target realization.

Despite never encountering the target realization during training, Leave-One-Out Scaling outperforms Single-Harness RL by 8.23–11.26 points. These results provide additional evidence of generalization to previously unseen tool and context realizations, beyond recombinations of observed components.

Q2. Harness Scaling and Structured Diversity

Harness Scaling investigates harness diversity as an additional dimension of agentic RL, rather than assuming that performance increases monotonically with the number of configurations. Our focus is on organizing this diversity effectively under finite training budgets.

(1) Coverage Distance Analysis. We compute the mean minimum Hamming distance from the 18 evaluation configurations to each training set.

Training SetMean Nearest DistanceIn-Space
Single-Harness3.3337.33
Three-Harness2.4448.76
Random-18 (seed 42)1.4447.90
Random-18 (seed 68)1.6746.90
Random-18 (seed 98)1.7248.02
Marginal-Balanced-181.5049.53
Structured-181.5652.78
Full-2430.0047.94

Qwen3-4B + GRPO, ClawGym-Bench. All methods are evaluated on the same 18 configurations. Full-243 includes these configurations during training and is therefore not a held-out generalization setting.

Structured-18 outperforms Random-18 (seed 42) and Marginal-Balanced-18 despite having a larger mean nearest-training distance (1.56 vs. 1.44/1.50). Thus, nearest-training proximity alone does not predict the performance ordering, supporting the value of structured configuration selection.

(2) Assignment Distance and Training Exposure. With K = 2 nearest-neighbor assignment, the second harness has Hamming distance 5, 2, and 1 in the Three-Harness, Structured-18, and Full-243 settings, respectively. We additionally evaluate Full-243 with distance-2 assignment, matching Structured-18.

Training SetAssignment DistanceWhite-BoxClaude Code
Structured-18252.7845.37
Full-243147.9441.52
Full-243247.2842.11

Qwen3-4B + GRPO, ClawGym-Bench, 300 optimization updates.

Structured-18 maintains higher performance even with matched assignment distance, further supporting the effectiveness of structured harness selection.

Increasing the number of harnesses under a fixed rollout budget reduces the average training exposure per configuration. To further assess the effectiveness of structured selection, we compare training sets with the same number of harnesses and matched optimization updates and rollouts. Structured-18 consistently outperforms Random-18 and Marginal-Balanced-18 (52.78 vs. 47.90/49.53), demonstrating the benefits of structured selection beyond simply increasing harness count.

AuthorsOpenReview ↗

Reply to Q3

Q3. Evaluation Variance and Strong Single-Harness Baselines

(1) Training-Seed Robustness. We extend the training-seed analysis from Table 14 to Tables 2 and 13, evaluating three independent training seeds for each configuration. ClawGym-Bench, JobBench-Easy, and PinchBench contain 200, 63, and 30 tasks, respectively.

ModelMethodOriginalThree-Seed Mean ± SD
4BBlack-Box SFT35.0836.29 ± 1.08
4BBlack-Box RL33.3832.67 ± 0.61
4BHarness Scaling37.2237.95 ± 0.92
30B-A3BBlack-Box SFT38.0839.02 ± 0.84
30B-A3BBlack-Box RL36.2836.24 ± 0.65
30B-A3BHarness Scaling41.2041.51 ± 0.43

Claude Code, three-benchmark average. Three-seed results report mean ± training-seed SD, with each checkpoint evaluated three times.

Harness Scaling maintains higher average performance than both black-box baselines across three training seeds and both model scales.

(2) Strong Single-Harness Comparison. We further evaluate the single-harness configurations in Table 13 across three independent training seeds.

Training ConfigurationIn-Space (Original → Three Seeds)Claude Code (Original → Three Seeds)
Single hbaseh_{\mathrm{base}}31.06 → 30.78 ± 0.9228.57 → 29.01 ± 0.74
Single h1h_135.79 → 35.99 ± 0.6536.11 → 36.13 ± 0.48
Single h2h_233.52 → 33.34 ± 0.3133.11 → 32.62 ± 0.52
Single h3h_335.15 → 35.04 ± 0.4232.72 → 31.90 ± 0.71
Harness Scaling40.86 → 40.94 ± 0.6837.22 → 37.95 ± 0.92

Qwen3-4B + GRPO, three-benchmark average. Three-seed results report mean ± training-seed SD.

Harness Scaling exceeds the strongest evaluated single-harness baseline by 4.95 points In-Space and 1.82 points on Claude Code in the three-seed averages.

(3) Cross-Harness Robustness. To further assess transfer robustness across different deployment harnesses, we additionally evaluate h1h_1, h2h_2, and h3h_3 on OpenClaw.

Training ConfigurationClaude CodeOpenClaw
Single hbaseh_{\mathrm{base}}28.5732.02
Single h1h_136.1130.41
Single h2h_233.1132.58
Single h3h_332.7233.72
Harness Scaling37.2236.07

Qwen3-4B + GRPO, three-benchmark average. Claude Code results are from Table 13; OpenClaw results for hbaseh_{\mathrm{base}} and Harness Scaling are from Table 10, while h1h_1–h3h_3 are newly evaluated.

The strongest evaluated single harness differs across deployment targets: h1h_1 performs best on Claude Code (36.11), whereas h3h_3 performs best on OpenClaw (33.72). Harness Scaling outperforms these respective baselines by 1.11 and 2.35 points, supporting more consistent cross-harness transfer without target-specific training-harness selection.

AuthorsOpenReview ↗

Reply to Q4

Q4. Black-Box RL Baselines

(1) Training Convergence. As described in Appendix C.1, Black-Box RL and Harness Scaling use the same backbone models, main GRPO hyperparameters, 300 optimization updates, and 128 rollouts per update. To examine potential undertraining, we extend Black-Box RL training to 500 updates for both Qwen3-4B and Qwen3-30B-A3B, evaluating intermediate checkpoints on OpenClaw (the training harness) and Claude Code (the unseen evaluation harness).

For Qwen3-4B, the Claude Code results are:

Training StepClaude Code
024.47
10028.64
20030.19
30033.38
40034.05
50033.47

Qwen3-4B + GRPO, three-benchmark average.

The training curves show substantial improvements on OpenClaw for both model scales, indicating effective native-harness optimization. On Claude Code, gains diminish at later stages. For Qwen3-4B, performance reaches 34.05 at step 400 before declining to 33.47 at step 500, remaining below Black-Box SFT (35.08). Qwen3-30B-A3B also exhibits diminishing transfer gains. Thus, extending training does not eliminate the observed transfer gap within the evaluated budget. We have added the Black-Box RL training curves to Appendix C.1 (Figure 6).

(2) Native-Harness Evaluation. We additionally evaluate the black-box baselines on their native training harness, OpenClaw, complementing the Claude Code results in Table 2.

ModelTraining MethodOpenClawClaude Code
4BBlack-Box SFT35.2635.08
4BBlack-Box RL36.9333.38
4BHarness Scaling36.0737.22
30B-A3BBlack-Box SFT37.1438.08
30B-A3BBlack-Box RL39.2236.28
30B-A3BHarness Scaling41.2041.20

Three-benchmark average. Claude Code results are from Table 2; Harness Scaling's OpenClaw results are from Table 10. Black-box baseline results on OpenClaw are newly evaluated.

On OpenClaw, Black-Box RL outperforms SFT by 1.67 and 2.08 points for 4B and 30B-A3B, respectively, whereas it underperforms SFT on Claude Code by 1.70 and 1.80 points. These results show that successful adaptation to the training harness does not necessarily translate into stronger cross-harness transfer.

Harness Scaling achieves comparable performance to Black-Box RL on OpenClaw at 4B (36.07 vs. 36.93), higher performance at 30B-A3B (41.20 vs. 39.22), and stronger transfer to Claude Code at both scales.

(3) Native Harness Implementation. We use OpenClaw v2026.4.5 and Claude Code v2.1.205, preserving their native system prompts, execution logic, and context management while replacing only the model backend.

Both white-box and black-box settings use the same context-budget mechanism. The 8K/4K prompt-length and 32K/60K response-length configurations in Table 8 correspond to total context budgets of 40K/64K tokens for Qwen3-4B/Qwen3-30B-A3B, respectively. The native OpenClaw system prompt and tool definitions contain 11,353 tokens, fitting within both budgets without truncation. The 8K/4K settings are not independently enforced as truncation limits on the system prompt. We have added this prompt-length clarification to Appendix C.1.

Tool calls use the Qwen-compatible parser with XML-formatted tool-call messages, which are mapped to native harness tool invocations. Tool execution, observations, and context management follow the native harness mechanisms, with consistent model I/O adaptation across checkpoints.

Reviewer_pAutOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Harness Scaling, the idea is instead of training an agent with RL inside one fixed harness, it trains across many variants of the harness. The method beats single-harness RL on held-out harness combinations and on two real harnesses not used in training (Claude Code and OpenClaw).

Critical strengths
  1. I like that the capability transfer to real harnesses not used in training is consistent. It also beats SFT and RL done inside OpenClaw when both are moved to Claude Code. This matters because models are routinely trained in one harness and deployed in another.
  2. The ablations are mostly well controlled. A Repeat-2 control separates harness variation from simply seeing a task twice, random and marginal-balanced 18-sets test the orthogonal design, Harness Scaling still beats all four single-harness baselines, and three training seeds show that the variance is small.
Critical weaknesses

I am not convinced by the "scaling" claim, and the closest related methods are not compared. The scores rise from 1 to 3 to 18 harnesses but fall at 243, even though the 243 set includes the evaluation harnesses. The paper blames the fixed training budget for the drop but never tests a larger budget. The method is essentially domain randomization over harness features, and the prior work on RL across heterogeneous harnesses that the paper cites (ClawGym II, OpenForgeRL) is not compared. The black-box baseline also uses a single real harness, so Table 2 compares diversity against no diversity. I think training on a mix of real harnesses would be the natural baseline.

Questions for authors

Could you break down the in-space gain by component, for example held-out harnesses with and without opaque tools (T2) or self-compaction (C1/C2)? For Claude Code, which is a single harness, this would need a training ablation, for example training without the T2 variants or without C1/C2 and testing again in Claude Code. That would show whether the gain is broad or comes from one or two perturbations.

3 comments
AuthorsOpenReview ↗

Reply to Comments

We sincerely thank the reviewer for the encouraging assessment and constructive suggestions, particularly the recognition of our cross-harness transfer results and careful ablations. We provide detailed responses and additional experimental evidence below.

Q1. Harness Scaling and Training Budget

Our work investigates harness diversity as an additional scaling dimension for agentic RL, with a particular focus on how to select and organize training harnesses effectively. Rather than assuming monotonic improvements with increasing harness count, we study the impact of structured harness diversity under finite training budgets.

To examine whether Full-243 underperforms due to insufficient training, we extend Structured-18 and Full-243 from 300 to 400 and 500 steps.

Training Set300 Steps400 Steps500 Steps
Structured-1852.78 / 45.3753.18 / 46.4353.04 / 46.37
Full-24347.94 / 41.5246.90 / 40.7146.45 / 39.29

Qwen3-4B + GRPO, ClawGym-Bench. White-box / Claude Code. Full-243 includes the white-box evaluation configurations.

Structured-18 maintains stable performance, whereas Full-243 does not recover the performance gap even with additional training up to 500 steps. These results suggest that effective Harness Scaling depends not only on the number of training configurations, but also on how harness diversity is organized.

Q2. Domain Randomization and Related Work

Harness Scaling systematically organizes harness diversity through a factorized harness space, orthogonal selection, and locality-aware task assignment. Rather than simply increasing the number of training harnesses, we investigate how structured selection and task-harness assignment affect generalization.

Our controlled comparisons show that Structured-18 outperforms Random-18 (Table 3) and Marginal-Balanced-18, while locality-aware assignment outperforms random assignment (Table 3). The consistent improvements demonstrate the benefits of structured selection and task-harness assignment beyond simply introducing harness variation.

ClawGym II focuses on scalable black-box RL and mixed-harness training, while OpenForgeRL provides a framework for training agents across diverse real-world harnesses and environments. In contrast, our work focuses on how harness diversity should be systematically constructed and organized to improve cross-harness generalization, through a factorized harness space, balanced pairwise coverage, and locality-aware task assignment. We additionally evaluate mixed real-harness RL in Q3.

Q3. Comparison with Mixed-Harness RL

We evaluate joint RL across two distinct real-world agent harnesses, OpenClaw and Hermes, rather than variants of a single harness. Claude Code is held out for evaluation, with optimization updates and rollout counts matched across methods.

Training MethodTraining HarnessesClaude Code
Black-Box RLOpenClaw33.38
Black-Box RLOpenClaw + Hermes35.14
Harness ScalingStructured-18 White-Box37.22

Qwen3-4B + GRPO, three-benchmark average.

Structured-18 outperforms Mixed-Harness RL on Claude Code (37.22 vs. 35.14) under matched updates and rollouts, demonstrating stronger cross-harness transfer in this comparison.

Q4. Component-Level Generalization Analysis

We examine whether gains are concentrated in specific tool or context variants through two analyses.

(1) In-Space Component Breakdown. We group the 18 held-out configurations by tool (T) and context (C) realizations.

VariantSingleScalingΔ
T040.4953.82+13.33
T137.6552.13+14.48
T233.8552.39+18.54
C038.2854.61+16.33
C137.0451.79+14.75
C236.6751.94+15.27

Qwen3-4B + GRPO, ClawGym-Bench. Each subset contains six configurations. Single denotes RL on the base harness.

Improvements are consistent across all variants, including T0 and C0, suggesting that gains are not confined to opaque tools or context compaction.

(2) Claude Code Training Ablation. We further remove selected variants during training and evaluate transfer to Claude Code.

Training ConfigurationClaude CodeΔ vs. Full
Full Structured-1837.22—
Without T236.07−1.15
Without C135.29−1.93
Without C236.14−1.08
Without C1 & C235.18−2.04

Qwen3-4B + GRPO, three-benchmark average.

The full Structured-18 model outperforms the strongest evaluated single-harness baseline on Claude Code (37.22 vs. 36.11). Even after excluding T2 or both C1 and C2, performance remains at 36.07 and 35.18, respectively, substantially above base Single-Harness RL (28.57). The consistent improvements across tool and context realizations, together with the strong performance under variant-excluded training, suggest broad and robust cross-harness generalization.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback. We have carefully addressed the concerns raised in your review and provided additional experiments and clarifications in our rebuttal.

If there are any remaining questions or points that would benefit from further clarification, we would be very happy to continue the discussion and provide additional details. We would also be grateful if you could consider revisiting your initial rating in light of our responses.

Thank you for your time and consideration.

AuthorsOpenReview ↗

A Gentle Follow-up Before the Discussion Deadline

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback. We sincerely appreciate the time and effort you have devoted to reviewing our work.

As the discussion period draws to a close, we wanted to kindly follow up on our rebuttal. We have carefully considered your comments and worked to address them through additional experiments and clarifications.

If there are any further questions, we would be more than happy to provide additional details. We would also be sincerely grateful if you could consider revisiting your initial rating in light of our responses.

Thank you once again for your time, valuable feedback, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.