- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies harness generalization in RL post-training, or more specifically how RL through a given harness affects transfer to other execution configurations. It proposes Harness Scaling, which varies the harness while keeping the underlying tasks and environments fixed. The authors define five components: system prompt, tool interface, observation, context management, and workflow. From there, they instantiate three variants of each, which results in 243 distinct configurations. They select 18 training configurations with balanced component variants and pairwise combinations.
The experiments use Qwen3-4B-Instruct and Qwen3-30B-A3B-Instruct with PPO and GRPO, evaluated on ClawGym-Bench, JobBench-Easy, and PinchBench. Using the various harnesses during training, the paper reports improvements on held-out harness combinations as well as transfer to Claude Code and OpenClaw (which demonstrates in-domain and out of domain generalization).
The contribution is an empirical training methodology showing that harness variation makes the models more robust to harness variation during deployment.
Critical strengths
-
The paper shows useful practical results, and the evidence goes beyond the constructed harness space which is what one would wish to see. Improvements hold across two model sizes and both PPO and GRPO, and transfer extends to Claude Code and OpenClaw. The results also control that the performance on the base harness does not degrade, which is crucial for practitioners who would consider using this method.
-
The controls are well chosen. They really help to make the empirical contribution stronger than the intuitive premise. The paper compares random selections, marginally balanced selection, random assignment, alternative fixed harnesses, and repeated exposure to the same task and harness. The proposed paradigm consistently increases the performance in ClawGym, and this scales with .
Critical weaknesses
-
It looks like most of the reported Claude Code gain can also be achieved by cherry picking a better single training harness. In the 4B/GRPO experiment, Harness Scaling scores 37.22, compared with 28.57 for the single harness used in the main comparison. However, Table 12 shows that training on another single harness, , already reaches 36.11. Thus most of the apparent 8.65 point improvement can also be obtained simply by choosing a different training harness, without training across multiple harnesses. The authors should discuss this result prominently - why is the quest a composition of harnesses and not the best harness there could be?
-
The proposition that the compact design (18 configurations) vs full (243) is better isn't fully explained. While structured vs random selection is useful, the 18 vs 243 comparison changes the nearest neighbour hamming distance (2 for the former, 1 for the latter). Coverage changes and as a result, within task variation changes as well. Given that updates and rollouts are matched but actual token consumption and execution cost are not reported, these results require further investigation to draw a precise conclusion.
Questions for authors
-
Related to Weakness #1: the paper needs a stronger fixed-harness baseline. Does the advantage over also hold on OpenClaw? This would help assess whether the smaller external-transfer advantage is robust; an exhaustive search over all fixed harnesses is not necessary.
-
Does nearest-neighbor assignment produce distance-1 pairs in the full-space experiment and distance-2 pairs in the structured-18 experiment? If so, please discuss whether this difference could contribute to the advantage of the structured set. Would a 243 configurations run with distance-2 pairs be a feasible new control?
-
A concrete composed prompt and the corresponding execution rules would clarify the implementation. I'm curious how the system-prompt and workflow variants are composed, while keeping non-target components fixed in the system prompt. I can imagine that these interactions between factors are compatible with a factorial design; still, for reproducibility concerns I think this information should be shared with the readers.
2 comments
A Gentle Follow-up on Our Rebuttal
Dear Reviewer,
Thank you again for your thoughtful and constructive feedback. We have carefully addressed the concerns raised in your review and provided additional experiments and clarifications in our rebuttal.
If there are any remaining questions or points that would benefit from further clarification, we would be very happy to continue the discussion and provide additional details. We would also be grateful if you could consider revisiting your initial rating in light of our responses.
Thank you for your time and consideration.
Reply to Q1, Q2, and Q3
We sincerely thank the reviewer for the thoughtful and constructive feedback, particularly for recognizing the practical value of cross-harness transfer and our controlled comparisons.
As the reviewer noted, a well-chosen single harness can achieve competitive performance on Claude Code (36.11 vs. 37.22). The key question is whether such a harness also transfers reliably to other execution configurations.
To examine this, we evaluate the Table 13 single-harness baselines on OpenClaw.
Qwen3-4B + GRPO, three-benchmark average.
The strongest single-harness configuration differs across targets: h1 performs best on Claude Code (36.11), but achieves only 30.41 on OpenClaw, where h3 performs best (33.72). Harness Scaling outperforms the strongest evaluated single-harness baseline on both targets, by 1.11 and 2.35 points, respectively.
These results highlight the value of training across diverse harness configurations: improved cross-harness generalization without relying on the selection of a particular training harness for each deployment target.
We appreciate the reviewer highlighting potential confounding factors in the Structured-18 versus Full-243 comparison. We provide additional analyses of assignment distance, coverage, and training cost.
(1) Assignment Distance. Nearest-neighbor pairs have distance 2 in Structured-18 and 1 in Full-243. We therefore add a distance-2 Full-243 control.
Qwen3-4B + GRPO, ClawGym-Bench, 300 updates. Full-243 includes the white-box evaluation configurations.
With matched assignment distance, Structured-18 retains higher performance on both White-Box and Claude Code evaluations, supporting the effectiveness of structured harness selection.
(2) Coverage Analysis. We compare the mean Hamming distance from each held-out configuration to its nearest training configuration.
Qwen3-4B + GRPO, ClawGym-Bench. All methods are evaluated on the same 18 held-out configurations.
Structured-18 outperforms Random-18 (seed 42) and Marginal-Balanced-18 despite greater nearest-training distance, supporting structured selection beyond marginal balance and proximity.
(3) Training Budget. At 300 updates, Structured-18 uses 1.4× policy tokens and 1.2× GPU hours relative to Single-Harness RL, compared with 1.2× and 1.1× for Full-243. We further extend Full-243 to approximately match Structured-18's cumulative policy tokens.
Qwen3-4B + GRPO, three-benchmark average. Token-matched checkpoints approximately match cumulative policy-token consumption.
With approximately matched policy tokens, Structured-18 leads Full-243 by 3.84 points In-Space and 2.11 on Claude Code.
We clarify how the harness components are composed in our implementation.
System Prompt (S) and Workflow (W) operate through different mechanisms. S determines the static system-message template, while W controls runtime message injections during agent execution. The Tool Interface (T) independently supplies tool guidance and schemas through a designated placeholder in the system-message template. Specifically, W1 requests an explicit plan before execution and then injects a continuation instruction after the plan is generated. These workflow instructions are added as user-role messages and do not modify the system prompt.
For example,
S1T0O0C0W1combines a product-oriented system-message template (S1) with plan-first message injections (W1), while retaining the base tool interface (T0), raw observations (O0), and full-context management (C0). Changing W1 to W0 leaves the S1 system message unchanged, while changing S1 to S0 leaves the workflow mechanism unchanged.We have added a concrete harness composition example in Appendix A.2 (Table 6 and Figure 5), illustrating how the system-message template, tool guidance, and runtime message injections are combined, together with the corresponding component substitution rules.