acceptodds
Under review as a conference paper at ICLR 2027

ClawGym II: Exploring Black-Box Reinforcement Learning on Agent Harness

Abstract

Agent harnesses have substantially improved performance on long-horizon tasks by coordinating model interactions with complex environments. However, reinforcement learning through such harnesses remains underexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. We first build a sandbox-based execution infrastructure that isolates task environments and native harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture faithful model calls, reconstruct structured training trajectories, and preserve training–inference consistency through black-box token-in-token-out. Crucially, rather than training separate harness-specific policies, we formulate each task–harness pair as a basic training instance and jointly optimize a shared policy from rollouts produced by heterogeneous harnesses. With Qwen3-30A3B jointly trained through OpenClaw and Claude Code, our model improves Pass@1 on ClawGym-Bench by points under OpenClaw and points under Claude Code, while improving PinchBench by and points, respectively. The same framework also delivers consistent gains on more challenging task settings from JobBench and OfficeQA. Overall, our results show that heterogeneous black-box harnesses can be jointly used as training interfaces for a single general agent policy, enabling effective and scalable optimization across diverse execution systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
general agent, agent harness, black-box reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ms2YOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ClawGym II, a unified black-box reinforcement learning framework for training agents through complex execution harnesses. The framework uses sandbox-based execution and a serving proxy to collect model interactions without modifying the internal logic of the harness. It further introduces prefix-tree-based trajectory reconstruction and filtering to recover training trajectories from fragmented model calls, while adopting token-in-token-out to maintain training–inference consistency. To support heterogeneous harnesses, the authors treat each task–harness pair as a training instance and jointly optimize a shared policy using GRPO. Experiments with OpenClaw and Claude Code show substantial improvements on ClawGym-Bench and PinchBench, with additional results on JobBench and OfficeQA.

Critical strengths
  1. The paper addresses a practical and important problem. Training agents through complex, production-grade harnesses is challenging, and enabling black-box RL without modifying the harness implementation is valuable for real-world agent development.

  2. The framework provides a unified training pipeline for heterogeneous harnesses. The proposed architecture integrates sandbox execution, model-call interception, trajectory reconstruction, and policy optimization into a common pipeline. This design is practical and potentially useful for training agents across different execution systems.

  3. The experimental results show substantial performance improvements.

Critical weaknesses
  1. Limited novelty compared to existing black-box agent RL frameworks. Many key components, including sandbox execution, serving proxies, trajectory reconstruction, and training–inference consistency, have already been explored in POLAR, Dressage, and OpenForgeRL. Dressage also supports multi-harness training. Although these works are cited, the paper does not clearly explain what is new or provide direct comparisons.

  2. Insufficient ablation studies. The paper introduces several components, including prefix-tree reconstruction, dead-leaf filtering, over-branching filtering, and auxiliary trajectory exclusion. However, their individual contributions are not evaluated, making it unclear which components are actually necessary or effective.

  3. Excluding auxiliary trajectories may discard useful training signals. The authors exclude subagent and context-compaction trajectories due to credit assignment challenges. However, these interactions are important parts of modern agent harnesses and may provide valuable learning signals. The paper does not report how much data is discarded or whether excluding these trajectories is better than including them.

  4. Limited experimental coverage and reproducibility. Joint training is tested with only two harnesses, while the JobBench and OfficeQA experiments use Claude Code alone. This limits the evidence for generalization across heterogeneous harnesses. In addition, important details such as training hyperparameters, data construction, filtering thresholds, and computational costs are missing.

Questions for authors
  1. Does joint training provide positive transfer between harnesses? Figure 2 mainly shows that joint training does not hurt performance. Can the authors further investigate whether training through one harness improves performance under another, and whether the two harnesses provide complementary learning signals?
7 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address each weakness and the question on cross-harness transfer below.

Response to W1: Contributions and comparison with contemporaneous black-box RL frameworks.

Our work provides a unified training interface that turns native, opaque harness execution into reliable learning signals for long-horizon workspace tasks. Its technical contribution connects model-boundary recording, reconstruction of fragmented interaction histories, exact sampled-token accounting, and task-aligned reward assignment. The same interface routes rollouts through heterogeneous harnesses and jointly optimizes one policy while preserving each harness's native execution logic. Experiments with OpenClaw and Claude Code demonstrate consistent improvements across both evaluated model scales and multiple task formats.

POLAR [1], Dressage [2], and OpenForgeRL [3] are closely related contemporaneous efforts on agentic RL through complex harnesses. We will expand the discussion of their respective designs and make our focus explicit: stable long-horizon training from native execution, together with joint optimization of a shared policy across different harnesses.

Adapting different training frameworks and their environment-handling pipelines requires substantial engineering. Frameworks differ in how they initialize workspaces, represent tools and observations, manage context, and execute verifiers. Comparisons therefore require aligning the task data and these execution conditions while preserving each framework's intended behavior.

Within our available compute, we first added an AgentLoop RL experiment using the same training data and Qwen3-30A3B initialization. This white-box baseline follows the basic ReAct cycle [4] of reasoning, tool execution, and observation feedback. On ClawGym-Bench under OpenClaw, the OpenClaw-trained model achieves 52.51, compared with 40.33 for the AgentLoop-trained model. This result demonstrates the practical benefit of optimizing the policy through its deployment harness:

ModelWhite-box ReAct AgentLoopOpenClaw
Qwen3-30A3B initialization41.6945.11
ClawII-OC-30A3B, OpenClaw black-box RL43.3753.41
WhiteBox-30A3B, ReAct AgentLoop RL49.9040.33

Each long-horizon run uses 64 H20 GPUs, with a step time of approximately 3,600 seconds. Step time increases during training as interaction turns and output tokens grow, and a complete experiment takes approximately 2 weeks. These demands make both framework adaptation and repeated training substantial commitments. We appreciate the reviewer's understanding and will progressively extend comparisons with these contemporaneous frameworks and additional harnesses as resources become available.

Response to W2: Reconstruction and filtering as requirements for stable long-horizon training.

These mechanisms were introduced to fix concrete failures encountered while building the framework. They are required safeguards in our training pipeline: each ensures that the optimizer receives valid interaction histories and appropriately assigned learning signals.

  • Prefix-tree reconstruction handles context changes caused by compaction, regeneration, and separate interaction chains. Simply concatenating successive calls can create a conditioning history that was never used during generation. The tree preserves the actual histories, while shared-token accounting ensures that each retained sampled-token occurrence contributes to the loss once (Section 3.2.1).
  • Dead-leaf filtering removes responses superseded by harness-side regeneration, such as rejected malformed tool calls (Section 3.2.2). Otherwise, an abandoned response can receive the outcome of a different, accepted continuation. This does not remove ordinary model-selected retries: a valid tool call, its error feedback, and the model's subsequent retry remain in the main trajectory.
  • Over-branching filtering discards rollouts whose prefix trees exceed 10 leaves, targeting the abandoned branches produced when repeated failed regeneration stalls a turn. Ordinary sequential retries extend the existing history; repeated tool use alone does not trigger this filter. Main-agent interactions that continue across context compaction remain eligible for training.
  • Auxiliary-trajectory exclusion keeps the training reward aligned with the original task. A subagent works toward a local assignment, while a compaction call summarizes existing history. The terminal reward does not directly assess either local objective. We therefore optimize the main-agent task-solving segments and retain those before and after compaction.

These features make the connection between rollout execution and policy optimization well defined. Leaving the corresponding failures unhandled can produce incorrect conditioning contexts, repeated token contributions, or misleading reward attribution, and can disrupt training itself. They were introduced as practical fixes needed for stable, sustained RL. We will explain these failure cases and their handling more explicitly in the method description.

Response to W3: Subagent credit assignment and the precise scope of compaction exclusion.

Subagents. Local subtask success and overall task success need not agree. A subagent can finish its assignment while the main task fails elsewhere; the main agent can also recover from an unsuccessful subagent and complete the task. Broadcasting the terminal reward to all these calls would mix different learning objectives. Keeping the main-agent task-solving segments aligns the current optimization target with the original task.

Before disabling these tools, our statistics for Qwen3-30A3B showed that the Task* tools associated with subagents accounted for about 5% of all tool calls. This includes subagent creation, status inspection, completion, and related lifecycle operations together. For the formal RL runs, we disabled subagent-related tools in the harness's native tool settings before rollout collection. We are continuing to explore local rewards and finer credit assignment for effectively incorporating subagent learning.

Context compaction. We exclude only the long-context, single-turn summary-generation interaction. Although its input is long, its objective is narrowly defined: summarize an existing history. This differs from the main agent's ongoing planning, tool execution, and use of environment feedback, and the task-completion reward gives limited direct supervision for the quality of that summary.

The main-agent trajectories before and after compaction both participate in training. Our training statistics show an average of 1.7 main-agent segments per task execution from compaction, with 30% of executions not split by compaction. With one summary call per compaction, this corresponds to 0.7 excluded summary calls per execution on average, while the task-solving segments remain trainable.

During framework development, including these summary calls in policy updates was accompanied by rapidly increasing policy entropy and unstable training. The resulting implementation preserves the main task-solving experience across compaction boundaries while excluding this auxiliary summarization objective, supporting stable long-horizon optimization.

Reviewer_qcqpOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to perform reinforcement learning (RL) through opaque agent harnesses (e.g., OpenClaw and Claude Code) and proposes a unified black-box RL framework. It targets two difficulties of black-box training: the model calls exposed by a harness are fragmented, forked, and redundant rather than complete trajectories, and the text-level interface between the harness and the model can break training–inference consistency. The authors show that stable policy optimization is achievable without modifying the harness, by isolating task environments in sandboxes, reconstructing structured training trajectories from captured model calls, and preserving consistency through black-box token-in-token-out.

The main contributions are:

1). Sandbox-based execution infrastructure. Task environments and native harnesses are isolated in temporary sandboxes, enabling large-scale concurrent rollouts.

2). Serving proxy with black-box token-in-token-out. A proxy at the model boundary records every model call issued by the harness. Training uses the token IDs produced by the inference engine directly, avoiding the mismatch introduced by re-tokenization.

3). Prefix-tree trajectory reconstruction. The calls in a rollout are organized into a prefix tree, where each call is attached to the node whose accumulated history is the longest prefix of its input. This also recovers non-model content inserted by the harness, such as tool outputs. Each root-to-leaf path is a candidate multi-turn trajectory; shared prefixes are stored once, and each sampled token contributes to the loss only once. Trajectories unsuitable for training (dead leaves, over-branching tasks, and auxiliary trajectories) are filtered out, and both PPO and GRPO are adapted to the tree structure.

4). Mix-harness training. Each task–harness pair is treated as a basic training instance, and a single policy is jointly optimized on rollouts from heterogeneous harnesses.

Experiments are conducted on Qwen3 models. Trained through OpenClaw and Claude Code on ClawGym-Bench, Qwen3-30A3B improves Pass@1 by 7.40 and 14.81 points, respectively, and training remains stable over 200–400 optimization steps. The authors also compare mix-harness training against single-harness training, and evaluate out-of-distribution generalization on JobBench and OfficeQA.

Critical strengths

The main strengths are:

  1. A more general and principled representation for fragmented black-box traces. The paper turns the unstructured call log of an opaque harness into a rollout-level prefix tree: each call is attached to the node whose accumulated history is the longest prefix of its input, and harness-inserted content such as tool outputs is recovered by comparing a child’s input with its parent’s history and recorded response. Compared with linear merging schemes that join consecutive calls only on an exact match and otherwise start a new sequence (e.g., Agent Lightning v1.0; He et al., 2026, arXiv:2608.17528), the tree can represent forks caused by context compaction and subagents, stores each shared prefix once, and ensures every sampled token contributes to the loss only once, so highly branched rollouts are not over-weighted. Adapting both PPO and GRPO to the tree, together with explicit trajectory filtering indicates that the authors have considered the failure modes of mature harnesses rather than assuming clean, linear trajectories.

  2. A practically motivated mix-harness setting that is shown to be non-degrading. The paper treats each task–harness pair as a training instance and mixes pairs from heterogeneous harnesses within the same batch, with GRPO grouping and normalization performed per pair. Under an equal per-step rollout budget (32 instances × 8 rollouts), joint training matches or slightly outperforms the single-harness runs under both OpenClaw and Claude Code, even though each harness receives roughly half of the instances per step. The framework also makes it straightforward to add new task families, as shown by the extension to JobBench-style and OfficeQA-style tasks.

Critical weaknesses

The main Weaknesses are:

  1. The need for the prefix tree and the three filtering rules is asserted rather than demonstrated. The paper states that mature harnesses may produce multiple leaves through context compaction or subagent scheduling, and introduces three filters (dead leaves, over-branching tasks, auxiliary trajectories). However, no statistics or case studies show how often such branching occurs in the evaluated rollouts. Evaluation uses a 64K context limit with longer trajectories handled by the harness's context management, so compaction is reachable, but its frequency is never reported. If most rollouts are linear, the prefix tree degenerates to a chain and offers no advantage over linear exact-match merging.

  2. Discarding over-branching tasks may bias the training distribution. Tasks where the harness spawns many subagents or branches heavily are plausibly the longer, more complex ones, which are exactly the long-horizon behaviors the paper targets. Moreover, all retained trajectories from a rollout share the same terminal reward, so subagent and auxiliary trajectories receive the same credit as the main one. The effect on data composition and credit assignment is not analyzed.

  3. The mix-harness evidence is limited and does not test the stated motivation. Joint training is motivated as a way to prevent the policy from specializing to a single harness, yet Section 4.2 evaluates only on the two training harnesses, and the conclusion is merely "no evident degradation"; no unseen harness is evaluated. Since GRPO normalization is performed per task–harness pair, mixing amounts to random data mixing within a batch, with no algorithmic component. Results appear to come from single runs without variance. Related work (OpenForgeRL, Sec. 5.2) already reports gains from mixing three harnesses, including on unseen harnesses; the paper should cite it and clarify the difference. Because each harness receives about half of the per-step instances in the joint setting, an equal per-harness-budget comparison would also help interpret the result.

  4. Baselines and evaluation scope. Improvements are reported only against the untrained Qwen3 models. There is no comparison with other ways of training through the same harnesses (e.g., per-call samples or linear merging as in Agent Lightning v1.0), so the contribution of the proposed reconstruction cannot be isolated. The training data and benchmark appear to come from the same ClawGym line of work, so the main gains are largely in-distribution, and no seeds or confidence intervals are reported.

  5. Section 4.3 shows task-format flexibility rather than generalization. The model is trained on synthesized JobBench-style and OfficeQA-style tasks with Claude Code only and evaluated on the corresponding benchmarks, so this is not out-of-distribution evaluation of the policy. JobBench is reported on the Easy split despite the "more challenging tasks" framing, there are no baselines, and the quality of the synthesized tasks and verifiers (and possible reward hacking) is not analyzed.

Questions for authors

Regarding Weakness 1, please answer the following:

  • For each harness (OpenClaw and Claude Code), what is the distribution of the number of leaves per rollout during training, and what fraction of rollouts are purely linear (a single leaf)?
  • How many of the extra leaves stem from genuine context compaction or subagent execution, and how many arise from prefix mismatches caused by the harness rewriting the history? Please include a few concrete case studies of multi-leaf trees.
  • What is the effect on final performance of turning off each filter individually, and of replacing the prefix tree with linear merging of consecutive calls or with per-call samples (reported over multiple seeds)?

Regarding Weakness 2, please answer the following:

  • What fraction of "dead leaves" arises from harness-side rewriting of the history rather than genuine branching, and how does token-in-token-out handle such prefix mismatches?

Regarding Weakness 3-5, please answer the following:

  • Does the jointly trained model improve over the base model and over single-harness models on a harness that was not used in training (e.g., Codex or another open harness)? Please also report the full cross-harness matrix, i.e., how the OpenClaw-only model performs under Claude Code and vice versa, so that the benefit of joint training can be separated from simple cross-harness transfer.
  • How does the method compare, in both in-harness performance and transfer to a held-out harness, with (a) a model trained in a white-box agent loop (e.g., verl AgentLoop), and (b) OpenForgeRL (Yu et al., 2026, arXiv:2607.21557), which also trains through unmodified harnesses and reports mixed-harness results (Sec. 5.2)?
6 comments

We sincerely thank the reviewer for the detailed and constructive feedback. We address the three question groups together with their corresponding weaknesses below.

Response to W1 and Q1: Why native histories require reconstruction, and comparison with a linear agent loop.

Native harnesses actively transform the interaction history: they normalize tool arguments, reserialize assistant messages, regenerate rejected responses, and replace long histories with summaries. Consequently, a later request need not contain the preceding sampled response verbatim. Even when execution proceeds sequentially, concatenating calls in chronological order can therefore create a training context that the model never saw. Our prefix-tree reconstruction and token-in-token-out (TITO) representation preserve the actual sampled tokens and their conditioning histories, as described in Sections 3.2.1 and 3.2.3.

Our Qwen3-30A3B training statistics show that compaction produces 1.7 main-agent segments per task execution on average, with 30% of executions not split by compaction. Thus, history reconstruction is relevant even with the 64K context window used in our experiments. Three concrete mechanisms illustrate the issue:

  • OpenClaw rejects a malformed tool call and regenerates the response from the same history. The rejected attempt becomes a dead leaf, while the accepted response continues the main trajectory.
  • In the Claude Code compaction pattern observed in our interactions, the summarized history is inserted as a separate user message before the original task. The next request starts from this reorganized context, so the main-agent interactions before and after compaction require separate training segments.
  • Claude Code can normalize a Bash command such as cd /workspace && ls to ls when /workspace is already the current directory. The recorded tool-call history then contains the normalized command rather than the original model output. This removal of the redundant cd <cwd> && sequence is also documented in a maintainer-confirmed report [1]. Exact text-prefix matching can consequently create an apparent branch even without a change in the task-solving path.

These cases explain why a sequential execution trace cannot simply be treated as an unchanged, append-only token sequence. Training each request as an independent sample can preserve its context, but turns a long rollout into many overlapping training sequences and repeatedly processes their shared histories during parameter updates. That cost becomes prohibitive for our long-horizon runs. We therefore combine tree-based prefix sharing with TITO: compatible histories share their prefixes, changed histories remain separate, and each retained sampled-token occurrence contributes to the policy loss once.

We also evaluated a controlled linear-history baseline using a basic ReAct AgentLoop [2]. In this white-box setting, we directly maintain the context and append reasoning, tool calls, and observations, making linear trajectory assembly well defined. The following comparison uses the white-box model under AgentLoop and the main experiment's jointly trained black-box model under its native harnesses, with the same Qwen3-30A3B initialization and task data:

Training approachEvaluation harnessInitializationAfter RL
White-box ReAct AgentLoop RLReAct AgentLoop41.6949.90
Joint black-box RLOpenClaw45.1152.51
Joint black-box RLClaude Code37.0651.87

ClawGym-Bench average scores (%). The two joint-training rows use the same checkpoint.

In their respective evaluation harnesses, the jointly trained black-box model reaches 52.51 on OpenClaw and 51.87 on Claude Code, compared with 49.90 for the white-box AgentLoop. Our framework supports these native harnesses with history transformations through one shared model. The reconstruction mechanisms address the execution conditions encountered in those native systems.

Each long-horizon experiment uses 64 H20 GPUs, with a step time of approximately 3,600 seconds. Interaction turns and output tokens increase during training, making later steps longer; a complete run takes approximately 2 weeks. We prioritized the native-harness and controlled AgentLoop experiments within these resources. We appreciate the reviewer's understanding of these demands and will extend the reconstruction and filtering comparisons, including multiple training seeds, as compute becomes available.

Response to W2 and Q2: Filtering scope, auxiliary credit assignment, and history rewriting.

Over-branching refers to a rollout becoming stuck regenerating a turn and accumulating many abandoned sibling responses, for example repeated malformed versions of the same tool call. We discard a rollout when its prefix tree exceeds 10 leaves. This relatively generous threshold is substantially above the 1.7 main-agent segments per execution observed from compaction. It targets stalled regeneration; repeated tool use along a valid continuing history does not by itself create a new leaf for each call.

Our dead-leaf filter removes rejected responses superseded by harness-controlled regeneration, as described in Section 3.2.2. A valid tool call, its error feedback, and a subsequent retry chosen by the model remain trainable. Low task reward or an unsuccessful action alone does not trigger removal, so the policy retains failure feedback and recovery behavior.

Subagent exclusion addresses a different issue: a subagent's local assignment and the original task have different success conditions. A subagent can succeed while the main task fails elsewhere, or fail while the main agent subsequently recovers. Broadcasting the final task reward to these calls would mix different learning objectives. Before disabling these tools, our Qwen3-30A3B statistics showed that subagent-related Task* operations accounted for about 5% of all tool calls, including creation, status inspection, completion, and related lifecycle operations together. In the formal RL runs, we disabled these tools through the harness's native tool settings before collecting rollouts. The retained training trajectories therefore pursue the original task, while finer subagent credit assignment remains a direction we are exploring.

For compaction, we exclude only the long-context, single-turn summary-generation interaction. Its objective is to summarize an existing history, whereas the main agent plans, executes tools, and responds to environment feedback to complete the task. Both main-agent segments before and after compaction remain available for training. Including the summary calls during development was accompanied by rapidly increasing policy entropy and unstable training, motivating this separation of objectives.

Our statistics on OpenClaw show that 15% of apparent dead leaves are attributable to harness-side history rewriting. Section 3.2.3 describes the corresponding tool-call normalization and message reserialization. TITO records the exact input tokens, generated tokens, and rollout log-probabilities at the model-serving boundary. Training uses the sampled response tokens directly, without rebuilding the learning target from decoded and harness-edited text. Reconstruction also preserves the actual context of each later request, including any harness changes. This keeps policy updates aligned with what the model actually sampled.

Response to W3 and Q3: Cross-harness training and comparisons.

The intended benefit is a single model that performs well under multiple deployment harnesses, whose system prompts, tools, and execution rules can differ substantially. We report the additional OpenClaw–Claude Code cross-evaluation together with the initialization and jointly trained checkpoint:

Training harnessClaude Code evaluationOpenClaw evaluation
Qwen3-30A3B initialization37.0645.11
Claude Code only50.1239.17
OpenClaw only43.2453.41
OpenClaw + Claude Code51.8752.51

ClawGym-Bench average scores (%).

Single-harness training leaves substantial gaps when the deployment interface changes. Under OpenClaw, the Claude-Code-trained model scores 39.17, below the initialization's 45.11 and the OpenClaw-trained model's 53.41. Under Claude Code, OpenClaw-only training reaches 43.24, compared with 50.12 for Claude-Code-only training. Joint training maintains strong performance under both harnesses. These results support learning directly from both interfaces to obtain an effective shared policy.

In the comparison in Section 4.2, both joint and single-harness training allocate 32 task–harness instances with 8 rollouts each. Balanced mixing gives each harness approximately 50% fewer direct training instances per step while maintaining comparable native-harness performance. Task–harness grouping keeps GRPO reward comparisons within the same execution conditions; the resulting gradients update one shared policy. This preserves meaningful credit assignment while combining experience from heterogeneous harnesses.

We also compare a basic ReAct AgentLoop with OpenClaw using models trained from the same Qwen3-30A3B initialization and task data. The OpenClaw-only checkpoint in this comparison is ClawII-OC-30A3B:

ModelWhite-box ReAct AgentLoopOpenClaw
Qwen3-30A3B initialization41.6945.11
ClawII-OC-30A3B, OpenClaw black-box RL43.3753.41
WhiteBox-30A3B, ReAct AgentLoop RL49.9040.33

ClawGym-Bench average scores (%).

The OpenClaw-trained model improves on the unseen AgentLoop from 41.69 to 43.37. The white-box model transfers to OpenClaw at 40.33, while training natively through OpenClaw reaches 53.41. The white-box model falls below OpenClaw's initialization score of 45.11, illustrating the difficulty of cross-harness transfer and motivating joint rollout routing. This AgentLoop transfer experiment evaluates the OpenClaw-only checkpoint; extending the evaluation to the joint checkpoint and additional harnesses is our next step.

OpenForgeRL [3] is contemporaneous work, and we will expand the discussion of its mixed-harness and transfer experiments in Section 5.2. It combines SFT and RL in its cross-harness study; our study directly optimizes Qwen3 through native OpenClaw and Claude Code. A comparable reproduction requires aligning initialization, training data, workspace provisioning, tools, context handling, and verification across the two implementations. Framework adaptation and long-horizon training are both substantial commitments: one of our runs occupies 64 H20 GPUs for approximately 2 weeks, with step time increasing as trajectories grow. We will extend these comparisons as resources permit.

Reviewer_ZVXkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework for RL training of LLM agents through unmodified, black-box agent harnesses (OpenClaw and Claude Code). Rollouts run in sandboxes, a proxy at the model endpoint records all model calls, and these calls are rebuilt into training trajectories via a prefix tree and filtered. Each task–harness pair forms its own GRPO group, so one policy is trained jointly across harnesses. Qwen3-8B and Qwen3-30A3B trained this way improve over their base models on ClawGym-Bench, PinchBench, JobBench, and OfficeQA, and joint training matches single-harness training.

Critical strengths
  1. Practical and timely. Training models directly through the production harnesses they are deployed in is an important, underexplored problem.
  2. One policy across harnesses. A single model is trained through two structurally different harnesses (OpenClaw and Claude Code).
  3. Well written and easy to follow.
Critical weaknesses
  1. Experimental rigor.

    • No comparison with other training methods. All baselines are untrained Qwen3 models (Tab. 1); other experiments compare only variants of the framework (Fig. 2) or before vs. after training (Fig. 3). This shows that RL on this data helps, not that the framework beats alternatives. Missing:
      • other harness-RL frameworks the paper cites (POLAR, Dressage, OpenForgeRL);
      • RL in a simple agent loop (e.g., ReAct), to test whether training through the harness matters;
      • SFT on the same data.
    • Reward not rigorous. Rubric-based rewards come from an LLM judge, but the rubrics are not included in the paper, and it is unclear whether they are grounded in human data.
  2. No statistical significance. Results are single runs with no confidence intervals (e.g., Tab. 1, Fig. 2c). PinchBench has 30 tasks, so a 17-point gain is about five tasks; rubric scores rely on one LLM judge (GPT-5.4).

  3. Limited novelty; components not ablated. The algorithm is standard GRPO, and sandboxing, the serving proxy, and token-in-token-out are existing techniques. The remaining choices (prefix tree, the three filters, per-harness grouping) are not ablated, and no statistics show how much data each filter removes. The filters also bias the data, as discarded retries and over-branching rollouts are mostly failures. The over-branching threshold is unspecified.

Minor (presentation): the prefix-tree construction in Fig. 1 is not very informative, and "SP" (likely system prompt) is never defined; Fig. 2a–b (training rewards) could move to the appendix.

Questions for authors
  1. How does the framework compare with other harness-RL frameworks (e.g., POLAR, Dressage, OpenForgeRL) on the same data?
  2. If the model itself decides to retry an action several times within a trajectory, are those retries removed as dead leaves?
  3. What is the intuition behind discarding over-branching rollouts? In stochastic domains with genuinely high branching, would this not discard valid data?
  4. Joint training across harnesses appears intended to provide a benefit, but it only matches single-harness training (Fig. 2). What benefit do you expect from it, and what evidence supports it?
14 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address the weaknesses and questions below, with overlapping concerns grouped under their corresponding review items.

Response to W1 and Q1: Training-method comparisons and reward design.

Comparison with a simple agent loop. To examine the value of training through a native harness, we added a white-box RL baseline using an AgentLoop built from the basic ReAct structure [1]: reasoning, tool execution, and observation feedback. Both training approaches start from Qwen3-30A3B and use the same task data. We evaluate the resulting models under both execution systems:

ModelWhite-box ReAct AgentLoopOpenClaw
Qwen3-30A3B initialization41.6945.11
ClawII-OC-30A3B, OpenClaw black-box RL43.3753.41
WhiteBox-30A3B, ReAct AgentLoop RL49.9040.33

ClawGym-Bench average scores (%).

Under OpenClaw, our black-box RL model exceeds the white-box baseline by 13.08 points (53.41 versus 40.33). The white-box model performs best under its own AgentLoop but falls below initialization on OpenClaw, whereas our model also improves over initialization on AgentLoop (43.37 versus 41.69). These results show the practical value of training through the intended deployment harness.

SFT on the same task data. We additionally report an SFT baseline initialized from Qwen3-30A3B and trained on demonstrations for the same task data. Under OpenClaw, it achieves 56.82 on ClawGym-Bench and 86.00 on PinchBench, compared with 52.51 and 73.22 for our jointly trained RL model. The demonstrations come from MiniMax-M2.5 and GLM-5.1, which have stronger agentic capabilities than the Qwen3-30A3B initialization. SFT therefore receives successful behavior from stronger teachers, whereas our RL model learns directly from its own interactions and task rewards. This additional supervision provides a natural explanation for the stronger SFT results and motivates combining such an initialization with harness-native RL.

Comparison with contemporaneous frameworks. POLAR [2], Dressage [3], and OpenForgeRL [4] are all contemporaneous efforts. We appreciate the importance of comparing these systems on common data. In practice, adapting different training frameworks and their environment pipelines is demanding: workspace initialization, tool interfaces, observation formats, context management, and verifier execution must all be aligned to make the results comparable.

The compute requirement is also substantial. Each long-horizon RL experiment uses 64 H20 GPUs, with a step time of approximately 3,600 seconds. Steps become longer as training progresses because the model produces more interaction turns and output tokens; a complete experiment occupies these GPUs for approximately 2 weeks. Within this budget, we prioritized the same-data AgentLoop comparison to directly address the value of native-harness training. We would very much like to extend the framework comparisons and will do so as resources permit. We appreciate the reviewer's understanding of these engineering and compute constraints.

Reward design. We use the public ClawGym-SynData tasks [5], as specified in Section 4.1.1. They provide task-specific executable checks and rubrics with defined scoring criteria and anchors. ClawGym-Bench contains 156 code-only tasks and 44 hybrid tasks; for hybrid tasks, code checks and rubric judgments receive weights of 0.7 and 0.3. Appendix A supplies the judging prompt, which requires criterion-level scores grounded in the task and output artifacts. The dataset documentation reports human review of 50 sampled training tasks and human review of benchmark instructions, resources, and verifiers [5]. We will make the rubric specifications and their quality-control process more explicit in the revised manuscript.

Response to W2: Interpretation and reliability of average scores.

The values labeled “Pass@1” in our manuscript are average task scores from one execution per task. We average the task-level verifier scores, rather than counting only tasks that pass every check. A task can therefore receive credit for satisfying some of its requirements. For hybrid ClawGym tasks, its score combines code checks and rubric judgments with weights of 0.7 and 0.3; PinchBench uses its supplied task verifiers. Thus, a 17-point increase on 30 tasks reflects higher average task completion and cannot simply be interpreted as five additional all-pass tasks.

The improvement is also consistent across models, harnesses, and task categories in Table 1. For Qwen3-30A3B, PinchBench increases from 55.60 to 73.22 under OpenClaw and from 54.14 to 71.42 under Claude Code. Both model sizes improve across all six ClawGym-Bench categories under both harnesses. Together, these results show a broad improvement in task-solving performance. Code-only tasks and the code component of hybrid tasks also provide deterministic verification alongside rubric-based assessment. As evaluation resources become available, we will supplement these results with avg@5, averaging five independent executions per task, and report the variation across executions.

Response to W3, Q2 and Q3: Contribution, training stability, and the scope of filtering.

Our contribution is a unified training framework that preserves native harness execution, reconstructs valid training histories from opaque model calls, and jointly optimizes a shared policy across heterogeneous harnesses. The reconstruction, filtering, and grouping mechanisms arose from concrete failures encountered during development. They make sustained long-horizon training work correctly: removing them without addressing the underlying failures can corrupt the conditioning context, duplicate token contributions, or assign rewards to unrelated actions.

  • Prefix-tree reconstruction preserves the histories actually used during generation, including separate segments created by compaction. Shared-token accounting ensures that each retained sampled-token occurrence enters the loss once (Section 3.2.1). Task–harness grouping computes relative advantages under comparable execution conditions, preventing differences in harness behavior from being mistaken for differences in action quality (Section 3.3).
  • Our framework removes responses rejected and superseded by harness-controlled regeneration, such as malformed tool calls or invalid arguments, as described in Section 3.2.2. These abandoned attempts never receive the continuation that determines the final reward. Retaining them would assign that reward to an execution path that was not followed. A valid tool call, its error observation, and a retry chosen by the model remain in the training history. The distinction concerns the accepted interaction history, not whether an action succeeds.
  • Over-branching targets rollouts that become stuck regenerating a turn, for example repeatedly producing malformed versions of the same tool call and accumulating abandoned sibling leaves. We discard a rollout when its prefix tree exceeds 10 leaves. This is a leaf-count threshold, not a limit of 10 tool calls or interaction turns: ordinary sequential retries extend the history and do not themselves create a new leaf for every attempt. This safeguard prevents stalled regeneration from dominating the training data.
  • Compaction itself is not a reason to discard a task-solving segment: main-agent histories before and after it remain eligible for training. We exclude only the separate, long-context, single-turn summary-generation interaction, whose summarization objective is not directly assessed by the task-completion reward. Our training statistics show 1.7 main-agent segments per execution from compaction on average, with 30% of executions not split by compaction. The cutoff of 10 leaves is therefore substantially above this typical segment count. For environments with genuinely extensive branching, the cutoff should account for their native branching structure. In our formal training runs, subagent tools were disabled through the harness's native tool settings; assigning rewards to subagent objectives separately remains an important direction for extending this setting.

These rules select coherent task-solving histories with appropriate reward attribution. Low reward or an unsuccessful action alone does not trigger their removal, so normal failure feedback and model-driven recovery remain available for learning. Adding superseded generations would not supply additional valid task-outcome supervision. The retained histories allow the policy to learn stably from the actions actually taken and their consequences. We will clarify these failure cases, the threshold, and the precise filtering scope in the method description.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.