- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents ClawGym II, a unified black-box reinforcement learning framework for training agents through complex execution harnesses. The framework uses sandbox-based execution and a serving proxy to collect model interactions without modifying the internal logic of the harness. It further introduces prefix-tree-based trajectory reconstruction and filtering to recover training trajectories from fragmented model calls, while adopting token-in-token-out to maintain training–inference consistency. To support heterogeneous harnesses, the authors treat each task–harness pair as a training instance and jointly optimize a shared policy using GRPO. Experiments with OpenClaw and Claude Code show substantial improvements on ClawGym-Bench and PinchBench, with additional results on JobBench and OfficeQA.
Critical strengths
-
The paper addresses a practical and important problem. Training agents through complex, production-grade harnesses is challenging, and enabling black-box RL without modifying the harness implementation is valuable for real-world agent development.
-
The framework provides a unified training pipeline for heterogeneous harnesses. The proposed architecture integrates sandbox execution, model-call interception, trajectory reconstruction, and policy optimization into a common pipeline. This design is practical and potentially useful for training agents across different execution systems.
-
The experimental results show substantial performance improvements.
Critical weaknesses
-
Limited novelty compared to existing black-box agent RL frameworks. Many key components, including sandbox execution, serving proxies, trajectory reconstruction, and training–inference consistency, have already been explored in POLAR, Dressage, and OpenForgeRL. Dressage also supports multi-harness training. Although these works are cited, the paper does not clearly explain what is new or provide direct comparisons.
-
Insufficient ablation studies. The paper introduces several components, including prefix-tree reconstruction, dead-leaf filtering, over-branching filtering, and auxiliary trajectory exclusion. However, their individual contributions are not evaluated, making it unclear which components are actually necessary or effective.
-
Excluding auxiliary trajectories may discard useful training signals. The authors exclude subagent and context-compaction trajectories due to credit assignment challenges. However, these interactions are important parts of modern agent harnesses and may provide valuable learning signals. The paper does not report how much data is discarded or whether excluding these trajectories is better than including them.
-
Limited experimental coverage and reproducibility. Joint training is tested with only two harnesses, while the JobBench and OfficeQA experiments use Claude Code alone. This limits the evidence for generalization across heterogeneous harnesses. In addition, important details such as training hyperparameters, data construction, filtering thresholds, and computational costs are missing.
Questions for authors
- Does joint training provide positive transfer between harnesses? Figure 2 mainly shows that joint training does not hurt performance. Can the authors further investigate whether training through one harness improves performance under another, and whether the two harnesses provide complementary learning signals?
7 comments
Response to W2: Reconstruction and filtering as requirements for stable long-horizon training.
These mechanisms were introduced to fix concrete failures encountered while building the framework. They are required safeguards in our training pipeline: each ensures that the optimizer receives valid interaction histories and appropriately assigned learning signals.
- Prefix-tree reconstruction handles context changes caused by compaction, regeneration, and separate interaction chains. Simply concatenating successive calls can create a conditioning history that was never used during generation. The tree preserves the actual histories, while shared-token accounting ensures that each retained sampled-token occurrence contributes to the loss once (Section 3.2.1).
- Dead-leaf filtering removes responses superseded by harness-side regeneration, such as rejected malformed tool calls (Section 3.2.2). Otherwise, an abandoned response can receive the outcome of a different, accepted continuation. This does not remove ordinary model-selected retries: a valid tool call, its error feedback, and the model's subsequent retry remain in the main trajectory.
- Over-branching filtering discards rollouts whose prefix trees exceed 10 leaves, targeting the abandoned branches produced when repeated failed regeneration stalls a turn. Ordinary sequential retries extend the existing history; repeated tool use alone does not trigger this filter. Main-agent interactions that continue across context compaction remain eligible for training.
- Auxiliary-trajectory exclusion keeps the training reward aligned with the original task. A subagent works toward a local assignment, while a compaction call summarizes existing history. The terminal reward does not directly assess either local objective. We therefore optimize the main-agent task-solving segments and retain those before and after compaction.
These features make the connection between rollout execution and policy optimization well defined. Leaving the corresponding failures unhandled can produce incorrect conditioning contexts, repeated token contributions, or misleading reward attribution, and can disrupt training itself. They were introduced as practical fixes needed for stable, sustained RL. We will explain these failure cases and their handling more explicitly in the method description.
Response to W3: Subagent credit assignment and the precise scope of compaction exclusion.
Subagents. Local subtask success and overall task success need not agree. A subagent can finish its assignment while the main task fails elsewhere; the main agent can also recover from an unsuccessful subagent and complete the task. Broadcasting the terminal reward to all these calls would mix different learning objectives. Keeping the main-agent task-solving segments aligns the current optimization target with the original task.
Before disabling these tools, our statistics for Qwen3-30A3B showed that the Task* tools associated with subagents accounted for about 5% of all tool calls. This includes subagent creation, status inspection, completion, and related lifecycle operations together. For the formal RL runs, we disabled subagent-related tools in the harness's native tool settings before rollout collection. We are continuing to explore local rewards and finer credit assignment for effectively incorporating subagent learning.
Context compaction. We exclude only the long-context, single-turn summary-generation interaction. Although its input is long, its objective is narrowly defined: summarize an existing history. This differs from the main agent's ongoing planning, tool execution, and use of environment feedback, and the task-completion reward gives limited direct supervision for the quality of that summary.
The main-agent trajectories before and after compaction both participate in training. Our training statistics show an average of 1.7 main-agent segments per task execution from compaction, with 30% of executions not split by compaction. With one summary call per compaction, this corresponds to 0.7 excluded summary calls per execution on average, while the task-solving segments remain trainable.
During framework development, including these summary calls in policy updates was accompanied by rapidly increasing policy entropy and unstable training. The resulting implementation preserves the main task-solving experience across compaction boundaries while excluding this auxiliary summarization objective, supporting stable long-horizon optimization.
We sincerely thank the reviewer for the thoughtful and constructive feedback. We address each weakness and the question on cross-harness transfer below.
Response to W1: Contributions and comparison with contemporaneous black-box RL frameworks.
Our work provides a unified training interface that turns native, opaque harness execution into reliable learning signals for long-horizon workspace tasks. Its technical contribution connects model-boundary recording, reconstruction of fragmented interaction histories, exact sampled-token accounting, and task-aligned reward assignment. The same interface routes rollouts through heterogeneous harnesses and jointly optimizes one policy while preserving each harness's native execution logic. Experiments with OpenClaw and Claude Code demonstrate consistent improvements across both evaluated model scales and multiple task formats.
POLAR [1], Dressage [2], and OpenForgeRL [3] are closely related contemporaneous efforts on agentic RL through complex harnesses. We will expand the discussion of their respective designs and make our focus explicit: stable long-horizon training from native execution, together with joint optimization of a shared policy across different harnesses.
Adapting different training frameworks and their environment-handling pipelines requires substantial engineering. Frameworks differ in how they initialize workspaces, represent tools and observations, manage context, and execute verifiers. Comparisons therefore require aligning the task data and these execution conditions while preserving each framework's intended behavior.
Within our available compute, we first added an AgentLoop RL experiment using the same training data and Qwen3-30A3B initialization. This white-box baseline follows the basic ReAct cycle [4] of reasoning, tool execution, and observation feedback. On ClawGym-Bench under OpenClaw, the OpenClaw-trained model achieves 52.51, compared with 40.33 for the AgentLoop-trained model. This result demonstrates the practical benefit of optimizing the policy through its deployment harness:
Each long-horizon run uses 64 H20 GPUs, with a step time of approximately 3,600 seconds. Step time increases during training as interaction turns and output tokens grow, and a complete experiment takes approximately 2 weeks. These demands make both framework adaptation and repeated training substantial commitments. We appreciate the reviewer's understanding and will progressively extend comparisons with these contemporaneous frameworks and additional harnesses as resources become available.