acceptodds
Under review as a conference paper at ICLR 2027

OpenForge-RL: Train Harness-native Agents in Any Environment

Abstract

Modern AI agents rely on elaborate *inference harnesses* such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. OpenForge-RL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on *any harness in any environment* at scale. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForge-Claw reaches 31.7 (pass^3) and 55.9 (pass@3) on ClawEval and 33.7 on QwenClawBench. OpenForge-GUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves *agentic reliability*, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agents, harness, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ghs9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenForge RL, a framework for training harness-native agents through reinforcement learning across diverse environments. The framework combines a lightweight proxy for intercepting model calls and reconstructing training trajectories with a Kubernetes-based orchestrator for managing remote rollout containers. This design enables existing RL frameworks to train agents within their deployment harnesses without reimplementing the underlying agent execution logic.

Critical strengths
  1. The work addresses an important gap between increasingly complex inference harnesses and existing RL training infrastructure. Decoupling harness execution from policy optimization through a proxy-based interface provides a practical way to support heterogeneous agent environments without redesigning the training pipeline.

  2. The framework is evaluated across multiple harnesses and six benchmarks covering tool-use and GUI agents. The consistent improvements from SFT to SFT+RL, together with the cross-harness analysis, provide useful evidence of its practical applicability.

Critical weaknesses
  1. Limited novelty. The framework primarily integrates proxy-based trajectory collection with containerized rollout orchestration, building upon existing systems such as veRL and Orchard. Concurrent work such as Polar [1] also explores harness-native RL. While the broader application coverage is valuable, the paper does not sufficiently establish which technical challenges are uniquely addressed by OpenForge RL beyond existing infrastructure.

  2. Insufficient validation of scalability and system efficiency. Although scalability is a central claim, the experiments primarily emphasize downstream benchmark performance rather than system-level measurements. There is no systematic evaluation of rollout throughput, GPU utilization, communication overhead, or scaling efficiency as the number of concurrent environments increases. Moreover, the performance improvements may partly arise from the curated training data and harness configurations rather than the proposed infrastructure itself. These limitations make it difficult to assess the framework's actual systems contribution.

References

[1] Xu, B., Zhang, H., Zhang, S., et al. (2026). Polar: Agentic RL on Any Harness at Scale. arXiv:2605.24220.

Questions for authors
  1. What are the key technical differences between OpenForge RL and existing systems such as Orchard and Polar? Which components or capabilities cannot be readily supported by these frameworks?

  2. Can the authors provide a scalability analysis, including rollout throughput, GPU utilization, and wall-clock training efficiency under different numbers of concurrent containers? How does the proposed framework compare with existing RL infrastructure under matched computational budgets?

  3. Can the authors further disentangle the contributions of training data, harness choice, and the RL infrastructure? In particular, would comparable improvements be achievable using the same training data and optimization procedure within an existing framework?

Reviewer_FmRHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenForge RL lets an RL codebase such as veRL train agents inside real harnesses (ZeroClaw, OpenClaw, Codex, Kimi-Agent, MolmoWeb). A proxy logs each harness model call as a prompt-response sample and a Kubernetes orchestrator runs each rollout in a remote container. The authors use SFT on distilled traces and GRPO on synthetic tasks, report gains on six benchmarks, and compare harness and SFT versus SFT+RL behavior.

Critical strengths

The framework lets a standard RL codebase train agents inside the real harnesses with a small proxy and an orchestrator, which closes a train-deploy gap that blocks open research. SFT+RL beats SFT on every benchmark, for example ClawEval pass^3 from 21.7 to 31.7 and WebVoyager from 61.5 to 72.3, and shell calls fall from 22.6% to 13.9% under ZeroClaw. These gains use 252 to 900 RL tasks on one 8×B200 node, so open labs can reproduce the recipe.

Critical weaknesses

The paper does not test its main claim that real-harness training beats simplified-harness training. Table 3 does not support the claim that some harnesses are harder to learn, since Codex gains most over the base (+20.3 pass@1) and RL adds +4.0 on ZeroClaw against +4.2 on OpenClaw over SFT. OpenClaw and Codex call tools through SKILL.md bash wrappers that Claude Opus 4.6 wrote, while ReACT and ZeroClaw use native tools. So the harness gap mixes harness design with tool interface. The paper reports no seeds or intervals, and gains such as +1.6 on QwenClawBench and +3.3 on OSWorld-Verified are close to benchmark noise. Baseline numbers come from reports with other harnesses, step limits and judges, and SFT alone beats most of them.

This paper delivers a systems and engineering contribution and adds little research. The proxy that logs harness model calls as prompt-response samples is a small engineering layer. The orchestrator comes from Orchard (Peng et al., 2026). The authors state that Polar (Xu et al., 2026) proposes a similar approach, and they separate their work from it by task breadth and added analysis. The paper adds no new algorithm, objective, credit assignment method or theory. It runs GRPO in veRL without change, discards failed rollouts and leaves credit assignment to future work. The authors state that data synthesis is not a core contribution. The analysis section could carry the research value, but it rests on one backbone, 100 trajectories per checkpoint and no reported seeds. The tool interface confounds the harness difficulty claim, and the error recovery finding stays a hypothesis without a test. The novelty therefore reduces to one claim that training inside the real harness helps, and the paper does not test that claim against a simplified harness baseline.

Questions for authors

Does training on ReACT only and testing on OpenClaw and Codex match your multi-harness gains? This isolates the value of in-harness training. Your error handling discards timeouts and harness crashes as non-policy failures. Your Codex format-robustness metric counts unparsed tool calls as session failures, so how many policy-caused failures does training discard? Which advantage estimator and loss normalization do you use when trajectories have different numbers of calls, and do you train on token IDs from the proxy or on re-tokenized text? How did you remove overlap between the synthetic tasks (seeded from ClawHub, ZClawBench and AgentNet) and ClawEval, QwenClawBench and OSWorld-Verified? What concurrency, step time and cost did you measure? Your runs use 64 rollouts per step but Figure 1 claims thousands of containers. Section 4.1 trains on four harnesses and Table A5 lists three, so which model produced the Table 3 SFT+RL row? How many seeds back Tables 3 and A5 and Figure 4?

Reviewer_1Bf7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents OpenForge-RL, an interface for training agents inside existing inference harnesses and remote environments. A proxy records model calls and reconstructs training samples, while a Kubernetes orchestrator manages containerized rollouts connected to veRL. The authors combine this infrastructure with synthesized tool-use and computer-use tasks, browser tasks, teacher-generated SFT trajectories, and GRPO training. Evaluation covers three tool-use and three GUI benchmarks. Additional experiments examine harness transfer and behavioral differences between SFT and SFT+RL agents.

Critical strengths
  1. The useful systems contribution is a reusable boundary between harness execution and policy training. A trainer normally needs the prompts, generated actions, and rewards for optimization, whereas a stateful harness controls its own nested calls, tools, and context. The proxy captures these model calls and reconstructs training samples without requiring the trainer to reproduce the harness's control flow; remote containers supply the environment lifecycle (Section 3.2, Figure 2). Demonstrations across tool-use, desktop, and browser settings show practical value beyond one task-specific integration. Orchard supplies orchestration, and the acknowledged concurrent Polar approach already addresses a similar proxy interface in software engineering. The demonstrated additional value here is its application across these heterogeneous harness/environment combinations, rather than evidence that prior proxy systems cannot support them.

  2. The within-recipe SFT-to-RL comparisons consistently demonstrate useful learning. ClawEval pass^3 increases from 21.7 to 31.7, while OSWorld-Verified, Online-Mind2Web, and WebVoyager improve from 34.4/57.4/61.5 to 37.7/63.0/72.3 (Tables 1–2). Table 3 and Table A5 further show training and transfer across harnesses. Together, these results substantiate the practical utility of the interface, although they do not isolate an efficiency advantage over other training infrastructure.

Critical weaknesses
  1. The paper explains how it captures model calls, reconstructs samples, and manages rollouts, but does not identify which capabilities required new engineering beyond Orchard or the similar approach acknowledged in Polar. One worked comparison of a difficult integration—such as multimodal observations or nested harness calls—would help establish that boundary. Appendix B already reports hardware and training times; efficiency leadership is not necessary for the systems contribution, and I do not require a comprehensive scaling study. However, the unrestricted “any harness × any environment” wording exceeds the interfaces actually demonstrated.

  2. Appendix D.3 gives the computer-use agent bash and a file editor in addition to GUI actions, while Table 2 mixes published baselines and step budgets without fully resolving comparable tool access. ClawEval tools are exposed directly to ReACT/ZeroClaw but through SKILL.md/bash for OpenClaw/Codex (D.1). These differences weaken attribution of cross-system rankings or harness gaps to model capability or intrinsic learnability. They do not, by themselves, invalidate SFT-to-RL improvements evaluated in the same configuration. Separately, discarding failed trajectories requires a clear distinction between infrastructure errors and policy-induced failures to interpret the learning signal.

Questions for authors

Were the SFT and SFT+RL checkpoints and the untrained Qwen baseline evaluated with identical tools, budgets, and harness modifications within each reported comparison? Please identify any differences and clarify which failures/timeouts are discarded during training, with their frequency. A mismatch affecting the within-recipe gains would weaken my main acceptance evidence; a difference confined to external baselines would chiefly require narrower comparison claims.

Comments on the paper

1 comment

Dear AC, SAC, and reviewers,


As the author-reviewer discussion period is closing soon, we would like to provide a brief summary of the discussions to better facilitate your final assessment.

During the discussion period, we completed all clarifications and experiments requested in each reviewer's main concerns. These mainly include: (1) a tabulated comparison of our work with Orchard and Polar; (2) a comparison of simplified-harness vs. harness-native training, including a new ReACT-only control; (3) rollout error statistics and standard deviations; and (4) GPU utilization and scalability measurements. At the time of this post, we have not received any further questions after our responses. We briefly summarize each reviewer's main concerns/weakness and our responses below.

  • Reviewer 1Bf7: Raised concerns about what is new beyond Orchard and Polar (W1), and whether tool differences and discarded error rollouts affect our results (W2). We added a comparison table to highlight the differences. Polar (a concurrent work under submission to ICLR 2027) also supports harness-native RL, but it only trains a 4B text model on SWE tasks. In contrast, OpenForge-RL trains both text and multimodal models, covering 6 harnesses and 6 benchmarks across tool-use, browser-use, and computer-use. This includes end-to-end multimodal GUI RL, which is non-trivial and which Polar does not study. To our knowledge, we are the first to enable harness-native RL training across both text- and GUI-based environments. We also showed that the additional tools used in Kimi-Agent do not change performance, and that infra-related errors are very rare (≤0.5%). We have not yet received a follow-up response, so we hope our rebuttal has already addressed the main concerns.
  • Reviewer FmRH: The main concern was that the paper does not test whether real-harness training beats simplified-harness training (W1). We showed that training with a simplified harness (ZeroClaw only) improves OpenClaw and Codex only slightly (+3.3 and +4.6), while training with these harnesses directly gives much larger gains (+9.5 and +20.3). Upon follow-up request, we additionally trained with ReACT-only, which gives near zero gain on OpenClaw and Codex. These results indicate the importance of harness-native training.
  • Reviewer Ghs9: Raised concerns about the technical differences from Orchard and Polar (W1), and about the validation of scalability and system efficiency (W2). We provided the comparison table to highlight the differences, similar to our response to reviewer 1Bf7. We also showed that our GPU utilization is on par with or better than prior work, and that our environment service scales with small overhead. We have not yet received a follow-up response, so we hope our rebuttal has already addressed the main concerns.

Overall, we thank all reviewers for their valuable feedback. The new results and clarifications are in the revised manuscript (marked in red).


Kind regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.