- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OpenForge RL, a framework for training harness-native agents through reinforcement learning across diverse environments. The framework combines a lightweight proxy for intercepting model calls and reconstructing training trajectories with a Kubernetes-based orchestrator for managing remote rollout containers. This design enables existing RL frameworks to train agents within their deployment harnesses without reimplementing the underlying agent execution logic.
Critical strengths
-
The work addresses an important gap between increasingly complex inference harnesses and existing RL training infrastructure. Decoupling harness execution from policy optimization through a proxy-based interface provides a practical way to support heterogeneous agent environments without redesigning the training pipeline.
-
The framework is evaluated across multiple harnesses and six benchmarks covering tool-use and GUI agents. The consistent improvements from SFT to SFT+RL, together with the cross-harness analysis, provide useful evidence of its practical applicability.
Critical weaknesses
-
Limited novelty. The framework primarily integrates proxy-based trajectory collection with containerized rollout orchestration, building upon existing systems such as veRL and Orchard. Concurrent work such as Polar [1] also explores harness-native RL. While the broader application coverage is valuable, the paper does not sufficiently establish which technical challenges are uniquely addressed by OpenForge RL beyond existing infrastructure.
-
Insufficient validation of scalability and system efficiency. Although scalability is a central claim, the experiments primarily emphasize downstream benchmark performance rather than system-level measurements. There is no systematic evaluation of rollout throughput, GPU utilization, communication overhead, or scaling efficiency as the number of concurrent environments increases. Moreover, the performance improvements may partly arise from the curated training data and harness configurations rather than the proposed infrastructure itself. These limitations make it difficult to assess the framework's actual systems contribution.
References
[1] Xu, B., Zhang, H., Zhang, S., et al. (2026). Polar: Agentic RL on Any Harness at Scale. arXiv:2605.24220.
Questions for authors
-
What are the key technical differences between OpenForge RL and existing systems such as Orchard and Polar? Which components or capabilities cannot be readily supported by these frameworks?
-
Can the authors provide a scalability analysis, including rollout throughput, GPU utilization, and wall-clock training efficiency under different numbers of concurrent containers? How does the proposed framework compare with existing RL infrastructure under matched computational budgets?
-
Can the authors further disentangle the contributions of training data, harness choice, and the RL infrastructure? In particular, would comparable improvements be achievable using the same training data and optimization procedure within an existing framework?
Dear AC, SAC, and reviewers,
As the author-reviewer discussion period is closing soon, we would like to provide a brief summary of the discussions to better facilitate your final assessment.
During the discussion period, we completed all clarifications and experiments requested in each reviewer's main concerns. These mainly include: (1) a tabulated comparison of our work with Orchard and Polar; (2) a comparison of simplified-harness vs. harness-native training, including a new ReACT-only control; (3) rollout error statistics and standard deviations; and (4) GPU utilization and scalability measurements. At the time of this post, we have not received any further questions after our responses. We briefly summarize each reviewer's main concerns/weakness and our responses below.
Overall, we thank all reviewers for their valuable feedback. The new results and clarifications are in the revised manuscript (marked in red).
Kind regards,
The authors