- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents POLAR, an asynchronous rollout framework for reinforcement learning over long-horizon language agents. Rather than requiring existing agent harnesses to be rewritten into the environment interface of a specific training framework, POLAR treats the harness as a black box and places a provider-compatible proxy at its model API boundary. The proxy forwards model requests and records prompt tokens, behavior-policy sampled tokens, log probabilities, and message structure, then reconstructs these interactions into trainer-ready trajectories. The system separates runtime preparation, harness execution, trajectory reconstruction, evaluation, and trainer callbacks into asynchronous stages, and provides two trajectory construction strategies: per-request and token-faithful prefix merging. The authors train Qwen3.5-4B with GRPO on SWE-Gym across four coding harnesses—Codex, Claude Code, Qwen Code, and Pi—and report absolute pass@1 gains of 22.6, 4.8, 0.6, and 6.2 points on SWE-Bench Verified, respectively. The paper also presents an efficiency ablation of prefix merging versus per-request reconstruction, and an offline SFT data generation case study using Qwen3.5-122B-A10B with the Pi harness.
Critical strengths
- The paper identifies and addresses an important, practical systems bottleneck: complex or closed-source agent harnesses are difficult to port losslessly into the environment abstractions prescribed by training frameworks.
- Using the model API as the unified observation and training boundary is a clean, pragmatic design. It covers harnesses shipped as CLIs, package-managed tools, or closed-source binaries without requiring changes to their internal event loops.
- The system architecture cleanly separates task scheduling, runtime preparation, harness execution, trajectory reconstruction, evaluation, and trainer callbacks, which is well suited to long-tail rollouts and expensive environment startup.
- Prefix merging handles token fidelity carefully: trainable tokens come from the behavior policy's raw sampled output, while canonical interstitial tokens are explicitly masked, avoiding drift from naive transcription and re-tokenization.
Critical weaknesses
- The claim of supporting 'any harness' is qualified by important preconditions: the harness must allow its model endpoint to be redirected, model traffic must use a protocol the proxy can emulate, and the proxy must faithfully mimic authentication, errors, streaming events, and provider-specific semantics. The current experiments cover only coding-style CLI harnesses and do not establish broad generality.
- All main online RL experiments use the same Qwen3.5-4B base model, the same SWE-Gym training set, and the same software-engineering task family. The paper does not validate on browser, OS, research, data-analysis, or true multi-agent settings, so external validity is limited.
- There is no direct end-to-end comparison against Agent Lightning, rLLM, SkyRL-Agent, OpenClaw-RL, or other low-intrusion training approaches. Table 3 is a qualitative judgment based on the authors' reading of code and documentation, which cannot substitute for empirical comparisons of compatibility, throughput, integration cost, and training outcomes.
- The main benchmark results do not report independent random seeds, confidence intervals, or variance. In particular, the 0.6-point gain on Qwen Code likely falls within training or SWE-Bench evaluation noise.
- Key control experiments are missing: e.g., training the same model in a standard agent loop without the native harness, supervised fine-tuning only, per-request reconstruction with proper session-level normalization, or training on one harness and evaluating transfer to another. This makes it hard to attribute the gains to POLAR's system design, GRPO, harness-specific adaptation, or training budget.
- The prefix-merging efficiency ablation is not fully fair. The two strategies produce 1,185 versus 218 trainer updates, which may also change effective batch construction, sequence lengths, optimizer steps, and reward propagation semantics. The 5.39× wall-clock difference therefore cannot be attributed solely to the trajectory reconstruction strategy as a pure systems efficiency gain.
Questions for authors
- Please crisply define the applicability boundary of 'any harness'. Which harnesses cannot be trained via POLAR—e.g., those with fixed remote endpoints, certificate pinning, end-to-end encryption, custom websockets, client-side local sampling, or those that depend on precise streaming timing?
- How many independent training runs and evaluations do the four SWE-Bench Verified results reflect? Can you report random seeds, standard deviations, or confidence intervals? In particular, is the 0.6-point gain on Qwen Code statistically meaningful?
- Please provide a direct comparison with the closest systems, including at least integration effort, amount of harness code that must change, supported protocols, throughput, end-to-end training time, and final performance. Qualitative judgments in Table 3 alone make it hard to confirm a substantive advantage over Agent Lightning or rLLM.
- In the 5.39× speedup experiment between prefix_merging and per_request: are total tokens, trainable tokens, optimizer steps, batch token caps, sequence-length distributions, and gradient update counts held constant? If not, how do you separate pure systems gains from changes in optimization semantics?
- What exactly does 'rollout GPU utilization' in Figure 5b measure—inference engine busy time, sampled device utilization, or actual kernel utilization? Why would the choice of trajectory representation change this metric from 20.4% to 87.7%?
We appreciate the reviewer for the specific review. These are critical comments that help us realized common questions in our approach. Here are our response to each of the concerns.
W1/Q1. Boundary of "any harness". Polar supports all common LLM API types (Anthropic Messages, OpenAI Chat, OpenAI Responses or Google generateContent over HTTP/SSE). By "any harness" it is not to say the proxy translation is fixed for any possible API types in the future, but to strengthen the innovation of using API-capture as the design boundary for modern Agentic RL rollout. To sustain our claim, we ctively maintain the framework's proxy for future API type changes. Harness url redirection is a native feature to existing agent harnesses to support multiple provider, and is assumed in Polar's proxy. Not supported: fixed remote endpoints, certificate pinning and proprietary weights. But these are intentional simplicity since we assume a RL framework must have access to its local policy and is self-served for token capture and weight update. Eleven mainstream harnesses run today (see preset examples) with no harness-side changes. We will add a "Scope and limitations" paragraph and qualify the abstract to "any harness with a redirectable, supported model API".
W2. Narrow example with Qwen 4B and SWE task. We further validated Polar over a large run: Qwen3.8-27B with the Pi harness, trained on the public Skill2Env corpus (7,496 terminal tasks), evaluated on Terminal-Bench 2.1: 49.4% → 58.4% pass@1 over 5 trials (details under Zf4Z W1). Multimodal, Web and Computer-uUe reconstruction statistics are under WgMP W3.
W3/Q3. Empirical comparison with other frameworks. We agree. Below is our additional comparison experiment using the same SWE tasks, Qwen3.5-4B with GRPO on 8 B200, each over Claude Code and Codex harnesses.
"No" means no Anthropic/Responses endpoint, or the loop must be re-implemented inside the framework with non-trivial effort. We do not claim better policies where a harness fits these systems; the claim is that more harnesses become trainable with no per-harness integration.
W4/Q2. Variance. We agree. Table 1 is one training run and one pass over 500 instances. We add evaluation intervals and repeated evaluations across 4 independent runs.
Qwen Code gain is within noise. The correct interpretation is that Polar RL did not degrade a model on the harness it was already tuned for. We will add this note to the main experiment section.
W5. Controls. We answer by argument and will add (a) and (b) as experiments in the final version. On attribution: the policy gain is from GRPO; Polar's claim is that GRPO becomes applicable to all agent harnesses, so that policy learns to solve problems respecting harness behaviors during training. (a)/(b) Table 1 already shows capability does not carry across harnesses: the same base model scores 34.2% on Pi and Qwen Code but 3.8% on Codex, whose Responses protocol and patch-style tools it has not seen. Training on a generic loop cannot be expected to close that gap. (c) SFT needs successful trajectories on the target harness, which only a rollout system such as Polar can collect (Sec. 3.2); it complements RL but its stage during training (usualy mid-training) is orthogonal. (d) Session-level normalisation removes per-request's sample-weighting bias but keeps its redundant prefix computation (see Zf4Z Q1).
W6/Q4. Ablation fairness. 1,185 and 218 are trainer samples, not gradient updates, and both arms take 3 optimiser steps. We kept the key variants constant: model, hardware, rollout workload (192 sessions), per-GPU token cap, and trainable tokens (2.31M vs 2.3M; every sampled token is preserved for training in prefix-merging algorithm). The only difference by design is the number of tokens forwarded through the trainer (30.8M vs 6.3M). The ablation states trainer-side speedup for a given model, dataset and hardware consumed over the same epoch.
Q5. Fig. 5b. The metric is utilisation of the 4 inference GPUs, averaged over the 3-step window. The representation does not change inference cost. It changes how fast the trainer consumes rollouts. During Async RL, a max async step is necessary to limit staleness between training and inference. With per-request the trainer is the bottleneck and rollout turns idle to wait for training updates. With prefix merging the trainer keeps pace and they stay busy (87.7%).