acceptodds
Under review as a conference paper at ICLR 2027

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Abstract

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. Building on advances in asynchronous RL efficiency and off-policy robustness, we study stable and effective single-rollout optimization for long-horizon agentic tasks. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement learning, language model, agent, long horizon

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UhfJOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Then propose an alternative to GPRO that gets rid of groups (hence less off-policy due to waiting for groups to fill), and add value functions back. They show then can train the value function by freezing attention, updating the critic 2x as often as the policy model, and using pretraining data. They also use skip-observation GAE to avoid predicting the value function of a noisy environment, and DIS clipping.

Critical strengths

Strong results and impressive that they did something that the field has been trying to work out how to do for a long time. Their method combines lots of sensible and interesting components which they have explained clearly and validated with impressive results.

Critical weaknesses
  • No major weaknesses but it would be good to explain how the GRPO-based baselines were tuned.
Questions for authors
  • Can you expand more on the difference in compute cost between SAO and GRPO.
2 comments

Thank you for acknowledge the contribution of our work.

No major weaknesses but it would be good to explain how the GRPO-based baselines were tuned.

The paper evaluates two GRPO-based baselines using the same backbone and asynchronous pipeline as SAO (Appendix B):

  • GRPO: Uses group-relative advantages, DAPO’s clip-higher and token-level loss, and importance sampling between the current training policy and the latest old-policy snapshot.
  • GRPO (w/ DIS): Retains group-relative advantages and replaces the clipped objective with DIS, using the direct current-to-rollout importance ratio with double-sided hard masking.

Can you expand more on the difference in compute cost between SAO and GRPO.

The training costs are summarized below:

MethodGPUsTraining timeGPU-hours
GRPO48 H80020 h960
SAO64 H80020 h1,280

Thank you for your response. I will maintain my score - I am somewhat surprised by the low scores of the other reviewers and can't see a strong reason to reduce mine.

Reviewer_sFCmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • SAO is a recipe for asynchronous agentic RL with two parts. (1) DIS: a per-token importance ratio formed directly from the logged rollout probability vs the current policy, with a hard double-sided mask (tokens outside [1-eps_l, 1+eps_h] get zero gradient) and the ratio detached, so the gradient only flows through log pi. (2) One rollout per prompt instead of GRPO groups, made workable by a learned value model with 2 critic updates per policy step, frozen attention in the critic, observation-skipping token-level GAE (from VAGEN), and a scaled value pre-training corpus (16 samples per prompt from the SFT model).
  • Experiments on Qwen3-30B-A3B: math with a Python tool, SWE-Bench Verified with OpenHands, ablations of the value-model tricks, a simulated online writing-style task scored by an LLM judge, and a frontier-scale run on GLM-5.2 with gains over SFT on five coding benchmarks.
  • Claim: SAO trains stably for ~1000 steps while vanilla GRPO and vanilla VAPO collapse early, and single-rollout training is the right fit for online settings with one feedback trajectory per prompt.
Critical strengths
  • The motivation is real and well explained. In async RL a GRPO group has to wait for its slowest member, and group sampling does not fit online settings with one trajectory per prompt. Fig. 2 makes the point clearly, and the stability story is consistent through the paper.
  • The value-model side has a reasonable set of ablations (critic update frequency, frozen attention, running-mean baseline, vanilla VAPO) and training-dynamics plots that give each trick some support.
  • Mostly clear writing, and a frontier-scale sanity check on GLM-5.2 that, if it holds, shows the recipe survives at scale.
Critical weaknesses
  • The numbers are extraordinary but the training data is never described. A 30B-A3B model reaching 97.3 on AIME 2025, 74.8 on BeyondAIME and 88.3 on HMMT is a very big claim. The paper does not say what the "RL training set" is: size, source, date range, or any decontamination against AIME25 / HMMT Nov 2025 / BeyondAIME / IMOAnswerBench. The TIR SFT data is distilled from GPT-OSS-120B on an unnamed set. Without this I cannot judge soundness. Also the proprietary rows in Table 1 are most likely no-tool numbers, so mixing them with tool-augmented results is misleading.
  • The SAO vs GRPO comparison is confounded. Both use batch 128, but SAO sees 128 unique prompts per step while GRPO sees 16 prompts x 8 samples, so SAO gets 8x the prompt diversity. SAO also trains a 30B critic and pre-trains it on 16 samples per prompt over the RL + SFT sets, which is a lot of extra compute GRPO does not get. There is no GRPO run with 128 prompts and no SAO run with 16 prompts, so I cannot tell whether "single rollout" or "more prompts + a critic" is what helps.
  • Novelty and related work. DIS in Eq. (3), a detached masked ratio times advantage times log pi, is essentially the CISPO objective from MiniMax-M1 with a hard mask instead of a clip, and masked / truncated importance sampling for training-inference mismatch is also published; neither is cited or compared, only IcePop is. Single-rollout + critic is PPO / VAPO with group size 1. Frozen-attention critic and 2x critic updates are small tricks. The paper itself says the contribution is "the combined design". The paper also does not connect to the recent multi-agent RL line (e.g., agent- and turn-wise grouping in STRONGER-MAS) where the same "groups must wait" problem shows up at every agent-turn and a single-rollout critic would be a natural alternative.
Questions for authors
  • What is the RL training set (source, size, dates) for the math-TIR and the SWE experiments, and what decontamination was done against AIME25, HMMT Nov 2025, BeyondAIME and IMOAnswerBench? Were the proprietary numbers in Table 1 measured with or without a Python tool?
  • Can you run (a) GRPO+DIS with 128 prompts x 8 samples and (b) SAO restricted to 16 prompts per step, so the effect of prompt diversity is separated from the single-rollout + critic design? Please also report total compute including critic pre-training.
  • How does DIS differ from CISPO (MiniMax-M1) and from truncated / masked importance sampling? A direct ablation would settle this. Also, why bounds of [0.7, 5.0]?
  • Are the Table 1 numbers from the final step or the best step, and did GRPO (w/ DIS) and SAO get the same number of optimizer steps and the same wall-clock? Fig. 3 shows the curves diverging after step 400, so this matters.
2 comments

We thank the reviewer for the detailed feedback and questions.

W1 & Q1: Training data and evaluation settings

The numbers are extraordinary but the training data is never described... What is the RL training set (source, size, dates) for the math-TIR and the SWE experiments, and what decontamination was done against AIME25, HMMT Nov 2025, BeyondAIME and IMOAnswerBench? Were the proprietary numbers in Table 1 measured with or without a Python tool?

A1. Our training data are drawn from public sources. Both the SFT and RL math-TIR problems are sourced from DAPO-Math-17k, NuminaMath 1.5, and OpenR1-Math-220k. TIR SFT uses 1,000 problems with trajectories distilled from GPT-OSS-120B, while RL uses 3,000 problems filtered to Qwen3-30B-A3B pass rates within [1/8,3/8][1/8, 3/8]. For software engineering, RL uses 2,000 instances from the open-source SWE-Dev dataset [1], evaluated on SWE-Bench Verified.

The external-model rows are reference scores, not tool-matched comparisons. They are quoted from their respective technical reports without Python tool access, whereas the SAO and GRPO-based RL results use Python execution under a common protocol. Our algorithmic claims rest on the comparisons among these Qwen-based RL methods.

[1] Haoran Wang, Zhenyu Hou, Yao Wei, Jie Tang, and Yuxiao Dong. SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling.


W2 & Q2: Single-rollout sampling and comparison budgets

The SAO vs GRPO comparison is confounded... Can you run (a) GRPO+DIS with 128 prompts x 8 samples and (b) SAO restricted to 16 prompts per step, so the effect of prompt diversity is separated from the single-rollout + critic design?

A2. Our experiments match the number of sampled trajectories to compare methods under similar training compute. Greater prompt diversity within the same rollout budget is an advantage of SAO's single-rollout design. Our 128-trajectory batches contain 128×1128\times1 samples for SAO and 16×816\times8 for GRPO. The critic allows SAO to use one trajectory per prompt; standard GRPO cannot do so because its group-relative advantages become zero at G=1G=1. Broader prompt coverage is therefore enabled by the single-rollout formulation, rather than by allocating more rollout samples to SAO.

The two requested runs would change the per-update trajectory budget in opposite directions. GRPO at 128×8128\times8 would require 1,024 trajectories—approximately eight times compute—which exceeds our available resources.

We also measured rollout-to-training KL to characterize policy mismatch. It is 0.000314 for SAO versus 0.001216 for GRPO, a 74.2% reduction consistent with the motivation to remove group-waiting-induced staleness. As for critic pretraining, it is a reusable, one-time initialization cost. It can be amortized across NN compatible RL runs (480/N480/N GPU-hours per run).

StageGPUsWall-clock timeGPU-hours
SAO online RL64 H80020 h1,280
SAO critic preparation (data generation + pretraining)48 H80010 h480

W3 & Q3: DIS, related work, and threshold choices

How does DIS differ from CISPO (MiniMax-M1) and from truncated / masked importance sampling? A direct ablation would settle this. Also, why bounds of [0.7, 5.0]?

A3. DIS uses direct current-to-rollout correction and double-sided token masking. DIS divides the current training-policy probability by the logged rollout-engine probability, jointly accounting for policy drift and training–inference engine mismatch.

CISPO caps importance weights; DIS masks out-of-range tokens. CISPO caps importance weights (with no lower bound in its experiments) without discarding tokens solely because their weights exceed the cap. DIS instead applies a double-sided binary mask, retaining token contributions only when 1−ϵℓ<r<1+ϵh1-\epsilon_{\ell}<r<1+\epsilon_h to limit the influence of off-policy updates. Appendix C contrasts its combined-ratio mask specifically with IcePop's engine-mismatch correction plus PPO clipping .

The bounds define an asymmetric retention interval. The bounds (0.7,5.0)(0.7, 5.0) correspond to ϵℓ=0.3\epsilon_{\ell}=0.3 and ϵh=4.0\epsilon_h=4.0, allowing larger upward importance-ratio changes before masking. These are the settings used in our experiments, not a claim of globally optimal thresholds.


Q4: Checkpoint selection and training budgets

Are the Table 1 numbers from the final step or the best step, and did GRPO (w/ DIS) and SAO get the same number of optimizer steps and the same wall-clock?

A4. Table 1 reports the best checkpoint selected by mean accuracy across the four math benchmarks. Selection uses AIME2025, BeyondAIME, HMMT November 2025, and IMOAnswerBench.

SAO and GRPO+DIS match policy-update counts and online RL wall-clock time (20 hours), not total compute. SAO uses additional GPUs for online critic training and incurs the one-time critic preparation cost reported in A2.

Area_Chair_GKuLOpenReview ↗

Reminder: Rebuttal Discussion Deadline

Dear Reviewer,

The discussion deadline is only a few hours away. Could you please respond to the authors' rebuttal before the deadline?

Best regards, Area Chair

Reviewer_PCNcOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAO for asynchronous agentic RL. It uses one rollout per prompt and a learned critic to estimate advantages, avoiding the need to wait for a group of responses as in GRPO. The method also applies double-sided masking to the current-to-rollout probability ratio and includes several critic training designs. Experiments on reasoning and coding tasks show improved performance and more stable training compared to the evaluated baselines.

Critical strengths
  1. The motivation is clear. Removing group-wise dependencies can reduce waiting in asynchronous agent training.
  2. The experiments show clear improvements over GRPO, including GRPO with the proposed DIS correction.
  3. The ablations help understand the effects of more frequent critic updates and frozen-attention critic training.
Critical weaknesses
  1. My main concern is the novelty of the method itself, as it seems very similar to existing actor–critic methods. It is unclear why the authors frame the method mainly around GRPO and emphasize that only one rollout per prompt is needed, since this is already standard in actor–critic methods.

  2. My second concern is that the proposed method seems to mainly combine existing training techniques and hyperparameter choices, rather than introduce a new algorithm. If my understanding is correct, I suggest that the authors tone down the claims of algorithmic novelty and present the contribution more clearly as a practical training recipe for asynchronous agentic RL.

  3. The ablation studies are not sufficient to explain the benefit of single-rollout sampling. SAO uses 128 prompts with one rollout each, while GRPO uses 16 prompts with eight rollouts each. It is unclear whether the improvement comes from reduced waiting and policy lag or from seeing more different prompts per batch. An ablation varying the number of rollouts per prompt within the same actor–critic framework, while keeping the total number of trajectories fixed, would help clarify this.

Questions for authors

see weakness

2 comments

We thank the reviewer for the thoughtful feedback and constructive suggestions.

W1 & W2: Novelty and technical contributions

My main concern is the novelty of the method itself, as it seems very similar to existing actor–critic methods... I suggest that the authors tone down the claims of algorithmic novelty and present the contribution more clearly as a practical training recipe for asynchronous agentic RL.

A1 & A2. We appreciate the reviewer's feedback and clarify that SAO's contribution lies in the technical designs that make single-rollout actor–critic training effective for asynchronous agentic RL, rather than in the actor–critic paradigm itself.

  • Why we contrast with GRPO: Recent large-scale LLM RL systems widely rely on GRPO-style group-based methods [1–4], which serve as our primary baselines. While standard actor–critic naturally supports learning from a single rollout per prompt, making it reliable for long-horizon, tool-using agents under asynchronous execution requires addressing off-policy staleness, unstable value estimation, and credit assignment across environment turns.
  • Making single-rollout actor–critic viable in this regime requires more than merely replacing group-relative advantages with a critic. SAO addresses these bottlenecks through techical designs DIS, skip-observation GAE, frozen-attention value tuning, and more frequent critic updates—supported by the ablations and training-dynamics analyses in Section 4.3 and Appendix A.

[1] DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. 2025. arXiv:2501.12948.

[2] Qwen Team. Qwen3 Technical Report. 2025. arXiv:2505.09388.

[3] Xiaomi LLM-Core Team. MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement. 2026. arXiv:2610.11959.

[4] GLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering. 2026. arXiv:2602.15763.


W3: Prompt diversity under a fixed trajectory budget

The ablation studies are not sufficient to explain the benefit of single-rollout sampling... An ablation varying the number of rollouts per prompt within the same actor–critic framework, while keeping the total number of trajectories fixed, would help clarify this.

A3. Greater prompt diversity within the same rollout budget is an advantage of SAO's single-rollout design. With a batch size of 128 trajectories, SAO uses 128×1128\times1, covering eight times as many prompts as GRPO's 16×816\times8 without generating additional rollouts. In contrast, GRPO requires group-wise sampling to compute relative advantages; we retain G=8G=8 because smaller groups are more likely to yield identical rewards and zero advantages. SAO also removes the need to wait for sibling rollouts before a completed trajectory becomes eligible for training. Consistent with this motivation, its observed rollout-to-training KL is 74.2% lower than GRPO's (0.000314 vs. 0.001216).

Matching SAO's prompt coverage would require GRPO to generate kk rollouts per prompt, increasing the rollout count by kk. In our setting, k=8k=8, so matching 128 prompts requires 128×8=1,024128\times8=1,024 trajectories per update and approximately eight times compute. This exceeds our available compute budget and would no longer be a matched-rollout-budget comparison. Our results therefore demonstrate the practical benefit of broader prompt coverage under a fixed trajectory budget.

Area_Chair_GKuLOpenReview ↗

Reminder: Rebuttal Discussion Deadline

Dear Reviewer,

The discussion deadline is only a few hours away. Could you please respond to the authors' rebuttal before the deadline?

Best regards, Area Chair

Comments on the paper

1 comment

Dear Reviewers,

I hope you're doing well.

Please respond to the authors' rebuttal if you have not already done so.

Thank you for your valuable time and engagement.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.