acceptodds
Under review as a conference paper at ICLR 2027

FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models

Abstract

Asynchronous reinforcement learning suits long-horizon agents with irregular rollout times, but group-relative updates require repeated rollouts of each prompt. However, group-based sampling limits prompt coverage at a fixed rollout budget and introduces group synchronization barriers. One rollout per prompt avoids these constraints but removes the variance-reducing group baseline, making updates noisier and more prone to instability on stale, variable-length trajectories. We present FlashREINFORCE, a critic-free, one-pass framework built on One-Batch REINFORCE, a Sequence Trust Region, and Sample-Mean Optimization. These components preserve prompt coverage and signed feedback while controlling policy drift and length-amplified negative updates. In our experiments, FlashREINFORCE remains stable through 6,000 updates on DeepSeek-R1-Distill-Qwen-1.5B at a policy lag of approximately four updates. On Qwen2.5-Math-1.5B, it reaches 38.0 mean accuracy across five benchmarks, outperforming the reported GRPO baseline by 1.7 percentage points with half the rollouts (256k versus 512k). Python-tool experiments extend to the 30B MoE model Qwen3-30B-A3B, with stable training at policy lag eight. On Qwen2.5-7B-Instruct, FlashREINFORCE maintains tool use where GRPO stops making tool calls and achieves 98.3%/96.5% seen/unseen ALFWorld success.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Reinforcement Learning, Agentic RL, Asynchronous Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4WxeOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FlashREINFORCE, a critic-free, one-rollout-per-prompt asynchronous RL update with three parts: (i) a reward baseline computed as the mean over each fresh batch, with one update per batch; (ii) a sequence-level admission gate that thresholds the per-token mean of a sampled-action Bernoulli KL proxy (from DPPO); (iii) sample-mean (per-trajectory) loss aggregation. Experiments cover R1-Distill-1.5B long-CoT, Qwen2.5-Math-1.5B, Python-tool use (7B and 30B MoE), and ALFWorld.

Critical strengths

1, The method is simple and practical. One pass, no critic, no reference model, no PPO clipping. 2, The evaluation is broad, including a 30B MoE at lag 8 and an agentic environment. 3, G=1 training is stable without negative-token filtering, whereas Wu et al. (2026) report that REINFORCE++ and C-RF crash at G=1.

Critical weaknesses

Reason 1: Poor Overall Performance

A close inspection of Table 1 reveals that FlashREINFORCE does not demonstrate clear superiority over standard baselines. Its mean improvement is primarily driven by a single domain outlier (OlympiadBench). Degradation Across Multiple Benchmarks: On the majority of reported tasks, e.g., MATH-500, AMC23, FlashREINFORCE performs worse than standard GRPO and C-RF + NTF. Gaining marginal benefits on one dataset while suffering performance drops on most standard reasoning benchmarks raises serious questions about the generalizability and robustness of the proposed method.

Reason 2: Flawed & Unfair Baseline Comparison

In Section 4.2.2 and Table 1, the authors compare FlashREINFORCE against prior published baselines (Wu et al., 2026). However, the paper evaluates FlashREINFORCE using avg@16, whereas the reported baselines from prior literature are evaluated using avg@32.

Reason 3: Highly Trivial Algorithmic Novelty

The technical contributions presented in this work are extremely limited and largely incremental:

1, Sample-Mean Reduction: The distinction from standard REINFORCE essentially boils down to changing the loss reduction from token-level mean to sample-level mean (trajectory normalization). This is a standard hyperparameter/configuration choice rather than a technical breakthrough.

2, Sequence-Level Trust Region Gate: Filtering stale or drifted rollouts using a sequence-level threshold is already an established technique in the literature and widely implemented in existing open-source RL infrastructure (e.g., veRL).

3, Over-Modularized/Trivial Changes: Practically speaking, both "sample-mean loss" and "sequence-level gating" are pre-existing, out-of-the-box features in mainstream LLM-RL frameworks (such as veRL). Implementing the proposed approach requires nothing more than modifying a few lines in a standard configuration YAML file (e.g., changing loss_reduction and enabling sequence_mask).

Questions for authors

1, An explanation of the OlympiadBench +9.6 gap.

2, Re-run GRPO, Dr. GRPO and REINFORCE++ (k=1) in the same codebase, with the same eval protocol, at matched rollouts and matched updates, with ≥3 seeds and CIs.

3, does k=1 give us higher diversity? Can we evaluate this?

Reviewer_dqWHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FlashREINFORCE, an RL approach suited to single rollout RL, which avoids problems that group-relative rollouts introduce. Departing from a recent approach to this problem, SAO, FlashREINFORCE uses no critic and instead computes advantages via batch-mean normalization, a la REINFORCE++. FlashREINFORCE also drops trajectories for which sampler (behavior) and trainer policies sufficiently disagree on token probabilities (using a mean KL formulation based on DPPO’s Bernoulli KL proxy), motivated by a theoretically bounded gap between the on and off policy objectives. Across a range of model sizes and tasks, FlashREINFORCE stably trains models using only a single rollout per prompt, even under off-policyness and many optimizer steps.

Critical strengths

Single rollout RL has many advantages/applications, and this paper provides a recipe in FlashREINFORCE that facilitates it.

Experiments utilize a variety of datasets and model scales.

Critical weaknesses

My main concern is that the submission lacks controlled comparisons to strong baselines. Currently, it’s difficult for readers to understand where benefits come from, as it’s unclear if the experiment setup changes or the method changes are driving results.

  • Tables 1 & 4: the baselines are published numbers, making it unclear if the RL codebase, agent evaluation harness, etc. are constant across methods. Furthermore, performance improves primarily on Olympiad in Table 1, and it's not clear if improvements over GRPO are simply due to FlashREINFORCE's seeing more prompts.
  • In Table 2, the baseline is collapsing in the middle of training (the GRPO result in Figure 3 is collapsing at just one step off-policy). Many fixes to vanilla GRPO have been proposed (e.g. Dr. GRPO, TBA-style KL reg, score centering, etc.), and such prior work could likely produce a more stable/strong baseline.
Questions for authors

Could you please add more controlled experiments? E.g., implement a strong baseline for single-rollout methods (e.g. SAO) and/or for group-relative rollouts (e.g. Dr. GRPO with IS and score centering). Then, using the same setup, drop in FlashREINFORCE.

Could you please add to the main text more details on the training/inference setup, for both the baselines and FlashREINFORCE?

Reviewer_hEThOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents FlashREINFORCE, a value-function-free, single-step unrolling asynchronous reinforcement learning method. The approach combines batch mean reward centering, sequence-level trust region masking, and sample mean optimization.

Critical strengths

The method is simple and achieves promising results, particularly in asynchronous training and tool-use tasks.

Critical weaknesses
  1. Limited novelty. The proposed components are closely related to existing techniques, and the differences from prior work as well as actual innovations have not been clearly articulated.

  2. Weak motivation. The paper highlights limitations of GRPO, such as repeated rollouts and group synchronization. However, these issues are absent in REINFORCE, upon which FlashREINFORCE is built.

  3. Missing baselines. The paper mainly compares against GRPO rather than the underlying REINFORCE method. This makes it difficult to assess the benefits of the proposed modifications.

Questions for authors
  1. What are the key differences between the proposed components and existing techniques? Which parts are genuinely novel?

  2. Why are REINFORCE and REINFORCE++ not included as baselines?

11 comments

Thank you for your response. Our contribution is not simply to introduce another configuration option to veRL, but to demonstrate, for the first time, that critic-free training with a single rollout can work very well. Our key contribution is showing that combining a sample-mean loss with batch-mean advantage estimation makes critic-free, single-rollout training scalable. To the best of our knowledge, no prior open-source work has achieved this.

The primary goal of single-rollout REINFORCE with batch-mean advantages is to demonstrate that a critic-free, single-rollout method can achieve performance comparable to methods using group-mean advantages. We consider this the main contribution: single-rollout training is faster, and achieving comparable performance at lower computational cost is a meaningful result in itself. Consistently outperforming GRPO while also training faster would be a substantially stronger objective. The gains should be assessed alongside the sample budget and computational cost. Our main experiments use only half as many samples as the reference experiments. In addition, the reference runs use synchronous training, whereas ours use asynchronous training, reducing GPU costs.

Consider TD3, a well-known and influential RL algorithm. Its core innovations can also be viewed as a combination of relatively simple modifications, yet their combined impact has made it widely adopted. The significance of a contribution lies in the capability it enables, rather than the number of configuration options it introduces. The goal is to solve the problem, not to solve it in a complex way。

In addition, veRL’s K3 rejection mechanism is difficult to tune: the importance sampling (IS) ratio is not a particularly reliable rejection criterion, and choosing an appropriate threshold is challenging. By contrast, Flash Reinforce’s default threshold of 0.05 works well across most settings. The K3 criterion uses a geometric mean of probability ratios, which we found difficult to control. Ratio-based criteria also respond differently to changes in high- and low-probability tokens, potentially penalizing low-probability tokens disproportionately (maybe also refer to DAPO clip-higher).

Thank you for the clarification. I have two remaining questions:

  1. What advantages does FlashREINFORCE offer over standard REINFORCE under the same training settings? Please provide a direct comparison to show the benefits of your modifications.

  2. REINFORCE already supports critic-free training with a single rollout per prompt. What exactly does “for the first time” refer to in your response?

Advantages over standard REINFORCE, and what "for the first time" refers to

We agree that REINFORCE is critic-free and single-rollout by definition. "For the first time" refers to making it work in long-horizon asynchronous training: FlashREINFORCE stays stable through 6,000 updates at policy lag ≈4, with no critic, PPO clipping, or negative-token filtering (Fig. 2, Sec. 4.2.1). Prior work does not cover this setting: C-RF + NTF (Wu et al., 2026) is shown only with synchronous training, and SAO (Hou et al., 2026) needs a learned value model. We will state this scope explicitly in the revision.

Direct comparison

  • Single-rollout REINFORCE baseline. Wu et al. (2026) report that REINFORCE++ and C-RF with G=1 collapse during training without Negative Token Filtering. Liu et al. (Tricks or Traps) likewise find batch-level normalization unstable under binary rewards, and their Lite PPO still relies on a group-level mean. FlashREINFORCE needs neither filtering nor groups.
  • Token-mean loss (standard) vs. sample-mean (ours). Token-mean collapses within about 150 updates (Table 6b).
  • IS only vs. sequence trust (ours). 30.6 vs. 33.7 on AIME24/25 (Table 8).
  • Positive-only vs. signed feedback (ours). 9.8 vs. 36.2 mean accuracy (Table 6a).

Comments on the paper

1 comment

Dear Reviewers,

I hope you're doing well.

Please respond to the authors' rebuttal if you have not already done so.

Thank you for your valuable time and engagement.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.