- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FlashREINFORCE, a critic-free, one-rollout-per-prompt asynchronous RL update with three parts: (i) a reward baseline computed as the mean over each fresh batch, with one update per batch; (ii) a sequence-level admission gate that thresholds the per-token mean of a sampled-action Bernoulli KL proxy (from DPPO); (iii) sample-mean (per-trajectory) loss aggregation. Experiments cover R1-Distill-1.5B long-CoT, Qwen2.5-Math-1.5B, Python-tool use (7B and 30B MoE), and ALFWorld.
Critical strengths
1, The method is simple and practical. One pass, no critic, no reference model, no PPO clipping. 2, The evaluation is broad, including a 30B MoE at lag 8 and an agentic environment. 3, G=1 training is stable without negative-token filtering, whereas Wu et al. (2026) report that REINFORCE++ and C-RF crash at G=1.
Critical weaknesses
Reason 1: Poor Overall Performance
A close inspection of Table 1 reveals that FlashREINFORCE does not demonstrate clear superiority over standard baselines. Its mean improvement is primarily driven by a single domain outlier (OlympiadBench). Degradation Across Multiple Benchmarks: On the majority of reported tasks, e.g., MATH-500, AMC23, FlashREINFORCE performs worse than standard GRPO and C-RF + NTF. Gaining marginal benefits on one dataset while suffering performance drops on most standard reasoning benchmarks raises serious questions about the generalizability and robustness of the proposed method.
Reason 2: Flawed & Unfair Baseline Comparison
In Section 4.2.2 and Table 1, the authors compare FlashREINFORCE against prior published baselines (Wu et al., 2026). However, the paper evaluates FlashREINFORCE using avg@16, whereas the reported baselines from prior literature are evaluated using avg@32.
Reason 3: Highly Trivial Algorithmic Novelty
The technical contributions presented in this work are extremely limited and largely incremental:
1, Sample-Mean Reduction: The distinction from standard REINFORCE essentially boils down to changing the loss reduction from token-level mean to sample-level mean (trajectory normalization). This is a standard hyperparameter/configuration choice rather than a technical breakthrough.
2, Sequence-Level Trust Region Gate: Filtering stale or drifted rollouts using a sequence-level threshold is already an established technique in the literature and widely implemented in existing open-source RL infrastructure (e.g., veRL).
3, Over-Modularized/Trivial Changes: Practically speaking, both "sample-mean loss" and "sequence-level gating" are pre-existing, out-of-the-box features in mainstream LLM-RL frameworks (such as veRL). Implementing the proposed approach requires nothing more than modifying a few lines in a standard configuration YAML file (e.g., changing loss_reduction and enabling sequence_mask).
Questions for authors
1, An explanation of the OlympiadBench +9.6 gap.
2, Re-run GRPO, Dr. GRPO and REINFORCE++ (k=1) in the same codebase, with the same eval protocol, at matched rollouts and matched updates, with ≥3 seeds and CIs.
3, does k=1 give us higher diversity? Can we evaluate this?
Thank you for your response. Our contribution is not simply to introduce another configuration option to veRL, but to demonstrate, for the first time, that critic-free training with a single rollout can work very well. Our key contribution is showing that combining a sample-mean loss with batch-mean advantage estimation makes critic-free, single-rollout training scalable. To the best of our knowledge, no prior open-source work has achieved this.
The primary goal of single-rollout REINFORCE with batch-mean advantages is to demonstrate that a critic-free, single-rollout method can achieve performance comparable to methods using group-mean advantages. We consider this the main contribution: single-rollout training is faster, and achieving comparable performance at lower computational cost is a meaningful result in itself. Consistently outperforming GRPO while also training faster would be a substantially stronger objective. The gains should be assessed alongside the sample budget and computational cost. Our main experiments use only half as many samples as the reference experiments. In addition, the reference runs use synchronous training, whereas ours use asynchronous training, reducing GPU costs.
Consider TD3, a well-known and influential RL algorithm. Its core innovations can also be viewed as a combination of relatively simple modifications, yet their combined impact has made it widely adopted. The significance of a contribution lies in the capability it enables, rather than the number of configuration options it introduces. The goal is to solve the problem, not to solve it in a complex way。
In addition, veRL’s K3 rejection mechanism is difficult to tune: the importance sampling (IS) ratio is not a particularly reliable rejection criterion, and choosing an appropriate threshold is challenging. By contrast, Flash Reinforce’s default threshold of 0.05 works well across most settings. The K3 criterion uses a geometric mean of probability ratios, which we found difficult to control. Ratio-based criteria also respond differently to changes in high- and low-probability tokens, potentially penalizing low-probability tokens disproportionately (maybe also refer to DAPO clip-higher).