acceptodds
Under review as a conference paper at ICLR 2027

RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). However, its effectiveness is substantially hindered by the prevalence of ineffective training data: many sampled prompts yield response groups that are either entirely correct or entirely incorrect, resulting in zero-variance rewards and limited learning signals. Recent state-of-the-art methods address this issue through extensive LLM rollouts to filter ineffective samples, but at the cost of considerable computational overhead. Alternative approaches, including predictive sampling and trajectory replay, aim to improve data efficiency but often remain insufficient and may introduce additional issues such as systematic bias or suboptimal constraints. To address these limitations, we propose Group Prioritized Off-Policy Optimization (POPO), a simple yet effective framework that fully exploits effective training batches without additional rollout overhead. POPO comprises two key components: prioritized group replay and decoupled off-policy optimization. The former replaces ineffective on-policy groups with effective off-policy groups via a recency-based replay mechanism that jointly considers sample quality and the degree of off-policiness. To further mitigate the off-policy gap, POPO employs decoupled importance sampling to correct off-policy bias while maintaining stable policy updates under consistent trust-region constraints. Empirical evaluations across diverse reasoning tasks, including mathematics, planning, and visual geometry, demonstrate that POPO substantially accelerates RL finetuning and achieves strong reasoning performance with significantly fewer rollouts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, large reasoning model, reinforcement learning, ineffective training data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8UE4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses ineffective response groups with zero reward variance in RLVR. POPO retains effective online groups and fills the remaining batch slots with recent effective groups from a FIFO replay buffer. Its optimization objective separates behavior-policy importance correction from PPO clipping around the current proximal policy. Experiments cover mathematics, Countdown, and visual geometry with models ranging from 1.5B to 7B.

Critical strengths
  • The paper targets an important practical bottleneck. POPO builds training data composed entirely of effective samples without incurring additional rollout cost.
  • The evaluation spans several tasks and model settings. The ablations examine group quality, replay recency, and the optimization objective. The DSR-1.5B comparison includes ReMix, REPO, RR, and GRESO, and the MoPPS integration suggests that replay can complement predictive sampling.
Critical weaknesses
  • Important buffer edge cases are underspecified. Algorithm 1 initializes the buffer as empty but immediately requests B−∣Gon−eff∣B-|G_{\mathrm{on-eff}}| replay groups. What happens when the first batch has too few effective groups, or when effective groups remain scarce for several iterations? Smaller batches, skipped updates, sampling with replacement, and additional rollouts have different optimization and cost implications. A FIFO capacity of BB also does not bound sample age when few new effective groups enter the buffer. These cases matter directly for the claims of full effective batches and controlled off-policy drift.

  • The experiments do not fully isolate the benefits of historical replay from optimization effort and run-to-run variation. GRPO-filter is a useful control, but it changes the effective batch size and may also change the number of optimizer updates and gradient noise. A complementary control would fill batches by repeating currently effective groups, or increase optimization epochs, while matching optimizer updates or training-token budgets. This would test whether historical replay contributes beyond greater reuse of informative data.

  • The insight provided in this paper is limited, and some claims are too strong, such as claims about off-policy bias correction and statistical correctness.

Questions for authors
  • Do Tables 1 and 2 report final checkpoints or checkpoints selected using a separate validation set? Are the main comparisons reproducible across multiple training seeds?
Reviewer_D3SPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Group Prioritized Off-Policy Optimization (POPO) to improve the sample efficiency of RLVR. POPO filters out response groups with zero reward variance and replenishes training batches with recent effective groups from a replay buffer. It also uses decoupled importance sampling to separate off-policy correction from policy-update clipping. Experiments on mathematical reasoning, Countdown, and visual geometry show performance comparable to DAPO with fewer rollouts and lower training time.

Critical strengths

The paper is generally well written and easy to follow, with a clear presentation of the motivation and proposed method.

Critical weaknesses
  1. The submission’s claimed innovations substantially overlap with prior work [1].
  • Prioritized Group Replay substantially overlaps with the batch construction mechanism in [1]: both filter out online groups with zero reward variance and fill the remaining batch capacity with recent informative historical groups. This submission admits historical groups based on nonzero reward variance and prioritizes the most recent groups, whereas [1] selects historical groups using a dynamic interval of group mean rewards and also prioritizes the most recent groups within a recent window. These differences are negligible.
  • Both methods use importance sampling to address the off-policy discrepancy introduced by historical data. This submission further factorizes the importance weights and uses the pre-update policy as a consistent reference for clipping. This constitutes a concrete distinction, but the decoupled formulation itself originates from prior work [2,3], and its incremental value over [1] in this setting remains to be demonstrated.
  1. Algorithm 1 initializes the replay buffer as empty but immediately requires retrieving B−∣Gon−eff∣B-|G_{\rm on-eff}| historical groups. In the first iteration, or when effective groups are scarce, the buffer may contain fewer groups than required. The paper does not explain how this case is handled.

  2. The experiments mainly focus on single-turn reasoning tasks and lack validation on multi-turn agent tasks.

[1] Xu Wan et al. Buffer matters: Unleashing the power of off-policy reinforcement learning in large language model reasoning. ICLR 2026

[2] Wei Fu et al. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. NeurIPS 2025.

[3] Jacob Hilton, Karl Cobbe, and John Schulman. Batch Size-Invariance for Policy Optimization. NeurIPS 2022.

Questions for authors

Without a direct comparison against [1] and a controlled experiment that holds batch construction fixed while varying only the optimization objective, it is difficult to assess the additional benefit of each component.

Reviewer_cj3vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Group Prioritized Off-Policy Optimization (POPO) to improve the rollout efficiency of RL with verifiable rewards. After collecting an ordinary batch, POPO retains response groups with nonzero reward variance and fills the remaining slots with recent, complete effective groups from a small FIFO buffer. For replayed data, it separates a behavior-to-proximal importance weight from the current-to-proximal clipping ratio. This permits replay without generating the additional candidate responses used by dynamic filtering. Experimental results on mathematical reasoning, Countdown, and visual geometry approach DAPO's accuracy without external rollout and runtime costs.

Critical strengths
  1. The good point of this paper is to realize a useful accuracy and cost trade-off across multiple reasoning settings. For example, Table 1 reports 63.3 versus 63.2 for the 7B model with 34 rather than 55 hours of DAPO. These results support a practical efficiency contribution, with the small accuracy shortfalls on the latter tasks stated explicitly.

  2. The paper compares recent-group replay against KL-based selection, on-policy filtering, ineffective-group replay, stale replay, and alternative off-policy objectives. Keeping the replay mechanism fixed when changing the objective is particularly useful. Figure 6 further supports compatibility with different group sizes, RLOO/PPO, and MoPPS.

Critical weaknesses
  1. Lemma 4.2 establishes equality of the two surrogate contributions and their gradients when the relevant clipping is inactive, but it does not establish unbiased estimation of an on-policy gradient or a training-stability guarantee.

  2. The upper bound of the outer importance weight is set as 2.0, but its contribution is not isolated from that of the decoupled objective. Comparing different value of upper bound might clarify this dependence.

Questions for authors
  1. For the first step, I was wondering how are empty replay buffers handled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.