- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses ineffective response groups with zero reward variance in RLVR. POPO retains effective online groups and fills the remaining batch slots with recent effective groups from a FIFO replay buffer. Its optimization objective separates behavior-policy importance correction from PPO clipping around the current proximal policy. Experiments cover mathematics, Countdown, and visual geometry with models ranging from 1.5B to 7B.
Critical strengths
- The paper targets an important practical bottleneck. POPO builds training data composed entirely of effective samples without incurring additional rollout cost.
- The evaluation spans several tasks and model settings. The ablations examine group quality, replay recency, and the optimization objective. The DSR-1.5B comparison includes ReMix, REPO, RR, and GRESO, and the MoPPS integration suggests that replay can complement predictive sampling.
Critical weaknesses
-
Important buffer edge cases are underspecified. Algorithm 1 initializes the buffer as empty but immediately requests replay groups. What happens when the first batch has too few effective groups, or when effective groups remain scarce for several iterations? Smaller batches, skipped updates, sampling with replacement, and additional rollouts have different optimization and cost implications. A FIFO capacity of also does not bound sample age when few new effective groups enter the buffer. These cases matter directly for the claims of full effective batches and controlled off-policy drift.
-
The experiments do not fully isolate the benefits of historical replay from optimization effort and run-to-run variation. GRPO-filter is a useful control, but it changes the effective batch size and may also change the number of optimizer updates and gradient noise. A complementary control would fill batches by repeating currently effective groups, or increase optimization epochs, while matching optimizer updates or training-token budgets. This would test whether historical replay contributes beyond greater reuse of informative data.
-
The insight provided in this paper is limited, and some claims are too strong, such as claims about off-policy bias correction and statistical correctness.
Questions for authors
- Do Tables 1 and 2 report final checkpoints or checkpoints selected using a separate validation set? Are the main comparisons reproducible across multiple training seeds?