How Off-Policy Can GRPO Be? Efficient $\mu$-GRPO with Negative-Advantage Veto

Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose $\mu$-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation–optimization stages. This design induces high rollout staleness while greatly reducing rollout–optimization switching overhead. To stabilize learning under stale data, $\mu$-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, $\mu$-GRPO matches or exceeds the performance of standard GRPO while achieving around 2× speedup in wall-clock training time, establishing a substantially improved performance–efficiency trade-off for LLM reinforcement learning.