Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Reinforcement learning with verifiable rewards (RLVR) often suffers from diversity collapse: Pass@$1$ improves while high-$k$ Pass@$k$ degrades. We study this divergence through the lens of overtraining. We first show that validation Pass@$1$ largely plateaus while training-set success continues to climb and high-$k$ coverage continues to decline. We then show that a problem can retain substantial Pass@$1$ headroom after its direct contribution to high-$k$ coverage has nearly saturated. A Bayesian analysis quantifies this distinction under few-rollout uncertainty. Early stopping reduces coverage loss. We also find that stopping updates on high-success problems preserves or improves Pass@$256$ relative to the base model while retaining Pass@$1$ gains, even as training continues on the remaining problems. Together, these results identify overtraining as a source of diversity collapse in RLVR: continued optimization outlasts its coverage benefit, and restricting its period or scope reduces the resulting coverage loss.