acceptodds
Under review as a conference paper at ICLR 2027

How Much, Not Where: RLVR's Capability Loss Is Dose-Controlled and Deep-Tail Only

Abstract

Reinforcement learning with verifiable rewards (RLVR) reliably improves pass@1, yet can shrink the set of problems a model solves at large sampling budgets. We ask where that damage lives, when it becomes visible, and what controls it. Instrumenting GRPO training of Qwen3-1.7B-Base on MATH and MBPP, we track a forward-only signal: the per-prompt decay rate λ̂ of the likelihood of frozen base-correct trajectories on boundary prompts, those the base model solves only rarely. We report three findings. The damage is invisible at standard sampling depths. Across our entire dose grid, pass@k stays significantly above base for every k ≤ 128 and entropy settles on a floor rather than collapsing; harm appears only beyond k ≈ 192, reaching −0.034 (95% CI [−0.057, −0.010]) at pass@256. Stopping at pass@64, the deepest endpoint in most of the inversion literature, would have shown no harm at all. The damage is dose-controlled, and the dose is measurable early. It grows monotonically with λ̂ × steps; λ̂ measured in the first 17% of training predicts the rate over the disjoint remainder (r = 0.79–0.94 on math) and grows superlinearly in learning rate. Rehearsal removes the damage in proportion to dose, not targeting. A dose- and schedule-matched control that anchors randomly chosen prompts protects as well as our signal-triggered scheme, and protection appears on prompts that were never anchored. The deliverable is a dose–response frontier trading deep-tail coverage against pass@1, not a per-prompt targeting rule. Code is available at https://anonymous.4open.science/r/rlvr-dose.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, RLVR, GRPO, pass@k, capability loss, training dynamics, entropy collapse, catastrophic forgetting, rehearsal, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Riz1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at why RLVR can in some cases improve pass@1 but hurt pass@k.

Critical strengths

N/A

Critical weaknesses

This paper appears to be largely if not entirely AI-written including experiments. The writing is hard to follow and littered with random statements.

Questions for authors

N/A

Reviewer_61RFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors investigate why RLVR improves pass@1 while reducing deep-tail coverage, and whether this loss can be anticipated and prevented. They use the squeeze-out rate λ̂ as a telemetry signal and introduce selective dynamic anchoring (SDA). Specifically, they (1) estimate λ̂ from the decline in mean per-token log-likelihood of frozen base-correct trajectories and (2) switch a prompt’s policy-gradient loss to an NLL loss when its rolling λ̂ exceeds a threshold, returning it to ordinary GRPO when λ̂ falls below the threshold minus a release margin. Experiments with Qwen3-1.7B-Base on MATH and MBPP training prompts report improved pass@1 but reduced pass@256; SDA reduces the deep-tail loss, while pseudo-anchoring, which replays SDA’s anchoring schedule on randomly chosen prompts with the same total anchored prompt-steps, provides similar protection.

Critical strengths

The motivation for studying deep-tail coverage is well justified. Table 1 reports that longer training improves pass@1 while reducing pass@256. This highlights the value of examining deep-tail coverage alongside pass@1 when evaluating RLVR.

The pseudo-anchoring control helps assess the value of targeting. It preserves SDA’s anchoring schedule and total anchored prompt-steps while randomly selecting the prompts (§3.3). The similar protection in both domains (§5.3) suggests that SDA’s benefit may not depend strongly on which prompts are anchored.

Critical weaknesses

The base references may overestimate deep-tail loss. The same base samples are used to select boundary prompts and estimate base pass@256 (§§3.1, 4). This can select prompts whose base performance was unusually high by chance, exaggerating the subsequent decline. An independent base evaluation is needed to verify the reported loss; Appendix B resamples the original data and therefore does not resolve this concern.

The squeeze-out rate is not directly validated as a predictor of deep-tail loss. Table 2 shows that early λ̂ predicts later likelihood slopes, but does not test whether it predicts final Δpass@256. Thus, the evidence does not yet support using this signal to anticipate coverage loss.

Questions for authors

Evaluation settings: Did base sampling and deep-tail evaluation use identical temperature, maximum generation length, and stopping rules? In particular, does the MBPP limit of 512 tokens and the stop sequence [DONE] in Table 5 also apply to the base references?

The MBPP comparison: Figure 1 presents the MATH pass@k curves. Could you provide MBPP results at k = 64, 128, and 192 to clarify whether the claim that damage is deep-tail only holds in both domains?

Individual extinction: Figure 5 restricts the analysis to prompts with at least four base successes. How does this criterion account for the possibility of observing zero successes in 256 samples without a decrease in the underlying success probability? Please clarify how sampling uncertainty affects the reported extinction and recovery counts.

Reviewer_7cbMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relationship between reinforcement learning with verifiable rewards (RLVR), training intensity, and large-budget pass@k coverage. Using GRPO on Qwen3-1.7B-Base, the authors observe that improvements in pass@1 can coexist with degradation at very large sampling budgets. They introduce a likelihood-decay statistic estimated from frozen base-correct trajectories and examine its relationship with training dose. The paper further studies negative-log-likelihood anchoring and finds that dose-matched random anchoring provides protection comparable to signal-triggered anchoring. Experiments are conducted on MATH and MBPP, with the main analysis focusing on rare base-solvable training prompts.

Critical strengths

1.The paper reveals that RLVR can improve pass@1 while degrading capability coverage at large sampling budgets, highlighting a limitation of conventional evaluations. 2.The dose-matched random anchoring experiments provide useful evidence that protection depends more on rehearsal dosage than prompt selection.

Critical weaknesses

1.The evidence for generalizable capability loss remains limited, as all pass@k evaluations use training prompts. The observed degradation therefore reflects reduced coverage over previously encountered problems rather than necessarily indicating a loss of reasoning capability on unseen problems. 2.Although the likelihood-decay signal predicts subsequent probability decay, its correlation with actual per-prompt pass@256 changes is negligible (r = -0.065). Moreover, the comparable performance of targeted and random anchoring leaves its practical advantage over simpler aggregate dose estimators unclear. 3.The effectiveness of the proposed dose-control strategy is not sufficiently isolated, since the experiments lack comparisons with KL regularization and constant-rate random rehearsal under matched training budgets. 4.Although the dose-response trends are examined across learning rates and training horizons, the deep-tail crossover is established within one model family and GRPO. Its stability across model scales, optimization algorithms, and sampling temperatures remains unclear.

Questions for authors

1.Does the deep-tail pass@k degradation persist on held-out problems under the same training and sampling configurations? 2.What practical predictive advantage does the proposed decay-rate signal provide over simpler training statistics, given that targeted anchoring performs similarly to random anchoring?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.