- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores an issue that arises when using RLVR in the context of large language models (LLMs): whether policies learned by practical, sample-based RLVR recover the optimum of the underlying KL-regularized objective. This paper finds that practical RLVR policies can deviate from the Gibbs-form optimum of the KL- regularized objective and introduces Reference-Anchored Policy Correction (RAPC), which explicitly models the policy model as its optimal formulation.
Critical strengths
- Most existing research on RLVR focuses on boosting scores on benchmarks or refining the formulation of the advantage function; in contrast, this paper explores whether the actual optimization outcomes deviate from the theoretical optimum of the objective function itself—a compelling perspective.
- This paper extends the sequence-level Gibbs optimal solution to the prefix level and interprets the correction applied under binary rewards as the success probability of the reference policy, offering interpretability.
- This paper proposes Reference-Anchored Policy Correction (RAPC), which explicitly models the policy model according to its optimal formulation.
Critical weaknesses
- Although is initially designed as an estimate of the subsequent success probability , end-to-end training using the clipped and group-normalized GRPO advantage function (i.e., ) does not guarantee that will recover the true .
- How can it be proven that minimizing causes the intermediate state estimate to truly converge to the actual ?
- Section 4.2 attributes the failure of standard RLVR to the variance of policy gradients estimated from samples under a fixed learning rate. However, since RAPC is also trained using empirical policy gradients via GRPO (Equation 12)—it faces the exact same issue of sampling noise.
- No ablation study evaluates where .
- In Table 1, the one-shot accuracy scores for Qwen2.5-7B (the baseline model) on MATH500, OlympiadBench, and SuperGPQA500 differ significantly from the results presented in the first row (“baseline sampling”) of Table 3 (inference phase). These discrepancies between the two tables, as well as the differences in experimental setup, warrant a clearer explanation.
- This study relies exclusively on the Qwen2.5-7B-math and Qwen3-4B models, which limits the scope of the experiments. There are several drawbacks: first, the models employed are outdated; second, the experiments do not encompass a diverse range of model scales, nor do they explore other model families.
Questions for authors
- Eq. (13): The first part is written as , while the second part is written as ; the notation is inconsistent.
- The main text mentions using Qwen2.5-Math-7B, while the tables and some reasoning experiments refer to Qwen2.5-7B; the specific name used needs to be clarified.
- In Table 4, both the 3B and 7B models have a validation score of 79.8; however, the main text states that the 3B model achieved the best validation performance. Clarify that they are tied for the best performance.
3 comments
W5 — Why does single-sample accuracy differ between the Qwen2.5 baseline in Table 1 and baseline sampling in Table 3?
Tables 1 and 3 use different decoding settings. Table 1 uses a sampling temperature of . Table 3 uses a temperature of . We use the higher temperature in Table 3 to encourage candidate diversity in the test-time sampling experiments.
W6 —How broadly do the results generalize beyond the two Qwen backbones evaluated?
We extend our evaluation to Qwen3.5, comparing RAPC with full-model GRPO. As shown in Table R7, RAPC improves the average across the eight benchmarks from 57.65% to 59.54%, a gain of 1.89 percentage points. RAPC scores higher on six benchmarks, ties on AIME25, and drops from 50.00% to 46.67% on AIME24. The average gain therefore extends to an additional Qwen backbone, with performance varying by benchmark.
| Method | GSM8K | MATH500 | OlympiadBench | SuperGPQA500 | AMC23 | AIME24 | AIME25 | AIME26 | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Full GRPO | 89.92 | 72.20 | 38.22 | 25.00 | 92.50 | 50.00 | 40.00 | 53.33 | 57.65 |
| RAPC | 90.67 | 79.00 | 42.07 | 26.20 | 95.00 | 46.67 | 40.00 | 56.67 | 59.54 |
Q1–Q3 — Which terminal prediction enters Eq. (13), which Qwen2.5 checkpoint was used, and why was the 3B estimator selected given the tie in Table 4?
Both terms of Eq. (13) should use the same terminal prediction , with as the observed verifier label. We will define this notation and use Qwen2.5-Math-7B consistently for the reference checkpoint throughout the paper. We will also correct the description of Table 4. The 3B and 7B estimators tie at 79.8% validation accuracy. We selected the 3B estimator for the main experiments because it achieves the same validation accuracy with fewer parameters.
Dear Reviewer xWeE,
Thank you again for your thoughtful review. We have provided point-by-point responses to your questions and revised the manuscript, adding theoretical analysis, the terminal-loss ablation, and an additional backbone evaluation, while clarifying the experimental settings and notation.
If you have an opportunity to look over these updates, we would be grateful for any further feedback. We would be happy to clarify any remaining questions.
Thank you for your time and consideration.
Best regards, The Authors of Submission 17775
We have added a convergence theorem and its proof to the appendix. It shows when approaching the joint population optimum yields pϕ(st,yt+1)→pref(st+1) for reference-supported actions at prefixes covered by terminal supervision.
First, adding terminal BCE preserves the Gibbs optimum. Substituting the true continuation probability pref(st+1) for pϕ(st,yt+1) in Eq. (9) gives the Gibbs policy π⋆, which maximizes the objective in Eq. (1). For a supervised termination action, the response ends immediately, so the true continuation probability equals the verifier’s binary label. The same choice therefore gives zero terminal BCE. Under our approximation assumption, the estimator can approach both optimal values simultaneously, so the joint loss has the same infimum as the policy objective alone, −J(π⋆).
Second, terminal BCE fixes the scale of the correction. Along a sequence approaching the joint infimum, both the policy objective gap and the terminal BCE tend to zero. Comparing Eq. (9) with the Gibbs transition shows that policy convergence determines the correction factors up to a common scale Zϕ(st)/h⋆(st). At a prefix where a supported termination action receives persistent supervision, the BCE drives the prediction for that action to its verifier label. This forces the common scale to converge to 1, so the predictions for all reference-supported actions at that prefix converge to their true continuation probabilities.
Clipping in Eq. (12) is used solely to stabilize finite-sample updates. In practice, it was active for fewer than 0.024% of token-level policy-loss terms in every reported training interval on both backbones. Thus, clipping affected only a negligible fraction of these terms (Table R6).
Table R6. Fraction of token-level policy-loss terms with active clipping.
In RAPC, repeated noisy updates cannot drive response odds arbitrarily far from the frozen reference. RAPC uses the same finite-sample GRPO loss and remains subject to sampling noise. At every training step, the policy is obtained by applying a bounded correction to the same frozen reference. Proposition 1 therefore bounds the total shift in next-token odds relative to that reference throughout training. For any two fixed responses to the same prompt, the resulting bound on their sequence-level log-odds shift depends only on β and the lengths of their suffixes after divergence, regardless of how many noisy updates have occurred.
We added the requested ablation with λanchor=0, which removes the terminal BCE. As shown in Table R4, removing the anchor lowers the average across the eight benchmarks from 55.35% to 51.43%, a drop of 3.92 percentage points. The default configuration with λanchor=0.05 scores higher on all eight benchmarks than the configuration with λanchor=0. The additional weights of 0.01 and 0.10 also improve the average over the zero-anchor setting, with 0.05 giving the highest average among the tested weights.