acceptodds
Under review as a conference paper at ICLR 2027

When Learning Misses the Target: Bridging the Gap Between RLVR and its Optimum

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning ability of large language models using outcome-based signals.However, it remains unclear whether policies learned by practical, sample-based RLVR recover the optimum of the underlying KL-regularized objective. To explore this, we derive the closed form solution of the desired optima, i.e., a distribution of reference policy reweighted by a Gibbs distribution, whereas the Gibbs term pushes policy to explore on tokens with higher expected reward. We further find that practical RLVR policies deviate substantially from this structure and concentrate probability on a narrower set of successful solutions.We therefore introduce Reference-Anchored Policy Correction (RAPC), which freezes the reference language model and learns a bounded process-level correction from terminal verifier feedback. The resulting acting policy combines the reference distribution with the learned reward-dependent correction. On reasoning benchmarks, RAPC more closely follows the predicted Gibbs structure and retains greater coverage of successful paths than updating the full language model under comparable evaluation protocols.These results suggest that practical RLVR can better exploit the information encoded in the KL-regularized optimum by explicitly learning its reward-induced correction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards (RLVR);KL-Regularized Reinforcement Learning;Large Language Model Reasoning;Gibbs Policy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xWeEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores an issue that arises when using RLVR in the context of large language models (LLMs): whether policies learned by practical, sample-based RLVR recover the optimum of the underlying KL-regularized objective. This paper finds that practical RLVR policies can deviate from the Gibbs-form optimum of the KL- regularized objective and introduces Reference-Anchored Policy Correction (RAPC), which explicitly models the policy model as its optimal formulation.

Critical strengths
  1. Most existing research on RLVR focuses on boosting scores on benchmarks or refining the formulation of the advantage function; in contrast, this paper explores whether the actual optimization outcomes deviate from the theoretical optimum of the objective function itself—a compelling perspective.
  2. This paper extends the sequence-level Gibbs optimal solution to the prefix level and interprets the correction applied under binary rewards as the success probability of the reference policy, offering interpretability.
  3. This paper proposes Reference-Anchored Policy Correction (RAPC), which explicitly models the policy model according to its optimal formulation.
Critical weaknesses
  1. Although pϕ(st,yt+1)p_\phi(s_t, y_{t+1}) is initially designed as an estimate of the subsequent success probability pref(st+1)p_{\text{ref}}(s_{t+1}), end-to-end training using the clipped and group-normalized GRPO advantage function (i.e., L^pol\hat{L}_{pol}) does not guarantee that pϕp_\phi will recover the true prefp_{\text{ref}}.
  2. How can it be proven that minimizing L^RAPC\hat{L}_{\text{RAPC}} causes the intermediate state estimate pϕ(st,yt+1)p_\phi(s_t, y_{t+1}) to truly converge to the actual pref(st+1)p_{\text{ref}}(s_{t+1})?
  3. Section 4.2 attributes the failure of standard RLVR to the variance of policy gradients estimated from samples under a fixed learning rate. However, since RAPC is also trained using empirical policy gradients via GRPO (Equation 12)—it faces the exact same issue of sampling noise.
  4. No ablation study evaluates where λanchor=0\lambda_{\text{anchor}} = 0.
  5. In Table 1, the one-shot accuracy scores for Qwen2.5-7B (the baseline model) on MATH500, OlympiadBench, and SuperGPQA500 differ significantly from the results presented in the first row (“baseline sampling”) of Table 3 (inference phase). These discrepancies between the two tables, as well as the differences in experimental setup, warrant a clearer explanation.
  6. This study relies exclusively on the Qwen2.5-7B-math and Qwen3-4B models, which limits the scope of the experiments. There are several drawbacks: first, the models employed are outdated; second, the experiments do not encompass a diverse range of model scales, nor do they explore other model families.
Questions for authors
  1. Eq. (13): The first part is written as pϕ(yi)p_\phi(y_i), while the second part is written as pϕ(yi)p_\phi(y_i); the notation is inconsistent.
  2. The main text mentions using Qwen2.5-Math-7B, while the tables and some reasoning experiments refer to Qwen2.5-7B; the specific name used needs to be clarified.
  3. In Table 4, both the 3B and 7B models have a validation score of 79.8; however, the main text states that the 3B model achieved the best validation performance. Clarify that they are tied for the best performance.
3 comments

W1–W2 — Why is the GRPO policy loss alone insufficient to identify prefp_{\rm ref}, and when does the joint RAPC objective recover it?

We have added a convergence theorem and its proof to the appendix. It shows when approaching the joint population optimum yields pϕ(st,yt+1)→pref(st+1)p_\phi(s_t,y_{t+1})\to p_{\rm ref}(s_{t+1}) for reference-supported actions at prefixes covered by terminal supervision.

First, adding terminal BCE preserves the Gibbs optimum. Substituting the true continuation probability pref(st+1)p_{\rm ref}(s_{t+1}) for pϕ(st,yt+1)p_\phi(s_t,y_{t+1}) in Eq. (9) gives the Gibbs policy π⋆\pi^\star, which maximizes the objective in Eq. (1). For a supervised termination action, the response ends immediately, so the true continuation probability equals the verifier’s binary label. The same choice therefore gives zero terminal BCE. Under our approximation assumption, the estimator can approach both optimal values simultaneously, so the joint loss has the same infimum as the policy objective alone, −J(π⋆)-\mathcal J(\pi^\star).

Second, terminal BCE fixes the scale of the correction. Along a sequence approaching the joint infimum, both the policy objective gap and the terminal BCE tend to zero. Comparing Eq. (9) with the Gibbs transition shows that policy convergence determines the correction factors up to a common scale Zϕ(st)/h⋆(st)Z_\phi(s_t)/h^\star(s_t). At a prefix where a supported termination action receives persistent supervision, the BCE drives the prediction for that action to its verifier label. This forces the common scale to converge to 11, so the predictions for all reference-supported actions at that prefix converge to their true continuation probabilities.

Clipping in Eq. (12) is used solely to stabilize finite-sample updates. In practice, it was active for fewer than 0.024% of token-level policy-loss terms in every reported training interval on both backbones. Thus, clipping affected only a negligible fraction of these terms (Table R6).

Table R6. Fraction of token-level policy-loss terms with active clipping.

BackboneSteps 1–50Steps 51–100Steps 101–150Steps 151–200
Qwen2.5-Math-7B0.0118%0.0093%0.0092%0.0090%
Qwen3-4B0.0239%0.0208%0.0198%0.0192%

W3 — RAPC uses the same empirical GRPO updates. What limits the drift caused by sampling noise?

In RAPC, repeated noisy updates cannot drive response odds arbitrarily far from the frozen reference. RAPC uses the same finite-sample GRPO loss and remains subject to sampling noise. At every training step, the policy is obtained by applying a bounded correction to the same frozen reference. Proposition 1 therefore bounds the total shift in next-token odds relative to that reference throughout training. For any two fixed responses to the same prompt, the resulting bound on their sequence-level log-odds shift depends only on β\beta and the lengths of their suffixes after divergence, regardless of how many noisy updates have occurred.

W4 —How much does the terminal anchor contribute, and what happens when λanchor=0\lambda_{\rm anchor}=0?

We added the requested ablation with λanchor=0\lambda_{\rm anchor}=0, which removes the terminal BCE. As shown in Table R4, removing the anchor lowers the average across the eight benchmarks from 55.35% to 51.43%, a drop of 3.92 percentage points. The default configuration with λanchor=0.05\lambda_{\rm anchor}=0.05 scores higher on all eight benchmarks than the configuration with λanchor=0\lambda_{\rm anchor}=0. The additional weights of 0.010.01 and 0.100.10 also improve the average over the zero-anchor setting, with 0.050.05 giving the highest average among the tested weights.

λanchor\lambda_{\rm anchor}GSM8KMATH500OlympiadBenchSuperGPQA500AMC23AIME24AIME25AIME26Avg. (%)
086.7376.8039.1124.6087.5040.0026.6730.0051.43
0.0189.6978.2039.2624.6087.5046.6730.0030.0053.24
0.05 (default)90.5279.8041.0424.8090.0050.0033.3333.3355.35
0.1088.1779.6039.7024.4092.5046.6730.0033.3354.30

W5 — Why does single-sample accuracy differ between the Qwen2.5 baseline in Table 1 and baseline sampling in Table 3?

Tables 1 and 3 use different decoding settings. Table 1 uses a sampling temperature of 0.20.2. Table 3 uses a temperature of 0.40.4 . We use the higher temperature in Table 3 to encourage candidate diversity in the test-time sampling experiments.

W6 —How broadly do the results generalize beyond the two Qwen backbones evaluated?

We extend our evaluation to Qwen3.5, comparing RAPC with full-model GRPO. As shown in Table R7, RAPC improves the average across the eight benchmarks from 57.65% to 59.54%, a gain of 1.89 percentage points. RAPC scores higher on six benchmarks, ties on AIME25, and drops from 50.00% to 46.67% on AIME24. The average gain therefore extends to an additional Qwen backbone, with performance varying by benchmark.

MethodGSM8KMATH500OlympiadBenchSuperGPQA500AMC23AIME24AIME25AIME26Avg.
Full GRPO89.9272.2038.2225.0092.5050.0040.0053.3357.65
RAPC90.6779.0042.0726.2095.0046.6740.0056.6759.54

Q1–Q3 — Which terminal prediction enters Eq. (13), which Qwen2.5 checkpoint was used, and why was the 3B estimator selected given the tie in Table 4?

Both terms of Eq. (13) should use the same terminal prediction pϕ(yi)p_\phi(y_i), with rir_i as the observed verifier label. We will define this notation and use Qwen2.5-Math-7B consistently for the reference checkpoint throughout the paper. We will also correct the description of Table 4. The 3B and 7B estimators tie at 79.8% validation accuracy. We selected the 3B estimator for the main experiments because it achieves the same validation accuracy with fewer parameters.

Dear Reviewer xWeE,

Thank you again for your thoughtful review. We have provided point-by-point responses to your questions and revised the manuscript, adding theoretical analysis, the terminal-loss ablation, and an additional backbone evaluation, while clarifying the experimental settings and notation.

If you have an opportunity to look over these updates, we would be grateful for any further feedback. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Best regards, The Authors of Submission 17775

Reviewer_UGQ7OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that GRPO-trained policies deviate from the closed-form optimum of KL-regularized RLVR (the reference policy reweighted by exp(R/β)) and attributes the deviation to finite-sample gradient noise. It proposes RAPC, which freezes the reference model and trains a separate 3B network with GRPO to output bounded per-token reweighting factors that mimic the optimum’s form. On math benchmarks, RAPC improves average accuracy over GRPO by 2.3 and 0.8 points (Qwen2.5-Math-7B and Qwen3-4B) and shows higher output diversity.

Critical strengths
  1. The question of whether practical RLVR reaches its own KL-regularized optimum is interesting, and framing that optimum as a reweighting of the reference policy gives a clear target to test against.
  2. Training only a small correction network on top of a frozen reference model and still outperforming full-parameter GRPO is a surprising result.
Critical weaknesses
  1. Poor readability. The language is often unclear or ungrammatical, so the method is hard to follow without substantial reconstruction.
  2. Many typos and typesetting errors. Examples: missing spaces and typos: “Definep_ref” (l.163), “Supposec_low” (l.320), “estimatorl” (l.715) grammar: “is in fact stay away” (l.178), “we modeling” (l.266) Undefined or inconsistent notation. p_φ^term (Eq. 13) is never defined. p_φ takes four different argument forms. Eq. 9 refers to Eq. 11 before it appears. Model names switch between Qwen2.5-7B and Qwen2.5-Math-7B.
  3. Estimator architecture is not specified. The paper never describes the output head, the inputs (the “prior” arrow in Fig. 1), the initialization, or the “frozen step-0 copy” (A.1).
  4. The method doesn’t address the diagnosed cause. The deviation is blamed on finite-sample gradient noise, yet RAPC is trained with the same sample-based GRPO loss. The argument behind Thm 4.1 applies equally to RAPC.
  5. D_seq is not a valid measure of distance to the optimum. The reference policy itself achieves D_seq = 0. The metric also grows with length and is dominated by a few tokens on off-policy samples. The objective gap J(π*) − J(π) = β·KL(π‖π*) would be a valid measure.
Questions for authors

Could authors specify the estimator architecture?

3 comments

W1–W2 — The method is difficult to follow, and the notation, equation references, and model names are inconsistent.

Thank you for pointing out these issues. We reorganized Section 5 to define the estimator’s inputs and outputs before introducing the next-token policy. We then describe rollout generation, verifier-based supervision, and inference in that order. We also clarified the estimator notation and corrected the typos and grammatical errors identified in the review.

W3 / Q1 — What are the estimator’s inputs and output head, and how are the prior and frozen step-0 copy used?

Estimator architecture. The estimator uses a Qwen2.5-3B causal LM with its language-model head. It reads the same token prefix sts_t as the frozen reference and outputs one logit per candidate next token in a single forward pass. The estimator score uϕ(st,a)u_\phi(s_t,a) is converted into the guidance probability pϕ(st,a)p_\phi(s_t,a) by the elementwise sigmoid in Eq. (11).

The step-0 copy provides the baseline for the estimator’s logits. For each candidate token aa at prefix ss, we compute pϕ(s,a)=σ(ℓϕ(s,a)−ℓstep0(s,a))p_\phi(s,a)=\sigma(\ell_\phi(s,a)-\ell_{\rm step0}(s,a)) with an elementwise sigmoid. At initialization, the two sets of logits coincide, so pϕ=0.5p_\phi=0.5 for every token. The resulting constant correction cancels in the softmax, and the initial acting policy equals the frozen reference.

The prior in Fig. 1 is the frozen 7B policy. n Fig. 1, the prior is the base next-token distribution πref(⋅∣st)\pi_{\mathrm{ref}}(\cdot\mid s_t) provided by the frozen 7B model. Equation (9) uses the estimator’s per-token guidance probabilities to reweight and renormalize this distribution. We will revise Fig. 1 so that the prior arrow points to the policy computation in Eq. (9).

W4 — Why would RAPC address finite-sample noise if it also uses GRPO?

In RAPC, repeated noisy updates cannot drive response odds arbitrarily far from the frozen reference. RAPC uses the same finite-sample GRPO loss and remains subject to sampling noise. At every checkpoint, the policy is obtained by applying a bounded correction to the same frozen reference. Proposition 1 therefore bounds the total shift in next-token odds relative to that reference throughout training. For any two fixed responses to the same prompt, the resulting bound on their sequence-level log-odds shift depends only on β\beta and the lengths of their suffixes after divergence, regardless of how many noisy updates have occurred.

W5 — Can DseqD_{\rm seq} support a claim of proximity to the Gibbs optimum given its sensitivity to length and individual tokens?

Figure 3 shows that full-model GRPO substantially violates a necessary condition for Gibbs optimality. For equally rewarded responses to the same prompt, the Gibbs policy preserves their probability ratio under πref\pi_{\rm ref}. On the evaluated pairs, full-model GRPO substantially distorts these ratios. RAPC reduces the mean DseqD_{\rm seq} from 45.31 to 10.45, showing that it better preserves the relative odds of equally rewarded responses. The frozen reference also satisfies this necessary condition and has Dseq=0D_{\rm seq}=0. To assess overall proximity to the optimum, we therefore report the objective gap separately in Table R2.

The score can be sensitive to suffix length and individual tokens. After the shared prefix cancels, DseqD_{\rm seq} is the absolute difference between two sums of token-level log-ratios. Longer suffixes contribute more terms, and a few large terms can dominate the score on off-policy candidate pairs. We therefore interpret Figure 3 as a comparison of within-reward odds distortion on the evaluated pairs.

To compare objective gaps directly, we report the estimated sequence-level KL to π⋆\pi^\star in Table R2 Under Eq. (1), the objective gap J(π⋆)−J(π)J(\pi^\star)-J(\pi) equals β\beta times the prompt-averaged sequence-level KL. RAPC has the smallest estimated KL among the three trained policies 4.793, compared with 8.679 for full-model GRPO and 7.710 for entropy-regularized GRPO.

Table R2. Estimated KL divergence from the evaluated policies to the Gibbs optimum.

MethodSequence-level KL ↓Token-level KL ↓
GRPO8.6790.578
GRPO + entropy regularization7.7100.554
RAPC4.7930.497

Thank you for the detailed responses and revisions. The methodology and experimental settings are now clear to me. The terminal-loss ablation, residual baseline, and additional evaluations strengthen the empirical case. The added KL evaluation also addresses my concern about relying on DseqD_{\rm seq} and DtokD_{\rm tok} alone to assess proximity to the optimum. I will raise my score.

I see an interesting connection to Section 4.1 of Your Language Model Is Secretly a Q-Function and Section 2.3 of Trajectory Bellman Residual Minimization. These works interpret policy logits as Q-values, providing a value-based perspective on language-model training. RAPC admits a similar interpretation through the mapping

Qϕ(s,a)=βlog⁡[1+(e1/β−1)pϕ(s,a)].Q_\phi(s,a)=\beta\log\left[1+(e^{1/\beta}-1)p_\phi(s,a)\right].

Its policy then satisfies πϕ(a∣s)∝πref(a∣s)exp⁡(Qϕ(s,a)/β)\pi_\phi(a\mid s)\propto\pi_{\rm ref}(a\mid s)\exp(Q_\phi(s,a)/\beta). Thus, training pϕp_\phi can equivalently be viewed as training a bounded Q-function parameterization whose induced soft policy is used for generation. This gives RAPC a value-based interpretation.

My remaining concern is whether intermediate pϕp_\phi predictions accurately estimate reference-policy continuation success. The new theorem establishes this under idealized assumptions, and the ablation demonstrates the terminal loss’s usefulness. However, intermediate predictions are trained through GRPO rather than directly against outcomes of reference-policy continuations from (s,a)(s,a). The practical objective therefore does not explicitly penalize errors in

\Pr_{\pi_{\rm ref}}(\text{successful continuation}\mid sa).$$ Better policy performance and successful PRM guidance do not establish the accuracy of these probability estimates. Comparing $p_\phi$ with success frequencies from reference-policy rollouts would help validate its intended interpretation.

Thank you for highlighting the connection to prior work on QQ-function parameterizations, which we will incorporate into our discussion of RAPC.

Theorem D.8 in Appendix D.5 establishes when the joint population objective recovers the Gibbs policy and identifies the reference-policy continuation probabilities at prefixes covered by terminal supervision. To assess the accuracy of the trained estimator directly, we compare pϕ(s,a)p_\phi(s,a) with success frequencies from frozen-reference rollouts. We compare against Q♯Q^\sharp because it also estimates the probability of success when continuing from (s,a)(s,a) under the reference policy.

Experimental setup. We evaluated both methods on the same 1,500 prefix–action pairs across 138 problems. For each pair, we fixed the prefix and candidate token, sampled 64 continuations from the frozen reference policy, and scored each continuation with the verifier. We computed Pearson correlation, RMSE, and MAE against the empirical success frequencies, and Brier scores against the binary verifier labels.

Table R8. Accuracy of continuation success predictions against frozen-reference rollouts.

MetricRAPCQ♯Q^\sharp
Pearson rr ↑0.92210.6839
RMSE ↓0.09460.2028
MAE ↓0.07050.1060
Brier score ↓0.07560.1078

RAPC predicts reference-policy continuation success more accurately than Q♯Q^\sharp on the evaluated pairs. As shown in Table R8, RAPC achieves a 29.9% lower Brier score. RAPC also has lower prediction errors on the 476 pairs with nonempty response prefixes.

Reviewer_Z3YuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper characterizes the Gibbs optimum of KL-regularized RLVR through reference-policy continuation success probabilities. It proposes Reference-Anchored Policy Correction (RAPC), combining a frozen language model with a bounded correction trained using GRPO and terminal outcome supervision. Experiments report improved reasoning scores and reduced deviations from the target structure.

Critical strengths

The prefix-level formulation clearly connects KL-regularized RL with reference-guided decoding. Comparisons include relevant baselines, which are entropy-regularized GRPO and Q♯Q\sharp. The 4B/7B models and standard reasoning benchmarks constitute meaningful experiments.

Critical weaknesses
  • The proposed correction does not establish the claimed Gibbs consistency. The true potential satisfies h(s)=∑aπref(a∣s)h(sa)h(s)=\sum_a\pi_{\mathrm{ref}}(a\mid s)h(sa), allowing token-level corrections to telescope into the sequence-level optimum. Eqs. (9)–(14) do not enforce this identity: terminal BCE anchors endpoint predictions, while Proposition 1 only bounds next-token odds. Consequently, calibrated reference-success estimates and preservation of relative probabilities among equally rewarded sequences do not follow from the architecture. Figure 3 does not close this gap: the unchanged reference itself has Dseq=0D_{\mathrm{seq}}=0, despite generally being suboptimal. Direct objective-gap or reward–KL evidence is needed to substantiate the central optimality claim.

  • The experiments do not isolate the mechanism or establish a robust practical advantage. RAPC adds a 3B estimator at inference (Appendix A.1), but Table 1 lacks a matched inference-budget comparison and an equally sized generic residual-correction control. The ablations also omit the terminal anchor. Average improvements over GRPO are 2.29 points for Qwen2.5 and only 0.79 for Qwen3; Qwen3’s AIME24 score decreases from 50.00 to 43.33. These results suggest potential benefit, but do not establish that Gibbs-specific modeling produces reliable gains beyond the architectural and regularization changes.

Questions for authors

Can the authors provide comparisons under matched-budget controls?

3 comments

Thanks for your review. Here are the replies.

W1a — What guarantees do Eqs. (9)–(14) provide for intermediate success-probability estimates and sequence-level Gibbs consistency?

We have added a convergence theorem and its proof to the appendix. It shows when approaching the joint population optimum yields pϕ(st,yt+1)→pref(st+1)p_\phi(s_t,y_{t+1})\to p_{\rm ref}(s_{t+1}) for reference-supported actions at prefixes covered by terminal supervision.

First, adding terminal BCE preserves the Gibbs optimum. Substituting the true continuation probability pref(st+1)p_{\rm ref}(s_{t+1}) for pϕ(st,yt+1)p_\phi(s_t,y_{t+1}) in Eq. (9) gives the Gibbs policy π⋆\pi^\star, which maximizes the objective in Eq. (1). For a supervised termination action, the response ends immediately, so the true continuation probability equals the verifier’s binary label. The same choice therefore gives zero terminal BCE. Under our approximation assumption, the estimator can approach both optimal values simultaneously, so the joint loss has the same infimum as the policy objective alone, −J(π⋆)-\mathcal J(\pi^\star).

Second, terminal BCE fixes the scale of the correction. Along a sequence approaching the joint infimum, both the policy objective gap and the terminal BCE tend to zero. Comparing Eq. (9) with the Gibbs transition shows that policy convergence determines the correction factors up to a common scale Zϕ(st)/h⋆(st)Z_\phi(s_t)/h^\star(s_t). At a prefix where a supported termination action receives persistent supervision, the BCE drives the prediction for that action to its verifier label. This forces the common scale to converge to 11, so the predictions for all reference-supported actions at that prefix converge to their true continuation probabilities.

Moreover, adding an explicit intermediate consistency loss reduced performance in our ablation. As shown in Table R1, both variants start at 64.6. By step 200, standard RAPC reaches 78.4, compared with 67.8 for the variant with the additional loss. Although intermediate consistency can theoretically provide extra supervision, it relies on success probability estimates at many prefixes. Noise in these estimates early in training may weaken that supervision.

Table R1. Effect of intermediate consistency supervision on RAPC performance during training.

Training variantStep 020406080100120140160180200
RAPC (terminal BCE)64.665.268.469.671.273.075.478.278.678.878.4
RAPC (terminal BCE + intermediate consistency)64.665.666.066.866.666.266.467.068.467.267.8

W1b — Does Figure 3 show that RAPC is closer to the Gibbs optimum, given that the frozen reference also has Dseq=0D_{\rm seq}=0?

Figure 3 tests a necessary condition for Gibbs optimality. Under π⋆\pi^\star, equally rewarded responses to the same prompt receive the same reweighting factor, so their probability ratios match those under πref\pi_{\rm ref}. Full-model GRPO substantially violates this condition on the evaluated pairs, while RAPC shows less distortion. Satisfying this condition alone is insufficient for Gibbs optimality. For example, the frozen reference has Dseq=0D_{\rm seq}=0 and is generally suboptimal under Eq. (1).

We therefore report the sequence-level KL to π⋆\pi^\star in Table R2 to compare objective gaps directly. Under Eq. (1), the objective gap J(π⋆)−J(π)J(\pi^\star)-J(\pi) equals β\beta times the prompt-averaged sequence-level KL. RAPC has the smallest estimated KL among the three trained policies (4.793), compared with 8.679 for full-model GRPO and 7.710 for entropy-regularized GRPO.

Table R2. Estimated KL divergence from the evaluated policies to the Gibbs optimum.

MethodSequence-level KL ↓Token-level KL ↓
GRPO8.6790.578
GRPO + entropy regularization7.7100.554
RAPC4.7930.497

W2a / Q1 — How does RAPC perform under a matched inference budget after accounting for its 3B estimator?

In Table 1 of the original submission, RAPC and Q♯ use the same training and inference compute budgets, including the cost of running RAPC’s 3B estimator during decoding. On Qwen2.5-Math-7B, RAPC scores 55.35% versus 45.84% for Q♯, a gain of 9.51 percentage points. On Qwen3-4B, RAPC scores 55.84% versus 50.14%, a gain of 5.70 percentage points.

Table R3 adds a generic residual control. We unfreeze the 7B backbone and train it jointly with a residual head that directly learns additive corrections to the token logits. The control uses the same inference budget as RAPC and requires more training compute. It scores 53.91% on average, 0.85 percentage points above full-model GRPO. RAPC scores 55.35%, 1.44 percentage points above the residual control.

Table R3. Benchmark performance of RAPC and the unfrozen residual control at matched inference cost.

MethodGSM8KMATH500OlympiadBenchSuperGPQA500AMC23AIME24AIME25AIME26Avg.
RAPC90.5279.8041.0424.8090.0050.0033.3333.3355.35
Unfrozen backbone + residual head88.7079.0040.7422.0087.5050.0036.6726.6753.91

W2b — How much does the terminal anchor contribute to RAPC’s benchmark performance?

The terminal BCE improves RAPC’s average score by 3.92 percentage points at the default weight. To measure its contribution, we set λanchor=0\lambda_{\rm anchor}=0 in Eq. (14), which removes the terminal BCE in Eq. (13). As shown in Table R4, this lowers the average across the eight benchmarks from 55.35% to 51.43%. The default weight of 0.050.05 improves all eight benchmarks over the zero-anchor setting and achieves the highest average among the four weights evaluated.

Table R4. Effect of the terminal BCE weight on RAPC’s benchmark performance.

λanchor\lambda_{\rm anchor}GSM8KMATH500OlympiadBenchSuperGPQA500AMC23AIME24AIME25AIME26Avg. (%)
086.7376.8039.1124.6087.5040.0026.6730.0051.43
0.0189.6978.2039.2624.6087.5046.6730.0030.0053.24
0.05 (default)90.5279.8041.0424.8090.0050.0033.3333.3355.35
0.1088.1779.6039.7024.4092.5046.6730.0033.3354.30

W2c —RAPC’s gains over GRPO vary across backbones, and its Qwen3-4B score drops on AIME24.

Thank you for highlighting the variation across backbones and the AIME24 drop on Qwen3-4B. We re-evaluated RAPC and full GRPO on Qwen3-4B using three matched evaluation seeds. In Table R5, RAPC improves the Table 1 average on all three seeds, with a mean paired difference of +1.28±0.48+1.28\pm0.48 percentage points. On AIME24, RAPC scores higher on two seeds and lower on one, with a mean paired difference of +1.11±6.94+1.11\pm6.94 percentage points. The drop reported in the original submission recurs on one seed, and the mean AIME24 gain is small relative to the variation across evaluation seeds.

Eval. seedGRPO Avg. (%)RAPC Avg. (%)Δ Avg. (pp)GRPO AIME24 Pass@32 (%)RAPC AIME24 Pass@32 (%)Δ AIME24 (pp)
155.3156.93+1.6243.3350.00+6.67
255.4956.98+1.4940.0043.33+3.33
355.1155.84+0.7350.0043.33−6.67
Mean ± SD55.30 ± 0.1956.58 ± 0.64+1.28 ± 0.4844.44 ± 5.0945.55 ± 3.85+1.11 ± 6.94

Thanks to the authors for the additional experiments and clarification. My main concerns have been addressed. I have updated my score accordingly.

Comments on the paper

2 comments

Dear Area Chair and reviewers,

Thank you for your careful reading and constructive feedback. We summarize the strengths highlighted in the reviews and the main findings and clarifications from our rebuttal.

Strengths recognized by reviewers

  • Clear motivation and a concrete target for RLVR (Z3Yu, UGQ7, xWeE). Reviewers highlighted the question of whether practical RLVR recovers the optimum of its KL-regularized objective. The Gibbs characterization gives a concrete target against which to evaluate trained policies.
  • Interpretable prefix-level guidance (Z3Yu, xWeE). Reviewers appreciated the connection between the sequence-level optimum and token-level decoding. For binary rewards, the ideal correction is determined by the reference policy’s continuation success probability.
  • A small trainable estimator and relevant baselines (UGQ7, Z3Yu). UGQ7 highlighted that a small estimator on top of a frozen reference can outperform full-model GRPO. Z3Yu noted the comparisons with entropy-regularized GRPO and Q♯Q^\sharp across two backbones and standard reasoning benchmarks.

Main concerns and our responses

  • Gibbs consistency and continuation probabilities (Z3Yu, xWeE). Z3Yu asked whether RAPC enforces the intermediate recursion needed for sequence-level Gibbs consistency, and xWeE asked whether its training loss recovers the true reference-policy continuation probabilities. We added Theorem D.8 and its proof in Appendix D.5 to establish when the population objective with terminal BCE recovers the Gibbs policy and identifies these probabilities at prefixes with terminal supervision.
  • Proximity to the Gibbs optimum (Z3Yu, UGQ7). Both reviewers asked whether Figure 3 establishes proximity to the Gibbs optimum, given that the frozen reference itself has Dseq=0D_{\rm seq}=0. We added Table R2 to compare objective gaps using estimated sequence-level KL to the Gibbs target under a common reference support. RAPC has the lowest estimate at 4.793, compared with 8.679 for full-model GRPO and 7.710 for entropy-regularized GRPO.
  • Finite-sample noise (UGQ7, xWeE). Both reviewers asked how RAPC limits policy drift while using the same sample-based GRPO updates. We show that the effect of repeated sampling errors on response odds cannot accumulate without bound. At every training step, RAPC applies a bounded correction to the same frozen reference, so the bound on the relative log-odds shift between two fixed responses to the same prompt is independent of the number of training updates.
  • Additional controls, ablations, and evaluation (Z3Yu, xWeE). The reviewers asked for a generic residual baseline, an ablation of the terminal anchor, and evidence across backbones and evaluation seeds. We added these comparisons. RAPC scores 1.44 percentage points above the jointly trained residual baseline, and removing terminal BCE lowers its average by 3.92 points. On Qwen3.5, RAPC improves over full-model GRPO by 1.89 points. On Qwen3-4B, it improves the overall average on all three matched evaluation seeds.

The individual replies give the experimental details and address the remaining questions about notation, and evaluation settings.

Best regards, Authors of Submission 17775

Dear Area Chair and Reviewers,

Thank you for your time and feedback on Submission 17775. As the author discussion draws to a close, we wanted to follow up on our revised manuscript and point-by-point responses. The rebuttal includes a new theorem on when the joint objective recovers continuation probabilities, a sequence-level KL evaluation, and the requested residual control and terminal-loss ablation.

We would appreciate any further feedback on the revision and would be happy to address any remaining questions.

Best regards, The Authors of Submission 17775

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.