acceptodds
Under review as a conference paper at ICLR 2027

Value Gradient Hypothesis of RL for LLMs

Abstract

Reinforcement learning substantially improves pretrained language models, but it remains understudied why critic-free methods such as GRPO and critic-free PPO variants work as well as they do, and when they should provide the largest gains. We develop a value-gradient perspective of critic-free RL for LLM post-training. First, under a differentiable rollout and additive-noise parameterization, we identify when the expected actor update admits a pathwise representation: the corresponding return costates have conditional expectation equal to the value gradient. Second, for discrete transformer policies, we characterize empirical adjoints through the layer-position computation graph and bound their discrepancy from a specified relaxed value signal by local-signal and transition residuals. Policy entropy controls one relaxation component of this bound, not the entire discrepancy. These results motivate a hypothesis decomposing RL impact into value-gradient signal and reachable reward headroom, yielding a criterion for when RL may be most effective along a pretraining trajectory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL, LLM, Value Function

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nhrLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why critic-free reinforcement learning methods, such as GRPO, can effectively improve LLMs without explicitly learning a value function. The authors propose a value-gradient perspective, showing that under differentiable rollouts, the expected costate propagated by backpropagation through time corresponds to the value gradient. For discrete Transformer policies, they analyze gradient propagation through attention and derive a conditional bound on the discrepancy between surrogate gradients and relaxed value gradients. Building on this analysis, they propose an RL-impact hypothesis that predicts RL gains based on two factors: usable value-gradient signal and remaining reward headroom. Experiments on OLMo-2 1B checkpoints using a controlled label-copying task report a moderate correlation between the proposed predictor and observed RL gains, providing preliminary empirical support for the hypothesis.

Critical strengths

Meaningful research question and a novel theoretical perspective. The paper addresses an important question: why critic-free RL methods such as GRPO can effectively improve LLMs without explicitly learning a value function. The proposed value-gradient perspective offers an interesting explanation beyond existing analyses based on group-relative advantages or implicit value estimation. By connecting actor gradient propagation to value gradients, the paper provides a potentially useful theoretical framework for understanding critic-free RL.

Critical weaknesses

Insufficient empirical validation. The experiments are limited to a simple label-copying task using OLMo-2 1B checkpoints, without evaluation on realistic reasoning tasks such as mathematics or coding. Moreover, the experiments do not directly verify the theoretical approximation bound, and the reported correlations provide only preliminary support for the RL-impact hypothesis. Stronger empirical evidence is needed to establish the generality and practical relevance of the proposed framewor

Questions for authors
  1. Serious concerns regarding the completeness and reliability of the experimental description. In Appendix A.13, the statements "The experimental description does not specify which such completion-dependent training reward or direct-gradient rule was used" and "This implementation detail must be resolved before the reported runs are identified with the reward-driven GRPO mechanism analyzed in this paper" show clear signs of an unfinished manuscript. Even under the most charitable interpretation, assuming that these issues arise from communication problems among co-authors rather than AI-automated research, the authors must explain this issue in the rebuttal. Otherwise, I will lower my rating to 1 (Clear Rejection).
  2. Limited experimental evidence for the proposed hypothesis. The empirical validation is primarily conducted on a simple label-copying task using OLMo-2 1B, which is substantially different from the long-horizon reasoning tasks that motivate this work. Moreover, the experiments do not directly establish whether the proposed value-gradient approximation holds in practical discrete-token RL training. Can the authors provide stronger evidence that the proposed RL-impact predictor captures meaningful RL gains beyond simple baselines such as initial performance or reward variance, ideally on realistic reasoning benchmarks? Such evidence would substantially strengthen the practical relevance of the proposed hypothesis.
1 comment

Thank you for the review. We are glad that you find the research question meaningful and the value-gradient perspective novel. Please see the answers below.

Q1: Appendix A.13

The sentences you quote come from the scope of the paper. A.13 is titled "Theoretical scope of the experimental diagnostics". Its purpose is to state, for each quantity measured in Section 6, which conclusions the theory supports and under which specification.

The theory analyzes the reward-driven GRPO update. This update exists only for a completion-dependent reward: if every completion of a prompt receives the same reward, all group-normalized advantages are zero and there is no reward-driven update (A.13). The score reported in Section 6, pθ(y⋆∣q)p_\theta(y^\star \mid q), is a property of the prompt rather than of a single completion. A.13 therefore records that the analysis applies to a completion-dependent reward whose expectation is this score, such as sampled-label correctness 1{o=y⋆}\mathbb{1}\{o = y^\star\}, and that a direct gradient of the score would be a different update, outside the analysis.

These are conditions on the scope of the theory, not uncertainty about the experiments. We agree, however, that the wording, in particular "the experimental description does not specify", reads as if the setup itself were undetermined. We will change it as follows:

  • state the training reward and update rule explicitly in Section 6;
  • rewrite A.13 as a scope remark that explains why the reward must be completion-dependent and what the theory then guarantees;
  • add the full implementation details to the appendix.

Q2: Stronger empirical evidence

  • Baselines. The paper does not claim out-of-sample superiority over competence-based predictors. A.13 states that the correlations are descriptive, and that a predictive claim requires separate calibration and evaluation checkpoints and comparison with competence-only baselines, such as the initial performance and reward variance you mention. For a binary reward, the variance is Rbefore(1−Rbefore)R_{\text{before}}(1 - R_{\text{before}}). Section 6 reports that the combined predictor correlates more strongly with the post-RL reward than current reward alone. We will move this scope statement into Section 6.
  • Why a controlled task. The predictor compares the actor costate with a gradient GtG_t. In the label task, Gt=∂pθ(y⋆∣q)/∂htG_t = \partial p_\theta(y^\star \mid q)/\partial h_t is the exact value gradient, so Σ\Sigma and Λm\Lambda_m are measured without approximation. For long reasoning completions with a binary reward, no closed form of GtG_t is available, and an estimator would add its own error to the comparison.
  • Realistic benchmarks. We agree that reasoning benchmarks are the main next step, and we will state this scope explicitly in the limitations.

Strengthened theory

We will also add an exact result that strengthens the central claim. Let hth_t be the complete hidden state and V~t\tilde V_t the value of the effective reward of Corollary 1. Then the actor backward pass of discrete GRPO satisfies E[λ^t∣ht]=∂hV~t(ht)\mathbb{E}[\hat\lambda_t \mid h_t] = \partial_h \tilde V_t(h_t) exactly, with no relaxation. The claim "critic-free does not mean value-free" therefore holds exactly for the discrete update that GRPO actually performs.

Thank you for your thoughtful feedback.

Reviewer_Gi1gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the relationship between actor updates and value gradients in critic-free RL. It proposes an influence score, which incorporates both gradient signal quality and reward headroom to predict the gains from reinforcement learning across various pre-training checkpoints. Experiments conducted on OLMo-2 1B model provide preliminary evidence for this hypothesis.

Critical strengths

1.This paper explores an interesting problem, i.e. why critic-free RL methods work, and under what conditions, it yields the significant benefits.

  1. A coherent analysis framework including actor updates, value gradients, and reward headroom is provided.

  2. Preliminary experiments show correlations between the proposed impact score and RL gains, indicating potential for checkpoint selection.

Critical weaknesses
  1. The theoretical contribution beyond existing mehtods needs to be explained or analysed more in depth. Specifically, the analysis is mainly based on existing score-function, pathwise-gradient, and adjoint methods. Beyond applying these well-established results to critic-free RL actor updates, the main theoretical contribution remains unclear.
  2. The theoretical guarantees with respect to the approximation theorem are inadequate. As the approximation theorem provides a conditional error bound, but the required local-signal and transition residuals are not shown to be sufficiently small in practical GRPO settings. Aligning hidden states does not necessarily guarantee improvements in the parameter space.
  3. The description of the outcome-based rewards or update rules employed by GRPO is not sufficiently validated in the experiments, making it difficult to establish a correspondence between the reported experiment results and the objectives under analysis. Furthermore, the experiments are conducted only on the OLMo-2 1B model and a single controlled label-replication task, as a result, the evidence provided to support the generalization capability of the proposed predictor seems insufficient. 4.(Small issues) The reference list contains some formatting inconsistencies and malformed entries.
Questions for authors

Please refer to weaknesses 1-3, more specifically,

  1. The discrepancy between empirical adjoints and relaxed value gradients is bounded by Theorem 1 under residual conditions. Can the authors clarify under what conditions these residuals become very small in practical GRPO? Furthermore, what theoretical guarantees does this result provide, compared to standard adjoint perturbation analysis?

2.The proposed scoring metric combines hidden-state gradient alignment with distributional reward headroom, yet it does not provide a direct guarantee of performance improvement in parameter space. Beyond empirical assumptions, can the authors provide a more rigorous theoretical basis to explain why this metric is capable of predicting RL gains under a limited budget?

2 comments

Thank you for the review. W1: Theoretical contribution

The paper uses established tools, but its results are not restatements of them:

  • Corollary 1 and the counterexample (A.5): the local GRPO gradient is the policy gradient of an effective reward that includes group normalization and the 1/Ti1/T_i weighting, and normalization is provably not a rescaling. Neither result follows from Fairbank & Alonso (2012), which treats neither group normalization nor transformers.
  • The transformer adjoint: the layer–position recursion (eq. 15) and the complete-state recursion including the KV cache (eq. 16).
  • The error decomposition (Theorem 1, A.9), the usable-signal bound (eq. 24), and the variational characterization of headroom (A.12).

We will also add an exact discrete result. Let hth_t be the complete hidden state and V~t\tilde V_t the value of the effective reward of Corollary 1. Differentiating V~t(h)=∑oπ(o∣h) Q~t(h,o)\tilde V_t(h) = \sum_o \pi(o \mid h)\,\tilde Q_t(h, o) gives the recursion of eq. 16 with Q~t\tilde Q_t in place of the advantage, so E[λ^t∣ht]=∂hV~t(ht)\mathbb{E}[\hat\lambda_t \mid h_t] = \partial_h \tilde V_t(h_t). The actor backward pass of discrete GRPO is therefore exactly a value-gradient estimate, without any relaxation.

W2 / Q1: The approximation theorem

The paper does not claim that the residuals are small in practical GRPO. Section 4.2 says the approximation holds "when the full right-hand side is small, not from low entropy alone", and A.9 calls Theorem 1 "a sufficient error decomposition, not a proof that every pretrained transformer is close to the relaxed regime". A conditional bound is the correct form for this statement. Moreover, the central claim does not depend on these residuals: the exact identity above has none. Theorem 1 is needed only to compare the discrete value gradient with a relaxed, pathwise one.

The residuals are small under three explicit conditions:

  1. Local signal: ∥bk−dk∥\|b_k - d_k\| is small, that is, the advantage-weighted score gradient matches the reward derivative.
  2. Transition: the through-action part of ∥Jk−Fk∥\|J_k - F_k\| is at most C0min⁡{1,Hk}/τrelC_0 \min\{1, H_k\}/\tau_{\mathrm{rel}} (Proposition 2), so low entropy controls it; the evaluation-point mismatch must also be small.
  3. Jacobian products: if ∥Jj∥≤M\|J_j\| \le M and ϵk≤ϵˉ\epsilon_k \le \bar\epsilon, the bound is at most ϵˉ/(1−γM)\bar\epsilon/(1 - \gamma M) when γM<1\gamma M < 1 (A.9).

Theorem 1 uses a standard adjoint perturbation argument, and we do not claim a new technique. What it adds is the instantiation: it compares a discrete score-function surrogate with a relaxed pathwise costate, splits the residual into the part entropy controls and the parts it does not (Remark 2), and keeps random, mutually dependent Jacobians inside the conditional expectation without an independence assumption.

W2 / Q2: Parameter space and finite budgets

The paper states and proves this limitation itself. Section 5 says that "hidden-state alignment must additionally be mapped through parameter Jacobians and optimizer geometry to control an actual update", and eq. 28 is "a finite-budget prediction to be tested, not a consequence of the signal bound alone". The criticism targets a guarantee the paper does not claim. The rigorous basis for the two factors is:

  • Rate (A.11): for an LFL_F-smooth objective, E[F(θ+ηPg)−F(θ)]≥η ∂θF⊤Pgˉ−LFη22(∥Pgˉ∥2+E∥P(g−gˉ)∥2)\mathbb{E}[F(\theta + \eta P g) - F(\theta)] \ge \eta\, \partial_\theta F^\top P \bar g - \tfrac{L_F \eta^2}{2}\big(\|P\bar g\|^2 + \mathbb{E}\|P(g - \bar g)\|^2\big), so the first-order gain is the alignment of the actual update with the true gradient.
  • Ceiling (A.12): Hα=sup⁡ν{Eν[R]−α−1KL(ν ∥ μ)}−Eμ[R]H_\alpha = \sup_\nu \{\mathbb{E}_\nu[R] - \alpha^{-1}\mathrm{KL}(\nu \,\|\, \mu)\} - \mathbb{E}_\mu[R] is the largest gain any KL-regularized reweighting of the current policy can reach.

A finite-budget guarantee would require a convergence analysis of GRPO for transformers, which no method has. This is why eq. 28 is stated as a hypothesis and tested.

W3 and W4

The theory applies to the reward-driven GRPO update for a completion-dependent reward whose expectation is the reported score pθ(y⋆∣q)p_\theta(y^\star \mid q), such as sampled-label correctness. A.13 states the scope of the diagnostics under this specification. We will rewrite it as a scope remark and state the training reward and implementation details explicitly in Section 6. The single model and task are stated as a limitation in Section 6. We will fix the reference formatting.

Thank you for the detailed response. However, my concern about the theoretical novelty, the practical implication of the conditional approximation bounds still remains. In this circumstance, I prefer to keep my original rating.

Reviewer_8mx2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the "value-gradient hypothesis": critic-free PPO/GRPO updates are not necessarily value-free, because the hidden-state adjoints (costates) propagated through the actor during backpropagation can be interpreted as samples of value gradients in expectation.

The theoretical analysis consists of three main components:

For differentiable rollouts under shift/additive-noise policies, the paper uses the BPTT costate identity from Fairbank and Alonso and the (\mathrm{SF}=\mathrm{PD}) lemma to derive a pathwise representation of local actor updates. It also establishes an identity showing that the local GRPO gradient is equivalent to the policy gradient of an effective reward with the other group members held fixed (Corollary 1).

For discrete transformers, the paper defines a layer-position recursion for empirical costates. It then uses an entropy-based bound on the softmax Jacobian (Proposition 2) and a residual recursion (Theorem 1) to bound the discrepancy between empirical costates and a relaxed value gradient in terms of local signal and transition residuals.

The paper proposes a multiplicative hypothesis for RL improvement, approximately (\kappa S_m(\theta)H_\alpha(\theta)), where (S_m=\Sigma-\Lambda_m) represents the "usable value-gradient signal" and (H_\alpha) measures the remaining reward improvement potential under KL regularization.

Empirically, the paper evaluates 20 pretrained checkpoints from OLMo-2 1B on a label-copying task, using (R=p_\theta(y^\star\mid q)) as a differentiable score. It reports a Spearman correlation of approximately 0.60 between (P_{\mathrm{gap}}) and RL improvement. When combined with (R_{\mathrm{before}}), the reported Spearman correlation with (R_{\mathrm{after}}) increases to approximately 0.73, although Figure 5 reports 0.70.

Critical strengths
  1. An interesting research question and a potentially useful perspective on critic-free RL. The paper reframes the question of "why critic-free methods work" as whether the actor's own backpropagation already contains value-gradient information. It further proposes using this information to predict which checkpoints along a pretraining trajectory are most suitable for subsequent RL training.

I consider this a potentially valuable research direction. If developed with a stronger theoretical connection to discrete language-model policies and more convincing empirical validation, it could provide useful insights into when and why RL improves pretrained models. This is the main reason I do not dismiss the paper entirely.

  1. A correct and self-contained effective-reward identity for GRPO.

Corollary 1 and the argument in Appendix A.5 are mathematically sound and internally consistent. Specifically, the local GRPO actor gradient is exactly equivalent to the policy gradient of an effective reward (\tilde{R}_0), with the other sampled group members held fixed.

The three-action counterexample also clearly demonstrates that group normalization cannot generally be reduced to a scalar rescaling of the gradient.

These observations complement the U-statistic perspective of Zhou et al. (2026) and are useful for understanding the objective that GRPO actually optimizes.

Critical weaknesses
  1. The theoretical results do not establish the paper's central claim, while the evidence shows in paper is largely known or elementary. The costate–value-gradient identity (Proposition 1, Equation 9) and the (SF=PD\mathrm{SF}=\mathrm{PD}) identity (Lemma 1) in Sections 2–3 are established results from Fairbank and Alonso (2012). The paper also omits Heess et al. (2015), Stochastic Value Gradients, a foundational reference for value-gradient methods in reinforcement learning. More importantly, the bridge developed in Section 3 requires shift/additive-noise policies of the form πθ(a∣s)=ν(a−aˉθ(s)).\pi_\theta(a\mid s)=\nu(a-\bar a_\theta(s)).

Categorical policies used by language models do not satisfy this structural assumption. The paper itself acknowledges that the construction "does not equate the two samplewise backward passes." Consequently, the central claim that critic-free RL for LLMs is not value-free has not actually been established for the discrete policies of interest.

Theorem 1 in Section 4 is obtained by unrolling the recursion ek=ηk+γJk⊤ek+1e_k=\eta_k+\gamma J_k^\top e_{k+1} and taking norms. Without any assumptions on, or estimates of, the residual terms (∣bk−dk∣)and(∣Jk−Fk∣|b_k-d_k|) and (|J_k-F_k|), the result essentially states that the discrepancy is small if the residuals are small. This provides limited explanatory or predictive value.

Similarly, Proposition 2 establishes ∣diag⁡(p)−pp⊤∣≤1−∑vpv2≤H,|\operatorname{diag}(p)-pp^\top| \leq 1-\sum_v p_v^2 \leq H,

while Proposition 3 gives the attention-Jacobian rank bound rank⁡(J)≤min⁡dmodel,nheadsdhead.\operatorname{rank}(J) \leq \min{d_{\mathrm{model}}, n_{\mathrm{heads}}d_{\mathrm{head}}}.

Both are elementary consequences of standard matrix properties rather than substantial new theoretical results.

More fundamentally, an exact value-gradient identity appears to be available without introducing the proposed relaxation. If the complete hidden state is used as the state representation for the discrete process, the score-function identity directly gives ∂htVt(ht),\partial_{h_t}V_t(h_t), with (R~0\tilde{R}_0) used as the effective reward in the GRPO case.

Under the interpretation in the paper, Theorem 1 merely bounds the difference between the value gradient of the original discrete process and that of an artificially relaxed process. Such a discrepancy is not intrinsic to GRPO. So, "entropy controls one relaxation component" loses much of its relevance to the paper's central claim. Appendix A.7 comes close to recognizing this exact identity but does not pursue its implications.

  1. The experiments do not test the proposed theory, and even the training algorithm used is not sufficiently specified, as acknowledged in the authors' own Appendix A.13. Appendix A.13 explicitly notes that if the training reward is literally defined as R=pθ(y⋆∣q)R=p_\theta(y^\star\mid q), then all completions sampled from the same prompt receive the same reward. Their GRPO advantages are therefore identically zero, and no reward-driven GRPO update can occur. The appendix further states: "The experimental description does not specify which such completion-dependent training reward or direct-gradient rule was used."

This is a fundamental ambiguity about the experimental method, not merely a missing implementation detail.

Appendix A.13 also acknowledges several additional limitations that directly undermine the interpretations presented in the main text:

The reported (\Lambda_m) is a sample-level statistic. By Jensen's inequality, it provides only an upper bound on the corresponding conditional quantity and "can be large even when the conditional mean is exactly correct." Thus, it does not necessarily measure the proposed value-gradient approximation error.

Under the stated definition, the within-prompt improvement potential measured by (H_\alpha) is identically zero. Any nonzero values shown in the figures must therefore arise from variation across prompts, but this distinction is not specified. The appendix acknowledges that it "must be specified before" Equation 27 can be given its proposed interpretation.

The "entropy bound" in Figure 2 is a rescaled curve rather than a theoretically established upper bound. As the appendix states, "No fitted constant can be interpreted as a proved uniform upper bound." The affine calibration is fitted on the evaluation data itself. The appendix acknowledges that the observed agreement "is not an out-of-sample prediction." These are substantial qualifications of the paper's central theoretical and empirical claims.

Beyond these issues, the empirical evaluation is extremely limited: it considers a single model family (OLMo-2 1B), one toy task, and only 20 pretrained checkpoints.

And the experiments omit the most basic predictive baselines. For example, the paper does not report $ \operatorname{Spearman} (R_{\mathrm{before}},R_{\mathrm{after}})$ or compare against predictors based solely on the current reward, such as $ R_{\mathrm{before}}(1-R_{\mathrm{before}}).$

For Bernoulli-type rewards, both (H_\alpha) and $ \Sigma= \mathbb{E} \left| \frac{\partial p}{\partial h} \right|^2 $ may be largely determined by the current success probability (p). Consequently, (PgapP_{\mathrm{gap}}) may effectively reduce to a nonlinear transformation of (RbeforeR_{\mathrm{before}}), rather than providing independent information about RL readiness.

Without controlling for this possibility, the reported correlations do not demonstrate that the proposed value-gradient quantities provide meaningful predictive information beyond current model performance. Essential implementation details are also missing, including the values of (\alpha) and (τrel\tau_{\mathrm{rel}}), the learning rate, GRPO group size, generation length, and the specific position at which (GtG_t) is evaluated.

Finally, the bibliography contains only 12 references and omits several directly relevant lines of work, for example stochastic value gradients, Gumbel-softmax and straight-through estimators, normalization analyses related to Dr. GRPO, and existing empirical studies of when and why RL improves pretrained language models.

Questions for authors
  1. What training algorithm and reward were actually used in Section 6? What was the completion-dependent training reward used by GRPO? If the reward was based on the correctness of sampled labels, please provide its exact mathematical definition.

  2. How do Theorem 1 and Proposition 2 remain relevant to GRPO in light of the exact value-gradient identity?

  3. Can the proposed RL-readiness predictor outperform simple reward-based baselines?

Please report the following correlations:

$ \operatorname{Spearman} (R_{\mathrm{before}},R_{\mathrm{after}}), $ and $ \operatorname{Spearman} \left( R_{\mathrm{before}} (1-R_{\mathrm{before}}), \overline{\Delta R} \right). $

Please also report the correlations obtained by using entropy alone and checkpoint index alone as predictors.

In addition, please evaluate the proposed predictor using leave-one-checkpoint-out calibration, rather than fitting and evaluating the calibration on the same checkpoints.

  1. How are (H_\alpha) and the reported correlations defined and computed? In Figure 4, what is the sampling axis used to compute (H_\alpha)? Is the variation measured within each prompt or across different prompts? What value of (\alpha) is used?

Also, Figure 5 reports a correlation of 0.70, while the main text reports 0.73. Which value is correct?

  1. What constitutes the complete hidden state in the theoretical analysis and implementation? Specifically, what variables are included in the "complete state" associated with (\lambda_t)? Is the KV cache included in the backpropagation computation? Furthermore, how is (\epsilon_t) in Figure 2 defined and evaluated across 60 generation positions when the task reward is based on the single-token probability (p(y^\star\mid q))? Please clarify the relationship between the token-position-dependent quantities and the reward used in the experiments.
2 comments

Thank you for the detailed review. We are glad that you found Corollary 1 and the three-action counterexample correct and useful. Your observation about an exact discrete identity is, in our view, the most important point of the review: it makes the central claim of the paper hold exactly for the actual discrete GRPO update. Please see the answers below.

W1: Relation to prior work

The paper does not present Proposition 1 or Lemma 1 as new. Both are attributed to Fairbank & Alonso (2012) in their titles and serve as background. The new results are:

  • Corollary 1 and the counterexample (A.5): the local GRPO gradient is the policy gradient of an effective reward, and group normalization is not a rescaling of the original reward.
  • The transformer adjoint: the layer–position recursion (eq. 15) and the complete-state recursion including the KV cache (eq. 16).
  • The error decomposition (Theorem 1, A.9), which separates the entropy-controlled sampling component from the local-signal and transition residuals.
  • The usable-signal bound (eq. 24) and the variational characterization of headroom (A.12).

We will cite Heess et al. (2015), Gumbel-softmax (Jang et al., 2017; Maddison et al., 2017), straight-through estimators (Bengio et al., 2013), and Dr. GRPO (Liu et al., 2025), and extend the related work with empirical studies of RL on pretrained models.

On categorical policies: Section 3 states that the bridge "does not equate the two samplewise backward passes", and Section 4 treats discrete policies separately. The identity below closes this gap.

W1 / Q2: The exact discrete identity

We agree, and we will add it as a proposition. Let hth_t be the complete state of Section 4 and V~t\tilde V_t the value of the effective reward R~0\tilde R_0 of Corollary 1, with the discrete history fixed. Since ot∼π(⋅∣ht)o_t \sim \pi(\cdot \mid h_t) and ht+1=f(ht,eot)h_{t+1} = f(h_t, e_{o_t}), differentiating V~t(h)=∑oπ(o∣h) Q~t(h,o)\tilde V_t(h) = \sum_o \pi(o \mid h)\,\tilde Q_t(h, o) gives

∂hV~t(h)=E[Q~t(ht,ot) ∂hℓt+Jt⊤∂hV~t+1(ht+1)∣ht=h].\partial_h \tilde V_t(h) = \mathbb{E}\big[\tilde Q_t(h_t, o_t)\,\partial_h \ell_t + J_t^\top \partial_h \tilde V_{t+1}(h_{t+1}) \mid h_t = h\big].

The empirical costate satisfies λ^t=A^t ∂hℓt+Jt⊤λ^t+1\hat\lambda_t = \hat A_t\,\partial_h \ell_t + J_t^\top \hat\lambda_{t+1} (eq. 16 with γ=1\gamma = 1), and E[A^t∣ht,ot]=Q~t\mathbb{E}[\hat A_t \mid h_t, o_t] = \tilde Q_t by Corollary 1. Backward induction gives

E[λ^t∣ht]=∂hV~t(ht).\mathbb{E}[\hat\lambda_t \mid h_t] = \partial_h \tilde V_t(h_t).

The actor backward pass of discrete GRPO is therefore, in conditional expectation, exactly the value gradient of its effective reward. This requires no relaxation and no shift policy, only that the backward pass keeps the cache edges (A.6). It proves the central claim, "critic-free does not mean value-free", directly in the discrete case, and we will make it the main result of Section 4.

Q2: Role of Theorem 1 and Proposition 2

With this identity, the left side of Theorem 1 becomes ∥∂hV~t−Gtrel∥\|\partial_h \tilde V_t - G^{\mathrm{rel}}_t\|, the distance between the discrete value gradient and a pathwise (BPTT) one. We agree that this distance is not intrinsic to GRPO. It is, however, the quantity the predictor of Section 5 depends on: Λm\Lambda_m compares λ^t\hat\lambda_t with a specified gradient GtG_t, in general the pathwise gradient of a differentiable score. Theorem 1 states when this comparison is small, and Proposition 2 isolates the single component that entropy controls (Remark 2). A.9 also gives an explicit sufficient condition: if ∥Jj∥≤M\|J_j\| \le M and ϵk≤ϵˉ\epsilon_k \le \bar\epsilon, the bound is at most ϵˉ∑r(γM)r\bar\epsilon \sum_r (\gamma M)^r.

Propositions 2 and 3 are indeed elementary. Their role is to show what entropy does and does not control, and we will present them as lemmas. We will restructure Section 4 accordingly: the exact identity first, then Theorem 1 as the discrete-versus-pathwise comparison.

Q1: Training algorithm and reward

The training algorithm is GRPO as defined in eq. 1, run from each checkpoint for K∈{10,20,25,30}K \in \{10, 20, 25, 30\} updates (Section 6). The theory applies to the reward-driven GRPO update for a completion-dependent reward whose expectation under the policy is the reported score pθ(y⋆∣q)p_\theta(y^\star \mid q), such as sampled-label correctness 1{o=y⋆}\mathbb{1}\{o = y^\star\}. In the one-step task, Gt=∂pθ(y⋆∣q)/∂htG_t = \partial p_\theta(y^\star \mid q)/\partial h_t is then the exact value gradient of the unnormalized reward, as A.13 notes. A.13 also records that a per-prompt constant reward gives zero advantages, so that case lies outside the analysis.

Appendix A.13 is a scope section, as its title states: it lists which conclusions the theory supports for each quantity in Section 6, and under which specification. Written as general conditions, its sentences read as if the experimental setup itself were undetermined, which was not their purpose. We will state the training reward and update rule explicitly in Section 6, rewrite A.13 as a scope remark, and add the full implementation details to the appendix: α\alpha, learning rate, KL coefficient, group size, generation length, and the layer and position at which GtG_t and λ^t\hat\lambda_t are evaluated.

Q4: Headroom, Λm\Lambda_m, and Figure 5

  • Headroom. For a completion-dependent binary reward, the within-prompt headroom is the Bernoulli expression of A.12, Hα(p)=α−1log⁡(1−p+peα)−pH_\alpha(p) = \alpha^{-1}\log(1 - p + p e^\alpha) - p, which is nonzero for 0<p<10 < p < 1. A.13 distinguishes this within-prompt quantity from variation across prompts, and we will state the sampling axis and the value of α\alpha explicitly in Section 6.
  • Λm\Lambda_m. As A.13 states, the samplewise statistic satisfies Λ~m≥Λm\tilde\Lambda_m \ge \Lambda_m, so Σ−Λ~m≤Sm\Sigma - \tilde\Lambda_m \le S_m. The reported score is therefore a conservative estimate of the usable signal, which is the right direction for a lower-bound diagnostic.
  • Figure 5. Both numbers are correct and refer to different predictors. The left panel plots Rbefore+Affine(Pgap)R_{\text{before}} + \mathrm{Affine}(P_{\text{gap}}) against RafterR_{\text{after}} (Spearman 0.70). The 0.73 in the text is for the scale-free z(Rbefore)+z(Pgap)z(R_{\text{before}}) + z(P_{\text{gap}}) in the right panel. We will state both in the caption.
  • Figure 2. As Section 6 states, the curve is a diagnostic and not a verification of Theorem 1. We will label it so and not call it a bound.

Q3: Simple baselines

The paper does not claim that the predictor outperforms competence-based predictors out of sample. A.13 states that the correlations are descriptive, and that a predictive claim requires separate calibration and evaluation checkpoints and comparison with competence-only baselines. These are exactly the comparisons you list. Section 6 reports only that the combined predictor correlates more strongly with the post-RL reward than current reward or the impact score alone. We will move this scope statement into Section 6.

On whether PgapP_{\text{gap}} reduces to a function of RbeforeR_{\text{before}}: for a binary reward, HαH_\alpha depends only on pp (A.12), but Σ\Sigma does not. Since ∂py⋆/∂z=py⋆(ey⋆−p)\partial p_{y^\star}/\partial z = p_{y^\star}(e_{y^\star} - p), the norm of ∂hpθ(y⋆)\partial_h p_\theta(y^\star) depends on the full output distribution and on the head and representation geometry. Λm\Lambda_m further depends on the costate mismatch. PgapP_{\text{gap}} is therefore not determined by RbeforeR_{\text{before}}.

Scope. The controlled task was chosen because it gives exact derivatives of the reward (Section 6), and the Limitations paragraph states the single model and task. We will extend the bibliography as described in Part 1.

Q5: Complete hidden state. In the theory, the complete state includes the KV cache, with the discrete history fixed (Section 4, A.6). Implementations that detach the cache define a different gradient graph (A.6). We will state which activations are used in the experiments, including the definition of ϵt\epsilon_t over generated positions in Figure 2.

Thank you for your thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.