- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why critic-free reinforcement learning methods, such as GRPO, can effectively improve LLMs without explicitly learning a value function. The authors propose a value-gradient perspective, showing that under differentiable rollouts, the expected costate propagated by backpropagation through time corresponds to the value gradient. For discrete Transformer policies, they analyze gradient propagation through attention and derive a conditional bound on the discrepancy between surrogate gradients and relaxed value gradients. Building on this analysis, they propose an RL-impact hypothesis that predicts RL gains based on two factors: usable value-gradient signal and remaining reward headroom. Experiments on OLMo-2 1B checkpoints using a controlled label-copying task report a moderate correlation between the proposed predictor and observed RL gains, providing preliminary empirical support for the hypothesis.
Critical strengths
Meaningful research question and a novel theoretical perspective. The paper addresses an important question: why critic-free RL methods such as GRPO can effectively improve LLMs without explicitly learning a value function. The proposed value-gradient perspective offers an interesting explanation beyond existing analyses based on group-relative advantages or implicit value estimation. By connecting actor gradient propagation to value gradients, the paper provides a potentially useful theoretical framework for understanding critic-free RL.
Critical weaknesses
Insufficient empirical validation. The experiments are limited to a simple label-copying task using OLMo-2 1B checkpoints, without evaluation on realistic reasoning tasks such as mathematics or coding. Moreover, the experiments do not directly verify the theoretical approximation bound, and the reported correlations provide only preliminary support for the RL-impact hypothesis. Stronger empirical evidence is needed to establish the generality and practical relevance of the proposed framewor
Questions for authors
- Serious concerns regarding the completeness and reliability of the experimental description. In Appendix A.13, the statements "The experimental description does not specify which such completion-dependent training reward or direct-gradient rule was used" and "This implementation detail must be resolved before the reported runs are identified with the reward-driven GRPO mechanism analyzed in this paper" show clear signs of an unfinished manuscript. Even under the most charitable interpretation, assuming that these issues arise from communication problems among co-authors rather than AI-automated research, the authors must explain this issue in the rebuttal. Otherwise, I will lower my rating to 1 (Clear Rejection).
- Limited experimental evidence for the proposed hypothesis. The empirical validation is primarily conducted on a simple label-copying task using OLMo-2 1B, which is substantially different from the long-horizon reasoning tasks that motivate this work. Moreover, the experiments do not directly establish whether the proposed value-gradient approximation holds in practical discrete-token RL training. Can the authors provide stronger evidence that the proposed RL-impact predictor captures meaningful RL gains beyond simple baselines such as initial performance or reward variance, ideally on realistic reasoning benchmarks? Such evidence would substantially strengthen the practical relevance of the proposed hypothesis.
Pt.1
Thank you for the review. We are glad that you find the research question meaningful and the value-gradient perspective novel. Please see the answers below.
Q1: Appendix A.13
The sentences you quote come from the scope of the paper. A.13 is titled "Theoretical scope of the experimental diagnostics". Its purpose is to state, for each quantity measured in Section 6, which conclusions the theory supports and under which specification.
The theory analyzes the reward-driven GRPO update. This update exists only for a completion-dependent reward: if every completion of a prompt receives the same reward, all group-normalized advantages are zero and there is no reward-driven update (A.13). The score reported in Section 6, pθ(y⋆∣q), is a property of the prompt rather than of a single completion. A.13 therefore records that the analysis applies to a completion-dependent reward whose expectation is this score, such as sampled-label correctness 1{o=y⋆}, and that a direct gradient of the score would be a different update, outside the analysis.
These are conditions on the scope of the theory, not uncertainty about the experiments. We agree, however, that the wording, in particular "the experimental description does not specify", reads as if the setup itself were undetermined. We will change it as follows:
Q2: Stronger empirical evidence
Strengthened theory
We will also add an exact result that strengthens the central claim. Let ht be the complete hidden state and V~t the value of the effective reward of Corollary 1. Then the actor backward pass of discrete GRPO satisfies E[λ^t∣ht]=∂hV~t(ht) exactly, with no relaxation. The claim "critic-free does not mean value-free" therefore holds exactly for the discrete update that GRPO actually performs.
Thank you for your thoughtful feedback.