acceptodds
Under review as a conference paper at ICLR 2027

Critic Error and Score Geometry in Token-Level Policy Gradients

Abstract

Token-level credit information need not reduce policy-gradient noise when token scores share parameters. We study this dependence in a deterministic, terminal-reward Token MDP. The advantage sequence is a martingale difference sequence, yielding an exact return-variance identity and a trajectory-gradient decomposition into an oracle token term and zero-mean cross-noise. The cross term has no fixed sign: a valid four-step shared-parameter policy reverses the second-moment ordering. Pathwise orthogonal scores permit an attainable ratio, or under homogeneous advantages when both estimators access the true value function. A spectral condition extends these conditional comparisons to random score geometry, but is uninformative on our measured neural-policy Grams. For estimated critics, a conditional mean-squared-error identity separates critic bias, its interaction with oracle error, and the full critic-error covariance; perturbation bounds and an exact-value GAE() spectrum complement this analysis. A pretrained GPT-2 prefix-continuation study exhibits a critic-budget-dependent estimator gap at substantial sampling cost. A separate three-seed, token-budget-matched LoRA diagnostic uses conditional- weights rather than oracle advantages and does not establish a training advantage. The results identify assumptions needed for an estimator separation and measurements needed before interpreting it as a practical efficiency gain.

Primary area:
reinforcement learning
Keywords:
policy gradients, token-level credit assignment, score geometry, critic error, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_syddOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper compares token-level and trajectory-level policy gradient estimates. It shows that token-level estimates do not always have lower variance. The authors also conduct experiments on language models to explore potential benefits of token-level estimation

Critical strengths

This paper studies an important question in LLM reinforcement learning: when it is beneficial to perform token-level policy gradient estimation.

Critical weaknesses

The motivation and contribution in this paper are not clear.
The abstract lists many technical terms, conditions, and formulas before clearly explaining why the results matter. It also present ratios involving HH without defining HH in the abstract. The introduction also does not clearly distinguish the central theoretical advance from the supporting results. Furthermore, the theoretical section, especially its latter half, reads more like a collection of related results. For example, the key theoretical decompositions in Sections 10.1 and 10.2 are placed within the empirical section, making the overall argument harder to follow. As a result, I struggle to follow the paper’s main contribution.

The experiments did not establish the practical applicability of the main theoretical findings.
I am unsure how useful the main theoretical results are for training LLMs. The variance reduction guarantees rely on strict conditions, such as orthogonal gradients across tokens. Empirical results on language models suggest that the spectral bound derived from these assumptions lacks practical relevance. Consequently, the experimental results seem to reveal the limitations of applying this theory rather than demonstrate under what conditions its expected benefits can be realized.

Questions for authors

1: Could you clarify the main theoretical contribution compared to prior work [1, 2]?

2: The spectral bound in Theorem 16 is uninformative for the tested language model. What practical guidance does the theory offer in this case?

[1]Greensmith et al. Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. JMLR, 2004. [2]Li et al. The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. ICML, 2026.

2 comments

Thank you for identifying both the positioning and practical-guidance gaps.

Relative to prior work, Greensmith et al. develop the classical theory of baselines and control variates for policy-gradient variance reduction, while Li et al. derive variance-minimizing token-baseline weights. Our question is different: when a centered terminal-return estimator is replaced by an exact token-advantage estimator, is the latter necessarily lower variance when every token score differentiates the same model parameters?

Our answer is no without an additional geometry condition. The main new result is the exact decomposition of the trajectory estimator into the oracle token estimator and zero-mean cross-noise, together with a valid bounded-reward shared-parameter construction in which their covariance reverses the expected ordering. We then establish the opposite, favorable direction under pathwise orthogonal scores. This is not another choice of action-independent baseline: it characterizes when changing the temporal credit weights can and cannot yield a factor-of-HH separation. For a learned critic, our conditional MSE identity additionally retains critic bias, its interaction with the oracle estimator error, and all cross-prefix critic-error covariances.

When the spectral bound is uninformative, the theorem should be treated as a diagnostic gate rather than a guarantee that every neural policy should satisfy. If the measured geometry fails the condition, one should not invoke the factor-of-HH conclusion. The comparison must instead be empirical, with three quantities reported together: the actual gradient variance or MSE, joint critic errors across prefixes, and the samples used to estimate those values.

The experiments show how to use that gate. In the controlled GPT-2 audit, the trajectory/token covariance-trace ratio grows from 3.30 at horizon 4 to 22.42 at horizon 32 at continuation budget 128, but the latter requires 2,001 generated continuation tokens per main trajectory. In our new fixed-policy Qwen2.5-1.5B measurements, the empirical spectral deviation is approximately one, so the theorem supplies no useful ranking. Across matched terminal/MC1/MC4/MC16 gradients, variance does not improve monotonically, while MC16 requires about 35 times the terminal estimator's sampled tokens. We report both results because they separate a demonstrable conditional mechanism from an unsupported general training claim.

We will sharpen the manuscript around this distinction. The decomposition and counterexample will be presented as the central contribution; the orthogonal and spectral results will be labeled as sufficient conditions; and the critic-budget measurements will be presented as evidence about whether those conditions can support a practical claim. The GAE and other extensions will move to the appendix, and H will be defined before any ratio appears.

Reviewer_syddOpenReview ↗

Maintain the original score

We thank you for the clarification. However, we have decided to maintain the original score for the following two reasons:

  1. As also noted by reviewers qnN2 and wMea, the manuscript's organization and the clarity of its contributions still fall short of the standards expected for acceptance at ICLR.
  2. The theoretical results rely on restrictive assumptions, which limit their explanatory power and practical relevance to LLMs with shared parameters.
Reviewer_wMeaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes the bias and variance of the commonly used gradient estimators of the Token MDP in LLM training. They provide an concrete example that although token-level estimator have smaller variance upper-bounds, it can have larger variance compared to trajectory-level estimators under shared parameters.

Critical strengths

None.

Critical weaknesses

The paper is just a collection of simple calculations. It's unclear what the authors are going to claim or to convey throught these calculations. The counter example is over simplified if it were to explain the variance of token-level estimators in practice. The experiment presented in the main text is incomplete that it demonstrates nothing.

Questions for authors

What are the claims and conclusions of this paper?

2 comments

Thank you. The paper makes one negative claim and one conditional positive claim.

Our negative result is that token-level credit does not universally lower policy-gradient variance when token scores share parameters. We prove the exact relation

centered trajectory estimator = oracle token estimator + zero-mean cross-noise.

The covariance between the last two terms has no fixed sign. Our four-step example is deliberately minimal because it serves as a counterexample to a universal ordering. It uses a valid shared-parameter stochastic policy, reward in [0,1], and a nonzero policy gradient; at the stated parameter, the token estimator has the larger second moment even though the average score Gram is the identity.

There is also a conditional positive result. A horizon-dependent improvement is attainable under stronger geometry. Pathwise orthogonal token scores give an exact factor-of-HH separation; when both estimators receive the true value function and advantage variances are homogeneous, the ratio is 2/(H+1)2/(H+1). With an estimated critic, however, the comparison also contains critic bias, its interaction with oracle estimation error, and the full covariance of value errors across prefixes. Thus the oracle result cannot be transferred to an LLM merely because token-level labels are available.

We agree that the original empirical section did not establish a practical training advantage, and we will frame it accordingly. The controlled GPT-2 audit does show the conditional mechanism: at continuation budget 128, the trajectory/token covariance-trace ratio rises from 3.30 at horizon 4 to 22.42 at horizon 32. That gain is not free—the horizon-32 estimate uses 2,001 generated continuation tokens per main trajectory.

We also added a matched fixed-policy diagnostic aimed directly at the geometry premise. Using one frozen Qwen2.5-1.5B policy, terminal, MC1, MC4, and MC16 use the same main trajectories. Across 32 minibatches per arm, increasing the continuation budget does not produce a stable variance ordering. The measured full-LoRA token Grams are extremely ill-conditioned: the empirical spectral deviation from a common isotropic scale is approximately one, making the sufficient bound uninformative. MC16 consumes roughly 35 times the terminal estimator's sampled-token cost. These measurements do not prove that token credit is ineffective; together with the controlled audit, they show both when the mechanism can appear and why it does not automatically imply an LLM training advantage.

We will revise the introduction and conclusion accordingly: the contribution is an estimator-level counterexample, an explicit set of sufficient conditions, and a critic-error/cost accounting framework. It is not a new training algorithm and does not claim that the short LoRA study demonstrates optimization superiority. We will move auxiliary calculations out of the main line so that these claims and conclusions are visible without reconstructing them from the appendices.

Thank you for your response. However, the quality of the paper is not good enough in presentation and informativeness. I will keey my score.

Reviewer_qnN2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

It was quite hard for me to understand what the main message or contribution of the paper was, so I can't provide a good summary. From my understanding, the authors calculate the variance of different token-level advantage estimators and compare their properties.

Critical strengths

None

Critical weaknesses
  • I couldn't figure out any clear narrative, main message, or primary contributions of the paper. My impression is that large parts of the paper are directly AI generated, which typically looks like unrelated technical sentences one after another, without any common theme or broader picture.

  • As for the technical details, they seem to be straightforward calculations of bias and variance of different advantage estimators. I'm not sure if they are really novel or provide any specific insights to the reader.

Questions for authors

My recommendation is that the authors spend more time trying to build the narrative and the main message of the paper. In case the authors have used generative AI extensively for writing the main text, I encourage them to keep in mind that current generative models lack the ability to properly write the narrative, and this specifically requires human insight and writing.

2 comments

Thank you for being direct about the presentation problem. We agree that the paper's main message is buried. What we should have said up front is simple:

Our main conclusion is that finer token-level credit does not by itself reduce policy-gradient noise. The ordering also depends on the geometry of token score vectors and on how accurately token values are estimated.

The paper supports this statement in three steps. First, the centered trajectory estimator is exactly the oracle token estimator plus a zero-mean cross-noise term. Because all token scores share policy parameters, that noise can be correlated positively or negatively with the token term. Second, our four-step construction is a valid stochastic policy with bounded reward and a nonzero gradient; it reverses the expected ordering even though the average score Gram is the identity. Its simplicity is intentional: it disproves a universal variance claim without relying on an invalid score sequence or an unbounded reward. Third, we identify conditions under which the favorable ordering does hold—pathwise orthogonal scores—and then show how critic bias and the full cross-prefix error covariance alter the oracle comparison.

We also added a neural-policy diagnostic to connect this argument to the setting that motivated it. Using one frozen Qwen2.5-1.5B policy, we measured full 1,089,536-dimensional LoRA gradients for matched terminal, MC1, MC4, and MC16 estimators across 32 minibatches per arm, and recovered token-level Grams for 128 trajectories. The empirical spectral deviation is approximately one, so the near-isotropy condition gives no useful guarantee. The variance traces show no stable monotone advantage as the continuation budget increases, while MC16 uses about 35 times as many sampled tokens as the terminal estimator. We view this as evidence for the paper's boundary, not as a training win.

We will rewrite the paper around this single argument. The decomposition, reversal, favorable construction, and critic-error identity will become one continuous main-text sequence. The GAE and secondary bounds will move to the appendix, and the empirical section will be labeled explicitly as an assumption audit. We will also replace the current abstract with a plain-language statement before introducing ratios or notation.

The disclosed AI-assisted tools were used during drafting and analysis, but responsibility for the unclear narrative is ours. The authors will rewrite the argument and independently recheck every mathematical claim, citation, experiment, and limitation in the revision.

Thank you for your response. Based on the paper's presentation issues and other reviews, I've decided to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.