- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper compares token-level and trajectory-level policy gradient estimates. It shows that token-level estimates do not always have lower variance. The authors also conduct experiments on language models to explore potential benefits of token-level estimation
Critical strengths
This paper studies an important question in LLM reinforcement learning: when it is beneficial to perform token-level policy gradient estimation.
Critical weaknesses
The motivation and contribution in this paper are not clear.
The abstract lists many technical terms, conditions, and formulas before clearly explaining why the results matter. It also present ratios involving without defining in the abstract. The introduction also does not clearly distinguish the central theoretical advance from the supporting results. Furthermore, the theoretical section, especially its latter half, reads more like a collection of related results. For example, the key theoretical decompositions in Sections 10.1 and 10.2 are placed within the empirical section, making the overall argument harder to follow. As a result, I struggle to follow the paper’s main contribution.
The experiments did not establish the practical applicability of the main theoretical findings.
I am unsure how useful the main theoretical results are for training LLMs. The variance reduction guarantees rely on strict conditions, such as orthogonal gradients across tokens. Empirical results on language models suggest that the spectral bound derived from these assumptions lacks practical relevance. Consequently, the experimental results seem to reveal the limitations of applying this theory rather than demonstrate under what conditions its expected benefits can be realized.
Questions for authors
1: Could you clarify the main theoretical contribution compared to prior work [1, 2]?
2: The spectral bound in Theorem 16 is uninformative for the tested language model. What practical guidance does the theory offer in this case?
[1]Greensmith et al. Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. JMLR, 2004. [2]Li et al. The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. ICML, 2026.
2 comments
Maintain the original score
We thank you for the clarification. However, we have decided to maintain the original score for the following two reasons:
- As also noted by reviewers qnN2 and wMea, the manuscript's organization and the clarity of its contributions still fall short of the standards expected for acceptance at ICLR.
- The theoretical results rely on restrictive assumptions, which limit their explanatory power and practical relevance to LLMs with shared parameters.
Thank you for identifying both the positioning and practical-guidance gaps.
Relative to prior work, Greensmith et al. develop the classical theory of baselines and control variates for policy-gradient variance reduction, while Li et al. derive variance-minimizing token-baseline weights. Our question is different: when a centered terminal-return estimator is replaced by an exact token-advantage estimator, is the latter necessarily lower variance when every token score differentiates the same model parameters?
Our answer is no without an additional geometry condition. The main new result is the exact decomposition of the trajectory estimator into the oracle token estimator and zero-mean cross-noise, together with a valid bounded-reward shared-parameter construction in which their covariance reverses the expected ordering. We then establish the opposite, favorable direction under pathwise orthogonal scores. This is not another choice of action-independent baseline: it characterizes when changing the temporal credit weights can and cannot yield a factor-of-H separation. For a learned critic, our conditional MSE identity additionally retains critic bias, its interaction with the oracle estimator error, and all cross-prefix critic-error covariances.
When the spectral bound is uninformative, the theorem should be treated as a diagnostic gate rather than a guarantee that every neural policy should satisfy. If the measured geometry fails the condition, one should not invoke the factor-of-H conclusion. The comparison must instead be empirical, with three quantities reported together: the actual gradient variance or MSE, joint critic errors across prefixes, and the samples used to estimate those values.
The experiments show how to use that gate. In the controlled GPT-2 audit, the trajectory/token covariance-trace ratio grows from 3.30 at horizon 4 to 22.42 at horizon 32 at continuation budget 128, but the latter requires 2,001 generated continuation tokens per main trajectory. In our new fixed-policy Qwen2.5-1.5B measurements, the empirical spectral deviation is approximately one, so the theorem supplies no useful ranking. Across matched terminal/MC1/MC4/MC16 gradients, variance does not improve monotonically, while MC16 requires about 35 times the terminal estimator's sampled tokens. We report both results because they separate a demonstrable conditional mechanism from an unsupported general training claim.
We will sharpen the manuscript around this distinction. The decomposition and counterexample will be presented as the central contribution; the orthogonal and spectral results will be labeled as sufficient conditions; and the critic-budget measurements will be presented as evidence about whether those conditions can support a practical claim. The GAE and other extensions will move to the appendix, and H will be defined before any ratio appears.