acceptodds
Under review as a conference paper at ICLR 2027

TPPO: Tracking-Value PPO For Language Model Reasoning

Abstract

Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While value evolution along trajectory characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, we analyze the relation between successive prefix values and show that they admit a Bayesian decomposition with additive updates in log-odds space. We use this additive structure to construct Tracking-Value PPO (\method), a cumulative value parameterization that sums position-wise scalar outputs into value logits. These outputs are learned through terminal-outcome supervision on the reconstructed values. % \method uses the same terminal outcomes as PPO, without additional rollouts or process-level supervision. % Across mathematical reasoning benchmarks, \method consistently outperforms PPO across model scales and variants, improving aggregate scores by 8.5 and 9.2 percentage points on Qwen3-4B-Base and Qwen3-8B-Base, respectively. These gains extend to countdown, science, and coding tasks. Further analysis confirms more accurate value-change prediction and advantage estimation. \method improves aggregate mathematical reasoning scores over PPO by 7.1 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, while improving out-of-distribution generalization. Gains extend to Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B and to countdown, and scientific coding tasks. Further analysis shows more accurate tracking of value evolution and more reliable advantage estimates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7qheOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TPPO, a variant of PPO that score prefix values through cumulative logit increments rather than independent scalar predictions. Motivated by a Bayesian decomposition of conditional success probabilities, TPPO trains the cumulative critic using binary cross-entropy against terminal outcomes while retaining the standard GAE and PPO actor objectives. Experiments across mathematical, Countdown, and coding tasks demonstrate consistent improvements over PPO.

Critical strengths
  1. TPPO introduces a lightweight change to the PPO critic without requiring additional rollout sampling or process-level supervision.
  2. TPPO shows improvement on mathematical, Countdown, and coding tasks, compared to PPO.
Critical weaknesses
  1. The advantage sign-accuracy metric may not meaningfully measure credit-assignment quality. Under binary rewards and γ=λ=1\gamma=\lambda=1, the sign of At=R−VtA_t=R-V_t is determined by the terminal reward when Vt∈(0,1)V_t\in(0,1). Thus, the substantial advantage sign-accuracy improvements in Figure 6 may reflect bounded value predictions rather than better token-level credit assignment.
  2. Table 1 shows only modest improvements in value-change sign accuracy (2.01–2.80 points), despite much larger downstream performance gains. The extent to which improved value tracking explains these gains remains unclear.
  3. Although the paper discusses HL-Gauss PPO in Section 2, it does not include this recent classification-based critic method as a baseline. Such comparisons would help establish the advantage of cumulative parameterization over alternative value-learning approaches.
Questions for authors
  1. Why does TPPO with MSE perform substantially worse than TPPO with BCE, according to Figure 8(a)? Does the cumulative parameterization introduce optimization difficulties that are specifically alleviated by BCE?
  2. Since the learned increments are not explicitly supervised to match the Bayesian log-likelihood ratios in Eq. (8), what evidence suggests that they recover the intended Bayesian value updates?
  3. Earlier increments receive gradients from more subsequent positions than later increments. How does this asymmetric gradient propagation affect value learning, particularly for long reasoning trajectories?
1 comment
AuthorsOpenReview ↗

Response Reviewer 7qhe (1/6)

Thank you for your time and insightful review. Following your insightful suggestions, we added

  1. further analysis of value evolution and credit assignment. TPPO better matches MC credit signals and identifies harmful edits (Tables R7–R8), maintains stronger tracking as PPO’s tracking deteriorates (Table R1), and outperforms HL-Gauss on all seven benchmarks (Table R9).
  2. further discussion of critic parameterization and optimization. Logit-space MSE outperforms PPO; value-space instability persists despite longer warmup (Tables R5–R6). TPPO also achieves lower increment MAE and learns faster with downstream gradients (Tables R10–R11).

We hope these additional results and clarifications fully address your concerns.

W1: The advantage sign-accuracy metric may not meaningfully measure credit-assignment quality

We appreciate this observation, which helps clarify the interpretation of advantage sign accuracy. Under binary rewards and γ=λ=1\gamma=\lambda=1, bounded value predictions determine the sign of At=R−V(st)A_t=R-V(s_t). This property is relevant to policy optimization: out-of-range baselines can reverse advantage signs and thus the direction of token updates. We complement it with advantage–MC Pearson correlation and controlled single-token interventions to assess quality of credit assignment more directly.

We additionally report the Pearson correlation between predicted advantages and their Monte Carlo references. Pearson correlation provides a direct diagnostic of the relative weighting signals used in token-level policy optimization compared to Monte Carlo references. As shown in Table R7, TPPO achieves higher correlation across all evaluated checkpoints, providing complementary evidence of better advantage alignment during training.

To directly test whether the critic assigns accurate local credit when an individual reasoning token is corrupted, we conduct a controlled token-level intervention experiment across 1,536 intervention cases (64 problems×8 rollouts×3 intervention positions64\text{ problems}\times8\text{ rollouts}\times3\text{ intervention positions}). Specifically, at each position before \boxed{}, we replace an intermediate numeric or operator token with an erroneous single-token alternative while keeping the preceding prefix identical. We then run independent Monte Carlo (MC) rollouts from the original and perturbed prefixes to identify harmful edits (ΔVMC=VMCorig−VMCpert>0\Delta V_{\text{MC}}=V_{\text{MC}}^{\text{orig}}-V_{\text{MC}}^{\text{pert}}>0), evaluated at two SNR thresholds. Because both branches share the same prefix baseline, the local advantage drop at the intervened token equals the predicted value drop (ΔA^t=ΔVt=Vtorig−Vtpert\Delta\hat{A}_t=\Delta V_t=V_t^{\text{orig}}-V_t^{\text{pert}}). As shown in Table R8, TPPO more reliably captures the advantage drop caused by corrupting a key token, improving the Drop Identification Rate (ΔA^t>0\Delta\hat{A}_t>0) over PPO from 53.70% to 76.60% and from 58.40% to 84.60% at the two SNR thresholds, respectively. TPPO also achieves lower MAE and substantially higher cosine similarity and Pearson correlation with ΔVMC\Delta V_{\text{MC}}, indicating more accurate local credit assignment.

We have incorporated these new analyses and discussions in Lines 1132–1175 (Appendix D.4).

Table R7. Pearson correlation between predicted advantages and their Monte Carlo references.

StepPPOTPPO
2000.579 [0.509, 0.653]0.691 [0.644, 0.739]
3000.574 [0.514, 0.637]0.649 [0.594, 0.707]
4000.561 [0.499, 0.622]0.677 [0.613, 0.736]
5000.641 [0.579, 0.702]0.691 [0.629, 0.750]

Table R8. Single-token perturbation diagnostics against Monte Carlo value differences.

SNRMethodDrop Identification Rate ↑\uparrowMAE ↓\downarrowCosine ↑\uparrowPearson ↑\uparrow
>0> 0PPO53.70%0.180.12630.0767
>0> 0TPPO76.60%0.140.44760.2359
>1> 1PPO58.40%0.290.15820.0648
>1> 1TPPO84.60%0.250.49320.1803
Reviewer_W8xFOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

TPPO parameterizes the critic as cumulative logit increments, applies a sigmoid to recover values, and trains them against terminal outcomes with binary cross-entropy. These values replace the standard PPO critic predictions. Experiments cover mathematical reasoning, Countdown, and coding.

Critical strengths
  1. TPPO improves average mathematics accuracy over PPO by 7.0 and 8.4 percentage points on Qwen3-4B-Base and Qwen3-8B-Base. Direct-BCE ablations and fixed-actor diagnostics provide evidence for the critic design.

  2. The method requires a small critic modification without additional training rollouts, process annotations, or a separate reward model.

Critical weaknesses
  1. Absolute AIME performance raises concerns about baseline strength. On Qwen3-4B-Base, GRPO and TPPO reach 12.1/11.4 and 18.5/18.4 on AIME24/25, while RePro [1] reports 23.5/19.1 for GRPO. The training and evaluation settings differ, but the gap remains unexplained despite several hundred training steps.

  2. The benefit of logit space is not isolated. The direct-BCE comparison supports cumulative prediction, but does not establish whether accumulating residuals in value space would provide similar gains.

[1] Liu et al. Rectifying LLM Thought from Lens of Optimization. arXiv:2512.01925, 2025.

Questions for authors
  1. Under matched training budgets, how do logit-space and value-space accumulation compare in value-estimation error and final policy accuracy?
Reviewer_5NEqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies token-level credit assignment in RLVR, and shows that standard critics such as in PPO often struggle to accurately measure value evolution across the trajectory. They propose Tracking-Value PPO (TPPO), a parametrization of the critic based on the accumulation of position-wise scalar outputs. They test TPPO with Qwen3-4B-Base and Qwen3-8B-Base, showing that TPPO substantially improves performance compared to methods such as PPO on mathematical reasoning, as well as extending to out-of-distribution generalization.

Critical strengths
  1. The paper identifies a problem in value-based RL for LLM reasoning and the proposed method is simple and well-motivated. TPPO is also compatible with standard PPO training and doesn’t require additional rollouts. This is consequential to the overall assessment as it provides substantial motivation (inaccurate value evolution across the trajectory, and the value parametrization using the Bayesian decomposition) and gives a simple solution that can be easily incorporated into existing implementations.

  2. The empirical performance improvements of TPPO are substantial and consistent across settings, performing better than several other baselines. They show improvements on different model sizes, rollout budgets, as well as across different tasks.

Critical weaknesses
  • The relationship between improved value tracking and the improved policy optimization is still a bit unclear. Even under TPPO, the sign accuracy seems to only improve by a few percentage points (as reported in Table 1), while staying relatively close to 50%. Given that this was part of the main motivation for the method, it would be helpful to discuss and analyze in more detail what exactly TPPO does that contributes to its improved performance.
  • Despite extensive benchmark coverage, the paper does not report performance variability across independent training runs (e.g. standard deviations across seeds in Tables 2 and 3). Including such results would strengthen confidence in the reported improvements.
Questions for authors
  1. Could the authors better isolate why cumulative value parametrization improves policy optimization? Given that the value-change sign accuracy benefits seem relatively small, could they analyze and identify what additional benefits that TPPO provides that contributes to its gains?
  2. How robust are the reported performance improvements across independent training runs? Could the authors report results across multiple seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.