- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TPPO, a variant of PPO that score prefix values through cumulative logit increments rather than independent scalar predictions. Motivated by a Bayesian decomposition of conditional success probabilities, TPPO trains the cumulative critic using binary cross-entropy against terminal outcomes while retaining the standard GAE and PPO actor objectives. Experiments across mathematical, Countdown, and coding tasks demonstrate consistent improvements over PPO.
Critical strengths
- TPPO introduces a lightweight change to the PPO critic without requiring additional rollout sampling or process-level supervision.
- TPPO shows improvement on mathematical, Countdown, and coding tasks, compared to PPO.
Critical weaknesses
- The advantage sign-accuracy metric may not meaningfully measure credit-assignment quality. Under binary rewards and , the sign of is determined by the terminal reward when . Thus, the substantial advantage sign-accuracy improvements in Figure 6 may reflect bounded value predictions rather than better token-level credit assignment.
- Table 1 shows only modest improvements in value-change sign accuracy (2.01–2.80 points), despite much larger downstream performance gains. The extent to which improved value tracking explains these gains remains unclear.
- Although the paper discusses HL-Gauss PPO in Section 2, it does not include this recent classification-based critic method as a baseline. Such comparisons would help establish the advantage of cumulative parameterization over alternative value-learning approaches.
Questions for authors
- Why does TPPO with MSE perform substantially worse than TPPO with BCE, according to Figure 8(a)? Does the cumulative parameterization introduce optimization difficulties that are specifically alleviated by BCE?
- Since the learned increments are not explicitly supervised to match the Bayesian log-likelihood ratios in Eq. (8), what evidence suggests that they recover the intended Bayesian value updates?
- Earlier increments receive gradients from more subsequent positions than later increments. How does this asymmetric gradient propagation affect value learning, particularly for long reasoning trajectories?
Response Reviewer 7qhe (1/6)
Thank you for your time and insightful review. Following your insightful suggestions, we added
We hope these additional results and clarifications fully address your concerns.
We appreciate this observation, which helps clarify the interpretation of advantage sign accuracy. Under binary rewards and γ=λ=1, bounded value predictions determine the sign of At=R−V(st). This property is relevant to policy optimization: out-of-range baselines can reverse advantage signs and thus the direction of token updates. We complement it with advantage–MC Pearson correlation and controlled single-token interventions to assess quality of credit assignment more directly.
We additionally report the Pearson correlation between predicted advantages and their Monte Carlo references. Pearson correlation provides a direct diagnostic of the relative weighting signals used in token-level policy optimization compared to Monte Carlo references. As shown in Table R7, TPPO achieves higher correlation across all evaluated checkpoints, providing complementary evidence of better advantage alignment during training.
To directly test whether the critic assigns accurate local credit when an individual reasoning token is corrupted, we conduct a controlled token-level intervention experiment across 1,536 intervention cases (64 problems×8 rollouts×3 intervention positions). Specifically, at each position before
\boxed{}, we replace an intermediate numeric or operator token with an erroneous single-token alternative while keeping the preceding prefix identical. We then run independent Monte Carlo (MC) rollouts from the original and perturbed prefixes to identify harmful edits (ΔVMC=VMCorig−VMCpert>0), evaluated at two SNR thresholds. Because both branches share the same prefix baseline, the local advantage drop at the intervened token equals the predicted value drop (ΔA^t=ΔVt=Vtorig−Vtpert). As shown in Table R8, TPPO more reliably captures the advantage drop caused by corrupting a key token, improving the Drop Identification Rate (ΔA^t>0) over PPO from 53.70% to 76.60% and from 58.40% to 84.60% at the two SNR thresholds, respectively. TPPO also achieves lower MAE and substantially higher cosine similarity and Pearson correlation with ΔVMC, indicating more accurate local credit assignment.We have incorporated these new analyses and discussions in Lines 1132–1175 (Appendix D.4).
Table R7. Pearson correlation between predicted advantages and their Monte Carlo references.
Table R8. Single-token perturbation diagnostics against Monte Carlo value differences.