- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper identifies a systematic failure mode in PPO critics called Value Flattening and proposes a simple, effective mitigation strategy. When state values are estimated via multiple Monte Carlo (MC) continuations from the same policy, these MC values often exhibit sharp local transitions across intermediate states. However, the PPO critic's predictions remain comparatively flat and sometimes even move in the opposite direction. To mitigate the problem, this paper proposes , which trains the critic on only a few well-separated states per response instead of on every token position. Experiments on Qwen3-4B-Base and Qwen3-8B-Base show that consistently outperforms standard PPO and GRPO on mathematical and out-of-distribution (OOD) reasoning benchmark.
Critical strengths
-
Identifies a novel and concrete failure mode. “Value Flattening” is indeed a well-defined phenomenon: the PPO critic predicts flat values while Monte Carlo estimates show sharp local transitions. This is a useful diagnostic contribution beyond just proposing another algorithm.
-
Systematic empirical evidence. The paper shows the phenomenon across training checkpoints, correct/incorrect responses, and in a controlled FrozenLake environment. Reproducing it outside LLMs strengthens the claim that it is not an artifact of one setup.
-
Propose a simple, practical solution. only changes where the value loss is applied, making it easy to implement on top of PPO. It does not require a new architecture or reward model.
Critical weaknesses
-
Narrow task coverage. Evaluation focuses on mathematical reasoning and OOD reasoning. Generalization to coding, agentic tasks, dialogue, RLHF, multimodal reasoning, or safety-critical applications is not established.
-
Causal isolation is incomplete. The paper shows that sparse value supervision reduces flattening and improves performance, but it does not fully prove that flattening is the primary causal bottleneck. Other factors, for example, better credit assignment, reduced gradient noise, or regularization could also explain the gains.
Questions for authors
-
Why only use fixed anchors for updating the critic model (0.3/0.6/0.9 + 0.95 tail). Why not adaptive anchors: split into K segments and pick the max hidden-state change per segment, or other more principal approach to determine when the critic should be updated? It keeps sparsity/separation and targets semantic shifts, but hidden-state change value change, anchors shift during training, and top changes may cluster or be noise. Can you do some additional experiments to test this and see if it can beat fixed anchors or destabilize the critic?
-
Does it survive dense/process rewards? Your mechanism assumes the same terminal reward at all anchors. With token-level rewards, for example, we obtain a PRM somehow, is flattening still the issue?
-
You report SP³O gives longer responses than PPO and the tail anchor cuts repetition from 18.33% to 1.12%. Could the gains be a length/repetition artifact rather than better critic values? Please report length-controlled results: same max token budget, length-matched comparisons, accuracy vs. length, repetition as a covariate, tail-anchor ablation at matched length, and short-answer tasks. How much of the in-domain and OOD gains remains? If it vanishes at matched length, the main effect may be generation stability, not fixing value flattening. How sensitive are and anchor placement to task, response length, and model scale? Is universal or tuned on math?
4 comments
Response to Reviewer 6GgM (2/2): Q2–Q3
Q2:
Thank you for suggesting experiments under different reward structures.
In LLM reasoning tasks, training commonly relies on terminal outcome rewards rather than dense token-level supervision. So we trained CNN-based PPO on the eight Atari games below with two seeds and 1,048,576 decisions per seed. These Atari games provide immediate environment rewards and have more constrained state spaces than the state space of LLM reasoning. In these games, PPO critics preserve meaningful within-trajectory value differences and closely track MC estimates.
MC values are estimated using 64 continuations per sampled state. The SD ratio compares critic and MC value variation within episodes; 1 indicates matched variation. The correlation measures how closely critic predictions track MC values across states. Together, these results indicate that PPO critics preserve meaningful within-episode value differences in these eight games, without pronounced value flattening.
Table. Critic value diagnostics on an eight-game Atari subset (means over two seeds).
| Metric | Freeway | Breakout | Space Invaders | Q*bert | Asterix | Ms. Pac-Man | River Raid | Seaquest |
|---|---|---|---|---|---|---|---|---|
| Critic/MC value SD ratio | 0.731 | 0.828 | 0.935 | 1.008 | 0.963 | 0.767 | 0.985 | 0.923 |
| Critic–MC correlation | 0.810 | 0.852 | 0.928 | 0.975 | 0.966 | 0.895 | 0.992 | 0.866 |
Exploring this phenomenon under different reward structures is a complex and valuable direction that needs further study. We thank the reviewer for suggesting this valuable direction and will explore it further in future work.
Q3:
Thank you for suggesting length-controlled comparisons and further analysis of generalization.
Our proposed causal explanation is that better critic estimates improve credit assignment and guide more effective policy optimization during training. Thus, reduced repetition and higher accuracy are both manifestations of improved policy behavior, rather than one causing the other.
In our main text, Figure 4 shows that SP³O’s critic has lower value errors and gradient discrepancies against MC references, alongside better outcome discrimination. This can support more effective policy optimization, consistent with the more stable actor updates and higher validation accuracy shown in Figure 5.
1. Length-controlled comparisons.
PPO and SP³O use identical maximum response lengths: 8,192 tokens during training and 24,576 during evaluation. So both methods follow identical training and evaluation protocols, ensuring a fair comparison. We further analyze the evaluation outputs underlying the paper’s 4B results using 4,096-token length bins.
Within each bin, we retain questions for which both methods have at least one response in that bin. We average scores within each question and method, then weight questions equally.
Table. Qwen3-4B-Base accuracy by generated length on shared questions (%).
| Generated length | Shared questions | PPO | SP³O |
|---|---|---|---|
| 0–4K | 1,527 | 51.15 | 61.78 |
| 4–8K | 927 | 25.33 | 36.06 |
| 8–12K | 525 | 18.65 | 23.34 |
| 12–16K | 213 | 13.04 | 14.80 |
| 16–20K | 62 | 9.54 | 15.02 |
| 20–24K | 267 | 8.84 | 10.04 |
SP³O achieves higher accuracy in all six question-matched length bins. These are descriptive comparisons conditioned on generated length.
2. Generalization and sensitivity analysis.
Beyond the 4B and 8B experiments, our paper evaluates SP³O on Qwen3-14B-Base trained on Polaris-V2-RL-14K. Compared with PPO, SP³O improves mathematical and OOD reasoning accuracy by 4.12 and 4.31 percentage points, respectively. On Qwen3.5-9B scientific coding, it improves the six-benchmark average by 2.38 points and SciCode subproblem accuracy by 1.97 points. These results support its applicability across model sizes, training datasets, and task domains.Our placement ablation also shows that both 0.2/0.5/0.8 and 0.3/0.6/0.9 outperform PPO, by 7.05 and 7.97 percentage points, respectively.
We hope these clarifications address your concerns. We welcome any further questions or suggestions.
References
[1] Improving Value Estimation Critically Enhances Vanilla Policy Gradient. ICML 2025. https://arxiv.org/abs/2505.19247
[2] High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016. https://arxiv.org/abs/1506.02438
The authors have adequately addressed most of my questions, especially regarding the dense reward. They have also provided stronger evidence for SP³O’s performance in a short time. I will retain my previous positive score because I still believe adaptive anchors could conceptually yield higher performance. Could you provide some analysis or discussion on why the adaptive approach falls behind the fixed anchors. I look forward to seeing the new results.
Thank you for your positive assessment and this helpful follow-up.
We agree that adaptive anchor placement is conceptually appealing because informative states may occur at different points in different responses. Our adaptive implementation selects the largest critic hidden-state change within each segment, and we conducted further diagnostics to understand why the tested hidden-change rule did not provide a clear improvement over fixed anchors.
We compared the two selection rules on saved trajectories from 64 prompts across four checkpoints through the training process. We find that hidden-change selection increases representation diversity but produces less uniform coverage.
From a representation perspective, adaptive selection indeed increases representation diversity. It reduces mean pairwise hidden-state cosine similarity by ~0.45% relative to fixed anchors, near 99% (in absolute value number). Since the selected representations remain highly similar, we believe that this modest reduction in similarity is insufficient to yield a noticeable improvement in policy performance.
From an anchor placement perspective, our diagnostics show that adaptive selection favors response openings and positions near line breaks, leaving larger gaps between the main anchors. This suggests that maximizing local representation changes may not allocate the supervision points effectively across the full reasoning process. Fixed anchors provide more uniform coverage, which may better support learning value changes throughout long reasoning trajectories.
Table. Anchor position statistics for fixed and adaptive selection.
| Position | Fixed Anchors | Adaptive Anchors |
|---|---|---|
| Mean first-anchor position | 30.00% | 10.50% |
| Anchor in beginning 5% | 0% | 46.90% |
| Anchors in a '\n' | 15.00% | 51.40% |
| Mean largest gap | 30.20% | 47.70% |
Although adaptive selection does not improve overall performance in this comparison, it achieves 1.06 percentage points higher accuracy in the 0–4K bin and better than PPO baseline at least. Fixed anchors perform better in all five longer-response bins, with advantages ranging from 2.52 to 7.08 percentage points. We think that local-change selection may identify useful supervision points in shorter responses, where gaps between anchors span fewer tokens and uneven coverage may matter less. We see how anchor placement interacts with response length as an open question and consider further controlled experiments on this mechanism a valuable direction.
We believe that adaptive anchor placement is a valuable direction for further improving critic learning, and we thank the reviewer for this insightful suggestion. More experiments are needed to understand how its benefits depend on the selection criterion, supervision coverage, and response length. We plan to investigate these questions and explore improved adaptive strategies in future work.
Response to Reviewer 6GgM (1/2): W1–Q1
Thank you for your careful reading and constructive comments. We provide further clarification below.
W1:
We appreciate the reviewer’s concern. In fact, we have already reported coding results across several benchmarks in Sec. 5.3 and Fig. 8 of our paper: SP³O improves the average score over PPO by 2.38 percentage points. This suggests that SP³O's benefits can extend beyond mathematical reasoning to coding. The other domains raised by the reviewer require further training experiments and empirical validation, which we consider an important direction for future work.
Table. Performance comparison on coding benchmarks (corresponding to Fig. 8 in our paper).
W2:
Thank you for the valuable suggestion. Our work focuses on identifying value flattening as a failure mode of PPO critics, investigating its possible causes, and exploring potential solutions.
Reducing value flattening can help the critic capture meaningful value differences across states. This can provide more informative token-level advantages and guide better policy optimization. Better credit assignment and reduced gradient noise are therefore part of our proposed explanation. This interpretation is consistent with the improved critic estimates against MC references in Figure 4, and the more stable actor updates and higher validation accuracy in Figure 5.
We do not claim that value flattening is the primary bottleneck in policy optimization. However, we do consider critic quality an important factor in effective policy optimization [1, 2]. Consistent with this view, our additional experiments(below table) showed that moderately increasing critic updates(better critic accuracy), following Wang et al. [1], improved PPO performance.Policy optimization is a complex process influenced by multiple factors, and other mechanisms may also contribute to the observed improvements. We will clarify the distinction between our measured improvements and the proposed causal explanation.
Table. Performance on Qwen3-4B-Base PPO
Q1:
Thank you for this insightful suggestion. Our fixed-anchor design is motivated by the two factors discussed in the paper: the implicit variance penalty in the sampled loss and similar updates from nearby states. Supervising a few well-separated positions is a simple empirical choice designed to address both. Our placement ablation shows that both tested fixed schedules outperform PPO, with a modest difference of 0.92 percentage points between them.
We also tested selecting the token with the largest pre-update critic hidden-state change within each segment. Its accuracy was 2.75% higher than the original PPO baseline, but 1.48% lower than fixed-anchor SP³O rerun under the same code and training budget.
Together with our critic diagnostics, these results support well-spaced fixed anchors as a simple empirical approach to reducing value flattening. More refined adaptive selection remains a valuable direction, and we will explore additional designs and broader experiments in future work.