acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

Abstract

In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PPO critics, which we call **value flattening**: state values, estimated from multiple Monte Carlo continuations, change sharply across intermediate states while critic predictions remain comparatively flat.We further observe this phenomenon in a controlled FrozenLake environment and find that it becomes more pronounced as the state space grows.Our theoretical and empirical analyses relate Value Flattening to an implicit variance penalty in the sampled loss and redundant updates from temporally correlated states with similar gradients. Motivated by these findings, we introduce SParse Proximal Policy Optimization (SP³O), which applies the value loss to only a few well-separated states in each response to mitigate both effects. Experiments on Qwen3-Base show that SP³O with only three states supervised per response can mitigate Value Flattening and consistently improve the learned policy across model sizes and evaluation suites. Together, our results identify Value Flattening as an important yet overlooked failure mode of critic learning in standard PPO and show that a simple sparse supervision strategy can mitigate it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6GgMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a systematic failure mode in PPO critics called Value Flattening and proposes a simple, effective mitigation strategy. When state values are estimated via multiple Monte Carlo (MC) continuations from the same policy, these MC values often exhibit sharp local transitions across intermediate states. However, the PPO critic's predictions remain comparatively flat and sometimes even move in the opposite direction. To mitigate the problem, this paper proposes SP3OSP^3O, which trains the critic on only a few well-separated states per response instead of on every token position. Experiments on Qwen3-4B-Base and Qwen3-8B-Base show that SP3OSP^3O consistently outperforms standard PPO and GRPO on mathematical and out-of-distribution (OOD) reasoning benchmark.

Critical strengths
  1. Identifies a novel and concrete failure mode. “Value Flattening” is indeed a well-defined phenomenon: the PPO critic predicts flat values while Monte Carlo estimates show sharp local transitions. This is a useful diagnostic contribution beyond just proposing another algorithm.

  2. Systematic empirical evidence. The paper shows the phenomenon across training checkpoints, correct/incorrect responses, and in a controlled FrozenLake environment. Reproducing it outside LLMs strengthens the claim that it is not an artifact of one setup.

  3. Propose a simple, practical solution. SP3OSP^3O only changes where the value loss is applied, making it easy to implement on top of PPO. It does not require a new architecture or reward model.

Critical weaknesses
  1. Narrow task coverage. Evaluation focuses on mathematical reasoning and OOD reasoning. Generalization to coding, agentic tasks, dialogue, RLHF, multimodal reasoning, or safety-critical applications is not established.

  2. Causal isolation is incomplete. The paper shows that sparse value supervision reduces flattening and improves performance, but it does not fully prove that flattening is the primary causal bottleneck. Other factors, for example, better credit assignment, reduced gradient noise, or regularization could also explain the gains.

Questions for authors
  1. Why only use fixed anchors for updating the critic model (0.3/0.6/0.9 + 0.95 tail). Why not adaptive anchors: split into K segments and pick the max hidden-state change per segment, or other more principal approach to determine when the critic should be updated? It keeps sparsity/separation and targets semantic shifts, but hidden-state change ≠\neq value change, anchors shift during training, and top changes may cluster or be noise. Can you do some additional experiments to test this and see if it can beat fixed anchors or destabilize the critic?

  2. Does it survive dense/process rewards? Your mechanism assumes the same terminal reward RR at all anchors. With token-level rewards, for example, we obtain a PRM somehow, is flattening still the issue?

  3. You report SP³O gives longer responses than PPO and the tail anchor cuts repetition from 18.33% to 1.12%. Could the gains be a length/repetition artifact rather than better critic values? Please report length-controlled results: same max token budget, length-matched comparisons, accuracy vs. length, repetition as a covariate, tail-anchor ablation at matched length, and short-answer tasks. How much of the +7.97+7.97 in-domain and +7.33+7.33 OOD gains remains? If it vanishes at matched length, the main effect may be generation stability, not fixing value flattening. How sensitive are KK and anchor placement to task, response length, and model scale? Is K=3K=3 universal or tuned on math?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6GgM (1/2): W1–Q1

Thank you for your careful reading and constructive comments. We provide further clarification below.

W1:

We appreciate the reviewer’s concern. In fact, we have already reported coding results across several benchmarks in Sec. 5.3 and Fig. 8 of our paper: SP³O improves the average score over PPO by 2.38 percentage points. This suggests that SP³O's benefits can extend beyond mathematical reasoning to coding. The other domains raised by the reviewer require further training experiments and empirical validation, which we consider an important direction for future work.
Table. Performance comparison on coding benchmarks (corresponding to Fig. 8 in our paper).

MethodSciCodeHumanEvalHumanEvalFix PyHumanEvalFix RustLiveCodeBench Exec v2LiveCodeBench Codegen v6Avg.
PPO32.6493.5082.7260.5797.8457.6070.81
SP³O34.6195.1284.5560.7796.8767.2073.19

W2:

Thank you for the valuable suggestion. Our work focuses on identifying value flattening as a failure mode of PPO critics, investigating its possible causes, and exploring potential solutions.
Reducing value flattening can help the critic capture meaningful value differences across states. This can provide more informative token-level advantages and guide better policy optimization. Better credit assignment and reduced gradient noise are therefore part of our proposed explanation. This interpretation is consistent with the improved critic estimates against MC references in Figure 4, and the more stable actor updates and higher validation accuracy in Figure 5.
We do not claim that value flattening is the primary bottleneck in policy optimization. However, we do consider critic quality an important factor in effective policy optimization [1, 2]. Consistent with this view, our additional experiments(below table) showed that moderately increasing critic updates(better critic accuracy), following Wang et al. [1], improved PPO performance.Policy optimization is a complex process influenced by multiple factors, and other mechanisms may also contribute to the observed improvements. We will clarify the distinction between our measured improvements and the proposed causal explanation.
Table. Performance on Qwen3-4B-Base PPO

SettingActor / critic updates per rolloutValset at rollout 129Valset at rollout 229Valset at rollout 329Valset at rollout 429Evalset at Best step
PPO baseline2 / 2 (1:1)34.62%37.13%35.24%40.58%37.60%
Repeat critic updates-22 / 4 (1:2)34.72%36.74%39.35%42.30%42.22%
Repeat critic updates-52 / 10 (1:5)16.40%17.97%Breakout (training collapse)Breakout (training collapse)—

Q1:

Thank you for this insightful suggestion. Our fixed-anchor design is motivated by the two factors discussed in the paper: the implicit variance penalty in the sampled loss and similar updates from nearby states. Supervising a few well-separated positions is a simple empirical choice designed to address both. Our placement ablation shows that both tested fixed schedules outperform PPO, with a modest difference of 0.92 percentage points between them.
We also tested selecting the token with the largest pre-update critic hidden-state change within each segment. Its accuracy was 2.75% higher than the original PPO baseline, but 1.48% lower than fixed-anchor SP³O rerun under the same code and training budget.
Together with our critic diagnostics, these results support well-spaced fixed anchors as a simple empirical approach to reducing value flattening. More refined adaptive selection remains a valuable direction, and we will explore additional designs and broader experiments in future work.

AuthorsOpenReview ↗

Response to Reviewer 6GgM (2/2): Q2–Q3

Q2:

Thank you for suggesting experiments under different reward structures.
In LLM reasoning tasks, training commonly relies on terminal outcome rewards rather than dense token-level supervision. So we trained CNN-based PPO on the eight Atari games below with two seeds and 1,048,576 decisions per seed. These Atari games provide immediate environment rewards and have more constrained state spaces than the state space of LLM reasoning. In these games, PPO critics preserve meaningful within-trajectory value differences and closely track MC estimates.
MC values are estimated using 64 continuations per sampled state. The SD ratio compares critic and MC value variation within episodes; 1 indicates matched variation. The correlation measures how closely critic predictions track MC values across states. Together, these results indicate that PPO critics preserve meaningful within-episode value differences in these eight games, without pronounced value flattening.
Table. Critic value diagnostics on an eight-game Atari subset (means over two seeds).

MetricFreewayBreakoutSpace InvadersQ*bertAsterixMs. Pac-ManRiver RaidSeaquest
Critic/MC value SD ratio0.7310.8280.9351.0080.9630.7670.9850.923
Critic–MC correlation0.8100.8520.9280.9750.9660.8950.9920.866

Exploring this phenomenon under different reward structures is a complex and valuable direction that needs further study. We thank the reviewer for suggesting this valuable direction and will explore it further in future work.

Q3:

Thank you for suggesting length-controlled comparisons and further analysis of generalization.
Our proposed causal explanation is that better critic estimates improve credit assignment and guide more effective policy optimization during training. Thus, reduced repetition and higher accuracy are both manifestations of improved policy behavior, rather than one causing the other.
In our main text, Figure 4 shows that SP³O’s critic has lower value errors and gradient discrepancies against MC references, alongside better outcome discrimination. This can support more effective policy optimization, consistent with the more stable actor updates and higher validation accuracy shown in Figure 5.

1. Length-controlled comparisons.
PPO and SP³O use identical maximum response lengths: 8,192 tokens during training and 24,576 during evaluation. So both methods follow identical training and evaluation protocols, ensuring a fair comparison. We further analyze the evaluation outputs underlying the paper’s 4B results using 4,096-token length bins.
Within each bin, we retain questions for which both methods have at least one response in that bin. We average scores within each question and method, then weight questions equally.
Table. Qwen3-4B-Base accuracy by generated length on shared questions (%).

Generated lengthShared questionsPPOSP³O
0–4K1,52751.1561.78
4–8K92725.3336.06
8–12K52518.6523.34
12–16K21313.0414.80
16–20K629.5415.02
20–24K2678.8410.04

SP³O achieves higher accuracy in all six question-matched length bins. These are descriptive comparisons conditioned on generated length.

2. Generalization and sensitivity analysis.
Beyond the 4B and 8B experiments, our paper evaluates SP³O on Qwen3-14B-Base trained on Polaris-V2-RL-14K. Compared with PPO, SP³O improves mathematical and OOD reasoning accuracy by 4.12 and 4.31 percentage points, respectively. On Qwen3.5-9B scientific coding, it improves the six-benchmark average by 2.38 points and SciCode subproblem accuracy by 1.97 points. These results support its applicability across model sizes, training datasets, and task domains.Our placement ablation also shows that both 0.2/0.5/0.8 and 0.3/0.6/0.9 outperform PPO, by 7.05 and 7.97 percentage points, respectively.


We hope these clarifications address your concerns. We welcome any further questions or suggestions.

References

[1] Improving Value Estimation Critically Enhances Vanilla Policy Gradient. ICML 2025. https://arxiv.org/abs/2505.19247
[2] High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016. https://arxiv.org/abs/1506.02438

The authors have adequately addressed most of my questions, especially regarding the dense reward. They have also provided stronger evidence for SP³O’s performance in a short time. I will retain my previous positive score because I still believe adaptive anchors could conceptually yield higher performance. Could you provide some analysis or discussion on why the adaptive approach falls behind the fixed anchors. I look forward to seeing the new results.

Thank you for your positive assessment and this helpful follow-up.

We agree that adaptive anchor placement is conceptually appealing because informative states may occur at different points in different responses. Our adaptive implementation selects the largest critic hidden-state change within each segment, and we conducted further diagnostics to understand why the tested hidden-change rule did not provide a clear improvement over fixed anchors.

We compared the two selection rules on saved trajectories from 64 prompts across four checkpoints through the training process. We find that hidden-change selection increases representation diversity but produces less uniform coverage.

From a representation perspective, adaptive selection indeed increases representation diversity. It reduces mean pairwise hidden-state cosine similarity by ~0.45% relative to fixed anchors, near 99% (in absolute value number). Since the selected representations remain highly similar, we believe that this modest reduction in similarity is insufficient to yield a noticeable improvement in policy performance.

From an anchor placement perspective, our diagnostics show that adaptive selection favors response openings and positions near line breaks, leaving larger gaps between the main anchors. This suggests that maximizing local representation changes may not allocate the supervision points effectively across the full reasoning process. Fixed anchors provide more uniform coverage, which may better support learning value changes throughout long reasoning trajectories.

Table. Anchor position statistics for fixed and adaptive selection.

PositionFixed AnchorsAdaptive Anchors
Mean first-anchor position30.00%10.50%
Anchor in beginning 5%0%46.90%
Anchors in a '\n'15.00%51.40%
Mean largest gap30.20%47.70%

Although adaptive selection does not improve overall performance in this comparison, it achieves 1.06 percentage points higher accuracy in the 0–4K bin and better than PPO baseline at least. Fixed anchors perform better in all five longer-response bins, with advantages ranging from 2.52 to 7.08 percentage points. We think that local-change selection may identify useful supervision points in shorter responses, where gaps between anchors span fewer tokens and uneven coverage may matter less. We see how anchor placement interacts with response length as an open question and consider further controlled experiments on this mechanism a valuable direction.

We believe that adaptive anchor placement is a valuable direction for further improving critic learning, and we thank the reviewer for this insightful suggestion. More experiments are needed to understand how its benefits depend on the selection criterion, supervision coverage, and response length. We plan to investigate these questions and explore improved adaptive strategies in future work.

Reviewer_Ez4GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identified a value flattening problem in PPO, where neural critics smoothed value prediction over regions with highly volatile value change in MC is used. It further connects with value flattening problem with an implicit variance penalty and gradient correlation across temporally related states. It then empirically shows that by reducing correlated gradient updates, such value flattening is mitagated and the proposed SPO can improve "small" LLM reasoning to some extend.

Critical strengths
  1. The paper identified a mechanistic problem in PPO critics learning. Which shows with standard PPO setting applied to LLM reasoning tasks, value learning can be too smooth, and does not capture the value value well. A toy Frozen lake study clearly shows the problem gets more severe as problem size increase.

  2. A mitigating solution is identified and tested on LLM reasoning that indeed improved the model. This shows the practical value of this observation.

Critical weaknesses
  1. There lacks a push of PPO itself to mitage the value flattening problem and checking the problem in more general RL setting. An obvious solution is to have much more samples and updates, especially for the FrozenLake problem. The bottleline is that with infinite samples and universal approximation ability of NNs, value estimation do converge, if we keep the policy fixed. So, in theory, this is a limitation to standard PPO setting. There was a paper: "Improving Value Estimation Critically Enhances Vanilla Policy Gradient" , maybe similar thing can happen to PPO. If the mechanism is supposed to be about PPO, then the paper should also test it on more serious classical RL problems such as atari games. It is of course, also possible that the problem only happens for sparse reward environments, in which case, the title is too big.

  2. The varience decomposition argument in eq 5 is a bit far-stretched. The PPO critics learning does not really model a mean value, then regress individual step value to that mean value. In fact, if the arguement holds, you can apply this line to any regression problem, then all regression will have a regression value flattening problem. Besides, no concrete algorithmic suggestion is based on this explaination.

Those points can be addressed by either narrowing the claim or perform more indepth analysis, either theoretically or empirically.

Questions for authors
  1. For FrozenLake is the network CNN or MLP? This does not appear to be specified. I suspect a tabular critics parameterization (hash the configuration into a learnable value) could have different result.

2: Are the authors aware of action chuncking? There are a few paper in offline RL that skip states for value estimation in long horizon tasks. The motivation and exact implementation is different from this paper, but I suspect the mechanism is actually similar.

  1. Have you included all the published sota work using the same baseline model on the same tasks? We understood it is not necessary to beat all varients of improvement as the motivation and methods differ. However, it is still worthy to have an idea about the gap for readers. Meanwhile, there are too many arxiv papers to track, so I am not sure about the AI literature review.
17 comments
AuthorsOpenReview ↗

Response to Reviewer Ez4G (1/4): W1 — Points 1–3

Thank you for your careful reading and constructive comments. We provide further clarification below.

W1:

Thank you for your valuable suggestions. We provide further clarification in four parts below.

1. PPO optimization under finite training budgets.
We agree that, for a fixed policy, sufficient state coverage, model capacity, and optimization can enable accurate value estimation. Our focus is learning under finite rollout budgets, as in widely used LLM reasoning training setups such as DeepSeek-R1 and DAPO [4, 5]. We have only a few rollouts (e.g., 8) for each prompt, and all states in the same rollout share the same target return, leading to the value flattening problem as analyzed in Secs. 4.1-4.2. SP³O is motivated by this: it reduces repeated supervision on nearby states without requiring additional rollouts. We will clarify this scope and present FrozenLake as a fixed-budget illustration, rather than evidence of an unavoidable limitation of PPO.

2. Scope of our research.
Our work focuses on value flattening in PPO critics for LLM reasoning, where supervision commonly consists of terminal outcome rewards. We focus on this scenario because it's widely used training signal for LLM reasoning, particularly when correctness can be verified but intermediate reasoning steps or tokens lack reliable supervision. Understanding critic learning in this de-facto setting is therefore important for practical LLM policy optimization.

Our method is inherently motivated by the specific challenge in LLM reasoning tasks, which was not prominent in traditional RL problems. In traditional RL benchmarks like Atari games, the input dimension and task horizon are not as large as LLM reasoning, and rewards can be received during an episode rather than only at its end. It is feasible to train a Q-network and V-network accompanied by the policy network, and very large batch of rollouts can be collected in simulation. In that case, we may not encounter the value flattening challenge, because Q-learning (e.g., DQN) can provide different return targets for samples in the same trajectory, or we can acquire large-batch MC estimates for each state using tree search. In contrast, our focus is learning under the finite rollout budgets of LLM reasoning, without the prerequisite of learning a Q-function or executing large-batch MC sampling.

We agree that the scope of our claims should be clearer. Our shared-return analysis specifically addresses the LLM reasoning setting described above; we do not claim that the same mechanism applies to all PPO applications or that value flattening occurs exclusively in sparse-reward environments. Its behavior under dense or process rewards remains an open question. We will state these boundaries explicitly in the main text and revise overly broad wording so that the paper accurately reflects its focus on critic learning for LLM reasoning.

3. Additional experiments on classical RL tasks.
Although Atari is beyond our paper’s main focus, we tested PPO on 8 games. The eight games below did not show the pronounced flattening pattern. We used CNN-based PPO with two seeds per game, 1,048,576 decisions per seed, and 64 MC continuations per sampled state.
Table. Value diagnostics for the eight games without pronounced flattening (means over two seeds; within-episode MC variance corrected for sampling noise).

MetricFreewayBreakoutSpace InvadersQ*bertAsterixMs. Pac-ManRiver RaidSeaquest
Critic/MC value SD ratio0.7310.8280.9351.0080.9630.7670.9850.923
Critic–MC correlation0.8100.8520.9280.9750.9660.8950.9920.866

We think that is because value learning is more tractable in these games. LLM reasoning has a vast state space, where exact prefixes are rarely revisited. With few independent rollout outcomes, state coverage, cross-trajectory averaging and generalization become difficult. This can allow finite-sample flattening pressure to persist during critic optimization. Our work focuses on this LLM setting and broader classical-RL behavior requires further investigation.We will clarify this scope in the main text and revise overly broad claims accordingly.

AuthorsOpenReview ↗

Response to Reviewer Ez4G (2/4): W1 — Point 4

4. Additional experiments with other critic optimization method.
We thank the reviewer for drawing our attention to the value estimation study by Wang et al. [1]. Their work improves value estimation by allocating more optimization steps to the critic, an approach that is largely orthogonal to ours. Inspired by this idea, we compared actor-to-critic update ratios of 1:1, 1:2, and 1:5 in Qwen3-4B-Base PPO. Relative to PPO, the 1:2 actor-to-critic update setting improved performance, whereas the 1:5 setting collapsed during training.
Table. Performance on Qwen3-4B-Base PPO

SettingActor / critic updates per rolloutValset at rollout 129Valset at rollout 229Valset at rollout 329Valset at rollout 429Evalset at Best step
PPO baseline2 / 2 (1:1)34.62%37.13%35.24%40.58%37.60%
Repeat critic updates-22 / 4 (1:2)34.72%36.74%39.35%42.30%42.22%
Repeat critic updates-52 / 10 (1:5)16.40%17.97%Breakout (training collapse)Breakout (training collapse)—

These results underscore the importance of critic learning for PPO in LLM reasoning and motivate a closer study of value flattening and ways to address it. We appreciate the reviewer’s insight and will explore how additional critic optimization strategies can be combined with our method.

AuthorsOpenReview ↗

Response to Reviewer Ez4G (3/4): W2–Q1

W2:

Thank you for raising these concerns. We clarify three points.

1. What the PPO critic learns. Equation (5) is an algebraic decomposition of the existing critic loss. It does not describe a procedure that first learns a response mean and then regresses individual predictions toward it. In our setting, PPO regresses each state-conditioned prediction Vϕ(st)V_\phi(s_t) to the sampled terminal return RR; the pointwise optimum of the population MSE is Vπ(st)=E[R∣st]V^\pi(s_t)=\mathbb{E}[R\mid s_t].

2. Why this does not apply to all regression problems. For general regression, let vtv_t denote the prediction and yty_t its target, with vˉ=1T∑t=1Tvt\bar v=\frac{1}{T}\sum_{t=1}^{T}v_t and yˉ=1T∑t=1Tyt\bar y=\frac{1}{T}\sum_{t=1}^{T}y_t. The squared loss admits the exact decomposition
1T∑t=1T(vt−yt)2=(vˉ−yˉ)2+1T∑t=1T[(vt−vˉ)−(yt−yˉ)]2.\frac{1}{T}\sum_{t=1}^{T}(v_t-y_t)^2=(\bar v-\bar y)^2+\frac{1}{T}\sum_{t=1}^{T}\left[(v_t-\bar v)-(y_t-\bar y)\right]^2.
The second term measures the mismatch between the variation in predictions and that in targets.
A key property of our setting is that all states within a sampled trajectory share the same terminal-return target: yt=Ry_t=R for all tt. Consequently, yˉ=R\bar y=R and yt−yˉ=0y_t-\bar y=0, yielding
1T∑t=1T(vt−R)2=(vˉ−R)2+1T∑t=1T(vt−vˉ)2⏟Var⁡t(vt).\frac{1}{T}\sum_{t=1}^{T}(v_t-R)^2=(\bar v-R)^2+\underbrace{\frac{1}{T}\sum_{t=1}^{T}(v_t-\bar v)^2}_{\operatorname{Var}_t(v_t)}.
Thus, the prediction-variance term arises from the shared target within each trajectory. With varying targets, the centered term instead encourages predictions to match the target variation. Our argument therefore does not imply that all regression problems suffer from flattening.

3. What the analysis motivates algorithmically. We use this identity to examine the empirical supervision structure in terminal-reward LLM reasoning. Under finite rollout budgets, dense critic supervision fits many states within each trajectory to the same sampled terminal return. This motivates Eq. (6): supervising a few well-separated positions while retaining the original return targets and token-level value predictions. The analysis concerns potential flattening pressure during finite-batch optimization; it does not establish inevitable flattening or a bias in the population optimum.

Q1:

Thank you for your careful attention to experimental setup. Figure 2 uses a MLP and we will state this explicitly in the revised paper.
We also evaluated a tabular critic that assigns an independent value to each nonterminal grid state. The learned table has one trainable logit per state, followed by a Sigmoid. We fitted it on the same sampled trajectories as the MLP used. We additionally report a direct MC table, whose entry is successes/visits for that state. Direct MC requires no optimizer and serves as a diagnostic rather than an optimization-matched critic. Results average five maps and 20 training seeds per map.
MAE is computed over visited states. Slope regresses predicted on true value differences across adjacent visited states; lower values indicate stronger flattening.(ideal to 1)
Table 1. FrozenLake critic comparison with 512 rollouts per run (visited-state MAE and visited-edge slope).

Maze sizeMLP MAE ↓MLP SlopeLearned table MAE ↓Learned table SlopeDirect MC MAE ↓Direct MC Slope
320.0820.2950.1530.6170.1520.972
640.1120.1950.1780.5080.2340.818
1280.1470.1230.1950.2890.2850.468

Table 2. FrozenLake critic comparison with matched visitation (visited-state MAE and visited-edge slope). Rollout budgets are 512, 1,922, and 7,394 for maze sizes 32, 64, and 128, respectively.

Maze sizeMLP MAE ↓MLP SlopeLearned table MAE ↓Learned table SlopeDirect MC MAE ↓Direct MC Slope
320.0820.2950.1530.6170.1520.972
640.0720.1750.1500.6140.1660.951
1280.0880.1040.1480.6170.1740.955

We find that both tabular methods have higher slopes than the MLP, so they alleviate value flattening. But their slopes still fall as maze size grows, showing that the trend reported in the main text persists under this fixed budget. Furthermore,we also increased the rollout budget with maze size to keep visitation per state approximately constant. The learned-table slope then stays around 0.61 and the direct-MC slope around 0.95 across all three sizes in line with the reviewer's expectation.
Our study is motivated by the challenges of RLVR for LLM reasoning, where state spaces are large and sampling rollouts is very costly.Under these conditions, it is unrealistic to excessively increase the budget and use tabular methods,so the value flattening phenomenon is not easy to solve. We will clarify the scope of our study and conclusions in the main text.

Reviewer_DJQ9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies value flattening: a PPO critic can predict nearly constant values along a response even when its probability of success changes. Repeated continuations from intermediate prefixes provide diagnostic value estimates. SP^3O trains the critic at a few spaced positions while using its predictions throughout the response. The proposed explanation involves shared return targets and similar gradients at nearby positions. Experiments cover FrozenLake, reasoning, and coding with several Qwen models.

Critical strengths
  • Distinguishing response-level accuracy from tracking value changes within a response is useful. Figure 1 makes the issue concrete, and the representation and gradient measurements help examine it. Appendix A.2 carefully distinguishes finite-batch effects from the population optimum and expected-gradient invariance; a short explanation of these limits would also help the main text.

PS: about the figure (nitpick): I would move the labels of each subplot and position them strategically to not overlap the curves.

  • The intervention is simple and the reported gains are substantial. On Qwen3-4B, mathematical accuracy rises from 37.60% to 45.57%, and general-reasoning accuracy from 51.95% to 59.28% (Tables 1–2). Results at larger scales and on coding strengthen the contribution. The placement, density, and tail-coverage ablations provide useful controls.
Critical weaknesses
  • Key concepts need clearer motivation when introduced. I found the roles of groups, advantages, and supervised states insufficiently connected when introduced. Lines 41–45 and 187–197 do identify the Monte Carlo estimates as diagnostic references. However, Eq. (4) resembles a GRPO group mean: it would be nice if the authors provide an explaination there that these continuations share an intermediate prefix, how this differs from grouping complete responses to a prompt, and which samples are used for training. This would also clarify how a learned critic avoids needing a fresh continuation group at every prefix and what that means for batch diversity. With γ=λ=1\gamma = \lambda = 1, Eq. (2) and lines 157–161 imply A^t=R(τ)=Vϕold\hat{A}_t = R(\tau) = V_{\phi_{old}}. I think the authors should state this directly and explain why the state-dependent baseline remains useful when the return target is constant; the discussion at lines 342–346 comes late in my opinion. Likewise, it would be better to define the supervised positions beside Eq. (6).

  • The cause of the gains remains unclear. I am less convinced by the causal explanation than by the practical gains. FrozenLake should be presented as a toy illustration under a fixed training configuration: visitation also decreases with maze size (Figure 2(c)), so lines 219–227 do not isolate state-space size. At lines 90–94, the observed gradient similarity need to be distinguished from its proposed effect on flattening. The conditions behind the value-head argument are not stated at lines 253–255; I think it should require similar hidden representations and prediction errors pointing in the same direction (both predictions above or both below the sampled return). Eq. (5) is a valid loss identity, but uniformly sampling positions preserves the expected mean loss for a fixed response. Also, Eq. (3) weights responses by length, whereas Eq. (6) weights them by their three or four anchors. How much comes from spacing, finite-batch optimization, or response reweighting? The main text needs this qualification before attributing the gains to a reduced variance penalty.

Questions for authors
  • At Figure 5(d), do you have further explanations on the reason of longer trajectories with SP^3O? Did you analyze reasoning traces? Do you observe more "doom-looping" behaviors? Note that I see this is briefly discussed at lines 436–439, but I thinks this deserves more discussion.
  • Does SP^3O retain its gains against dense supervision with matched response weights and optimizer settings? How does adding the fourth anchor affect these weights?
  • What are the training-time and memory costs relative to PPO and GRPO? A learned critic and longer responses both have costs; this trade-off should accompany the motivation for retaining a critic.
9 comments
AuthorsOpenReview ↗

Response to Reviewer DJQ9 (1/3): W0–W1

Thank you for your careful reading and constructive comments. We provide further clarification below.

W0:

PS: about the figure (nitpick)

Thank you for your careful reading and attention to detail. We have repositioned the subplot labels of Figure 1 to avoid overlap with the curves and improve readability.

W1:

Thank you for suggesting clearer explanations of the key concepts and their connections.
1. Clarification of key concepts and Monte Carlo value estimation.
Given a prompt xx and a sampled response y=(y1,…,yT)y=(y_1,\ldots,y_T), the state st=(x,y<t)=(x,y1,…,yt−1).s_t=(x,y_{<t})=(x,y_1,\ldots,y_{t-1}).
We fix this prompt and intermediate prefix, and independently sample KK continuations from the same actor policy: y~≥t(k)∼i.i.d.π(⋅∣x,y<t),k=1,…,K.\tilde y_{\ge t}^{(k)}\overset{\mathrm{i.i.d.}}{\sim}\pi(\cdot\mid x,y_{<t}),\qquad k=1,\ldots,K.
The corresponding completed trajectory is τt(k)=(x,y1,…,yt−1,y~t(k),…,y~Tk(k)).\tau_t^{(k)}=\left(x,y_1,\ldots,y_{t-1},\tilde y_t^{(k)},\ldots,\tilde y_{T_k}^{(k)}\right).
The prompt and prefix tokens are identical across all continuations; Only the remaining tokens are resampled. Averaging the terminal rewards of these(KK) continuations gives the MC estimate of the conditional expected return:
V^MCπ(st)=1K∑k=1KR(τt(k)),Vπ(st)=Eπ[R(τ)∣st].\hat V_{\mathrm{MC}}^\pi(s_t)=\frac1K\sum_{k=1}^K R(\tau_t^{(k)}),\qquad V^\pi(s_t)=\mathbb E_\pi[R(\tau)\mid s_t].
In contrast, GRPO samples complete responses conditioned on the original prompt:
y(k)=(y1(k),…,yTk(k))∼i.i.d.π(⋅∣x),k=1,…,K.y^{(k)}=(y_1^{(k)},\ldots,y_{T_k}^{(k)})\overset{\mathrm{i.i.d.}}{\sim}\pi(\cdot\mid x),\qquad k=1,\ldots,K.
Its group mean reward is Rˉ(x)=1K∑k=1KR(τ(k))\bar R(x)=\frac1K\sum_{k=1}^K R(\tau^{(k)}), where τ(k)=(x,y(k))\tau^{(k)}=(x,y^{(k)}). GRPO uses group rewards to construct relative advantages; Our continuation average instead estimates the value of a fixed intermediate prefix for diagnostics. We will add these clarifications to the main text.
2.Conceptual relationships in critic-based policy optimization.
The equality stated in the comment A^t=R(τ)=Vϕold\hat A_t=R(\tau)=V_{\phi_{\mathrm{old}}} does not follow from Equation (2).
With γ=λ=1\gamma=\lambda=1, the advantage of a given action is its action-value minus the state-value as A(st,at)=Q(st,at)−V(st)A(s_t,a_t)=Q(s_t,a_t)-V(s_t).
Here, we use the single-trajectory Monte-Carlo computation as the action-value estimate as Q(st,at)≈Gt=R(τ)Q(s_t,a_t)\approx G_t=R(\tau), and use a critic to approximate the state-value as V(st)≈Vϕold(st)V(s_t)\approx V_{\phi_{\mathrm{old}}}(s_t). Hence, we have A^t=Gt−Vϕold(st)\hat A_t=G_t-V_{\phi_{\mathrm{old}}}(s_t), which is also implied by Eq. (2).
The return target R(τ)R(\tau) is constant for all state-action pairs in trajectory τ\tau due to the single-trajectory MC estimation, but the expected return varies for different state-action pairs. PPO uses a critic network Vϕ(st)V_\phi(s_t) to approximate the expected state-value, which varies for different states in the same trajectory τ\tau. Hence, even when the MC return target remains constant, introducing the critic still works under the critic-based PPO framework.
The vanilla policy gradient is ∇θJ(θ)=Est,at∼πθ[∇θlog⁡πθ(at∣st)⋅Q(st,at)]\nabla_\theta J(\theta)=\mathbb E_{s_t,a_t\sim\pi_\theta}[\nabla_\theta\log\pi_\theta(a_t\mid s_t)\cdot Q(s_t,a_t)], where Q(st,at)Q(s_t,a_t) is the expected return when executing action ata_t from state sts_t following policy πθ\pi_\theta. In practice, we use the single-trajectory Monte-Carlo computation as the action-value estimate as Q(st,at)≈Gt=R(τ)Q(s_t,a_t)\approx G_t=R(\tau). As we know, MC estimation can have very large variance (especially when samples are few), hindering the stability and efficiency of policy gradient update. Hence, critic-based PPO introduces a state-dependent baseline function V(st)V(s_t) to reduce the variance of policy gradient estimation as ∇θJ(θ)=Est,at∼πθ[∇θlog⁡πθ(at∣st)⋅(Q(st,at)−V(st))]\nabla_\theta J(\theta)=\mathbb E_{s_t,a_t\sim\pi_\theta}[\nabla_\theta\log\pi_\theta(a_t\mid s_t)\cdot(Q(s_t,a_t)-V(s_t))], where the advantage is A(st,at)=Q(st,at)−V(st)A(s_t,a_t)=Q(s_t,a_t)-V(s_t). That is the mechanism of using a state-dependent baseline.
3. Supervised positions and critic training signals.
Thank you for this suggestion. We will revise the text beside Eq. (6) to specify both the states receiving critic supervision and the target used at those states.
Supervised positions. Let TτT_\tau be the number of valid response tokens in an on-policy trajectory τ\tau. We will define I(τ)⊆{1,…,Tτ}\mathcal I(\tau)\subseteq\{1,\ldots,T_\tau\} as the response positions receiving the critic regression loss. Our default selection takes the valid positions nearest relative locations 0.30.3, 0.60.6, and 0.90.9, and adds 0.950.95 when Tτ≥6,144T_\tau\ge 6{,}144. Critic training targets. In our terminal-reward setting with γ=λ=1\gamma=\lambda=1, we use the sampled trajectory’s terminal reward R(τ)R(\tau) as the value target at each selected state. The extra continuations used to estimate V^MCπ(st)\widehat V_{\mathrm{MC}}^\pi(s_t) are collected only for value diagnostics; they are not used as actor or critic training samples or as critic targets. Thus, Eq. (6) does not require additional rollouts for training.

AuthorsOpenReview ↗

Response to Reviewer DJQ9 (2/3): W2

W2:

1.FrozenLake results with matched state visitation.
We thank the reviewer for pointing out the role of state visitation. We added a visitation-matched control experiment by scaling rollout budgets.We use “fixed → scaled” to compare the fixed 512-rollout budget with the visitation-matched budget. Results are averaged over five maps and 20 training seeds per map.
Table. FrozenLake MLP critic under fixed and scaled rollout budgets (visited-state MAE and visited-edge slope).

Maze size nRolloutsVisits/stateState coverageVisited-state MAE ↓Visited-edge slope ↑
32512 → 5124.151 → 4.1510.899 → 0.8990.082 → 0.0820.295 → 0.295
48512 → 1,3011.633 → 4.1860.678 → 0.8970.095 → 0.0660.254 → 0.247
64512 → 1,9221.106 → 4.1470.575 → 0.8970.112 → 0.0720.195 → 0.175
80512 → 3,2230.659 → 4.1530.420 → 0.8940.120 → 0.0690.179 → 0.187
96512 → 4,2360.502 → 4.1610.354 → 0.8970.128 → 0.0730.149 → 0.142
112512 → 6,0600.351 → 4.1600.269 → 0.8910.140 → 0.0790.145 → 0.136
128512 → 7,3940.287 → 4.1230.231 → 0.8950.147 → 0.0880.123 → 0.104

Although matching visitation substantially reduces MAE, it yields little improvement in the local value-difference slope (obtained by regressing predicted value differences on true value differences across adjacent visited states,ideal to 1). Thus, even with approximately matched visitation,local value flattening still remains more pronounced in larger state spaces.

2.Uniform sampling and the expected loss.
We agree that uniformly sampling positions preserves the expected mean loss for a fixed response as loss=Et[(vt−R)2]\mathrm{loss}=\mathbb E_t[(v_t-R)^2]. In Eq. (5), the conventional method uses the sampled mean to approximate the expected loss as Et[(vt−R)2]≈1T∑t=1T(vt−R)2\mathbb E_t[(v_t-R)^2]\approx\frac1T\sum_{t=1}^T(v_t-R)^2. SP^3O uses only a few samples to approximate the expected loss as Et[(vt−R)2]≈1∣I(τ)∣∑t∈I(τ)(vt−R)2\mathbb E_t[(v_t-R)^2]\approx\frac1{|\mathcal I(\tau)|}\sum_{t\in\mathcal I(\tau)}(v_t-R)^2. Hence, the variance penalty in Eq. (5) comes from the optimization perspective (penalty for all positions vs. sparse positions), not the theoretical expectation.We will make this distinction explicit in the main text.

3. Gradient similarity and the conditions of the value-head argument.
We agree that the connection between the observed gradient similarity and value flattening needs to be explained more clearly. For a linear value head Vϕ(st)=w⊤htV_\phi(s_t)=w^\top h_t, the per-position gradient is$g_t^{(w)}=2\bigl(V_\phi(s_t)-R\bigr)h_t.$Thus, similar hidden representations yield aligned value-head gradients when the prediction errors have the same sign. Errors with opposite signs reverse the gradient alignment. We will make these conditions explicit.
We sampled 81,920 responses at regular intervals during PPO training to analysis. Using the pre-update residuals et=Vϕold(st)−Re_t=V_{\phi_{\mathrm{old}}}(s_t)-R, the table reports sign agreement between adjacent eligible positions, averaged equally across responses.
Table. Residual-sign agreement between neighboring token positions in PPO (%).

ResponsesCountSame sign
All responses81,92098.15
Successful (R = 1)31,73698.57
Unsuccessful (R = 0)50,18497.88

Our measurements suggest an optimization difficulty in learning state-dependent values. Under shared return targets, highly aligned gradients tend to move neighboring predictions together. This makes state-dependent value differences harder to learn under finite rollout budgets and contributes to flattening. This motivates supervising a few well-separated states. We have discussed this in Sec. 4.2 and Fig. 3(b), and will further clarify the alignment conditions and add residual-sign statistics.

4. Response normalization and token weighting.
In our code, the critic loss uses response-level averaging, so trajectories of all lengths receive the same total loss weight.
For a batch of BB responses, let TiT_i denote the response length, Ii\mathcal I_i the anchor set, Ki=∣Ii∣K_i=|\mathcal I_i|, and ℓit\ell_{it} the per-position critic loss. The implemented objectives are
LPPO=1B∑i=1B1Ti∑t=1Tiℓit,LSP3O=1B∑i=1B1Ki∑t∈Iiℓit.\mathcal L_{\mathrm{PPO}}=\frac1B\sum_{i=1}^{B}\frac1{T_i}\sum_{t=1}^{T_i}\ell_{it},\qquad \mathcal L_{\mathrm{SP^3O}}=\frac1B\sum_{i=1}^{B}\frac1{K_i}\sum_{t\in\mathcal I_i}\ell_{it}.
PPO weights each token by 1/(BTi)1/(BT_i), while SP^3O weights only anchors by 1/(BKi)1/(BK_i). Both give each response total weight 1/B1/B, so adding a fourth anchor does not change the weight of each response.

AuthorsOpenReview ↗

Response to Reviewer DJQ9 (3/3): Q1–Q3

Q1:

Thank you for suggesting a closer analysis of response length and reasoning behavior.

1.Analysis of reasoning behavior in SP³O.
SP³O produces longer responses and we find its policy revisits earlier reasoning, reflects on its conclusions, and checks intermediate results. These are desirable behaviors in reasoning tasks. The table below provides a quantitative summary of related reasoning trigger word frequencies.
Table. Statistics of reasoning trigger words (per 1,000 generated tokens).

MethodsothereforethuswaitTotal
PPO1.450.370.360.0372.21
SP³O4.260.330.210.7665.57

As for ablation experiment, we find that without tail supervision on long trajectories, the policy often continues checking and restating the same result after producing a boxed answer. This can lead to unproductive post-answer loops. Adding tail supervision reduces the repetition from 18.33% to 1.12%. It also reduces average post-answer continuation from 3,639 to 151 tokens.

2. Explanations for these behaviors.
SP³O mitigates value flattening and preserves value differences across reasoning steps. This can improve credit assignment, which in turn helps reinforce useful reasoning patterns during training.More frequent reflection and verification steps contribute to the longer responses generated by SP³O.
On successful trajectories, low tail-value estimates can give repetitive continuations positive advantages. The tail anchor directly trains the selected late-state value toward the observed return. This can reduce those positive advantages and weaken the reinforcement of repetition.
How credit assignment shapes behavioral patterns is an interesting and complex question with broad implications, and we consider it an important direction for our future work.

Q2:

Thank you for asking us to clarify response weighting and the effect of the fourth anchor.
SP³O retains its gains over dense PPO with matched response-level weights and optimizer hyperparameters. As detailed in our response to W2, Point 4, both methods assign each response total weight 1/B; adding a fourth anchor does not change the relative weights of long and short responses in the overall batch loss. It only redistributes weight within that response.

Q3:

Thank you for suggesting a comparison of training-time and memory costs.
SP³O changes only where PPO’s critic is supervised, requiring no additional model forward or backward passes and maintaining comparable training costs.Relative to PPO, SP³O uses 1.06× PPO’s total time and 1.07× its aggregate sampled GPU memory. Relative to GRPO, these ratios are 1.13× and 1.59×, respectively.
Table. Qwen3-8B-Base costs at same iteration.

MethodRollout(s/iteration)Update(s/iteration)Total(s/iteration)Sampled peak/GPU(GiB)Sampled peak total(GiB)
GRPO84.4143.7238.729.50220.73
PPO75.0159.5252.345.97329.60
SP³O79.1170.5268.748.81351.96

Despite the additional cost relative to GRPO, actor–critic methods remain valuable for long-horizon reasoning. A learned critic supports token-level credit assignment and single-rollout training [1–3].
A broader comparison between PPO and GRPO is beyond this paper’s scope. However, settings with only one rollout per prompt highlight the need to study PPO-based alternatives to GRPO. At N=1, standard GRPO’s group-centered task-reward advantage vanishes, while PPO and SP³O remain applicable, as our 4B results below show.
Table. Qwen3-4B-Base with N=1 training (avg@32 accuracy, %).

MethodAIME24–26 avg.AMC23MATH500MinervaOlympiadOverall avg.
PPO12.2954.8476.1942.8040.6135.90
SP³O11.4956.9577.1143.1141.6036.18

We hope these clarifications address your concerns. We welcome any further questions or suggestions.

References

[1] Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning. https://arxiv.org/abs/2607.07508
[2] Step-DeepResearch Technical Report. https://arxiv.org/abs/2512.20491
[3] MiniCPM5-2B. Hugging Face model card. https://huggingface.co/openbmb/MiniCPM5-2B

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.