- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines the learning behavior of PPO critics in LLM RL. The authors analyze critic-derived learning signals at the task, group, and token levels. The analysis identifies two failure modes: "soft silencing" and "length preference". The authors propose Critic-Policy Sample Decoupling (CPSD), which allows different sample selection rules for critic and policy training processes. Experiments cover mathematical reasoning with two base models and agentic coding. CPSD improves the augmented PPO baseline in mathematical reasoning and mitigates instability in agentic coding, and its coding performance is comparable to GRPO.
Critical strengths
- This paper provides a useful empirical analysis of PPO critics beyond overall training performance. The analysis connects three scales in two settings. It makes a concrete comparison between PPO and GRPO.
- This paper proposes CPSD as an effective method that improves the PPO baseline on both models. The coding experiments also demonstrate a practically relevant training failure.
Critical weaknesses
- Figure 4 shows smaller average advantage penalties for longer failed trajectories. However, this does not directly show that policy updates favor longer responses, as averaging over additional tokens with near-zero advantages can dilute earlier penalties. The claimed mechanism of length preference needs stronger evidence.
- The exact benefit of critic-policy decoupling is unclear. CPSD excludes groups from policy optimization while retaining them for critic training. An ablation is needed to isolate the benefit of preserving broader coverage for the critic.
Questions for authors
- In Figure 2, are the advantages shown before or after whitening? How does the GRPO-like pattern mentioned in the paper differ from task-difficulty awareness?
- The paper highlights PPO’s support for single-sample rollouts and asynchronous training. How does CPSD’s group-based sample selection affect these advantages?
- How consistent are the results across different training seeds?
- What is the additional training time and memory overhead of CPSD compared with PPO and GRPO?
6 comments
Response to Reviewer cZG6 (2/2)
Question
- GRPO-like pattern mentioned vs. task-difficulty awareness
The advantages reported in this paper are whitened, as advantage whitening is a common default in PPO training. In this setting, whitened advantages are the signals actually used for policy updates, making their analysis more suitable for understanding these updates.
For the terminology, the "GRPO-like pattern" largely corresponds to task-difficulty awareness. Without relying on group-averaged rewards, the critic learns the difficulty of individual problems and yields higher advantages for correct responses to harder problems. The advantage curve also converges toward the shape observed in GRPO. We use "GRPO-like pattern" to emphasize the similarity and set the stage for our subsequent findings about PPO that build on this connection.
- CPSD's group-based sample selection
In the experiments presented in Section 5, we use group rewards as the filtering criterion to match the sample selection strategy of GRPO with dynamic sampling. The motivation is to enable a controlled comparison and thereby help us understand the critic's benefits during training. Under this setting, using group rewards does require waiting for rollouts within each group and thus limits asynchronous training.
However, CPSD in its general form does not require rollouts within a group to wait for one another. In Appendix E.3, we note that advantage amplitude is strongly correlated with sample informativeness and can serve as a filtering feature independent of group metrics. We have added experiments using advantage amplitude for filtering (see the revised manuscript for details; Page 23), so that filtering itself does not rely on group information and can support fully asynchronous training. The results show that this approach maintains stable training and comparable performance (Table 4). We hope these clarifications help address your question.
Table. 4 Performance comparison across reasoning benchmarks.
| Method | AIME 24 | AIME 25 | AIME 26 | AMC 23 | Beyond AIME | Brumo | HMMT 25-Feb | GPQA | LiveCodeBench | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| GRPO | 31.25 | 18.54 | 16.30 | 76.41 | 6.53 | 13.18 | 2.86 | 37.85 | 12.26 | 23.91 |
| PPO-CPSD (acc) | 36.46 | 18.70 | 16.61 | 77.89 | 6.39 | 15.73 | 3.44 | 39.08 | 12.95 | 25.25 |
| PPO-CPSD (adv) | 35.95 | 20.00 | 16.77 | 77.34 | 6.17 | 15.05 | 3.91 | 38.32 | 12.53 | 25.12 |
- Results across different training seeds
We have conducted an additional training run of PPO-CPSD, with the results shown in Table 5. Neither run exhibits pronounced training instability, and achieve similar performance.
Table. 5 Performance comparison across reasoning benchmarks.
| Method | AIME 24 | AIME 25 | AIME 26 | AMC 23 | Beyond AIME | Brumo | HMMT 25-Feb | GPQA | LiveCodeBench | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| PPO-CPSD (Try-1) | 36.46 | 18.70 | 16.61 | 77.89 | 6.39 | 15.73 | 3.44 | 39.08 | 12.95 | 25.25 |
| PPO-CPSD (Try-2) | 35.68 | 19.11 | 16.30 | 76.56 | 6.52 | 16.25 | 3.65 | 39.36 | 13.13 | 25.17 |
- Training time and memory overhead
Thank you for this question. Regarding memory overhead, CPSD introduces no additional model parameters compared with PPO and incurs only minor overhead, mainly from additional rollout buffering. Compared with GRPO, the main memory overhead comes from the critic model and its training states.
Regarding training time, we would like to clarify the two factors affecting the runtime of PPO-CPSD relative to GRPO:
-
Runtime overhead from critic: PPO-CPSD maintains an additional critic model that is trained at each step. In LLM RL, rollouts account for most of the runtime, while critic training takes roughly as much time as policy training. In agentic coding settings, critic training adds 13.1% runtime overhead generally.
-
Potential speedups from asynchronous training: Potential speedups from asynchronous training: However, PPO-CPSD also offers potential runtime savings. It supports more aggressive asynchronous settings, such as advantage amplitude filtering, which can reduce waiting among rollouts within each group. The resulting speedups depend strongly on the RL framework’s asynchronous scheduling strategy and implementation, with greater benefits expected for longer trajectories.
Compared with PPO, the additional runtime of PPO-CPSD mainly comes from generating extra samples, analogous to the overhead of GRPO with dynamic sampling relative to GRPO without it. The per-step overhead depends on the proportion of all-correct and all-incorrect groups at that step and ranges from approximately 80% to 250% in agentic coding. However, this extra sampling time should be understood as the cost of obtaining sufficient informative training samples rather than simply as overhead. Without dynamic sampling, some samples contribute zero or near-zero gradients, which can amplify latent biases and training instability.
Thank you for the additional analyses. The PPO-dyn ablation addresses the second weakness. It keeps the policy filtering rule fixed and shows the benefit of retaining all samples for critic training. Figure 13 also addresses my concern about averaging over later tokens. However, the figure still compares advantage estimates for generated responses, and it does not show that PPO updates favor longer responses. The claimed mechanism of length preference still needs stronger evidence. I am keeping my rating unchanged for now.
Response to Reviewer cZG6 (1/2)
Thank you for your detailed feedback. We will respond to your comments point by point below. Revisions to the manuscript are highlighted in blue.
Weakness 1
Thank you for this comment. We provide additional empirical evidence that we hope could help address your concern.
To rule out dilution by later tokens as the sole explanation for this pattern, we compare advantages at the same token positions across trajectories of different lengths (Table 1). As shown in Tab. 1, Longer trajectories still receive weaker penalties at matched positions, ruling out dilution by later tokens as a standalone explanation.
We further examine whether longer failed trajectories still receive weaker penalties than shorter ones when both are generated for the same prompt. Specifically, we compute one Spearman correlation coefficient ρi for each group i analyzed in Figure 4, using trajectory lengths and advantages within that group. As shown in Table 2, the distribution of {ρi} is shifted toward positive values, with a mean of 0.5. Figure 13 (Page 20) in the paper shows the visualization, which is also available via an anonymous link for your convenient viewing (https://anonymous.4open.science/r/Fig13-additional-length-analysis-28D0/). These results indicate that longer failed trajectories tend to receive weaker penalties even within the same prompt, providing complementary evidence of length preference.
Additionally, we emphasize that length preference is a general phenomenon for the critic. Similar length preference is also observed in mathematical reasoning settings at the early stage of critic training (see Figure 14 in the paper at Page 20 (https://anonymous.4open.science/r/Fig14-length-analysis-6696/)). An intuitive picture for this phenomenon is that longer trajectories tend to have lower success rates across the overall data distribution, so the critic may learn to assign them lower values generally. The critic may exploit this tendency even among trajectories generated for the same prompt, translating lower value estimates into weaker penalties for incorrect trajectories. In agentic coding, where trajectories are longer and rewards are sparser, this effect may substantially bias optimization.
We hope the additional evidence on length preference presented above helps address your concern.
Table 1. Advantage at different token positions for failed trajectories grouped by response length.
Table 2. Quantiles and summary statistics of Spearman ρ.
Weakness 2
Thank you for your suggestion. We have added an ablation study to evaluate the effectiveness of preserving broader data coverage for the critic. Starting from PPO-CPSD, we filter out samples from all-correct and all-incorrect groups for both critic and policy updates and denote this variant as PPO-dyn. Table 3 reports its performance, and Figure 9 in the paper (Page 9) shows the training reward and explained variance dynamics (https://anonymous.4open.science/r/Fig9-critic-broad-convergence-A468/). PPO-dyn exhibits substantially slower critic convergence, accompanied by lower training rewards and worse performance. These results show that broader data coverage can improve critic learning and ultimately benefit policy training.
Table. 3 Performance comparison across reasoning benchmarks.