acceptodds
Under review as a conference paper at ICLR 2027

Revealing the Emergent Learning Mechanisms of Critic Models in PPO for LLM Reinforcement Learning

Abstract

Reinforcement learning (RL) has proven effective at improving the reasoning capabilities of large language models (LLMs) and has recently been extended to complex, long-horizon agentic tasks. This shift has renewed interest in Proximal Policy Optimization (PPO) over group-sampling-based methods such as Group Relative Policy Optimization (GRPO), owing to its finer-grained credit assignment and greater compatibility with asynchronous training. While prior work has largely focused on improving the overall performance, the learning dynamics of the critic model have received relatively little attention. In this work, we systematically investigate the emergent behaviors and learning mechanisms of the critic model in PPO across three complementary scales: *task*, *group*, and *token*. At the task level, our analysis of trajectory-level advantages reveals that the critic spontaneously develops a GRPO-like group-based normalization pattern, yet without incurring the cost of group sampling. At the group and token levels, we introduce a belief-ranking analysis to demonstrate that the critic indeed provides fine-grained learning signals, while also revealing its susceptibility to potential bias. Motivated by these findings, we identify potential failure modes of PPO and propose to decouple critic and policy training samples to better exploit informative samples. Our analysis offers a new perspective on PPO and GRPO training dynamics and mechanisms, with implications for RL algorithm design for LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Proximal Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cZG6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the learning behavior of PPO critics in LLM RL. The authors analyze critic-derived learning signals at the task, group, and token levels. The analysis identifies two failure modes: "soft silencing" and "length preference". The authors propose Critic-Policy Sample Decoupling (CPSD), which allows different sample selection rules for critic and policy training processes. Experiments cover mathematical reasoning with two base models and agentic coding. CPSD improves the augmented PPO baseline in mathematical reasoning and mitigates instability in agentic coding, and its coding performance is comparable to GRPO.

Critical strengths
  • This paper provides a useful empirical analysis of PPO critics beyond overall training performance. The analysis connects three scales in two settings. It makes a concrete comparison between PPO and GRPO.
  • This paper proposes CPSD as an effective method that improves the PPO baseline on both models. The coding experiments also demonstrate a practically relevant training failure.
Critical weaknesses
  • Figure 4 shows smaller average advantage penalties for longer failed trajectories. However, this does not directly show that policy updates favor longer responses, as averaging over additional tokens with near-zero advantages can dilute earlier penalties. The claimed mechanism of length preference needs stronger evidence.
  • The exact benefit of critic-policy decoupling is unclear. CPSD excludes groups from policy optimization while retaining them for critic training. An ablation is needed to isolate the benefit of preserving broader coverage for the critic.
Questions for authors
  • In Figure 2, are the advantages shown before or after whitening? How does the GRPO-like pattern mentioned in the paper differ from task-difficulty awareness?
  • The paper highlights PPO’s support for single-sample rollouts and asynchronous training. How does CPSD’s group-based sample selection affect these advantages?
  • How consistent are the results across different training seeds?
  • What is the additional training time and memory overhead of CPSD compared with PPO and GRPO?
6 comments
AuthorsOpenReview ↗

Response to Reviewer cZG6 (1/2)

Thank you for your detailed feedback. We will respond to your comments point by point below. Revisions to the manuscript are highlighted in blue.

Weakness 1

Stronger evidence for mechanism of length preference

Thank you for this comment. We provide additional empirical evidence that we hope could help address your concern.

To rule out dilution by later tokens as the sole explanation for this pattern, we compare advantages at the same token positions across trajectories of different lengths (Table 1). As shown in Tab. 1, Longer trajectories still receive weaker penalties at matched positions, ruling out dilution by later tokens as a standalone explanation.

We further examine whether longer failed trajectories still receive weaker penalties than shorter ones when both are generated for the same prompt. Specifically, we compute one Spearman correlation coefficient ρi\rho_i for each group ii analyzed in Figure 4, using trajectory lengths and advantages within that group. As shown in Table 2, the distribution of {ρi}\{\rho_i\} is shifted toward positive values, with a mean of 0.5. Figure 13 (Page 20) in the paper shows the visualization, which is also available via an anonymous link for your convenient viewing (https://anonymous.4open.science/r/Fig13-additional-length-analysis-28D0/). These results indicate that longer failed trajectories tend to receive weaker penalties even within the same prompt, providing complementary evidence of length preference.

Additionally, we emphasize that length preference is a general phenomenon for the critic. Similar length preference is also observed in mathematical reasoning settings at the early stage of critic training (see Figure 14 in the paper at Page 20 (https://anonymous.4open.science/r/Fig14-length-analysis-6696/)). An intuitive picture for this phenomenon is that longer trajectories tend to have lower success rates across the overall data distribution, so the critic may learn to assign them lower values generally. The critic may exploit this tendency even among trajectories generated for the same prompt, translating lower value estimates into weaker penalties for incorrect trajectories. In agentic coding, where trajectories are longer and rewards are sparser, this effect may substantially bias optimization.

We hope the additional evidence on length preference presented above helps address your concern.

Table 1. Advantage at different token positions for failed trajectories grouped by response length.

Token position1k3k5k7k9k
Long-0.3046-0.3916-0.3491-0.1911-0.1108
Mid-0.3871-0.4562-0.2497-0.2969-0.1331
Short-0.3252-0.5059-0.5310-0.3577-0.2804

Table 2. Quantiles and summary statistics of Spearman ρ\rho.

StatisticP20P40P60P80MeanMedian
Spearman ρ\rho0.20950.50000.66670.82380.49640.5952

Weakness 2

The benefit of preserving broader coverage for the critic

Thank you for your suggestion. We have added an ablation study to evaluate the effectiveness of preserving broader data coverage for the critic. Starting from PPO-CPSD, we filter out samples from all-correct and all-incorrect groups for both critic and policy updates and denote this variant as PPO-dyn. Table 3 reports its performance, and Figure 9 in the paper (Page 9) shows the training reward and explained variance dynamics (https://anonymous.4open.science/r/Fig9-critic-broad-convergence-A468/). PPO-dyn exhibits substantially slower critic convergence, accompanied by lower training rewards and worse performance. These results show that broader data coverage can improve critic learning and ultimately benefit policy training.

Table. 3 Performance comparison across reasoning benchmarks.

MethodAIME 24AIME 25AIME 26AMC 23Beyond AIMEBrumoHMMT 25-FebGPQALiveCodeBenchAvg.
PPO-CPSD36.4618.7016.6177.896.3915.733.4439.0812.9525.25
PPO-dyn32.6618.9115.1675.236.4512.503.7540.0213.4324.23
AuthorsOpenReview ↗

Response to Reviewer cZG6 (2/2)

Question

  1. GRPO-like pattern mentioned vs. task-difficulty awareness

The advantages reported in this paper are whitened, as advantage whitening is a common default in PPO training. In this setting, whitened advantages are the signals actually used for policy updates, making their analysis more suitable for understanding these updates.

For the terminology, the "GRPO-like pattern" largely corresponds to task-difficulty awareness. Without relying on group-averaged rewards, the critic learns the difficulty of individual problems and yields higher advantages for correct responses to harder problems. The advantage curve also converges toward the shape observed in GRPO. We use "GRPO-like pattern" to emphasize the similarity and set the stage for our subsequent findings about PPO that build on this connection.

  1. CPSD's group-based sample selection

In the experiments presented in Section 5, we use group rewards as the filtering criterion to match the sample selection strategy of GRPO with dynamic sampling. The motivation is to enable a controlled comparison and thereby help us understand the critic's benefits during training. Under this setting, using group rewards does require waiting for rollouts within each group and thus limits asynchronous training.

However, CPSD in its general form does not require rollouts within a group to wait for one another. In Appendix E.3, we note that advantage amplitude is strongly correlated with sample informativeness and can serve as a filtering feature independent of group metrics. We have added experiments using advantage amplitude for filtering (see the revised manuscript for details; Page 23), so that filtering itself does not rely on group information and can support fully asynchronous training. The results show that this approach maintains stable training and comparable performance (Table 4). We hope these clarifications help address your question.

Table. 4 Performance comparison across reasoning benchmarks.

MethodAIME 24AIME 25AIME 26AMC 23Beyond AIMEBrumoHMMT 25-FebGPQALiveCodeBenchAvg.
GRPO31.2518.5416.3076.416.5313.182.8637.8512.2623.91
PPO-CPSD (acc)36.4618.7016.6177.896.3915.733.4439.0812.9525.25
PPO-CPSD (adv)35.9520.0016.7777.346.1715.053.9138.3212.5325.12
  1. Results across different training seeds

We have conducted an additional training run of PPO-CPSD, with the results shown in Table 5. Neither run exhibits pronounced training instability, and achieve similar performance.

Table. 5 Performance comparison across reasoning benchmarks.

MethodAIME 24AIME 25AIME 26AMC 23Beyond AIMEBrumoHMMT 25-FebGPQALiveCodeBenchAvg.
PPO-CPSD (Try-1)36.4618.7016.6177.896.3915.733.4439.0812.9525.25
PPO-CPSD (Try-2)35.6819.1116.3076.566.5216.253.6539.3613.1325.17
  1. Training time and memory overhead

Thank you for this question. Regarding memory overhead, CPSD introduces no additional model parameters compared with PPO and incurs only minor overhead, mainly from additional rollout buffering. Compared with GRPO, the main memory overhead comes from the critic model and its training states.

Regarding training time, we would like to clarify the two factors affecting the runtime of PPO-CPSD relative to GRPO:

  • Runtime overhead from critic: PPO-CPSD maintains an additional critic model that is trained at each step. In LLM RL, rollouts account for most of the runtime, while critic training takes roughly as much time as policy training. In agentic coding settings, critic training adds 13.1% runtime overhead generally.

  • Potential speedups from asynchronous training: Potential speedups from asynchronous training: However, PPO-CPSD also offers potential runtime savings. It supports more aggressive asynchronous settings, such as advantage amplitude filtering, which can reduce waiting among rollouts within each group. The resulting speedups depend strongly on the RL framework’s asynchronous scheduling strategy and implementation, with greater benefits expected for longer trajectories.

Compared with PPO, the additional runtime of PPO-CPSD mainly comes from generating extra samples, analogous to the overhead of GRPO with dynamic sampling relative to GRPO without it. The per-step overhead depends on the proportion of all-correct and all-incorrect groups at that step and ranges from approximately 80% to 250% in agentic coding. However, this extra sampling time should be understood as the cost of obtaining sufficient informative training samples rather than simply as overhead. Without dynamic sampling, some samples contribute zero or near-zero gradients, which can amplify latent biases and training instability.

Thank you for the additional analyses. The PPO-dyn ablation addresses the second weakness. It keeps the policy filtering rule fixed and shows the benefit of retaining all samples for critic training. Figure 13 also addresses my concern about averaging over later tokens. However, the figure still compares advantage estimates for generated responses, and it does not show that PPO updates favor longer responses. The claimed mechanism of length preference still needs stronger evidence. I am keeping my rating unchanged for now.

Reviewer_YhKPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

To address the lack of a systematic understanding of how the critic learns in PPO, the paper studies its behavior at the task, group, and token levels. It finds that the critic naturally learns group normalization similar to GRPO and provides more fine-grained advantage signals, but also suffers from soft silencing and length preference. Based on these findings, the authors propose Critic-Policy Sample Decoupling, which trains the critic on all rollouts while selecting informative samples to update the policy, improving training stability and performance.

Critical strengths
  1. The paper provides valuable insights into how the critic learns at the task, group, and token levels. In particular, Figure 2 and Appendix Figure 12 show that it can learn advantage patterns similar to GRPO even without explicit group sampling.

  2. The analysis identifies two meaningful failure modes, soft silencing and length preference. The observation that length preference accompanies longer actions and declining success rates offers a useful explanation for PPO’s instability in agentic coding.

  3. The findings lead to a simple and practical method, CPSD, which uses all rollouts to train the critic while selecting informative groups for policy updates. The experiments on both mathematical reasoning and agentic coding give the analysis practical relevance beyond single-turn tasks.

Critical weaknesses

W1. CPSD has yet to fully exploit PPO’s ability to train without group sampling. Figure 12 shows that the critic can learn behavior resembling GRPO-style advantage normalization even in a single-rollout setting. However, CPSD still relies on group sampling (G=8/16) for sample filtering, and the mathematical reasoning experiments allow up to five sampling rounds to collect enough valid groups. Although Appendix E.3 explores group-free filtering based on advantage magnitude, its effectiveness in actual training has not been evaluated. It therefore remains unclear whether CPSD can realize PPO’s potential advantages in single-rollout or asynchronous RL settings.

W2. The performance gains of CPSD on complex agentic tasks remain limited. Although the paper emphasizes the value of PPO for long-horizon agent RL, PPO-CPSD still underperforms GRPO on SWE-bench Verified under the strict setting (18.8% vs. 19.6%) and only marginally outperforms it under the patch-based setting (25.8% vs. 25.4%). These results suggest that, while CPSD mitigates the training collapse observed with vanilla PPO, they do not yet establish a consistent and substantial advantage over GRPO on complex agentic tasks.

Questions for authors

Q1. Group size directly affects the fraction of samples retained for actor training, since CPSD filters out groups with identical rewards. How robust are the gains with smaller groups? Results with group sizes of 2, 4, 8, and 16 under similar rollout budgets would help clarify this.

Q2. It remains unclear whether the gains come mainly from actor sample filtering or from retaining all rollouts for critic training. Please compare PPO with dynamic sampling for both the actor and critic against PPO-CPSD, which filters only actor samples, and report final reward and critic explained variance.

Q3. Does CPSD actually mitigate the critic's length preference, or does it mainly avoid updating the actor on affected samples? Comparing advantage distributions across trajectory lengths before and after applying CPSD would help clarify the mechanism behind the improved stability.

Q4. Figure 3 shows informative value rankings, but does this translate into better policy learning? A comparison between token-level GAE and trajectory-level advantages under the same training settings would help isolate the benefit of fine-grained credit assignment.

5 comments
AuthorsOpenReview ↗

Response to Reviewer YhKP (1/3)

Thanks for your detailed feedback and support of our work. Here, we address your questions point by point below. Revisions to the manuscript are highlighted in blue.

Weakness 1:

Can CPSD realize PPO's potential advantages in single-rollout or asynchronous RL settings

Thanks for this suggestion. The PPO-CPSD variant presented in Section 5 relies on group reward for filtering. A key motivation for this experimental setting is to enable a controlled comparison with GRPO using dynamic sampling, thereby providing evidence that a well-converged critic can benefit training. It does require rollouts within each group to wait for one another and therefore does not fully realize the potential of asynchronous RL training.

However, CPSD in its general form does not require rollouts within a group to wait for one another. In Appendix E.3, we show a strong correlation between advantage amplitude and sample informativeness. We further supplement this analysis with experiments that use advantage amplitude as the filtering criterion. The setup is as follows.

  • For the first 60 training steps, we continue to filter using group rewards, allowing time for the critic to converge.

  • After step 60, continuing until approximately step 200, we compute π+\pi_+ and π−\pi_- from each completed step: the fractions of trajectories in the correct and incorrect subsets belonging to all-correct and all-incorrect groups, respectively. The corresponding quantiles of absolute advantage amplitude within each subset yield thresholds A+A_+ and A−A_-.

  • We smooth each threshold using a moving average over the most recent N=5N=5 steps. In the next step, we discard trajectories whose absolute advantage amplitudes fall below their corresponding thresholds. This avoids waiting for complete reward information from the "current group", supporting asynchronous training.

We refer to the above method as PPO-CPSD (adv) and the original method based on group accuracy as PPO-CPSD (acc). Table 1 reports the performance of advantage-based filtering, and Figure 19 in the paper (Page 23) shows its training dynamics, which is also available via an anonymous link for convenient viewing (https://anonymous.4open.science/r/Fig19-training-dynamics-4E29/). PPO-CPSD (adv) achieves performance comparable to PPO-CPSD (acc) while maintaining stable training.

Table. Performance comparison across reasoning benchmarks.

MethodAIME 24AIME 25AIME 26AMC 23Beyond AIMEBrumoHMMT 25-FebGPQALiveCodeBenchAvg.
GRPO31.2518.5416.3076.416.5313.182.8637.8512.2623.91
PPO-CPSD (acc)36.4618.7016.6177.896.3915.733.4439.0812.9525.25
PPO-CPSD (adv)35.9520.0016.7777.346.1715.053.9138.3212.5325.12

Weakness 2:

The performance gains of CPSD on complex agentic tasks

Thank you for this comment. We would like to further clarify the performance of PPO and PPO-CPSD in agentic coding. Our goal is not to demonstrate that PPO (and its variants) could outperform GRPO in every setting, but to examine a broader range of scenarios to understand its strengths, limitations, and applicability.

For example, in strict settings, truncation noise caused by reaching context-length or runtime limits can exacerbate the difficulty of credit assignment. PPO is more susceptible to a feedback loop in which longer trajectories have near-zero success rates, leading to lower value predictions, weaker penalties, and further boosting increases in trajectory length. In contrast, GRPO computes advantages solely from rewards alone and is less affected. In settings that aim to promote efficient reasoning, for example, by substantially shortening the model’s output-length distribution, the greater robustness of GRPO may make it a more appropriate choice.

In the patch-based setting, producing a correct patch is sufficient to earn a reward. This removes the comparatively easy route to higher scores of simply shortening previously overlong responses to fit within the limits, making further improvements on SWE-bench Verified more challenging. For example, GRPO improves over the base model by 2.2%, from 23.2% to 25.4%. Against this backdrop, although PPO-CPSD’s absolute improvement of 0.4% over GRPO (25.8% versus 25.4%) is modest, it could still provide certain evidence of the benefit of using a critic.

Thank you again for this suggestion. We have added the corresponding discussion to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer YhKP (2/3)

Question 1

Gains with smaller groups

We have added experiments with a group size of 4 to evaluate CPSD’s performance with smaller groups. As shown in Table 2, CPSD continues to achieve better performance in this setting. Furthermore, as discussed in our response to Weakness 1, filtering based on advantage amplitude remains effective. We hope these results together can help address your concern.

Table. 2 Performance comparison with group size of 4.

MethodAIME 24AIME 25AIME 26AMC 23Beyond AIMEBrumoHMMT 25-FebGPQALiveCodeBenchAvg.
GRPO28.9517.4514.2774.185.4214.064.2736.5511.7822.99
PPO-CPSD31.9318.1814.9075.825.5213.374.3337.0312.4123.72

Question 2

PPO with dynamic sampling for both the actor and critic

Thank you for your suggestion. We have added additional ablation study to further demonstrate the effectiveness of preserving broader data coverage for the critic. Starting from PPO-CPSD, we filter out samples from all-correct and all-incorrect groups for both critic and policy updates and denote this variant as PPO-dyn. Table 3 reports its performance, and Figure 9 in the paper (Page 9) shows the explained variance dynamics (https://anonymous.4open.science/r/Fig9-critic-broad-convergence-A468/). PPO-dyn exhibits substantially slower critic convergence, accompanied by lower training rewards and worse performance. These results show that broader data coverage can improve critic learning and ultimately benefit policy training.

Table. 3 Performance comparison across reasoning benchmarks.

MethodAIME 24AIME 25AIME 26AMC 23Beyond AIMEBrumoHMMT 25-FebGPQALiveCodeBenchAvg.
PPO-CPSD36.4618.7016.6177.896.3915.733.4439.0812.9525.25
PPO-dyn32.6618.9115.1675.236.4512.503.7540.0213.4324.23
AuthorsOpenReview ↗

Response to Reviewer YhKP (3/3)

Question 3

Mitigate the critic's length preference or avoid updating the actor on affected samples

Thank you for your question. We would like to clarify that CPSD does not alter the critic's length preference, since we do not modify the critic's optimization objective. PPO-CPSD works by avoiding actor updates on the affected samples. We view this length preference as a relatively general characteristic of critic models.

To illustrate this, we additionally report advantage statistics in Table 4 for all-failure groups in mathematical reasoning settings before critic convergence (step 40–60).The length preference phenomenon is also observed in mathematical reasoning, although it does not lead to pronounced training instability in this setting. An intuitive picture is that longer trajectories tend to have lower success rates across the overall data distribution. The critic may therefore learn to assign them lower values, which can translate into weaker penalties for incorrect trajectories.

Therefore, PPO-CPSD does not aim to change this characteristic of the critic, but instead aims to prevent it from dominating the optimization process. During policy updates, we aim to filter out samples from all-correct and all-incorrect groups, which provide no discriminative reward signals within the group but only introduce latent biases. This helps ensure that optimization is driven by sufficient informative signals rather than latent preferences.

Table 4. Trajectory advantage for all-failure groups across response-length quintiles. Q1 represents the shortest and Q5 represents the longest.

MethodQ1Q2Q3Q4Q5
PPO-0.484-0.442-0.392-0.371-0.317

Question 4

Informative value rankings of value and token-level advantage

Thank you for this suggestion. We did not conduct a comparative analysis of token-level advantages considering that a ground-truth reference for their evaluation is not readily available. By contrast, value ranking analysis can be grounded in the following statistical expectation: Among trajectories generated for the same prompt, correct trajectories are expected to receive higher value estimates than incorrect trajectories on average,

E[V∣x,correct]>E[V∣x,incorrect].\mathbb E[ V\mid x,\mathrm{correct}] > \mathbb E[ V\mid x,\mathrm{incorrect}].

This motivates our analysis of value rankings, which shows that the critic's predictions are more informative than a flat prior and improve as training progresses.

For advantages, it is difficult to establish an analogous criterion that can be evaluated efficiently. Monte Carlo rollouts from individual prefixes are computationally expensive for LLMs. Under computational budget, this would limit the number of rollouts and lead to high estimation variance. Therefore, to provide empirical evidence that a critic can offer finer-grained signals that benefit training, we aim to illustrate this indirectly through the final training results. PPO-CPSD and GRPO differ only in that PPO-CPSD uses a critic to provide these signals. The methods PPO-CPSD and GRPO differ only in that PPO-CPSD uses a critic to provide these signals. PPO-CPSD's gains over GRPO thus provide indirect evidence for the effectiveness of the critic’s finer-grained training signals

Reviewer_YH9fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the emergent behaviors and learning mechanisms of critic models in PPO for LLM reinforcement learning at the task, group, and token levels. It reports that PPO critics develop a GRPO-like task-level normalization pattern, provide fine-grained value signals, and exhibit a length-related bias within all-failure groups. Based on these observations, the paper proposes Critic-Policy Sample Decoupling, which uses all rollout samples for critic training while filtering all-success and all-failure groups from policy updates. Experiments are conducted on mathematical reasoning and agentic coding tasks.

Critical strengths
  1. The finding that PPO exhibits a length-related bias in all-failure groups is valuable.

  2. The paper is clearly written and well organized.

Critical weaknesses
  1. The paper’s findings have relatively limited value. For example, the emergence of GRPO-like difficulty adjustment at the task level and the presence of outcome-related information in token-level value changes are both intuitive from theoretical and conceptual perspectives. The empirical evidence therefore provides limited additional support, and the paper does not clearly establish their connection to the main storyline. In particular, the observation that longer trajectories have lower success rates may simply be because longer trajectories correspond to more difficult tasks, which consequently have lower accuracy. The paper does not rule out this confounding factor.

  2. The paper’s main finding is insufficiently substantiated. The group-level, length-related difference in advantages is the core contribution: the paper identifies a length bias in PPO for all-failure groups. However, this central conclusion requires broader analysis, both from theoretical and experimental perspectives. Evidence from a limited number of models and coding-only tasks is insufficient to support this conclusion and the subsequent method design.

  3. The paper’s discussion of related work is insufficient. The paper does not discuss any methods that improve GRPO or PPO, nor does it examine how these methods may affect the paper’s conclusions and results.

Questions for authors

See Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YH9f (1/2)

Thank you for your detailed feedback. We will address your concerns and questions point by point below. Revisions to the manuscript are highlighted in blue.

Weakness 1:

The emergence behavior is intuitive from theoretical and conceptual perspectives

From a theoretical perspective, a converged critic can provide valid learning signals for policy updates in PPO. However, in LLM reasoning settings, it involves extremely long horizons and a correspondingly vast space of possible action sequences, while sparse outcome-based rewards further complicate critic training. Against this backdrop, understanding the learning dynamics of critic models for LLM is a non-trivial empirical question. Previous studies have typically evaluated critic behavior using Monte Carlo estimates obtained from repeated rollouts. However, these evaluations are computationally expensive, and limited sampling budgets can leave substantial estimation variance. Findings on whether critics provide informative signals remain mixed [1–3]; for example, Ref. [1] reports that value networks “barely outperform a random baseline”.

In this study, we provide a systematic empirical understanding of critic learning in LLM reasoning settings: what information critics acquire, how their signals evolve during training, and whether these signals benefit policy optimization:

  • We introduce a computationally efficient approach to trajectory-level comparative analysis that requires no additional MC rollouts, enabling more systematic and extensive evaluation of critic learning dynamics.
  • We further conduct aligned comparisons between PPO-CPSD and GRPO with dynamic sampling. Under otherwise identical settings, the two methods differ only in whether a critic model is used. Together, these analyses provide complementary evidence of the informativeness and practical utility of critic-derived signals.

The observation that longer trajectories have lower success rates may simply be because longer trajectories correspond to more difficult tasks

We would like to emphasize that our finding is that longer failed trajectories systematically receive higher critic-derived advantages (i. e. weaker penalties). We view the observation that longer trajectories have lower success rates as an underlying factor driving the above phenomena (detailed see rebuttal to Weakness 2), rather than as the findings we aim to highlight.

Furthermore, we would like to point out that Figure 4 in the paper restricts the analysis to all-failure groups, ruling out the possibility that some tasks are easier than others.

To further rule out task difficulty as a confounding factor, we systematically compare trajectory advantages within the same prompt. Specifically, we compute one Spearman correlation coefficient ρi\rho_i for each group ii analyzed in Figure 4, using trajectory lengths and advantages within that group. As shown in Table 1, the distribution of ρi\rho_i is clearly shifted toward positive values, with a mean of 0.5. This indicates that longer failed trajectories tend to receive weaker penalties even within the same prompt. Furthermore, Table 2 shows that longer trajectories receive weaker penalties even when compared at the same token index, providing additional support for the argument. Visualizations are presented in Figure 13 of the paper (Page 20). For view convenience, we also provide an anonymous link to Figure 13 (https://anonymous.4open.science/r/Fig13-additional-length-analysis-28D0/).

Table 1. Quantiles and summary statistics of Spearman ρ\rho.

StatisticP20P40P60P80MeanMedian
Spearman ρ\rho0.210.500.670.820.500.60

Table 2. Advantage at different token positions for failed trajectories grouped by response length.

Token position1k3k5k7k9k
Long-0.30-0.39-0.35-0.19-0.11
Mid-0.39-0.46-0.25-0.30-0.13
Short-0.33-0.51-0.53-0.36-0.28

Connection of empirical evidence to the main storyline

Thank you for your suggestion. We have added additional signposting sentences to the main text to clarify the logical flow of the paper. Our paper follows a three-part structure: Characterization of critic behavior and learning dynamics (Section 3), Influence of these characteristics on policy training (Section 4), and Improvement of training based on these insights (Section 5). For example:

  • We find that, after the critic converges, the trajectory advantages from critic approach those of GRPO (Characterization). This, in turn, leads to soft silencing of all-correct and all-incorrect samples (Influence). Based on this observation, we propose CPSD to filter out these samples and maintain sufficient learning signals for policy updates (Improvement).
AuthorsOpenReview ↗

Response to Reviewer YH9f (2/2)

Weakness 2:

Central conclusion requires broader analysis, both from theoretical and experimental perspectives.

Thanks for your suggestions. On the theoretical side, we have extended the analysis with an intuitive mathematical description of the possible mechanism underlying this phenomenon.

Intuitively, longer trajectories tend to have lower success rates across the overall data distribution, and the critic may consequently learn to assign them lower values,

E[Vˉ∣T]≈E[R∣T].\mathbb E[\bar V\mid T]\approx\mathbb E[R\mid T].

The critic may exploit this tendency even among trajectories generated for the same prompt, translating lower value estimates into weaker penalties for incorrect trajectories.

Here we provide an intuitive mathematical description. Consider a fixed, finite set of all-failure trajectories for the same prompt. Assume terminal-only rewards and a common failure reward rr. We hypothesize that the critic can learn to associate prefixes characteristic of longer failed trajectories with lower values (Evidence see Figure 13 in the paper):

\qquad \Delta_t\geq0,$$ The trajectory-averaged GAE advantage satisfies $$\bar A =-\frac{1}{T}\sum_{t=1}^{T}\lambda^{t-1}(V_t-r).$$ Consequently, $$\frac{\partial \bar A}{\partial V_t} =-\frac{\lambda^{t-1}}{T}\leq 0.$$ For a fixed trajectory, lowering critic predictions from $V_t$ to $V_t-\Delta_t$, with $\Delta_t\geq0$, therefore increases its mean advantage by $$\Delta\bar A =\frac{1}{T}\sum_{t=1}^{T}\lambda^{t-1}\Delta_t\geq0.$$ Thus, lower critic predictions weaken the average penalty on a failed trajectory. This identifies a mechanism through which a learned association between greater length and lower critic values can contribute to length preference. **On the experimental side, we emphasize that length preference is a general phenomenon that extends beyond agentic coding. It is particularly pronounced in agentic coding because of the extremely long response trajectories.** As additional evidence, we present an analysis of length preference in mathematical reasoning setting. The results show that critic-based advantages likewise impose weaker penalties on longer trajectories in all-failure groups (Table 3) Figure 14 in the paper (Page 20) shows the visualization, which is also available via an anonymous link for convenient viewing ([https://anonymous.4open.science/r/Fig14-length-analysis-6696/](https://anonymous.4open.science/r/Fig14-length-analysis-6696/)). The same pattern is also observed in the within-group analysis. Since mathematical problems involve much shorter contexts than agentic coding tasks, we did not observe a substantial impact of this characteristic on training. This suggests that length preference is a relatively general phenomenon. **Table 3.** Trajectory advantage for all-failure groups across response-length quintiles. Q1 represents the shortest and Q5 represents the longest. | Method | Q1 | Q2 | Q3 | Q4 | Q5 | |:--|--:|--:|--:|--:|--:| | PPO | -0.484 | -0.442 | -0.392 | -0.371 | -0.317 | | GRPO | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | ***Weakness 3:*** > The paper does not discuss any methods that improve GRPO or PPO Thank you for raising this point. As described in Section 3.1, **in fact, our experiments incorporate empirically validated improvements to both GRPO and PPO to account for recent advances in LLM RL**. Specifically, GRPO includes clip-higher and dynamic sampling (DAPO [4]), while PPO includes critic pretraining (VC-PPO [5]), decoupled GAE (VAPO [6]), and skip-observation token-level GAE (SAO [7]). These configurations were chosen to reflect recent algorithmic advances and provide strong baselines. We stated in Section 3.1: "For brevity, we still refer to these enhanced variants simply as PPO and GRPO." We recognize that this important convention may have been easy to overlook because it was stated only once. In the revised manuscript, we have reinforced this clarification in subsequent sections, making explicit that "PPO" and "GRPO" refer to the enhanced variants used in our experiments.
AuthorsOpenReview ↗

References

[1] Kazemnejad et al. VinePPO: Refining credit assignment in RL training of LLMs, ICML (2025)

[2] Chen et al. Hista and numca: Estimate state value effectively for large language model reinforcement learning, ICML (2026)

[3] Zhou et al. Start classifying: Categorical critics for LLM reinforcement learning, COLM (2026)

[4] Yu et al. DAPO: An open-source LLM reinforcement learning system at scale, NeurIPS (2025)

[5] Yuan et al. What’s behind ppo’s collapse in long-cot? value optimization holds the secret, (2025)

[6] Yue et al. VAPO: Efficient and reliable reinforcement learning for advanced reasoning tasks, (2025)

[7] Hou et al. Single-rollout asynchronous optimization for agentic reinforcement learning, (2026)

Dear Reviewer YH9f,

We sincerely appreciate your detailed feedback and the time you have devoted to the review process. As the discussion period draws to a close, we would appreciate any further thoughts or suggestions you may have on our response. Thank you again for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely appreciate the time and effort you have devoted to the review process. This work investigates the learning dynamics of critic models in PPO for LLM RL, identifies failure modes, and proposes improvements to address them. As the rebuttal period draws to a close, we would like to provide a brief overview of the current review status.

We would first like to express our sincere appreciation to the reviewers for their recognition of our work, which we briefly summarize below:

  • Valuable Insights: "The paper provides valuable insights into how the critic learns at the task, group, and token levels" (Reviewer YhKP); "The finding that PPO exhibits a length-related bias in all-failure groups is valuable" (Reviewer YH9f).

  • Significance: "offers a useful explanation for PPO’s instability in agentic coding" (Reviewer YhKP); "provides a useful empirical analysis of PPO critics beyond overall training performance" (Reviewer cZG6).

  • Practicality: "an effective method that improves the PPO baseline on both models" (Reviewer cZG6); "a simple and practical method, CPSD" (Reviewer YhKP).

We appreciate the reviewers' insightful comments and suggestions and have provided additional experimental results, clarifications, and manuscript revisions during the rebuttal phase. We summarize the main updates below.

  • Further analysis of length preference phenomenon: We provide additional empirical evidence that the length-related advantage bias persists at matched token positions (Figure 13b), among rollouts generated for the same prompt (Figure 13a), and across both mathematical reasoning and agentic coding (Figure 14). We also provide a theoretical analysis of how this advantage bias could influence policy updates.

  • Validation and ablation in broad settings: We demonstrate the effectiveness of preserving broader data coverage for the critic (Figure 9) and present training results using advantage magnitude as a group-free proxy for sample filtering (Figure 19).

  • Clarifications: Added additional signposting sentences to clarify the paper’s logical flow.

We are pleased that our responses have addressed the concerns of Reviewers YhKP and cZG6, and we sincerely appreciate your follow-up feedback. At the time of writing, we have not yet received further feedback from Reviewer YH9f, and we have provided additional clarifications and analyses addressing the concerns. As the rebuttal period draws to a close, we may not have sufficient time to provide further responses. Overall, we believe the additions and clarifications have improved the technical rigor, clarity, and completeness of our work.

We sincerely thank the Area Chair again for coordinating the review process, and the reviewers for the valuable feedback on our paper.

Best Regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.