acceptodds
Under review as a conference paper at ICLR 2027

EasyPPO: Stabilizing the Critic Is Key

Abstract

A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning to estimate expected returns and reduce policy-gradient variance. However, we find that the critic is also a major source of instability in reinforcement learning for large language models (LLMs). We identify two critic failure modes that destabilize PPO. First, filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, allowing truncation to increase even as conditional reward improves. Second, heterogeneous return noise can cause high-variance prompts to dominate critic updates in finite batches. We introduce **EasyPPO** to address these failures. Actor-only overlong filtering trains the critic on returns from both completed and truncated rollouts. Noise-normalized critic regression weights each prompt's critic loss by the inverse standard deviation of its sampled returns, balancing noise contributions across prompts. Moderately smaller critic mini-batches confine outlier influence to fewer rollouts during gradient clipping. Across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1, EasyPPO remains stable throughout the full training horizon and consistently outperforms vanilla PPO, VAPO, and HL-Gauss PPO. Its best validation scores show relative gains of %, %, and % over PPO, respectively.

Primary area:
reinforcement learning
Keywords:
Proximal Policy Optimization, LLM Post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z9e9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Problem: This paper examines using PPO for LLM training and focuses on two sources of instability: 1) the use of truncated rollouts (and the practice of filtering them), which bias actor and critic estimations; and 2) return variability, which affects the gradient’s magnitude.

Method: The paper tackles these instabilities by introducing three modifications to standard PPO: 1) truncated rollouts are removed from the actor update but not from the critic update, 2) critic updates are normalized by the induced prompt variability, and 3) they study how batch size and gradient clipping impact gradient estimates and suggest that moderately small batches for critic training are better.

Experiments: The authors trained and validated their method using three different setups: FrontierCS (train Frontiersmith, evaluate FrontierCS), AIML (train DAPO-Math-17K, evaluate AIME24), and Search-R1 (on seven validation tests defined there).

Critical strengths

Importance. Using RL to fine-tune LLMs is certainly a relatively new and very hot research area. Therefore, it is important to understand how to properly apply reinforcement learning to language problems. The authors identify a bias source in RL policy updates caused by truncated rollouts. Understanding how to mitigate this bias can improve real-world applications.

Noise Normalization. I think that, although it has little novelty, the noise normalization is well thought out and well analyzed. It makes sense to use it per prompt, since we can get several rollouts from the same prompt and then easily assess return variance. Figure 3 clearly shows how noise normalization affects the critic's gradient magnitude.

Critical weaknesses

Low novelty. I think that the authors' contributions to the problem are rather limited.

  • The first contribution is to keep truncated rollout in the critic update while excluding it from the actor. While this is certainly novel, it is a very small modification of PPO.
  • The second contribution is about normalizing the noise by its variance. As the authors also note, this has already been done in both the ML and RL literature. I think it is smart that the authors apply it to prompt-induced variance, because the variance is easily accessible (whereas in classic RL, the agent is almost never in the same state twice, and assessing the variance there is more complicated).
  • The third contribution is to study the impact of the minibatch size, since the clipping mechanism introduces a non-linear effect on the empirical mean estimator. I regard this as a simple ablation study

Weak Arguments Supporting Filtering Truncated Rollouts: I'm not sure I fully agree with the author's reasoning. Both EasyPPO and PPO discourage truncated rollouts, in different ways: PPO assigns them the lowest possible reward (thus zero), while EasyPPO achieves a similar effect by including truncated, zero-reward rollouts in the critic, effectively lowering the baseline, and this has the effect of increasing the “reinforcement” of positive returns (of untruncated rollouts), since in this setting, the baseline does affect the expected gradient update (differently from classic PPO). Given this, it is not clear to me why one method should be better than the other. Nevertheless, the experiments support the authors’ argument. The reason I remain skeptical, though, is that the results do not show strong statistical significance (i.e., Figure 4 uses one seed, and Figure 6 uses three).

I know that running LLM experiments is expensive. But even if we assume the results are statistically significant, it is hard to tell whether the performance increase is due to the filtering itself, for example, because the value magnitude changes, or because decoupling the data between the critic and actor is, in general, beneficial. In short, I think many hypotheses can explain the results, and they do not necessarily support the author’s claim.

Questions for authors

Clarifications

  • Do the steps in Figure 3 refer to "PPO iterations"?

Future Work

  • The advantage A(s, a) = Q(s, a) - V(s), which is used in PPO and estimated via the generalized advantage estimator, can be understood as a baseline subtraction technique: with λ=1\lambda=1, the Q-function is estimated by Monte-Carlo rollouts. Theoretically, since the value function is action-independent, we can see that E[v(s)∇θlog⁡π(a∣s)]=0E[v(s)\nabla_\theta \log \pi(a|s)]=0, and thus E[(Q(s,a)−V(s))∇θlog⁡π(a∣s)]=E[Q(s,a)∇θlog⁡π(a∣s)]E[(Q(s, a) - V(s))\nabla_\theta \log \pi(a|s)]=E[Q(s, a) \nabla_\theta \log \pi(a|s)]. This reasoning actually can be applied to any function of the states g(s)g(s), i.e., E[(Q(s,a)−g(s))∇θlog⁡π(a∣s)]=E[Q(s,a)∇θlog⁡π(a∣s)]E[(Q(s, a) - g(s))\nabla_\theta \log \pi(a|s)]=E[Q(s, a) \nabla_\theta \log \pi(a|s)]. For this reason, it took me quite some time to understand how the way you train the critic can affect the bias of the actor update. The culprit is that, by modifying the distribution (by filtering out samples), baseline subtraction actually affects the expected update. The paper should state this effect more clearly.

  • Episode truncation is also a problem beyond language tasks. To validate the authors' theory, they could use toy tasks or small-scale experiments and increase the number of seeds to improve statistical significance.

  • In classical RL, we usually use γ<1\gamma < 1 to deal with infinite horizons properly, and we use bootstrapping (i.e., λ>1\lambda > 1) to predict the future return of truncated episodes. It would be interesting to see the shortcomings of discounting and bootstrapping compared with the filtering system the authors propose.

3 comments
AuthorsOpenReview ↗

Response to Reviewer z9e9

We thank the reviewer for recognizing the truncation-induced bias we identify, and for finding our noise normalization well thought out and well analyzed.

1. Statistical significance for overlong filtering. We agree that a single run cannot rule out alternative hypotheses, and we appreciate the reviewer's understanding of the cost of LLM experiments. We did our best within our compute budget and added Appendix D.3 (Figure J), which collects FrontierCS runs across seeds, configurations, and base algorithms (PPO vs. GRPO):

FilteringMethodVariantBest Val.Truncation (after step 80)
Actor-onlyEasyPPO4 critic mini-batches14.83.6%
Actor-onlyEasyPPO4 critic mini-batches14.33.9%
Actor-onlyEasyPPO4 critic mini-batches14.07.4%
Actor-onlyEasyPPO2 critic mini-batches15.33.0%
Actor-onlyEasyPPO8 critic mini-batches15.93.0%
Actor-onlyEasyPPO4 critic mini-batches, Huber critic loss14.54.5%
Actor-onlyGRPO$^\dagger$–12.519.9%
Actor + criticPPO–6.992.3%
Actor + criticPPOw/ noise normalization8.489.0%
Actor + criticGRPO$^\dagger$–7.786.4%

†^\daggerGRPO has no learned critic; its group mean reward plays the role of the value baseline. Under actor-only filtering, truncated rollouts are excluded from the policy loss but still enter the group mean and standard deviation; under actor + critic filtering, the group statistics are computed over completed rollouts only, so the baseline estimates E[R∣s,not truncated]\mathbb{E}[R \mid s, \text{not truncated}], as the critic does under joint filtering in PPO.


Across all six EasyPPO runs with actor-only filtering, training remains stable, with the truncation ratio averaging only 3–7% after step 80. In contrast, all three joint-filtering runs fail in the same way: the truncation ratio rises above 86% and the best validation score stays below 8.5. Compared with the original submission, this increases the evidence for actor-only filtering from 3 →\rightarrow 6 EasyPPO runs, and for joint filtering from 1 →\rightarrow 3 runs across two base algorithms.

2. Alternative hypotheses. We further agree that statistical significance alone cannot distinguish between hypotheses. The reviewer raises two alternatives: a change in value magnitude, and a general benefit from decoupling critic and actor data. To test them, we additionally include GRPO runs, which have no learned critic, so neither alternative applies. Under the same setup, GRPO with actor-only filtering remains stable, whereas excluding truncated rollouts from the group baseline computation produces the same failure as in PPO. This rules out both alternatives and is consistent with our explanation that excluding truncated rollouts from the baseline removes the penalty on truncation.

3. Steps in Figure 3. Yes. Each step is one PPO iteration: we sample a rollout batch (16 prompts × 32 responses), compute advantages, and update the actor and the critic on that batch. Steps 30, 60, and 90 are the corresponding checkpoints of the training run, and step 30 is the end of the critic warmup. We have clarified this in the revision.

4. Effect of the baseline under filtering. Thanks for this insightful observation, and we agree that the paper should state it more clearly. Without filtering, a state-dependent baseline does not change the expected policy gradient, since E[b⋅∇log⁡π(τ)]=0\mathbb{E}[b \cdot \nabla \log \pi(\tau)] = 0. Filtering breaks this identity: when the actor update is restricted to non-truncated trajectories CC, the baseline term becomes b⋅∇log⁡P(C)≠0b \cdot \nabla \log P(C) \neq 0, so the critic's training data affects the expected actor update. We have stated this explicitly in Section 3 and added the derivation at the beginning of Appendix B.2.

5. Discounting and bootstrapping. Thanks for this excellent suggestion. Concurrent work (Wen et al., 2026) also bootstraps with the critic value at chunk boundaries in LLM RL, and we have added it to the related work. Bootstrapping does not change our conclusion that the critic should keep truncated rollouts. What it changes is the actor side: truncated rollouts could enter the actor update with a bootstrapped return instead of being filtered, provided the deployment limit is longer than the training limit. In our setting, the two limits coincide, so truncation is a true termination.

We hope the additional runs, the clarified role of the baseline under filtering, and the discussion of bootstrapping address your concerns. If so, we would greatly appreciate it if you would consider raising your score.

Reviewer_z9e9OpenReview ↗

Thanks for the Clarifications

Dear authors, I've read your reply carefully. I will update my review accordingly and consider your changes to the paper for the reviewers-AC discussion time.

Thanks so much for taking the time to read our response! We hope it helps!

Reviewer_Qh1rOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes four modifications to PPO to improve the performance of PPO on LLM post-training tasks. The modifications are including truncated rollouts in the batch for the critic updates, weighting different prompts in a batch inversely-proportionally to their standard deviation, critic gradient clipping, and using smaller mini-batch sizes.

Critical strengths

The proposed techniques for stabilizing the critic are valuable for practitioners of LLM post-training with PPO.

Critical weaknesses

Although the proposed modifications to PPO are valuable for practitioners, the writing and presentation quality of the paper can be significantly improved. In Figure 1, it is hard to understand what the different shapes and arrows are trying to communicate.

Most of the abstract in lines 14–22 is unclear. Many sentences or phrases in the text are unclear and hard to understand. Examples of such sentences or phrases are:

  • “overlong rollouts” was not defined
  • “in the accurate-critic limit, filtering both networks shifts the policy objective to reward conditioned on completion, E[R |s,not truncated]” — What does filtering a network mean? What is accurate-critic limit?
  • “filtering both actor and critic updates” — What does filtering actor and critic updates mean?
  • “EasyPPO weights prompts inversely by their standard deviation” — Standard deviation of what?
  • “Smaller critic mini-batches limit outlier influence through gradient clipping” — What is outlier influence? Influence on what? What does “through gradient clipping mean? Can’t smaller mini-batches and gradient clipping be implemented independently?
  • “while controlling how heterogeneous return noise shapes finite-batch updates” — what does shaping an update mean?
  • “to focus critic updates on prediction error normalized by return variability”
  • “because they confine the remaining outliers’ influence to fewer rollouts”
  • “these effects”
  • “Our implementation retains token-level GAE, the full clipped PPO objective, and KL regularization.”
  • “we use the prompt’s return standard deviation across its token states” — what does this mean?
  • “We estimate ˆσ(s) from rollout groups including truncated responses” — It is unclear how ˆσ(s) is calculated.
  • “Smaller mini-batches confine the joint rescaling caused by outliers to fewer rollouts, but average out less return noise.”

In the preliminary, the paper mentions “We omit PPO clipping and KL regularization, yielding the actor objective,” which results in REINFORCE with baseline in equation 1. However, shortly after the paper mentions “Our implementation retains token-level GAE, the full clipped PPO objective, and KL regularization,” which seems to contradict the earlier sentence.

Questions for authors

No questions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Qh1r (2/2)

Thank you for your review. Your concerns are about clarity, and we agree that several terms appeared before we defined them. We revised the paper accordingly; every change is marked in blue in the updated PDF. Below, each point gives the meaning and the revised text.

EasyPPO makes three changes, not four. Gradient-norm clipping, which is separate from PPO's ratio clipping, is already used by our PPO baseline and by every other method in the paper, with threshold 1.0 for both actor and critic, as Appendix A.2 lists. EasyPPO keeps it unchanged. The third change is the number of critic mini-batches per rollout batch: four instead of one, where each mini-batch takes its own clipped optimizer step.

Can smaller mini-batches and gradient clipping be implemented independently? Yes. They are separate settings, and we keep clipping as in the baseline. Under clipping, the mini-batch size decides how many rollouts one clipping operation rescales. If one rollout has an unusually large critic gradient, an outlier, it dominates the direction of its mini-batch gradient and pushes the norm above the threshold. Clipping then scales the whole gradient down, which also shrinks the contribution of every other rollout in that mini-batch. With four mini-batches of 128 responses on FrontierCS and AIME, this affects 128 of the 512 rollouts instead of all of them. Smaller mini-batches, however, average out less return noise before clipping. Appendix B.4 bounds both effects, and we chose four mini-batches from the sweep in Figure 8.

Overlong rollouts. A rollout is truncated, or overlong, when its response reaches the maximum response length before it ends, for example 32,768 tokens on FrontierCS. Its reward is computed on an incomplete output. Section 1 now defines the term at first use.

Filtering an update; filtering both networks. An update is filtered when its loss excludes truncated rollouts, that is, gives them zero weight. Filtering both actor and critic updates removes truncated rollouts from both losses. EasyPPO filters the actor update only, so truncated rollouts still enter the critic loss and the GAE targets. Section 1 now says: "we call an update filtered when its loss excludes truncated rollouts."

Accurate-critic limit. This is the idealized case in which the critic predicts exactly the mean return of the rollouts it is trained on. Under joint filtering that mean is E[R∣s,not truncated]\mathbb{E}[R\mid s,\text{not truncated}], and Eq. 4 shows that the expected policy gradient then equals ∇θE[R∣s,not truncated]\nabla_\theta\mathbb{E}[R\mid s,\text{not truncated}], the gradient of the reward of completed rollouts only. The revised sentence reads: "if the critic predicts this mean exactly, filtering both the actor and critic updates shifts the policy objective to reward conditioned on completion."

Standard deviation of what. Of the returns of the responses sampled for that prompt in the current batch: 32 responses per prompt on FrontierCS, 16 on AIME and Search-R1. The Figure 1 caption now writes the weight as 1/Var(R∣s)1/\sqrt{\mathrm{Var}(R\mid s)}.

How σ^(s)\hat\sigma(s) is computed. Take a prompt ss with nn sampled responses and terminal returns Rs,1,…,Rs,nR_{s,1},\dots,R_{s,n} in the current rollout batch, truncated responses included. Then σ^(s)=1n∑j=1n(Rs,j−Rˉs)2,Rˉs=1n∑j=1nRs,j.\hat\sigma(s)=\sqrt{\frac1n\sum_{j=1}^{n}\bigl(R_{s,j}-\bar R_s\bigr)^2},\qquad \bar R_s=\frac1n\sum_{j=1}^{n}R_{s,j}. Eq. 7 normalizes the weight 1/max⁡{σ^(s),ε}1/\max\{\hat\sigma(s),\varepsilon\} to mean one over the prompts of the batch. The weight is computed once per rollout batch, before the critic mini-batch split, and applied to every token of every response to ss. This is what "the prompt's return standard deviation across its token states" meant: one weight per prompt for all its token states, instead of a separate variance estimate at each prefix. By the law of total variance, the prompt-level variance bounds the average prefix-level variance; Appendix B.3 gives the derivation. Section 4 now states both points.

AuthorsOpenReview ↗

Response to Reviewer Qh1r (2/2)

"Shapes finite-batch updates", "prediction error normalized by return variability", "remaining outliers' influence", "these effects". These phrases came from one paragraph of the introduction, which now reads:

First, actor-only overlong filtering filters actor updates only and keeps truncated rollouts in the critic loss. Second, noise-normalized critic regression weights each prompt's critic loss by the inverse standard deviation of its sampled returns, so that prompts with noisier returns do not dominate critic updates. Third, we split each rollout batch into four critic mini-batches, each clipped separately: clipping scales down a whole mini-batch when one rollout has an outlier gradient, so smaller mini-batches confine this effect to fewer rollouts, while larger ones average out more return noise.

"These effects" were the last two. "Prediction error normalized by return variability" refers to Eq. 6: after weighting, the expected squared critic gradient of a prompt depends, apart from the feature norm, only on its prediction error divided by its return standard deviation. Section 4 now also states the rescaling directly: "An outlier gradient that pushes ∥g∥2\|g\|_2 above cc makes clipping scale down every rollout in its mini-batch; smaller mini-batches confine this rescaling to fewer rollouts, but average out less return noise."

Preliminaries: Eq. 1 versus full PPO. The two sentences describe different things, and the old text did not say which is which. Eqs. 1 and 2 are a simplified model used only for the analysis in Sections 3 and 4: one response is one action, with no clipping and no KL term. As you note, Eq. 1 is REINFORCE with a learned baseline. All experiments use full PPO with token-level GAE, the clipped objective and a KL loss, with the settings in Appendix A.2. The two connect directly. Every method takes one actor step per rollout batch on freshly sampled responses, so the importance ratio is one and the clip is inactive in that step; the step is a policy gradient with a learned baseline, as in Eq. 1, applied per token and with a small KL term. With γ=λ=1\gamma=\lambda=1 and a single terminal reward, the per-token advantage telescopes to At=∑k≥t(rk+Vϕ(sk+1)−Vϕ(sk))=R−Vϕ(st)A_t=\sum_{k\ge t}\bigl(r_k+V_\phi(s_{k+1})-V_\phi(s_k)\bigr)=R-V_\phi(s_t) before verl normalizes advantages over the batch, and Eq. 1 uses this advantage at the prompt state. Section 2 now labels Eqs. 1 and 2 as the analysis model and states this link.

Figure 1. The top row shows the failure modes of PPO, and the bottom row shows EasyPPO.

  • Left: ✓ and × mark whether completed rollouts, which end with EOS, and truncated rollouts enter the actor loss and the critic loss. The bar gives the fraction of truncated rollouts and the curve gives the reward distribution over all rollouts, dashed for an earlier policy and solid for a later one. When both losses are filtered, truncation rises and reward moves toward 0. When only the actor loss is filtered, truncation falls and reward moves toward 1.
  • Middle: each arrow is the critic gradient of one prompt. Red arrows belong to prompts with high return variance; they are long and pull the summed update, the black arrow, toward them. Weighting each prompt by 1/Var(R∣s)1/\sqrt{\mathrm{Var}(R\mid s)} shortens them, and dashed arrows show them before weighting.
  • Right: the dashed circle is the clipping norm. In one large mini-batch, the raw mean gradient points along the single red outlier, and clipping keeps that direction. With four smaller mini-batches clipped one after another, the outlier changes only the step that contains it.

The revised caption explains these marks.

Abstract, lines 14 to 22. The revised sentences read:

First, when truncated rollouts (those cut off at the length limit) are filtered from both the actor and critic losses, PPO optimizes only the reward of completed rollouts, so the truncation ratio can increase even as this reward improves. Second, prompts with more variable returns can dominate critic updates in finite batches. [...] Actor-only overlong filtering excludes truncated rollouts from the actor loss but keeps them in the critic loss. Noise-normalized critic regression weights each prompt's critic loss by the inverse standard deviation of its sampled returns. Clipping the critic gradient separately on four mini-batches of each rollout batch keeps one outlier rollout from shrinking the update of the whole batch.

We hope the revision makes the paper easy to follow. If it resolves your clarity concerns, we would be grateful if you would reconsider your score.

I thank the authors for their detailed clarifications and revising the paper. My concerns about the clarity of the paper have been addressed, and I raise my rating to 3.

Thank you for taking the time to read our revisions and for raising your rating. We’re glad the changes addressed your concerns!

Reviewer_VAkyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identified two issues with PPO based llm training: filtering out truncated completion induce a sampling bias; varience in rewards too high in small batch causing too low signal to noise ratio for gradient estimation. To remedy the first issue, the paper propose to keep critics learning on the truncated completion. For the varience problem, it introduce a GRPO like reward varience normalization on the critic loss with solid theoretical justification. Intrinsic analysis does show the methods improve the stability of ppo. Modest performance improvement is also observed on a few llm reasoning tasks with Qwen3.5-9B-Base model.

Critical strengths
  1. The identified instability has indeed been alleviated to some extent with quite insightful and extensive intrinsic analysis.
  2. The critics loss normalization analysis is simple yet insightful.
Critical weaknesses
  1. the main experiment table should have grpo and dr. grpo for comparison. The current draft left the question of how far is a stablized ppo from being a plausible alternative to grpo completely open. this is a big question that most readers want to know.
Questions for authors
  1. ", allowing truncation to increase even as conditional reward improves" What do you mean? This is unclear grammatically.
  2. In figure 4, search R1, can you keep training the easy ppo for a bit longer? it looks like the performance is dropping
3 comments
AuthorsOpenReview ↗

Response to Reviewer VAky

We thank the reviewer for finding our analysis of critic instability insightful and extensive, and our critic loss normalization simple yet insightful.

1. Comparison with GRPO and Dr. GRPO. Thank you for the suggestion. We invested approximately $20K in additional compute to compare EasyPPO with GRPO and Dr. GRPO in our FrontierCS setting. We use the same initialization, rollout budget, and 240 training steps as EasyPPO. Training rewards are averaged over the same number of policy-update steps for all methods.

MetricEasyPPO (4 minibatches)EasyPPO (8 minibatches)GRPODr. GRPO
Avg. Training Reward30.78‾\underline{30.78}33.20\textbf{33.20}26.4426.4429.9729.97
Max. Validation Reward14.82‾\underline{14.82}15.94\textbf{15.94}12.5312.5314.6214.62

Both EasyPPO configurations outperform GRPO and Dr. GRPO on training and validation reward, showing that a stabilized critic offers a clear advantage over critic-free baselines. Full training curves are in Appendix D (Figure H) of the revised manuscript.

2. Clarity of the abstract sentence. Thank you for catching this unclear sentence. We meant that when truncated rollouts are filtered from both the actor and critic losses, PPO optimizes only the reward of completed rollouts, so the policy is not penalized for truncation, and the truncation ratio can increase even as the reward of completed rollouts improves (Figure 2). We have rewritten the sentence in the revised abstract (marked in blue).

3. Search-R1 performance at the end of training. Thank you for pointing this out. Search-R1 trains on the training splits of NQ and HotpotQA and validates on held-out splits of seven datasets, so NQ and HotpotQA are in-distribution while the other five are out-of-distribution. For the run in Figure 4, training accuracy saturates after step 100; from step 100 to 195, NQ still improves (42.5 → 44.4), while the out-of-distribution average drops (42.5 → 38.9). The late decrease is thus consistent with mild overfitting after saturation rather than a collapse.

Since the checkpoint of this run is no longer available, we provide an independent EasyPPO run as a robustness check similar to a multi-seed experiment (same setting except slightly different noise-normalization hyperparameters). Validation scores are reported every 25 steps; training accuracy is averaged over the preceding 25 steps.

StepNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleVal. Avg.Train Acc.
10038.164.441.239.337.715.049.640.838.3
12542.264.138.836.440.812.848.040.441.9
15041.264.138.835.837.111.845.639.239.1
17541.965.039.136.438.912.852.040.938.8
19543.865.040.337.739.613.145.640.740.8

This run remains stable through the end of training, with NQ and TriviaQA still improving, indicating that EasyPPO training on Search-R1 has saturated rather than collapsed. We added this explanation to Section 5 and the run, with full curves and settings, to Appendix D.2 (marked in blue).

We hope these additional comparisons with GRPO-based methods, the clarified abstract, and the extended Search-R1 analysis address your concerns. If so, we would greatly appreciate it if you would consider raising your score.

Thanks for the reply. Although the time is quite limited, can you attempt PPO and EasyPPO on some classical RL problems? like envpool Atari games or some sparse reward problem from https://github.com/instadeepai/jumanji . I understand the current paper is focusing on LLM reasoning, which is the main stream research, but since the claiming is quite broad, so should the experiments. It is also fine if you add some scoping claims in the paper to make it more clear that experiments and analysis mainly holds for LLM reasoning, and leaving the general RL problems to future.

Regardless, I raise my evaluation to 3.

Thank you for the positive feedback and for raising your score! Yes, EasyPPO is primarily designed for LLM RL, as the failure mode we address in vanilla PPO—differing return variances across prompts within the same batch—mainly arises in LLM tasks.

As suggested, we also evaluated EasyPPO on a classical RL task: sparse-reward TSP20 from Jumanji (https://github.com/instadeepai/jumanji). Using Jumanji’s default configuration, we train separate actor and critic networks from scratch (2 Transformer blocks, hidden dim 128, 8 heads, FFN dim 512; ~0.5M parameters each). We compare vanilla PPO, EasyPPO, and PPO with only Critic Mini-Batch Updates (splitting each critic batch into 4 mini-batches). All runs use grouped sampling (32 instances × 8 trajectories) and train fully on-policy for ~100M environment steps (20K full-batch actor updates).

The table below reports tour lengths (lower is better):

MethodTraining mean ↓\downarrowMean Validation ↓\downarrowBest validation ↓\downarrow
PPO4.118874.245453.92629
EasyPPO4.028774.230563.93210
PPO + Critic Mini-Batch Updates4.156804.226833.92786

While EasyPPO improves the training mean, validation performance is comparable across all three methods (with vanilla PPO slightly ahead on the best checkpoint), consistent with our focus on LLM RL. We believe this is because task instances here exhibit much less variation and return variance than LLM prompts; even so, EasyPPO does not hurt performance, and we will clarify our scope and report these results in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.