- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Problem: This paper examines using PPO for LLM training and focuses on two sources of instability: 1) the use of truncated rollouts (and the practice of filtering them), which bias actor and critic estimations; and 2) return variability, which affects the gradient’s magnitude.
Method: The paper tackles these instabilities by introducing three modifications to standard PPO: 1) truncated rollouts are removed from the actor update but not from the critic update, 2) critic updates are normalized by the induced prompt variability, and 3) they study how batch size and gradient clipping impact gradient estimates and suggest that moderately small batches for critic training are better.
Experiments: The authors trained and validated their method using three different setups: FrontierCS (train Frontiersmith, evaluate FrontierCS), AIML (train DAPO-Math-17K, evaluate AIME24), and Search-R1 (on seven validation tests defined there).
Critical strengths
Importance. Using RL to fine-tune LLMs is certainly a relatively new and very hot research area. Therefore, it is important to understand how to properly apply reinforcement learning to language problems. The authors identify a bias source in RL policy updates caused by truncated rollouts. Understanding how to mitigate this bias can improve real-world applications.
Noise Normalization. I think that, although it has little novelty, the noise normalization is well thought out and well analyzed. It makes sense to use it per prompt, since we can get several rollouts from the same prompt and then easily assess return variance. Figure 3 clearly shows how noise normalization affects the critic's gradient magnitude.
Critical weaknesses
Low novelty. I think that the authors' contributions to the problem are rather limited.
- The first contribution is to keep truncated rollout in the critic update while excluding it from the actor. While this is certainly novel, it is a very small modification of PPO.
- The second contribution is about normalizing the noise by its variance. As the authors also note, this has already been done in both the ML and RL literature. I think it is smart that the authors apply it to prompt-induced variance, because the variance is easily accessible (whereas in classic RL, the agent is almost never in the same state twice, and assessing the variance there is more complicated).
- The third contribution is to study the impact of the minibatch size, since the clipping mechanism introduces a non-linear effect on the empirical mean estimator. I regard this as a simple ablation study
Weak Arguments Supporting Filtering Truncated Rollouts: I'm not sure I fully agree with the author's reasoning. Both EasyPPO and PPO discourage truncated rollouts, in different ways: PPO assigns them the lowest possible reward (thus zero), while EasyPPO achieves a similar effect by including truncated, zero-reward rollouts in the critic, effectively lowering the baseline, and this has the effect of increasing the “reinforcement” of positive returns (of untruncated rollouts), since in this setting, the baseline does affect the expected gradient update (differently from classic PPO). Given this, it is not clear to me why one method should be better than the other. Nevertheless, the experiments support the authors’ argument. The reason I remain skeptical, though, is that the results do not show strong statistical significance (i.e., Figure 4 uses one seed, and Figure 6 uses three).
I know that running LLM experiments is expensive. But even if we assume the results are statistically significant, it is hard to tell whether the performance increase is due to the filtering itself, for example, because the value magnitude changes, or because decoupling the data between the critic and actor is, in general, beneficial. In short, I think many hypotheses can explain the results, and they do not necessarily support the author’s claim.
Questions for authors
Clarifications
- Do the steps in Figure 3 refer to "PPO iterations"?
Future Work
-
The advantage A(s, a) = Q(s, a) - V(s), which is used in PPO and estimated via the generalized advantage estimator, can be understood as a baseline subtraction technique: with , the Q-function is estimated by Monte-Carlo rollouts. Theoretically, since the value function is action-independent, we can see that , and thus . This reasoning actually can be applied to any function of the states , i.e., . For this reason, it took me quite some time to understand how the way you train the critic can affect the bias of the actor update. The culprit is that, by modifying the distribution (by filtering out samples), baseline subtraction actually affects the expected update. The paper should state this effect more clearly.
-
Episode truncation is also a problem beyond language tasks. To validate the authors' theory, they could use toy tasks or small-scale experiments and increase the number of seeds to improve statistical significance.
-
In classical RL, we usually use to deal with infinite horizons properly, and we use bootstrapping (i.e., ) to predict the future return of truncated episodes. It would be interesting to see the shortcomings of discounting and bootstrapping compared with the filtering system the authors propose.
3 comments
Thanks for the Clarifications
Dear authors, I've read your reply carefully. I will update my review accordingly and consider your changes to the paper for the reviewers-AC discussion time.
Thanks so much for taking the time to read our response! We hope it helps!
Response to Reviewer z9e9
We thank the reviewer for recognizing the truncation-induced bias we identify, and for finding our noise normalization well thought out and well analyzed.
1. Statistical significance for overlong filtering. We agree that a single run cannot rule out alternative hypotheses, and we appreciate the reviewer's understanding of the cost of LLM experiments. We did our best within our compute budget and added Appendix D.3 (Figure J), which collects FrontierCS runs across seeds, configurations, and base algorithms (PPO vs. GRPO):
†GRPO has no learned critic; its group mean reward plays the role of the value baseline. Under actor-only filtering, truncated rollouts are excluded from the policy loss but still enter the group mean and standard deviation; under actor + critic filtering, the group statistics are computed over completed rollouts only, so the baseline estimates E[R∣s,not truncated], as the critic does under joint filtering in PPO.
Across all six EasyPPO runs with actor-only filtering, training remains stable, with the truncation ratio averaging only 3–7% after step 80. In contrast, all three joint-filtering runs fail in the same way: the truncation ratio rises above 86% and the best validation score stays below 8.5. Compared with the original submission, this increases the evidence for actor-only filtering from 3 → 6 EasyPPO runs, and for joint filtering from 1 → 3 runs across two base algorithms.
2. Alternative hypotheses. We further agree that statistical significance alone cannot distinguish between hypotheses. The reviewer raises two alternatives: a change in value magnitude, and a general benefit from decoupling critic and actor data. To test them, we additionally include GRPO runs, which have no learned critic, so neither alternative applies. Under the same setup, GRPO with actor-only filtering remains stable, whereas excluding truncated rollouts from the group baseline computation produces the same failure as in PPO. This rules out both alternatives and is consistent with our explanation that excluding truncated rollouts from the baseline removes the penalty on truncation.
3. Steps in Figure 3. Yes. Each step is one PPO iteration: we sample a rollout batch (16 prompts × 32 responses), compute advantages, and update the actor and the critic on that batch. Steps 30, 60, and 90 are the corresponding checkpoints of the training run, and step 30 is the end of the critic warmup. We have clarified this in the revision.
4. Effect of the baseline under filtering. Thanks for this insightful observation, and we agree that the paper should state it more clearly. Without filtering, a state-dependent baseline does not change the expected policy gradient, since E[b⋅∇logπ(τ)]=0. Filtering breaks this identity: when the actor update is restricted to non-truncated trajectories C, the baseline term becomes b⋅∇logP(C)=0, so the critic's training data affects the expected actor update. We have stated this explicitly in Section 3 and added the derivation at the beginning of Appendix B.2.
5. Discounting and bootstrapping. Thanks for this excellent suggestion. Concurrent work (Wen et al., 2026) also bootstraps with the critic value at chunk boundaries in LLM RL, and we have added it to the related work. Bootstrapping does not change our conclusion that the critic should keep truncated rollouts. What it changes is the actor side: truncated rollouts could enter the actor update with a bootstrapped return instead of being filtered, provided the deployment limit is longer than the training limit. In our setting, the two limits coincide, so truncation is a true termination.
We hope the additional runs, the clarified role of the baseline under filtering, and the discussion of bootstrapping address your concerns. If so, we would greatly appreciate it if you would consider raising your score.