- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes PrePO to reduce the cost of updating multiple times on the same rollout batch in RLVR. It takes two temporary steps, extrapolates where K steps would end, moves part of the way there, and then takes one real step. The cost stays at three backward passes regardless of K. The paper provides some theoretical analysis and reports faster training than GRPO and SFPO on math RLVR, with additional results on SFT and distillation.
Critical strengths
The method is simple, easy to plug in, and its cost does not grow with K. The observation that the short-term optimization path on a batch is quite predictable is interesting.
Critical weaknesses
- The gains may come mainly from a larger effective step size. The paper's own analysis shows that a PrePO step behaves like a larger momentum step, and the fitted trends stay close to one, in which case the method is close to simply taking larger steps. All methods use the same learning rate, and there is no baseline with a tuned learning rate or with the same number of backward passes. It is hard to tell how much the prediction itself contributes.
- The theory feels weaker than the claims made about it. The error bound is fairly loose, and the empirical check of the bound does not really validate it. The method also seems to depend heavily on the entropy gate, which is not covered by the analysis.
Questions for authors
- How does PrePO compare with GRPO using a tuned or larger learning rate?
- How does it compare with baselines that use the same number of backward passes, e.g., more epochs per batch?
9 comments
Response on a Tuned or Larger Learning Rate (Part 2/5)
Learning-rate baseline: How does PrePO compare with GRPO using a tuned or larger learning rate?
We agree this is the most direct test of our claim. We ran GRPO at 3.5η and 5η in the setting of Table 2 with 3 seeds, with and without the fallback to η at PrePO's own entropy gate (Table 9 and Figure A4, Appendix D.7). The fallback gives the larger rate a simple schedule: large early, then the original rate once entropy breaks its trend.
| Method | Steps to GRPO peak | Wall-clock speedup | Peak | Final (step 400) |
|---|---|---|---|---|
| GRPO, η | 392 | 1.00× | 36.29 | 36.1 |
| GRPO, 3.5η | 139 | 2.82× | 37.4 | 35.3 |
| GRPO, 3.5η → η | 144 | 2.72× | 38.1 | 37.6 |
| GRPO, 5η | 132 | 2.97× | 37.1 | 32.8 |
| GRPO, 5η → η | 134 | 2.93× | 37.3 | 36.1 |
| PrePO | 113 | 2.33× | 40.30 | 39.1 |
Why a larger rate is fast but falls short. Early in training the gradient keeps pointing the same way, so a larger step covers that path faster. This is why every larger rate reaches GRPO's peak sooner, and in wall-clock time even sooner than PrePO, as Table 9 shows. The cost is that each update stretches one gradient by the same amount in every direction. That pulls the policy hard toward answers it already samples well, so token entropy is spent early (Figure A4). Once the rollouts become similar, each batch carries less new signal, but the large step still amplifies its noise. Without the fallback, accuracy declines after step 200 to 250.
Tuning the rate with a schedule. The fallback runs test the natural fix: start large, then return to η. This removes the late decline, but it cannot restore the entropy already spent. These runs plateau below PrePO, and the run that spent more plateaus lower. Against a higher target the order reverses. The plain larger rates never reach 37.5. To reach 38.0, 3.5η → η needs 246 steps and PrePO needs 149, about 10% less wall-clock time even with its extra passes. Only PrePO reaches 38.5. The schedule is the same for every run; the state each run brings into it is not.
Response on Whether PrePO Is Just a Larger Step (Part 3/5)
Effective step size: The paper's own analysis shows that a PrePO step behaves like a larger momentum step, and the fitted trends stay close to one, in which case the method is close to simply taking larger steps.
Part 2 shows the outcome. Here we explain why a PrePO step differs from a larger one.
The trends near one are Muon's. The value 0.96 in Section 7.3 is Muon's trend, one number per weight matrix. AdamW, which gives all the main results in Table 2, fits a separate retention for each coordinate. We logged it (Figure A6, Appendix D.11). It settles between 0.72 and 0.78, with a mean of about 0.74, for every K and α we tested. So for AdamW the trend is clearly below one.
Similar size, different shape. With a retention of 0.74, the K-step look-ahead adds up to about 3.0 steps at K = 5, not 5. On average, then, PrePO does take a step about three times larger. The difference is how that step is spread. A larger learning rate stretches every coordinate by the same amount. PrePO gives the large step only to coordinates that keep moving the same way. A coordinate that reverses gets at most about two plain steps, whatever K is (Proposition 2). The uniform version of a step this size is the 3.5η run in Part 2, which ends at 35.3 while PrePO ends at 39.1.
Where the gain comes from. PrePO's probes give it something a larger rate never sees: a second gradient on the same batch, taken where the optimizer is heading (Lemma 3). The probes are only measurements and are discarded, so the model still takes one update per batch, as GRPO does. That update moves only a fraction α toward the prediction and then takes one real step with a fresh gradient, so any prediction error is scaled down by α (Appendix C).
Why tuning alone does not close the gap. Proposition 3 makes this precise in a simplified noisy quadratic. A learning rate tuned to move as fast as PrePO settles at a higher noise level, and one tuned to match PrePO's noise moves more slowly. We read this as an explanation of the table rather than a guarantee for every setting. It matches what the table shows: the larger rates win on speed to GRPO's level but end lower.
Summary of Responses to Reviewer CWwN's Comments (Part 1/5)
Thank you for such a careful and expert review. We appreciate your recognition that PrePO is simple and easy to plug into an existing trainer, and that its cost stays at three backward passes however far ahead it predicts. We are also glad you found it interesting that the short-term optimization path on a batch is so predictable. That predictability is the fact the whole method rests on, and we are glad it came through. Your questions on the learning rate, matched compute and the theory led us to run the most direct controls, and we answer each below.
Summary of our response