acceptodds
Under review as a conference paper at ICLR 2027

Skip What You Can Predict: Predictive Repositioning for Policy Optimization for Efficient LLM Training

Abstract

Reinforcement learning with verifiable rewards (RLVR) can improve the reasoning ability of large language models, but repeatedly updating a policy on the same rollout batch is expensive: every additional update requires another backward pass, and multi-step methods pay for all intermediate optimization steps. We introduce Predictive Repositioning for Policy Optimization (PrePO), which uses two observed optimizer transitions to estimate a farther point along the same-batch optimization trajectory, moves partway toward that point, evaluates the original objective there, and applies a corrective update. This gives PrePO a fixed active cost that does not grow with the virtual optimization depth. Our analysis gives finite-horizon error bounds for AdamW and Muon and shows that sufficiently accurate endpoint estimates preserve the usual descent and convergence behavior of smooth gradient descent. In RLVR experiments, PrePO reaches matched performance targets in fewer optimization steps and lower wall-clock time than the corresponding baselines. We further evaluate the same update mechanism in supervised fine-tuning on a different dataset, showing that its use is not restricted to the original RLVR setting. Together, these results suggest that PrePO provides a practical mechanism for approximating repeated same-batch optimization while avoiding the cost of explicitly executing every intermediate update.

Primary area:
optimization
Keywords:
Large language models, efficient LLM training, Predict-and-correct optimization, Trajectory prediction, Reinforcement learning with verifiable rewards, Supervised fine-tuning, Knowledge distillation, Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CWwNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PrePO to reduce the cost of updating multiple times on the same rollout batch in RLVR. It takes two temporary steps, extrapolates where K steps would end, moves part of the way there, and then takes one real step. The cost stays at three backward passes regardless of K. The paper provides some theoretical analysis and reports faster training than GRPO and SFPO on math RLVR, with additional results on SFT and distillation.

Critical strengths

The method is simple, easy to plug in, and its cost does not grow with K. The observation that the short-term optimization path on a batch is quite predictable is interesting.

Critical weaknesses
  1. The gains may come mainly from a larger effective step size. The paper's own analysis shows that a PrePO step behaves like a larger momentum step, and the fitted trends stay close to one, in which case the method is close to simply taking larger steps. All methods use the same learning rate, and there is no baseline with a tuned learning rate or with the same number of backward passes. It is hard to tell how much the prediction itself contributes.
  2. The theory feels weaker than the claims made about it. The error bound is fairly loose, and the empirical check of the bound does not really validate it. The method also seems to depend heavily on the entropy gate, which is not covered by the analysis.
Questions for authors
  1. How does PrePO compare with GRPO using a tuned or larger learning rate?
  2. How does it compare with baselines that use the same number of backward passes, e.g., more epochs per batch?
9 comments
AuthorsOpenReview ↗

Summary of Responses to Reviewer CWwN's Comments (Part 1/5)

Thank you for such a careful and expert review. We appreciate your recognition that PrePO is simple and easy to plug into an existing trainer, and that its cost stays at three backward passes however far ahead it predicts. We are also glad you found it interesting that the short-term optimization path on a batch is so predictable. That predictability is the fact the whole method rests on, and we are glad it came through. Your questions on the learning rate, matched compute and the theory led us to run the most direct controls, and we answer each below.

Summary of our response

  1. We trained GRPO with the learning rate enlarged 3.5× and 5×, with and without returning to η at PrePO's gate, over 3 seeds (Table 9 and Figure A4, Appendix D.7).
  2. We logged AdamW's per-coordinate retention (Figure A6, Appendix D.11) and explained why a PrePO step is not a uniformly larger step.
  3. We added GRPO with three epochs per batch, which uses the same three backward passes per update as PrePO (Table 9).
  4. We narrowed the paper's theoretical claims, explained why the error bound is loose, and added a check of the bound that can fail (Appendix D.5).
  5. We extended Theorems 3 and 4 to hold for any gate, and related the gate to the fallback ablation (Table 7).
AuthorsOpenReview ↗

Response on a Tuned or Larger Learning Rate (Part 2/5)

Learning-rate baseline: How does PrePO compare with GRPO using a tuned or larger learning rate?

We agree this is the most direct test of our claim. We ran GRPO at 3.5η and 5η in the setting of Table 2 with 3 seeds, with and without the fallback to η at PrePO's own entropy gate (Table 9 and Figure A4, Appendix D.7). The fallback gives the larger rate a simple schedule: large early, then the original rate once entropy breaks its trend.

MethodSteps to GRPO peakWall-clock speedupPeakFinal (step 400)
GRPO, η3921.00×36.2936.1
GRPO, 3.5η1392.82×37.435.3
GRPO, 3.5η → η1442.72×38.137.6
GRPO, 5η1322.97×37.132.8
GRPO, 5η → η1342.93×37.336.1
PrePO1132.33×40.3039.1

Why a larger rate is fast but falls short. Early in training the gradient keeps pointing the same way, so a larger step covers that path faster. This is why every larger rate reaches GRPO's peak sooner, and in wall-clock time even sooner than PrePO, as Table 9 shows. The cost is that each update stretches one gradient by the same amount in every direction. That pulls the policy hard toward answers it already samples well, so token entropy is spent early (Figure A4). Once the rollouts become similar, each batch carries less new signal, but the large step still amplifies its noise. Without the fallback, accuracy declines after step 200 to 250.

Tuning the rate with a schedule. The fallback runs test the natural fix: start large, then return to η. This removes the late decline, but it cannot restore the entropy already spent. These runs plateau below PrePO, and the run that spent more plateaus lower. Against a higher target the order reverses. The plain larger rates never reach 37.5. To reach 38.0, 3.5η → η needs 246 steps and PrePO needs 149, about 10% less wall-clock time even with its extra passes. Only PrePO reaches 38.5. The schedule is the same for every run; the state each run brings into it is not.

AuthorsOpenReview ↗

Response on Whether PrePO Is Just a Larger Step (Part 3/5)

Effective step size: The paper's own analysis shows that a PrePO step behaves like a larger momentum step, and the fitted trends stay close to one, in which case the method is close to simply taking larger steps.

Part 2 shows the outcome. Here we explain why a PrePO step differs from a larger one.

The trends near one are Muon's. The value 0.96 in Section 7.3 is Muon's trend, one number per weight matrix. AdamW, which gives all the main results in Table 2, fits a separate retention for each coordinate. We logged it (Figure A6, Appendix D.11). It settles between 0.72 and 0.78, with a mean of about 0.74, for every K and α we tested. So for AdamW the trend is clearly below one.

Similar size, different shape. With a retention of 0.74, the K-step look-ahead adds up to about 3.0 steps at K = 5, not 5. On average, then, PrePO does take a step about three times larger. The difference is how that step is spread. A larger learning rate stretches every coordinate by the same amount. PrePO gives the large step only to coordinates that keep moving the same way. A coordinate that reverses gets at most about two plain steps, whatever K is (Proposition 2). The uniform version of a step this size is the 3.5η run in Part 2, which ends at 35.3 while PrePO ends at 39.1.

Where the gain comes from. PrePO's probes give it something a larger rate never sees: a second gradient on the same batch, taken where the optimizer is heading (Lemma 3). The probes are only measurements and are discarded, so the model still takes one update per batch, as GRPO does. That update moves only a fraction α toward the prediction and then takes one real step with a fresh gradient, so any prediction error is scaled down by α (Appendix C).

Why tuning alone does not close the gap. Proposition 3 makes this precise in a simplified noisy quadratic. A learning rate tuned to move as fast as PrePO settles at a higher noise level, and one tuned to match PrePO's noise moves more slowly. We read this as an explanation of the table rather than a guarantee for every setting. It matches what the table shows: the larger rates win on speed to GRPO's level but end lower.

Reviewer_buEHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PrePO uses two temporary optimizer updates to estimate a local motion trend, predicts the parameter position after a preset number of same-batch updates, and then repositions and corrects the parameters with a real gradient. This requires three backward passes whenever prediction is enabled. Experiments across model sizes, optimizers, and training objectives show faster progress and higher final accuracy under the tested budgets, supported by conditional error and convergence analyses.

Critical strengths
  1. The method addresses an important practical cost and demonstrates actual wall-clock savings. The main RLVR experiments report approximately 2.1–3.4× speedups in reaching GRPO’s peak score, including rollout generation and state transfers.

  2. Evaluation covers multiple model sizes, two optimizers, and several training objectives. Comparisons with SFPO and measurements of prediction error help substantiate the approach.

Critical weaknesses
  1. Computational savings depend on the workload. For 3B and 7B OPD, PrePO reduces training steps but increases elapsed time. The paper should more clearly identify when the additional update cost is outweighed by fewer rollout rounds.

  2. Stability depends substantially on disabling prediction. In the fallback ablation, average accuracy rises from 31.07 with prediction always enabled to 38.88 with permanent fallback. The reliability and sensitivity of the entropy-based trigger across tasks remain unclear.

  3. Applicability to asynchronous training is untested. Overlapping sampling and training changes the computational bottleneck and introduces policy lag. It is unclear whether PrePO retains its efficiency and performance benefits in this setting.

Questions for authors
  1. With sufficient training time for both methods to approach convergence, can PrePO at least match the baseline’s final performance? Longer training results would help establish whether its faster progress preserves eventual performance.

  2. Does the entropy drop reliably correspond to increasing prediction error? How sensitive is the fallback threshold, and can it be reused across models and tasks without tuning?

  3. How much do the reported time savings depend on synchronous rollout collection? Can PrePO be applied directly to asynchronous sampling while retaining wall-clock gains and final performance when sampling overlaps with training and rollouts become stale?

9 comments
AuthorsOpenReview ↗

Summary of Responses to Reviewer buEH's Comments (Part 1/5)

Thank you for your thoughtful and constructive review. We appreciate your recognition that PrePO gives practical wall-clock gains, not only fewer steps. We also appreciate your note on the breadth of our evaluation, across model sizes, optimizers and training objectives, and on our direct measurements of prediction error, where the real same-batch updates run beside the prediction. Your questions ask when the extra update cost pays off, whether faster early progress costs later performance, whether the gate reliably tells when prediction should stop, and whether the gain depends on synchronous rollouts. We answer each below.

Summary of our response

  1. We stated when PrePO saves wall-clock time: when its step reduction exceeds the extra cost of each step, which is small when rollouts dominate and larger in OPD with one sample per prompt.
  2. We trained GRPO and PrePO to 1,000 steps. PrePO stays ahead at every checkpoint (Table 10 and Figure A5, Appendix D.8).
  3. We tested what the entropy gate protects against, with entropy breaking its trend upward and downward, from the same checkpoint (Table 8, Appendix D.6).
  4. We explained why the gate works and why its threshold, fixed in advance, needs no tuning per run.
  5. We ran GRPO and PrePO with asynchronous rollouts, one step behind the learner (Section 7.6 and Figure 7 in the main text, Table 11 in Appendix D.9).
AuthorsOpenReview ↗

Response on When the Extra Update Cost Pays Off (Part 2/5)

Workload dependence: For 3B and 7B OPD, PrePO reduces training steps but increases elapsed time. The paper should more clearly identify when the additional update cost is outweighed by fewer rollout rounds.

Thank you for raising this. We agree the condition should be stated plainly. Section 7.4 gives it in short form, and a rough estimate makes it concrete.

A rough rule. A training step has two parts: generating rollouts and updating the model. PrePO keeps the generation and makes the update three backward passes instead of one. So a PrePO step costs roughly 1 + 2u plain steps, where u is the update's share of a plain step. This ignores smaller costs such as moving saved states, but it is consistent with the step costs reported in the paper. PrePO saves wall-clock time when its step reduction is larger than this factor.

The three settings in the paper sit at different points on this rule.

SettingUpdate share of a stepCost of a PrePO stepStep reductionWall-clock
RLVR (several samples per prompt)smallabout 1.5×3.2× to 4.9×2.1× to 3.4× faster
SFT (no rollouts)allabout 3×about 4×1.2× to 1.4× faster
OPD (one sample per prompt)large2.4× to 3×1.9× to 2.0× at 3B and 7B0.79× to 0.82×

In RLVR, generation takes most of each step, so the extra passes are cheap (Section 7.2). OPD samples only one response per prompt, so generation is short and the update dominates. At 3B and 7B, PrePO halves the steps, but each step costs more than twice as much, so elapsed time rises by 22% to 27% (Section 7.4). At 1.5B the step reduction is 4.75×, larger than the cost, and PrePO is 1.58× faster.

What this means in practice. PrePO saves time when rollouts are the expensive part of training: many samples per prompt, long responses, or slow generation. This is the usual case in RLVR at scale. When rollouts are cheap, as in OPD with one sample, PrePO still needs about half the rollouts and reaches a higher score (Table 3), but it should not be expected to save time. With asynchronous rollouts, the extra passes run while the next batch is generated, so their cost is hidden further (Part 5).

AuthorsOpenReview ↗

Response on Longer Training (Part 3/5)

Longer training: With enough training for both methods to approach convergence, does PrePO at least match the baseline's final performance?

We extended GRPO and PrePO to 1,000 steps on Qwen2.5-Math-1.5B with the same training and evaluation settings. The table gives average held-out accuracy over six benchmarks at matched steps (Table 10). Figure A5 shows the full curves (Appendix D.8).

Training step1002504005007501,000
GRPO33.7735.3236.1237.5539.0638.54
PrePO35.6838.5539.1339.5540.0839.66

Why the early advantage lasts. GRPO narrows the gap until about step 750 and then levels off below PrePO, while PrePO keeps what it gained early. We believe the reason is where PrePO's speed comes from. A larger learning rate also makes a run faster, but it stretches one gradient by the same amount in every direction. It pulls the policy quickly toward answers it already samples well and spends exploration early, so it tends to plateau lower (Appendix D.7). PrePO is fast in a different way. Its two probes only measure where GRPO's own updates on the batch are heading. They are then discarded, so the model still takes one update per batch, as GRPO does. That update moves only part of the way toward the prediction and then takes one real gradient step (Lemma 3). PrePO therefore skips ahead along GRPO's own path instead of taking a more aggressive one, and its token entropy stays close to GRPO's (Figure 6).

When the gate closes, both methods use the same optimizer. PrePO simply starts from a better point with its exploration largely intact, so its early progress is kept rather than paid back. Within the tested budget, PrePO's early speed does not cost later accuracy. We cannot say how the two would compare with unlimited training.

Reviewer_615JOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PrePO to reduce the cost of updating several times on the same rollout batch in RLVR. It first takes two temporary updates on the batch, fits a per-coordinate (AdamW) or per-matrix (Muon) motion trend from the ratio of the two displacements, and extrapolates the endpoint of KK updates. It then restores the original parameters and optimizer state, moves a fraction α\alpha toward the predicted endpoint, computes one real gradient there and takes a single update, so each step uses three backward passes. An entropy gate permanently switches off prediction once token entropy drops sharply, after which training falls back to ordinary updates. Experiments run mathematical RLVR on Qwen2.5-Math and Llama-3.2, and extend to Muon, SFT, on-policy distillation and a small code task. In mathematical RL, the number of steps needed to reach the baseline's peak drops by about 3 to 5 times, and wall-clock time by about 2 to 3 times.

Critical strengths

The gains in mathematical RL are of practical value. Wall-clock time includes rollout and parameter offloading, and the final average score is higher on all three models; the paper also reports honestly that OPD becomes slower in wall-clock time at 3B and 7B.

The method is simple, leaves the loss and data unchanged, and its cost of three backward passes does not grow with KK. The appendix directly compares the predicted endpoint with the endpoint of KK real updates on the same batch, and the two point in nearly the same direction, which shows that the same-batch trajectory is indeed very regular.

Critical weaknesses

The gains can be explained by a larger effective learning rate, and the paper lacks this most direct control. Lemma 3 shows that one PrePO step is a momentum step with an enlarged per-coordinate step size, whose upper bound in the main experiments is about 3.5 times the ordinary step. The fitted trends are almost all close to 1, and in the appendix simply repeating the last update gives nearly the same result as PrePO, so the trend extrapolation plays little role; after the entropy gate closes, training returns to ordinary updates, which acts like a learning-rate decay. The authors should compare GRPO with the learning rate enlarged by 3.5 and 5 times under the same setting, optionally falling back to the original learning rate when entropy drops, and report the steps and wall-clock time to reach the target, the peak and final accuracy, the training reward curve, and whether training remains stable. Since PrePO amounts to a momentum step, a baseline with a larger momentum coefficient for Muon or a larger β1 for AdamW should also be compared. If these simple alternatives reach similar results, the extra backward passes of PrePO are unnecessary.

The paper does not compare training dynamics with the baseline, so it is hard to tell how PrePO changes the training process. Token entropy and response length are shown only within PrePO's own sweeps over α and K, not against GRPO in the main experiments. Yet the Muon results in the appendix show that a larger α leads to longer responses and that the difference persists after prediction stops, and the entropy gate switches off prediction based on a sharp drop in entropy. The authors should provide a comparison of response length and entropy between the baseline and PrePO in the main experiments and, if the rebuttal allows revising the manuscript, preferably show these trends over training as figures.

Questions for authors
  1. Please compare GRPO with the learning rate enlarged by 3.5 and 5 times under the same setting, and report the steps and wall-clock time to reach the target, the peak and final accuracy, the training reward curve, and training stability.

  2. Please compare a baseline with a larger momentum coefficient for Muon or a larger β1 for AdamW, and show whether the gain of PrePO goes beyond the effect of stronger momentum.

  3. Please provide a comparison of response length and entropy between the baseline and PrePO in the main experiments; if the rebuttal allows revising the manuscript, please show them as training curves.

Update: Thank you for the author's rebuttal. I have updated my score.

7 comments
AuthorsOpenReview ↗

Summary of Responses to Reviewer 615J's Comments (Part 1/5)

Thank you for the careful review. We appreciate your recognition that the gains in mathematical RL are of practical value, measured in wall-clock time with rollout and offloading included, with higher final averages on all three models and an open report of the OPD slowdown. We also appreciate your recognition that PrePO is simple, leaves the loss and data unchanged, and keeps its cost at three backward passes for any K, and that Appendix D.3 confirms the same-batch trajectory is regular. That regularity is the premise of the method.

Summary of our response

  1. We trained GRPO with the learning rate enlarged 3.5× and 5×, with and without returning to η at PrePO's gate, over 3 seeds (Table 9 and Figure A4, Appendix D.7).
  2. We explained why a larger rate is fast early but ends lower, and why a decay alone does not close the gap.
  3. We logged AdamW's per-coordinate retention (Figure A6, Appendix D.11) and explained why a PrePO step is neither a uniformly larger step nor a repeated one.
  4. We added a stronger-momentum baseline, AdamW with a larger β₁ (Table 9).
  5. We added token entropy and response length of GRPO and PrePO for all six main runs to the main text (Figure 6, Section 7.3).
AuthorsOpenReview ↗

Response on a Larger Learning Rate and Its Decay (Part 2/5)

Larger learning rate: Please compare GRPO with the learning rate enlarged by 3.5 and 5 times under the same setting, and report the steps and wall-clock time to reach the target, the peak and final accuracy, the training reward curve, and training stability.

We ran both rates in the setting of Table 2 with 3 seeds, with and without the fallback to η. Results are in Table 9, and the training curves, including the training reward, are in Figure A4 (Appendix D.7).

MethodSteps to GRPO peakWall-clock speedupPeakFinal (step 400)
GRPO, η3921.00×36.2936.1
GRPO, 3.5η1392.82×37.435.3
GRPO, 3.5η → η1442.72×38.137.6
GRPO, 5η1322.97×37.132.8
GRPO, 5η → η1342.93×37.336.1
PrePO1132.33×40.3039.1

Why a larger rate peaks early and then declines. Early in training the gradient keeps pointing the same way. A larger step covers that path faster, and in wall-clock time it reaches GRPO's peak even sooner than PrePO, as Table 9 shows. The cost comes later. Each large update moves the policy further from the one that generated its rollouts, toward answers it already samples well, so exploration is spent early. GRPO learns by comparing the answers sampled for one problem. Once the policy has narrowed, those answers look alike, and what remains of the gradient is mostly sampling noise. A large step turns that noise into a large move. The training reward also levels off lower, at about 0.55 against PrePO's 0.60 (Figure A4), so the early speed costs accuracy on the training problems too.

What returning to η changes. With the fallback, each larger rate returns to η under PrePO's own gate. This directly tests your point that the gate acts as a decay. The decay removes the late decline, but it cannot give back the entropy already spent. These runs plateau below PrePO, and the run that spent more plateaus lower. So the decay is shared by all runs; what differs is the state each run brings into it. The plain larger rates never reach 37.5. To reach 38.0, 3.5η → η needs 246 steps and PrePO needs 149, about 10% less wall-clock time even with its extra passes. Only PrePO reaches 38.5. A larger rate is the faster route to GRPO's level; PrePO is the faster route past it.

AuthorsOpenReview ↗

Response on Fitted Trends and Repeating the Last Update (Part 3/5)

Trend extrapolation: The fitted trends are almost all close to 1, and in the appendix simply repeating the last update gives nearly the same result as PrePO, so the trend extrapolation plays little role.

Part 2 shows the outcome. Here we explain why a PrePO step differs from a larger or repeated step.

The trends near one are Muon's. The value 0.96 in Section 7.3 is Muon's trend, one number per weight matrix. AdamW, which gives all the main results in Table 2, fits a separate retention for each coordinate. We logged it (Figure A6, Appendix D.11). It settles between 0.72 and 0.78, with a mean of about 0.74, for every K in {3, 5, 10} and every α in {0.1, 0.3, 0.5, 0.8}. So for AdamW the trend is clearly below one. Appendix D.3 mentions both: the trends near one, and AdamW's retention of about 0.74.

Similar size, different shape. With a retention of 0.74, the K-step look-ahead adds up to about 3.0 steps at K = 5 and 3.7 at K = 10, not 5 and 10. On average, then, PrePO does take a step about three times larger, close to your 3.5η baseline. The difference is how that step is spread. A larger learning rate stretches every coordinate by the same amount. PrePO gives the large step only to coordinates that keep moving the same way. A coordinate that reverses gets at most about two plain steps, whatever K is (Proposition 2). The uniform version of a step this size is the 3.5η run in Part 2, which ends at 35.3 while PrePO ends at 39.1. So the size of the step does not explain the gain.

Why repeating the last update lands close. You are right that in Figure A2 (Appendix D.3), repeating the last update lands near PrePO's predicted endpoint. We read this as follows. Both methods agree on coordinates that keep moving steadily, and these appear to set most of the endpoint. They differ on coordinates that slow down or reverse, where repetition keeps the full step and PrePO shortens it (Proposition 2). We therefore see the fitted trend as a safeguard, not the main source of the gain, as Appendix D.3 also states. Repetition alone is again a uniform stretch, which is what the 3.5η run tests.

Where the gain comes from. PrePO's probes give it something a larger rate never sees: a second gradient on the same batch, taken where the optimizer is heading (Lemma 3). The probes are only measurements and are discarded, so the model still takes one update per batch, as GRPO does (Section 5). That update moves only a fraction α toward the prediction and then takes one real step from there, so any prediction error is scaled down by α (Appendix C). We do not claim PrePO is more stable in every setting. In our runs, this partial step and the gate let PrePO reach the gate with entropy close to GRPO's, and its final accuracy is above every baseline's peak (Figure A4).

Comments on the paper

1 comment

Dear all, The discussion phase will end soon. The authors have provided their responses to your comments. If the provided responses do not address (even in part) your concerns, this is your chance to raise them. Please engage with the authors timely. We really appreciate your efforts! Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.