acceptodds
Under review as a conference paper at ICLR 2027

Inter-Iteration Policy Improvement for LLM Reinforcement Learning

Abstract

Reinforcement learning is central to post-training LLMs for reasoning and agentic tasks. However, existing methods optimize rewards, advantages, or distillation targets constructed from the current batch without using subsequent performance observations to inform later optimization. This open-loop structure leaves recent policy progress outside the learning signal and makes optimization sensitive to finite sampling, generation stochasticity, and noisy feedback. We introduce Policy Improvement Reinforcement Learning (PIRL), a trajectory-level formulation that makes inter-iteration policy improvement an explicit optimization signal while retaining cumulative equivalence to terminal task performance. Building on PIRL, we propose Policy Improvement Policy Optimization (PIPO), a plug-in framework that closes the loop around policy updates. At each iteration, PIPO compares current empirical performance with a sliding-window historical anchor and uses the resulting progress signal to retrospectively modulate the base algorithm's local attribution. We establish local alignment with isolated base-step improvement under explicit trend-coherence conditions and characterize signed retrospective replay for group-relative optimization. Across two model scales and mathematical reasoning, code, tool-use, and self-distillation tasks, PIPO consistently improves PPO, GRPO, GSPO, DAPO, and SDPO under matched training and evaluation settings across all reported experiments.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reinforcement Learning with Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_w3anOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PIPO uses reward trends relative to a historical window to control the sign and strength of retrospective updates on the previous batch. The paper provides conditional alignment analysis and evaluates this plug-in across several RL optimizers, two model scales, and mathematics, coding, tool-use, and self-distillation tasks.

Critical strengths

Modularity and consistent empirical gains. PIPO improves aggregate performance across PPO, GRPO, GSPO, and DAPO in multiple task domains. Supplementary multi-seed results and accuracy–wall-clock comparisons strengthen the evidence of practical utility. These are substantive strengths because the method is reusable rather than specific to one optimizer or task.

Critical weaknesses

The central methodological contribution is insufficiently established. The fixed-replay control and negative-feedback ablation provide partial evidence, but the fixed-replay baseline uses only an untuned coefficient c=1c=1. These comparisons do not adequately distinguish informative, update-specific feedback from a better choice of replay strength or schedule. Consequently, the results establish a useful adaptive replay heuristic, but do not convincingly establish the distinctive value of the proposed policy-improvement feedback.

The theory leaves the mechanism’s central premise unverified. The alignment result assumes that historical reward trends agree with isolated base-step improvements, rather than establishing this relationship for the actual training process. It also requires attribution matching; the separate groupwise proportionality result does not establish aggregate direction matching when normalization weights differ across groups. These are central applicability gaps: the analysis does not validate whether the implemented controller reliably distinguishes beneficial from harmful updates.

Questions for authors

Clarification: What justifies attribution matching for the actual batch-level, group-normalized updates? Was agreement between historical reward trends and isolated base-step improvements measured during training?

Additional experimental evidence: Does PIPO outperform tuned fixed-strength replay and shuffled-feedback controls that preserve the coefficient distribution within comparable training stages, under equal compute and identical attribution normalization? Such evidence would directly affect my assessment of the central contribution.

1 comment

We appreciate reviewer w3an’s detailed review and constructive questions about tuned fixed-strength and shuffled-replay controls, batch-level attribution matching, and the relationship between historical reward trends and isolated base-step gains. We have revised the manuscript accordingly, with all additions and revisions highlighted in red in the updated PDF.

W1 & Q2: Fixed and shuffled replay controls

We added fixed-replay controls with c=0.1,0.5c=0.1,0.5 and shuffled-feedback controls at both model scales, alongside the existing c=1c=1 control. All replay variants use the same attribution normalization and rollout budget. For the shuffled control, we randomly permuted the policy-improvement reward coefficients recorded from the PIPO run within consecutive blocks of 16 training steps.

Qwen3-4B-Base

MethodMATH500AIME25AMC23MinervaOlympiadAvg
GRPO79.318.560.021.040.543.9
GRPO + fixed replay (c=0.1c=0.1)75.922.255.022.440.043.1
GRPO + fixed replay (c=0.5c=0.5)76.314.855.022.139.441.5
GRPO + fixed replay (c=1c=1)75.914.867.522.141.144.3
GRPO + shuffled replay76.518.562.525.039.244.3
GRPO + PIPO80.918.560.026.844.446.1

Qwen3-8B-Base

MethodMATH500AIME25AMC23MinervaOlympiadAvg
GRPO80.122.267.527.642.448.0
GRPO + fixed replay (c=0.1c=0.1)75.925.962.524.340.845.9
GRPO + fixed replay (c=0.5c=0.5)76.714.865.023.242.344.4
GRPO + fixed replay (c=1c=1)80.122.260.025.435.644.7
GRPO + shuffled replay78.114.862.527.238.944.3
GRPO + PIPO82.322.267.529.043.648.9

PIPO exceeds the strongest fixed-replay control by 1.8 points on 4B and 3.0 points on 8B, and also outperforms shuffled replay at both scales. These results support the effectiveness of PIPO’s feedback-guided replay.

W2 & Q1: Clarification on Attribution Matching and Agreement with Isolated Base-Step Gains

Batch-level matching. We measured cosine similarity between the GRPO policy gradient and PIPO’s normalized replay policy gradient before scalar feedback, using the same full batch at identical post-base parameters over 20 consecutive post-warm-up steps. All 20 cosines were positive, with a median of 0.944 and a range of 0.863–0.956, supporting batch-level directional matching in our local analysis.

Agreement with isolated base-step gains. PIPO uses the next batch’s mean reward to estimate post-update performance and compares it with historical rewards to guide replay. Over 20 post-warm-up updates, we compared the subsequent K=8K=8 feedback with changes in the five-benchmark average from paired pass@1 evaluations immediately before and after each base update. Sign agreement was 13/20 (65%) at batch size 128 and 16/20 (80%) at batch size 256. Across these updates, positive contributions to the estimated leading alignment term exceeded negative contributions in magnitude, with positive-to-negative ratios of 1.19 for B128 and 4.55 for B256.

We also supplemented the local analysis with a feedback-error bound (Appendix C.3, Proposition C.3 of the revised manuscript). Decomposing the reward difference as δ^t=Δt+bt+εt\widehat\delta_t=\Delta_t+b_t+\varepsilon_t, with base-step gain Δt\Delta_t, historical-window discrepancy btb_t, and sampling error εt\varepsilon_t, gives, for 0<λ≤10<\lambda\le1,

\frac{\phi_\lambda(\Delta_t)\Delta_t-|\Delta_t|(|b_t|+|\varepsilon_t|)}{s_t},$$ where $s_t>0$ is the normalization denominator. The proof uses the rectification function’s positive homogeneity and Lipschitz continuity, then substitutes this inequality into the original local expansion to bound the effect of feedback error on alignment.
Reviewer_7kjrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focused on RL for LLMs. The main motivation is to use information beyond the per-step policy gradient (which traditional RL methods do). Their proposed framework of PIRL involves examining how much the realized expectation of total returns for each policy iteration improves. This is realized via the algorithm PIPO. In PIPO, the current performance is compared to a Gaussian distribution fit over past iteration performance. When the relative performance of the current policy is high, the algorithm further attenuates this update. This is implemented via an additional "phase 2" update which in essence takes an additional policy gradient step on current batch of rollouts, scaled on its improvement vs. the historical average. Empirical results are examined on Qwen models using various base RL algorithms on MATH and tool-use tasks, along with a self-distillation benchmark.

Critical strengths

The proposed algorithm appears novel and empirical results show it provides a consistent improvement over prior methods. The implementation of the method appears simple and does not require much additional compute, instead simply using the same rollouts again for a second gradient step. The evaluation suite is thorough and covers many base RL algorithms, tasks, and models.

Critical weaknesses

The way the method is presented is confusing, using unstandard RL terminology such as "attribution" and "feedback". In this way, the novel portion of the method is hard to understand. The paper would benefit from a didactic example focusing in on one case (e.g. GRPO) and providing concrete equations for each step. The algorihtm box only contains a high-level description. For example, it is hard to tell whether the additional policy gradient step is taken on the current batch of data (in which case, how does it differ from the standard RL update?), or the previous batch (in which case, what motivates this?).

The method itself adds a number of additional hyperparameters, such as the weighting of the additional update, an algorithm-specific epsilon, and a sign-dependent rectification term.

The evaluation setting leaves questions. As to my understanding this method involves taking a dynamically-sized second policy gradient step, was the base learning rate or clip ratio of the base methods swept to provide a fair baseline?

The related work section is minimal and more related work should be cited.

Questions for authors

As to my understanding this method involves taking a dynamically-sized second policy gradient step, was the base learning rate or clip ratio of the base methods swept to provide a fair baseline?

2 comments

We appreciate reviewer 7kjr’s detailed review and constructive comments on the clarity of PIPO’s update procedure, its additional hyperparameters, learning-rate and clipping comparisons, and the coverage of related work. We have revised the manuscript accordingly, with the corresponding additions and revisions highlighted in red in the updated PDF.

W1: Clarity of terminology and the update procedure

Terminology. PIPO applies to PPO, GRPO, and SDPO, so we use “attribution” as a common term for the learning signal derived from each method, such as GRPO advantages or the teacher–student log-probability difference in SDPO. “Feedback” refers to the normalized difference between the new batch’s mean reward and the average over the preceding KK batches. This normalization uses historical means and standard deviations; no Gaussian distribution is fitted or assumed.

Replay batch and motivation. PIPO performs its additional retrospective update on the previous batch. Since the next batch is sampled from the updated policy, its mean reward offers a natural estimate of post-update performance. PIPO uses the normalized feedback to modulate the previous batch’s attribution and construct the policy improvement reward for replay.

GRPO case. At step tt, we sample Bt\mathcal B_t, compute advantages within each prompt group, and perform a GRPO update. At step t+1t+1, we sample Bt+1\mathcal B_{t+1} and compare its mean reward with the historical average to obtain policy improvement feedback. We use this feedback to replay Bt\mathcal B_t. To see how this feedback affects a replayed response, suppose yty_t in Bt\mathcal B_t has a normalized replay advantage of 1. Feedback of 0.5 then gives a policy improvement reward of 0.5, reinforcing its original update direction. The coefficient λ\lambda controls negative-feedback strength: with λ\lambda set to 0.1, feedback of −0.5 gives a policy improvement reward of −0.05, counteracting that direction. Algorithm 1 in the revised manuscript now explicitly specifies the feedback computation, policy improvement reward construction, and separate updates on the previous and current batches.

W2: Additional hyperparameters

Weight of the additional update. PIPO uses the baseline learning rate for both updates, without a separately tuned replay learning rate. The additional update is weighted by the policy improvement reward defined in Eq. (12).

Algorithm-specific epsilon. The epsilon terms are fixed numerical safeguards for normalization. They handle near-zero historical reward variation and zero group-attribution sums; their values are reported in Appendix A for reproducibility.

Sign-dependent rectification. The coefficient λ\lambda controls the strength of correction under negative feedback, while positive feedback retains its original magnitude. Our 4B ablation (Table 4) shows a five-benchmark average of 46.1 with the default λ=0.1\lambda=0.1, compared with 44.5 when negative feedback is not attenuated (λ=1\lambda=1). Across the tested values from 0.05 to 0.5, averages range from 45.6 to 46.1, above GRPO’s 43.9, indicating that the gains are not limited to the default setting.

W3 & Q1: Learning-rate and clipping sensitivity

We evaluated 4B GRPO with learning rates {0.5,1.0}×10−6\{0.5,1.0\}\times10^{-6} and clipping intervals [0.8,1.2][0.8,1.2] and [0.9,1.1][0.9,1.1]. The table includes the default GRPO and GRPO+PIPO results for reference. The strongest GRPO configuration in this comparison achieves a five-benchmark average of 43.9, while GRPO+PIPO reaches 46.1, retaining a 2.2-point advantage over the baseline configurations examined.

MethodLR (×10⁻⁶)Clip intervalMATH500AIME25AMC23MinervaOlympiadAvg
GRPO0.5[0.9, 1.1]75.514.862.525.038.043.2
GRPO0.5[0.8, 1.2]78.518.560.022.439.543.8
GRPO1.0[0.9, 1.1]76.718.552.523.936.541.6
GRPO1.0[0.8, 1.2]79.318.560.021.040.543.9
GRPO+PIPO1.0[0.8, 1.2]80.918.560.026.844.446.1

W4: Coverage of related work

We have expanded Appendix D with discussions of Safe Policy Iteration and High Confidence Policy Improvement, and a more detailed comparison with HA-DW. We discuss their connections to PIPO and clarify the differences in their update mechanisms.

Thank you to the authors for the detailed response, which is acknowledged. At the same time, my main concerns about the contents of this paper remain, and I opt to maintain my current score.

Reviewer_shk5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the idea of using policy improvement as a signal for further policy optimization. Specifically, it compares current empirical performance with a historical one and uses the resulting signal to modulate a retrospective update on previously collected data. While the idea is interesting, I have several concerns:

  1. The broader problem of evaluating and controlling policy improvement has been studied in previous work, including:
  • Safe Policy Iteration, Matteo Pirotta, Marcello Restelli, Alessio Pecorino, Daniele Calandriello. ICML 2013.
  • High Confidence Policy Improvement, Philip Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, ICML 2015.

Although these methods a bit differ from the proposed retrospective update, the paper should discuss them and explain more clearly what is new about its formulation and feedback mechanism. The related-work discussion is too shallow and does not adequately compare the contribution of this paper with previous works. Note that these are classic papers related to this paper; however, they are from 2013 and 2015, and there are more related works.

  1. The use of a historical anchor also warrants a detailed comparison with [*], which introduces History-Aware Adaptive Difficulty Weighting (HA-DW). Since this work appeared on arXiv in Jan-2026, it is fair to ask.

HA-DW uses historical performance to construct an evolving difficulty anchor, and its Appendix F describes an anchor based on recent batch accuracies. Although the present paper cites HA-DW and briefly distinguishes it in Appendix D, the comparison needs more detail. The authors should clarify the novelty of the historical anchor separately from that of the signed retrospective update, ideally with a direct comparison or ablation.

[*] Your Group-Relative Advantage Is Biased
Fengkai Yang, Zherui Chen, Xiaohan Wang, Xiaodong Lu, Jiajun Chai, Guojun Yin, Wei Lin, Shuai Ma, Fuzhen Zhuang, Deqing Wang, Yaodong Yang, Jianxin Li, Yikun Ban, NeurIPS 2026.

  1. The computational budget requires clearer discussion in the main experimental comparison. PIPO reuses historical samples and adds an optimization pass. The main results should more clearly distinguish comparisons at matched rollout budgets from comparisons at matched total computation.

  2. The evaluation configuration is not sufficiently clear. Math reasoning benchmark scores can depend on the maximum generation length, so the evaluation token budget matters when reading the results. Table 5 specifies a maximum response length of 4K tokens for training/(eval?) rollouts, but I could not find information about the evaluation limit. The results would be different if you used 16K tokens rather than 4K tokens.

  3. Confusing notation. For example, q denotes a query in Eq. (1), while Eq. (6) denotes the teacher. Also, in general, Section 4 is hard to follow. Since this is the main method of this paper, the expectation is to have more clarity. Also, when reading Algorithm 1, I need to go back and forth between it and the main paper because Eqs referenced but not discussed directly in the algorithm, which makes it hard to use.

Overall, while the paper follows an interesting direction, I am not convinced that it sufficiently establishes its novelty relative to prior work. These concerns, together with the limited comparative discussion and unclear evaluation details, l'd recommend against acceptance at ICLR in its current form.

Critical strengths

See above.

Critical weaknesses

See above.

Questions for authors

See above.

1 comment

We appreciate reviewer shk5’s detailed review and constructive comments on PIPO’s relationship to prior policy-improvement methods and HA-DW, the fairness of training-budget comparisons, and the clarity of our evaluation settings, notation, and algorithm description. We have revised the manuscript accordingly, with all additions and revisions highlighted in red in the updated PDF.

Concern 1: Relationship to prior work

Safe Policy Iteration uses a lower bound on improvement to select the next policy, while High Confidence Policy Improvement uses logged trajectories to test whether a candidate policy meets a specified return threshold with high confidence.

These classical RL methods use performance bounds to select or validate candidate policies. We share their motivation of using policy performance to guide learning, but introduce retrospective feedback for LLM post-training. PIRL formulates inter-iteration performance changes as an explicit learning signal, and PIPO uses the reward trend from the next batch to control the sign and strength of replay on the preceding batch.

Concern 2: Comparison with HA-DW

Both methods use historical reward statistics. HA-DW’s main formulation uses adaptive smoothing, while Appendix F discusses a simplified formulation based on recent batch averages. PIPO uses the mean reward of the preceding K=8K=8 batches as its reference.

The key difference lies in how this feedback controls learning. HA-DW compares each prompt group’s success rate with the difficulty anchor and reweights advantages in the current batch. PIPO compares the new batch’s mean reward with its historical reference and uses the resulting feedback to control the sign and strength of replay on the preceding batch.

The two mechanisms can be combined: HA-DW constructs the current-batch advantages, which PIPO reuses when replaying that batch at the next iteration. The tables include the reported GRPO and GRPO+PIPO results alongside our HA-DW comparisons. Adding PIPO to HA-DW improves the best five-benchmark average from 44.1 to 46.0 on 4B and 48.2 to 49.0 on 8B.

Qwen3-4B-Base

MethodMATH500AIME25AMC23MinervaOlympiadAvg
GRPO79.318.560.021.040.543.9
GRPO + HA-DW77.918.560.024.339.844.1
GRPO + PIPO80.918.560.026.844.446.1
GRPO + HA-DW + PIPO78.318.567.526.539.446.0

Qwen3-8B-Base

MethodMATH500AIME25AMC23MinervaOlympiadAvg
GRPO80.122.267.527.642.448.0
GRPO + HA-DW82.522.265.028.742.748.2
GRPO + PIPO82.322.267.529.043.648.9
GRPO + HA-DW + PIPO83.925.965.026.843.549.0

Concern 3: Matched budget

Our main experiments use the same rollout budget for each baseline and its PIPO variant. PIPO reuses cached responses for an additional optimization pass, which increases optimization cost without requiring additional rollouts.

We also evaluated wall-clock efficiency in Appendix B.5 of the original submission: by the time each baseline finishes training, PIPO has already reached higher accuracy. For PPO, we measure only a 1.6% increase in total wall-clock training time when adding PIPO (Figure 5 in Appendix B.5).

We further extend GRPO, GSPO, and DAPO training and compare the best five-benchmark averages reached within a common wall-clock window for each pair.

MethodWall-clock budget(h)MATH500AIME25AMC23MinervaOlympiadAvg
GRPO12.778.722.257.525.443.045.4
GRPO+PIPO12.780.918.560.026.844.446.1
GSPO11.681.318.557.524.642.044.8
GSPO+PIPO11.680.322.260.024.641.845.8
DAPO26.181.322.260.027.246.347.4
DAPO+PIPO26.182.722.270.025.046.349.2

PIPO achieves higher five-benchmark averages in all three 4B comparisons. These results complement the matched-rollout evaluation by examining performance under a shared training-time budget.

Concern 4: Evaluation generation limit

Math evaluation uses a maximum of 4,096 generated tokens per response, matching the training rollout limit reported in Table 5.

Concern 5: Notation and algorithm clarity

We agree that the notation and algorithm description could be clearer. We reserve qq for the query and use πteacher,t\pi_{\mathrm{teacher},t} for the teacher in Eq. (6). Algorithm 1 now explicitly includes the feedback calculation, replay weights, and update order, reducing the need to consult equations elsewhere in the manuscript. We also add a GRPO case in Section 4.2 to clarify the procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.