- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PIPO uses reward trends relative to a historical window to control the sign and strength of retrospective updates on the previous batch. The paper provides conditional alignment analysis and evaluates this plug-in across several RL optimizers, two model scales, and mathematics, coding, tool-use, and self-distillation tasks.
Critical strengths
Modularity and consistent empirical gains. PIPO improves aggregate performance across PPO, GRPO, GSPO, and DAPO in multiple task domains. Supplementary multi-seed results and accuracy–wall-clock comparisons strengthen the evidence of practical utility. These are substantive strengths because the method is reusable rather than specific to one optimizer or task.
Critical weaknesses
The central methodological contribution is insufficiently established. The fixed-replay control and negative-feedback ablation provide partial evidence, but the fixed-replay baseline uses only an untuned coefficient . These comparisons do not adequately distinguish informative, update-specific feedback from a better choice of replay strength or schedule. Consequently, the results establish a useful adaptive replay heuristic, but do not convincingly establish the distinctive value of the proposed policy-improvement feedback.
The theory leaves the mechanism’s central premise unverified. The alignment result assumes that historical reward trends agree with isolated base-step improvements, rather than establishing this relationship for the actual training process. It also requires attribution matching; the separate groupwise proportionality result does not establish aggregate direction matching when normalization weights differ across groups. These are central applicability gaps: the analysis does not validate whether the implemented controller reliably distinguishes beneficial from harmful updates.
Questions for authors
Clarification: What justifies attribution matching for the actual batch-level, group-normalized updates? Was agreement between historical reward trends and isolated base-step improvements measured during training?
Additional experimental evidence: Does PIPO outperform tuned fixed-strength replay and shuffled-feedback controls that preserve the coefficient distribution within comparable training stages, under equal compute and identical attribution normalization? Such evidence would directly affect my assessment of the central contribution.
We appreciate reviewer w3an’s detailed review and constructive questions about tuned fixed-strength and shuffled-replay controls, batch-level attribution matching, and the relationship between historical reward trends and isolated base-step gains. We have revised the manuscript accordingly, with all additions and revisions highlighted in red in the updated PDF.
We added fixed-replay controls with c=0.1,0.5 and shuffled-feedback controls at both model scales, alongside the existing c=1 control. All replay variants use the same attribution normalization and rollout budget. For the shuffled control, we randomly permuted the policy-improvement reward coefficients recorded from the PIPO run within consecutive blocks of 16 training steps.
Qwen3-4B-Base
Qwen3-8B-Base
PIPO exceeds the strongest fixed-replay control by 1.8 points on 4B and 3.0 points on 8B, and also outperforms shuffled replay at both scales. These results support the effectiveness of PIPO’s feedback-guided replay.
Batch-level matching. We measured cosine similarity between the GRPO policy gradient and PIPO’s normalized replay policy gradient before scalar feedback, using the same full batch at identical post-base parameters over 20 consecutive post-warm-up steps. All 20 cosines were positive, with a median of 0.944 and a range of 0.863–0.956, supporting batch-level directional matching in our local analysis.
Agreement with isolated base-step gains. PIPO uses the next batch’s mean reward to estimate post-update performance and compares it with historical rewards to guide replay. Over 20 post-warm-up updates, we compared the subsequent K=8 feedback with changes in the five-benchmark average from paired pass@1 evaluations immediately before and after each base update. Sign agreement was 13/20 (65%) at batch size 128 and 16/20 (80%) at batch size 256. Across these updates, positive contributions to the estimated leading alignment term exceeded negative contributions in magnitude, with positive-to-negative ratios of 1.19 for B128 and 4.55 for B256.
We also supplemented the local analysis with a feedback-error bound (Appendix C.3, Proposition C.3 of the revised manuscript). Decomposing the reward difference as δt=Δt+bt+εt, with base-step gain Δt, historical-window discrepancy bt, and sampling error εt, gives, for 0<λ≤1,
\frac{\phi_\lambda(\Delta_t)\Delta_t-|\Delta_t|(|b_t|+|\varepsilon_t|)}{s_t},$$ where $s_t>0$ is the normalization denominator. The proof uses the rectification function’s positive homogeneity and Lipschitz continuity, then substitutes this inequality into the original local expansion to bound the effect of feedback error on alignment.