- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EPIC, a prefix-level clipping method for off-policy reinforcement learning of autoregressive language models. The method constructs adaptive clipping bounds using cumulative prefix ratios and the exact reverse-KL coordinate, together with behavior-policy probability calibration and a geometric-mean optimization weight. Experiments on Qwen3-30B-A3B-Base show that EPIC outperforms baselines such as CTPO and GSPO on average across four mathematical reasoning benchmarks.
Critical strengths
-
The paper derives a prefix-level performance-difference identity for autoregressive generation and establishes an exact relationship between the expected reverse-KL coordinate and prefix-distribution KL divergence, providing a theoretical motivation for the proposed method.
-
EPIC achieves promising empirical results across four mathematical reasoning benchmarks, outperforming baselines such as CTPO and GSPO in the evaluated off-policy MoE training setting.
-
EPIC can be integrated into existing PPO-style training pipelines without additional model forward passes or expert-routing replay. The paper also provides useful analyses of clipping behavior and gradient norms to examine training stability.
Critical weaknesses
-
The paper contains extensive mathematical derivations but lacks clear and intuitive explanations of its core motivation and contributions. In particular, CTPO already employs cumulative prefix ratios and position-dependent clipping bounds. The paper does not sufficiently clarify which limitations of CTPO remain unresolved or what the key methodological advances of EPIC are.
-
The paper does not adequately justify the choice of the exact reverse-KL coordinate or the proposed position-dependent coordinate budgets. The theoretical analysis does not sufficiently establish the advantages of these design choices over existing clipping methods.
-
The experimental validation remains insufficient. The ablations in Appendix E.4 do not directly isolate the contribution of the exact reverse-KL coordinate or comprehensively validate the proposed coordinate budgets. Sensitivity analyses of the budget parameters and are also missing. Furthermore, the experiments are limited to a single sparse MoE model, without evaluation across other model scales or architectures.
Questions for authors
-
CTPO already employs cumulative prefix ratios and position-dependent clipping bounds. Could the authors clarify which specific limitations of CTPO are addressed by EPIC and what its main methodological contributions are?
-
Could the authors provide further justification for adopting the exact reverse-KL coordinate and the proposed position-dependent coordinate budgets? What specific advantages do these designs offer over existing clipping methods, particularly CTPO?
-
Could the authors provide more targeted ablation studies to isolate the effects of the exact reverse-KL coordinate and the proposed coordinate budgets, along with sensitivity analyses of (\delta^-) and (\delta^+)? Additionally, could the authors evaluate EPIC on other model scales or dense architectures to assess its generalizability?
3 comments
Thank you for providing the additional information, which has addressed some of my concerns. However, my main concern remains unresolved, so I will maintain my original rating.
Dear reviewer,
We sincerely thank you for considering our response and acknowledging that the additional information addressed some of your concerns. We appreciate your time and would like to further emphasize EPIC’s theoretical contribution and practical value.
EPIC Provides a Principled Connection Between Autoregressive Policy Improvement and Clipping
EPIC addresses a fundamental design question: which policy-induced distribution should a trust region constrain, and how should that constraint determine clipping boundaries? Starting from the autoregressive MDP, EPIC identifies the prefix ratio as the exact correction for changes in both context visitation and current-token selection, then connects prefix-distribution KL to clipping through an exact coordinate. This provides a coherent theoretical basis for both the choice of prefixes and the construction of their clipping bounds.
Although CTPO also uses prefix correction, its position-dependent thresholds are motivated by cumulative log-ratio variance growth. EPIC isn't come from a modification of CTPO, it's contribution is the connection between performance improvement, prefix-distribution divergence, and clipping design.
EPIC’s Theoretical Design Choices Have Demonstrable Practical Value
EPIC combines this theoretical foundation with stronger empirical performance. In the evaluated off-policy MoE setting, it outperforms CTPO and the other clipping baselines, while exhibiting lower cross-seed variability than CTPO. Crucially, the exact coordinate contributes beyond the shared prefix structure: with budgets and other settings fixed, EPIC outperforms both quadratic-coordinate alternatives on all four benchmarks. These results directly support the practical value of the theoretically motivated coordinate.
EPIC Offers an Efficient and Robust Approach to Off-Policy Training
EPIC integrates into PPO-style pipelines without additional model forward passes, auxiliary networks, or routing-state storage. EPIC demonstrates strong effectiveness in MoE training, while additional experiments provide evidence of its potential to generalize beyond this setting. Together, these findings support EPIC as a theoretically motivated method with practical benefits in performance, robustness, and implementation efficiency.
If possible, we would appreciate further clarification of the specific concern that remains unresolved, so that we can respond more directly and guide further improvements. We will continue refining the theoretical presentation and strengthening the empirical evaluation. Thank you again for your careful review and constructive input.
Rebuttal by Authors
We thank the reviewer for recognizing EPIC’s theoretical motivation, empirical performance, and practical efficiency. We address the concerns below:
Q1. Main contributions and distinction from CTPO
We thank the reviewer for highlighting this distinction. EPIC asks which policy-induced distribution a trust region should constrain and how its divergence budget should determine clipping boundaries. We start from autoregressive MDPs and TRPO-style policy improvement, rather than modifying CTPO. A policy update changes both the probability of reaching a generated context and selecting the current token. Our performance-difference identity identifies the prefix ratio as the exact correction for this joint change, motivating a prefix-distribution trust region.
We have recognized that CTPO also derives prefix correction, but motivates log-space thresholds proportional to t through position-dependent cumulative log-ratio variance growth. EPIC instead connects performance improvement, prefix-distribution KL, and clipping boundaries through the exact KL coordinate, with an effective practical implementation. Its contribution thus extends beyond using prefix ratios.
In our experiments, we observed that CTPO still exhibits larger gradient spikes and less balanced position-wise clipping despite its prefix correction. CTPO and EPIC achieve four-benchmark means of 33.98±2.04% and 39.34±1.14%, respectively. Peak gradient norms range from 2.83–9.67 for CTPO versus 1.34–1.73 for EPIC. These observations highlight limitations of CTPO in the evaluated off-policy MoE setting and support EPIC as a complete method; the ablations below isolate the coordinate’s contribution.
Q2. Justification and coordinate ablations
We thank the reviewer for encouraging an intuitive explanation. Sections 3.2–3.3 provide the theoretical justification for these designs, which we summarize below alongside targeted empirical evidence.
Theoretical justification. Three steps connect the theory to clipping:
Correction. In the autoregressive MDP, the state and current action identify the current prefix. Expressing each improvement term under behavior-policy sampling requires the prefix importance ratio, motivating measurement of policy change on these same distributions.
Trust region. Bounding prefix KL by tδ at every length t bounds the maximum KL per prefix token, preventing large deviations at one length from being averaged away by smaller deviations elsewhere.
Clipping boundaries. The exact identity is
Kx,t(μ,π)=Eμ[ϕ(rtpre)],ϕ(r)=r−1−logr.
Since ϕ is minimized at one and increases away from it on either side, a samplewise budget determines lower and upper ratio boundaries.
Thus, an intuitive benefit of this coordinate is that it preserves the exact prefix-KL relationship, with is the distribution-level quantity constrained by the trust region; a quadratic, like GRPO or CTPO, retains only its local approximation around one.
Coordinate ablation. We replace only the constrained coordinate with (r−1)2/2 or (logr)2/2, keeping budgets and other settings fixed:
Q3. Budget sensitivity and dense-model generalization
We thank the reviewer for encouraging more comprehensive evidence. Q2 discusses the exact coordinate and proposed budgets; below we examine other sensitivity and generalization.
Budget sensitivity. Budgets are empirical hyperparameters and the second-order approximation and prior empirical settings informed the 10−3 budget scale. Sensitivity tests within this scale, using the same training recipe, showed effective learning across all tested settings:
Dense models. Our primary focus is off-policy training stability, particularly in MoE models where routing discrepancies introduce rollout–training mismatch. To explore applicability beyond this setting, we also evaluated GRPO and EPIC on dense Qwen3-0.6B and Qwen-1.7B models:
EPIC outperforms GRPO on the four-benchmark average, suggesting potential for transfer to dense architectures.
We hope our response and the additional evidence adequately address your concerns. We kindly invite you to reconsider the overall evaluation of our work.