acceptodds
Under review as a conference paper at ICLR 2027

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-ratio Clipping

Abstract

Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-policy sampling. The resulting change of measure identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each stepwise policy-improvement term. Starting from the monotonic-improvement principle of trust-region policy optimization, we then construct a prefix-level trust region using the exact reverse-KL coordinate , whose behavior-policy expectation equals the corresponding prefix-distribution KL divergence. This yields EPIC, a PPO-style sequence-clipping method that directly regulates prefix-level policy drift without relying on a local quadratic approximation or expert-routing replay. Experiments on off-policy reinforcement learning with a sparse mixture-of-experts language model show that EPIC consistently outperforms standard and sequence-level clipping baselines across mathematical-reasoning benchmarks, while remaining competitive with routing-replay-based alternatives.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Off-Policy Reinforcement Learning, Reinforcement Learning with Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZFg3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EPIC, a prefix-level clipping method for off-policy reinforcement learning of autoregressive language models. The method constructs adaptive clipping bounds using cumulative prefix ratios and the exact reverse-KL coordinate, together with behavior-policy probability calibration and a geometric-mean optimization weight. Experiments on Qwen3-30B-A3B-Base show that EPIC outperforms baselines such as CTPO and GSPO on average across four mathematical reasoning benchmarks.

Critical strengths
  1. The paper derives a prefix-level performance-difference identity for autoregressive generation and establishes an exact relationship between the expected reverse-KL coordinate and prefix-distribution KL divergence, providing a theoretical motivation for the proposed method.

  2. EPIC achieves promising empirical results across four mathematical reasoning benchmarks, outperforming baselines such as CTPO and GSPO in the evaluated off-policy MoE training setting.

  3. EPIC can be integrated into existing PPO-style training pipelines without additional model forward passes or expert-routing replay. The paper also provides useful analyses of clipping behavior and gradient norms to examine training stability.

Critical weaknesses
  1. The paper contains extensive mathematical derivations but lacks clear and intuitive explanations of its core motivation and contributions. In particular, CTPO already employs cumulative prefix ratios and position-dependent clipping bounds. The paper does not sufficiently clarify which limitations of CTPO remain unresolved or what the key methodological advances of EPIC are.

  2. The paper does not adequately justify the choice of the exact reverse-KL coordinate or the proposed position-dependent coordinate budgets. The theoretical analysis does not sufficiently establish the advantages of these design choices over existing clipping methods.

  3. The experimental validation remains insufficient. The ablations in Appendix E.4 do not directly isolate the contribution of the exact reverse-KL coordinate or comprehensively validate the proposed coordinate budgets. Sensitivity analyses of the budget parameters δ−\delta^{-} and δ+\delta^{+} are also missing. Furthermore, the experiments are limited to a single sparse MoE model, without evaluation across other model scales or architectures.

Questions for authors
  1. CTPO already employs cumulative prefix ratios and position-dependent clipping bounds. Could the authors clarify which specific limitations of CTPO are addressed by EPIC and what its main methodological contributions are?

  2. Could the authors provide further justification for adopting the exact reverse-KL coordinate and the proposed position-dependent coordinate budgets? What specific advantages do these designs offer over existing clipping methods, particularly CTPO?

  3. Could the authors provide more targeted ablation studies to isolate the effects of the exact reverse-KL coordinate and the proposed coordinate budgets, along with sensitivity analyses of (\delta^-) and (\delta^+)? Additionally, could the authors evaluate EPIC on other model scales or dense architectures to assess its generalizability?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for recognizing EPIC’s theoretical motivation, empirical performance, and practical efficiency. We address the concerns below:

Q1. Main contributions and distinction from CTPO

We thank the reviewer for highlighting this distinction. EPIC asks which policy-induced distribution a trust region should constrain and how its divergence budget should determine clipping boundaries. We start from autoregressive MDPs and TRPO-style policy improvement, rather than modifying CTPO. A policy update changes both the probability of reaching a generated context and selecting the current token. Our performance-difference identity identifies the prefix ratio as the exact correction for this joint change, motivating a prefix-distribution trust region.

We have recognized that CTPO also derives prefix correction, but motivates log-space thresholds proportional to t\sqrt{t} through position-dependent cumulative log-ratio variance growth. EPIC instead connects performance improvement, prefix-distribution KL, and clipping boundaries through the exact KL coordinate, with an effective practical implementation. Its contribution thus extends beyond using prefix ratios.

In our experiments, we observed that CTPO still exhibits larger gradient spikes and less balanced position-wise clipping despite its prefix correction. CTPO and EPIC achieve four-benchmark means of 33.98±2.04%33.98\pm2.04\% and 39.34±1.14%39.34\pm1.14\%, respectively. Peak gradient norms range from 2.83–9.67 for CTPO versus 1.34–1.73 for EPIC. These observations highlight limitations of CTPO in the evaluated off-policy MoE setting and support EPIC as a complete method; the ablations below isolate the coordinate’s contribution.

Q2. Justification and coordinate ablations

We thank the reviewer for encouraging an intuitive explanation. Sections 3.2–3.3 provide the theoretical justification for these designs, which we summarize below alongside targeted empirical evidence.

Theoretical justification. Three steps connect the theory to clipping:

  1. Correction. In the autoregressive MDP, the state and current action identify the current prefix. Expressing each improvement term under behavior-policy sampling requires the prefix importance ratio, motivating measurement of policy change on these same distributions.

  2. Trust region. Bounding prefix KL by tδt\delta at every length tt bounds the maximum KL per prefix token, preventing large deviations at one length from being averaged away by smaller deviations elsewhere.

  3. Clipping boundaries. The exact identity is

    Kx,t(μ,π)=Eμ[ϕ(rtpre)],ϕ(r)=r−1−log⁡r.K_{x,t}(\mu,\pi)=\mathbb{E}_\mu[\phi(r_t^{\mathrm{pre}})],\qquad\phi(r)=r-1-\log r.

    Since ϕ\phi is minimized at one and increases away from it on either side, a samplewise budget determines lower and upper ratio boundaries.

Thus, an intuitive benefit of this coordinate is that it preserves the exact prefix-KL relationship, with is the distribution-level quantity constrained by the trust region; a quadratic, like GRPO or CTPO, retains only its local approximation around one.

Coordinate ablation. We replace only the constrained coordinate with (r−1)2/2(r-1)^2/2 or (log⁡r)2/2(\log r)^2/2, keeping budgets and other settings fixed:

CoordinateAIME24AIME25AMC23HMMT25Mean
(r−1)2/2(r-1)^2/234.5820.4274.067.534.14
(log⁡r)2/2(\log r)^2/230.4223.7572.816.6733.41
r−1−log⁡rr-1-\log r (EPIC)36.672576.5614.1738.1

Q3. Budget sensitivity and dense-model generalization

We thank the reviewer for encouraging more comprehensive evidence. Q2 discusses the exact coordinate and proposed budgets; below we examine other sensitivity and generalization.

Budget sensitivity. Budgets are empirical hyperparameters and the second-order approximation and prior empirical settings informed the 10−310^{-3} budget scale. Sensitivity tests within this scale, using the same training recipe, showed effective learning across all tested settings:

(δ−,δ+)(\delta^-,\delta^+)AIME24AIME25AMC23HMMT25Mean
(5×10−4,2×10−3)(5\times10^{-4},2\times10^{-3})36.672576.5614.1738.1
(10−3,10−3)(10^{-3},10^{-3})27.9225.8375.318.3334.35
(8×10−4,1.6×10−3)(8\times10^{-4},1.6\times10^{-3})33.9627.2975.1612.7137.28

Dense models. Our primary focus is off-policy training stability, particularly in MoE models where routing discrepancies introduce rollout–training mismatch. To explore applicability beyond this setting, we also evaluated GRPO and EPIC on dense Qwen3-0.6B and Qwen-1.7B models:

ModelMethodAIME24AIME25AMC23HMMT25Mean
0.6BGRPO0.752.523.750.256.81
EPIC2.081.25270.177.63
1.7BGRPO7.926.3937.660.4113.09
EPIC7.366.7439.581.1813.72

EPIC outperforms GRPO on the four-benchmark average, suggesting potential for transfer to dense architectures.


We hope our response and the additional evidence adequately address your concerns. We kindly invite you to reconsider the overall evaluation of our work.

Thank you for providing the additional information, which has addressed some of my concerns. However, my main concern remains unresolved, so I will maintain my original rating.

Dear reviewer,

We sincerely thank you for considering our response and acknowledging that the additional information addressed some of your concerns. We appreciate your time and would like to further emphasize EPIC’s theoretical contribution and practical value.

EPIC Provides a Principled Connection Between Autoregressive Policy Improvement and Clipping

EPIC addresses a fundamental design question: which policy-induced distribution should a trust region constrain, and how should that constraint determine clipping boundaries? Starting from the autoregressive MDP, EPIC identifies the prefix ratio as the exact correction for changes in both context visitation and current-token selection, then connects prefix-distribution KL to clipping through an exact coordinate. This provides a coherent theoretical basis for both the choice of prefixes and the construction of their clipping bounds.

Although CTPO also uses prefix correction, its position-dependent thresholds are motivated by cumulative log-ratio variance growth. EPIC isn't come from a modification of CTPO, it's contribution is the connection between performance improvement, prefix-distribution divergence, and clipping design.

EPIC’s Theoretical Design Choices Have Demonstrable Practical Value

EPIC combines this theoretical foundation with stronger empirical performance. In the evaluated off-policy MoE setting, it outperforms CTPO and the other clipping baselines, while exhibiting lower cross-seed variability than CTPO. Crucially, the exact coordinate contributes beyond the shared prefix structure: with budgets and other settings fixed, EPIC outperforms both quadratic-coordinate alternatives on all four benchmarks. These results directly support the practical value of the theoretically motivated coordinate.

EPIC Offers an Efficient and Robust Approach to Off-Policy Training

EPIC integrates into PPO-style pipelines without additional model forward passes, auxiliary networks, or routing-state storage. EPIC demonstrates strong effectiveness in MoE training, while additional experiments provide evidence of its potential to generalize beyond this setting. Together, these findings support EPIC as a theoretically motivated method with practical benefits in performance, robustness, and implementation efficiency.


If possible, we would appreciate further clarification of the specific concern that remains unresolved, so that we can respond more directly and guide further improvements. We will continue refining the theoretical presentation and strengthening the empirical evaluation. Thank you again for your careful review and constructive input.

Reviewer_5o4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at PPO-style clipping for LLM policies at the prefix level. Starting from a performance-difference identity under behavior-policy sampling, each token's advantage is weighted by the cumulative prefix ratio between the target and behavior policies. The authors then use the exact KL coordinate phi(r) = r - 1 - log r to turn a prefix KL budget into ratio bounds, calibrate the bounds by the geometric-mean behavior probability, and clip the length-normalized prefix ratio, with a straight-through gradient on the current token and a sign-aware dual clip. The extra cost is small. On Qwen3-30B-A3B-Base trained on DAPO-Math-17K, EPIC gets the best four-benchmark average against DAPO, GSPO and GRPO (three seeds), and roughly matches GRPO with R2/R3 routing replay without storing any routing state.

Critical strengths
  1. It is a cheap drop-in change that seems to really help. There is no extra model pass, auxiliary network or routing state (Listing 1). The three-seed mean curve is ahead of every baseline on the four-benchmark average from roughly step 200 (Fig. 2a), and at the final step it is still about 3 points above DAPO, so most of the Table 1 margin does not come from checkpoint selection. It is also not just "more conservative updates": GSPO clips a fraction of tokens on the same order as EPIC (Appendix E.2) but is well behind in Table 1, and EPIC's entropy does not collapse (Fig. 6b). Matching R2/R3 replay without storing routing decisions (Fig. 4) is a real practical win for MoE RL.

  2. The calibration finding is useful beyond this paper. If the budget is allocated by the raw prefix probability, which decays exponentially in t, clipping collapses into the first position bin (Fig. 8c) and lower-bound coverage drops sharply (Fig. 3c). The geometric-mean calibration keeps clipping active and roughly even across long responses (Fig. 3a-b). The mechanism is clear. The accuracy comparison in Appendix E.4 looks like a single final-step AIME24 run, though, so I treat it as supporting evidence only.

Critical weaknesses
  1. What makes this paper distinctive is its claim about why the method works, an exact-KL prefix trust region, and the experiments do not isolate that. The guarantee does not cover the method that is actually run. Proposition 1 follows from the exact identity and assumes exact advantages, and the paper itself says lambda only controls regularization (l.263-264). Yet contribution 1 presents it as a monotonic-improvement extension, while Eq. (22) changes almost everything: a group-relative response-level advantage, length normalization, weighting by g instead of r^pre, and a straight-through gradient. Nothing from Prop. 1 is shown to carry over. In the same way, the calibration does not quite "retain" the per-token budget of Eq. (16) (l.300-302). Hard enforcement of Eq. (18) gives E_mu[phi] <= t delta * E_mu[1/p_bar_t]. That is a reasonable approximation, but it should be called one.

The exact coordinate in the title is also never compared with the quadratic one. Appendix E.3 explains how phi differs from the log-space quadratic (tighter upper ratio bound, looser lower one, l.1175-1178), and the authors note this does not isolate its effect (l.1179-1180). In EPIC's regime the difference is not small: at the mean length of about 9,000 tokens, and assuming p_bar is about 0.7, the length-normalized band is tighter on the upside than on the downside. A matched quadratic run would show whether this shape, and not prefix clipping itself, drives the gain. The negative-advantage cap (Eq. 36), the delta values and the 1/T_i normalization are not ablated either; Appendix E.4 only varies the calibration and the forward weight.

  1. No uncertainty is reported, and the routing-replay comparison looks like one run per method. For each seed, Table 1 takes the best of roughly 60 evaluations on the same 130-prompt average that it then reports (Appendix D.4), with no per-seed numbers or spread. The ordering survives at the final step (Fig. 2a), but the reported margins include a selection bonus of about 1-4 points depending on the method.

Fig. 4 is weaker. Appendix D.4 says only the six-method comparison uses three seeds, EPIC's Fig. 4a average (38.10) differs from its Table 1 average (39.34) under the same protocol, and all 16 Fig. 4a entries sit on the single-run mean@8 grid. So "EPIC slightly outperforms both replay-based variants" (l.468-469) is not supported, though "competitive" is.

The scope is also narrow: one MoE model, one dataset, one update depth. "Update depth 8" means eight minibatches in one PPO epoch (Appendix D.3), not the asynchronous lag the introduction uses as motivation. The baselines use published default clipping (l.367-368), and I could not tell whether they share EPIC's loss aggregation and negative-advantage cap.

Questions for authors
  1. Could you run EPIC with the log-space quadratic s^2/2 (the E.3 comparator) instead of phi, with matched budgets and everything else fixed? Could you also ablate the negative-advantage cap, and say how delta- and delta+ were chosen and how sensitive the results are to them?

  2. How many seeds are behind Fig. 4 and Appendix E.4? For Table 1 and Fig. 4a, can you report per-seed selected scores and the final-step (or steps 280-300) average, with standard deviations?

  3. Do the baselines use the same 1/T_i aggregation and negative-advantage cap? Which budget do you claim for Eq. (18), given that hard enforcement gives t delta * E_mu[1/p_bar_t]? And which properties of Prop. 1, if any, do you claim for Eq. (22)?

Minor points that do not affect my rating:

  • l.133-135 says ratios in [1 - eps, 1 + eps] bound the KL by eps^2/2. They don't: mu = (0.5, 0.5) and pi = (0.4, 0.6) have both ratios in [0.8, 1.2], but KL(mu || pi) = -0.5 log 0.96 = 0.0204 > 0.02. The tight bound is -0.5 log(1 - eps^2), so this is a second-order approximation, which matters in a paper built on quadratic versus exact.
  • B.2 l.826 calls the clipped objective "pessimistic", but for A_hat < 0 and g above the upper bound, Eq. (36) gives A_hat u > A_hat g.
  • The 77.5% / 93.0% memory numbers should carry the E.5 caveat that part of R3's footprint comes from int32 versus uint8 routing storage.
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for recognizing EPIC as a “cheap drop-in change” and its practical value for MoE RL. Our goal is to stabilize off-policy training without additional model passes or routing state. Due to space constraints, we address the reviewer’s main questions below:

Q1. Coordinate and parameter ablations

We thank the reviewer for encouraging targeted ablations of our design choices. The results are listed below:

Coordinate. We replace only the constrained coordinate by two approximations, (r−1)2/2(r-1)^2/2 and (log⁡r)2/2(\log r)^2/2, keeping budgets and other settings fixed.

CoordinateAIME24AIME25AMC23HMMT25Mean
(r−1)2/2(r-1)^2/234.5820.4274.067.534.14
(log⁡r)2/2(\log r)^2/230.4223.7572.816.6733.41
r−1−log⁡rr-1-\log r (EPIC)36.672576.5614.1738.1

The exact coordinate outperforms both quadratic approximations on all four benchmarks, supporting the practical benefit of retaining the exact KL coordinate.

Negative-advantage cap. Below listed three-seed means ± sample SD result:

MethodAIME24AIME25AMC23HMMT25Mean
EPIC without cap34.86±6.5126.11±3.8775.00±3.6014.17±5.4537.53±4.73
EPIC with cap38.06±1.7325.28±1.6877.92±2.6216.11±2.1039.34±1.14

Though EPIC without cap already outperformances over baselines, the cap primarily stabilizes performance, reducing cross-seed SD on every benchmark and making performance more consistent across independent runs.

Budget selection and sensitivity. The budgets are empirical hyperparameters. Our second-order interpretation of PPO-style clipping and related methods' practical settings motivated budgets at the 10−310^{-3} scale; we then tuned their values empirically. We assess sensitivity within the same order of magnitude under the same training recipe. All tested settings enabled effective learning, suggesting that EPIC does not rely on a single finely tuned choice:

AIME24AIME25AMC23HMMT25Mean
(5×10−4,2×10−3)(5\times10^{-4},2\times10^{-3})36.672576.5614.1738.1
(10−3,10−3)(10^{-3},10^{-3})27.9225.8375.318.3334.35
(8×10−4,1.6×10−3)(8\times10^{-4},1.6\times10^{-3})33.9627.2975.1612.7137.28

Q2. Uncertainty and checkpoint selection

We thank the reviewer for highlighting uncertainty reporting. Table 1 selects each seed's checkpoint by the four-benchmark mean, then averages across seeds. We acknowledge possible selection bias and also average steps 280-300 within each seed below.

Selected per seedSelected mean±SD280–300 per seedWindow mean±SD
GRPO29.53/27.42/29.0128.65±1.1027.13/26.43/26.4526.67±0.40
DAPO33.98/34.38/37.9935.45±2.2133.44/32.17/37.4634.36±2.76
GSPO31.88/32.40/34.2432.84±1.2529.10/30.04/33.1530.76±2.12
CTPO31.77/34.35/35.8133.98±2.0431.29/31.75/34.0232.35±1.46
RIPO30.60/34.66/33.6732.98±2.1229.07/32.72/30.6130.80±1.83
EPIC38.10/40.34/39.5839.34±1.1437.32/38.87/38.9638.38±0.92

EPIC remains strongest in the fixed window, with relatively low cross-seed SD under both protocols, indicating that its advantage is consistent across seeds and training stages.

For routing replay, we added two seeds for GRPO + R2/R3 (denoted R2/R3 below), extending the comparison about benchmark performance and CPU RSS to three seeds.

MethodAIME24AIME25AMC23HMMT25Mean
Selected checkpointEPIC38.06±1.7325.28±1.6877.92±2.6216.11±2.1039.34±1.14
R238.75±7.8130.83±4.3579.38±5.9514.17±5.5140.78±5.85
R338.47±6.7727.78±3.7678.33±3.0413.47±3.0739.51±3.67
Steps 280–300EPIC35.78±1.2525.39±0.7076.56±0.8815.81±1.4638.38±0.92
R236.83±8.5228.83±4.1378.83±4.8913.53±6.4339.51±5.97
R335.28±6.0727.31±3.8878.15±1.9313.14±3.3738.47±3.77
MethodRSS (GiB)EPIC savings (%)
EPIC5.395±0.163—
R221.477±2.89374.49±4.32
R364.192±8.53091.48±1.28

EPIC remains competitive with routing replay, with similar means, lower cross-seed variability, and substantially lower process memory, supporting its use as a memory-efficient alternative in the evaluated MoE setting.

Q3. Baseline implementation

We thank the reviewer for drawing attention to these implementation differences.

We follow the usual method-specific practices for loss aggregation and negative-advantage handling: GRPO, DAPO, and RIPO use token means; GSPO, CTPO, and EPIC use seq-mean-token-mean. The cap ablation above isolates its contribution to EPIC.

Eq. (18) retains linear length scaling but approximates the original budget: hard enforcement implies Kx,t≤tδ Eμ[1/pˉt]K_{x,t}\le t\delta\,\mathbb E_\mu[1/\bar p_t]. Eq. (22) preserves the prefix clipping boundaries under geometric transformation, while modifying the other components.


We hope our response and the additional evidence adequately address your concerns. We kindly invite you to reconsider the overall evaluation of our work.

AuthorsOpenReview ↗

Clarification of theoretical claims

We thank the reviewer for the careful assessment and helpful suggestions.EPIC emphasizes a theoretical foundation for prefix-level clipping, while acknowledging a gap between the idealized formulation and practical training. This comment clarifies several concerns raised in the weaknesses and addresses the minor points. The corresponding revisions are highlighted in the revised manuscript.

Theoretical foundation.

We thank the reviewer for highlighting the gap between the theoretical formulation and the practical training objective. Our analysis starts from the autoregressive MDP and establishes that the cumulative prefix ratio is the exact change-of-measure factor for each stepwise performance-improvement term. This motivates constructing a trust region on the corresponding prefix distributions. The identity Kx,t(μ,π)=Eμ[ϕ(rtpre)]K_{x,t}(\mu,\pi)=\mathbb{E}_\mu[\phi(r_t^{\mathrm{pre}})], with ϕ(r)=r−1−log⁡r\phi(r)=r-1-\log r, then connects this distribution-level divergence to the coordinate used to construct clipping boundaries. The theoretical contribution therefore explains both why prefixes are the relevant unit of correction and how their KL divergence can inform clipping design. Proposition 1 establishes a conditional monotonic-improvement result for this exact-prefix formulation under its stated assumptions.

Practical objective and its relationship to theory.

Eq. (22) retains the prefix-based structure established by our theory and adopts a GRPO/PPO-style training objective, incorporating group-relative advantages and response-length normalization alongside geometric-mean weighting and a current-token-only straight-through gradient. Group-relative response-level advantages provide a critic-free surrogate for the exact state–action advantages.

Response-length normalization corresponds to first sampling a response from the behavior policy and then uniformly averaging the token-level surrogate over its valid positions, giving each response equal total aggregation weight. This provides a well-defined sampling interpretation of the response-normalized objective, while the original finite-horizon performance-difference identity averages trajectory sums.

Geometric averaging adjusts the optimization-weight scale, with exact preservation of prefix boundary-crossing decisions guaranteed by applying the same strictly increasing transformation to the ratio and its endpoints. The straight-through construction retains cumulative prefix information in the forward weight while restricting differentiation to the current token.

Thus, the exact prefix correction and KL identity provide the theoretical basis for our design, while these adaptations specify the optimization objective and gradient estimator used in training. Following the reviewer’s helpful comments, we have clarified their relationship to Proposition 1, the approximate nature of probability calibration and the symmetric ratio-bound interpretation, and the wording concerning the dual-capped objective.

Clarifications on minor points

We thank the reviewer for these helpful corrections. We clarify the approximation and memory measurements below and have updated the corresponding manuscript statements:

  1. Second-order approximation. The ϵ2/2\epsilon^2/2 expression is a second-order approximation, as indicated by the approximation sign in Appendix B.2. The main-text wording inadvertently omitted this qualification; we have corrected “would bound” to “would approximately bound.”

  2. Memory measurements. Appendix E.5 already analyzes the implementation differences: R2 stores routing indices as uint8, whereas the evaluated R3 implementation uses int32, contributing to its larger memory footprint. We have added this clarification to the main-text comparison. The reported 77.5% and 93.0% reductions refer to the observed median driver-process resident CPU memory relative to these implementations. EPIC does not store or replay expert-routing decisions and therefore avoids the associated routing-state storage requirement regardless of the index representation.

Reviewer_CYZiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes EPIC, a prefix-level importance-sampling objective for LLM reinforcement learning, combining geometric-mean probability ratios, clipping, and a straight-through gradient estimator. It is evaluated on four mathematical reasoning benchmarks against GRPO and some other related algorithms.

Critical strengths

None

Critical weaknesses
  1. The central trust-region justification is mathematically incorrect. eps^2/2 is only an approximation.

  2. It seems that there is overlap between the train set (DAPO-math-17k) and the test set AMC.

  3. Geometric mean transformation makes a big gap between the theory and the actual algorithm used in practice. It is only intuitive. Nothing is guaranteed.

Questions for authors

Please consider a sequence containing a few severely off-policy tokens whose extreme probability ratios are diluted or canceled by other tokens, leaving the geometric-mean ratio within the clipping range. How does EPIC prevent such locally unstable updates, and can the authors provide a theoretical guarantee or empirical analysis for this failure mode?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for closely examining EPIC’s theoretical foundation and practical design. Our additional evidence and explanations below are provided below:

W1. The quadratic approximation and theoretical justification

We thank the reviewer for raising this point and giving us the opportunity to clarify the connection between PPO clipping and KL-based trust-region optimization. PPO clipping is motivated by TRPO’s conservative-update principle and provides a first-order mechanism for discouraging excessive policy changes by removing the incentive to move probability ratios further beyond prescribed boundaries [1]. This connection also admits a local geometric interpretation: DKL(μ∥π)≈12Eμ[(r−1)2]D_{\mathrm{KL}}(\mu\|\pi)\approx \frac12\mathbb E_\mu[(r-1)^2], so symmetric ratio boundaries correspond to bounds on a quadratic KL coordinate [2]. Our discussion follows this connection. The correspondence involving ϵ2/2\epsilon^2/2 is a second-order approximation, as already indicated by the approximation signs in our appendix. We have clarified the main-text wording accordingly.

W2. Potential overlap involving AMC

We appreciate the concern regarding potential overlap between the datasets. Using DAPO-based training data and evaluating on AMC is a commonly adopted experimental setting in prior work [2–7]. To maintain comparability with these studies and assess EPIC under a widely used training setup, we use the released training dataset without any special dataset processing. All compared methods use the same training data and evaluation protocol, and EPIC’s performance gains are not confined to any single benchmark.

W3. Geometric-mean transformation and the theory–implementation relationship

We thank the reviewer for raising this important question about the relationship between our theoretical construction and practical optimization. We would like to clarify that the geometric-mean transformation preserves our prefix trust-region design and is applied only when constructing the optimization objective. Therefore, the prefix ratio is used for clipping underlying prefix-KL construction, only the optimization weight and gradient scaling is modified.

The empirical motivation for this optimization-scale design is examined in Appendix E.4. The Cumulative-ratio variant retains the same probability calibration but directly uses the cumulative prefix ratio in the objective. Across training steps 1–300, the temporal variance of the logged mean optimization ratio is 42.39 for this variant, compared with 2.49×10−82.49\times10^{-8} for EPIC. The cumulative variant also exhibits a higher average gradient norm over steps 201–300 (0.240 versus 0.165) and lower four-benchmark performance (33.13% versus 38.10%). These observations provide empirical support for geometric normalization as a practical means of reducing optimization-weight fluctuations and improving training performance while preserving the original prefix-boundary criterion.

Q1. Extreme token ratios diluted or canceled within a prefix

We thank the reviewer for this helpful example. Geometric averaging itself neither hides a prefix-boundary violation through dilution nor introduces cancellation: as discussed above, geometric averaging is applied only when constructing the optimization objective and does not change the clipping decisions. For illustration, consider token ratios of 4 and 0.25, yielding cumulative prefix ratios of 4 and 1. If the first-position upper bound is 2, EPIC detects the first prefix as exceeding its bound and applies the corresponding clipping rule. The subsequent cancellation does not alter that earlier decision. At the second position, the prefix ratio of one correctly reflects equal joint probability for this particular two-token prefix under the two policies. This cancellation occurs in the cumulative product itself, independently of geometric averaging, and is consistent with our performance-difference identity: the exact correction accounts jointly for reaching the current context and selecting the current token, so offsetting probability changes should be reflected in the weight.


We hope our response and the additional evidence adequately address your concerns. We kindly invite you to reconsider the overall evaluation of our work.

[1] Schulman et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.

[2] Cai et al. Beyond Euclidean Clipping. arXiv:2607.10169, 2026.

[3] Wang et al. Beyond the 80/20 Rule. arXiv:2506.01939, 2025.

[4] Becker et al. TROLL: Trust Regions Improve Reinforcement Learning for Large Language Models. arXiv:2510.03817, 2026.

[5] Lei et al. Stabilizing Off-Policy LLM Optimization with Prefix Importance Ratio. arXiv:2601.22718, 2026.

[6] Zheng et al. Act Only When It Pays. NeurIPS, 2025.

[7] Liang et al. SwS: Self-Aware Weakness-Driven Problem Synthesis in Reinforcement Learning for LLM Reasoning. NeurIPS, 2025.

  1. Your response (W1) reminds me another related work [1]. However, this related work is not selected as a baseline. This paper points out the relationship between TV divergence and ratio. (You mention KL divergence and ratio here). I think you should compare the proposed method with this related work.

  2. It's true that AMC is widely adopted benchmark. However, you claim non-overlap in the paper.

  3. I have no misunderstanding of the objective. And I think this transformation departs from the theory.

Considering my above concerns, I keep my current rating (score: 1).

Reference:

[1] Qi et al. Rethinking the Trust Region in LLM Reinforcement Learning. arXiv 2602.04879.

We appreciate the reviewer for the follow-up and for pointing us to DPPO. We address the remaining concerns.

1. Relation to DPPO and additional comparison.

DPPO is closely related to our work: both connect policy divergence to update control. DPPO considers the conditional next-token distribution at a given context and uses Binary or Top-K approximations to TV or KL divergence to determine whether token-level updates should be masked. EPIC starts from the autoregressive MDP: a policy update changes both the probability of reaching a context and selecting the current token. Our performance-difference identity identifies the cumulative prefix ratio as the exact correction for this joint change, motivating clipping boundaries derived from prefix-distribution KL.

Thus, the distinction is not simply TV versus KL. DPPO improves how conditional policy divergence is measured, while EPIC uses the prefix joint distribution to define the correction and clipping criterion.

Following the reviewer's suggestion, we evaluated DPPO-Binary-KL under our main experimental setup, providing a comparison with another KL-based method:

MethodAIME24AIME25AMC23HMMT25Average
DPPO-Binary-KL26.6720.8374.386.2532.03
EPIC36.6725.0076.5614.1738.10

DPPO outperforms most of the evaluated clipping baselines in this setting, supporting its effectiveness. EPIC achieves higher scores on all four benchmarks in this comparison, supporting the practical value of its prefix-based design.

2. Wording concerning dataset overlap.

Thank you for highlighting this concern. We confirm that we used the released training dataset without special modifications. We will carefully review the manuscript and remove wording that could imply verified non-overlap between the training and evaluation data.

3. Relationship between the transformation and theory.

We appreciate the reviewer's further comments on the relationship between the theoretical formulation and practical optimization. Our intention is to use the prefix-level analysis to guide the clipping design, while geometric normalization adjusts the optimization scale for long responses. Applying the same transformation to the ratio and its bounds preserves the original prefix boundary-crossing decisions. We understand that the concern extends beyond this equivalence to the broader connection between theory and implementation, and will further clarify the influence of these design choices in the presentation.

We appreciate the reviewer's careful consideration and the suggestions that helped us improve the comparison and clarify the scope of our claims. We will continue refining the presentation and strengthening the experimental evaluation to further assess the method's strengths and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.