acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Divergence Regularization in LLM RL

Abstract

Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream methods such as PPO and GRPO approximate this control with a ratio-clipping mechanism, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO addresses this mismatch by replacing ratio-based clipping with a divergence-based mask, yielding a trust region defined by the sampled token's absolute probability shift. However, DPPO still relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded entirely. To address this, we propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift. DRPO keeps DPPO's trust-region geometry while inducing bounded, continuous gradient weights that attenuate diverging updates by their absolute probability shift and can reverse them beyond the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves the stability and efficiency of LLM RL training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Large Language Models, Trust Region, Off Policy, GRPO, Divergence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5KB1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper mainly studies the off-policy errors in reinforcement learning caused by the mismatch between the behavior policy and the training policy. Similar to methods such as TRPO, this paper follows the approach of constraining the magnitude of policy updates. The main contribution of the paper is combining the TV divergence constraint of DPPO with the smooth objective function of SPO. The experimental results show that the proposed method is effective.

Critical strengths
  1. The off-policy error problem studied in this paper is a classical problem in reinforcement learning. It is valuable and broadly applicable.
  2. The paper cleverly uses the quadratic objective function of SPO to implement a smooth constraint, while adopting the TV constraint of DPPO to address the scale mismatch problem of KL.
  3. The experimental results are good, and the method is relatively clean and easy to reproduce.
Critical weaknesses
  1. The contributions of the paper are limited. The main method simply combines the smooth constraint of SPO with the TV constraint of DPPO. Essentially, it is a combination of two existing methods, making the contribution relatively limited. The authors could have strengthened the contribution by providing more theoretical analysis. However, the current theoretical section mainly analyzes the properties of DRPO and lacks a more substantial contribution that directly and systematically explains, in an end-to-end manner, why DRPO leads to improved RL performance. The current version looks more like an incremental improvement based on combining existing methods.
  2. The experiments could be further expanded. Although AIME24 and AIME25 are both mainstream benchmarks, they are still too similar, which may limit the evaluation perspectives. Adding other benchmarks, such as Olympiad and HMMT (just as examples), would strengthen the credibility of the experimental results.
  3. Some derivations are not sufficiently rigorous. For example, the probabilities given by Equation 11 may not sum to 1, meaning that they do not form a valid probability distribution. The authors should carefully check the mathematical rigor throughout the paper and make corrections. Taking this case as an example, the authors could obtain a more rigorous solution using Lagrange multipliers. They would only need to add a constant to Equation 11 (when delta is sufficiently small) and introduce clipping (when delta is sufficiently large).
Questions for authors

What do you consider to be the main contributions of this submission?

Reviewer_1DrjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Divergence Regularized Policy Optimization (DRPO), which replaces the hard divergence-based mask in DPPO with a smooth, advantage-weighted quadratic regularizer. The method aims to improve training stability by controlling absolute probability shifts rather than importance ratios. Experiments across different model scales, architectures, and precision settings demonstrate improvements over existing policy optimization methods.

While the approach is practically motivated and shows promising empirical results, its methodological novelty is relatively limited, as it primarily combines existing ideas from SPO and DPPO. Furthermore, the experiments mainly focus on single-turn mathematical reasoning, with additional validation on only one code-generation benchmark. The effectiveness of DRPO in more complex RL settings remains insufficiently explored.

Critical strengths
  1. Clear motivation and intuitive design. The paper is well written and identifies meaningful limitations of ratio-based clipping and hard divergence masks. The proposed smooth regularization is simple and can be integrated into existing LLM RL frameworks.

  2. Promising empirical improvements. DRPO demonstrates improved training stability and performance across different model scales, architectures, and precision settings. The comparisons with existing optimization methods and the ablation studies provide useful supporting evidence.

Critical weaknesses
  1. Limited methodological novelty. The proposed DRPO essentially combines the smooth quadratic regularization principle of SPO with the Binary-TV geometry introduced by DPPO. Given the extensive existing literature on PPO-style optimization and trust-region regularization, the methodological contribution appears relatively incremental. The paper does not sufficiently establish what fundamentally new insights or optimization capabilities DRPO provides beyond these existing approaches.

  2. Limited task diversity and insufficient validation of the proposed mechanism. The experiments are predominantly conducted on single-turn mathematical reasoning tasks, with additional code-generation evaluation limited to LiveCodeBench v6. It remains unclear whether the proposed method offers similar benefits in more complex settings, such as multi-turn agentic tasks or RL with learned reward models. Furthermore, the main experiments use δ = 12.5, under which the proposed gradient reversal mechanism cannot be activated. This limits the empirical evidence supporting the claimed trust-region correction behavior.

Questions for authors
  1. Given the extensive existing literature on PPO-style optimization and trust-region regularization, including closely related methods such as SPO and DPPO, could the authors provide a more comprehensive discussion and comparison with prior work? In particular, what is the distinct methodological contribution of DRPO beyond combining existing smooth regularization and divergence-based trust-region designs?

  2. The current experiments primarily focus on mathematical reasoning and a single code-generation benchmark. Could the authors provide evidence that DRPO remains effective in more challenging settings, such as multi-turn agentic RL or optimization with learned reward models?

  3. Since the main experiments use δ = 12.5, the gradient reversal mechanism discussed in the theoretical analysis is never activated. Could the authors clarify which aspects of the proposed trust-region mechanism actually account for the observed performance improvements?

  4. Could the authors provide more detailed comparisons with carefully tuned DPPO baselines to establish whether DRPO offers consistent practical advantages beyond existing divergence-based optimization methods?

Reviewer_CEDFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript studies regularization for LLM reinforcement learning under training–inference mismatch and policy staleness. It proposes Divergence Regularized Policy Optimization (DRPO), which multiplies SPO's advantage-weighted quadratic ratio penalty by the sampled token's behavior probability. Under the behavior-policy expectation, this produces an advantage-weighted squared Euclidean penalty on probability displacement. The resulting policy-gradient multiplier varies continuously with the sampled token's absolute probability shift and remains bounded, unlike SPO's ratio-dependent multiplier. The method is motivated as a smooth alternative to DPPO's Binary-TV mask. Experiments cover six mathematical-reasoning configurations with dense and MoE models and BF16/FP8 precision, plus a code-generation experiment. DRPO has the highest reported final accuracy in the main comparison tables. Ablations examine advantage weighting, alternative penalties, reference-KL regularization, and masked variants. All evaluated DRPO coefficients are in the nonnegative-weight regime, gradient reversal beyond a reachable boundary is not tested.

Critical strengths

(1)The modification is simple and interpretable. With fixed behavior probabilities and advantages, Eqs. (8)-(10) correctly yield an advantage-weighted squared Euclidean penalty and a continuous, bounded additional gradient multiplier, unlike SPO's potentially unbounded ratio-dependent multiplier. (2)Experiments cover dense/MoE models, multiple scales, BF16/FP8 precision, and both mathematical reasoning and code generation. Tables 3 and 5 report gains of 0.4–7.2 percentage points over the strongest listed math baseline and 1.7 points over DPPO on LiveCodeBench v6. Ablations and partial baseline tuning provide some useful supporting evidence.

Critical weaknesses

(1)The methodological novelty is incremental relative to SPO and DPPO. Substituting the token-adaptive ratio tolerance implied by DPPO into SPO's quadratic penalty directly yields DRPO and its continuous, bounded additional gradient multiplier. This interpretable combination could be valuable, but its significance rests on robust empirical gains or new mechanistic insight. (2)The mask ablation is difficult to interpret from the current formulation. Appendix D.9 uses δ for both the mask threshold and the regularization scale, whereas Section 4 specifies a DPPO threshold of 0.2 and a DRPO scale of 12.5. If both parameters are set to 12.5, the mask is identically zero. The actual configuration used in Figure 11 therefore needs clarification. Exact configurations and pseudocode or an anonymized implementation would help resolve this ambiguity. (3)I am not yet convinced that the reported advantages would persist under more extensive tuning and repeated training runs. Independent-run counts and variability are not clearly reported, and D.7 shows that tuned DPPO achieves similar FP8-E2E performance. Comparisons under comparable tuning budgets, together with independent runs on a few representative configurations, would help establish whether DRPO's advantages are consistent. If training efficiency refers to runtime, wall-clock time or GPU-hours should also be reported.

Questions for authors

(1)Given DRPO's close connections to SPO and DPPO, could the authors clarify the main added value of this integration? It would be helpful to identify which empirical findings most directly support that contribution. (2)What mask threshold and regularization scale were actually used for Mask-DRPO, Mask-SPO, and Mask-KL in Figure 11? Were 0.2 and 12.5 used as separate parameters? Provide the exact configurations with pseudocode or an anonymized code. (3)D.7 reports similar FP8-E2E performance for tuned DPPO. Does DRPO retain an advantage on representative configurations when SPO, DPPO, and DRPO receive comparable tuning budgets? Please report the search ranges and selection criterion. If efficiency refers to runtime, please also compare wall-clock time or GPU-hours under the same hardware setup. (4)How many independent training runs underlie the main results and the code-generation results? For a few representative comparisons, please report seeds, per-run scores, and uncertainty at common, prespecified training steps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.