- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper targets at improving reinforcement learning with verifiable rewards, with emphasizes on the exploration stage during the training process.
- To improve the policy diversity, they propose correctness-conditioned entropy regularization, which regularizes the policy's projection onto the optimal policy space to preserve optimality.
- They also empirically investigate the effectiveness of the proposed method across multiple mathematical reasoning benchmarks, and demonstrate superiority compared with baseline methods such as DAPO.
Critical strengths
- Exploration in reinforcement learning is a critical part, this work introduce CCE to regularize this stage to enhance policy diversity is straightforward and meaningful, and also demonstrate superiority compared with previous methods.
- This work has provided solid theory prove for the optimality analysis of the regularization term, provides deeper understanding for readers.
Critical weaknesses
- One of the key weakness of this work is the novelty of the idea, entropy regularization has been widely investigated in reinforcement learning, for example an entropy regularization term is frequently used in model training.
- As shown in Table 1, the difference between the proposed method and previous DivPO is whether applied correctness condition.
Questions for authors
What about the effectiveness of the proposed method on models larger than 7B or MoE models, which could provide more additional insights.
6 comments
Response to Reviewer SBuj (2/3)
W1: novelty
Thank you for your concern regarding the novelty of our idea. However, unlike existing methods that aim simply to "increase diversity," our motivation is to achieve this without compromising optimality.
Specifically, existing methods all aim to optimize a conventional regularization term: where the diversity metric $ \mathcal{H} $ typically prevents the optimality metric $ \mathcal{L} $ from reaching its optimum. In contrast, our objective is to maximize diversity while preserving optimality: CCE in Table 1 is a method we identify that can achieve this objective, whereas conventional regularization methods cannot. Therefore, CCE arises from a different objective, rather than being a simple variant of existing methods.
Driven by this new objective, our method also takes a different theoretical framework from conventional methods. We did not develop our method by simply modifying certain components of DivPO or existing entropy regularization approaches. Instead, we first identified the geometric nature of regularization terms (Section 4), and then, from this geometric perspective, investigated what types of regularization are compatible with preserving optimality, which led us to find CCE. However, the conditional probabilities involved in CCE cannot be computed directly. We therefore further derived an unbiased estimator of the CCE gradient, incorporated it into an offline estimation framework for RLVR, and established error control guarantees. This process makes CCE a new method derived from a new theoretical framework for a new objective, rather than a simple variant of existing approaches.
Furthermore, we provide a variational regression perspective on CCE (Section 5), which offers an additional interpretability analysis from an new viewpoint.
Regarding conventional methods such as entropy regularization mentioned by the reviewer, we have cited them, discussed them in Section 4, and compared against them experimentally. These results demonstrate the superiority of CCE.
W2: difference with DivPO
Thank you for raising the question regarding the distinction between CCE and existing methods. However, CCE and DivPO differ fundamentally in their objectives, theoretical frameworks, underlying principles, and implementation strategies. As outlined in the brief overview of our work, we first formulate a different objective and then identify CCE through geometric analysis as a method that satisfies this objective, rather than simply modifying an existing approach.
1. Why CCE May Appear Similar to DivPO
The purpose of Table 1 is to provide an abstract representation of different methods rather than an exhaustive comparison. At this level of abstraction, the differences between various methods may appear relatively subtle. For example:
- The difference in Table 1 between DivPO and entropy regularization is whether the expectation is conditioned;
- The differences among MaxRL, GRPO, and Dr-GRPO mainly lie in how the advantages are normalized.
However, this does not mean that these methods are not fundamentally different from one another. They are built upon distinct theoretical frameworks, which provide the most fundamental basis for distinguishing.
2. Theoretical Differences of CCE
These are the most significant differences between our work and DivPO, and constitute the main contributions of our paper:
-
Objective: Existing methods focus solely on increasing diversity, whereas CCE aims to increase diversity while simultaneously preserving optimality.
-
Theoretical Framework: CCE first introduces a novel geometric perspective for characterizing different methods (Section 4), based on which we derive CCE as a regularization method that preserves optimality. This work does not follow any existing line of research.
-
Interpretability: We prove that CCE regularization is equivalent to variationally minimizing the divergence from the optimal policy (Section 5), providing an interpretation that is absent from existing work.
-
Estimation Reliability: Due to the intractability of directly computing the conditional probabilities, we derive an estimator for the CCE gradient and further establish an error-control method for offline estimation (Section 6). These are also novel contributions not found in existing methods.
Response to Reviewer SBuj (3/3)
3. Implementation Differences from DivPO
The different theoretical frameworks naturally lead to different implementation strategies:
- We propose directly estimating the CCE gradient, rather than the CCE term itself, through conditional sampling, which distinguishes our approach from existing methods (Eq. 14).
- To account for the behavior-target policy mismatch in RLVR, we develop an offline estimation method for the CCE gradient. (Eq.16) In contrast, existing methods generally overlook this aspect.
- To ensure the validity of offline estimation, we derive an upper bound on its estimation error (Theorem 3). Based on this analysis, we incorporate the CCE gradient into PPO-clip (Eq.15), which serves to control the offline estimation error. In contrast, other methods simply add a regularization term to the loss function.
This leads to substantial differences in their loss functions:
- DivPO:
- CCE (Eq.16):
Response to Reviewer SBuj (1/3)
Thank you for the feedback! We have responded to all the weaknesses and questions raised in the review:
Weakness 1 (novelty): Our goal is not simply to propose another method for "increasing diversity," as conventional approaches do, but rather to achieve this without compromising optimality. This perspective has not been explored by existing methods and represents a novel objective. We have also discussed and empirically compared against existing approaches, including entropy regularization.
Weakness 2 (Distinction from DivPO): The fundamental difference between our method and DivPO lies in their objectives: CCE aims to increase diversity while preserving optimality, whereas methods such as DivPO inevitably compromise optimality to some extent. To achieve this objective, which existing methods cannot accomplish, we develop a different theoretical framework directly from the basic RL formulation (Section 4), without relying on existing regularization methods. Furthermore, to make CCE practical, we address several new challenges, which also lead to differences between CCE and DivPO in terms of gradient estimation methods, the placement in the loss function, and other key aspects. Therefore, CCE is an new approach rather than a variant of DivPO.
Question (Larger Models): We have provided additional experimental results.
We have also incorporated the corresponding additions into the latest version of the PDF:
A Brief Overview of Our Work
Given the primarily theoretical nature of our work (i.e., it is task-agnostic, introduces a fundamental algorithmic improvement, and is theoretically motivated, as in [1–4]), its distinction from other methods should be understood from the perspective of its underlying theoretical approach. Since the reviewer's concerns mainly focus on our novelty and distinction from DivPO, we believe a brief overview of our work would be helpful. Below, we revisit the research approach underlying our work, which illustrates why it represents a new contribution rather than a simple variant of existing entropy regularization methods:
Our objective and motivation are to improve diversity in RLVR without sacrificing optimality, rather than simply increasing diversity itself, which entropy regularization has already achieved (Abstract and paragraphs 1–2 of the Introduction).
We theoretically identify why entropy regularization compromises optimality (Section 4.3). Based on this insight, we propose CCE regularization and prove that it preserves optimality (Section 4.4).
We establish that CCE regularization is equivalent to variationally encouraging the LLM to approach the optimal policy (Section 5), providing a theoretical interpretation of our method.
Since CCE involves the distribution π(⋅∣A∗), which cannot be directly computed, we derive and design an estimator for its gradient (Eq. 14).
Given that RLVR relies on offline Monte Carlo estimation, we further develop an offline estimator for the CCE gradient (Eq. 16) to reuse RLVR rollouts, incorporate it into PPO-clip (Eq. 15), and provide theoretical justification for this approach (Theorem 3).
Finally, we implement CCE in two algorithms, DAPO and MaxRL, to demonstrate its effectiveness (Section 7).
To summarize, we would like to emphasize two key distinctions:
Our method is not simply "another diversity-enhancing method." Rather, it addresses the fundamental problem of existing diversity-enhancing methods: their tendency to compromise optimality. This represents a new objective.
Our method is not merely a variant of existing entropy regularization approaches. Rather, it is a new method developed from a different objective through new theoretical analysis.