- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
PPO and GRPO rely on local policy gradient updates, which may require many steps to approach the optimal policy, leading to inefficient training. This paper proposes a different approach by leveraging the closed-form optimal policy in KL-regularized RL. Since the optimal policy cannot be computed directly, the authors use reparameterization to construct a tractable training target from rollout rewards. Instead of following local policy gradients, the method trains the policy to match this target using cross-entropy loss, aiming to achieve faster and more efficient convergence.
Critical strengths
-
The motivation is clear and well justified, as local policy gradient updates in PPO/GRPO may not efficiently move the policy toward the optimum. Policy-space regression is a natural and intuitive way to address this limitation.
-
The authors use the closed-form optimal policy in KL-regularized RL and reparameterize it into a tractable training target without enumerating the entire action space. This offers an interesting alternative to standard policy gradient methods.
-
MCPO achieves better final performance, faster convergence, and more stable training than GRPO in the reported experiments. The faster reward improvement is consistent with the paper’s motivation, though it does not directly establish faster convergence to the optimal policy.
Critical weaknesses
I like the paper’s motivation, but it is not yet clear whether MCPO’s empirical gains come from the core mechanism of policy-space regression. Lemma 1, Theorem 2, and Figure 1 support more direct progress toward the optimal policy in an idealized tabular setting, while the LLM experiments only show faster reward improvement. More evidence is needed to connect the theoretical advantage to the gains observed in LLM training.
Questions for authors
Q1. The theory and Figure 1 suggest that MCPO approaches the optimal policy faster, but the LLM experiments only show faster reward improvement. Could you compare for MCPO and PPO/GRPO in a neural network setting where the optimal policy is known?
Q2. MCPO changes several components at once, including the objective, reward weighting, baseline, and clipping, while the current ablations mainly examine the clipping threshold and . Could you ablate these components using the same rollouts and training budget to clarify what drives the gains?
Q3. MCPO treats a full response as one action, while actions in multi-turn tasks affect future states and rewards. How would you construct the optimal policy target in this setting? A discussion or a small multi-turn experiment would help clarify its applicability.
Response to Reviewer DY2c
Thank you for your suggestion regarding the source of the gains! Our response is as follows.
Weakness & Q1
Thank you for pointing out the need for a more direct evaluation metric. Since estimating $ D_ {KL}(\pi^ *|| \pi) $ requires sampling from $ \pi^ * $ , which is generally intractable to compute accurately, we initially did not include this metric and instead reported only the rewards. We have now supplemented our experiments with $ D_{KL}(\pi^ *| | \pi) $ comparisons in the neural network setting, as well as an indirect comparison of $ D_{KL}(\pi^ * || \pi) $ in our LLM experiments.
1. Comparison of $ D_{KL}(\pi^*||\pi) $ in the Neural Network
Thank you for suggesting experiments in a "neural network setting where the optimal policy is known."
We constructed state features using standard Gaussian distributions, with 4 states and 2 dimensions per state. We used a two-layer MLP with ReLU activations as the policy network, where the final layer outputs 2 numbers that are passed through a softmax function to obtain the policy. Rewards were sampled from a Bernoulli distribution, the reference policy was uniform, and $ \beta $ was set to 0.1.
Under this setting, we trained GRPO and MCPO from the same initialization, using the same learning rate of $ 1\mathrm{e}{-2} $ for 1,000 iterations.
The results are as follows:
We repeated the experiments with multiple random seeds and observed consistent results. We believe these results demonstrate the advantages of MCPO, at least in the neural network setting. We will also incorporate these experimental results into the next version of the paper.
2. indirect comparison in LLM
Although $ D_{KL}(\pi^*||\pi) $ is difficult to measure directly, we can decompose it as follows:
$ D_ {KL}(\pi^ *||\pi) = \sum_ a \pi^ *(a) \log \frac{\pi^ *}{\pi} = \frac{1}{\mathbb{E}_ {\pi _ {ref}}[e^ {\frac{r}{\beta}}]} \left( \mathbb{E} _ {\pi _ {ref}} [e^ {\frac{r}{\beta}} \log \pi _ {ref} + \frac{r}{\beta} e^ {\frac{r}{\beta}} ] \right) - \log \mathbb{E} _ {\pi _ {ref}}[e^ {\frac{r}{\beta}}] - \frac{1}{\mathbb{E}_ {\pi _ {ref}}} \mathbb{E} _ {\pi _ {ref}}[e^ {\frac{r}{\beta}} \log \pi] $
where the only term that differs across different policies π is Eπref[eβrlogπ], which can be estimated by sampling from πref.
Therefore, we calculated Eπref[eβrlogπ] for the Llama-8B model trained with different methods on ArenaHard. The results for GRPO, BNPO, DPO, and MCPO are -1791749, -1518047, -1062195, and -2042607, respectively. MCPO achieves the smallest value, indicating that its Eπref[eβrlogπ] is also the smallest.
Q2: Ablation Study
Since MCPO introduces a new objective function, we conducted ablation studies on the ArenaHard v2 benchmark using Llama-8B to investigate the contributions of its other components. Specifically, we successively removed the baseline and clipping mechanisms, and replaced reward weighting with advantage weighting.
We observe that removing the clipping mechanism leads to the most significant performance degradation, possibly due to increased variance. All these components contribute to MCPO, with the full MCPO achieving the best overall performance.
Q3: actions in multi-turn tasks
Thank you for considering the multi-turn setting. In fact, we previously attempted to extend MCPO to the token level, where the weight of each token is computed independently, which is also similar to the multi-turn setting.
However, we were unable to obtain theoretically satisfactory results. This is because estimating the target requires independently estimating the exponential of the expected future return for each token, which is generally difficult to do without bias. An exception is when we have perfectly accurate process rewards, in which case each token only needs to maximize its own process reward. We can then treat each token as an action and re-derive Eq. 8 accordingly.
However, perfectly accurate process rewards are unavailable in most cases. We also experimented with directly using the return of the entire sequence as the reward for each token and applying Eq. 8 directly. The resulting performance was very similar to that of sequence-level MCPO, with a difference of only approximately one point. Nevertheless, since sequence-level MCPO has a more theoretically sound formulation, we ultimately chose to present the sequence-level version.
We will incorporate the above discussion into the revised version of the paper.