acceptodds
Under review as a conference paper at ICLR 2027

A Distribution-Matching Perspective on LLM Training Algorithms

Abstract

In this work, we propose a unified framework for training algorithms across different stages of modern large language model development, based on the perspective of distribution matching. We show that a broad class of language model training algorithms can be interpreted as minimizing a certain divergence between the current model and a target distribution, and that characterizing the properties of this target distribution provides substantial insight into the behavior of the corresponding training algorithms. By analyzing specific forms of the target distribution, we establish that On-Policy Distillation (OPD) enjoys an advantage over classical Knowledge Distillation (KD) when the current model distribution is close to the target distribution, while exhibiting a disadvantage when the two distributions are far apart. Furthermore, when the target distribution itself depends on the model being optimized, we show that the accumulation of estimation variance can induce a deterministic degradation of the target distribution. These results provide a theoretical explanation, at least in part, for the long-term behavior of existing Reinforcement Learning (RL) algorithms. With this intuition we further designed Conditional KL (CKL) regularization for mitigating the degradation of the target distribution. CKL can effectively preserve mode diversity while relaxing the constraint imposed by classical KL regularization on reward improvements. We believe this theoretical perspective can bring some new understanding of language model training algorithms and provide useful guidance for future research.

Primary area:
learning theory
Keywords:
Large Language Model, Distribution Matching, Machine Learning Theory, Reinforcement Learning, Distillation Algorithms

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kMeXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper seeks to provide a general perspective on LLM (pre to post) training as distribution matching. It makes a few statements:

On-policy distillation gradient is more statistically stable than SFT when the student is closer to the teacher. Argues that while both match to the teacher distribution, OPD has lower gradient variance when student is closer to the teacher (and zero when they coincide), while SFT generally would not. When the student is far from the teacher, there are some cases where OPD can have higher variance, particularly if the OPD ratio blows up due to low teacher probabilities.

Variance accumulation leads to entropy decrease. They analyze the setting of SFT-ing a model on its own generation. Using a martingale argument and assuming entropy is concave in parameters, they show expected entropy can only stay the same or decrease. This is used as a motivating example to motivate entropy collapse in long term RL training.

Conditional KL regularization Motivated by the entropy of the correctness-conditioned distribution possibly collapsing when it should not if the target distribution for e.g. MaxRL is well-optimized, the authors call for a new KL regularizer which effectively regularizes only the positive samples. They show this can prevent collapse to an extent and also has higher rewards than regularizing both the correct and incorrect parts of the distribution.

Critical strengths

The paper seeks to provide theoretical understanding in a number of relevant post-training algorithms. As summarized in the above section, each section provides useful and interesting hypotheses about observations such as stability and entropy collapse. This also leads to a number of possibly actionable algorithmic choices, such as when to use SFT vs. OPD, and a new CKL regularizer.

I enjoyed reading about target distribution degradation as the reason for collapse. To me, one key thread in the paper is that success-conditioned entropy can decrease from training when they should not if the target is perfectly optimized. This feels distinct from simply “entropy decreasing” which can be due to removing probability mass from negative sample as well. This also motivates ways to try to mitigate this, such as via the conditional KL regularizer.

Critical weaknesses

The paper is overly broad; the novelty & value of the unified framework is unclear

The paper attempts to provide a general distribution-matching framework that explains several practically meaningful observations in LLM training. However, I find the scope overly ambitious, and led to a paper that tries to explore a few (interesting) directions but ultimately without investigating any of them sufficiently deeply scientifically. It also generally lacks clarity in expositing the various research directions.

There are essentially two distinct research directions:

  1. Understanding the trade-off between OPD and SFT/KD, and when one should be preferred over the other.
  2. Understanding how the correctness-conditioned distribution can degrade during RL training with binary rewards, and whether the proposed Conditional KL (CKL) regularization can mitigate this degradation.

The distribution-matching framework is motivated to help with both. Yet, I do not find it very informative. The connection between cross-entropy and forward KL is standard (e.g., [1]), as is the reverse-KL formulation of OPD. Iterative matching to reward-conditioned distributions has also been studied previously, e.g. EM interpretations of RL [2, 3, 4, many more]. While the authors do acknowledge some related literature (e.g. Korbak et al 2022), it is missing others above, and I do not think the paper establishes what is fundamentally new compared with these earlier perspectives. The unified framework currently occupies a substantial part of the paper, and in my opinion does not actually help me understand the rest of the paper.

The proposed correctness-conditioned entropy collapse mechanism is insufficiently investigated.

Focusing just on the second research direction, I understand it as:

  1. Although an idealized RL update preserves the correctness-conditioned distribution, practical training can cause this distribution to degrade, reducing its entropy.
  2. This degradation may be explained by the variance-accumulation mechanism studied through the martingale result (Theorem 4.1).
  3. CKL regularization can mitigate this degradation while allowing greater reward improvement than standard KL regularization.

I find this interesting. However, it is missing a number of connections to fully establish this narrative:

First, the generality and consequences of correctness-conditioned entropy collapse are unclear. The authors demonstrate this phenomenon for GRPO in Figure 3. However, it would be useful to measure how the correctness-conditioned entropy evolves across different objectives and gradient estimators, such as GRPO, RLOO, and MaxRL, to establish how general and significant the phenomenon is. It would also be useful to connect the decrease directly to measurable consequences, e.g. does more degradation of the conditional target lead to more reduced diversity of correct solutions, worse pass@k, etc, and in a way that is not predicted by the unconditional entropy? Basically what I am asking is to more convincingly and thoroughly establish the conditional entropy collapse as the quantity we should care about.

Second, the variance accumulation / martingale result does not directly establish conditional entropy collapse. Theorem 4.1 considers iterative self-distillation, where the model trains on samples from its own distribution and the expected parameter update is zero. This relies on a strong assumption that entropy is concave in the parameters (not generally true?). Further, the theorem studies the entropy of the full distribution, whereas the phenomenon motivating CKL is the degradation of the correctness-conditioned distribution? I do not see a theoretical argument establishing that the martingale result extends to this setting?

A more direct investigation would be nice. For instance, do higher-variance gradient estimators cause greater correctness-conditioned entropy reduction? Such experiments could help establish whether variance accumulation is actually responsible for the observed collapse.

Overall, I believe the individual directions explored in this paper are interesting, and each could be the central contributions of a more focused paper. Yet, the breadth came at the cost of depth, and I do not feel the current paper’s overall scientific narrative is coherent or sufficiently investigated to be convincing.


[1] GX-Chen, Anthony, et al. "KL-regularized reinforcement learning for generative modelling is designed to mode collapse." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Dayan, Peter, and Geoffrey E. Hinton. "Using expectation-maximization for reinforcement learning." Neural Computation 9.2 (1997): 271-278.

[3] Abdolmaleki, Abbas, et al. "Learning from negative feedback, or positive feedback or both." International Conference on Learning Representations. Vol. 2025. 2025.

[4] Kaddour, Jean. "Target policy optimization." arXiv preprint arXiv:2604.06159 (2026).

Questions for authors

Q1: What is the point of Theorem 2.1, when later you go on to motivate a different target distribution for RL in Table 1, bottom? It would also be worthwhile to establish how this is novel since I believe it is identical to Equation 8 of [1] and Remark 3.3 of [2].

Q2: I believe the idea behind CKL is very similar to the “differential regularization / smoothing” idea of [3] which to my understanding is effectively an entropy regularizer only on the positive samples. Can you comment on this? A conceptual and empirical comparison would be helpful.


[1] Grill, Jean-Bastien, et al. "Monte-carlo tree search as regularized policy optimization." International Conference on Machine Learning. PMLR, 2020.

[2] GX-Chen, Anthony, et al. "KL-regularized reinforcement learning for generative modelling is designed to mode collapse." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Gai, Jingchu, et al. "Differential smoothing mitigates sharpening and improves llm reasoning." arXiv preprint arXiv:2511.19942 (2025).

7 comments

Thank you very much for your thoughtful and thorough review. We have carefully prepared a detailed rebuttal addressing your comments, and we sincerely appreciate the time and effort you devote to reading our responses. We hope that our explanations help address your concerns, and we would be more than happy to engage in further discussion if you have any additional questions.

AuthorsOpenReview ↗

Q1: The Use of Theorem 2.1 and Research Ideas

Theorem 2.1 is actually the start point of our work which inspires all the following researches. Here we would like to further clarify the insights behind our research and how these directions are connected.

As you point out in the review, the optimal solution of RL with KL restriction in the policy space is a classical result that can be found in many other works. Our novel observation over this theorem is: Responses with the same reward receive the same probability multiplier, preserving their conditional distribution (The same result also hold for reverse KL restriction). And if we take the limit β→0\beta \to 0, the ideal RL process will eventually transform the model distribution to its true conditional distribution. This yields the distribution-matching form of RL.

We know that real RL usually causes collapse in diversity and may not preserve this property. This naturally introduces two questions:

  • What might be the cause of the distribution drift and what is the consequence?
  • How to better align real RL to the ideal case?

Theorem 2.1 provides strong theoretical guidance for the second question. Specifically, it suggests that both the conditional distribution and conditional entropy should remain invariant under ideal RL, and that neither quantity should depend on accuracy. Therefore, from a theoretical perspective, conditional entropy is expected to be a more informative metric than entropy, which further motivates the design of our CKL algorithm.

AuthorsOpenReview ↗

Further Declaration on Theoretical Results

Our main theoretical analyses focus on the issue of diversity collapse in RL. Before diving into the technical details of our theoretical proofs, we would like to first elaborate on the motivation behind our investigation. In particular, we will explain why we were led to explore the seemingly counterintuitive theory of variance accumulation, and why we believe it provides a more accurate explanation of the underlying mechanism.

Existing Explanations on Diversity Collapse

Most existing studies attribute diversity collapse in RL to two factors: selection bias and reinforcement bias [1][2][3]. We have cited and discussed some of these studies in Lines 300–306 of our manuscript, and here we wont to further explain this point. The paper you mentioned [1] also adopts the same theoretical explanation. While this explanation appears highly intuitive and easy to understand, it is fundamentally incorrect. In the following, we will take [1] as an example to illustrate why this explanation fails.

The bias explanation says that, the trajectories with higher probability are sampled more frequently and receive higher enhancement during RL. This sharpens the distribution of correct samples. However, the same deduction could give some wrong predictions. In the section 3.2 of [1] they said: "Crucially, the analysis of reinforcement and selection biases pertains specifically to correct trajectories. Conversely, incorrect trajectories exhibit an inverse dynamic: high-probability errors are subject to stronger penalization. Consequently, the probability distribution over incorrect trajectories is flattened rather than sharpened." This claim is false since in our experiments in Figure 1, Figure 3 and Figure 7, the entropy of incorrect trajectories is still decreased instead of increased.

If we assume that bias is the primary cause of diversity collapse, then reversing the sign of the bias for negative samples would naturally lead to the counterfactual prediction described above. Motivated by this observation, we designed experiments such as reverse ISD, where all samples are treated as negative samples. However, even under this setting, we consistently observe a decrease in entropy over prolonged training.

In fact, existing theoretical derivations usually contain several subtle issues. For instance, UCPO [2] overlooks a term that plays a crucial role in training (please refer to our second response to Reviewer Ae1y for details). In differential smoothing, the two types of bias essentially represent the same phenomenon counted twice, and this phenomenon itself does not lead to harmful diversity collapse.

These observations led us to reconsider the underlying mechanism of diversity collapse and hypothesize that its primary driving force is the variance inherent in RL, rather than bias.

[1]: Differential smoothing mitigates sharpening and improves llm reasoning. Gai, Jingchu, et al.

[2]: Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity. Anamika Lochab, et al.

[3]: On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models. Shumin Wang, et al.

Reviewer_CZHTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper interprets pretraining, SFT, distillation and binary-reward RL from a distribution-matching perspective, and discusses three questions: the gradient variance of on-policy distillation (OPD) and SFT when the student is close to or far from the teacher; how sampling noise in iterative self-distillation decreases the expected entropy, which the paper uses to explain entropy collapse in RL; and conditional KL (CKL) regularization, which constrains only the conditional distribution over correct responses. Experiments mainly cover distillation, self-distillation and mathematical RL with Qwen3 models.

Critical strengths

The CKL objective is clearly defined and the derivation is correct. At similar correct-conditional entropy, CKL obtains a higher reward than KL (Figure 4).

The paper decomposes the total entropy into the success rate and the conditional entropies of correct and incorrect responses (Lemma 5.1), showing that a drop in total entropy during RL need not mean a loss of diversity among correct solutions. This is a useful distinction.

Critical weaknesses

The three parts are loosely connected, and the distillation part is weak. The unified distribution-matching view already exists, and the distillation results are unrelated to the later RL analysis and to CKL. The proximal result is fairly direct: when the student equals the teacher, the OPD gradient estimate is zero by construction. The distal result only constructs an example with very large variance, but the mean grows as well, so it does not show a worse signal-to-noise ratio. Empirically, the gap between proximal OPD and SFT is small and based on a single seed, and the distal case has a single failure, for a model whose context length is shorter than the response length allowed in the experiment, so the failure may have other causes.

The evidence for CKL is limited. All CKL experiments use one model, one dataset and one seed, and compare only with global KL. CKL does not constrain incorrect samples or the success rate, so it is a weaker constraint by design, and a higher reward is expected. It also acts only on groups with at least two correct samples, so in practice it mainly affects easier prompts. The paper does not compare with entropy regularization or other methods that aim to preserve the diversity of correct samples.

Questions for authors
  1. Please give a measurable criterion that separates the proximal and distal regimes, repeat the distillation experiments with several seeds, and rule out factors such as the context length in the OPD failure.

  2. Please compare CKL with KL over several seeds and on another model or task, add controls without centering and with matched difficulty, and compare with entropy regularization.

Thank you for the author's reply. I have updated my score.

4 comments

Thank you for taking the time to review our paper and provide valuable feedback. We would first like to clarify the overall reasoning behind our work and explain how the different components of the paper are logically connected.

AuthorsOpenReview ↗

Further Declaration on the Theoretical Results

Our work primarily focuses on the properties of post-training algorithms for large language models, particularly the widely observed phenomenon of diversity collapse during RL training. This issue has been extensively studied in prior work, with most existing theoretical explanations attributing diversity collapse to the inherent bias of RL training. However, our theoretical and empirical analyses challenge this prevailing explanation. Specifically, under both the Iterative Self-Distillation (ISD) and reverse ISD settings, entropy collapse consistently occurs even in the absence of training bias. Moreover, the entropy dynamics of negative samples during RL training are inconsistent with the predictions of previous theories. We have discussed these findings in greater detail in our responses to the other two reviewers. If you are interested in the details, please refer to the second official comment for the other two reviewers.

Motivated by these observations, we develop a variance-based theoretical explanation for diversity collapse and provide empirical evidence to support it. The comparison between OPD and SFT emerged naturally from our investigation into training variance. However, this comparison is not a central conclusion of our paper, and we therefore did not pursue an exhaustive analysis of it. Nevertheless, we agree that comparing these two algorithms and characterizing the boundaries of their effectiveness are both interesting research directions, which we are also actively exploring in our ongoing work.

More importantly, the significance of this investigation in the context of our paper lies in the observation that the nonzero proximal variance of SFT can lead to a decrease in expected entropy under iterative training. This result contradicts previous theoretical predictions, while our experimental findings support our analysis. We further investigate the implications of this phenomenon in RL training. The theoretical study of distillation algorithms then provides the foundation for our subsequent analysis. The fundamental connection among these algorithms is that they can all be viewed as different forms of distribution matching. This unified perspective, together with the implications of Theorem 2.1, ultimately motivates the design of CKL. Since ideal RL training should preserve the conditional distribution, introducing an additional CKL constraint to explicitly control this property offers a promising way to achieve better performance.

We acknowledge that the direct discussion of distillation in the current manuscript is relatively limited. Nevertheless, we hope that the above clarification helps illustrate the logical connections among the different components of our work, as well as the broader significance of our findings. In the following, we provide additional experimental results that further substantiate our existing theoretical and empirical findings.

AuthorsOpenReview ↗

Distillation Experiments

We have additionally conducted six repeated training and evaluation runs to supplement the results reported in Table 3 of our paper. All runs follow the same training configuration as in the original paper, using the same teacher model obtained after two epochs of RL training, but with different random seeds. Below we report the accuracy evaluated by 8 samples per prompt. The resulting test accuracies are reported below. As shown, these results remain consistent with our theoretical predictions.

methodAIME2024AMC23MATH500
OPD-seed424.1729.758.8
OPD-seed432.9228.457.9
OPD-seed445.8330.057.9
SFT-seed424.1729.457.6
SFT-seed435.8325.958.5
SFT-seed441.6730.658.2
OPD average4.3129.758.2
SFT average3.8928.658.1

In the proximal distillation setting, the difference between OPD and SFT is relatively small, since the training target is already close to the original model. More importantly, the significance of our proximal variance analysis lies in establishing a theoretical foundation for the subsequent analysis of iterative methods. In iterative distillation, the effect of this variance can be substantially amplified, leading to pronounced diversity collapse. This provides valuable insights into understanding the underlying mechanisms of RL and related algorithms.

Regarding the collapse of OPD under distal distillation, our paper primarily presents one representative case of such failure. Since both the teacher and student originate from the same base model, context length should not be a contributing factor in this setting. In fact, the stability of OPD has also been investigated in several recent empirical studies, some of which we have cited in our paper [1][2]. These studies have identified various experimental settings in which OPD suffers from training collapse, typically when the teacher and student distributions differ substantially in certain regions. Meanwhile, SFT-based distillation rarely encounters such stability issues, even in distal settings. Our work provides further theoretical foundations for understanding these empirical observations.

[1]: Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, Yaxuan Li, et al.

[2]: KL for a KL: On-Policy Distillation with Control Variate Baseline, Oh Minjae, et al.

AuthorsOpenReview ↗

CKL Regularization

We have also conducted additional experiments to further evaluate CKL. First, we would like to clarify that we are aware that CKL is inherently a weaker constraint than KL by design. Therefore, to ensure a fair comparison, we actually used different constraint coefficients of 1.0 and 0.3 for CKL and KL, respectively, in our main experiments. As shown in Figure 4, we empirically observe that, under these settings, CKL and KL exert comparable regularization effects on the conditional distribution, while the difference in overall entropy can be largely attributed to improvements in reward. Even under these conditions, CKL still achieves higher rewards.

The main findings in our original paper were obtained using Qwen3-1.7B-Base. To further validate our conclusions, we have conducted additional experiments with different random seeds and hyperparameter settings, as well as extended our evaluation to Qwen3-8B-Base. These additional results consistently support our main conclusions. These experiments are all trained on the MATH dataset for 4 epochs if not specified.

Experiments on Qwen3-1.7B-Base

methodΔ\Deltaseq. cond. entropyΔ\Deltaaccuracy
KL-1.0-163.540.067
KL-0.3-201.130.130
KL-0.1-246.660.245
CKL-1.0-190.420.175
CKL-0.3-199.180.255
CKL-0.1-258.630.310
CKL-0.1 (non-centering)-285.280.281

Experiments on Qwen3-8B-Base

methodΔ\Deltaseq. cond. entropyΔ\Deltaaccuracy
KL-0.3-141.820.167
KL-0.1-149.230.285
CKL-0.3-130.920.300
CKL-0.1-161.780.310
Differential Smoothing 0.011986.240.251
Entropy Regularization 0.0120133.070.243

Across these experiments, CKL-0.1 achieves an accuracy improvement of approximately 0.310 on both the 1.7B and 8B models, with final accuracies of 0.6183 and 0.7968, respectively. Interestingly, under other experimental settings, CKL appears to perform even better on the larger model, achieving greater performance improvements at a lower entropy cost.

Additionally, we would like to provide further clarification regarding the comparison with the last two entropy-based control methods. Differential Smoothing (DS)[1] is a related method proposed by another reviewer, KMeX. It can be viewed as an improved variant of direct entropy regularization. In our experiments, we use the same hyperparameter settings as those adopted in the main experiments of the DS paper [1].

Entropy regularization methods have long faced challenges related to hyperparameter sensitivity and training instability, which may partly explain their limited adoption in practice. For instance, the official AReaL training framework we use does not provide an implementation of entropy loss by default. This also highlights an important distinction: entropy can serve as a reliable metric only when it is not directly optimized as a training objective.

In our experiments, both of the last two methods remain stable during the first two epochs. However, starting from the third epoch, both exhibit a substantial increase in entropy that persists until the end of training. Crucially, this does not indicate that these methods have discovered more diverse problem-solving strategies. Instead, they artificially increase entropy by appending meaningless or garbled tokens to the end of their responses. This is clearly not the type of diversity we aim to encourage. In contrast, constraint-based methods such as KL and CKL exhibit substantially more stable behavior. For further discussion, please also refer to our response to Reviewer KMeX.

Here are some samples generated by Qwen3-8B-Base trained with entropy loss regularization whose reward indeed increased after training.

最大化DGまったく życiu با콱公共 św磡קצבadministr espec_

[1]: Differential smoothing mitigates sharpening and improves llm reasoning. Gai, Jingchu, et al.

Reviewer_Ae1yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a distribution-matching view of LLM training, derives proximal and distal variance comparisons between KD and OPD, and argues that accumulated gradient noise can cause entropy collapse in iterative training. It further proposes Conditional KL (CKL), a reverse KL between correctness-conditioned policy distributions, and evaluates these claims through single-seed mathematical-reasoning experiments comparing SFT/OPD, variance-reduction methods, and CKL/standard KL.

Critical strengths

The proximal/distal variance analysis offers an interpretable, testable explanation for why OPD may be stable near the teacher yet fail under substantial teacher–student mismatch, and the reported distillation experiments qualitatively support this prediction.

Critical weaknesses

The central entropy-collapse explanation is not established for LLM RL. Theorem 4.1 requires entropy to be concave along every parameter-update segment, which is generally false even for a binary softmax. Moreover, the theorem covers zero-mean self-distillation with plain SGD, not nonzero-mean RL with GRPO/AdamW; single-seed trajectories cannot validate an expectation-level claim. Thus, the paper’s main explanation of long-term RL entropy collapse is unsupported.

The novelty and implemented objective of CKL are unclear. Earlier CoKL already proposed regularizing correctness-conditioned distributions for essentially the same motivation, while UCPO analyzed stochastic collapse within the correct-response set. The reverse-KL direction and centered estimator may be incremental contributions, but CoKL is not cited or compared. Furthermore, Eq. (13) scales each prompt’s expected CKL gradient by its current success probability, so the implementation does not optimize the stated uniformly prompt-averaged CKL and provides almost no regularization for difficult prompts.

Typo in Equation (4).

References:

[1] Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning. Li Wang, et al.

[2] Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity. Anamika Lochab, et al.

Questions for authors

The CKL minimizes DKL(πθ+∥πref+)D_{\mathrm{KL}}(\pi_\theta^+\|\pi_{\mathrm{ref}}^+), whereas the earlier CoKL of Wang et al. [1] minimizes the opposite KL direction for essentially the same correctness-conditioned preservation goal. Why is this closest work omitted, and what benefit specifically follows from reversing the KL direction and using the centered estimator? Please provide a direct comparison and explain how the success-rate-dependent scaling in Eq. (13) corresponds to the stated prompt-averaged CKL objective.

References:

[1] Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning. Li Wang, et al.

4 comments

Thank you for your careful review! Here I want to make more explanations on your concerns.

AuthorsOpenReview ↗

Citation and Comparison with CoKL

Thank you for pointing out this work. As far as We know, this work was released on Aug 3, 2026, when our work is almost done. So we are more like concurrent and independent works. Though it we share some initial ideas on regularization methods in RL, the ultimately proposed algorithms are completely different in practice.

Let's begin with our method which is much simpler. CKL regulates DKL(πθ+∣∣πref+)D_\mathrm{KL} (\pi_\theta^+ || \pi_\mathrm{ref}^+), and we can use a direct estimator similar to "k1 in reward" in classical KL regularization. As shown in theorem 5.1, we need the samples from current policy, and the log probability of the policy model and reference model on these samples to calculate CKL. It is equivalent to adding the following term to the reward:

rCKL,i+=ri−β(log⁡πθ(Yi)πref(Yi)−1N∑j∈I+log⁡πθ(Yj)πref(Yj))r_\mathrm{CKL, i}^+ = r_i - \beta \left(\log\frac{\pi_\theta (Y_i)}{\pi_\mathrm{ref} (Y_i)} - \frac1N \sum_{j\in I^+}\log\frac{\pi_\theta (Y_j)}{\pi_\mathrm{ref} (Y_j)}\right)

CKL is quite easy to be integrated into existing RL frameworks by modifying several lines of code and CKL do not require additional computation.

CoKL optimizes the opposite KL direction, which introduces much more complexity. Their implementation is primarily built on Eq. 10 in their paper, which says

∇θLCoKL(x)=−Ey∼πref+(⋅∣x)[∇θlog⁡πθ(y∣x)]+Ey∼πθ+(⋅∣x)[∇θlog⁡πθ(y∣x)]\nabla_\theta \mathcal{L}_\mathrm{CoKL} (x) = -\mathbb{E}_{y\sim\pi_\mathrm{ref}^+ (\cdot | x)} [\nabla_\theta \log \pi_\theta (y | x)] +\mathbb{E}_{y\sim\pi_\theta^+ (\cdot | x)} [\nabla_\theta \log \pi_\theta (y | x)]

The estimator for this objective requires samples from both the policy model and the reference model. Implementing CoKL requires a large modification on the training pipeline. Moreover, sampling from reference model could be as expensive as RL itself. To handle this problem, CoKL uses less batchsize to compute the first term, and reuses some samples from the reference model. In our experiments CoKL still slows down the whole training process by around 49.2% compared to CKL with these optimizations.

To provide an empirical comparison, we conducted additional experiments using Qwen3-1.7B-Base on the MATH dataset, with four training epochs and comparable regularization strengths. The results are summarized below.

methodΔ\Deltaseq. cond. entropyΔ\Deltaaccuracy
KL-246.660.245
CoKL-258.320.309
CKL-258.630.310

We find that CoKL achieved comparable balance in divergence of true samples and accuracy with CKL, but CoKL comsumes significantly higher computational resources and introduces more complexity in the training pipeline. Hence we believe our CKL is still a more elegant algorithm.

AuthorsOpenReview ↗

Theoretical Concerns and Relationship with UCPO

Aside from CKL regularization, another important part in this work focuses on exploring the mechanism behind the diversity collapse especially within the correct-response set. The diversity collapse is already widely studied in previous works while there are evidence that the diversity of the correct samples are of more importance. It is usually attributed to the sampling bias or gradient amplification in existing literature [1][2][3]. We have cited more related works in Section 4, and showed that These explanations could be wrong. We have briefly discussed this in section 4 and here I would like to provide more comments.

Sampling bias says that since samples with higher probability are sampled more frequently, then they will receive larger updates as their gradient is proportional to the current probability, say ∼πθ∇θlog⁡πθ\sim \pi_\theta \nabla_\theta \log \pi_\theta. the gradient amplification mechanism is actually discussing similar things. But they omitted an important term ∇θlog⁡πθ\nabla_\theta \log \pi_\theta which is also heavily dependent on the probability. The actual updates may not satisfy this property and thus this mechanism fails to explain results in self-distillation.

Self-distillation shares some important properties with RL but is easier to analyse. This is achieved by setting all rewards to 1 or -1 in RL. If all rewards are set to 1, we can prove the expected gradient is identically 0. Here the sampling bias still exists but it has no contribution to the expected gradient. Their effects are precisely cancelled by the term ∇θlog⁡πθ\nabla_\theta \log \pi_\theta. Even so we still observed entropy collapse in experiments.

A more important evidence come from the reversed Iterative Self-Distillation (ISD), where all rewards are set to -1. In this case if we agree with the sampling bias explanation, the responses with higher probability will naturally receive higher penalty and thus increases the entropy. But the expected gradient is still zero and we constantly observed entropy collapse in Figure 1 and Figure 7 in the appendix.

We then proposed a new explanation for this phenomenon through variance analysis. I hope you could also find it novel and interesting.

Our main idea is that, since the conditional entropy collapse still exists even when the expected gradient is 0, then it may be a consequence of the variance instead of the expectation of the objective. The following study comes from a simple observation, that since entropy is a concave function of the probability, if we add a random noise ϵ\epsilon to the probability in the policy space, then the entropy HH satisfies:

E[H(πθ+ϵ)]<E[H(πθ)]\mathbb{E} [H(\pi_\theta + \epsilon)] < \mathbb{E} [H(\pi_\theta)]

This indicates random noise can result in a deterministic collapse in the long term, and the expected behaviour can exhibit after the accumulation through steps. Apparently the random noise also exists in RL and affects the conditional distribution. We know that entropy is a concave function of probabilities. From the distribution matching perspective, we use parameter gradients induced by the corresponding loss functions to approximate changes in the probability distribution. Consequently, this concavity can be expected to hold approximately with respect to the model parameters, at least to some extent. Aside from these theoretical results, we also conducted many experiments on this and observed many supportive results in both ISD and RL, see Section 4.

Due to the complexities of model architectures, optimization algorithms, and other practical factors, rigorously proving this result in practical RL settings remains highly challenging. In fact, many theoretical analyses in ML can not be fully rigorously established either. For instance, theoretical analyses of optimizers often rely on convexity assumptions that are not generally satisfied, while previous studies on the mechanisms underlying RL algorithms have not sufficiently accounted for the effects of different optimizers, model structure and certain additional terms. We hope that our proposed variance accumulation theory provides a better explanation of the observed phenomena and offers additional insights into the underlying mechanisms. We therefore hope that the reviewer will rethink and recognize the value of our contribution to understanding the mechanisms behind diversity collapse in RL.

[1]: Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity. Anamika Lochab, et al.

[2]: Differential smoothing mitigates sharpening and improves llm reasoning. Gai, Jingchu, et al.

[3]: On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models. Shumin Wang, et al.

AuthorsOpenReview ↗

Cross-Prompt Average Behaviour of CKL

Here you mentioned a subtle issue in the design of CKL. As mentioned in Line 173 - 175 in our paper, our main discussion is built on the conditional KL divergence on a single sample and that is where RL and MaxRL coincide. However, their behaviour differs in cross-prompt average schemes and can thus affect the behaviour of KL or CKL regularization.

We adopted the simplest approach in implementing CKL: directly subtracting the mean for centering, without applying any additional scaling, but this is indeed a valuable direction to explore. We have then empirically evaluated both variants, with and without scaling, and found no significant difference in their performance under comparable strength. That said, we acknowledge that there is still considerable room for further exploration regarding CKL, particularly its cross-prompt averaging strategies and their interactions with different algorithms. We will include additional discussion of these aspects in the revised manuscript.

methodΔ\Deltaseq. cond. entropyΔ\Deltaaccuracy
CKL-258.630.310
normed CKL-251.040.303

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.