- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper seeks to provide a general perspective on LLM (pre to post) training as distribution matching. It makes a few statements:
On-policy distillation gradient is more statistically stable than SFT when the student is closer to the teacher. Argues that while both match to the teacher distribution, OPD has lower gradient variance when student is closer to the teacher (and zero when they coincide), while SFT generally would not. When the student is far from the teacher, there are some cases where OPD can have higher variance, particularly if the OPD ratio blows up due to low teacher probabilities.
Variance accumulation leads to entropy decrease. They analyze the setting of SFT-ing a model on its own generation. Using a martingale argument and assuming entropy is concave in parameters, they show expected entropy can only stay the same or decrease. This is used as a motivating example to motivate entropy collapse in long term RL training.
Conditional KL regularization Motivated by the entropy of the correctness-conditioned distribution possibly collapsing when it should not if the target distribution for e.g. MaxRL is well-optimized, the authors call for a new KL regularizer which effectively regularizes only the positive samples. They show this can prevent collapse to an extent and also has higher rewards than regularizing both the correct and incorrect parts of the distribution.
Critical strengths
The paper seeks to provide theoretical understanding in a number of relevant post-training algorithms. As summarized in the above section, each section provides useful and interesting hypotheses about observations such as stability and entropy collapse. This also leads to a number of possibly actionable algorithmic choices, such as when to use SFT vs. OPD, and a new CKL regularizer.
I enjoyed reading about target distribution degradation as the reason for collapse. To me, one key thread in the paper is that success-conditioned entropy can decrease from training when they should not if the target is perfectly optimized. This feels distinct from simply “entropy decreasing” which can be due to removing probability mass from negative sample as well. This also motivates ways to try to mitigate this, such as via the conditional KL regularizer.
Critical weaknesses
The paper is overly broad; the novelty & value of the unified framework is unclear
The paper attempts to provide a general distribution-matching framework that explains several practically meaningful observations in LLM training. However, I find the scope overly ambitious, and led to a paper that tries to explore a few (interesting) directions but ultimately without investigating any of them sufficiently deeply scientifically. It also generally lacks clarity in expositing the various research directions.
There are essentially two distinct research directions:
- Understanding the trade-off between OPD and SFT/KD, and when one should be preferred over the other.
- Understanding how the correctness-conditioned distribution can degrade during RL training with binary rewards, and whether the proposed Conditional KL (CKL) regularization can mitigate this degradation.
The distribution-matching framework is motivated to help with both. Yet, I do not find it very informative. The connection between cross-entropy and forward KL is standard (e.g., [1]), as is the reverse-KL formulation of OPD. Iterative matching to reward-conditioned distributions has also been studied previously, e.g. EM interpretations of RL [2, 3, 4, many more]. While the authors do acknowledge some related literature (e.g. Korbak et al 2022), it is missing others above, and I do not think the paper establishes what is fundamentally new compared with these earlier perspectives. The unified framework currently occupies a substantial part of the paper, and in my opinion does not actually help me understand the rest of the paper.
The proposed correctness-conditioned entropy collapse mechanism is insufficiently investigated.
Focusing just on the second research direction, I understand it as:
- Although an idealized RL update preserves the correctness-conditioned distribution, practical training can cause this distribution to degrade, reducing its entropy.
- This degradation may be explained by the variance-accumulation mechanism studied through the martingale result (Theorem 4.1).
- CKL regularization can mitigate this degradation while allowing greater reward improvement than standard KL regularization.
I find this interesting. However, it is missing a number of connections to fully establish this narrative:
First, the generality and consequences of correctness-conditioned entropy collapse are unclear. The authors demonstrate this phenomenon for GRPO in Figure 3. However, it would be useful to measure how the correctness-conditioned entropy evolves across different objectives and gradient estimators, such as GRPO, RLOO, and MaxRL, to establish how general and significant the phenomenon is. It would also be useful to connect the decrease directly to measurable consequences, e.g. does more degradation of the conditional target lead to more reduced diversity of correct solutions, worse pass@k, etc, and in a way that is not predicted by the unconditional entropy? Basically what I am asking is to more convincingly and thoroughly establish the conditional entropy collapse as the quantity we should care about.
Second, the variance accumulation / martingale result does not directly establish conditional entropy collapse. Theorem 4.1 considers iterative self-distillation, where the model trains on samples from its own distribution and the expected parameter update is zero. This relies on a strong assumption that entropy is concave in the parameters (not generally true?). Further, the theorem studies the entropy of the full distribution, whereas the phenomenon motivating CKL is the degradation of the correctness-conditioned distribution? I do not see a theoretical argument establishing that the martingale result extends to this setting?
A more direct investigation would be nice. For instance, do higher-variance gradient estimators cause greater correctness-conditioned entropy reduction? Such experiments could help establish whether variance accumulation is actually responsible for the observed collapse.
Overall, I believe the individual directions explored in this paper are interesting, and each could be the central contributions of a more focused paper. Yet, the breadth came at the cost of depth, and I do not feel the current paper’s overall scientific narrative is coherent or sufficiently investigated to be convincing.
[1] GX-Chen, Anthony, et al. "KL-regularized reinforcement learning for generative modelling is designed to mode collapse." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Dayan, Peter, and Geoffrey E. Hinton. "Using expectation-maximization for reinforcement learning." Neural Computation 9.2 (1997): 271-278.
[3] Abdolmaleki, Abbas, et al. "Learning from negative feedback, or positive feedback or both." International Conference on Learning Representations. Vol. 2025. 2025.
[4] Kaddour, Jean. "Target policy optimization." arXiv preprint arXiv:2604.06159 (2026).
Questions for authors
Q1: What is the point of Theorem 2.1, when later you go on to motivate a different target distribution for RL in Table 1, bottom? It would also be worthwhile to establish how this is novel since I believe it is identical to Equation 8 of [1] and Remark 3.3 of [2].
Q2: I believe the idea behind CKL is very similar to the “differential regularization / smoothing” idea of [3] which to my understanding is effectively an entropy regularizer only on the positive samples. Can you comment on this? A conceptual and empirical comparison would be helpful.
[1] Grill, Jean-Bastien, et al. "Monte-carlo tree search as regularized policy optimization." International Conference on Machine Learning. PMLR, 2020.
[2] GX-Chen, Anthony, et al. "KL-regularized reinforcement learning for generative modelling is designed to mode collapse." International Conference on Learning Representations. Vol. 2026. 2026.
[3] Gai, Jingchu, et al. "Differential smoothing mitigates sharpening and improves llm reasoning." arXiv preprint arXiv:2511.19942 (2025).
7 comments
Q1: The Use of Theorem 2.1 and Research Ideas
Theorem 2.1 is actually the start point of our work which inspires all the following researches. Here we would like to further clarify the insights behind our research and how these directions are connected.
As you point out in the review, the optimal solution of RL with KL restriction in the policy space is a classical result that can be found in many other works. Our novel observation over this theorem is: Responses with the same reward receive the same probability multiplier, preserving their conditional distribution (The same result also hold for reverse KL restriction). And if we take the limit , the ideal RL process will eventually transform the model distribution to its true conditional distribution. This yields the distribution-matching form of RL.
We know that real RL usually causes collapse in diversity and may not preserve this property. This naturally introduces two questions:
- What might be the cause of the distribution drift and what is the consequence?
- How to better align real RL to the ideal case?
Theorem 2.1 provides strong theoretical guidance for the second question. Specifically, it suggests that both the conditional distribution and conditional entropy should remain invariant under ideal RL, and that neither quantity should depend on accuracy. Therefore, from a theoretical perspective, conditional entropy is expected to be a more informative metric than entropy, which further motivates the design of our CKL algorithm.
Further Declaration on Theoretical Results
Our main theoretical analyses focus on the issue of diversity collapse in RL. Before diving into the technical details of our theoretical proofs, we would like to first elaborate on the motivation behind our investigation. In particular, we will explain why we were led to explore the seemingly counterintuitive theory of variance accumulation, and why we believe it provides a more accurate explanation of the underlying mechanism.
Existing Explanations on Diversity Collapse
Most existing studies attribute diversity collapse in RL to two factors: selection bias and reinforcement bias [1][2][3]. We have cited and discussed some of these studies in Lines 300–306 of our manuscript, and here we wont to further explain this point. The paper you mentioned [1] also adopts the same theoretical explanation. While this explanation appears highly intuitive and easy to understand, it is fundamentally incorrect. In the following, we will take [1] as an example to illustrate why this explanation fails.
The bias explanation says that, the trajectories with higher probability are sampled more frequently and receive higher enhancement during RL. This sharpens the distribution of correct samples. However, the same deduction could give some wrong predictions. In the section 3.2 of [1] they said: "Crucially, the analysis of reinforcement and selection biases pertains specifically to correct trajectories. Conversely, incorrect trajectories exhibit an inverse dynamic: high-probability errors are subject to stronger penalization. Consequently, the probability distribution over incorrect trajectories is flattened rather than sharpened." This claim is false since in our experiments in Figure 1, Figure 3 and Figure 7, the entropy of incorrect trajectories is still decreased instead of increased.
If we assume that bias is the primary cause of diversity collapse, then reversing the sign of the bias for negative samples would naturally lead to the counterfactual prediction described above. Motivated by this observation, we designed experiments such as reverse ISD, where all samples are treated as negative samples. However, even under this setting, we consistently observe a decrease in entropy over prolonged training.
In fact, existing theoretical derivations usually contain several subtle issues. For instance, UCPO [2] overlooks a term that plays a crucial role in training (please refer to our second response to Reviewer Ae1y for details). In differential smoothing, the two types of bias essentially represent the same phenomenon counted twice, and this phenomenon itself does not lead to harmful diversity collapse.
These observations led us to reconsider the underlying mechanism of diversity collapse and hypothesize that its primary driving force is the variance inherent in RL, rather than bias.
[1]: Differential smoothing mitigates sharpening and improves llm reasoning. Gai, Jingchu, et al.
[2]: Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity. Anamika Lochab, et al.
[3]: On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models. Shumin Wang, et al.
Thank you very much for your thoughtful and thorough review. We have carefully prepared a detailed rebuttal addressing your comments, and we sincerely appreciate the time and effort you devote to reading our responses. We hope that our explanations help address your concerns, and we would be more than happy to engage in further discussion if you have any additional questions.