- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper tries to resolve the entropy collapse difficulty during LLM RL training, where the entropy becomes too low and stops exploring. The authors propose Entrocraft, which selectively trains on positive or negative-advantage responses to keep the model's entropy close to a chosen curve. On math reasoning benchmarks, the authors report that this helps both GRPO and GSPO keep improving over multiple benchmarks.
Critical strengths
- The presentation of the paper is pretty clear and overall method is easy to understand.
- Entropy collapse is a important problem for LLM RL.
- The paper compares with relevant entropy technique baselines over multiple math benchmarks.
Critical weaknesses
- The assumptions make the theoretical results unrealistic
- The strongest assumption is deferred to appendix, which requires all the tokens to have the same direction towards their advantage. Note that in GRPO, the token advantage is same across a full rollout. However, because they share model parameters, their gradients can conflict, so if I understand it correctly, an update that increases the probability of a positive-advantage rollout does not necessarily increase the conditional probability of every token in it. The assumption that also looks strong. If I interpret it correctly, together with token-level probability alignment, it already guarantees the desired first-order entropy direction, so it reads a bit like handy-crafted theory for specific strong condition.
-
I believe entropy collapse should be applied to long-rollout model, but all current models are base models and the authors capped the response to only 3K, so the scalability to actual models that really suffer from entropy collapse remains unvalidated.
-
The experiment and training setup also lowers the GRPO baseline performance. Similar paper (e.g., arxiv.org/pdf/2602.11748v1) with avg@32 reaches 18.7 on AIME25, but this paper reports 9.0 for avg@32 on AIME25, which deviates too much. Previous works report Qwen3-4B-Base, before any training, would reach avg@32 of 6.9 on AIME25, so I guess this require some serious clarification. A 450k-step RL can only raise GRPO to around 10.0, maybe cap it to 3K is too restrictive to model performance? Also, it would be better if you can use DeepScaleR or MATH training set, which is more standard to RL-LLM than Numina-Math you used.
-
The scalability experiment in appendix to larger models and different model families are not compared with the other entropy technique baselines, so it remains unclear whether this method can scale better comparing to the other techniques.
Questions for authors
Please address the concerns and questions in the weakness section.
2 comments
Q4: Comparing Entropy Baselines with Other Models
Due to time and resource limit, we evaluate the closest baseline -- EntroPIC, on Qwen3-8B-Base and Llama-3.1-8B-Instruct to demonstrate Entrocraft's improvement over existing baselines. We will finish and add all compared models and baselines in the final version.
Qwen3-8B-Base:
| Method | MATH-500 mean@32 | MATH-500 pass@32 | AMC-23 mean@32 | AMC-23 pass@32 | AIME-25 mean@32 | AIME-25 pass@32 |
|---|---|---|---|---|---|---|
| GRPO | 77.2 | 91.8 | 59.7 | 92.5 | 14.4 | 43.3 |
| EntroPIC | 77.8 | 93.2 | 62.8 | 96.7 | 13.4 | 44.5 |
| GRPO + Entrocraft | 78.4 | 94.0 | 63.5 | 97.5 | 13.8 | 46.7 |
Llama-3.1-8B-Instruct:
| Method | MATH-500 mean@32 | MATH-500 pass@32 | AMC-23 mean@32 | AMC-23 pass@32 | AIME-25 mean@32 | AIME-25 pass@32 |
|---|---|---|---|---|---|---|
| GRPO | 56.2 | 82.2 | 30.8 | 75.0 | 10.4 | 16.7 |
| EntroPIC | 52.2 | 80.6 | 33.8 | 77.5 | 10.3 | 20.0 |
| GRPO + Entrocraft | 57.1 | 85.2 | 36.8 | 85.0 | 18.8 | 26.7 |
Thank you so much for taking time to read the details of our paper. The raised concerns are all helpful and constructive for improving the clarity. If you find the responses below sufficient to address your concerns, we truly appreciate your re-consideration for the score.
Q1: Assumptions in Theoretical Analysis
We acknowledge that the current version of Theorem 2's proof relies on over-strong assumptions for Eq. 12, which undermines the generality of our theoretical results. We therefore update the proof of Theorem 2 as follows:
First, we denote Ly=logπθ(y∣x) to be the log-likelihoods. Then, let's assume a question-response pair (x,y) triggers an parameter update δθ. The token-prob update can be written as δpt,i=<gradθpt,i,δθ>+O(∣∣δθ∣∣22)=ηA^(x,y)<gradθpt,i,gradθLy>+O(η2).
We take the above equation into Eq. 11 to simplify the expression: δH=−∣y∣1∑t=1∣y∣∑i∈Vlogpt,i⋅<gradθpt,i,gradθLy>+O(η2)=<gradθH,δθ>+O(∣∣δθ∣∣22)=ηA^(x,y)⋅<gradθH,gradθLy>+O(η2).
If we denote γy=<gradθH,gradθLy>, we can express the relationship between entropy changes and advantages as: δH=−ηA^(x,y)γy+O(η2), and the directions of entropy changes is determined by the sign of γy.
Then we empirically analyze the value of γy across a Numina-Math 1K subset. Similar to Table 2(a), we show the results in the table below. Under all three cases, the majority of trajectories have positive γy, validating the entropy changing trend.
Q2: Entropy Collapse in Long-Rollout Settings
We agree that evaluation beyond single-turn tasks would strengthen the paper. We chose mathematical reasoning as a controlled setting with established RL baselines, which aligns with previous papers' experiment settings [1, 2, 3]. To evaluate Entrocraft on more challenging cases, we already include AIME as the hard benchmarks, and scale up the training steps to see if the performance improvement is persistent (Section 5.3).
Entrocraft operates on policy entropy and estimated advantages, without relying on single/multi-turn trajectory structures, making application to longer trajectories plausible. We will clarify the scope of our empirical claims and explicitly acknowledge the generalization to multi-turn tasks as an open limitation.
[1] https://arxiv.org/abs/2511.15248
[2] https://arxiv.org/abs/2505.22617
[3] https://arxiv.org/abs/2506.01347
Q3: Training Setup and Baseline Numbers
We would like to clarify the numerical differences between this paper and the mentioned paper [4]. The most important training setting differences are shown in the table below. [4] allows for longer trajectories and use a newer training dataset, which explains why its reported numbers are consisently higher than ours.
We also compare other training settings in the table below, validating that the context length is more responsible for the numerical differences.
Therefore, we add additional evaluations based on our save checkpoint, enabling 8K context length. The results are shown in the table below. The GRPO baseline results match the mentioned papers. We will also separately add the 8K-context evaluation to our final paper.
[4] https://arxiv.org/abs/2602.11748
[5] https://huggingface.co/datasets/project-numina/NuminaMath-CoT
[6] https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k
[7] https://arxiv.org/abs/2510.13554