acceptodds
Under review as a conference paper at ICLR 2027

Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control

Abstract

Reinforcement learning (RL) has enabled complex reasoning abilities in large language models (LLMs). However, most RL algorithms suffer from performance saturation, preventing continued gains as RL training scales. This problem can be characterized by the collapse of entropy, a key diagnostic for exploration in RL. Current approaches focus on preventing entropy collapse through regularization or clipping. However, their resulting entropy curves often exhibit instability in the long term, which hinders performance gains. In this paper, we introduce Entrocraft, a simple rejection-sampling approach that realizes user-customized entropy schedule by biasing the advantage distributions. Entrocraft requires no objective regularization and is advantage-estimator-agnostic. Theoretically, we relate per-step entropy change to the advantage distribution under minimal assumptions. Our analysis explains the behavior of existing RL and entropy-preserving methods and enables a systematic study of entropy schedules. It shows that linear annealing, which starts high and decays to a slightly lower target, achieves strong performance. Empirically, Entrocraft addresses performance saturation, significantly improving generalization, output diversity, and long-term training. It enables a 4B model to outperform an 8B baseline, sustains improvement for up to 400K training samples, and significantly raises pass@K over the baseline.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Entropy Control, Performance Saturation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Wwa9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper tries to resolve the entropy collapse difficulty during LLM RL training, where the entropy becomes too low and stops exploring. The authors propose Entrocraft, which selectively trains on positive or negative-advantage responses to keep the model's entropy close to a chosen curve. On math reasoning benchmarks, the authors report that this helps both GRPO and GSPO keep improving over multiple benchmarks.

Critical strengths
  1. The presentation of the paper is pretty clear and overall method is easy to understand.
  2. Entropy collapse is a important problem for LLM RL.
  3. The paper compares with relevant entropy technique baselines over multiple math benchmarks.
Critical weaknesses
  1. The assumptions make the theoretical results unrealistic
  • The strongest assumption is deferred to appendix, which requires all the tokens to have the same direction towards their advantage. Note that in GRPO, the token advantage is same across a full rollout. However, because they share model parameters, their gradients can conflict, so if I understand it correctly, an update that increases the probability of a positive-advantage rollout does not necessarily increase the conditional probability of every token in it. The assumption that bt≥0b_t \ge 0 also looks strong. If I interpret it correctly, together with token-level probability alignment, it already guarantees the desired first-order entropy direction, so it reads a bit like handy-crafted theory for specific strong condition.
  1. I believe entropy collapse should be applied to long-rollout model, but all current models are base models and the authors capped the response to only 3K, so the scalability to actual models that really suffer from entropy collapse remains unvalidated.

  2. The experiment and training setup also lowers the GRPO baseline performance. Similar paper (e.g., arxiv.org/pdf/2602.11748v1) with avg@32 reaches 18.7 on AIME25, but this paper reports 9.0 for avg@32 on AIME25, which deviates too much. Previous works report Qwen3-4B-Base, before any training, would reach avg@32 of 6.9 on AIME25, so I guess this require some serious clarification. A 450k-step RL can only raise GRPO to around 10.0, maybe cap it to 3K is too restrictive to model performance? Also, it would be better if you can use DeepScaleR or MATH training set, which is more standard to RL-LLM than Numina-Math you used.

  3. The scalability experiment in appendix to larger models and different model families are not compared with the other entropy technique baselines, so it remains unclear whether this method can scale better comparing to the other techniques.

Questions for authors

Please address the concerns and questions in the weakness section.

2 comments

Thank you so much for taking time to read the details of our paper. The raised concerns are all helpful and constructive for improving the clarity. If you find the responses below sufficient to address your concerns, we truly appreciate your re-consideration for the score.

Q1: Assumptions in Theoretical Analysis

We acknowledge that the current version of Theorem 2's proof relies on over-strong assumptions for Eq. 12, which undermines the generality of our theoretical results. We therefore update the proof of Theorem 2 as follows:

First, we denote Ly=log⁡πθ(y∣x)L_y = \log \pi_{\theta}(y | x) to be the log-likelihoods. Then, let's assume a question-response pair (x,y)(x, y) triggers an parameter update δθ\delta\theta. The token-prob update can be written as δpt,i=<gradθpt,i,δθ>+O(∣∣δθ∣∣22)=ηA^(x,y)<gradθpt,i,gradθLy>+O(η2).\delta p_{t,i} = < \text{grad}_{\theta}p_{t,i}, \delta\theta > + O(||\delta\theta||_2^2) = \eta \hat{A}(x,y) < \text{grad}_{\theta}p_{t,i}, \text{grad}_{\theta}L_y > + O(\eta^2).

We take the above equation into Eq. 11 to simplify the expression: δH=−1∣y∣∑t=1∣y∣∑i∈Vlog⁡pt,i⋅<gradθpt,i,gradθLy>+O(η2)=<gradθH,δθ>+O(∣∣δθ∣∣22)=ηA^(x,y)⋅<gradθH,gradθLy>+O(η2).\delta\mathcal{H} = -\frac{1}{|y|}\sum_{t=1}^{|y|}\sum_{i\in\mathbb{V}} \log p_{t,i} \cdot <\text{grad}_{\theta}p_{t,i}, \text{grad}_{\theta}L_y> + O(\eta^2) = <\text{grad}_{\theta}\mathcal{H}, \delta\theta> + O(||\delta\theta||_2^2) = \eta\hat{A}(x,y) \cdot <\text{grad}_{\theta}\mathcal{H}, \text{grad}_{\theta}L_y> + O(\eta^2).

If we denote γy=<gradθH,gradθLy>\gamma_y = <\text{grad}_{\theta}\mathcal{H}, \text{grad}_{\theta}L_y>, we can express the relationship between entropy changes and advantages as: δH=−ηA^(x,y)γy+O(η2),\delta\mathcal{H} = -\eta\hat{A}(x,y)\gamma_y + O(\eta^2), and the directions of entropy changes is determined by the sign of γy\gamma_y.

Then we empirically analyze the value of γy\gamma_y across a Numina-Math 1K subset. Similar to Table 2(a), we show the results in the table below. Under all three cases, the majority of trajectories have positive γy\gamma_y, validating the entropy changing trend.

CaseMean γy\gamma_yMedian γy\gamma_yγy>0\gamma_y > 0
Positive-Adv Samples3,072.774,349.9272.2%
Negative-Adv Samples8,482.917,875.7289.7%
All Samples6,610.175,511.7483.7%

Q2: Entropy Collapse in Long-Rollout Settings

We agree that evaluation beyond single-turn tasks would strengthen the paper. We chose mathematical reasoning as a controlled setting with established RL baselines, which aligns with previous papers' experiment settings [1, 2, 3]. To evaluate Entrocraft on more challenging cases, we already include AIME as the hard benchmarks, and scale up the training steps to see if the performance improvement is persistent (Section 5.3).

Entrocraft operates on policy entropy and estimated advantages, without relying on single/multi-turn trajectory structures, making application to longer trajectories plausible. We will clarify the scope of our empirical claims and explicitly acknowledge the generalization to multi-turn tasks as an open limitation.

[1] https://arxiv.org/abs/2511.15248

[2] https://arxiv.org/abs/2505.22617

[3] https://arxiv.org/abs/2506.01347

Q3: Training Setup and Baseline Numbers

We would like to clarify the numerical differences between this paper and the mentioned paper [4]. The most important training setting differences are shown in the table below. [4] allows for longer trajectories and use a newer training dataset, which explains why its reported numbers are consisently higher than ours.

SettingOurs: GRPO Baseline[4]: GRPO Baseline
Training response length limit3,072 tokens8,192 tokens
Training dataNumina-Math [5]DAPO-Math-17K [6]

We also compare other training settings in the table below, validating that the context length is more responsible for the numerical differences.

Paper and settingMATH-500AMC23AIME25
Ours trained on Numina-Math: 3K responses75.357.48.9
[7] trained on DAPO-Math-17K: 1K context74.255.15.2
[7] trained on DAPO-Math-17K: 8K context81.057.616.1

Therefore, we add additional evaluations based on our save checkpoint, enabling 8K context length. The results are shown in the table below. The GRPO baseline results match the mentioned papers. We will also separately add the 8K-context evaluation to our final paper.

Paper and settingMATH-500AMC23AIME25
[4] trained on DAPO-Math-17K: 8K context80.455.218.7
[7] trained on DAPO-Math-17K: 8K context81.057.616.1
Ours trained on Numina-Math: 3K responses75.357.48.9
Ours trained on Numina-Math: 7K responses80.257.616.7

[4] https://arxiv.org/abs/2602.11748

[5] https://huggingface.co/datasets/project-numina/NuminaMath-CoT

[6] https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k

[7] https://arxiv.org/abs/2510.13554

Q4: Comparing Entropy Baselines with Other Models

Due to time and resource limit, we evaluate the closest baseline -- EntroPIC, on Qwen3-8B-Base and Llama-3.1-8B-Instruct to demonstrate Entrocraft's improvement over existing baselines. We will finish and add all compared models and baselines in the final version.

Qwen3-8B-Base:

MethodMATH-500 mean@32MATH-500 pass@32AMC-23 mean@32AMC-23 pass@32AIME-25 mean@32AIME-25 pass@32
GRPO77.291.859.792.514.443.3
EntroPIC77.893.262.896.713.444.5
GRPO + Entrocraft78.494.063.597.513.846.7

Llama-3.1-8B-Instruct:

MethodMATH-500 mean@32MATH-500 pass@32AMC-23 mean@32AMC-23 pass@32AIME-25 mean@32AIME-25 pass@32
GRPO56.282.230.875.010.416.7
EntroPIC52.280.633.877.510.320.0
GRPO + Entrocraft57.185.236.885.018.826.7
Reviewer_m4jeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This is a paper dealing with entropy collapse on RL training. Instead of regularization or clipping, this paper introduces Entrocraft, which is a rejection sampling approach that realizes user-customized entropy schedule. In other words, this work proposes a simple rejection sampling technique to filter out positive/ negative-advantage rollout samples when entropy is lower/higher than a threshold. The experiments are mainly performed on math reasoning tasks on three Qwen models and one llama model and compares agains GRPO, GSPO, W-reinforce and EntroPIC.

Critical strengths

Provides a simple algorithm and theoretical proof that can easily be implemented.

Critical weaknesses

This work has significant overlap with prior works with little novelty added. Basically rejection is same as setting weights as zero, so I do not see a big difference between existing methods with weights set as zero. In addition Comparisons to cited baselines are not present, such as Entropy Scheduling and Flexible Entropy Control

Misc. Confidence intervals are not present, so not sure if this is a result of noise Only single-turn, math tasks are evaluated, so not sure if this is generalizable.

Questions for authors

Have you seen similar works? This work seems very close to QwenLong-L1.5’s entropy-controlled rollout masking There are also similar works like Jin et al. on regulating entropy through positive-advantage reweighting, and Wu et al. on quantile advantages that selectively activate positive or negative response updates? These work seem very similar to this work.

2 comments

We regret to learn that you do not like this paper, but we are still grateful for your time and effort in reading it. We hope the following rebuttals may address your concerns, and would truly appreciate your re-consideration if you acknowledge the insight and contribution of this paper.

Q1: Overlaps with Prior Works

We agree that rejection sampling can be expressed as zero weighting. Our contribution is the specific entropy-control rules and its supporting analysis: Entrocraft removes positive-advantage rollouts when entropy is below a scheduled target range, removes negative-advantage rollouts when above it, and retains both within the range.

There are meaningful partial overlaps with prior works, but the control rules and a lot of details differ:

  • [1] includes a trick to switch between positive-only and mixed updates; Entrocraft additionally uses negative-only updates to actively raise entropy.
  • [2] adjusts positive-advantage weights through entropy feedback; Entrocraft directly selects either advantage sign, which make entropy control sufficiently precise so that we can customize entropy curves.
  • [3] selects updates through a reward-quantile baseline tied to query success rates; Entrocraft preserves the existing advantage estimator and selects updates using deviation from the entropy target.
  • [4, 5] use adaptive clipping to influence the entropy schedule; Entrocraft instead uses adaptive rejection sampling to shape the entropy curves. Through visualized results in Fig. 5, we are confident that rejection-sampling-based entropy control is more accurate.

In summary, our novelty claim is around the particular rollout-selection controller, its theoretical analysis for the connection between rollout-selection and entropy, and its evaluation over long-term training. We will include the distinction with prior works to the final version.

[1] https://arxiv.org/abs/2512.12967

[2] https://arxiv.org/abs/2511.05993

[3] https://arxiv.org/abs/2509.22611

[4] https://aclanthology.org/2026.findings-acl.206/

[5] https://arxiv.org/abs/2602.09782

Q2: Confidence Intervals

Thanks for raising this question! We attached the confidence intervals (CI) for the difference between Entrocraft and GRPO in the table below. The raw evaluation results are from our saved repeated responses for each question. We use the paired bootstrap resampling [6] to calculate the 95% CI (re-sampled responses are picked with replacement, and the size of the re-sampled sets are identical to the original.). The improvement from GRPO to our method is statistically stable.

MATH-500AMC-23AIME-25
Entrocraft ACC v.s. GRPO ACC+ 3.7%+ 7.6%+ 5.5%
95% CI[2.906%, 4.600%][4.656%, 11.469%][3.250%, 9.188%]

[6] https://aclanthology.org/W04-3250/

Q3: Only Single-Turn Math Tasks Are Evaluated

We agree that evaluation beyond single-turn math would strengthen the paper. We chose mathematical reasoning as a controlled setting with established RL baselines, which aligns with previous papers' experiment settings [7, 8, 9]. To evaluate Entrocraft on more challenging cases, we already include AIME as the hard benchmarks, and scale up the training steps to see if the performance improvement is persistent (Section 5.3).

Entrocraft operates on policy entropy and estimated advantages, without relying on mathematical content, making application to other policy-gradient settings plausible. We will clarify the scope of our empirical claims and explicitly acknowledge broader task generalization as an open limitation.

[7] https://arxiv.org/abs/2511.15248

[8] https://arxiv.org/abs/2505.22617

[9] https://arxiv.org/abs/2506.01347

I want to clarify regarding "like" or "dislike" of the paper. I don't dislike the paper, I just do not think that it adds significant novelty compared to existing work. As mentioned, rejection sampling is same as zero weight sampling, which has already been published. The overlap is too great to warrant another publication, so I will keep my score.

Reviewer_XxkoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on performance saturation caused by entropy collapse. They propose a new update mechanism based on rejection sampling. When the entropy of the current policy distribution lies within the target interval, the update remains unchanged. When entropy exceeds the upper bound, only positive samples contribute to the update. When it falls below the lower bound, only negative samples contribute. Moreover, this work establishes that the entropy should be scheduled during the training process. Specifically, a linearly decaying schedule is recommended.

Critical strengths

The paper focuses on a critical topic and the algorithm Entrocraft has a simple and practical design.

Critical weaknesses
  1. The finding that advantage and entropy change have opposite signs under certain conditions has been established in "On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models", which is cited by this paper but not discussed sufficiently. The present paper claims that a key distinction is the absence of the assumption that the model parameters are the logits, i.e., θ=z\theta=z. However, the first-order entropy expansion does not rely on parameterization. The previous work uses the logit assumption only to characterize the required condition more explicitly. The theoretical novelty therefore remains unclear. The sequence-level result still assumes that every sampled token's probability change has the same sign as the advantage. The paper does not show when this assumption holds under updates with shared parameters.

  2. EntroPIC already uses an entropy-dependent function to adjust the weights of positive- and negative-advantage samples. Entrocraft’s rejection sampling mechanism can be viewed as a specific choice of the entropy-dependent weighting function, which is also mentioned in EntroPIC. The authors should clarify how their particular choice advances beyond this existing principle. Moreover, in the experimental comparison, Entrocraft uses scheduled entropy targets while EntroPIC uses fixed targets. Therefore, the effects of entropy scheduling and the rejection-sampling mechanism cannot be clearly separated. A controlled comparison should evaluate both methods under matched fixed or scheduled targets.

  3. Configurations of some experiments are ambiguous. There are also inconsistencies that require clarification. Figure 4's caption describes the entropy trajectory in the long-term experiment as linearly decaying, while Table 5 addresses cosine decay for the full-dataset experiment. In addition, Table 2 reports a MATH-500 pass@32 of 93.0% for GRPO + Entrocraft, while the corresponding endpoint in Figure 3c appears below the base-model reference line at approximately 92%. These make the empirical evidence difficult to assess.

  4. Other closely related studies on entropy-guided reweighting and dynamic entropy control are not sufficiently discussed, such as [1-3].

Overall, although the reported improvements are promising, the current manuscript needs clearer positioning and presentation. The authors should explicitly identify the new insights and contributions beyond prior work and organize the paper around them. Moreover, the experimental configurations should be stated clearly, and the inconsistencies also need to be clarified.

[1] Jin et al., Revisiting Entropy in Reinforcement Learning for Large Reasoning Models

[2] Hao et al., Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective

[3] Gai et al., Demystifying Entropy Control in LLM RL Training: Theoretical Analysis and Dynamic Scheduling

Questions for authors

See weakness

2 comments

Thank you so much for taking time to read the details of our paper. The raised concerns are all helpful and constructive for improving the clarity. If you find the responses below sufficient to address your concerns, we truly appreciate your re-consideration for the score.

Q1: Theoretical Results

Q1.1: Parameterization Assumptions

We agree that the first-order Taylor expansion of entropy is independent of the model parameterization. Our discussion did not sufficiently distinguish the assumptions used to characterize the probability changes induced by a policy-gradient update. We will clarify this distinction.

The role of assuming z=θz=\theta arises in the latter step. For example, the policy update in [1] takes the form δz=ηA(ek−p),\delta z=\eta A(e_k-p), corresponding to gradient ascent directly on the logits. For a general differentiable model z=z(θ)z=z(\theta), applying gradient ascent to the model parameters instead gives

\qquad J_z=\frac{\partial z}{\partial\theta}.$$ Thus, setting $z=\theta$ removes the model-dependent matrix $J_zJ_z^\top$. This affects the first-order update direction and consequently the sign of entropy change. With shared parameters across sequence positions, cross-position terms involving $J_tJ_u^\top$ also enter at first order. We therefore agree that the Taylor expansion itself is not a theoretical novelty. The novelty lies in the accurate characterization of entropy change under the induced parameter-space update. [1] https://arxiv.org/abs/2602.03392 ### Q1.2: Derivation of Theorem 2 We acknowledge that the current version of Theorem 2's proof relies on over-strong assumptions for Eq. 12, which undermines the generality of our theoretical results. We therefore update the proof of Theorem 2 as follows: First, we denote $L_y = \log \pi_{\theta}(y | x)$ to be the log-likelihoods. Then, let's assume a question-response pair $(x, y)$ triggers an parameter update $\delta\theta$. The token-prob update can be written as $$\delta p_{t,i} = < \text{grad}_{\theta}p_{t,i}, \delta\theta > + O(||\delta\theta||_2^2) = \eta \hat{A}(x,y) < \text{grad}_{\theta}p_{t,i}, \text{grad}_{\theta}L_y > + O(\eta^2).$$ We take the above equation into Eq. 11 to simplify the expression: $$\delta\mathcal{H} = -\frac{1}{|y|}\sum_{t=1}^{|y|}\sum_{i\in\mathbb{V}} \log p_{t,i} \cdot <\text{grad}_{\theta}p_{t,i}, \text{grad}_{\theta}L_y> + O(\eta^2) = <\text{grad}_{\theta}\mathcal{H}, \delta\theta> + O(||\delta\theta||_2^2) = \eta\hat{A}(x,y) \cdot <\text{grad}_{\theta}\mathcal{H}, \text{grad}_{\theta}L_y> + O(\eta^2).$$ If we denote $\gamma_y = <\text{grad}_{\theta}\mathcal{H}, \text{grad}_{\theta}L_y>$, we can express the relationship between entropy changes and advantages as: $$\delta\mathcal{H} = -\eta\hat{A}(x,y)\gamma_y + O(\eta^2),$$ and the directions of entropy changes is determined by the sign of $\gamma_y$. Then we empirically analyze the value of $\gamma_y$ across a Numina-Math 1K subset. Similar to Table 2(a), we show the results in the table below. Under all three cases, the majority of trajectories have positive $\gamma_y$, validating the entropy changing trend. | Case | Mean $\gamma_y$ | Median $\gamma_y$ | $\gamma_y > 0$ | |---|---:|---:|---:| | Positive-Adv Samples | 3,072.77 | 4,349.92 | 72.2% | | Negative-Adv Samples | 8,482.91 | 7,875.72 | 89.7% | | All Samples | 6,610.17 | 5,511.74 | 83.7% | # Q2: Comparison with EntroPIC We add the direct comparison across methods and entropy schedules in the table below. For annealed curves, we choose the linear decay from [0.6, 0.7] to [0.4, 0.5], and for EntroPIC, we use the mid-point of the annealed range (0.65 to 0.45). EntroPIC tends to anneal the actual entropy values to the specified range, but the entropy control is not responsive enough. Our Entrocraft performs better in terms of entropy control precision and empirical performance. | Method | MATH-500 | AMC-23 | AIME-25 | Step-50 entropy / target | Step-100 entropy / target | |---|---:|---:|---:|---|---| | EntroPIC, fixed target 0.5 | 73.7 | 55.3 | 10.9 | **0.2095 / 0.5000** | **0.3729 / 0.5000** | | EntroPIC, annealed curve | 6.7 | 57.0 | 13.9 | **0.2779 / 0.5510** | **0.3215 / 0.4520** | | Entrocraft, fixed target 0.5 | 77.1 | 64.3 | 14.1 | **0.3927 / [0.4000, 0.6000]** | **0.4193 / [0.4000, 0.6000]** | | Entrocraft, annealed curve | 79.0 | 65.0 | 14.5 | **0.4962 / [0.5010, 0.6010]** | **0.4074 / [0.4020, 0.5020]** |

Q3: Experimental configurations

For the mismatch between Fig. 4's caption and Table 5, we consisently use linear decaying in long-term training experiments. Table 5's "cosine" is a typo.

For the mismatched MATH-500 pass@32 results, we confirm that the GRPO + Entrocraft row in Table 2 is from an older checkpoint with another set of hyperparameters, while Fig. 3(c) shows the latest numbers. The updated row from the latest checkpoint should be:

MethodMATH-500 mean@32MATH-500 pass@32AMC-23 mean@32AMC-23 pass@32AIME-25 mean@32AIME-25 pass@32Avg. mean@32Avg. pass@32
GRPO + Entrocraft78.991.264.894.714.946.752.977.5

Q4: Related Studies

Thanks for mentioning these related works! We will include them in the final version. Specifically, [2] used reweighted postive-advantages which is also guided by entropy, [3] adjusted the token weights based on the token-wise entropy, and [4] introduced an adaptive entropy regularization loss. These prior works have investigated the effect of positive and negative samples to entropy, and their corollaries are also aligned with ours. Our paper takes one step further to the entropy curve control in LLM RL. We precisely maintain linearly decaying entropy schedules with rejection sampling, which achieves the best performance among baselines. Our method is also compatible with any possible entropy curves. If some better entropy curves are observed in the future, we can easily implement them by our method.

[2] https://arxiv.org/abs/2511.05993

[3] https://arxiv.org/abs/2510.10150

[4] https://openreview.net/pdf?id=hq2MPYXAko

Q5: Citation Typos

Thanks for marking this citation typo for the Llama3 paper! We will update the citation as follows:

@article{grattafiori2024llama,
  title={The llama 3 herd of models},
  author={Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others},
  journal={arXiv preprint arXiv:2407.21783},
  year={2024}
}

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Message to AC and Reviewers about the Novelty Assessment

The entropy in LLM RL is a crowded research topic, and we believe it is better to have papers with diverse perspectives presented to the community. You can always find minor technical similarity across different papers of the same topic, but we respectfully disagree that this is the reason for a clear rejection.

We acknowledge the overlaps, including that rejection sampling can be expressed as zero weighting. Our novelty claim concerns the specific entropy controller and its supporting analysis:

Prior workShared idea / existing approachEntrocraft’s distinction
AEPO [1]Switches between positive-only and mixed updates using entropy thresholds.Adds negative-only updates below the target range to actively raise entropy.
Positive-Advantage Reweighting [2]Adjusts positive-advantage weights through entropy feedback.Selects positive-only, negative-only, or mixed rollout updates according to a scheduled entropy range.
Quantile Advantage Estimation [3]Uses a reward-quantile baseline to select updates according to query success rates.Preserves the existing advantage estimator and selects rollouts according to deviation from the entropy target.
Entropy Scheduling [4] and Flexible Entropy Control [5]Adjust clipping thresholds to shape entropy trajectories.Uses adaptive rollout rejection as the mechanism for tracking entropy targets.

In summary, Entrocraft’s contribution combines a three-mode rollout-selection controller, theoretical analysis connecting rollout selection to entropy, and evaluation over long-term training. This provides a distinct approach to shaping customized entropy curves. We respectfully ask that novelty be assessed on this combined contribution, and we will clarify these distinctions in the revised paper.

[1] https://arxiv.org/abs/2512.12967

[2] https://arxiv.org/abs/2511.05993

[3] https://arxiv.org/abs/2509.22611

[4] https://aclanthology.org/2026.findings-acl.206/

[5] https://arxiv.org/abs/2602.09782

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.