acceptodds
Under review as a conference paper at ICLR 2027

Beyond Max-Ent Targets: Disentangling Entropy and Diversity in LLM Post-Training

Abstract

Entropy-regularized trajectory-balance objectives have been proposed for LLM post-training as an alternative to GRPO-like objectives. By matching a distribution over trajectories rather than concentrating only on the highest-reward ones, they are expected to preserve diverse solutions, expand problem reachability, and improve accuracy on difficult tasks. We test these expectations under binary verifiable rewards. To expose the available target-design space, we expand the classical Shannon entropy with -exponential targets in a common Tsallis family, used here as an analytical and experimental degree of freedom. We show that group-normalized binary rewards reduce every reward tilt to a single incorrect-to-correct reweighting profile . This reduction reveals a coupling between support and scale, compresses apparently different target constructions into a narrow region, and provides a direct criterion for choosing trajectory-balance parameters. In the experiments, we separate nominal target geometry from learned-policy behavior and evaluate the latter after quantifying procedural variation. Although our sweep changes the -profile substantially, its behavioral effects reveal a clear separation between nominal target geometry and learned-policy properties: reachability and accuracy mostly remain within measured variation, while the strongest intervention reduces both. A matched GRPO comparison shows that Max-Ent objectives need not retain higher token entropy, even as an LLM judge reproduces the previously reported semantic-diversity effect. We thus identify nominal Max-Ent structure, token-level entropy, and semantic solution diversity as empirically distinct axes, showing that theoretical target changes cannot substitute for behavioral validation.

Primary area:
reinforcement learning
Keywords:
reinforcement learning from verifiable rewards, LLM post-training, GFlowNets, trajectory balance, entropy regularization, Tsallis entropy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xq89OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether max-ent objectives actually improves the diversity and reasoning capabilities of LLMs. The experiments found that these objectives do not actually expand the reachability of the optimized model, and do not improve the accuracy consistently. The results also highlight that various measures of "diversity" might not always be consistent, which calls to attention whether we are optimizing the right objective when it comes to encouraging diverity.

Critical strengths
  1. The paper is quite insightful and the theoretical characterization is quite informative, particularly the fact that under binary rewards, diff max-ent objectives can be understood through a single ratio weighing incorrect v correct responses.
  2. The experiments are thorough in accounting for all sources of variation from random seeds, init, and evaluation settings before interpreting performance differences.
  3. This is quite interesting negative result that challenges the assumption about encouraging diversity in solution, all levels/measures of diversity, and their significance.
Critical weaknesses
  1. Main thing is that there is only one model (Qwen2.5-7B) evaled on 90 AIME problems, which is a limited experimental setting. I'm curious how the observations might change or stay consistent with the size of the model, as bigger models might perform better on these tasks, and also become more confident (lower entropy);
  2. Are tasks with binary rewards like math truly align with the type of diversity that max-ent objective is optimizing towards? Would be good to actually look into the difference in the generated solutions as case studies rather than simply relying on LLM judge. It would be interesting to understand whether these objectives actually induce alternative reasoning strategies and novel approaches that didn't pan out versus just rewording an incorrect solution.
  3. Also the semantic diversity evaluation is not sufficiently robust, since it is only relying on a single LLM judge. Need to try different judges to avoid biases.
  4. The paper lacks more fine-grained analysis, for example, does semantic diversity actually correlate with specific-problem solving success? Like do problems with more distinct correct reasoning strategies have higher pass@k? also given the original problem pass rate distribution, which types of problems actually benefit more from max-ent, do the higher-pass-rate ones become more diverse or do the lower ones become more diverse (i.e. do models try more creative strategies for something that it already know how to solve pretty well or for something that it can barely solve), is this related to the rollout entropy of samples from the base model on those problems respectively?
  5. The explanation for the performance collapse for the strongest intervention seems to be lacking.
  6. Also there seems to be a disconnect between what is shown in the theory part, which is really relevant to what is shown experimentally?
Questions for authors

Already included above

Reviewer_GBTcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether Maximum-Entropy (Max-Ent) reinforcement learning objectives are able to preserve solution diversity, expand problem reachability, and increase accuracy on difficult tasks in LLM post-training under Reinforcement Learning with Verifiable Rewards (RLVR). The authors show that binary right/wrong grading mathematically simplifies the complex nominal target distributions into a single scalar factor (the weight given to incorrect versus correct answers). The authors also established a baseline for experimental noise, finding that minor changes like formatting or random seeds can shift results a not-negligible amount. When they adjusted target parameters, the expected improvements in reachability and accuracy didn't consistently rise above this noise, and extreme changes severely hurt performance. Finally, the paper reveals that training with a Maximum Entropy loss doesn't automatically increase token-level randomness. In fact, compared to a standard baseline, these checkpoints actually showed lower token-level entropy and less variety in final answers, even though an external judge rated their high-level problem-solving strategies as more diverse.

Critical strengths

The paper brings clarity to the different interpretations of diversity in LLM post-training. By showing that standard GRPO achieves higher accuracy and more random token generation, while Max-Ent models simply look more diverse to an LLM judge, the paper clearly proves a key point: semantic diversity, answer diversity, and policy entropy are distinct in practice and cannot be used interchangeably. Furthermore, by measuring how much simple procedural changes might impact the final results, the authors establish a quantitative bound for measuring improvements across different RLVR techniques.

Critical weaknesses
  1. The authors mention that in practice, the surrogate in (2) is used for training instead of the nominal target in (1). This undermines the theoretical contribution of this work as the main theoretical derivations are for the nominal target. To me, it seems that the experiments mainly acknowledge the flaws of the surrogate objective. That is also a valuable result but does something different than confirming the theoretical contribution of this work.

  2. The mismatches in some parts of the experiments weaken the final conclusions. In particular:

  • Claims that were made from q=1.9q = 1.9 and GRPO results are hard to trust because these models use different checkpoints compared to the main group. They were also trained on completely different GPUs which might alter the final performance due to the different low-level library implementations of different GPUs.
  • Furthermore, LLM judge evaluated the earlier exploratory sweep with smaller test batch instead of the main 90-problem test batch where GRPO actually performed better. While scaling the LLM judge tests might be costly, because the paper itself proves that small setup changes can hugely impact performance, these mismatched setups might weaken the final conclusions.
  • Finally, as the authors acknowledge in Section 5.6, the semantic judge is model-dependent. The experiments are done only using a single model which might make the final results too noisy to analyse.

Minor Weaknesses:

  • Because of heavy reliance on cross-referencing, the current organization of the experiments and the related appendix is too difficult to follow. Especially the numerous tables in the appendices would be much easier to follow by minimizing the frequent cross-referencings and explicit mentioning of the corresponding subsection in the caption of tables. For example, subsection 5.1 refers to appendix I for more complete results but the text in appendix I never explicitly points the reader to the corresponding Tables.
  • The first line of Appendix A includes repetitions ("Equation equation 1" and "Equation equation 2").
Questions for authors
  1. Given that the experiments optimize the surrogate loss in Equation (2) rather than the nominal target in Equation (1), can you provide any empirical evidence to show whether the negative behavioral results are a failure of Max-Ent targets, or simply a result of the length-normalized surrogate failing to optimize the target properly?
  2. Why were the q=1.9q = 1.9 and GRPO models trained from different starting checkpoints, for different step counts, and on different hardware? Was it because of the unavailability of the hardware?
  3. Does the LLM-judged semantic variety advantages over GRPO still hold when evaluated on the horizon-matched 400-step checkpoints from the controlled 90-problem AIME protocol where GRPO retained higher token-level entropy? To avoid the costs of judging all trajectories, could you evaluate a smaller random subset of samples (e.g., n=8n=8) per problem from the controlled checkpoints to clarify if the semantic result persists?
Reviewer_djFmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether maximum-entropy training targets can improve language-model accuracy, problem-solving coverage, and diversity under binary rewards. The authors introduce qq-exponential targets to expand the target-design space and reduce within-group reward transformations to an incorrect-to-correct weighting ratio. The experiments show that, despite substantial changes in target weights, changes in accuracy and problem-solving coverage mostly remain within training and evaluation variability, while assessments of token entropy and semantic solution diversity may not align.

Critical strengths
  • The authors reduce target transformations under binary rewards to an incorrect-to-correct weighting ratio, providing a unified comparison of different targets' reweighting effects in this particular setting.
  • The paper provides empirical observations showing differences between assessments of token entropy and solution diversity.
Critical weaknesses
  • Prior studies have shown that maximum entropy can mislead policy optimization and that entropy minimization can benefit LLM reasoning. The paper's additional insights beyond these precedents remain unclear, and its discussion of related work could be strengthened.
  • The core controlled analysis focuses on one model and 90 AIME problems, leaving the applicability of the main findings to other settings unclear.
Questions for authors
  • An error-type analysis at q=1.9q=1.9 could be added to examine whether answer-extraction failures and output truncation affect performance.
  • The length normalization in Eq. (2) changes the target distribution specified in Eq. (1). The authors might include a control that preserves this target to reassess the accuracy and problem-solving coverage results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.