- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates whether max-ent objectives actually improves the diversity and reasoning capabilities of LLMs. The experiments found that these objectives do not actually expand the reachability of the optimized model, and do not improve the accuracy consistently. The results also highlight that various measures of "diversity" might not always be consistent, which calls to attention whether we are optimizing the right objective when it comes to encouraging diverity.
Critical strengths
- The paper is quite insightful and the theoretical characterization is quite informative, particularly the fact that under binary rewards, diff max-ent objectives can be understood through a single ratio weighing incorrect v correct responses.
- The experiments are thorough in accounting for all sources of variation from random seeds, init, and evaluation settings before interpreting performance differences.
- This is quite interesting negative result that challenges the assumption about encouraging diversity in solution, all levels/measures of diversity, and their significance.
Critical weaknesses
- Main thing is that there is only one model (Qwen2.5-7B) evaled on 90 AIME problems, which is a limited experimental setting. I'm curious how the observations might change or stay consistent with the size of the model, as bigger models might perform better on these tasks, and also become more confident (lower entropy);
- Are tasks with binary rewards like math truly align with the type of diversity that max-ent objective is optimizing towards? Would be good to actually look into the difference in the generated solutions as case studies rather than simply relying on LLM judge. It would be interesting to understand whether these objectives actually induce alternative reasoning strategies and novel approaches that didn't pan out versus just rewording an incorrect solution.
- Also the semantic diversity evaluation is not sufficiently robust, since it is only relying on a single LLM judge. Need to try different judges to avoid biases.
- The paper lacks more fine-grained analysis, for example, does semantic diversity actually correlate with specific-problem solving success? Like do problems with more distinct correct reasoning strategies have higher pass@k? also given the original problem pass rate distribution, which types of problems actually benefit more from max-ent, do the higher-pass-rate ones become more diverse or do the lower ones become more diverse (i.e. do models try more creative strategies for something that it already know how to solve pretty well or for something that it can barely solve), is this related to the rollout entropy of samples from the base model on those problems respectively?
- The explanation for the performance collapse for the strongest intervention seems to be lacking.
- Also there seems to be a disconnect between what is shown in the theory part, which is really relevant to what is shown experimentally?
Questions for authors
Already included above