- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why reinforcement learning with verifiable rewards lowers a model's entropy, by looking at the weight change it makes. The authors keep the size and structure of the RLVR update fixed but randomly change its direction. The learned direction lowers entropy, while random directions barely do, so direction is what matters. They then show that a random search around the SFT model can reach accuracy close to RLVR's without lowering entropy, suggesting that the accuracy gain does not require the entropy drop.
Critical strengths
The main control rotates the learned update while keeping its size, structure, and singular values fixed, so any difference in entropy can only come from direction. The result holds across two model families, several settings, and both fixed and model-generated text. This matters because it makes the central comparison easy to interpret and gives a reusable tool for studying what weight updates do.
Critical weaknesses
None
Questions for authors
Does the analysis also generalize to other RL algorithm like RLHF or DPO?
3 comments
Thank you for the detail rebuttal. I would keep my positive score.
reply to Reviewer nnWW's comment
Dear Reviewer, Many thanks for your support. We are glad that our rebuttal was appreciated!
Author Response
Thank you for your review, and to answer your question we used the Olmo-3-7B-DPO checkpoint to test for these effects. DPO shows a much weaker entropy drop, albeit hinting at a similar connection with orientation. We have also substantially rewritten and updated the paper to address the points raised in the reviews.