- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies distributional biases in reasoning post-training and inference scaling using a Markov-chain framework. It theoretically analyzes the tendency of RLVR and reward-guided inference to favor easy reasoning trajectories and proposes a Doob -transform-based process reward.
Critical strengths
The paper addresses an interesting question about reasoning diversity in post-training. The unified theoretical framework is appealing and provides an interesting perspective on the role of exploration in preserving rare reasoning trajectories.
Critical weaknesses
The main conclusion appears to depend heavily on the modeling assumptions.
Definition 2.2(iii) assumes that the probability of a valid CoT being correct is proportional to its likelihood under the base model. This directly implies that easy CoTs have higher expected rewards than hard CoTs.
Consequently, the tendency of reward-maximizing algorithms to favor easy CoTs seems to be partly built into the assumptions rather than emerging independently from the analysis.
While this may be reasonable for a simplified theoretical model, it is unclear whether the main conclusions would hold without this assumption.
Questions for authors
See weakness.