- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper analyzes why GRPO-based updates can fail on single-token binary classification tasks even if they modify the sampling procedure to contain at least one correct and incorrect answer in each group, while SFT succeeds. They show that changing the update weighting (removing zero-centering, allocating more weight to questions the model gets wrong) substantially improves learning. Additionally, they show that linear probes can recover label information despite near-chance direct answers.
Critical strengths
S1: The paper distinguishes decodability (whether a linear classifier can recover label information from hidden states) from learnability (the model can learn to answer correctly) S2: The paper compares group-relative advantage formulations and update weighting rules under matched sampling procedures
Critical weaknesses
The paper is difficult to follow due to insufficient motivation and dense, imprecisely defined terminology. The implications of the single-token binary classification setting and its differences from standard generative GRPO could be explained more clearly.
The controlled experiments use a fixed training budget without demonstrating convergence. Longer runs and learning-rate sensitivity checks are needed to distinguish a persistent limitation from slower optimization.
The broader significance for RLVR remains unclear. Although the paper includes standard GRPO comparisons, its strongest controlled evidence concerns single-token binary classification under forced sampling. Further evidence is needed to establish whether the identified update sensitivity extends to learning multi-step reasoning.
Questions for authors
Do longer runs or learning-rate sweeps close the gap between the centered update and SFT? How do training and held-out correctness probabilities evolve for the freely sampled answers? Which conclusions do the authors expect to extend beyond single-token binary classification to multi-token reasoning?