- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper addresses a limitation of standard RL training for reasoning: Binary reward gives same credit to correct trajectories, but cannot distinguish the quality of reasoning. The method proposed is BSR, which measures the quality of the reasoning trace by the probability the model generates the target output. Experiment is conducted on math reasoning, agentic search and code-output prediction tasks, and BSR out-performs standard binary reward under different RL approaches and models.
Critical strengths
The experiment is abundant, covering three different task types, multiple model sizes and different RL methods. The performance is also significant despite tested on a single seed.
Critical weaknesses
- The presentation is unclear and hinders understanding. BSR scoring is simple on its own: Replacing the output text with a unified format, and measure the probability of generating the output. But the main text does not explain this at all, and specifically, how to compute in Equation (8) is actually computed. The implementation details are all deferred to Section C.1. In addition, the paper tries to build a complex theory around this already simple and easily understood method, for instance, using a telescoping method in Equation (6) and (7) to explain that intermediate reasoning probability does not need to be computed. The paper tries to place it as a motivation, but the final BSR scoring formula (Equation (5)) is better motivated than the derivation.
- The paper hides key implementation choice from main text. In appendix C.1, the paper mentions that they truncate the concluding answer-formatting block, and replace it with another answering format ("Therefore, the answer is "), which is not the natural generation of the model. The paper calls it "target-leakage prevention", but this actually changes nothing but the final output format, so it is unnecessary. It does not solve the leakage issue neither. From Table 5 we see that both responses indicate at the end that the answer should be 0, so the probability of the final answer cannot reflect the quality of the intermediate reasoning at all. More seriously, it forces the model to predict a sequence that the model is not naturally trained to generate, so it is an unreliable measure of generation quality. The paper fails to justify why such a replacement is implemented and why this key implementation is not put in the main text.
Overall, I admit the empirical performance gain but I highly question the novelty and motivation of the method.
Questions for authors
Can the author explain why the output truncation and replacement in Appendix C.1 is needed? Can the authors post an example of raw response, along with the processed response?