- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RAWR for generating step-level supervision in verifiable domains without rollouts. For each question, RAWR samples multiple CoT and uses validator to judge the correctness of the final output. For every reasoning prefix, RAWR computes the score between the most supported correct answer and the most supported wrong answer. And the final score will be thresholded into binary step labels and be used to train the PRM. Comprared with manual labelling and rollout-based labelling, RAWR achieves comparable results regarding best-of-K with lower costs.
Critical strengths
- Clear motivation and practical method design. For domains with applicable final-answer validator, the proposed method is effective.
- The experiments are thorogh and ablation analysis is quite detail.
- The writting is overall clear and easy to follow.
Critical weaknesses
- The 0-1 label definition is derived from threshold, which might leads to model overconfidence. These scores might be better suitable for ranking of the trace. Is there any ablation study about this alternatives like soft labels?
- There might be differences in trace steps. The granularity of the "STEP" definition in different traces might be different. The author might provide more evidences showing the method robustness given different step length/step budget etc.
Questions for authors
- The likelihood score is sum of all tokens, which might be sensitive to the trace length. It would be interesting to see how the answer length bias would affects the model performance.
- How to define threshold for new problem domains? How sensitive is the method to this parameter?