- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper observes that test-time reinforcement learning, which rewards trajectories that match the majority vote, can fail when the answer space is narrow, such as multi-choice questions. Therefore, they propose PROSE, process-reward-guided self-training. It rewards the trajectory that has their reasoning process correct instead of the answer, with an auxiliary answer format constraints. More specifically, it uses the lowest process score as the overall reward of the whole reasoning trajectory. By adopting another process reward model, PROSE improves the performance of LLMs on multi-choice medical questions.
Critical strengths
- The paper is well motivated: It observes the failure modes of existing test-time RL, and then propose a solution that targets this issue.
- The method outperforms the baseline methods: A clear performance gain is shown in the experiment section.
- The authors compare the design choices of using mean or minimum process rewards as the training signal, and demonstrate that min consistently outperforms the mean solution. This can be an useful strategy for future process reward reinforcement learning.
Critical weaknesses
- This paper adopts an existing process reward model and applied it to their own pipeline. The process reward model, Med-PRM, already shows it superior performance on various QA benchmarks. Although Med-PRM was designed for inference time selection, it still remains a question if the performance gain of PROSE comes from its own design or the performance of Med-PRM.
- The paper works only with multi-choice questions, which are known to be prone to spurious signals in reinforcement learning. More scenarios, such as free form medical question answer, should be studied to fully verify the hypothesis in this paper.
- The results of the paper is based on the evaluation performance. More details, such as the process reward scores and response lengths, should be shared.
Questions for authors
- The improvement of PROSE over baseline is not observed at the scale of 1.7B, and it starts to show only at and after 4B model sizes. Do you think this can be related to reasoning lengths or quality?