- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper propose SGRC, a new rewarding approach for LLM training. The standard GRPO training rewards sparsely, and thus, fails to recognize and reward equivalent expressions or formatting variants (they get zero rewards). The proposed method addresses this problem by having a repair system to let the system realize other possible correct representations, resulting in denser rewarding. The experiment is demonstrated using a subset of MathV360k.
Critical strengths
- The paper presents a well-motivated method to make a better reward to train LLM models on structured reasoning tasks. It points out a suboptimal mechanism of the popular GRPO method when it has incorrect outcomes from verifiers. The paper reframes reward repair as hidden-partition recovery problem, which is insightful. The method is presented consistently and empirical results are delivered.
- The quality of presentation is good overall, offering a fair clarity of explanations of essential concepts and information.
Critical weaknesses
- The experiments are solely based on a single dataset, even it is a subset of MathV360K dataset. Although it is clarified that it contains image-question pairs from different sources, the result on a single dataset would not be enough to support empirically the claim introduced in the paper. As GRPO is successfully applied and demonstrated in various tasks and domains, it would not be fair to claim that the proposed method outperformed GRPO with a single dataset. It is not clear if the proposed method enhances the performance universally in different scenarios, or it worked well specifically on this dataset.
Questions for authors
How the method will scale further tasks and datasets? Would it enhance the performance overall, beating the GRPO baseline?