- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper propose a R-zero style self-play framework that includes two key differences: 1) the proposer is replaced by a store of generator programs that generate problems and correct answers directly; 2) the solver evolves these programs with several heuristics centered on a metric called RQ (reasoning quality), which is a slight change to existing learnability signals. Results show that the propose pipeline improves over R-zero on several mathematical benchmarks, on average, slightly.
Critical strengths
The proposed pipeline is relatively complete engineering-wise, and similar ideas could be more broadly applied to other domains that can benefit from generator programs with sufficient human heuristics are priors.
Critical weaknesses
The major weakness is with the scientific contributions of this paper, as I do not clearly see how this advances the communities' understandings. Individual components, namely using program generators for math questions to produce synthetic training signals, the RQ design, and the evolution algorithm, are mostly methods broadly applied in previous works with minor changes. I do like the overall formulation of evolving the program stores as an replacement of the proposer in the original R-zero framework, yet there is no convincing evidences or ablation studies on whether the mutation itself actually contributes to the gains. As a result, the gains could largely be from models getting more consistent (via the program-generated questions) and moving slightly to regions that they were good at in the first place (as shown by the RQ selection beating random). I do not see any clear evidence that any real and sustainable evolution from the proposed framework is possible.
Questions for authors
Please see the weaknesses.