- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a benchmark HorizonMath with an automatic verification framework for evaluate mathematical discovery ability of LLM. The method is demonstrated and validated through the process of verifying the novel solutions of GPT. The work is significant for AI applications in mathematic research fields.
Critical strengths
- The benchmark consists of 113 predominantly unsolved problems which are selected carefully and resistant to memorization so that the mathematical reasoning ability of an LLM can be evaluated strictly and correctly.
- The automated verification pipeline and the associated verification scripts using the generator-verifier gap are provided in open-source so that the verification can be performed efficiently and conveniently.
Critical weaknesses
None
Questions for authors
None
Thank you!
Dear reviewer, thank you for your positive review! We're glad you found the problem selection, resistance to memorization, and open-source verification pipeline valuable. These were our main goals in creating HorizonMath.
Since submission, we have added a new set of 15 unsolved existence problems that bring the eval to 128 problems in total. The problem statements and their verifiers are in the anonymized repo. We are continuously growing HorizonMath as a community resource and unsaturated evaluation for frontier models.
Thank you again!