- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MathIdeaBench, a benchmark for evaluating mathematical research ideas generated by LLMs. It provides a dataset constructed from arXiv papers and their follow-up works, together with MathIdeaJudge for evaluating proposal soundness, feasibility, novelty, and topical diversity.
Strengths
Critical strengths
-
The paper constructs a reasonably large dataset covering multiple mathematical subfields and provides an automated pipeline that can potentially refresh the benchmark over time.
-
The evaluation prompts and rubrics are described in detail, and the MSC-based diversity metric is more interpretable than generic embedding similarity.
Critical weaknesses
-
Limited novelty. The framework largely applies existing idea-benchmark methodology to mathematics. The math-specific rubrics and MSC metric are useful but incremental.
-
Limited model coverage. Only two proposer models and one primary judge model are evaluated, with no frontier proposer included.
-
Questionable human baseline. The baseline proposals are reconstructed by an LLM from completed follow-up papers rather than directly written by mathematicians.
-
Insufficient judge validation. The human study covers only 24 proposals from four source papers, which is too small to validate the judge across mathematical domains.
Questions for authors
same as weaknesses
We thank the reviewer for the constructive feedback.