acceptodds
Under review as a conference paper at ICLR 2027

MathIdeaBench: A Benchmark For Mathematical Research Idea Generation

Abstract

As large language models (LLMs) continue to advance in mathematical reasoning and progress from problem solving to research discovery, rigorously evaluating their capacity for research ideation becomes vital. However, measuring the quality of mathematical research proposals remains an open challenge due to its technical and subjective nature. In this work, we introduce MathIdeaBench, the first benchmark designed to measure the quality of LLM-generated mathematical research proposals against a human baseline. MathIdeaBench consists of (1) an automated data generation pipeline which produces a refreshable evaluation dataset, and (2) an LLM-based evaluator, MathIdeaJudge, which scores proposals using detailed evaluation rubrics designed by experts. Furthermore, we propose a tree-based diversity metric for proposals based on mathematical subject classification. Our experiments show that compared to ideas from human-authored publications, LLMs tend to generate proposals which are less sound, less feasible and less diverse but conceptually deeper and more original.

Primary area:
datasets and benchmarks
Keywords:
AI for Mathematics, Research Idea Generation, Large Language Models, Benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_bneNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MathIdeaBench, a benchmark for evaluating mathematical research ideas generated by LLMs. It provides a dataset constructed from arXiv papers and their follow-up works, together with MathIdeaJudge for evaluating proposal soundness, feasibility, novelty, and topical diversity.

Strengths

Critical strengths
  1. The paper constructs a reasonably large dataset covering multiple mathematical subfields and provides an automated pipeline that can potentially refresh the benchmark over time.

  2. The evaluation prompts and rubrics are described in detail, and the MSC-based diversity metric is more interpretable than generic embedding similarity.

Critical weaknesses
  1. Limited novelty. The framework largely applies existing idea-benchmark methodology to mathematics. The math-specific rubrics and MSC metric are useful but incremental.

  2. Limited model coverage. Only two proposer models and one primary judge model are evaluated, with no frontier proposer included.

  3. Questionable human baseline. The baseline proposals are reconstructed by an LLM from completed follow-up papers rather than directly written by mathematicians.

  4. Insufficient judge validation. The human study covers only 24 proposals from four source papers, which is too small to validate the judge across mathematical domains.

Questions for authors

same as weaknesses

1 comment

We thank the reviewer for the constructive feedback.

  1. We acknowledge the limited model coverage due to high inference costs for our detailed evaluation, and we remark that the proposed framework readily applies to all frontier models.
  2. We emphasize that while the reconstructed proposals were extracted using LLMs, we used the frontier model GPT-5.6-Terra with reasoning effort set to high. The authors believe that such a strong model is reasonably reliable to automate the baseline construction process, which concerns only summarization and rephrasing.
  3. Evaluating mathematical research proposals is a highly specialized and demanding task, which makes large-scale recruitment challenging. To ensure rigor despite a smaller sample size, we recruited reviewers with diverse expertise, carefully matched them to relevant proposals (Appendix D), and had each participant spend roughly three hours grading multiple proposals in depth. Because each evaluation reflects substantial and domain-specific effort, we believe our findings offer meaningful insights, though we acknowledge that a larger-scale study remains an important direction for future work.
Reviewer_xy8hOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MathIdeaBench, a benchmark for evaluating large language models’ ability to generate mathematical research ideas. The benchmark is constructed from source papers and their subsequent citing papers, with the latter serving as a human-research baseline. The authors also develop MathIdeaJudge, an LLM-based evaluator that assesses proposals along dimensions of soundness, feasibility, and novelty, and propose a topical diversity metric based on the MSC2020 classification system. Experiments with two idea-generation models suggest that their proposals generally lag behind published research in soundness and feasibility, although some models achieve competitive or higher novelty scores. A small human evaluation provides preliminary evidence for the alignment between MathIdeaJudge and human judgments. Overall, the paper offers a structured benchmark for an important task, but the reliability of its evaluation framework and the validity of its comparisons with published research remain key questions.

Critical strengths
  1. Meaningful and promising research direction. Evaluating the ability of large language models to generate valuable mathematical research ideas is an interesting and worthwhile direction. Mathematics offers certain opportunities to assess the correctness and feasibility of proposed ideas through relatively direct checks, such as examining definitions, assumptions, simple examples, and potential counterexamples. This paper takes a meaningful step toward systematizing such evaluations and assessing models’ ability to generate mathematical research ideas.
  2. A structured framework for evaluating research ideas. Rather than focusing solely on novelty, the paper decomposes evaluation into multiple dimensions, including soundness, feasibility, and novelty, with more fine-grained criteria for each. This structure can make assessments more systematic and model comparisons more informative, providing a useful starting point for future research on the automated evaluation of mathematical research ideas.
Critical weaknesses
  1. Limited methodological novelty. The overall workflow of this paper is relatively straightforward, consisting primarily of constructing a dataset from existing papers, using LLMs to generate research ideas, and evaluating them against predefined rubrics. Although the research direction is meaningful, the paper does not sufficiently establish how its methodology makes substantial progress beyond existing approaches to research idea generation and evaluation. Consequently, the contribution lies primarily in integrating benchmark construction and evaluation procedures rather than introducing a clearly novel methodology.
  2. Limitations in the design of the novelty evaluation mechanism. The paper uses an LLM-as-a-Judge approach to assess the originality of research ideas relative to existing mathematical literature, prompting the model to actively search for related work. However, delegating both literature retrieval and originality assessment to an LLM does not ensure comprehensive and systematic coverage of relevant literature, making the evaluation results susceptible to limitations in the model’s retrieval and reasoning capabilities. Assessing the originality of mathematical research ideas often requires extensive and in-depth literature investigation, and relying solely on LLM-driven retrieval may overlook substantially relevant prior work. A more rigorous design would explicitly incorporate systematic literature retrieval and relevance screening into the evaluation workflow, allowing the evaluator to assess originality based on the retrieved evidence rather than delegating the entire retrieval process to the LLM.
  3. Insufficient experimental validation and lack of comparisons with existing research idea generation frameworks. The paper evaluates only two LLMs and does not compare them with existing agent-based frameworks for mathematical research idea generation. Consequently, the experiments do not sufficiently demonstrate whether the benchmark can effectively evaluate more sophisticated, multi-step research idea generation systems, nor do they establish its practical value relative to existing approaches. Furthermore, the authors use only a single LLM as the primary evaluator, meaning that the conclusions may be influenced by the capabilities and evaluation biases of that particular model.
  4. Limited human validation and insufficient evidence for the validity of the evaluation framework. The human evaluation involves only 24 research ideas and seven reviewers, which is insufficient to establish that MathIdeaJudge is reliable across different mathematical fields and types of research ideas. In particular, assessing the feasibility and originality of mathematical ideas often requires in-depth expert judgment, and the limited human evaluation does not provide sufficient evidence to validate the agreement between automated scores and expert assessments. Therefore, the paper’s claims regarding the reliability of the evaluation framework require more extensive experimental support.
Questions for authors
  1. Novelty Evaluation and Literature Retrieval. The novelty evaluation relies on an LLM judge that is prompted to retrieve related work and assess the originality of generated ideas. Could the authors clarify how the retrieved literature is obtained and verified, and whether the judge has access to a predefined corpus of relevant mathematical papers or relies entirely on its own retrieval capabilities? How do the authors mitigate the risk of overlooking relevant prior work, which could lead to overestimating the novelty of generated ideas?

  2. Scope and Generalizability of the Benchmark. The experiments evaluate only two idea-generation models. Could the authors clarify whether MathIdeaBench is intended primarily to evaluate individual LLMs or whether it is also suitable for evaluating more complex, multi-step research idea generation agents? If the latter, what aspects of the benchmark design support such evaluations? Furthermore, could the authors explain why existing agent-based research idea generation systems were not included in the evaluation? Comparing against such systems would help establish the benchmark’s applicability to more sophisticated research workflows and its value relative to existing approaches.

  3. Reliability of Human Evaluation. The human evaluation involves 24 research ideas and seven reviewers. Could the authors provide more details about the agreement between reviewers and the agreement between MathIdeaJudge and individual human reviewers, particularly for novelty and feasibility? Given the limited evaluation sample, how do the authors interpret the strength and generalizability of the reported alignment between automated and human judgments?

1 comment

We thank the reviewer for the detailed feedback.

Response to Q1: The LLM judge evaluates novelty with a web search tool and its parametric knowledge. It is not possible to construct a static and reasonably small pool of candidate papers in advance given the open-ended nature of proposals. We make sure that no contamination occurs by eliminating retrieved papers published after the source paper. More accurate verification can be achieved by employing a dedicated literature review pipeline. While this can easily be integrated into our pipeline, we had to leave this out due to funding constraints. We also comment that exclusion of relevant prior work may also lead to higher novelty scores for follow-up proposals, and we cannot conclude that this risk disproportionately affects one class over the other without extensive evaluation.

Response to Q2: The evaluation is designed to be largely agnostic to the idea generation pipeline by focusing on the important aspects of the generated proposals. MathIdeaJudge can always give numeric scores and qualitative feedback on the feasibility, novelty, soundness and topical diversity of generated proposals. The constraint for the human baseline comparison to apply is that the input to the ideation system must be one paper which has sufficiently many follow-up papers. The inclusion tools for literature review and theorem proving does not affect the applicability of our benchmark. The authors agree that the evaluation can benefit from the inclusion of agentic systems but have to leave them out due to prohibitively expensive inference costs.

Response to Q3: The mean scores given by human reviewers and the GLM-based judge are given in the following table. For all dimensions except example-checker feasibility, the two groups are not statistically significantly different under two-sided Wilcoxon signed-rank tests. This means that human evaluators and the GLM-based judge assign similar scores on average, which means that the LLM judge could be used as a proxy for expensive human reviewers. Possible reasons why GLM gives lower example-check feasibility scores include 1) genuinely identifying counterexamples which human reviewers missed and 2) incorrect judgment by the LM due to limited exploration depth.

CriterionHuman meanGLM meanGLM − humanHolm-adjusted pResult
Well-defined terms3.2363.083−0.1530.545Not significant
Sound assumptions3.3123.125−0.1881.000Not significant
Supporting evidence2.4582.5830.1251.000Not significant
Conceptual depth2.2432.5000.2570.545Not significant
Originality2.3962.292−0.1041.000Not significant
Example-check feasibility3.2712.500−0.7710.012GLM significantly lower
Obstruction feasibility3.3963.167−0.2291.000Not significant

The mean absolute human-human disagreement (HH) and the mean absolute human–GLM disagreement (HL) is given in the following table (same as Fig 4). Except for well-defined terms and originality, a human reviewer does not give significantly different judgment from another human reviewer than from the GLM judge. The greater variation among human reviewers in the two dimensions might be attributed to the reviewers’ varied background, given that the proposals may not lie in the areas of their strongest expertise.

CriterionMean HHMean HLHL − HHHolm-adjusted pResult
Well-defined terms1.0560.681−0.3750.021HL significantly lower
Sound assumptions0.8750.9100.0351.000Not significant
Supporting evidence1.1670.875−0.2920.245Not significant
Conceptual depth0.9580.785−0.1740.408Not significant
Originality1.2080.854−0.3540.040HL significantly lower
Example-check feasibility1.0971.2150.1181.000Not significant
Obstruction feasibility0.6810.9240.2430.865Not significant

Evaluating mathematical research proposals is a highly specialized and demanding task, which makes large-scale recruitment challenging. To ensure rigor despite a smaller sample size, we recruited reviewers with diverse expertise, carefully matched them to relevant proposals (Appendix D), and had each participant spend roughly three hours grading multiple proposals in depth. Because each evaluation reflects substantial and domain-specific effort, we believe our findings offer meaningful insights, though we acknowledge that a larger-scale study remains an important direction for future work.

Reviewer_d3kXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper builds a benchmark for testing LMs' capabilities in generating math research idea. The grader is also an LM judge, with ruburics designed by human experts. A human-written idea baseline is provided. Results show that, according to the LM judge, compared to human-written ideas, LM-generated ideas are less sound, feasible, diverse, but conceptually deeper and more original.

Critical strengths

None

Critical weaknesses
  1. No techincal novelty. Similar automated idea generation pipelines and ruburics have been studied in many prior work. But this is not a key weakness for a benchmark paper.
  2. Limited benchmark quality evaluation. The paper relies on a prompted-LM as the major grader. Thus, whether the results can be trusted highly depends on the judge's quality. The paper indeed tries to demonstrate the quality of the LM judge by showing that human-human agreement is not significantly higher than human-LM agreement. But it's important to note that human experts often disagree with each other when judging research idea qualities. Is any relatively objective experiments we can run to get more confidence about the LM judge? For example,
  • Ablations on prompts/rubrics. Do they really matter? Or in fact that with frontier LMs (e.g. Fable 5.1, Opus 5.5, GPT Astra), LMs have stable performance when judging research ideas.
  • Can the LM judge accelerate math research? Given a math research problem, we can iteratively propose and execute ideas, w/ or w/o using the LM judge to pick ideas. If the LM judge is good, it should lead to objectively faster, more successful math research.
Questions for authors

See weaknesses

1 comment

We thank the reviewer for the constructive feedback and suggestions.

  1. We emphasize that while several idea generation benchmarks exist, none of them can be properly applied to mathematics due to their respective flaws. In particular, the idea generation processes in Guo et al. (2025) and Qiu et al. (2025) depend on influential citations within the target paper. This approach to construct a counterfactual target imposes unnecessary constraints and is unnatural in the inherently forward-looking research process. We address this limitation by comparing generated proposals with proposals extracted from human-written follow-up papers instead. Other rubrics-based benchmarks such as Chen et al. (2026a) fall short because their generic grading prompts are ill-adapted for reliable mathematical research idea evaluation. We address this limitation by carefully designing a collection of grading dimensions used by mathematicians. Neither of the aforementioned contributions is present in the literature.
  2. We point out that evaluating research ideation is inherently subjective due to differences in prior knowledge, technical capabilities and tastes. This work aims to provide a comprehensive evaluation framework which eliminates some subjectivity by constraining the scope of each important evaluation direction. The rubrics in MathIdeaJudge are designed by mathematicians to capture evaluation dimensions which are relevant. We emphasize that the goal of MathIdeaBench is not just to accelerate research, but to do so in a controlled and transparent manner so that mathematicians are well-informed. In this light, ablations on prompts/rubrics are irrelevant because these prompts/rubrics are precisely what defines performance, without which no systematic comparison is possible. As for whether the LM judge can accelerate math research, we have indeed identified this as one important future direction. As the main contribution of this paper is to introduce the detailed grading schemes, this is beyond the scope of the current work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.