acceptodds
Under review as a conference paper at ICLR 2027

MathDB: A Dynamic Evaluation Platform for Frontier AI in Research Mathematics at Scale

Abstract

As AI systems advance at mathematical reasoning, static benchmarks saturate quickly, and repeatedly constructing expert-curated replacements is costly and hard to scale. We introduce MathDB, a continuously updated, structured database of more than 80,000 research-level mathematical problems with literature provenance, domain labels, status histories, verification evidence, and a knowledge graph of mathematical relationships. MathDB grows through automated literature ingestion and community contributions, and its structured metadata supports the construction and later refresh of benchmark releases without resourcing an entirely new problem set from scratch. As a first demonstration of this framework, we construct MathDB-Bench from 100 tasks adapted from 981 resolved MathDB records, spanning five mathematical domains and four reasoning-demand dimensions: structural, symbolic, quantitative/computational, and geometric/spatial reasoning. The tasks use deterministic checkers reviewed by mathematicians for fidelity and resistance to guessing. Evaluating four frontier and open-weight models under a requested 15,000-token completion budget, pass@1 ranges from to . Token-budget truncation, rather than incorrect completed answers, dominates failures: all four models are correct – of the time when they produce a checkable answer. A targeted 30,000-token probe on 10 tasks that were truncated for every model and run at 15k further shows substantial budget sensitivity: GPT-6 Astra solves 6 of 10, while Kimi K3 solves 1 and Qwen3.8 Max solves none. We release benchmark tasks, prompts, deterministic checkers, reasoning-demand annotations, and raw model transcripts, while MathDB serves as the durable contribution: a growing, community-editable resource from which new, unsaturated evaluations can be constructed as existing benchmarks saturate.

Primary area:
datasets and benchmarks
Keywords:
mathematical reasoning, benchmark, evaluation, dataset, mathematics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_yNmbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a live database of over 80k math problems sourced from arXiv and other repositories. It also includes a subset of 100 problems for use as a benchmark and reports evaluation results on several models like GPT-6 Astra and several open-source ones. The evaluation is deterministic and does not rely on proofs. The models generally do very well on the benchmark; Astra scores 69%, and it turns out most of the failures are because the model exceeds the 15k token budget.

Critical strengths

The database is large and is seemingly formatted well. They use deterministic verifiers for each problem, meaning that proofs or formalized statements are not necessary. It is also interesting how weaker models tend to fail more because they don't arrive at a final answer within the token budget.

Critical weaknesses

Overall, there seem to be many critical issues with this paper.

  • First, the dataset seems to provide minimal value to the AI/math research community. While the database is large, most problems are published and already have solutions. Therefore these problems are not unsolved, and the very high saturation rate indicates the problems are already far too easy for frontier models

  • A 15k token budget is used, which severely limits model performance. They never justify why this token limit is used, since most LLMs support a ~128k token limit. Importantly, most failed problems are because they run out of tokens. Answers that are finished pass between 92 to 100% of the time.

  • Astra's 69% pass rate is basically exactly its finish rate. And they even increase the budget to 30k tokens for an additional evaluation of Astra, which lets it solve 6 more of the 10 tasks every model had previously missed.

  • The paper makes the conclusion that stopping and length rater than wrong math are a bottleneck. But this is obvious... there is no reason to arbitrarily limit model reasoning.

  • It's unclear where the problems came from. The sources are arXiv, AIM lists, Erdos problems, wikipedia pages, but there are no details on the distribution or scraping. They also don't provide the prompts to the LLM for how the problems are rewritten.

Questions for authors
  • Why did you use a 15k token budget? What happens when you give the models the full token budget?

  • How was the 100-problem eval set selected from the full 80k dataset?

  • What prompts were given to the model for rewriting? What model was used? How many of the problems came from arXiv, Wikipedia, etc? Was there any review/quality inspection of the problems (this is critical).

  • Can you discuss contamination risks in greater detail? It seems like nearly all problems are from published literature where the solution already exists in theory.

  • Perhaps this is a naive question, but can you please explain the point of these deliberately separate problem-solving phases for the agents? Why not give the agents a single but larger budget to solve the problems?

1 comment

Thank you for the review. MathDB is the main contribution of this paper; MathDB-Bench is its first evaluation release. Several concerns arise from treating these as the same resource.

  1. Most problems in MathDB are open.

The claim that most of MathDB is already solved is incorrect. MathDB contains over 80,000 mathematical research problems, most of which remain open, together with literature sources, solution histories, and relationships between results. It supports problem discovery, tracking mathematical progress, and future benchmark construction. Performance on 100 adapted tasks does not establish saturation of this database.

  1. MathDB-Bench deliberately uses solved problems for verifiable evaluation.

We selected 100 tasks from 981 resolved records across five mathematical domains. All tasks and their checkers were individually reviewed by two mathematicians. Known results provide reliable reference points for designing checkable tasks, with a range of difficulties that helps distinguish model capabilities.

A problem being solved by mathematicians does not make it uninformative for evaluating models. This release demonstrates how MathDB can support benchmark construction; it does not define the scope of the database.

  1. Our results measure performance under a stated budget.

The requested 15,000-token budget defines the evaluation condition. We explicitly tested its effect: on ten tasks missed by all evaluated models, increasing the budget to 30,000 tokens allowed Astra to solve six, Kimi K3 one, and Qwen3.8 Max none.

These results show budget sensitivity and differences between models. We do not claim they establish maximum mathematical capability. A larger, uninterrupted budget is also a valid evaluation protocol.

  1. Provenance, review, and contamination require separate treatment.

MathDB preserves problem sources and mathematical context, while the benchmark undergoes expert review. We will clarify the source breakdown, adaptation prompts, and models used.

MathDB-Bench demonstrates one use of MathDB. Its initial scores do not justify dismissing the broader database or its value to mathematical research and future evaluation.

Reviewer_j3NkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MathDB, a continuously updated database and evaluation platform for research-level mathematics.

  • Database: Approximately 87,000 mathematical problems, with sources, subject labels, solution status, and verification evidence. It grows through automated literature extraction and community contributions.
  • Benchmark: A manually curated set of 100 tasks adapted from 981 resolved records, covering five mathematical domains and using programmatic answer verification.
  • Results: Under a 15,000-token budget, four models achieve 13.5%–69% accuracy. Most failures result from exhausting the budget; accuracy among completed answers is approximately 92%–100%.
  • Core idea: Replace saturated benchmark items with new problems as models improve, keeping the evaluation informative.
Critical strengths
  1. Reusable infrastructure for mathematical evaluation. MathDB connects a large collection of mathematical problems with literature sources, solution status, verification evidence, and revision histories, while separating the continuously updated database from frozen benchmark releases. This can reduce problem-discovery and provenance-tracking costs for future evaluations.

  2. A rigorous task-verification process. All 100 benchmark tasks and their checkers were reviewed individually by two mathematicians, and answers are evaluated using deterministic programs. This improves scoring reproducibility.

Critical weaknesses
  1. Insufficient evidence that adapted tasks retain research-level difficulty. A problem’s origin in research literature does not establish that an adapted instance requires similarly deep mathematical reasoning. For example, the paper’s group-construction task can be solved through simple random search. Without random-guessing, simple-search, and standard computational baselines, it is difficult to determine how much model performance reflects research-level mathematical reasoning. This directly undermines the benchmark’s core validity.

  2. The claimed reduction in benchmark-maintenance costs lacks empirical support. Task selection, adaptation, checker construction, and mathematical review remain entirely manual in the first release. The paper neither demonstrates a subsequent refresh cycle nor quantifies human effort or the rate at which suitable new tasks become available. Although the database offers reuse value, the evidence does not establish that the framework can sustainably produce high-quality benchmarks at low cost, weakening its central scalability claim.

Questions for authors
  1. Clarification request: How was the difficulty of the adapted tasks validated? For the group-construction example, randomly generated matrices satisfy the target condition approximately 4% of the time under the stated group law. Are there additional constraints omitted from the main text? Do existing reviews or baseline results indicate how frequently other tasks can be solved through simple guessing or search? This would directly affect my assessment of whether the benchmark measures research-level mathematical reasoning.

  2. Clarification request: What existing evidence supports the cost advantage of dynamic updates? Task selection, adaptation, checker construction, and mathematical review were all performed manually for the first release. Do existing workflow records quantify the human effort at each stage, the specific work saved by the database, and the proportion of newly resolved problems that become eligible benchmark tasks? Has a subsequent refresh cycle already been completed? These details would affect my assessment of the framework’s scalability.

2 comments
AuthorsOpenReview ↗

Problem Selection and How MathDB Facilitates MathDB-Bench future releases

Thank you for the thoughtful review and for recognizing the value of MathDB. We would like to clarify two points.

1. On the difficulty of adapted mathematical tasks.

We agree that some adapted tasks may be relatively easy, and your observation about the group-construction example is valuable. We do not claim that every task requires research-level reasoning. MathDB itself contains well-established conjectures, but we carefully select those solved problems from MathDB and redesign them so that they are challenging and can well distinguish models at different levels. Indeed, from our experiments, frontier models like Astra couldn't complete all the tasks within our budget limit and weaker models can solve some of them so that we know their limits.

Task difficulty was assessed by domain experts, including mathematicians specializing in algebra, geometry, probability, and related fields. Each task was reviewed for mathematical correctness, expected solution methods, and difficulty. Our empirical results also indicate substantial remaining challenges: even the best-performing model solved only 69 of the 100 tasks under a 15,000-token budget.

2. On the scalability and cost of dynamic updates.

This is our first release, so we do not yet have empirical comparisons of maintenance costs across multiple benchmark versions. However, MathDB provides infrastructure to make future task selection and updates more systematic and reusable. In particular:

  • Mathematical artifact graphs connect theorems, conjectures, and related results, helping identify mathematically central problems and their dependencies.
  • Subject labels, literature references, and solution histories help researchers find and assess suitable benchmark candidates without repeatedly reconstructing their mathematical context.
  • Community contributions and submitted LLM solution attempts provide additional evidence of which problems may be approachable by current models, helping inform future difficulty assessment and task selection.

These features allow MathDB to support not only a fixed benchmark, but also a continuously evolving mathematical evaluation ecosystem. We used the first release of MathDB-Bench as an example to illustrate this and the data set MathDB is also our valuable contribution to the AI research community.

Thank you for the detailed rebuttal. Most of my concerns have been resolved, rasing the score to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Problem Selection and How MathDB Facilitates MathDB-Bench future releases

Thank you for your detailed review and for carefully examining our released artifacts. We appreciate your suggestions and would like to clarify the following points on MathDB adn MathDB-Bench.

  1. On the difficulty of adapted mathematical tasks. We agree that some adapted tasks may admit simple solutions, including the examples you identified. We do not claim that every task requires research-level reasoning. Rather, our goal is to construct tasks with different levels of difficulty that can distinguish models with different mathematical capabilities. Task selection and difficulty were reviewed by domain experts, including mathematicians specializing in algebra, geometry, probability, and related fields. Our empirical results also indicate substantial remaining challenges: even the strongest model, Astra, solved only 69% of the tasks under our budget.

  2. On the value and scalability of MathDB. MathDB is our central contribution, while MathDB-Bench is its first demonstration. We agree that this initial release does not yet provide quantitative evidence of reduced maintenance costs across benchmark updates. However, MathDB provides infrastructure that makes future updates more systematic:

  • Mathematical artifact graphs connect conjectures, theorems, and related results, helping identify important problems and their mathematical dependencies.
  • Domain labels, literature references, and solution histories make it easier to discover and assess suitable benchmark candidates.
  • Community contributions and LLM solution attempts provide additional information about which problems may be approachable by current models.

These features reduce the need to reconstruct mathematical context and provenance from scratch for each new release. Expert review remains essential, but the underlying database can be continuously reused and expanded. We agree that measuring expert time and conducting further refresh cycles would provide stronger evidence of the resulting efficiency gains.

  1. On the sandbox environment and evaluation integrity Our evaluation is designed to separate agents' problem-solving environments from the reference solutions and evaluation checkers. Agents work on the mathematical tasks in a sandboxed environment, where they can perform computations and develop their own solutions without access to the reference answers. Answer verification is handled separately through deterministic checkers, which were individually reviewed by mathematicians for correctness and fidelity to the original tasks. This separation is intended to prevent answer leakage through the evaluation infrastructure while allowing agents to use computational tools. Together, sandbox isolation and deterministic verification support an evaluation of independent mathematical problem-solving rather than answer retrieval.

  2. Reproducibility of results. Thank you for independently re-scoring our released artifacts. We agree that the differences between the reported scores (69.0% for Astra and 20.5% for Qwen) and the scores reproducible from the available logs (68.0% and 20.0%) are not evident and within the accepted range of error.

We believe MathDB's main value extends beyond the difficulty of any individual benchmark task. It provides a growing, structured mathematical knowledge base with provenance, verification evidence, and relationships between mathematical results, supporting the construction of reproducible evaluations as AI capabilities evolve.

We appreciate your careful review and the concrete suggestions for strengthening both the benchmark and its reproducibility.

We thank all reviewers for their careful reading. Several concerns arise from evaluating MathDB through the lens of its first benchmark release, so we would like to restate what each part contributes.

MathDB is the main contribution. MathDB is a living, structured map of research mathematics rather than a fixed problem set. At the snapshot used in the paper it contained 87,372 records, of which 84,655 are open problems and conjectures, collected from arXiv papers, AIM problem lists, Erdős problems, curated collections, and community submissions. Each record links the statement to its sources, subject tags, status history (open, claimed, solved), verification evidence, and tracked revisions.

It is dynamic. New records arrive through automated literature ingestion and community contributions, and a record's status changes as the mathematics changes: when a conjecture is proved or disproved, the record moves from open to solved together with the evidence. Exports are frozen with manifests (date, scope, row counts, hashes), so any analysis or benchmark can refer to an exact snapshot while the database keeps growing. This is what allows evaluations to be renewed from a pool that is constantly replenished by research itself, rather than rebuilt from scratch.

The relation graph locates each problem within mathematics. Records are connected by typed relationships such as generalizes, implies, and implied by, and the graph is ranked by centrality. This lets researchers and model developers see where a conjecture sits within its field: which problems it depends on, which results would follow from it, and which open problems are central rather than isolated. To our knowledge, no existing mathematics benchmark provides this structure. It supports problem discovery, tracking of mathematical progress, and the selection of evaluation targets by their mathematical position, not only by topic.

It is a resource for AI and mathematics alike. Because MathDB records open conjectures with their provenance and context, it can serve as a source of research targets for AI systems, a record of which problems have been attempted and resolved, and a foundation for new evaluations as existing ones saturate.

MathDB-Bench is one example of what MathDB enables. We built the first release from the 981 records with an established resolution, adapting 100 of them into tasks with deterministic checkers, each reviewed by two mathematicians against its original source. Its purpose is to demonstrate the pipeline from database to frozen, verifiable evaluation, not to define the scope of MathDB. The release already separates current systems: pass@1 ranges from 13.5% to 69.0%, 28 tasks are solved by no model under the 15,000-token budget, and a 30,000-token probe recovers 6 of 10 such tasks for GPT-6 Astra but at most 1 for the other models we probed. We agree that individual adapted tasks vary in difficulty; the range of difficulty is what lets one release distinguish systems at different levels. Future releases can draw from the growing resolved pool, retire saturated items, and target problems selected through the relation graph.

Revisions. We revised the paper to state the evaluation protocol precisely (a requested 15,000-token completion budget, including reasoning and output), to describe the budget probe for the models it covers, and to sharpen the description of how tasks were reviewed.

We hope this clarifies that the value of this work lies primarily in MathDB as a growing, structured, and community-maintained map of open research mathematics, with MathDB-Bench as a first demonstration of the evaluations it makes possible. We would be grateful if the reviewers could take this into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.