- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a live database of over 80k math problems sourced from arXiv and other repositories. It also includes a subset of 100 problems for use as a benchmark and reports evaluation results on several models like GPT-6 Astra and several open-source ones. The evaluation is deterministic and does not rely on proofs. The models generally do very well on the benchmark; Astra scores 69%, and it turns out most of the failures are because the model exceeds the 15k token budget.
Critical strengths
The database is large and is seemingly formatted well. They use deterministic verifiers for each problem, meaning that proofs or formalized statements are not necessary. It is also interesting how weaker models tend to fail more because they don't arrive at a final answer within the token budget.
Critical weaknesses
Overall, there seem to be many critical issues with this paper.
-
First, the dataset seems to provide minimal value to the AI/math research community. While the database is large, most problems are published and already have solutions. Therefore these problems are not unsolved, and the very high saturation rate indicates the problems are already far too easy for frontier models
-
A 15k token budget is used, which severely limits model performance. They never justify why this token limit is used, since most LLMs support a ~128k token limit. Importantly, most failed problems are because they run out of tokens. Answers that are finished pass between 92 to 100% of the time.
-
Astra's 69% pass rate is basically exactly its finish rate. And they even increase the budget to 30k tokens for an additional evaluation of Astra, which lets it solve 6 more of the 10 tasks every model had previously missed.
-
The paper makes the conclusion that stopping and length rater than wrong math are a bottleneck. But this is obvious... there is no reason to arbitrarily limit model reasoning.
-
It's unclear where the problems came from. The sources are arXiv, AIM lists, Erdos problems, wikipedia pages, but there are no details on the distribution or scraping. They also don't provide the prompts to the LLM for how the problems are rewritten.
Questions for authors
-
Why did you use a 15k token budget? What happens when you give the models the full token budget?
-
How was the 100-problem eval set selected from the full 80k dataset?
-
What prompts were given to the model for rewriting? What model was used? How many of the problems came from arXiv, Wikipedia, etc? Was there any review/quality inspection of the problems (this is critical).
-
Can you discuss contamination risks in greater detail? It seems like nearly all problems are from published literature where the solution already exists in theory.
-
Perhaps this is a naive question, but can you please explain the point of these deliberately separate problem-solving phases for the agents? Why not give the agents a single but larger budget to solve the problems?
Thank you for the review. MathDB is the main contribution of this paper; MathDB-Bench is its first evaluation release. Several concerns arise from treating these as the same resource.
The claim that most of MathDB is already solved is incorrect. MathDB contains over 80,000 mathematical research problems, most of which remain open, together with literature sources, solution histories, and relationships between results. It supports problem discovery, tracking mathematical progress, and future benchmark construction. Performance on 100 adapted tasks does not establish saturation of this database.
We selected 100 tasks from 981 resolved records across five mathematical domains. All tasks and their checkers were individually reviewed by two mathematicians. Known results provide reliable reference points for designing checkable tasks, with a range of difficulties that helps distinguish model capabilities.
A problem being solved by mathematicians does not make it uninformative for evaluating models. This release demonstrates how MathDB can support benchmark construction; it does not define the scope of the database.
The requested 15,000-token budget defines the evaluation condition. We explicitly tested its effect: on ten tasks missed by all evaluated models, increasing the budget to 30,000 tokens allowed Astra to solve six, Kimi K3 one, and Qwen3.8 Max none.
These results show budget sensitivity and differences between models. We do not claim they establish maximum mathematical capability. A larger, uninterrupted budget is also a valid evaluation protocol.
MathDB preserves problem sources and mathematical context, while the benchmark undergoes expert review. We will clarify the source breakdown, adaptation prompts, and models used.
MathDB-Bench demonstrates one use of MathDB. Its initial scores do not justify dismissing the broader database or its value to mathematical research and future evaluation.