acceptodds
Under review as a conference paper at ICLR 2027

Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs

Abstract

Large language models (LLMs) are becoming increasingly capable mathematical collaborators, but static benchmarks are no longer sufficient for evaluating progress: they are often narrow in scope, quickly saturated, and rarely updated. This makes it hard to compare models reliably and track progress over time. Instead, we need evaluation platforms: continuously maintained systems that run, aggregate, and analyze evaluations across many benchmarks to give a comprehensive picture of model performance within a broad domain. In this work, we build on the original MathArena benchmark by substantially broadening its scope from final-answer olympiad problems to a continuously maintained evaluation platform for research mathematics with LLMs. We regularly evaluate new models, review their execution, and publish detailed analyses that inform updates to benchmarks and evaluation protocols. MathArena now includes three research benchmarks: ArXivMath for final-answer questions, BrokenArXiv for responses to false mathematical claims, and ArXivLean for formal proof generation in Lean. Currently, GPT-6 Astra is the best-performing model on MathArena, achieving 88% expected performance on the non-deprecated benchmarks.

Primary area:
datasets and benchmarks
Keywords:
AI4math, benchmark, llm, evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_A4ZJOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a extension to MathArena benchmark by combining three types of mathematical tasks: ArXivMath for final-answer questions, BrokenArXiv for responses to false mathematical claims, and ArXivLean for formal proof generation in Lean. While ArxivMath and ArxivLean type benchmarks have been proposed before, BrokenArxiv adds a complementary evaluation of whether models reject false claims rather than invent proofs. Its main contribution is bringing these evaluations together with regular benchmark updates. The authors state that the benchmark will be updated at regular frequency with latest research papers. The experiments show substantial improvements in mathematical performance and restraint when facing false claims, while formal proof generation remains harder.

Critical strengths
  1. The idea of reducing contamination by regularly updating with latest papers and evaluating models which wouldn't have seen these new research is very effective.
  2. The authors have taken measures to prevent internet search which can conflate the models own knowledge.
  3. The idea of BrokenArxiv is very interesting and novel.
  4. The pipeline of question generation and validation check is very thorough and explained well.
  5. The authors address the prohibitive cost of full benchmark evaluations by using an IRT model to blend observed scores with statistically predicted performance for unattempted questions, while fitting a log-additive model to estimate missing costs separately.
Critical weaknesses
  1. Since the benchmark uses research work, which in some can be generated by heavy AI assistance. Such data points may favor problems which can be easier to current models, potentially biasing the benchmark’s coverage.
  2. In final answer benchmark, the evaluation does not bring out the mechanism of answering the question. The result could be obtained by guessing or shortcut. Since the benchmark uses LLM as a judge, it would also benefit to look at how LLM arrived at the answer, its reasoning etc.
  3. Although the paper reports high agreement between two BrokenArxiv judges, it does not show how judge choice affects per-model scores or leaderboard rankings. Reporting these differences would strengthen confidence in the comparisons.
  4. Since this paper is more towards introduction of a continuously updated benchmark, it would be really useful to know what is the cost of creating the problems, the screening, the validation etc.
Questions for authors
  1. How do the authors validate that every BrokenArxiv statement is false under precisely the assumptions given to the model? Although the construction includes automated checks and human review, have they independently audited the questions or investigated model generated proofs for possible errors in the benchmark labels?
  2. Can the authors comment on how a source paper which heavily relied on LLMs to find a proof or solution, can affect the benchmark quality and ease of newer models solving them?
  3. Can the authors provide more details on the rationale behind using different models within the source extraction pipeline across the distinct benchmark subsections, and how this choice might impact consistency?
  4. Can authors report some per-model scores with different judges?
  5. Does the three formats of benchmark problems here cover all or most of the type of research in Mathematics? Can you provide a distribution chart on any underrepresented areas if any and comment on the same?
2 comments

We thank the reviewer for their review. We are happy to see they found the refreshed research questions, BrokenArXiv task, and documented generation and validation valuable. In this rebuttal, we address their questions.

Q1: How do you verify that every BrokenArXiv statement is false under precisely the assumptions provided to the model?
A large part of our automated pipeline is constructed to stress the importance of including all necessary assumptions, as we think this is very important. We also inspect strong-model outputs after evaluation to investigate claims of a proof. Earlier releases sometimes contained ambiguous statements that these checks helped identify. Across the August, September, and October versions, we have not found an ambiguous formulation.

Q2: Have the statements been independently audited, and have model-generated proofs been investigated for possible errors in benchmark labels?
Please see Q2 of our main response.

Q3: How might heavy LLM assistance in source papers affect benchmark quality, coverage, and ease for newer models?
Please see Q3 in our main response.

Q4: Why are different models used across extraction stages and benchmark subsections, and how does that choice affect consistency?
Two practical considerations explain the changing model choices:

  • We use the best available model at the time of construction. GPT-6 Astra was not available when the June ArXivLean release was generated, but was used for September.
  • Funding also influenced earlier choices. Access to GCP credits made Gemini 3.1 Pro cheaper for much of the pipeline through June. However, its formalization performance was noticeably poor, so we used GPT models for the most demanding stages in ArXivLean. We now use GPT models for all production benchmarks.

In Q1 of our main response, we show that model choice does not significantly affect consistency.

Q5: Can you report per-model scores and changes in leaderboard rankings under different judges?
Yes. We re-graded September BrokenArXiv responses with Sol 6.1 (high), Opus 5.5 (high), and Gemini 4 Argon(high), using the existing judge prompt and Gemini 3.8 Flash baseline. Spearman rank correlations are 0.9727, 0.9909, and 0.9909, respectively. Appendix E.2 details the experiment with full results and model scores.

Q6: How broadly do the three task formats cover mathematical research?
We added three pie charts grouping source papers' primary arXiv topics into mathematical areas, covering all questions since August. Geometry and Topology is the largest group for ArXivMath at 48/157 (30.6%), Analysis for BrokenArXiv at 42/163 (25.8%), and Combinatorics for ArXivLean at 27/53 (50.9%). Overall, we find that all topics are represented, but some are less prominent than the raw distribution of arXiv. For instance, in September, analysis formed 33% of papers, while only being represented with 6% in arXivMath and 13% in ArXivLean. The distribution of BrokenArXiv is closer to the actual source distribution, because its restrictions in the question generation procedure are more independent of the field.

Q7: Can you assess whether correct final answers arise from valid reasoning, guessing, or shortcuts?
Great question! We do not evaluate the full reasoning behind model answers for two reasons:

  • LLM judges can introduce bias and measurement errors. Although our judge achieves 99.7% accuracy when verifying final answers, we expect its accuracy to be substantially lower when evaluating the reasoning behind them, since this is a much harder task. This could disadvantage models that are honest about their uncertainty: plausible but incorrect arguments might receive credit, while an explicit admission of guessing would not.
  • We do not require models to provide a full derivation. Some models therefore give only a brief explanation of how they arrived at their answer, making it difficult to evaluate reasoning across existing answers.

Nevertheless, we asked an AI agent to categorize a subset of ArXivMath responses for the September benchmark, assuming the model outputs were truthful about how they obtained the answer. Among judge-correct responses, 42.5% claim an analytical derivation, 18.7% use an exact computation, 27.3% use numerical reconstruction or extrapolation, and 7.7% use a heuristic guess (mostly between two unexcluded answers). Importantly, we explicitly request a best guess because some models otherwise gave multiple candidates, which earned no credit and disadvantaged truthful models.

Q8: What does creating and maintaining the benchmarks cost?
The budget is roughly \100perbenchmarkeditionplusaboutonedayofhumanreview.Asanexample,BrokenArXivSeptembercost100 per benchmark edition plus about one day of human review. As an example, BrokenArXiv September cost \\111.93: \18.58forscreening,18.58 for screening, \\60.40 for generation, and \32.95forverification.Atthebenchmarksizewehave(roughly50−60questions),asingleevaluationpasscostsroughly32.95 for verification. At the benchmark size we have (roughly 50-60 questions), a single evaluation pass costs roughly \\400 for an expensive model and \$50 for a typical model.

Reviewer_A4ZJOpenReview ↗

Response to author rebuttal

Thank you very much for clarifying my questions. I believe all my questions are answered. I would like to keep my current rating.

Reviewer_FtEoOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends MathArena into a regularly updated evaluation platform for research-level mathematics. The authors collect and review problems from recent arXiv papers to build ArXivMath, BrokenArXiv, and ArXivLean. These benchmarks evaluate final-answer correctness, responses to false statements, and Lean proof generation. The authors use experiments on multiple models, human review, and additional analyses to study model behavior and the robustness of the results. They find that some stronger models more often admit that they cannot prove false statements, while Lean proof generation remains challenging.

Critical strengths
  1. BrokenArXiv distinguishes between claiming to prove false statements, silently modifying them, admitting an inability to prove them, and identifying them as false. The authors also use self-correction experiments to analyze these behaviors. This supports systematic evaluation of fidelity and reliability in mathematical collaboration and shows how some models keep repairing proofs without questioning the false statements.

  2. The authors collect and review research-level mathematical problems from recent papers and regularly update the benchmarks. ArXivLean also provides proof tasks that can be checked automatically. These resources help reduce contamination and saturation risks and support future evaluation of models and theorem-proving systems.

Critical weaknesses

The authors evaluate different models using their recommended execution frameworks and resource settings. Although they report the costs and settings, the paper lacks an analysis of how these choices affect the main rankings. This makes it hard to tell how much of the performance gap comes from the models themselves and whether the rankings would remain stable under other reasonable settings.

Questions for authors
  1. Have the authors compared the same model under different execution frameworks or resource budgets? If so, do these changes substantially affect model performance and the main rankings?
  2. For the stronger models on BrokenArXiv, have the authors also evaluated their responses to the corresponding true statements? How often do these models label true statements as false or admit that they cannot prove them?
2 comments

We thank the reviewer for their review. We are happy to see they found the reliability and fidelity analysis, self-correction experiments, and automatically verified Lean tasks useful. In this rebuttal, we address their questions and concerns.

Q1: Have you compared the same models across execution frameworks and resource budgets?
Yes. We compared Gemini 3.8 Flash on August ArXivMath across Antigravity, Qwen Code, and Kimi Code, and DeepSeek V4.1 Flash 10/17 on September ArXivMath across DeepSeek Harness, Qwen Code, Kimi Code, and OpenCode. The observed accuracy ranges span 2.3 and 2.8 percentage points, respectively. These results suggest very limited harness sensitivity in these runs.

Gemini 3.8 Flash on ArXivMath August

HarnessAccuracyCost/attempt
Antigravity40.35%$0.916
Qwen Code41.52%$0.859
Kimi Code42.69%$0.833

DeepSeek-v4.1-Flash-1017 on ArXivMath September

HarnessAccuracyCost/attempt
Qwen Code48.11%$0.396
OpenCode45.28%$0.329
Kimi Code45.28%$0.446
DeepSeek Harness45.28%$0.373
This experiment has now been added in Appendix E.3.

Q2: How often do strong models label corresponding true statements false or admit that they cannot prove them?
We ran Sol 6.1 on the true statements from BrokenArXiv September. It rejected a true statement in 2/106 attempts, both on the same one of 53 questions. Both rejections invoke a well-known theorem from 1996 that the 2026 source paper refutes by finding gaps in the original proof. It admitted an incomplete proof in 34/106 attempts (32%) and claimed a complete proof in 70/106 (66%). On the corresponding false statements, it claimed a proof in 2/106 attempts, again on that same question, admitted an incomplete proof in 49/106 (46%), and rejected the statement in 55/106 (52%).

Thank you for running these additional experiments. The small differences in accuracy across frameworks for the two models tested are reassuring. The comparison between the matched true and false statements also answers my question about BrokenArXiv. I have no further questions and will keep my rating of clear acceptance.

Reviewer_LYU1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MathArena expands from an uncontaminated competition benchmark into a continuously updated evaluation platform for LLM mathematics. Each month, it extracts problems from recent arXiv papers via an LLM-assisted, human-reviewed pipeline across three tasks: ArXivMath for final-answer questions (287 total, 145 active), BrokenArXiv for catching false statements scored 0--3 by an LLM judge (308 total, 160 active), and ArXivLean for proving LLM-formalized theorems checked with Comparator (87 total, 46 active). An IRT model aggregates the partially observed evaluations into a single expected performance score, where GPT-6 Astra leads overall at 88.0% (about 35 points ahead of the best open model) and makes zero false proof claims on the August BrokenArXiv set.

Critical strengths
  1. The platform releases all questions, prompts, per-run model outputs, and evaluation code, and reproduces provider-reported results before publication. Moreover, Appendix C documents eight operational under-elicitation bugs caught and fixed in practice---including tool-call caps, a prompt phrase that shortened Gemini's reasoning, API timeouts, and a deprecated Claude parameter---providing practical insights critical for trustworthy leaderboard numbers.
  2. The suite combines automatically scored research-level tasks with rare human-graded competition data. BrokenArXiv (August) draws natural false claims directly from prior results refuted in the source papers, ArXivLean uses Comparator to prevent statement tampering in formal proofs, and the deprecated benchmarks include Putnam 2025 and blind-graded Mikl'os Schweitzer 2025 solutions scored by official juries (Appendix B.3), alongside human-versus-LLM grading comparisons on USAMO (Table 6, Fig. 5).
Critical weaknesses
  1. BrokenArXiv cannot separate reliability from abstention. Because the main benchmark contains only false statements, the August rubric (App. F.5.2) awards 3 points for saying a statement is incorrect (without checking mathematical validity) and 2 points for admitting inability to find a proof, while the original 0--2 rubric gave full credit for admitting inability (App. A.3.3). A model that always answers "this statement is false" would score 100%, and one that always answers "I cannot prove this" would score 66.7%---beating Kimi K3 (61.9%), Gemini 3.8 Flash (27.4%), and Muse Spark 1.3 (20.2%), and nearly matching Qwen3.8-Max (69.6%). Moreover, the August prompt tells models to attempt the statement "regardless of the fact it is open" (p. 49), which cues abstention. GPT-6 Astra's headline result of never claiming a false proof consists of 77 rejections and 91 admissions of inability out of 168 responses, which without true-statement controls is indistinguishable from simply declining hard problems. The true-statement control in App. D.1 tests only 56 March items on three older models, leaving the claims in Section 5.2 unsupported without sensitivity and specificity on matching true statements.

  2. The top-ranked model built the August benchmarks, and the aggregate rankings lack uncertainty estimates. GPT-6 Astra runs all three stages of the August extraction (screening, generation, verification; Section 4.1) and serves as one of the two judges in Section 4.2. While it also leads on the Gemini 3.1 Pro-built June sets (94.4% vs. 82.6% on ArXivMath; 99.1% vs. 90.7% on BrokenArXiv), its edge over Claude Fable 5.1 rests entirely on the August sets it helped construct: 88.6% vs. 87.7% on ArXivMath (101 vs. 100 of 114 responses) and 81.9% vs. 79.8% on BrokenArXiv. In addition, Fig. 2 places GPT-6 Astra's release around September 2026, after the August source papers appeared, invalidating the one-month pre-release contamination guard (Section 3.1). Finally, the 4.9-point IRT gap between GPT-6 Astra (88.0%) and Fable 5.1 (83.1%, observed only on the two August sets and otherwise imputed in Table 2) is smaller than the IRT model's own leave-one-benchmark-out MAE of 5.53 points (App. D.2), and the temporal progress claim (o4-mini 15.8% to 88.0%; Fig. 2) pools imputed scores and June tool-free runs with August tool-assisted harness runs without confidence intervals.

  3. Benchmark quality is unmeasured, with potential post-execution selection bias and simplified formalizations. Items are drafted and filtered by LLMs (Gemini 3.1 Pro, GPT-6 Astra, GPT-5.5) and checked by a single author, with no reported item error rates, human acceptance rates, inter-annotator agreement, or human-agreement rates for the BrokenArXiv LLM judges (only 97.6% inter-LLM agreement). The paper also omits how many items were dropped after model execution (App. A.3.1), a post-hoc filtering step triggered by model outputs that can bias scores---especially given that 12 of 24 difficulty-selected items in Apex were found flawed late in construction (App. C). In ArXivLean, Comparator only checks that a proof matches the Lean statement, not that the GPT-5.5 formalization (restricted to "a couple of lines", Section 4.3) is faithful to the paper; a 65.2% solve rate on recent main theorems warrants auditing whether the statements were weakened.

  4. The methodological delta over prior work is thin, and key analyses are deferred to external blogs. MathArena (Balunovic et al., 2025) was already a live, contamination-aware platform, BrokenArXiv ports BrokenMath (Petrov et al., 2025b) from competitions to research claims, and ArXivMath follows existing refreshable arXiv benchmarks while omitting RealMath (Zhang et al., NeurIPS 2025). Beyond time-bound leaderboard numbers already on the website, the qualitative failure analysis claimed as contribution 3 in Section 5 is deferred to non-peer-reviewed blog posts rather than systematically analyzed across benchmark versions in the paper.

Questions for authors

See weaknesses.

3 comments

We thank the reviewer for their review. We are happy to see they found the open nature, execution safeguards, and complementary grading procedures valuable. We address their questions below.

Q1: Can matched true-statement controls distinguish reliability from abstention?
Not really. We excluded true statements because we found that models are more likely to accept a user statement than to reject it. However, we regularly revisit this assumption. For instance, we ran Sol 6.1 on the true statements from BrokenArXiv September. It rejected a true statement in 2/106 attempts, admitted an incomplete proof in 32% and claimed a proof in 66%. On the false statements, it claimed a proof in 2/106 attempts, admitted an incomplete proof in 46%, and rejected the statement in 52%.

Additionally, trivial baselines would score 0 in ArXivMath and ArXivLean. The complementary nature of our benchmarks therefore allows us to mitigate individual weaknesses.

Q2: Does the August prompt's reference to the statement being open cue abstention?
We added this instruction to reduce abstention. Initially, we found that models often recognize a statement as an open conjecture and stop quickly. Our prompt reduces this behavior and was taken from a mathematician that uses AI in their daily work and uses it to encourage an attempt.

Q3: How do you assess bias from Astra performing generation?
Please see Q1 in our main response.

Q4: How does Astra's September release reconcile with August source papers?
This was ambiguous in our paper, so we clarified it. Models are evaluated on all active benchmarks. We have found that performance between editions preceding and following model release is almost always very consistent.

Q5: Can you report uncertainty in benchmark scores?
Yes. The revised paper reports 95% confidence intervals in Appendix E.6. We agree that our differentiation between Fable 5.1 and GPT-6 Astra was too strong. The current paper instead includes GPT-6.1 Sol and Opus 5.5, where the gap is significant.

Q6: How robust is the temporal-progress claim when scores mix imputation?
Imputation makes the exact historical scores uncertain, but this claim does not rely on comparing disconnected models. Consecutive model generations share many evaluated questions. All these consecutive models improved, supporting the conclusion that performance has increased steadily.

Q7: Can you independently audit benchmark quality?
Please see Q2 of our main response.

Q8: How is it possible that models achieve 65% on ArXivLean?
We specifically choose easier questions for inclusion in ArXivLean in our human review. In September, we tried to sample questions more uniformly, but scores were near 0%, which is useless for a benchmark that differentiates between model capabilities. A member of our team with significant experience in Lean validates all statements and ensures they are correctly formalized.

Q9: How well do the BrokenArXiv judges agree with human judgments?
We manually graded 100 responses from the September BrokenArXiv answers. The original Gemini 3.8 Flash judge agreed on 95/100. Review of the five disagreements identified three human oversights.

Q10: How many items were removed after observing outputs?
Approximately six questions were corrected or removed across August, September, and October, which contain 373 questions. We make such changes rarely and only after confirming a ground-truth error. The sentence in A.3.1 was too broad, so we have corrected it.

Q11: What is the methodological advance over prior work, and how does this work relate to RealMath?
We have added the missing RealMath citation. Our contribution includes:

  • Sustained maintenance. EternalMath, LiveMathematicianBench, and RealMath propose refreshable benchmarks, but updates have been mostly absent, with only LiveMathematicianBench posting an update in June where Gemini-3.5-Flash is still the best performing model. Maintaining a quality benchmark requires ongoing verification and investigation of unexpected results. For example, LiveMathematicianBench ranks DeepSeek-R1 (released in 01/25) above GPT-5.2 (released in 12/25), which seems highly unplausible. On MathArena, DeepSeek-R1 and GPT-5.2 have estimated scores of 12.3% and 32.4%, respectively.
  • Relevance and difficulty. Increasing the relevance and difficulty of benchmarks is challenging. BrokenMath relies on (easier) competition-level problems, while BrokenArXiv asks for open conjectures that were recently disproven. MathArena initially focused on now long-saturated competition problems, this new version focuses on research and more clearly measures performance on three distinct axes.

Q12: Can the qualitative failure analysis be included in the paper?
The revised text now includes some qualitative examples in Appendix F. Importantly, the blog posts are an essential part of the submission, and are available through the supplementary material.

Thank you for the detailed response and the additional experiments. The confidence intervals (App. E.6), the human and cross-judge validation of the BrokenArXiv judge (App. E.2), the generator ablation, and the clarification of the Astra–Fable comparison address most of my concerns. I will take these into account and re-evaluate my score.

I would still encourage the authors to discuss in the final version that the BrokenArXiv rubric gives partial credit to abstention (e.g., by reporting an always-abstain baseline), and to state explicitly that ArXivLean deliberately selects easier statements.

Dear Reviewer LYU1,

Thank you for your feedback. We have now updated the paper to refer to the always-abstain baseline (L319, L413) and that we select easier statements for ArXivLean (L343-345).

Reviewer_6UXUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper describes the design and maintenance of MathArena, a platform for evaluating LLMs' math capabilities.

Critical strengths
  • The authors devoted substantial efforts to developing and maintaining the benchmark, which is updated regularly.

  • There are many reasonable and thoughtful design decisions to ensure the validity of the benchmark.

  • The live and continuously updated benchmark provides useful information in comparing models.

Critical weaknesses
  • The benchmark relies on public results to construct and update the benchmark. So all answers (or important clues to answers) are publicly available. This is a significant weakness. Many models are closed source, and can only be accessed by API calls. What happens on the server is opaque and cannot be controlled or guaranteed. Training or release cutoff dates do not guarantee that there is no internet access happening on the server side or the model doesn't use any internal cache of the latest arXiv.

  • The problem selection protocol makes the results hard to interpret. The problems in ArXivMath are restricted to be those that involve a final answer that can be verified automatically. This excludes many math problems. In addition, the problems are filtered by LLMs, in particular, by the leading model Astra. This may introduce significant selection bias. Can the filtering include only those problems easy to the selection model? This is not clear. In principle, a benchmark should be constructed without the involvement of the model to be tested.

  • Another potential source of selection bias is that AI is now increasingly used to write math papers submitted to arxiv, and not all AI use is disclosed and there is no reliable way to know. The papers might be on arxiv precisely because they are within reach of AI.

  • Another issue with problem selection is that appearing on arxiv does not tell us how difficult or significant the result is. This makes the benchmark less informative.

  • My main concern is that the results of the benchmark are difficult to interpret and not very informative due to the way the problems are selected. The results may simply tell us how well the models can solve problems with a final answer that are within reach of GPT-Astra.

  • The paper claims to provide "continuous" evaluation, but in practice it is just multiple versions of static benchmarks.

Questions for authors

I'd like the authors to address the issue of selection bias in terms of benchmark construction.

1 comment

We thank the reviewer for their review. We are happy to see they found the sustained development effort, evaluation safeguards, and regularly updated model comparisons valuable. In this rebuttal, we address their questions and concerns.

Q1: How do biases affect model scores?
Please see Q3 in our main response.

Q2: How do you address possible access to public answers through opaque API infrastructure?
We consider undisclosed access to the source material highly unlikely for three reasons:

  • We inspect model outputs extensively. Models often recognize a well-known conjecture but do not appear to know that a recent paper resolved it, e.g., they frequently state the given problem is open. If providers were supplying the new paper behind the scenes, they are also preventing models from revealing that access, which is extremely challenging to do reliably.
  • When we explicitly enable internet access, for example to check a formulation, models readily find the source paper and its answer. Hidden access when internet access is disabled should therefore produce much higher scores than we observe.
  • This concern applies to any public benchmark. Evidence for privileged access would get noticed, since it would mean that closed model providers are lying about what their model has access to on public benchmarks to improve their performance.

Q3: Does using evaluated models to select and filter problems introduce selection bias?
Please see Q1 in our main response.

Q4: How do you establish the mathematical difficulty and significance of results selected from arXiv?
We do not independently rank the mathematical significance of every result. Our aim is to construct difficult questions that support relative model comparisons. It is therefore not necessary to differentiate between a major breakthrough and a smaller extension. We only check whether the answer follows from prior work to mitigate contamination. This novelty check relies on the source authors' account of prior work, which we expect to be reliable in most cases.

Q5: Can the benchmark construction procedure extract questions that are out of reach of current models?
Yes. The generator only needs to extract a question from a paper that already explains its solution. This is substantially easier than solving the question without the paper. It can therefore extract questions beyond its own problem-solving ability.

Q6: What distinguishes the platform's continuous evaluation from successive versions of static benchmarks?
The platform contributes more than successive releases of fixed task formats. We also:

  • Publish regular analyses. Since submission, we studied Sol 6.1 as a tool for finding errors in arXiv papers and compared four agents on writing a MathArena blog post from scratch. The revised paper now includes the latter finding. Update the benchmark suite. We regularly consider new tasks and retire benchmarks that no longer provide useful comparisons.
  • Revise construction and evaluation procedures. Monthly experience informs changes to question generation, verification, prompts, tools, and resource limits as model capabilities improve.
  • Correct earlier releases. We actively investigate unexpected outputs. These checks have led to approximately six question corrections or removals. Although small relative to the full benchmark, these corrections matter for evaluation quality.
Reviewer_jomDOpenReview ↗
Rating
4
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Authors provide several benchmarks to evaluate LLMs on mathematical proofs, and indicate they will be continously maintained.

Critical strengths

If the claim that existing bencmarks in the area are unsatisfactory (which I am not familiar enough with to verify), their contribution is important for LLM evaluation. Among other benchmarks they provide a benchmark for testing whether LLM is too eager to accept false claims/say it has proven things that are false. They also indicate they will maintain/update benchmarks.

Critical weaknesses

None, assuming this is sufficient improvement over existing benchmarks.

Questions for authors

Do you plan to grow benchmarks beyond just mathematical proofs, to adjacent fields (e.g. physics or chemistry problems)?

2 comments

We thank the reviewer for their review. We are happy to see they found the reliability benchmark and continued benchmark maintenance valuable. In this rebuttal, we address their questions and concerns.

Q1: Are existing benchmarks insufficient?
We believe existing benchmarks leave important gaps. The closest alternatives illustrate why:

  • FrontierMath: its final-answer tiers have recently saturated. The evaluation questions are also private, which limits independent validation and reuse by researchers.
  • EternalMath, LiveMathematicianBench, and RealMath: these are close to ArXivMath and also propose refreshable benchmarks. However, regular maintenance and evaluation of current models have been limited. Only LiveMathematicianBench has released a second version, and this dates from June. Its leaderboard still identifies Gemini 3.5 Flash as the best model. These benchmarks also cover only final-answer reasoning, just one of the three capabilities evaluated by MathArena.
  • Lean AI Formalization leaderboard: this recent Lean theorem-proving benchmark from the creators of Lean is a useful comparison, but its questions remain fixed and submissions do not use a uniform execution setup. This creates contamination concerns and makes direct model comparisons difficult.

Additionally, MathArena is the only mathematical benchmark we are aware of with either of the following features:

  • Active maintenance with regular changes to questions and methodology to limit hill-climbing and maintain difficulty.
  • Evaluation of several complementary mathematical capabilities.

We also publish questions, model outputs, code, and detailed results, making the evaluations open to inspection and reuse.

Q2: Do you plan to expand into adjacent fields?
We have no immediate plans to expand into adjacent fields for three reasons. First, our expertise is in mathematics, where we can review benchmark construction more rigorously. Second, we have direct access to mathematicians who help us refine the benchmarks and discuss new tasks. Third, we have tried before to apply the ArXivMath pipeline to theoretical computer science and physics, but GPT-6 Astra solved 101 of the 104 generated questions. These simple extensions therefore offer little additional discrimination between strong models.

Thank you for your response, and for running additional experiments other reviewers requested. I will maintain my score.

Comments on the paper

1 comment

We thank all reviewers for their feedback. We are happy to see that reviewers highlight the value of regular updates (jomD, 6UXU, FtEo, A4ZJ), BrokenArXiv's complementary evaluation of reliability (jomD, FtEo, A4ZJ), and our prevention of evaluation errors (6UXU, LYU1, A4ZJ).

We address common questions here and provide an overview of the main additions to the paper. All these changes are marked in red in the updated paper.

  • Updated results: We included new versions of ArXivMath, BrokenArXiv, and ArXivLean. We also included the evaluations of the latest LLMs.
  • BrokenArXiv judge validation (§4.3): We manually labeled 100 samples to validate the LLM judge used in BrokenArXiv, obtaining 95% exact agreement. After review, we believe three mismatches were caused by a human labeling error.
  • Blog post conclusions (§5.3): We highlight the results of two of our blog posts.
  • Added discussion (§6): We discuss the possibility of biases in our benchmarks and why we believe they do not significantly affect results.
  • New generation models (§E.1): We regenerate ArXivMath September with three models, showing that results are consistent no matter which model is used for the generation procedure.
  • Harness comparison (§E.3): We compare scores on ArXivMath with two models using several harnesses. Scores between harnesses are very similar.
  • True statement analysis (§E.4): We show that GPT-6.1 Sol scores better on the true statements compared to the false statements.
  • Topic distribution (§E.5): We show the topic distribution in the benchmarks.
  • Qualitative analysis (§F): We added a qualitative analysis of some questions.

Q1: Does using evaluated models to select and filter problems introduce selection bias?
We regenerated September ArXivMath with Sol 6.1, Opus 5.5, and Gemini 4 Argon. We randomly selected 65 questions per generator and evaluated Sol (max), Opus (high), DeepSeek (max), and Argon (medium) without tools. Scores are shown in the table below. Rankings between questions are consistent between models, with only questions written by Gemini 4 Argon showing a more significant drop for all models except Sol. Historically, we have also not observed a model-ranking discontinuity when changing generators, including the switch from Gemini 3.1 Pro in June to GPT-6 Astra in August.

SolverSol 6.1 questionsOpus 5.5 questionsGemini 4 Argon questions
GPT-6.1 Sol (max)93.08%90.00%89.23%
Opus 5.5 (high)83.85%82.31%70.77%
Gemini 4 Argon (medium)76.15%72.31%51.54%
DeepSeek V4.1 Flash 10/17 (max)58.46%60.77%47.69%

Q2: Can you independently audit benchmark quality and report item error rates?
Yes, existing external use and feedback provide evidence about benchmark errors:

  1. Two companies searched for errors while evaluating their models on the March and June ArXivLean releases. We removed two of the original 89 questions after their analysis.
  2. Feedback from five other individuals led us to edit six questions. This is small compared to the 1,444 questions MathArena contains.
  3. We also inspect unexpected successes and failures. The website lists the five most surprising cases by comparing observed solve rates with IRT-predicted probabilities. Reviewing such cases has identified approximately three to six issues that were corrected.
  4. In the past, our benchmarks have regularly been saturated near 100% by models months after benchmark publication. If our benchmarks contained many errors, saturation performance would lie lower.

Unfortunately, we cannot add further independent audits during the rebuttal stage.

Q3: How do biases affect model scores?
Some reviewers identify several plausible sources of bias, e.g., LLM usage in our source papers. The answers below exclude the effects for some of them, but we cannot exclude all bias. However, two observations support that they likely have only small effects:

  • Our scores correlate strongly with FrontierMath Tier 4, which is constructed by mathematicians without relying on arXiv extraction. On the same 28 matched model configurations, Pearson correlation is 0.954 for ArXivMath.
  • We also receive consistent feedback from mathematicians that improvements on MathArena usually correspond to useful gains in their own work. This agreement is difficult to quantify, but provides an additional practical check on our measurements.

Further, there is an inherent tradeoff between independent expert authorship and scale. FrontierMath devoted substantial resources to just 50 Tier 4 questions, while First Proof, another famous initiative, is only able to construct 10 questions every few months. This makes it impossible to easily and regularly update such benchmarks. We believe there is a need for both expert authorship and scale, and the current active benchmarks in MathArena lean towards scale while making them as accurate as possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.