- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a extension to MathArena benchmark by combining three types of mathematical tasks: ArXivMath for final-answer questions, BrokenArXiv for responses to false mathematical claims, and ArXivLean for formal proof generation in Lean. While ArxivMath and ArxivLean type benchmarks have been proposed before, BrokenArxiv adds a complementary evaluation of whether models reject false claims rather than invent proofs. Its main contribution is bringing these evaluations together with regular benchmark updates. The authors state that the benchmark will be updated at regular frequency with latest research papers. The experiments show substantial improvements in mathematical performance and restraint when facing false claims, while formal proof generation remains harder.
Critical strengths
- The idea of reducing contamination by regularly updating with latest papers and evaluating models which wouldn't have seen these new research is very effective.
- The authors have taken measures to prevent internet search which can conflate the models own knowledge.
- The idea of BrokenArxiv is very interesting and novel.
- The pipeline of question generation and validation check is very thorough and explained well.
- The authors address the prohibitive cost of full benchmark evaluations by using an IRT model to blend observed scores with statistically predicted performance for unattempted questions, while fitting a log-additive model to estimate missing costs separately.
Critical weaknesses
- Since the benchmark uses research work, which in some can be generated by heavy AI assistance. Such data points may favor problems which can be easier to current models, potentially biasing the benchmark’s coverage.
- In final answer benchmark, the evaluation does not bring out the mechanism of answering the question. The result could be obtained by guessing or shortcut. Since the benchmark uses LLM as a judge, it would also benefit to look at how LLM arrived at the answer, its reasoning etc.
- Although the paper reports high agreement between two BrokenArxiv judges, it does not show how judge choice affects per-model scores or leaderboard rankings. Reporting these differences would strengthen confidence in the comparisons.
- Since this paper is more towards introduction of a continuously updated benchmark, it would be really useful to know what is the cost of creating the problems, the screening, the validation etc.
Questions for authors
- How do the authors validate that every BrokenArxiv statement is false under precisely the assumptions given to the model? Although the construction includes automated checks and human review, have they independently audited the questions or investigated model generated proofs for possible errors in the benchmark labels?
- Can the authors comment on how a source paper which heavily relied on LLMs to find a proof or solution, can affect the benchmark quality and ease of newer models solving them?
- Can the authors provide more details on the rationale behind using different models within the source extraction pipeline across the distinct benchmark subsections, and how this choice might impact consistency?
- Can authors report some per-model scores with different judges?
- Does the three formats of benchmark problems here cover all or most of the type of research in Mathematics? Can you provide a distribution chart on any underrepresented areas if any and comment on the same?
2 comments
Response to author rebuttal
Thank you very much for clarifying my questions. I believe all my questions are answered. I would like to keep my current rating.
Rebuttal
We thank the reviewer for their review. We are happy to see they found the refreshed research questions, BrokenArXiv task, and documented generation and validation valuable. In this rebuttal, we address their questions.
Q1: How do you verify that every BrokenArXiv statement is false under precisely the assumptions provided to the model?
A large part of our automated pipeline is constructed to stress the importance of including all necessary assumptions, as we think this is very important. We also inspect strong-model outputs after evaluation to investigate claims of a proof. Earlier releases sometimes contained ambiguous statements that these checks helped identify. Across the August, September, and October versions, we have not found an ambiguous formulation.
Q2: Have the statements been independently audited, and have model-generated proofs been investigated for possible errors in benchmark labels?
Please see Q2 of our main response.
Q3: How might heavy LLM assistance in source papers affect benchmark quality, coverage, and ease for newer models?
Please see Q3 in our main response.
Q4: Why are different models used across extraction stages and benchmark subsections, and how does that choice affect consistency?
Two practical considerations explain the changing model choices:
In Q1 of our main response, we show that model choice does not significantly affect consistency.
Q5: Can you report per-model scores and changes in leaderboard rankings under different judges?
Yes. We re-graded September BrokenArXiv responses with Sol 6.1 (high), Opus 5.5 (high), and Gemini 4 Argon(high), using the existing judge prompt and Gemini 3.8 Flash baseline. Spearman rank correlations are 0.9727, 0.9909, and 0.9909, respectively. Appendix E.2 details the experiment with full results and model scores.
Q6: How broadly do the three task formats cover mathematical research?
We added three pie charts grouping source papers' primary arXiv topics into mathematical areas, covering all questions since August. Geometry and Topology is the largest group for ArXivMath at 48/157 (30.6%), Analysis for BrokenArXiv at 42/163 (25.8%), and Combinatorics for ArXivLean at 27/53 (50.9%). Overall, we find that all topics are represented, but some are less prominent than the raw distribution of arXiv. For instance, in September, analysis formed 33% of papers, while only being represented with 6% in arXivMath and 13% in ArXivLean. The distribution of BrokenArXiv is closer to the actual source distribution, because its restrictions in the question generation procedure are more independent of the field.
Q7: Can you assess whether correct final answers arise from valid reasoning, guessing, or shortcuts?
Great question! We do not evaluate the full reasoning behind model answers for two reasons:
Nevertheless, we asked an AI agent to categorize a subset of ArXivMath responses for the September benchmark, assuming the model outputs were truthful about how they obtained the answer. Among judge-correct responses, 42.5% claim an analytical derivation, 18.7% use an exact computation, 27.3% use numerical reconstruction or extrapolation, and 7.7% use a heuristic guess (mostly between two unexcluded answers). Importantly, we explicitly request a best guess because some models otherwise gave multiple candidates, which earned no credit and disadvantaged truthful models.
Q8: What does creating and maintaining the benchmarks cost?
The budget is roughly \100perbenchmarkeditionplusaboutonedayofhumanreview.Asanexample,BrokenArXivSeptembercost111.93: \18.58forscreening,60.40 for generation, and \32.95forverification.Atthebenchmarksizewehave(roughly50−60questions),asingleevaluationpasscostsroughly400 for an expensive model and \$50 for a typical model.