acceptodds
Withdrawn from ICLR 2027

ReMath: Benchmarking Theorem Retrieval in Research-Level Mathematics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mathematical proofs often use known theorems. We introduce ReMath, a bench- mark for retrieving theorems needed in unfinished research proofs. Given the global setup and a local proof prefix, a large language model (LLM) uses external search to retrieve a theorem statement with its source. The source lets researchers check the statement and credit prior work. The retrieved theorem must justify the intended next step using the available facts. The task allows suitable alternative theorems and sources. We build a dataset from published papers across five mathe- matical domains. Across the tested models, most outputs that pass the grounding check do not pass the reference-based sufficiency check. Case studies show that a theorem may assume a fact the proof still needs to establish, while a conclusion weaker than the reference may suffice for the next step. Evaluation must therefore check how the theorem applies to the intended step, including whether the theorem concerns the objects in the proof. We release the dataset and code for construction and evaluation

Primary area:
datasets and benchmarks
Keywords:
AI4Math, Retrieve, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bppnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark and evaluation protocol for retrieving relevant theorems during theorem proving, and provides evaluations of some models.

Critical strengths
  • The task setting is interesting and appears to have some novelty.

  • The evaluations reveal some interesting findings about certain models.

Critical weaknesses
  • The benchmark has two evaluation setups: raw and oracle. Both have major issues.

  • The raw setting is unrealistic and unreasonable. The model can query google scholar only once, and can only read the metadata of only the top 20 results (not full text), and can only read the full text of one paper. This is unnecessarily restrictive. It is not even clear top human mathematicians can do this with a reasonable accuracy.

  • The oracle setting uses LLMs as judges, which introduces questions about how reliable the judges are and the validity of the evaluation. But LLMs as judges are not necessary, because a model can be given the full text of the source paper, and asked to pin point the exact line, which is knowable at the time of benchmark construction.

  • There's no human baseline. It is not clear how hard these tasks are for human experts.

  • Figuring out what theorems are needed can require a lot of test-time compute, because it can involve working out the full strategy of the proof. The paper does not provide any such analysis about how the results are affected by the allowed test time compute.

Questions for authors

None.

Reviewer_HfqgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a benchmark that measures how good models are at retrieving a theorem and its source for the next step of a given partial proof. They extract the dataset and offer the formalization and the judge prompts needed to perform the benchmark.

Critical strengths
  • Clear writing
  • Very honest about limitations
  • Useful benchmark for marginalized groups such as mathematicians
Critical weaknesses
  • The judge compares outputs to the reference statement, not the proof step itself, so it rejects valid theorems that may be weaker but sufficient
  • About 29% of successes are wrong or come from citing the paper itself.
  • Scores are very low, and the cited source can't be retrieved in most cases, and the search budget is just one short query. This means the bottleneck might not be the models.
Questions for authors
  • What are the ToolAcc counts after excluding the citing paper as a source?
  • Why limit to one Scholar query of at most 5 words and a single opened source? This is far below how researchers actually search.
Reviewer_YJqfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Re²Math, a benchmark evaluating whether large language models can retrieve a theorem and its source to justify the next step in an unfinished mathematical proof.

  • Dataset: 860 proof-gap instances across five mathematical domains, with 200 selected for evaluation.
  • Results: Across seven models, only 1%–7% of outputs pass both the source-grounding and theorem-sufficiency checks.
  • Key finding: Finding a real, relevant theorem is insufficient; its assumptions must hold, and its conclusion must advance the current proof.
  • Main limitations: Reference-based scoring and LLM judges can misclassify answers, and some instances withhold information needed to identify the intended next step.

The paper emphasizes evaluating how a theorem applies to the current proof, beyond whether the model retrieves relevant literature.

Critical strengths
  1. The task addresses a meaningful research need. The paper studies identifying and retrieving a theorem needed in an unfinished research-level mathematical proof, together with a verifiable source. This goes beyond topical literature retrieval and closely reflects mathematicians’ practical needs. The dataset spanning five mathematical domains provides a valuable foundation for studying this capability.

  2. The distinction between theorem applicability and evaluation errors is insightful. The case studies expose missing premises, mismatched mathematical objects, and reference-based rejection of potentially sufficient results. These analyses demonstrate that source support does not establish applicability and provide concrete guidance for improving mathematical retrieval evaluation, strengthening the paper’s methodological value.

Critical weaknesses
  1. The primary scoring criterion is misaligned with the task definition and lacks direct validation on the main experimental outputs. The task permits any theorem sufficient to advance the intended proof step, but scoring requires the returned theorem to be equivalent to or stronger than the reference, potentially rejecting mathematically valid answers systematically. Moreover, the human audit uses historical outputs that cannot be linked individually to the main scoring run, so it does not directly establish the reliability of the reported scores. Consequently, it remains unclear how much the low pass rates reflect model limitations versus evaluation bias, weakening confidence in the central empirical findings.

  2. Some instances withhold information needed to identify the retrieval target. The paper acknowledges that the intended next step may become clear only in the hidden proof continuation. A model could therefore propose a reasonable way to advance the proof yet fail because it does not anticipate the original author’s next move. Without quantifying the prevalence of this ambiguity, the benchmark risks conflating theorem retrieval ability with guessing a hidden proof plan, undermining its validity as a reliable evaluation benchmark.

Questions for authors
  1. Have the main experimental scores been directly validated by human experts? Have domain experts reviewed outputs passing both checks and a sample of rejected outputs from the main scoring run? If so, please describe the sample, error rates, and implications for the results. Is there a verifiable correspondence between the historical audit and the main experimental outputs?

  2. How does the implemented scoring procedure handle theorems that are weaker than the reference but sufficient for the intended step? Can such answers receive positive labels? If so, how does this reconcile with the stated “equivalent to or stronger than” criterion? If not, is there evidence that this restriction does not substantially affect the reported pass rates?

  3. How many Eval-200 instances lack enough visible information to identify the intended next step? Has the prevalence of this ambiguity been assessed, and do the main findings remain valid when such instances are excluded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.