- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark and evaluation protocol for retrieving relevant theorems during theorem proving, and provides evaluations of some models.
Critical strengths
-
The task setting is interesting and appears to have some novelty.
-
The evaluations reveal some interesting findings about certain models.
Critical weaknesses
-
The benchmark has two evaluation setups: raw and oracle. Both have major issues.
-
The raw setting is unrealistic and unreasonable. The model can query google scholar only once, and can only read the metadata of only the top 20 results (not full text), and can only read the full text of one paper. This is unnecessarily restrictive. It is not even clear top human mathematicians can do this with a reasonable accuracy.
-
The oracle setting uses LLMs as judges, which introduces questions about how reliable the judges are and the validity of the evaluation. But LLMs as judges are not necessary, because a model can be given the full text of the source paper, and asked to pin point the exact line, which is knowable at the time of benchmark construction.
-
There's no human baseline. It is not clear how hard these tasks are for human experts.
-
Figuring out what theorems are needed can require a lot of test-time compute, because it can involve working out the full strategy of the proof. The paper does not provide any such analysis about how the results are affected by the allowed test time compute.
Questions for authors
None.