- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a new benchmark consisting of 300 proof tasks from academic papers in theoretical computer science. In addition, it develops an LLM-as-a-judge-based verifier that achieves 90% agreement with human experts. In the evaluation, the best performance achieved by the models and agents on this benchmark is around 70%, demonstrating that the benchmark poses a certain level of challenge.
Critical strengths
-
Whether AI can be used to prove theorems in research papers is a question of interest to many scientists, and the motivation behind this benchmark is clear.
-
The verifier developed in this paper achieves 90% agreement with human experts, which is a very good result for a verifier.
Critical weaknesses
-
Lack of empirical analysis of benchmark difficulty. The paper uses Dependency Rank to measure task complexity and increases difficulty by hiding intermediate lemmas. However, the authors do not sufficiently demonstrate whether model performance actually decreases as the rank increases or more dependencies are hidden.
-
Lack of fine-grained analysis of model reasoning capabilities. Beyond the overall success rates, the paper does not provide analyses of common failure modes, Pass@k, performance across different subfields of theoretical computer science, or the impact of long dependency chains on model capabilities.
Questions for authors
-
The authors design the benchmark to be self-contained, and Section 3.5 explicitly states that model responses must follow the provided context and cannot rely on external sources: “All justifications must follow from results present in the provided context and the solver is prohibited from accessing the original paper or external sources.” However, how is this restriction enforced in practice? A model might use a higher-level mathematical theorem memorized during pretraining to complete a proof, even if that theorem is not provided in the context. Can the verifier identify cases where a model uses mathematical results beyond the provided context? How can the authors distinguish between general mathematical knowledge that is permissible to use in a proof and external research results that should not be cited? Since the benchmark is derived from papers published between 2020 and 2026, is it possible that some models encountered the original proofs during pretraining, resulting in data contamination?
-
Are there any analyses of model performance on this benchmark beyond the final success rates? For example, are there any common failure modes? Is model failure correlated with the benchmark difficulty factors discussed by the authors, such as dependency rank and the hiding of intermediate lemmas? Can models improve their success rates through multiple attempts (Pass@k)? Are the specific differences or challenges involved in proving theorems from research papers, as opposed to mathematical theorems in general, reflected in the solutions generated by LLMs?
-
In Section 4, the authors introduce an LLM-as-a-judge-based verifier agent to evaluate the tasks, achieving 90% agreement with human expert evaluations. Could this verifier be used to supervise model training or being used as a test-time verification to improve model performance on this benchmark?
-
Were the data used to optimize the verifier prompt with GEPA independent of the data used to report the final 90% accuracy? Does the verifier achieve comparable accuracy across proofs of different difficulty levels and types?
I would be happy to reconsider my rating if the authors can adequately address the questions and concerns raised above.