- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether structured mathematical knowledge can improve LLM-based theorem proving in Lean 4. It introduces MathKG, a knowledge graph of 364 Mathlib theorems and definitions connected by 9,434 typed semantic relations, and MathAgent, a multi-agent framework that uses the graph alongside Mathlib retrieval to support formal proof generation. Experiments on miniF2F, PutnamBench, and MathOlympiadBench show that Lean-specialized training substantially outperforms inference-time knowledge augmentation, while static augmentation provides limited and model-dependent benefits, sometimes even hurting stronger models. However, different augmentation strategies solve complementary problems: an oracle combining their successes achieves higher coverage, with particularly large relative gains on harder benchmarks. These findings suggest that the key challenge is not simply providing more knowledge, but adaptively selecting the most useful knowledge and augmentation strategy for each problem.
Critical strengths
- The paper introduces MathKG, a structured knowledge graph of mathematical concepts and theorems, integrated into a multi-agent framework for Lean 4 theorem proving.
- It systematically compares knowledge augmentation strategies across models with different capabilities, revealing that their effectiveness is model-dependent and can even be negative for stronger models.
- It demonstrates the complementarity of different augmentation strategies through oracle-union experiments, motivating future research on adaptive knowledge selection.
Critical weaknesses
- The benefits of knowledge augmentation are limited and inconsistent across models, with some stronger models experiencing performance degradation.
- The oracle-union results represent an upper bound rather than a practical adaptive selection method, leaving the key challenge of selecting the best augmentation strategy unresolved.
- The evaluation focuses primarily on established theorem-proving benchmarks, while the proposed open-ended mathematical exploration capabilities are not experimentally validated.
Questions for authors
- Why does knowledge augmentation help weaker models but hurt stronger models? English
The results show that knowledge augmentation can substantially improve weaker models but provides limited or even negative gains for stronger models. Can the authors analyze the underlying reasons for this difference? For example, is it due to knowledge redundancy, context interference, or differences in the models' ability to utilize structured knowledge?
- How does MathKG differ from existing graph-based theorem-proving methods? English
How does MathKG differ fundamentally from existing graph-based theorem-proving and premise-retrieval methods, such as LeanDojo/ReProver and recent graph-based retrieval approaches? Can the authors clarify the novelty of the proposed knowledge representation and provide direct empirical comparisons with these methods?
We thank the reviewer for the insightful feedback and for their patience. Points shared with the other reviewers are answered in the general response.
The oracle. See point 1 of the general response: the union is the aggregation of pipelines that are all implemented and run, not a learned selector, and point 2 for the equal-budget comparison with independent restarts.
Relation to LeanDojo/ReProver and graph-based retrieval. ReProver retrieves premises for the current proof state with a dense retriever trained on premise usage, inside a tactic-level prover. MathKG is a different object used at a different point: named theorems and definitions with natural-language statements and verified Mathlib anchors, connected by LLM-inferred semantic relations under a typed ontology rather than by premise co-occurrence, and supplied once, before the first attempt, to an off-the-shelf prover. Recent graph-based retrievers (Khalov et al. and Ataeva et al.; KG-Prover) are closer in spirit and we cite them as such; our contribution relative to them is the cross-model comparison, not a retrieval method. On the empirical comparison: the Mathlib condition was deliberately lexical, and learned premise selection is out of the paper's scope (Section 5.3 and the revised Limitations section). A head-to-head with ReProver would change the prover and the proof procedure at once, not only the knowledge source, so it would not answer the paper's question; we leave it as future work.
Open-ended exploration. Agreed: the discovery loop (Explorer → Conjecturer → Prover → Integrator) is not evaluated in the paper, and the revised Limitations section says that all results are for targeted proving. Before submission we did run the system to open problems: open conjectures from the Formal Conjectures repository, 9 Erdős problems (pass@32) and 10 OEIS conjectures (pass@64) with Sonnet 4.6, and all 134 OEIS conjectures with Goedel-Prover-V2-32B (pass@64), under the four augmentation modes. What we learned there shaped our view of what open-ended proving needs: generating a whole proof per attempt is the wrong unit for these problems, because the retry loop gives no credit for the parts of an attempt that compile. That argues for stating the goal and its subgoals explicitly, keeping the parts that compile and retrying only the subgoals that fail. Supporting that direction means changing which part of the execution the system captures and iterates on, from whole proofs to subgoals, and since the targeted pipeline and its evaluation were already the heavy part of this work, we chose not to split the paper's focus.