- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper’s goal is to use small, local language models as math critics. The main approach, named Co-PEER, is to use a language model to find counterexamples to the given claim. This approach is motivated by the idea that finding counterexamples to an argument is easier than constructing a complete proof. The input includes a claim and its assumptions, while the output is a counterexample, a proof-of-error by Lean, and “a review of whether the objection applies to the source and what it establishes”.
The paper also introduces a benchmark, named BugBank, which is built from running Co-PEER. It includes 147 human-approved (claim, assumptions, counterexample) tuples across 137 sources. Findings from running LLMs on this benchmark are also presented.
Critical strengths
- Section 6 apparently documents examples where the founded counterexample helps advance proof generation. But the presentation requires substantial improvement.
Critical weaknesses
- The paper lacks a formal problem setup, and its presentation is very difficult to follow. A few examples are given below:
- The motivation is poorly presented. The introduction argues: “Constructing a complete proof can demand substantial reasoning, while a counterexample or false step may provide a shorter route to identifying an error.” This statement is unclear whether it refers to the theorem-proving process or the proof-verification process.
- Several terms are used without clear definition or are difficult to infer from the context; for example, the term “translation” in line 082
- The exact form of input and output of the Co-PEER framework is not rigorously presented. It’s unclear from the writing whether the input is a mathematical proof or a conjecture.
- The paper does not compare Co-PEER against existing baselines or alternative approaches, making it difficult to evaluate the relative effectiveness of the proposed framework.
- The details regarding sources of BugBank are not presented. It’s unclear in terms of what kind of claims or which domains are included.
Questions for authors
Please refer to the weaknesses.
Thank you for the feedback. We have rewritten the abstract and main text, made the setup and sources easier to find, and added comparisons with other search methods.
1. What problem are we solving?
We aim to find a concrete counterexample for either an error in a proof of a true statement or to the statement itself and formally verify this counterexample. A theorem can be true even when its proposed proof contains a false step. Proving the theorem another way does not identify that mistake. A counterexample to a step in the proof does.
Co-PEER handles both false proof steps and false conjectures. It is not limited to small models. We do not claim that finding counterexamples is always easier than proving theorems.
2. What are the input, output, and formal setup?
The submitted paper gave the formal setup in Appendix Q.1 (p. 79) and pointed to it in Section 2.1. We have now placed it directly in Section 2.
Input: A problem statement and proof, or a standalone conjecture. For a proof, Co-PEER first selects a claim and states its assumptions.
Goal: For a claim ∀x, A(x)⇒C(x), find an example w for which A(w) is true and C(w) is false. In plain terms, the example meets the assumptions but breaks the claim.
Output: A counterexample report, called a dossier. It contains the original claim, its assumptions, the counterexample, a Lean proof, and a review checking that the counterexample applies to the original claim. It also states whether the error is in one proof step or in the theorem itself.
“Translation” means writing the claim, assumptions, and counterexample in Lean. The submitted Section 2.1 introduced this meaning in lines 076–083. Our revised Section 2.3 explains it directly.
Lean checks the formal proof. A separate human review checks that the formal statement matches the original text. These are different checks. Finding no counterexample does not prove that a claim is true.
Section 3 now explains the workflow step by step: select a claim, find a counterexample, check it with a separate program, prove it in Lean, and review the counterexample correctly disproves the original proof step or statement under the correct assumptions.
3. What comparisons have we added for this rebuttal?
The submitted Section 4.1 included simple baselines for selecting error lines. We now also compare methods for finding counterexamples.
We tested four models using the following methods against our method (verified search) on 60 false claims not used to develop the search. The topics of the 60 false claims include graph theory, algorithms, algebra, number theory, and probability. Each model tried each claim three times per method. Success means returning a counterexample that passes the Python checker.
VS more than doubled success over directly asking: 34.7% to 78.2%. All methods had the same maximum 65,536-token cap, but naturally used different average amounts due to the prompts used in each method.
Checking also helped without generating more answers. From exactly the same saved candidates, checker-based selection succeeded in 72.1% of trials, compared with 48.2% for voting. That is 172 more successes out of 720, at even a smaller model-token cost.
These new experiments can be found in Section 4.4 and Appendix D.7.
4. Where do counterexamples in BugBank come from?
The source collections and counts appeared in Table 20 of the submitted paper (p. 72). We now summarize them in Section 4.1 and Appendix C.7.1.
The 147 human-approved dossiers come from Reddit (42), the two Stack Exchange collections (34), ProofWiki (28), QEDBench (25), arXiv (15), and reference-proof additions (3).
They cover graph theory, number theory, algebra, logic and sets, algorithms, analysis, combinatorics, geometry and topology, and probability. Table 7 gives the subject counts.