acceptodds
Under review as a conference paper at ICLR 2027

Co-PEER: Auditing Mathematics with Proofs of Error

Abstract

A precise counterexample can reveal a recurring failure pattern, motivate a correction, and lead to a theorem about an entire family. We introduce Co-PEER, a workflow for developing and checking such counterexamples, alongside BugBank, a collection of 147 dossiers from 137 source units. Each dossier connects a source claim and its assumptions to a counterexample, a Lean-checked proof, and a separate assessment of whether the objection applies to the source. We also introduce a novel benchmark based off BugBank. On 100 texts known to contain an error, the best-performing model selects a line marked incorrect as its first choice in 73.0% of cases, compared with 18.0% for the strongest rule-based baseline. In author-led mathematical follow-ups with AI-assisted proofs, we extend individual witnesses into complete characterizations and sharp bounds. We characterize and count frozen six-colorings of doubled odd cycles, where no vertex can legally change color: for odd cycle length , they exist exactly when three divides , yielding colorings for labeled vertices and colors. We also prove that is the optimal limiting fraction of ordered triples with three different colors within a two-threshold integer-coloring family, exceeding the conjectured bound of .

Primary area:
foundation or frontier models, including LLMs
Keywords:
counterexamples, math verification, Lean

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_o6EcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper’s goal is to use small, local language models as math critics. The main approach, named Co-PEER, is to use a language model to find counterexamples to the given claim. This approach is motivated by the idea that finding counterexamples to an argument is easier than constructing a complete proof. The input includes a claim and its assumptions, while the output is a counterexample, a proof-of-error by Lean, and “a review of whether the objection applies to the source and what it establishes”.

The paper also introduces a benchmark, named BugBank, which is built from running Co-PEER. It includes 147 human-approved (claim, assumptions, counterexample) tuples across 137 sources. Findings from running LLMs on this benchmark are also presented.

Critical strengths
  • Section 6 apparently documents examples where the founded counterexample helps advance proof generation. But the presentation requires substantial improvement.
Critical weaknesses
  • The paper lacks a formal problem setup, and its presentation is very difficult to follow. A few examples are given below:
    • The motivation is poorly presented. The introduction argues: “Constructing a complete proof can demand substantial reasoning, while a counterexample or false step may provide a shorter route to identifying an error.” This statement is unclear whether it refers to the theorem-proving process or the proof-verification process.
    • Several terms are used without clear definition or are difficult to infer from the context; for example, the term “translation” in line 082
    • The exact form of input and output of the Co-PEER framework is not rigorously presented. It’s unclear from the writing whether the input is a mathematical proof or a conjecture.
  • The paper does not compare Co-PEER against existing baselines or alternative approaches, making it difficult to evaluate the relative effectiveness of the proposed framework.
  • The details regarding sources of BugBank are not presented. It’s unclear in terms of what kind of claims or which domains are included.
Questions for authors

Please refer to the weaknesses.

1 comment

Thank you for the feedback. We have rewritten the abstract and main text, made the setup and sources easier to find, and added comparisons with other search methods.

1. What problem are we solving?

We aim to find a concrete counterexample for either an error in a proof of a true statement or to the statement itself and formally verify this counterexample. A theorem can be true even when its proposed proof contains a false step. Proving the theorem another way does not identify that mistake. A counterexample to a step in the proof does.

Co-PEER handles both false proof steps and false conjectures. It is not limited to small models. We do not claim that finding counterexamples is always easier than proving theorems.

2. What are the input, output, and formal setup?

The submitted paper gave the formal setup in Appendix Q.1 (p. 79) and pointed to it in Section 2.1. We have now placed it directly in Section 2.

Input: A problem statement and proof, or a standalone conjecture. For a proof, Co-PEER first selects a claim and states its assumptions.

Goal: For a claim ∀x, A(x)⇒C(x)\forall x,\ A(x)\Rightarrow C(x), find an example ww for which A(w)A(w) is true and C(w)C(w) is false. In plain terms, the example meets the assumptions but breaks the claim.

Output: A counterexample report, called a dossier. It contains the original claim, its assumptions, the counterexample, a Lean proof, and a review checking that the counterexample applies to the original claim. It also states whether the error is in one proof step or in the theorem itself.

“Translation” means writing the claim, assumptions, and counterexample in Lean. The submitted Section 2.1 introduced this meaning in lines 076–083. Our revised Section 2.3 explains it directly.

Lean checks the formal proof. A separate human review checks that the formal statement matches the original text. These are different checks. Finding no counterexample does not prove that a claim is true.

Section 3 now explains the workflow step by step: select a claim, find a counterexample, check it with a separate program, prove it in Lean, and review the counterexample correctly disproves the original proof step or statement under the correct assumptions.

3. What comparisons have we added for this rebuttal?

The submitted Section 4.1 included simple baselines for selecting error lines. We now also compare methods for finding counterexamples.

We tested four models using the following methods against our method (verified search) on 60 false claims not used to develop the search. The topics of the 60 false claims include graph theory, algorithms, algebra, number theory, and probability. Each model tried each claim three times per method. Success means returning a counterexample that passes the Python checker.

MethodSuccess rateAverage model tokens per trial
Directly ask34.7%5,315
Ask in a fixed answer format42.2%3,560
Self-consistency-style voting (Wang et al., 2023)48.2%53,891
Reflexion-style feedback (Shinn et al., 2023)70.3%19,126
Verified restarts (Brown et al., 2024)72.5%18,514
PAL/Program of Thoughts-style search (Gao et al., 2023; Chen et al., 2023)74.6%17,105
Our Verified Search (VS)78.2%21,111

VS more than doubled success over directly asking: 34.7% to 78.2%. All methods had the same maximum 65,536-token cap, but naturally used different average amounts due to the prompts used in each method.

Checking also helped without generating more answers. From exactly the same saved candidates, checker-based selection succeeded in 72.1% of trials, compared with 48.2% for voting. That is 172 more successes out of 720, at even a smaller model-token cost.

These new experiments can be found in Section 4.4 and Appendix D.7.

4. Where do counterexamples in BugBank come from?

The source collections and counts appeared in Table 20 of the submitted paper (p. 72). We now summarize them in Section 4.1 and Appendix C.7.1.

The 147 human-approved dossiers come from Reddit (42), the two Stack Exchange collections (34), ProofWiki (28), QEDBench (25), arXiv (15), and reference-proof additions (3).

They cover graph theory, number theory, algebra, logic and sets, algorithms, analysis, combinatorics, geometry and topology, and probability. Table 7 gives the subject counts.

Reviewer_mEnuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Co-PEER, a pipeline for detecting and proving errors in mathematics. The pipeline consists of a critic (an LLM) which detects the erroneous line in a source text and produces a counterexample called the witness. A separate program then tests whether the witness satisfies the assumptions and violates the conclusion of the claim. The witness and the claim are then translated into Lean, where they are machine-checked. A second human/model-assisted review of the proof of error is then done to ensure that the Lean code actually corresponds to the witness and claim, that the assumptions are fulfilled, and whether the error establishes a false intermediate step or a false theorem. All steps are collected into a dossier and deposited into BugBank. The initial 425 dossiers were reduced through multi-stage human review, where only 147 unambiguously approved dossiers made the final cut in BugBank. 100 texts known to contain an error were then extracted from this collection to serve as a benchmark. 13 models' ability to detect the marked erroneous line was tested using this benchmark. The best model selected a marked erroneous line as its first choice in 73% of cases, compared with 18% for the strongest rule-based baseline. Lastly, several mathematical findings are presented, which the authors made by following up on witnesses produced by Co-PEER.

Critical strengths
  1. The article does not only validate that the pipeline works as intended but also shows several examples where it practically helped the authors uncover errors in source texts, including errors in model-generated proofs that had received perfect human rubric scores, and a counterexample to a published conjecture.
  2. The pipeline separates a proof of error into distinct parts, which are each handled and verified separately. An LLM proposes the witness, a separate program checks it, Lean machine-checks the refutation, and human/model-assisted review verifies the correspondence and the nature of the discovered error. This separation is a contribution that reduces the risk of errors passed through the pipeline or dissonance between steps.
  3. A rigorously controlled benchmark is presented. It has gone through several steps of human verification to ensure correctness, and the tests using this benchmark demonstrate that it discriminates between models in their ability to select a line marked incorrect in a source text
Critical weaknesses
  1. The core auditing loop builds on existing approaches. FaithSieve already combines Lean checks with assessments of whether formal statements match the source, and step-level error benchmarks already exist. Co-PEER's advance is primarily the added layers of human verification, and the packaging into a reusable resource, rather than a fundamentally new method. The approach is also not scalable. Each dossier required multistage human review, so extending to larger corpora would demand substantial human effort.
  2. Several limitations of the evaluation are acknowledged but not resolved. First, the benchmark comparisons are not budget-matched. The models differ widely in usable answers and recorded tokens so Hit@1 mixes detection ability with answering within budget. Second, the benchmark cannot measure false positives, since all 100 texts contain certified errors by design, and Hit@1 scores agreement with the annotated line rather than error detection in general. In the authors' own analysis of 30 misses, 12 were valid objections to other errors but received no credit. Together these limit what the 73% headline establishes.
Questions for authors
  1. The benchmark texts derive from public sources. Did you check whether the evaluated models' training data includes these texts? Could such exposure affect the Hit@1 scores?
  2. Section 4.1 reports 13 evaluated model configurations, but Section 4.2 and Table 2 report results for 12 model configurations (plus two baselines). Is there a reason for this?
3 comments

Thank you for the careful review and for recognizing the value of our methods and benchmark. We rewrote the abstract and Sections 2–3 and added experiments addressing your concerns.

1. Contribution and relation to FaithSieve

Thank you for raising this comparison. FaithSieve is relevant work, which we cite in our paper. The difference in methodology is not that Co-PEER uses Lean or adds human review; it is how the counterexample is found, checked, and used.

A different search and verification process. FaithSieve organizes proof steps into a tree, translates selected claims into Lean, and tries to prove or disprove them. Co-PEER first finds and independently checks a counterexample, then builds its Lean proof. A search program uses exact arithmetic to find the example. A separately written program checks it against the claim’s assumptions and conclusion. Lean then checks the formal proof (Section 3.1; Appendix K).

A different requirement for a completed result. FaithSieve uses formal evidence to identify the first error or judge a proof error-free. Co-PEER produces a checked counterexample to a specific claim. Lean verifies the counterexample under the encoded assumptions; a separate review checks that it refutes the original claim. We also state whether it disproves one proof step or the theorem itself. Co-PEER does not need to find the first error or build a proof-state tree (Section 2).

A different role for formal evidence in benchmarking. FaithSieve uses Lean checks to help decide where a proof is wrong. BugBank uses checked counterexamples and Lean proofs to justify the answers used to score models in our benchmark. The tested model sees neither the counterexample nor its certificate. We also reuse these counterexamples in proof-repair experiments and mathematical follow-ups (Sections 4–5).

Thus, Co-PEER contributes a distinct counterexample-construction and validation workflow, together with a benchmark constructed from its methods, not simply additional human review.

2. Human effort and scalability

Human review was only required for the benchmark core, not every BugBank record or each new search. We designed the benchmark so that the reviewed benchmark can test new models without repeating the human review. Expanding the core still requires human checks of source claims, but once a Lean statement is reviewed and in the benchmark, it need not be reviewed again (Sections 3–4).

3. Budgets and usable answers

All models had the same 16,000-token output allowance and 600-second limit, but token use naturally differed. Gemini’s 73% counts failed responses as misses; it scored 78.5% (73/93) on usable answers. (Appendices F.2–F.4).

For this rebuttal, we tested three token limits using the same experimental setup. The table shows correct error-line selections out of 100 texts:

Model4,000 tokens8,00016,000
GPT-OSS 120B485151
GPT-OSS 20B323941
Nemotron 3.5 Lightning3619

More tokens improved scores, but the ranking stayed the same. Appendix F.4 also reports how often each model returned a usable answer.

4. Partial labels and false positives

The benchmark accepts multiple error lines, not a single target line. We checked and expanded the accepted sets to include restatements and instances of the challenged claim; 57 of the 100 texts have multiple accepted lines. The reported 73% includes choices matching any accepted line, not just a single one.

Our reported 12/30 analysis concerns an earlier 191-text evaluation, not the current 100-text benchmark. (Section 4.2; Appendices D.7 and F).

For this rebuttal, we also tested our verified search (VS) framework on 60 additional problems. On 60 false claims not used to develop the search, success rose from 34.7% to 78.2% across four local models when directly asking for a counterexample versus using VS. Both methods had a 65,536-token allowance. (Section 4.4; Appendix D.7).

True claims: On 24 true claims tested for this rebuttal, VS returned 0 false counterexamples in 96 trials, compared with 10/96 when asked directly and 44/96 with voting.

5. Training exposure

We have not verified whether these texts appeared in the models’ training data. However, our new VS experiments compare the same model checkpoint on the same claims, with and without search, so training history is held fixed. These comparisons measure the benefit of the search procedure for the tested models (Appendix D.7).

6. Thirteen models versus twelve

We evaluated 13 model configurations: 12 appear in Table 1, and Haiku 4.5 appears in Appendix F. Haiku scored 31.0% when session-limit interruptions counted as misses and 56.0% after those interrupted runs were resumed. These are two results for the same model, not two different models. Table 26 therefore contains 14 model-result rows for 13 configurations.

Thanks for the efforts that went into this rebuttal. I am keeping my score.

Thank you for the reply, for reading our rebuttal, and for the positive assessment!

Reviewer_1sLqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the use of proofs of error, i.e. counterexamples, and proposes BugBank.

The paper shows how such proofs can be of use in proving specific results.

The claimed gains are interesting, but not remarkable.

Critical strengths

The paper covers an interesting topic of research.

Critical weaknesses

Although the paper is well-motivated, the experimental results come through as lacking.

Questions for authors

None.

1 comment

Thank you for recognizing the paper’s motivation and topic. To address your concern that the experiments were lacking, we added tests of counterexample search itself (not just error-line identification) and a controlled comparison at different token limits using the same model-generated answers (Section 4.4; Appendix D.7). Throughout this rebuttal, we call our method Verified Search (VS).

1. Verified search more than doubled success across four local models.

Across all six models, verified search increased the average success rate from 49.5% to 83.9%; this is a gain of 34.4 percentage points.

ModelDirectly askVerified search
DeepSeek R1 Qwen45.6% (82/180)83.3% (150/180)
Gemma 3 27B11.7% (21/180)66.1% (119/180)
Phi-412.8% (23/180)70.6% (127/180)
Qwen3 32B68.9% (124/180)92.8% (167/180)
GPT-OSS 120B76.1% (137/180)96.7% (174/180)
GPT-OSS 20B82.2% (148/180)93.9% (169/180)
Average across all six models49.5% (535/1,080)83.9% (906/1,080)

Each model ran three trials on each of the same 60 false claims, giving 180 trials per procedure. A trial succeeded only if the counterexample passed the task-specific Python checker. The average gives equal weight to each model. The procedures shared a 65,536 token cap.

This allowance governs new calls, not an ongoing response; actual token use differs.

Directly ask: The model answers once without a required format or checker feedback. We convert proposed counterexamples into the checker’s format for evaluation.

2. Checking also helped when the model answers and token use were identical.

After the runs, we took the same saved candidate answers and compared two ways of choosing an answer: voting and selecting the first candidate that passed the checker.

Voting (Wang et al., 2023): This self-consistency-style baseline keeps an initial “true” answer. Otherwise, it gathers fresh answers and selects “true” if those answers outnumber counterexample answers; otherwise, it selects the most frequent counterexample. Selection does not use a checker (unlike our verified search procedure).

Selection ruleSuccessful trials
Voting347/720 (48.2%)
Checker-based selection519/720 (72.1%)

Checking recovered 172 additional successes, a gain of 23.9 percentage points (95% confidence interval: 19.6–28.3) over voting. No new model answers were generated, so both used exactly the same model-token cost: 53,891 per trial.

3. We added stronger baselines and tests on true claims.

On the same false claims, Reflexion-style feedback achieved 70.3%, verified restarts 72.5%, and PAL-style program search 74.6%, compared with 78.2% for our verified search procedure.

These comparisons use adaptations of prior methods:

Verified restarts (Brown et al., 2024): Makes fresh attempts after a rejected counterexample or an unresolved answer, without showing earlier attempts or checker feedback. Unlike VS, it stops after an initial “true” answer and does not retry a failed first call.

Reflexion-style feedback (Shinn et al., 2023): Gives the model up to six failed candidates and their checker messages to guide later attempts, without a separate reflection-writing step. VS instead starts each attempt fresh, without past candidates or feedback.

PAL-style program search (Gao et al., 2023; Chen et al., 2023): Lets the model write and run Python programs to find counterexamples and use execution feedback. It can also propose a counterexample directly for checking. VS instead generates candidates through fresh model answers, using Python to check them rather than to run model-written search programs.

Compared with all three baselines, VS never returns a counterexample that has not passed the checker.

The latter two also stop after an initial “true” answer.

On 24 true claims tested with four models, verified search returned 0 false counterexamples in 96 trials, compared with 10 for directly asking and 44 for voting.

References

Brown et al. (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787. Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR. Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS. Gao et al. (2023). PAL: Program-Aided Language Models. ICML. Chen et al. (2023). Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall comment with new experimental results and updated revision

Thank you for your feedback. We rewrote the abstract and much of the main text for clarity. New experiments are marked in blue in Section 4.4 and Appendices D.7 and F.4. Writing changes are not highlighted.

1. Reviewer 1sLq: “The experimental results come through as lacking.”

In addition to the experiments we presented in our paper for larger foundational models, for this rebuttal, we also tested whether smaller, local models could find counterexamples to 60 false mathematical claims using our search framework. These specific claims were not used to develop the framework in our paper. Across four local models, directly asking (named Directly Ask) for a counterexample succeeded in 34.7% of trials. Our verified search (VS) framework, which checks proposed counterexamples and keeps trying if none passes, raised this to 78.2%, more than doubling the success rate. In later tests, our VS succeeded in 96.7% of trials with GPT-OSS 120B and 93.9% with GPT-OSS 20B compared to 76.1% and 82.2%, respectively, without our framework.

These experiments show that our novel search framework helps both foundational and local models find proof and statement errors via counterexamples (Section 4.4; Appendix D.7).

2. Reviewer o6Ec: “Compare against existing baselines”

We compared our search procedure with three alternatives, using the same four local models and 60 false claims.

ProcedureHow it searchesSuccess rate
Self-consistency-style voting (Wang et al., 2023)Generates several answers and uses voting to select one.48.2%
Reflexion-style feedback (Shinn et al., 2023)Uses earlier failed attempts and checker feedback to guide later attempts.70.3%
Program search inspired by PAL and Program of Thoughts (Gao et al., 2023; Chen et al., 2023)Writes and runs Python programs to find counterexamples.74.6%
Our verified searchMakes fresh attempts until a counterexample passes the checker or the search-budget rule stops further attempts.78.2%

Our procedure had the highest success rate in these tests (Appendix D.7).

References

Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR.

Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS.

Gao et al. (2023). PAL: Program-Aided Language Models. ICML.

Chen et al. (2023). Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR.

3. Reviewer mEnu: “The benchmark comparisons are not budget-matched.”

The original benchmark already used matched token and time allowances. Every model received a 16,000-token output allowance, a 120,000-token total-usage ceiling, and a 600-second time limit per attempt. However, the models used different numbers of tokens naturally. We now clarify this in the revision (Appendix F.2).

The new search comparisons also used matched token allowances. Every procedure received the same 65,536-token allowance per trial.

We also tested self-consistency-style voting (Wang et al., 2023), a natural alternative to our VS procedure which asks the same model several times and select an answer by voting agreement. VS instead checks each candidate and can stop as soon as one passes.

Under the same token allowance, voting used an average of 53,891 model tokens per trial, compared with 21,111 for our VS. Yet, VS achieved 78.2% success versus 48.2% which is 30 percentage points higher success with less than half as many model tokens (Appendix D.7).

4. Reviewer mEnu: “Hit@1 scores [use] annotated line.”

We choose to use line identification in our benchmark because it provides a simple, automatically scored test of how well models find errors in mathematical reasoning. We show that this benchmark successfully distinguishes models. The new search experiment above accepts any counterexample that passes the checker (Section 4.4; Appendix D.7).

5. Reviewer mEnu: “The benchmark cannot measure false positives”

Our VS returned no false counterexamples on true claim tests. We tested 24 true claims with four models, giving 96 trials per procedure (Appendix D.7):

ProcedureFalse counterexamples returned
VS0/96 (0%)
Directly ask10/96 (10.4%)
Verified restarts, Reflexion, and PAL36/96 (37.5%) each
Self-consistency-style voting44/96 (45.8%)

6. Reviewer o6Ec: “The paper lacks a formal problem setup,”

We've rewritten Sections 2–3 to be clearer. We also explicitly distinguish an incorrect proof step from a false theorem.

7. Reviewer o6Ec: “sources of BugBank.”

Section 4.1 and Appendix C.7.1 contain the source collections, mathematical subjects, and their counts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.