acceptodds
Withdrawn from ICLR 2027

Are We Ready for Proof Weaving? Understanding Agent Swarms through Fermat's Last Theorem

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A collection of correct proofs is not yet a proof of a common goal. We study how agent systems can build a common proof from earlier results, a process we call proof weaving. The public formalization of Fermat's Last Theorem (FLT) provides a completed reference for studying this ability. We introduce FLTBench, ten Lean reconstruction tasks drawn from this development. Each task withholds a target proof and a specified set of supporting reference proofs, while retaining the target statement and its permitted mathematical foundation. Continuing a proof also requires recognizing what earlier work has established, so we examine both reconstruction and the interpretation of work records. In our evaluation, all four evaluated systems complete the same one of the ten tasks. Case analysis shows checked cross-agent reuse, preserved unfinished constructions, and correction of a false completion report. In a separate diagnostic, an agent assesses 100 constructed work records, including normal proof reports, partial work, and conflicting claims of success. Its judgments disagree with the assigned completion status in 20 cases. These findings highlight proof weaving as a challenge for agent research. FLTBench and the diagnostic cases offer the community a concrete basis for studying how local contributions can become complete mathematical proofs.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
proof weaving, agent swarms, multi-agent systems, formal theorem proving, mathematical reasoning, Fermat's Last Theorem, lean

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L7zrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces proof weaving, the process of constructing a complete formal proof by creating, sharing, and integrating intermediate results across agents and time. To evaluate this capability, the authors propose FLTBench, a benchmark containing 10 Lean proof reconstruction tasks derived from the formalization of Fermat's Last Theorem, spanning local constructions to full theorem proving. Four multi-agent systems are evaluated under a common four-hour budget. All systems successfully solve only the simplest task, with no successes on the remaining nine. The paper further presents qualitative case studies of cross-agent proof reuse, unfinished proof obligations, and correction of incorrect completion reports.

Critical strengths
  • The paper highlights a meaningful distinction between solving individual proof obligations and integrating distributed contributions into a complete mathematical proof.
Critical weaknesses
  • The paper offers limited experimental insights due to near-zero success rates. All four systems fail on Q2–Q10, making it difficult to compare their proof-weaving capabilities or identify which approaches are more effective. The substantial difficulty gap between the easiest and remaining tasks limits the benchmark's ability to distinguish current systems.

  • The work also lacks quantitative evaluation of proof weaving itself. Although proof weaving is the central contribution, the primary metric is binary target completion. The paper does not systematically measure intermediate progress, successful cross-agent reuse, integration efficiency, or the proportion of generated lemmas that contribute to the target. The qualitative examples are insufficient to establish broader conclusions about collaboration

  • There is insufficient evidence that multi-agent coordination is the main bottleneck. The experiments do not isolate failures caused by individual mathematical reasoning from those caused by coordination, information sharing, or integration. Without single-agent baselines or ablations, it remains unclear whether the observed failures specifically reflect limitations in proof weaving rather than the inherent difficulty of the mathematical tasks.

Questions for authors
  1. How can the benchmark better distinguish current systems? Since none of the systems solves Q02–Q10, could the authors introduce intermediate checkpoints or tasks of more gradually increasing difficulty to obtain more informative comparisons?

  2. Can proof weaving be measured quantitatively? For example, how many verified intermediate lemmas are produced, how many are reused by other agents, and how many meaningfully reduce unresolved obligations? Could these metrics distinguish systems even when they fail to complete the final target?

  3. What evidence suggests that proof weaving, rather than individual proving capability, is responsible for the failures? Would comparisons against single-agent systems or independent parallel proof attempts help isolate the benefits and limitations of collaborative construction?

Reviewer_3n3FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces “proof weaving,” the process of developing and combining intermediate results into a common proof. It proposes FLTBench, ten Lean reconstruction tasks drawn from a formalization of Fermat’s Last Theorem. Four agent systems are evaluated, and all complete only the simplest task. The paper also presents examples of reuse and unfinished constructions, alongside a diagnostic in which an observer judges completion from 100 constructed work records and agrees with the assigned labels in 80 cases.

Critical strengths
  1. A concrete benchmark with checkable outcomes. FLTBench specifies the target statements and permitted foundations while withholding supporting proofs. This provides a useful testbed for extended theorem proving, with success determined by Lean rather than subjective judgments.

  2. Useful analysis of intermediate work. The case studies trace actual lemma reuse, unfinished constructions, and correction of a false completion report. These examples provide information beyond binary success rates and help identify what agents have established and what mathematics remains unresolved.

Critical weaknesses
  1. The research question itself is insufficiently motivated. In mathematical research, lemmas are typically formulated and proved to serve a larger argument; choosing them and understanding how they support the target are integral to developing the proof. The observation that a collection of correct lemmas does not establish the target is therefore unsurprising.

  2. The benchmark produces no discriminative signal. Table 4 is 0/3 on Q02–Q10 for all four systems; the only completed task, Q01, is a single-theorem anchor that is closer to a sanity check than a weaving task. The paper motivates FLTBench as a way to “compare different ways of organizing reconstruction”, but with this outcome it cannot distinguish any organizational choice from any other.

Questions for authors

Please see my critical weaknesses

Reviewer_ruPkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

FLTBench defines ten Lean reconstruction tasks by removing target proofs. These connect supporting material from a completed Fermat's Last Theorem proof. Four systems use a common model and resource cap. Three continuation cases examine reuse and unfinished components. A separate constructed-record diagnostic studies completion judgments. Contextual proving and repository engineering already have benchmarks such as miniCTX and APE-Bench. FLTBench's useful distinction is controlled removal of connected support while retaining exact targets and permitted foundations.

Critical strengths

The completion semantics are precise. A checked conditional result is useful without satisfying the missing mathematical construction. The cases make this distinction concrete through a reused conditional lemma, an outstanding realization gap, and correction of an invalid inference.

Critical weaknesses

Nine of ten tasks are at the floor despite markedly different foundations and recon- struction sizes. This identifies difficult work, but supplies limited guidance for comparing methods, lo- cating an achievable frontier, or understanding which interventions help.

The auxiliary diagnostic does not adequately compensate for that limited resolution. Its 80% agreement is with constructed labels, as the paper discloses, rather than replayed target proofs.

Questions for authors
  1. Can intermediate withholding settings produce informative variation in completion or prerequisite con- struction while preserving the benchmark's central reconstruction challenge?
  2. Can the continuation cases be situated in a systematic inventory of the runs, and can each positive diag- nostic label be tied unambiguously to the requested target?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.