- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces proof weaving, the process of constructing a complete formal proof by creating, sharing, and integrating intermediate results across agents and time. To evaluate this capability, the authors propose FLTBench, a benchmark containing 10 Lean proof reconstruction tasks derived from the formalization of Fermat's Last Theorem, spanning local constructions to full theorem proving. Four multi-agent systems are evaluated under a common four-hour budget. All systems successfully solve only the simplest task, with no successes on the remaining nine. The paper further presents qualitative case studies of cross-agent proof reuse, unfinished proof obligations, and correction of incorrect completion reports.
Critical strengths
- The paper highlights a meaningful distinction between solving individual proof obligations and integrating distributed contributions into a complete mathematical proof.
Critical weaknesses
-
The paper offers limited experimental insights due to near-zero success rates. All four systems fail on Q2–Q10, making it difficult to compare their proof-weaving capabilities or identify which approaches are more effective. The substantial difficulty gap between the easiest and remaining tasks limits the benchmark's ability to distinguish current systems.
-
The work also lacks quantitative evaluation of proof weaving itself. Although proof weaving is the central contribution, the primary metric is binary target completion. The paper does not systematically measure intermediate progress, successful cross-agent reuse, integration efficiency, or the proportion of generated lemmas that contribute to the target. The qualitative examples are insufficient to establish broader conclusions about collaboration
-
There is insufficient evidence that multi-agent coordination is the main bottleneck. The experiments do not isolate failures caused by individual mathematical reasoning from those caused by coordination, information sharing, or integration. Without single-agent baselines or ablations, it remains unclear whether the observed failures specifically reflect limitations in proof weaving rather than the inherent difficulty of the mathematical tasks.
Questions for authors
-
How can the benchmark better distinguish current systems? Since none of the systems solves Q02–Q10, could the authors introduce intermediate checkpoints or tasks of more gradually increasing difficulty to obtain more informative comparisons?
-
Can proof weaving be measured quantitatively? For example, how many verified intermediate lemmas are produced, how many are reused by other agents, and how many meaningfully reduce unresolved obligations? Could these metrics distinguish systems even when they fail to complete the final target?
-
What evidence suggests that proof weaving, rather than individual proving capability, is responsible for the failures? Would comparisons against single-agent systems or independent parallel proof attempts help isolate the benefits and limitations of collaborative construction?