acceptodds
Withdrawn from ICLR 2027

LeanSwarm: An Agent Swarm for Long-Horizon Lean 4 Theorem Proving and Counterexample Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon theorem proving requires more than solving isolated proof states. Large developments involve evolving proof plans, auxiliary lemmas, failed decompositions, and dependencies that must persist across many proof-assistant interactions. We present **LeanSwarm**, a guide-centric multi-agent system for autonomous long-horizon theorem proving in Lean 4. Specialized agents handle proof planning, construction, repository search, research, cycle analysis, and repair, while durable proof state is maintained outside any individual agent context. When ordinary proof search stalls, the system can escalate to deeper analysis, revise the proof structure, or diagnose that a target is false or under-specified. We evaluate **LeanSwarm** on the 600-problem Lean split of NTP4VC and on long-horizon correctness proofs from Strata. On NTP4VC, **LeanSwarm** proves 563/600 (93.8%) verification conditions, which to the best of our knowledge is the highest reported end-to-end solve rate on this split. In the Strata case studies, controlled comparisons against a standalone Claude Code agent using the same primary model show faster proof development, including 25% lower wall-clock time, 12% lower cost, and 17% shorter proofs on a substantial transformation-correctness task. **LeanSwarm** also closes a previously open SMT correctness target, discovers proof architectures different from an existing human development, and identifies false or under-specified specifications that the standalone Claude Code agent continues trying to prove. These results suggest that durable state, specialization, and structured escalation substantially extend strong language models on long-horizon formal reasoning.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
swarm of agents, neuro-symbolic AI, LLMs, theorem-proving, software verification, formal proof search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Sm5FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes an agentic system for long-horizon proof writing in Lean. The system uses a deterministic orchestration workflow to invoke specialized agents for tasks such as proposing proof strategies, writing proofs, and conducting research. Stateful agents keep their conversation history across rounds, while stateless agents start each task with a fresh context. On the 600-problem Lean split of NTP4VC, the system achieves a higher score than the reported model-only baselines. On one instance, it is cheaper and produces shorter proofs than Claude Code on average across three runs.

Critical strengths

The evaluation includes Lean proof problems on different topics. The large scale main evaluation is on real world program verification tasks, and the case study is evaluated on long-horizon program transformations tasks.

Critical weaknesses

Limited evaluation

  • On the 600-problem NTP4VC evaluation, LeanSwarm is only compared against model-only evaluation. There is no comparison with any other similar agentic system discussed in related work, such as Goedel-Architect and LeanMaraton.
  • The comparison with Claude Code in Table 3 only includes one task with 3 samples. This experiment is too small to support any advantage of their system. On the Factory.quant_correct case, Claude Code is cheaper than Lean Swarm, which already conflicts with the conclusion of Table 3 that LeanSwarm is more efficient.
  • There is not any ablation analysis to justify the design of LeanSwarm and help readers to understand how different components contribute.

Missing details and supplementary material

  • The paper describes different roles at a high level, but does not provide full prompts or the implementation of LeanSwarm. As the paper says that the difference comes from instrcution, context and tool access, these details are necessary to understand the system.
  • Several mechanisms in the system are not clearly explained. For example, how is 'estimated generality' determined when selecting the next obligation? How are messages in the persistent mailbox stored and maintained when LeanSwarm is working?
  • No supplementary material was submitted, so I cannot figure out the missing details above from its actual implementation.

Unclear difference from similar works

  • The paper differ their system from existing work on 'durable search state', which inlcudes keeping failed attempts, proof status, and communication history. However, it does not clearly explain how these mechanisms differ from similar works, such as Goedel-Architect and LeanMarathon. This weakness is also partially from many missing details in the paper and no supplementary material available.
Questions for authors

I don't have additional questions. Please refer to Weaknesses section.

Reviewer_sR9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

LeanSwarm is a Lean proving harness that coordinates specialized agents through shared proof state and a sequential workflow. It evaluates the system on software-verification conditions and several larger proof-development tasks.

Critical strengths

The paper addresses practical verification problems and produces Lean-checked proofs.

Critical weaknesses
  1. The main benchmark lacks a fair baseline. On NTP4VC, the paper should compare LeanSwarm and standalone Claude Code using the same Opus 5 model and comparable budget. Without this, the 93.8% result does not establish that the harness improves performance.
  2. It is unclear which added components help. LeanSwarm adds several agent roles, shared memory, cycle detection, and repair. The paper does not test whether these additions improve over a simpler setup. The small Strata comparisons show some speed gains, but do not establish which parts are useful.
Questions for authors
  1. What does standalone Claude Code + Opus 5 achieve on NTP4VC under a comparable budget?
  2. Does removing the additional agent roles, memory, or repair mechanisms reduce performance?
Reviewer_H7ZrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LeanSwarm, a multi-agent system for long-horizon proof development in Lean 4. It separates proof planning, implementation, search, and repair, while maintaining dependencies, failure histories, and communication histories externally. When progress stalls, it detects circular decompositions or diagnoses incorrect specifications and missing assumptions. It solves 563/600 NTP4VC problems and reports shorter development time than a standalone agent using the same primary model on Strata case studies. Its main contribution is the design of a proof harness using existing models and its application to program verification.

Critical strengths
  1. The problem setting is important. The paper addresses practical proof development, including lemma decomposition, invariants, dependencies, and recovery from failures.
  2. The system design is concrete. It combines persistent proof state, specialized roles, cycle detection, and staged repair with Lean validation and checks against the original theorem.
  3. The results are promising. It achieves high coverage on NTP4VC. On a Strata transformation-correctness theorem, comparisons using the same primary model and Lean-facing harness show approximately 25% lower average development time over three runs per condition.
Critical weaknesses
  1. There is no experimental comparison with the closest existing harnesses. LeanMarathon and Goedel-Architect already include related mechanisms such as persistent proof graphs, role separation, and failure-driven restructuring. Comparison with standalone Claude Code does not establish an advantage over these structured systems.
  2. There are no component ablations. The individual contributions of persistent memory, a separate search agent, semantic cycle detection, and escalation remain unclear. The paper does not establish whether a simpler configuration could achieve similar improvements.
  3. The large-scale evaluation does not separate architectural gains from model and inference-budget effects. NTP4VC baselines use different models, inference methods, and budgets. Without a strong standalone baseline using the same backbone and budget, the 93.8% success rate cannot be attributed to the proposed architecture.
  4. Evaluation coverage is limited. NTP4VC is the only standardized benchmark, while matched-primary-model comparisons cover only a few cases within Strata. Evaluation on complementary benchmarks such as miniCodeProps, VERINA ProofGen, or VeriSoftBench, or an explanation for their exclusion, would strengthen the evidence for generalization.
  5. Specification diagnosis is evaluated qualitatively. Concrete counterexamples or formal disproofs are insufficiently presented, and diagnosis success rates, false diagnosis rates, and costs are not systematically evaluated.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.