- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work proposes an agentic system for long-horizon proof writing in Lean. The system uses a deterministic orchestration workflow to invoke specialized agents for tasks such as proposing proof strategies, writing proofs, and conducting research. Stateful agents keep their conversation history across rounds, while stateless agents start each task with a fresh context. On the 600-problem Lean split of NTP4VC, the system achieves a higher score than the reported model-only baselines. On one instance, it is cheaper and produces shorter proofs than Claude Code on average across three runs.
Critical strengths
The evaluation includes Lean proof problems on different topics. The large scale main evaluation is on real world program verification tasks, and the case study is evaluated on long-horizon program transformations tasks.
Critical weaknesses
Limited evaluation
- On the 600-problem NTP4VC evaluation, LeanSwarm is only compared against model-only evaluation. There is no comparison with any other similar agentic system discussed in related work, such as Goedel-Architect and LeanMaraton.
- The comparison with Claude Code in Table 3 only includes one task with 3 samples. This experiment is too small to support any advantage of their system. On the Factory.quant_correct case, Claude Code is cheaper than Lean Swarm, which already conflicts with the conclusion of Table 3 that LeanSwarm is more efficient.
- There is not any ablation analysis to justify the design of LeanSwarm and help readers to understand how different components contribute.
Missing details and supplementary material
- The paper describes different roles at a high level, but does not provide full prompts or the implementation of LeanSwarm. As the paper says that the difference comes from instrcution, context and tool access, these details are necessary to understand the system.
- Several mechanisms in the system are not clearly explained. For example, how is 'estimated generality' determined when selecting the next obligation? How are messages in the persistent mailbox stored and maintained when LeanSwarm is working?
- No supplementary material was submitted, so I cannot figure out the missing details above from its actual implementation.
Unclear difference from similar works
- The paper differ their system from existing work on 'durable search state', which inlcudes keeping failed attempts, proof status, and communication history. However, it does not clearly explain how these mechanisms differ from similar works, such as Goedel-Architect and LeanMarathon. This weakness is also partially from many missing details in the paper and no supplementary material available.
Questions for authors
I don't have additional questions. Please refer to Weaknesses section.