- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ProofCouncil, a multi-agent system for research-level mathematical proofs. The system iteratively coordinates an author agent and a critic agent; the critic is periodically reset and followed by a second audit by a fresh critic, the author may request assistance from a multi-model council or a compute node on demand, and the workflow is orchestrated as a conditional directed acyclic graph. The paper evaluates the system on the FirstProof benchmark and a set of researcher-provided open problems, reports several results accepted by human referees or showing partial progress, and analyzes failure modes such as critic misjudgment, attribution, and readability.
Critical strengths
1.Periodic reset of the stateful critic and audit by a fresh critic expose path dependence in LLM reviewers. 2.A decontaminated human-refereed benchmark using unpublished problems and expert referees supports fine-grained failure attribution. 3.An open-source conditional DAG library with conditional edges lowers the engineering barrier. Command-line compute nodes for mathematical software and human-review nodes support workflows in strict domains.
Critical weaknesses
1.Methodological novelty is limited, as prior work already includes numerous integrated systems. 2.Although the experimental execution is rigorous and commendable, the comparative design has gaps, and the evidence for the core performance claim is insufficient to support the conclusion of “6/10 best in the same batch.”
Questions for authors
1.The necessity of the fresh critic, council, compute node, and critic reset has not been isolated through systematic ablation; component-level comparative evidence is also lacking for whether the current combination is superior to contemporaneous integrated systems. 2.Distinguishing architectural gains from computational gains requires a single-model baseline with equal rounds and equal tool access. 3.Complete solutions to open problems should be clarified as to whether they were validated independently of the problem authors.