acceptodds
Withdrawn from ICLR 2027

ProofCouncil: An LLM Agent for Solving Open Mathematical Problems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) have shown increasing promise in solving open problems in mathematics. However, their performance can be further improved through agentic workflows tailored to real-world mathematical practice. To this end, we introduce ProofCouncil, a mathematical agent that is designed to tackle open problems using an author-critic architecture. ProofCouncil served as a submission to the second batch of FirstProof, a challenge consisting of 10 real-world mathematical problems that agents must solve autonomously. Its submissions for 6 of the 10 problems were judged by the referees to be correct up to at most minor revisions, showing the best performance among participating teams. We also evaluate ProofCouncil on 30 open problems collected from mathematical researchers. Among the 21 solutions that received human feedback, 5 were judged completely correct, 2 more were judged promising pending final verification, and a further 8 contained useful partial progress. In this paper, we describe the development of ProofCouncil and the agent-building library used to create it, which we release as open source to the community.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mathematics, LLMs, agentic problem solving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_T29YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ProofCouncil, a multi-agent system for research-level mathematical proofs. The system iteratively coordinates an author agent and a critic agent; the critic is periodically reset and followed by a second audit by a fresh critic, the author may request assistance from a multi-model council or a compute node on demand, and the workflow is orchestrated as a conditional directed acyclic graph. The paper evaluates the system on the FirstProof benchmark and a set of researcher-provided open problems, reports several results accepted by human referees or showing partial progress, and analyzes failure modes such as critic misjudgment, attribution, and readability.

Critical strengths

1.Periodic reset of the stateful critic and audit by a fresh critic expose path dependence in LLM reviewers. 2.A decontaminated human-refereed benchmark using unpublished problems and expert referees supports fine-grained failure attribution. 3.An open-source conditional DAG library with conditional edges lowers the engineering barrier. Command-line compute nodes for mathematical software and human-review nodes support workflows in strict domains.

Critical weaknesses

1.Methodological novelty is limited, as prior work already includes numerous integrated systems. 2.Although the experimental execution is rigorous and commendable, the comparative design has gaps, and the evidence for the core performance claim is insufficient to support the conclusion of “6/10 best in the same batch.”

Questions for authors

1.The necessity of the fresh critic, council, compute node, and critic reset has not been isolated through systematic ablation; component-level comparative evidence is also lacking for whether the current combination is superior to contemporaneous integrated systems. 2.Distinguishing architectural gains from computational gains requires a single-model baseline with equal rounds and equal tool access. 3.Complete solutions to open problems should be clarified as to whether they were validated independently of the problem authors.

Reviewer_XePDOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents ProofCouncil, an agentic workflow for solving mathematical open problems. The workflow consists of an author, a critic, an LLM council, and a compute node agent. The author proposes solutions to the critic and can communicate with the council and the compute nodes for assistance. When the author believes it has a solution, it proposes it to a critic for assessment. The critic's history is cleared every kk rounds. If the critic rejects the solution, it's returned to the author to repair. When the critic accepts, another critic round with fresh history will also run. The LLM council consists of multiple independent models providing ideas for the author. The paper uses gpt-5.5-pro for the author and critic and as one of the member of the council, gpt-5.5 in Codex for the compute node agent, and Gemini 3.1 Pro and Claude Opus 4.7 as additional members of the council. To implement this workflow, the authors implement an open-source and more general DAG-based agentic system.

The workflow was submitted to the second batch of FirstProof challenge. Setting aside the problem where the OpenAI API returned apparently incidental timeouts, FirstProof solved two additional problems comparing to UCLA Moonshot and plain gpt-5.5-pro. However, the cost per problem for the FirstProof was ~30x the cost per problem of plain gpt-5.5-pro. The workflow was further evaluated on a set of researcher provided problems were it was some times successful in solving the open problems. ProofCouncil has also provided important partial results for Erdős problem 539.

Critical strengths
  1. The workflow is valuable and substantive since it has proven several open mathematical problems. Specifically, the partial progress on Erdős #539 and being the "top" performer of the first proof competition.
  2. I see the open-source agentic system as a nice contribution for the community.
Critical weaknesses
  1. The contribution of the parts of the architecture are not clear since there is no ablation on them. Moreover, it's not clear whether running the plain gpt-5.5-pro more (in parallel, or sequential, or a combination of both) would or would not improve its performance. Although I understand this is costly both in costs and human time to evaluate, I still would value this high into understanding the capabilities of the agent.
Questions for authors

None

Reviewer_t4ALOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ProofCouncil, an LLM agent for solving open mathematical problems through iterative proof generation and revision. An author maintains a proof draft, research notes, and references, while a critic provides feedback, with periodic resets and an additional fresh-critic audit before internal acceptance. The author can also consult a council of models and a dedicated compute node equipped with mathematical software.

The evaluation includes the ten-problem FirstProof challenge, 30 researcher-submitted problems, and three public Erdős problems. The paper also examines execution traces, component usage, costs, and failure cases to characterize the system’s behavior.

Critical strengths
  1. Independent expert evaluation. FirstProof submissions were assessed by expert referees appointed by the organizers. This provides credible evidence of mathematical correctness beyond the system’s internal LLM judgments.
  2. Informative analysis of component usage and failure modes. The paper provides useful qualitative insights into component roles, such as checking intermediate claims, and identifies failure modes including repeated local proof repairs and shared problem misinterpretations. These observations help explain the system’s behavior and offer practical guidance for designing future mathematical research agents.
Critical weaknesses
  1. The methodological novelty is insufficiently established. The system combines an author–critic loop, critic resets, a multi-model council, and a compute node, but the paper does not clearly identify a new methodological principle beyond their integration.

  2. The experiments do not adequately establish the architecture’s contribution. There are no controlled component ablations or budget-matched comparisons, while the FirstProof improvement over the single-model baseline is 6/10 versus 5/10 at reported costs of $354 versus $12 per problem. The ten-problem comparison provides limited evidence of a reliable advantage.

Questions for authors

Can the authors provide budget-matched comparisons and controlled ablations of the council, compute node, and critic-reset/audit mechanisms? A broader evaluation with a fixed configuration would help establish which components account for them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.