acceptodds
Withdrawn from ICLR 2027

A Closed-Loop System of 3 Mathematical Agents: Conjecturer, Refuter, and Prover

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present a closed-loop autonomous system for mathematical discovery. In this paper, “autonomous” refers to execution-level autonomy: after initialization, one command runs conjecture generation, counterexample search, formal proving, outcome reconciliation, and the next generation without human intervention between stages. The system generates conjectures in the form of linear inequalities, searches for counterexamples, and sends surviving statements to a Lean 4 prover with a zero-sorry policy. The concrete setting is polyhedral combinatorics. The system separates proposal, verification, and learning. Outcome-dependent signals may influence the conjecture generator, but do not modify the counterexample validator, exhaustive search procedure, or proof-acceptance gates. Accepted counterexamples contain explicit witness graphs and pass five validation checks; accepted proofs are compiling Lean artifacts whose root theorems match the generated statements. The system also separates arithmetic feasibility from geometric realizability. In the current repository version, the campaign records 204 witness-verified refutations, four end-to-end machine-proved conjectures (C104, C124, C201, and C215), and three open conjectures (C193, C195, and C198). The main contribution is an artifact-producing autonomous workflow that preserves refutations, proofs, and unresolved cases as reusable computational or formal objects.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Automated mathematical discovery, autonomous reasoning systems, automated conjecture generation, counterexample-guided search, automated theorem proving, formal verification, Lean 4, combinatorial enumeration, polyhedral computing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LdNoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript proposes a closed-loop autonomous system for mathematical discovery, which integrates conjecture generation, counterexample search, formal proof, result coordination and subsequent conjecture generation into a single unattended workflow. The system targets affine inequalities over face-size histograms of simple convex 3-polytopes, where arithmetic feasibility does not necessarily imply geometric realizability. Its key design is a realizability-aware verification pipeline that combines reusable witness replay, Eberhard-lattice search, bounded exhaustive plantri enumeration, and multiple stochastic search methods, followed by a five-check validator for candidate counterexamples. Surviving conjectures undergo an entailment pre-check and are accepted as proved only when the corresponding Lean 4 artifact compiles under a zero-sorry policy. Results are then saved as structured artifacts and fed back to the conjecture generator to guide subsequent exploration while keeping the verification gates fixed. Authors validate the system through a repository-level autonomous campaign. The reported campaign produces 204 witness-verified refutations, four end-to-end machine-checked Lean proofs, and three unresolved conjectures, with more than half of the refutations obtained through replay of previously validated witnesses. The manuscript also provides provenance information for the refutations, explicit witness graphs and validation records, formal proof artifacts, and an audit of the formal inventory underlying the Lean proofs.

Critical strengths
  1. By integrating conjecture generation, counterexample search, independent validation, formal proving, and outcome-driven learning into a coherent autonomous workflow, the manuscript clearly separates the proposal and verification components, making them easy for the reader to understand.

  2. The authors chose a field with real technical difficulties. Instead of choosing the scenario that "arithmetical establishment equals problem solving", but instead chooses simple convex 3-polytopes: a p-vector can satisfy the Eberhard equation without a corresponding polytope existing, indicating that arithmetic validity does not imply geometric existence. This choice makes the verification of the system not stop at the numerical level, and it is necessary to construct a real plane cubic diagram as a witness. Therefore, counterexample search is not a simple numerical check, but a process that includes substantive verification steps.

  3. The campaign produces 204 witness-verified refutations, 4 formal proofs, and 3 remaining open conjectures, demonstrating that the proposed system can execute a nontrivial mathematical discovery loop rather than only generating isolated conjectures or proofs.

  4. The realizability-aware counterexample pipeline and five-check validation provide a meaningful guarantee against spurious arithmetic counterexamples, while the Lean 4 zero-sorry proof gate and structured result records improve the reliability and reproducibility of the reported results.

Critical weaknesses
  1. The manuscript reports an impressive number of refutations and proofs, but provides limited controlled ablations or matched-budget comparisons that isolate the contributions of the four counterexample search tracks (witness replay, Eberhard-lattice search, plantri screening, and stochastic tracks), the outcome-driven learning mechanism, the five-check validator, and the closed-loop feedback from outcomes to conjecture generation.

  2. Although the system executes the discovery loop autonomously after initialization, substantial human design remains in the choice of mathematical domain, conjecture language, search components, formalization inventory, and system configuration. Moreover, the evaluation is restricted to affine inequalities over face-size statistics of simple convex 3-polytopes, making it difficult to assess whether the proposed architecture generalizes to substantially different mathematical domains.

  3. The reported Lean proofs are zero-sorry at the generated theorem level, but the formalization relies on a hand-audited inventory containing 13 sorry-marked lemmas. However, the paper does not clearly state that the formal guarantees are relative to this trusted inventory rather than being fully established from a completely formalized foundation.

  4. The manuscript is generally understandable, but the presentation is overly procedural and sometimes reads more like a detailed system description than a focused scientific narrative. The central insight and scientific contribution are somewhat diluted by extensive discussion of implementation details and outcome bookkeeping.

Questions for authors
  1. This manuscript reports 3 open conjectures, but it is unclear whether they are mathematically open or merely unresolved under the current configuration. For example, Open Conjecture 2 appears to require only one missing realizability lemma. A sensitivity analysis over the enumeration bound (e.g., ) would clarify the status of these conjectures.

  2. The manuscript identifies outcome-driven learning as a key component of the proposed system, but the current evaluation does not clearly isolate its contribution from that of the underlying conjecture generator and search tracks. Could the authors compare the full system against a variant that does not use previous outcomes to guide subsequent conjecture generation, under the same computational budget? This would help establish whether the feedback mechanism materially improves the campaign beyond simply integrating existing components.

  3. The manuscript reports that 103 of the 204 refutations come from witness replay and interprets this as evidence that the counterexample pool strengthens the discovery campaign. However, this result could also reflect reliance on previously discovered evidence rather than new discoveries. Could the authors report how many replayed refutations apply to genuinely new conjectures, as opposed to duplicate or near-duplicate conjectures? This would help clarify the extent to which witness replay contributes to the discovery of new results.

  4. The provenance analysis shows that the search tracks contribute very unevenly, with some tracks producing few or no independently discovered refutations. Could the authors report the marginal contribution of each track, for example by measuring how many unique validated counterexamples are found only by that track under a fixed search budget? This would clarify whether the current multi-track design provides complementary coverage or includes components with limited practical value.

  5. The manuscript reports four end-to-end Lean proofs with no new sorry, while acknowledging that the underlying formal inventory contains 13 sorry-marked lemmas. Could the authors clarify whether any of these lemmas are used in the dependency chains of the four reported theorems, and identify the resulting trusted assumptions for each theorem? A theorem-level dependency summary would make the scope of the formal guarantees easier to assess.

  6. The manuscript addresses a relevant problem and reports a substantial system implementation. However, the current evaluation primarily reports the overall outputs of the system, such as the number of verified refutations and formal proofs, without sufficiently clarifying the incremental benefits of the proposed framework over the underlying search and proving components. It remains unclear whether the closed-loop organization and outcome-driven feedback provide substantial advantages beyond merely integrating these existing components. The paper would be strengthened by a more explicit characterization of the framework's unique benefits, supported by controlled comparisons, ideally a framework-level comparison against a non-closed-loop pipeline using the same components, supplemented by component ablations where appropriate.

2 comments
AuthorsOpenReview ↗

Value of the closed loop, track contributions, and generality (1/2)

We thank the Reviewer for the positive assessment, in particular for recognizing that this domain makes counterexample search a substantive verification problem, and for the concrete questions. We have uploaded a revised PDF (changes in blue); each of your six questions is answered by a new analysis in App. K. In short, the contribution is, to our knowledge, the first system that closes the full generate-refute-prove-learn loop without human intervention, and the new analyses show that the loop's components are complementary rather than redundant.

W1 / Q2 / Q6 (does the closed loop add value beyond its components?). Yes. We added the framework-level comparison you suggested, and it favors the closed loop. Four pieces of evidence, each isolating a different part of the loop. (1) Feedback to the generator: in the generator-level ablation on the same frozen registry, outcome feedback raises hypothesis-cell coverage from 26 to 40 cells (+54%) and accepted conjectures from 27 to 41; the 14 extra cells are unreachable without feedback. (2) Witness reuse, which exists only because outcomes flow back into the loop: 103 of the 204 refutations replay a previously certified witness, and 99 of these 103 refute a conjecture in a different hypothesis cell (Q3). (3) The validator: in the open-conjecture audit, 238 of 271 arithmetically feasible candidates are non-realizable, so an arithmetic-only pipeline would have accepted every one of them as a counterexample; in an instrumented run of 4,475 validations, the realizability check rejected 32 such candidates. (4) A new loop ablation (App. K): closed vs. open loop over two generations from the same frozen registry, five seeds, identical budgets, gates, and shared sources, with the open arm withholding every outcome signal (which disables the two feedback-only sources). On the pre-registered primary metric, hypothesis cells of accepted conjectures not covered by the frozen registry, feedback wins in 4 of 5 seeds (41.4 vs. 29.8 on average), and on distinct accepted cells it wins in all 5 (54.6 vs. 34.8). The gain comes from the feedback-only sources, the mutation engine and dynamic cells, while the shared sources perform on par: feedback opens directions the open loop never proposes. Those two sources add about 73 evaluated candidates per run (about 40% more wall time); every shared component runs with an identical budget. For time, downstream search used the deterministic tiers only and the prover was disabled.

Q4 (marginal contribution of each track). Thank you; App. K now reports this, and it shows the tracks are complementary. Under a static reachability model, all 17 RL first finds and 14 of the 23 LLM first finds lie beyond the reach of every deterministic tier. Running plantri, the Eberhard lattice, and Hopper each alone on the 101 first-find conjectures, with the campaign's per-batch timeouts, recovers only 1 of the 40 conjectures first refuted by RL or LLM, and none of the 17 refuted first by RL. Conversely, the deterministic tiers run first in the campaign, so first-find counts reflect the order in which tracks run rather than what each can find: run alone, these tracks recover all three Eberhard-lattice first finds and the Hopper first find, and plantri contributes exhaustive non-realizability verdicts that no stochastic track can provide (App. K).

W2 (human design and generality). We agree, and the paper now states it plainly: autonomy concerns execution: humans chose the domain, implemented the components, and audited the inventory. The domain is chosen deliberately: by Steinitz's theorem every counterexample can be certified by a graph check, whereas already in dimension 4 deciding realizability is complete for the existential theory of the reals (Richter-Gebert, 1996). The conclusion lists what another domain must provide to support the loop.

(continued in Part 2/2)

AuthorsOpenReview ↗

Trusted base, presentation, open conjectures, and replay novelty (2/2)

(Part 2/2, continued from above)

W3 / Q5 (trusted base). Thank you; we added the theorem-level dependency summary you asked for. The formal guarantees are relative to the hand-audited inventory, as the abstract, Sec. 4.4, and Limitations now state. App. A lists every sorry-marked declaration, and App. K gives a theorem-level dependency table: C124 uses only p_range; C201 and C215 use six elementary face-count and occupation facts and no deep bound; C104 uses the same six plus Barnette's bound. For this revision we exhaustively checked the base over all 58,716 simple polytopes with at most 13 faces and replaced one mis-stated declaration by a proved lemma; the generated proofs are unchanged and recompile, and the base now has eight sorry-marked declarations (thirteen in the submission).

W4 (presentation). Thank you; we refocused the narrative. The abstract and contributions now state the central claim directly, low-level procedures moved to the appendices, a terminology table was added (App. J), and App. I traces one proved and one refuted conjecture end to end.

Q1 (open, or unresolved under the configuration?). Thank you for this suggestion; App. K now gives the bound sensitivity you proposed: at f2<=20 and f2<=45 (all candidates in fact have f2<=28), no decided candidate of the three open conjectures is realizable, and the remaining 33 are undecided only because of the 15 s plantri budget. C193 and C198 are genuinely open: every candidate has S=1, and their residual case, a single large face of unbounded size, cannot be closed by any finite enumeration. For C195 (your Open Conjecture 2), which stays formally open, the loop's diagnosis is exact: the case S>=2 is immediate, the elementary fact f2>=m+1 closes S=1, and the only remaining obstruction is the finite hole {4:6, 6:1}, which plantri proves non-realizable but the formal inventory cannot yet express (Sec. 5.3).

Q3 (are replayed refutations genuinely new?). Yes. 99 of the 103 replays refute a conjecture in a different hypothesis cell from the one where the witness was first found. Of the other four, two are coefficient variants within one cell, one is a different statement in the same cell, and one pair (C24/C23, both from the early seed batch) is a semantic duplicate, which we disclose. The 103 replays draw on 34 distinct witnesses; the most reused, an RL witness with f2=64, refuted 17 later conjectures (App. I).

Revisions made. (i) The central claim in the abstract and contributions; (ii) App. A: the trusted base and the base audit; (iii) App. K: a five-seed loop ablation, per-track contribution, replay novelty, bound sensitivity, validator evidence, and theorem-level dependencies; (iv) App. I: a worked example; (v) App. J: terminology; (vi) a corrected description of the generator, including its LLM review step. We hope these address the concerns; if they do, we would be grateful if you would consider updating your assessment, and we welcome further discussion.

Reviewer_z7PbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents a closed loop system that proposes linear inequalities on the face counts of simple 3-polytopes, searches for counterexamples that must be explicit witness graphs, and sends the surviving statements to a Lean 4 prover, reporting 204 refutations, four proved theorems, and three open conjectures.

Critical strengths

This paper connects conjecture generation, counterexample search, and Lean proving in one loop. Its reported results can be checked: every refutation comes with a witness graph that is verified again by independent code.

Critical weaknesses
  1. The evaluation does not show which parts of the system work. The paper gives the final counts, but not the total number of conjectures or how many were removed at each stage. Without this, it is hard to judge what 204 refutations mean, since the system refutes conjectures it proposed itself. There is no ablation of the outcome signals. The LLM, the RL policy, and the compute budget are also not described.
  2. The results do not show that the designed loop can reach nontrivial mathematics. The four proved theorems are simple. The fourth follows from the definition of f2, and Theorem 2 only needs an argument for S = 1. The Lean proofs also do not cover the hard steps: these come from thirteen sorry-marked lemmas, including the Jučovič and Barnette bounds, which are assumed rather than proved.
Questions for authors
  1. What does the closed loop add beyond simple enumeration? The conclusions are linear in a few face counts, and plantri can enumerate candidates up to f2 ≤ 36. In this setting, listing all realizable p-vectors and fitting the tightest linear bounds might already recover most of the reported results. Could the authors compare the loop with this simple baseline?
  2. What do the LLM and RL tracks contribute? The LLM appears in three roles (proposer, constructor, and prover), and the RL track in one, but none is described. The model name appears only in Appendix F, and the RL algorithm (PPO) can only be inferred from the same appendix. Did the LLM and RL tracks find counterexamples that plantri with a larger bound would miss? If they did not, the role of learning in the system seems small.
2 comments
AuthorsOpenReview ↗

Soundness of the trusted base, loop ablation, and the enumeration baseline (1/2)

We thank the Reviewer, especially for noting that our results are independently checkable. We wish to stress that it is very rare to have a mathematical topic where automated conjecturing, refutation, and proving can exist as a closed-loop of automated mathematics. The reason is simple, in many areas of mathematics, one has no generating principles that connect all objects (e.g., in our case Steinitz operations), or one has no ways to conjecture automatically (in our case the data of multiple polyhedra allows for p-vector conjecturing). Note that one cannot even construct a similar 3 agent closed loop system for 4-dimensional polyhedra because there are no analogues of Steinitz operations: there is no Steinitz-type theorem, and deciding whether a combinatorial type is realizable as a 4-polytope is complete for the existential theory of the reals (Richter-Gebert, 1996). Our contribution is to have implemented such a rare agent system.

What changed for your concerns, in brief. (1) Soundness: the Jučovič bound is proved, not assumed, and Barnette is the only assumed bound (Thm 1 only); vacuous proofs are excluded by construction and tested exhaustively - the base is guarded against fabricated instances, every base lemma a proof uses was checked against its exact Lean statement (exhaustively up to 13 faces for the adjacency-dependent ones), and the one mis-statement this audit caught is fixed with the four proofs byte-for-byte unchanged (W2). (2) A new pre-registered, five-seed loop ablation: feedback wins on novel hypothesis cells in 4 of 5 seeds and on distinct accepted cells in 5 of 5 (W1). (3) Your enumeration baseline: all 41 counterexamples first found by LLM/RL/Hopper lie outside any range enumeration can reach, and plantri, the Eberhard lattice, and Hopper, each run alone, recover only 1 of the 40 found by RL or LLM (Q1/Q2).

Weakness 1 (funnel). Thank you; the revision now reports the full funnel. 214 registered -> 206 refuted (204 carry a rendered witness graph, the paper's count; 2 validated but unrendered) -> 8 survived -> 6 entered the prover (2 were not entailed by the inventory) -> 4 proved, 3 open, and 1 degenerate test statement on which the prover failed. Refuting its own proposals is the loop working as intended: the generator deliberately targets extremal inequalities, and each refutation is a new explicit polytope, decided by an independent witness graph and a five-check validator, never by the proposing model. The LLM/RL/Hopper tracks first-found 41 and, with the 72 replays reusing those witnesses, account for 113 of 204 refutations (55%). Gate logging covered nine late-campaign runs (93 of 749 candidates accepted, ~12%); the caps bound the campaign at ~3,200 constructor calls, ~1.3e5 RL episodes, and ~1e6 Hopper steps. The LLM, the RL policy, and the compute budget are now described in the main text (Sec. 4.2; see also Q2).

Weakness 1 (outcome-signal ablation). We added this ablation at the level of the full loop (App. K). Over two generations with five seeds, identical budgets for every shared component, and a pre-registered metric, withholding all outcome signals loses novel hypothesis cells in 4 of 5 seeds (29.8 vs. 41.4 on average) and distinct accepted cells in all 5 (34.8 vs. 54.6). The gain comes from the feedback-only mutation engine and dynamic cells, which add about 73 evaluated candidates per run (about 40% more wall time); downstream search used the deterministic tiers only, and the prover was disabled. A complementary generator-level ablation (one generation from the same registry, gates fixed, deterministic) agrees: feedback adds +54% hypothesis-cell coverage (a cell = a conjunction of face conditions, e.g. p4=0, p5<=2), +52% accepted conjectures, and 14 cells that only feedback reaches.

Q1 ("enumerate realizable p-vectors to f2<=36 and fit the tightest linear bounds"). Thank you for proposing this concrete baseline; a new paragraph in Sec. 5.1 compares against it. In short, enumeration can reach fewer than half of our refutations. Exhaustive enumeration is infeasible - the number of simple 3-polytopes explodes long before f2=36 - so "enumerate to f2<=36" is feasible only as our plantri screen (enumerate candidates, realize each), which caps large faces. Of the 204 witness-verified refutations, at most 96 (47%) fall in any range the screen can enumerate, and only 54 within its default caps; the screen first-found 57 (43 inside the default caps, 14 via per-hypothesis cap-widening), and the loop refuted the remaining 147 via witness replay (103), the stochastic tracks (41), and the Eberhard-lattice search (3). Run alone on the 101 first-find conjectures, plantri, the Eberhard lattice, and Hopper recover only 1 of the 40 conjectures first refuted by RL or LLM (App. K). And fitting is not proving - no analog for the four Lean theorems or the open-case localization.

(continued in Part 2/2)

AuthorsOpenReview ↗

Stochastic-track contribution, trusted base, and informativeness (2/2)

(Part 2/2, continued from above)

Q2 (do the LLM/RL tracks find counterexamples a larger-bound plantri would miss?). Yes, and this is where learning matters most. All 41 counterexamples first found by the stochastic tracks (LLM 23, RL 17, Hopper 1) fall outside the screen's enumeration caps; 37 exceed the large-face cap (S up to 18; 16 also exceed the max face size) rather than the f2 cap, so a larger f2 bound does not recover them, and raising the large-face cap is intractable: the multiset count rises from 8,568 at a cap of 5 to ~2e8 at S=18. The remaining 4 exceed only the f2 cap. The RL track is PPO over a GNN editing a planar cubic graph (truncate a vertex / stop), reward +100 per validated counterexample; the LLM (Claude Opus 4.7; Sonnet 4.6 in runs before July 2026) is co-proposer, reviewer, and counterexample constructor, and Opus 4.7 is the prover, gated by compilation and the zero-sorry audit; Hopper applies random local moves to the dual triangulation. Run alone, plantri, the Eberhard lattice, and Hopper recover none of the 17 RL first finds (App. K).

Weakness 2 (soundness / the trusted base). Thank you for pressing on this; it lets us correct a point that bears directly on soundness: the Jučovič bound is not assumed. It is proved inside the artifact, with zero sorry, from elementary face-count and occupation facts, and Thms 2, 3, and 4 use no deep bound at all (Thm 4 uses only p_range). The only assumed published bound is Barnette's, and only Thm 1 uses it (per-theorem dependencies in App. K). The trusted base is now eight declarations (App. A corrects the submitted count of 13): six elementary face-count and occupation facts that carry sorry only because Mathlib lacks planar-embedding theory (e.g., Euler's formula for plane maps), one existence statement used by no proof, and the Barnette bound; all four proofs and every helper above the base are sorry-free. Second, vacuity. A proof could be vacuous only if the base were inconsistent or a lemma it uses were false; we exclude the first by construction and test the second exhaustively (App. A). (i) Consistency: every base lemma is guarded by the opaque predicate IsMap, which has no introduction rule, so no fabricated instance can satisfy the base (without the guard, the all-zero map would falsify Euler's formula and yield False); IsMap can be obtained only from the theorem's own hypotheses or from an existence statement that no proof uses. (ii) Truth: we evaluated the exact Lean statement of every sorry-marked lemma a proof uses. Euler, handshake, regularity, p_range, and Barnette hold on all 204 witnesses and 110 further realized polytopes; the occupation facts, which depend on face adjacency, hold exhaustively on all 58,716 simple polytopes with at most 13 faces (Barnette too, on the 33,547 meeting its hypotheses). (iii) The audit has teeth: it caught one mis-stated occupation lemma in the submitted inventory. Every use of it in the four proofs occurs where it holds (p4=0), so we added the hypothesis p4=0 or p3=0, discharged automatically at every call site, and proved the lemma; the generated proofs are byte-for-byte unchanged and recompile.

Nontrivial mathematics / informativeness. We agree that the four theorems are elementary: Thm 4 follows from the definition of f2, and Thm 2 needs only the case S=1, which the agent closed on its own with the inventory's Jučovič inequality. They certify end to end that the formal stage works and are not the contribution. As discussed at the beginning, our contribution is the autonomous loop and its checkable outputs, and there are few mathematical settings where discovery can be automated end to end like this. The loop's mathematical output is of a different kind. It produced 204 refutations, each certified by an explicit polytope that independent code re-verifies, and it sharpens the open cases: for open conjecture C195, its diagnosis is exact - the only remaining obstruction is the single finite case {4:6, 6:1}, which exhaustive plantri enumeration of all 7-face simple 3-polytopes shows non-realizable but the Lean inventory cannot yet express, so C195 stays formally open. For C193, C195, and C198 jointly, our search finds no realizable counterexample among all 271 candidates up to f2<=45 (238 non-realizable, 33 undecided at a 15 s timeout). We highlight these in Sec. 5.3, with a bound-sensitivity table in App. K, and added a terminology table (App. J).

If these resolve your Q1/Q2 and soundness concerns, we would be grateful if you would reconsider your rating and the Soundness score.

Reviewer_y6yzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a fully autonomous AI system to work on mathematical problems, composed of 3 main agents, the conjecturer, the refuter and the prover. They apply their framework to a family of problems in polyhedral combinatorics, obtaining along this way several results and conjectures. Their system supports Lean formalization, and a conjecture is concluded when either a Lean proof or a counterexample has been found.

The family of problems considered is narrow enough so that a counterexample is easy to verify. Once a conjecture is proposed, the system first tries to find a counterexample with methods in increasing order of cost. If no counter-example is found, then the conjecture proceeds to the next stage of Lean proving.

Critical strengths

-None

Critical weaknesses

-It is unclear how novel this approach is.

-Very little ablation studies, maybe simpler systems with less engineering do as well ?

-There is very little context provided for the mathematical results obtained. Are they of any interest?

-A lot of fancy terms (often not defined) which make the paper hard to follow. Examples "entailment-gated lean formalization”, “Hopper search”, “exhaustive plantri screen”, “Eberhard-lattice search”, etc

Questions for authors

-see weaknesses section above

5 comments
AuthorsOpenReview ↗

Full funnel, enumeration baseline, learned-track contribution, and the trusted base

We thank the Reviewer and address the four weaknesses in order. Your comments led directly to the main changes in the revised PDF (summarized at the end): every term is defined at first use, a terminology table and a worked example are added, and two new experiments test simpler variants of the system. Basic terms used throughout: a p-vector (p3, p4, ...) lists the face counts of a simple 3-polytope with f2 faces in total; it is realizable if some polytope attains exactly those counts; a witness is such a polytope, given explicitly as a planar cubic graph, whose p-vector violates the conjecture under test.

Weakness 1 (novelty). In short: to our knowledge, ours is the first system that closes the full discovery loop without human intervention (Table 1). There are few areas of mathematics where one has the ability to create agents that conjecture, prove and search for counterexamples. The reason is mathematical. Few areas of math have the necessary mathematical infrastructure to allow for such. Our contribution is not an individual component but rather the closed loop itself: the system generates conjectures, searches for counterexamples, formalizes and proves the survivors in Lean, automatically derives hints from every outcome (refutations, proofs, and prover failures), and the generator learns from these hints to propose the next generation, all without human intervention. Every conjecture is carried to a verifiable outcome: a witness-certified refutation, a sorry-free Lean proof (relative to an audited axiom base, App. A), or a documented open case. In Table 1, each prior system covers only a subset. Graffiti/TxGraffiti generate conjectures but neither search for counterexamples nor prove them; the RL counterexample line (Wagner, 2021) refutes but has no proving stage; conjecture-and-prove loops (STP, LeanConjecturer) have no refutation channel, so a false conjecture can only fail to be proved; and the closest concurrent system, AutoGraphForge (released near our submission), screens conjectures against ~348,000 graphs and establishes the surviving relations by hand, so its loop is not closed automatically. Here a counterexample must be an explicit realizable polytope, not an arithmetic solution, so refutation is itself non-trivial; automating it jointly with proving, and feeding every outcome back into the generator, is the contribution we claim.

Weakness 2 (do simpler systems do as well?). No: both the simplest baseline and two simpler variants of our system do measurably worse. The simplest baseline, enumerating realizable p-vectors up to f2<=36 and fitting bounds, can only be run as our plantri screen (Weakness 4), one of our refutation tiers. Of the 204 witness-certified refutations, at most 96 lie in any range it can enumerate, and it finds 57 first. The rest come from the other tracks: 103 by replaying previously found witnesses, 41 from the stochastic tracks (RL 17, LLM constructor 23, Hopper 1), and 3 from the Eberhard-lattice search. Two findings show the extra components are not redundant. First, all 41 stochastic counterexamples lie outside the screen's caps; 37 exceed its cap on the number of large faces (16 also its maximum face size), which no f2 bound relaxes. Second, witnesses are shared across conjectures: 72 of the 103 replays reuse a witness first found by a stochastic track, so stochastic-track witnesses refute 113 of the 204 (55%). The loop's learning step also matters: a generator-level ablation (same frozen conjecture registry, deterministic) shows that feedback from outcomes raises hypothesis-cell coverage by 54% (40 vs 26 cells; a cell is a conjunction of face conditions such as p4=0, p5<=2), and the 14 extra cells are unreachable without it. Two new experiments test simpler variants directly (App. K). Removing outcome feedback, i.e. an open loop with identical budgets for every shared component, loses novel hypothesis cells in 4 of 5 seeds (29.8 vs. 41.4 on average). Dropping the RL and LLM tracks and running plantri, the Eberhard lattice, and Hopper each alone recovers only 1 of the 40 conjectures that RL or LLM refuted first.

(continued in Part 2/2)

AuthorsOpenReview ↗

Context, open conjectures, terminology, and revisions made (2/2)

(Part 2/2, continued from above)

Weakness 3 (context / are the results of interest?). Yes: the underlying question is classical, and the loop produces specific, verifiable statements about it. As mentioned earlier, automated discovery, proof and refutation is simply not possible in all mathematical areas (pick almost any area and it will miss at least one of the elements). A 3 agent loop like this is not even possible anymore in dimension 4: there is no Steinitz-type theorem there, and deciding whether a combinatorial type is realizable as a 4-polytope is complete for the existential theory of the reals (Richter-Gebert, 1996), so witnesses could not be certified by a graph check. Thus ours is a truly unique opportunity to create automated mathematics (outside of a commercial LLM). We agree the submission gave too little context. Which face-count vectors are realized by an actual simple 3-polytope, as opposed to merely satisfying the Eberhard equation, is a classical question: it goes back to Eberhard (1891), the question of which p6 values are realizable is posed explicitly by Jučovič (1971), it is surveyed by Grünbaum (2003), and it remains only partially resolved. The four proved theorems are elementary, and we do not present them as the contribution. The novelty is the rare example of a fully automated mathematical loop. The pipeline does, however, yield specific, verifiable statements. For the open conjecture C195 (if p5=0 and f2>=7 then p6 >= 2 - S, where S is the number of faces with at least seven sides), the loop's diagnosis is exact: the only remaining obstruction is the single finite case {4:6, 6:1} (six quadrilaterals and one hexagon), which exhaustive enumeration of all 7-face simple polytopes shows non-realizable but our Lean inventory cannot yet express, so C195 stays formally open. For C193, C195, and C198 jointly, extending the audit to f2<=45 covers 271 arithmetic candidates and yields no realizable counterexample (238 shown non-realizable, 33 undecided under a 15 s per-candidate timeout). We highlight these in Sec. 5.3.

Weakness 4 (undefined terms). Thank you; we agree, and every term is now defined at first use, with a terminology table (App. J). The Eberhard-lattice search is a random walk over integer p-vectors satisfying the Eberhard equation sum_{k>=3}(6-k)p_k = 12, seeking one that violates the conjecture; the exhaustive plantri screen enumerates candidate p-vectors in a bounded range and decides each one's realizability with the plantri graph generator; the Hopper search applies random local moves to the dual triangulation, seeking a realizable violating face count; and entailment-gated formalization checks, before the Lean prover is invoked, that the conclusion follows arithmetically from the inventory's axioms, and otherwise skips the prover and records the case as "not entailed" (open, never counted as a proof).

Revisions made. We have uploaded a revised PDF (changes in blue). It (i) defines each term at first use and adds a terminology table (App. J); (ii) moves low-level procedures to the appendices and brings the model, RL, and budget summary into the main text (Sec. 4.2); (iii) adds a fully worked example tracing C201 from proposal to Lean proof and C27 from proposal to an RL-found witness refutation (App. I); (iv) audits the formal base: an exhaustive check over all 58,716 simple polytopes with at most 13 faces revealed one mis-stated base declaration, which we replaced by a proved lemma, so the trusted base now has eight sorry-marked declarations and the generated proofs are unchanged (App. A); and (v) adds further analyses (App. K), including a five-seed loop ablation in which outcome feedback raises the number of novel hypothesis cells in 4 of 5 seeds. We hope this addresses the presentation and significance concerns; if it does, we would be grateful if you would consider updating your assessment, and we welcome further discussion.

I thank the authors for the time taken to reply to my comments.

As you say yourself, the areas where your approach is applicable is very narrow. Already in dimension 4 it does not work anymore according to your comment. This would not be an issue to me if your agent obtained an interesting mathematical results. Here by "interesting" I mean a result that human experts on these topics would be interested to know. It is not clear to me that this is the case, as I quote "The four proved theorems are elementary, and we do not present them as the contribution".

In summary, I agree that the approach is novel, but my main concern is the combination of 2 things : (1) the approach applies to a very narrow set of problems, and (2) it does not discover truly interesting and new mathematical theorems.

I would be happy to reconsider my score if any of these two points could be resolved, but I do not know if this is possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.