- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript proposes a closed-loop autonomous system for mathematical discovery, which integrates conjecture generation, counterexample search, formal proof, result coordination and subsequent conjecture generation into a single unattended workflow. The system targets affine inequalities over face-size histograms of simple convex 3-polytopes, where arithmetic feasibility does not necessarily imply geometric realizability. Its key design is a realizability-aware verification pipeline that combines reusable witness replay, Eberhard-lattice search, bounded exhaustive plantri enumeration, and multiple stochastic search methods, followed by a five-check validator for candidate counterexamples. Surviving conjectures undergo an entailment pre-check and are accepted as proved only when the corresponding Lean 4 artifact compiles under a zero-sorry policy. Results are then saved as structured artifacts and fed back to the conjecture generator to guide subsequent exploration while keeping the verification gates fixed. Authors validate the system through a repository-level autonomous campaign. The reported campaign produces 204 witness-verified refutations, four end-to-end machine-checked Lean proofs, and three unresolved conjectures, with more than half of the refutations obtained through replay of previously validated witnesses. The manuscript also provides provenance information for the refutations, explicit witness graphs and validation records, formal proof artifacts, and an audit of the formal inventory underlying the Lean proofs.
Critical strengths
-
By integrating conjecture generation, counterexample search, independent validation, formal proving, and outcome-driven learning into a coherent autonomous workflow, the manuscript clearly separates the proposal and verification components, making them easy for the reader to understand.
-
The authors chose a field with real technical difficulties. Instead of choosing the scenario that "arithmetical establishment equals problem solving", but instead chooses simple convex 3-polytopes: a p-vector can satisfy the Eberhard equation without a corresponding polytope existing, indicating that arithmetic validity does not imply geometric existence. This choice makes the verification of the system not stop at the numerical level, and it is necessary to construct a real plane cubic diagram as a witness. Therefore, counterexample search is not a simple numerical check, but a process that includes substantive verification steps.
-
The campaign produces 204 witness-verified refutations, 4 formal proofs, and 3 remaining open conjectures, demonstrating that the proposed system can execute a nontrivial mathematical discovery loop rather than only generating isolated conjectures or proofs.
-
The realizability-aware counterexample pipeline and five-check validation provide a meaningful guarantee against spurious arithmetic counterexamples, while the Lean 4 zero-sorry proof gate and structured result records improve the reliability and reproducibility of the reported results.
Critical weaknesses
-
The manuscript reports an impressive number of refutations and proofs, but provides limited controlled ablations or matched-budget comparisons that isolate the contributions of the four counterexample search tracks (witness replay, Eberhard-lattice search, plantri screening, and stochastic tracks), the outcome-driven learning mechanism, the five-check validator, and the closed-loop feedback from outcomes to conjecture generation.
-
Although the system executes the discovery loop autonomously after initialization, substantial human design remains in the choice of mathematical domain, conjecture language, search components, formalization inventory, and system configuration. Moreover, the evaluation is restricted to affine inequalities over face-size statistics of simple convex 3-polytopes, making it difficult to assess whether the proposed architecture generalizes to substantially different mathematical domains.
-
The reported Lean proofs are zero-sorry at the generated theorem level, but the formalization relies on a hand-audited inventory containing 13 sorry-marked lemmas. However, the paper does not clearly state that the formal guarantees are relative to this trusted inventory rather than being fully established from a completely formalized foundation.
-
The manuscript is generally understandable, but the presentation is overly procedural and sometimes reads more like a detailed system description than a focused scientific narrative. The central insight and scientific contribution are somewhat diluted by extensive discussion of implementation details and outcome bookkeeping.
Questions for authors
-
This manuscript reports 3 open conjectures, but it is unclear whether they are mathematically open or merely unresolved under the current configuration. For example, Open Conjecture 2 appears to require only one missing realizability lemma. A sensitivity analysis over the enumeration bound (e.g., ) would clarify the status of these conjectures.
-
The manuscript identifies outcome-driven learning as a key component of the proposed system, but the current evaluation does not clearly isolate its contribution from that of the underlying conjecture generator and search tracks. Could the authors compare the full system against a variant that does not use previous outcomes to guide subsequent conjecture generation, under the same computational budget? This would help establish whether the feedback mechanism materially improves the campaign beyond simply integrating existing components.
-
The manuscript reports that 103 of the 204 refutations come from witness replay and interprets this as evidence that the counterexample pool strengthens the discovery campaign. However, this result could also reflect reliance on previously discovered evidence rather than new discoveries. Could the authors report how many replayed refutations apply to genuinely new conjectures, as opposed to duplicate or near-duplicate conjectures? This would help clarify the extent to which witness replay contributes to the discovery of new results.
-
The provenance analysis shows that the search tracks contribute very unevenly, with some tracks producing few or no independently discovered refutations. Could the authors report the marginal contribution of each track, for example by measuring how many unique validated counterexamples are found only by that track under a fixed search budget? This would clarify whether the current multi-track design provides complementary coverage or includes components with limited practical value.
-
The manuscript reports four end-to-end Lean proofs with no new sorry, while acknowledging that the underlying formal inventory contains 13 sorry-marked lemmas. Could the authors clarify whether any of these lemmas are used in the dependency chains of the four reported theorems, and identify the resulting trusted assumptions for each theorem? A theorem-level dependency summary would make the scope of the formal guarantees easier to assess.
-
The manuscript addresses a relevant problem and reports a substantial system implementation. However, the current evaluation primarily reports the overall outputs of the system, such as the number of verified refutations and formal proofs, without sufficiently clarifying the incremental benefits of the proposed framework over the underlying search and proving components. It remains unclear whether the closed-loop organization and outcome-driven feedback provide substantial advantages beyond merely integrating these existing components. The paper would be strengthened by a more explicit characterization of the framework's unique benefits, supported by controlled comparisons, ideally a framework-level comparison against a non-closed-loop pipeline using the same components, supplemented by component ablations where appropriate.
2 comments
Trusted base, presentation, open conjectures, and replay novelty (2/2)
(Part 2/2, continued from above)
W3 / Q5 (trusted base). Thank you; we added the theorem-level dependency summary you asked for. The formal guarantees are relative to the hand-audited inventory, as the abstract, Sec. 4.4, and Limitations now state. App. A lists every sorry-marked declaration, and App. K gives a theorem-level dependency table: C124 uses only p_range; C201 and C215 use six elementary face-count and occupation facts and no deep bound; C104 uses the same six plus Barnette's bound. For this revision we exhaustively checked the base over all 58,716 simple polytopes with at most 13 faces and replaced one mis-stated declaration by a proved lemma; the generated proofs are unchanged and recompile, and the base now has eight sorry-marked declarations (thirteen in the submission).
W4 (presentation). Thank you; we refocused the narrative. The abstract and contributions now state the central claim directly, low-level procedures moved to the appendices, a terminology table was added (App. J), and App. I traces one proved and one refuted conjecture end to end.
Q1 (open, or unresolved under the configuration?). Thank you for this suggestion; App. K now gives the bound sensitivity you proposed: at f2<=20 and f2<=45 (all candidates in fact have f2<=28), no decided candidate of the three open conjectures is realizable, and the remaining 33 are undecided only because of the 15 s plantri budget. C193 and C198 are genuinely open: every candidate has S=1, and their residual case, a single large face of unbounded size, cannot be closed by any finite enumeration. For C195 (your Open Conjecture 2), which stays formally open, the loop's diagnosis is exact: the case S>=2 is immediate, the elementary fact f2>=m+1 closes S=1, and the only remaining obstruction is the finite hole {4:6, 6:1}, which plantri proves non-realizable but the formal inventory cannot yet express (Sec. 5.3).
Q3 (are replayed refutations genuinely new?). Yes. 99 of the 103 replays refute a conjecture in a different hypothesis cell from the one where the witness was first found. Of the other four, two are coefficient variants within one cell, one is a different statement in the same cell, and one pair (C24/C23, both from the early seed batch) is a semantic duplicate, which we disclose. The 103 replays draw on 34 distinct witnesses; the most reused, an RL witness with f2=64, refuted 17 later conjectures (App. I).
Revisions made. (i) The central claim in the abstract and contributions; (ii) App. A: the trusted base and the base audit; (iii) App. K: a five-seed loop ablation, per-track contribution, replay novelty, bound sensitivity, validator evidence, and theorem-level dependencies; (iv) App. I: a worked example; (v) App. J: terminology; (vi) a corrected description of the generator, including its LLM review step. We hope these address the concerns; if they do, we would be grateful if you would consider updating your assessment, and we welcome further discussion.
Value of the closed loop, track contributions, and generality (1/2)
We thank the Reviewer for the positive assessment, in particular for recognizing that this domain makes counterexample search a substantive verification problem, and for the concrete questions. We have uploaded a revised PDF (changes in blue); each of your six questions is answered by a new analysis in App. K. In short, the contribution is, to our knowledge, the first system that closes the full generate-refute-prove-learn loop without human intervention, and the new analyses show that the loop's components are complementary rather than redundant.
W1 / Q2 / Q6 (does the closed loop add value beyond its components?). Yes. We added the framework-level comparison you suggested, and it favors the closed loop. Four pieces of evidence, each isolating a different part of the loop. (1) Feedback to the generator: in the generator-level ablation on the same frozen registry, outcome feedback raises hypothesis-cell coverage from 26 to 40 cells (+54%) and accepted conjectures from 27 to 41; the 14 extra cells are unreachable without feedback. (2) Witness reuse, which exists only because outcomes flow back into the loop: 103 of the 204 refutations replay a previously certified witness, and 99 of these 103 refute a conjecture in a different hypothesis cell (Q3). (3) The validator: in the open-conjecture audit, 238 of 271 arithmetically feasible candidates are non-realizable, so an arithmetic-only pipeline would have accepted every one of them as a counterexample; in an instrumented run of 4,475 validations, the realizability check rejected 32 such candidates. (4) A new loop ablation (App. K): closed vs. open loop over two generations from the same frozen registry, five seeds, identical budgets, gates, and shared sources, with the open arm withholding every outcome signal (which disables the two feedback-only sources). On the pre-registered primary metric, hypothesis cells of accepted conjectures not covered by the frozen registry, feedback wins in 4 of 5 seeds (41.4 vs. 29.8 on average), and on distinct accepted cells it wins in all 5 (54.6 vs. 34.8). The gain comes from the feedback-only sources, the mutation engine and dynamic cells, while the shared sources perform on par: feedback opens directions the open loop never proposes. Those two sources add about 73 evaluated candidates per run (about 40% more wall time); every shared component runs with an identical budget. For time, downstream search used the deterministic tiers only and the prover was disabled.
Q4 (marginal contribution of each track). Thank you; App. K now reports this, and it shows the tracks are complementary. Under a static reachability model, all 17 RL first finds and 14 of the 23 LLM first finds lie beyond the reach of every deterministic tier. Running plantri, the Eberhard lattice, and Hopper each alone on the 101 first-find conjectures, with the campaign's per-batch timeouts, recovers only 1 of the 40 conjectures first refuted by RL or LLM, and none of the 17 refuted first by RL. Conversely, the deterministic tiers run first in the campaign, so first-find counts reflect the order in which tracks run rather than what each can find: run alone, these tracks recover all three Eberhard-lattice first finds and the Hopper first find, and plantri contributes exhaustive non-realizability verdicts that no stochastic track can provide (App. K).
W2 (human design and generality). We agree, and the paper now states it plainly: autonomy concerns execution: humans chose the domain, implemented the components, and audited the inventory. The domain is chosen deliberately: by Steinitz's theorem every counterexample can be certified by a graph check, whereas already in dimension 4 deciding realizability is complete for the existential theory of the reals (Richter-Gebert, 1996). The conclusion lists what another domain must provide to support the loop.
(continued in Part 2/2)