acceptodds
Under review as a conference paper at ICLR 2027

Automated Conjecturing: Generation, Selection, and Evaluation

Abstract

Deciding which conjectures are worth proving is a crucial part of mathematical discovery, and a system that learns mathematics from its axioms alone must make that decision without human guidance. Self-play provers such as Minimo learn a conjecturer alongside the prover and reward it for producing hard conjectures, but a conjecture that is harder to prove is not necessarily more useful for improving a prover: the learned conjecturer drifts toward long, artificial statements. We study conjecturing in a controlled, single-round setting in the Peano formal environment, across propositional logic, abelian groups, and natural-number multiplication, evaluating a set of conjectures by how much a fixed prover improves on held-out theorems after training on their proofs. We show that a far simpler generator does better: making small structural edits to statements the prover already knows yields conjectures that, at an equal training budget and under best-first search, improve the prover by 6 to 13 percentage points more than Minimo's learned conjectures. The gain holds even when conjectures are drawn at random from the generated pool. In contrast to the common practice of selecting the most difficult statements, we also propose Hard Neighbours (HN), a score that favors conjectures whose proof difficulty grows disproportionately relative to their structural distance from easier reference statements; HN gives the largest selection gains in propositional logic, but no selection method is best in every domain. In the current setting, how conjectures are generated matters more than how they are selected, and structural perturbation is the baseline that future conjecturers should be measured against. Apart from improvements in proving ability, we also study other attributes of generated proofs: the amount of search required and the length of the proofs.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
conjecturing, discovery, interestingness, theorem proving

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Mbw9OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper is well written, so I can read it directly without AI assistance. However, in case I miss something, I still use AI to summarise the content because it does better than I do. I use AI to help me understand the paper while I verify the content myself.

The paper asks an interesting question: Which mathematical problems should an automated prover practise on?

An obvious strategy is to give it difficult problems. But difficulty can be misleading: a generator can make statements longer and more cumbersome without making them particularly educational.

Their alternative is simple:

  1. Start with elementary mathematical statements produced by a rule-based baseline.
  2. Make small changes to their structure.
  3. Filter the resulting statements and attempt to prove them.
  4. Select some successful proofs as training data.
  5. Measure whether training on those proofs helps the prover solve separate benchmark problems.

They study propositional logic, natural-number arithmetic centred on multiplication, and abelian groups.

Their main finding is that training on proved structural perturbations beats training on Minimo-generated conjectures under best-first search in all three domains—even when the perturbations are selected randomly. Their new selector, Hard Neighbours (HN), adds a substantial benefit in logic, but does not consistently win elsewhere.

Critical strengths
  • The paper is clearly written and easy to follow. I enjoyed reading it.

  • I find the use of policy-guided best-first search to measure difficulty, together with Hard Neighbours (HN) for training-data selection, interesting and potentially useful for future work.

  • The controlled comparison with Minimo provides useful empirical evidence: under best-first search, the perturbation-based approach consistently performs better in the tested domains.

Critical weaknesses
  • HN’s methodological novelty appears relatively narrow: it selects statements by their increase in proof-search difficulty relative to syntactically nearby, easier references. Although this is intuitively appealing, the paper provides insufficient justification for the particular formulation. The motivating examples do not explain the choices of logarithmic hardness, token edit distance, averaging over easier references, or the reference-update procedure.

  • The experiments remain confined to the small-scale Peano setting used by Minimo (Poesia et al., 2024). This supports a controlled comparison, but leaves the applicability of the findings to richer mathematical domains and stronger provers uncertain.

  • This paper lacks a modern LLM baseline for the generator as an ablation study compared to the rule-based proposer. I think an average ICLR reader will expect this.

  • My AI (GPT-6.1-sol) flagged this paper as an omission: Alchemy: Amplifying Theorem-Proving Capability through Symbolic Mutation (Wu et. al.), which also perturbs symbolic mutation. Please clarify the difference between yours and theirs in the paper.

Questions for authors
  • Did you consider alternative formulations of HN or other selection strategies, including approaches that were unsuccessful? A brief account of the alternatives explored and the reasons for choosing the final formulation would help me assess the contribution. If no alternatives were tested, please clarify this and explain the design rationale.

  • I am not requesting a comprehensive ablation table during the rebuttal. A concise but convincing explanation of these choices would strengthen my confidence in the soundness and could move my assessment towards clear acceptance.

Reviewer_E7pNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper explores conjecture in which they look for what helpful conjectures are to be proven to consequently support models from being more performant and efficient provers. They conduct this work in the Peano environment. They generate conjecture using two methods, minimo self-play generation and by altering existing statements. They also consider a new way of measuring helpfulness of a conjecture, Hard Neighbours, by looking at perturbation of existing statement and observing how much the permutation force the model to follow more steps for the proof.

Critical strengths
  • the idea and approach is well-motivated, I clearly understood why this is important and how it is different from the other conjecturing papers
  • the idea of usefulness of conjecture and how it's defined as making the proof more efficient and shorter is a great way of defining it
  • the results and the comparison of Minimo and HN is very appropriate, the rigour is there.
Critical weaknesses
  • the paper is difficult to read and understand. The big idea is understandable and I could follow it but the detail are often difficult to fully grasp from just one pass.
  • Lack of examples in the main paper showing the reduction of the proof steps, to encourage why something is more useful, weakens the legibility of the paper.
  • the use of Peano seems very specific in this paper especially when STP uses Lean. It would be good to explain the specific choice of Peano over something like Lean.
Questions for authors
  • Why do you use Peano over Lean or even Isabelle?
  • Can you give an example of how usefulness of conjecture helps?
Reviewer_HzJrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes an automated conjecture generation technique for training formalized Peano prover. Starting from seed conjectures (partially from a self-play style prover Minimo), conjectures are structually perturbed in their syntax trees. Second, conjectures are filtered by prover to formally check correctness, and sampled in favor of conjectures with difficult proofs. At last, the selected conjectures are used to train the prover once (without self-play bootstrap). Empirical results show that this simple heuristic can improve prover performance compared to Minimo-generated conjectures, even though there is not a learned generator.

Critical strengths

The empirical results of this paper surprisingly reveal that structural perturbations combined with simple selection heuristics can outperform a learned conjecture generator. For example, even random selection from structural perturbations improves solve rate over training on Minimo-generated conjectures, leading to above 5% improve on all the three mathematical domains considered in the paper. This discovery challenges the necessity for learning to generate conjectures, and potentially inspire a simplification for self-play style formalized prover.

Critical weaknesses

The major weakness is the failure to validate the conjecture generation approach in the self-play framework. The current empirical results are constrained to generating conjectures for the initial prover, and the prover is trained on the generated conjectures once. Similar to Minimo, one can generate conjectures for the improved prover and train the prover on the new batch of conjectures again. The self-play style technique generates conjectures adapting to the increasing capability of prover across training. However, the current empirical results fail to establish the efficacy of proposed conjecture generation technique when adapting to the prover capability. It is therefore not an absolutely fair comparison to Minimo.

Also, the major improvement of the proposed technique can be attributed to the structural perturbations, while the heuristic sampling leads to a smaller gain compared to random sampling.

Questions for authors

Can the authors provide the solve rate of prover when iterating the optimization of prover and the conjectures together, in the self-play framework, and compare to that of Minimo?

2 comments

Thank the author for their response.

The author demonstrates that iteratively optimizing conjecture and solver consistently improves over Minimo, which addresses the main concern.

Therefore, I adjust the score from 2 to 3.

We thank the reviewer for the constructive feedback and for raising their score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.