- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper is well written, so I can read it directly without AI assistance. However, in case I miss something, I still use AI to summarise the content because it does better than I do. I use AI to help me understand the paper while I verify the content myself.
The paper asks an interesting question: Which mathematical problems should an automated prover practise on?
An obvious strategy is to give it difficult problems. But difficulty can be misleading: a generator can make statements longer and more cumbersome without making them particularly educational.
Their alternative is simple:
- Start with elementary mathematical statements produced by a rule-based baseline.
- Make small changes to their structure.
- Filter the resulting statements and attempt to prove them.
- Select some successful proofs as training data.
- Measure whether training on those proofs helps the prover solve separate benchmark problems.
They study propositional logic, natural-number arithmetic centred on multiplication, and abelian groups.
Their main finding is that training on proved structural perturbations beats training on Minimo-generated conjectures under best-first search in all three domains—even when the perturbations are selected randomly. Their new selector, Hard Neighbours (HN), adds a substantial benefit in logic, but does not consistently win elsewhere.
Critical strengths
-
The paper is clearly written and easy to follow. I enjoyed reading it.
-
I find the use of policy-guided best-first search to measure difficulty, together with Hard Neighbours (HN) for training-data selection, interesting and potentially useful for future work.
-
The controlled comparison with Minimo provides useful empirical evidence: under best-first search, the perturbation-based approach consistently performs better in the tested domains.
Critical weaknesses
-
HN’s methodological novelty appears relatively narrow: it selects statements by their increase in proof-search difficulty relative to syntactically nearby, easier references. Although this is intuitively appealing, the paper provides insufficient justification for the particular formulation. The motivating examples do not explain the choices of logarithmic hardness, token edit distance, averaging over easier references, or the reference-update procedure.
-
The experiments remain confined to the small-scale Peano setting used by Minimo (Poesia et al., 2024). This supports a controlled comparison, but leaves the applicability of the findings to richer mathematical domains and stronger provers uncertain.
-
This paper lacks a modern LLM baseline for the generator as an ablation study compared to the rule-based proposer. I think an average ICLR reader will expect this.
-
My AI (GPT-6.1-sol) flagged this paper as an omission: Alchemy: Amplifying Theorem-Proving Capability through Symbolic Mutation (Wu et. al.), which also perturbs symbolic mutation. Please clarify the difference between yours and theirs in the paper.
Questions for authors
-
Did you consider alternative formulations of HN or other selection strategies, including approaches that were unsuccessful? A brief account of the alternatives explored and the reasons for choosing the final formulation would help me assess the contribution. If no alternatives were tested, please clarify this and explain the design rationale.
-
I am not requesting a comprehensive ablation table during the rebuttal. A concise but convincing explanation of these choices would strengthen my confidence in the soundness and could move my assessment towards clear acceptance.
Thank the author for their response.
The author demonstrates that iteratively optimizing conjecture and solver consistently improves over Minimo, which addresses the main concern.
Therefore, I adjust the score from 2 to 3.