- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper works on auto research workflow with LLMs, especially Gemini 3.1 pro. They focus on the Busy Beaver number problem, which asks about how much steps can a terminating program of a fixed size run. Their agentic workflow combines both informal and formal methods, and established several concrete theoretical results on this problem.
Critical strengths
- The major results in this paper is very sound as theoretical verifiable discoveries with formal verification. These are concrete contributions in the domain of fundamental computer science.
- The main workflow adopted in the paper appears to be fairly simple, which gives it good scalability and broad potential for practical applications.
Critical weaknesses
- This paper is not very informative for researchers in the AI community. These results demonstrate the improved capabilities of language models, but they offer limited guidance for future research on AI4Math and autonomous AI discovery. Overall, the paper does not provide sufficiently many new insights for researchers in the AI community.
- The significance of the discoveries in this work appears to be overstated in some places (Line 086 - Line 089 in Page 2). Although further declared in conclusions, we still prefer more grounded and pragmatic presentation.
Questions for authors
- What do you think researchers working on AI4Math can learn from this work that could meaningfully inform or benefit their future research?
Thank you for your thoughtful review! We are glad you found the workflow simple and scalable. We have updated the manuscript accordingly, with all revisions marked in blue, and would sincerely appreciate it if you would consider updating your review in light of our responses below.
Q1 / W1. What can AI4Math researchers learn from this work?
We added three general takeaways, backed by new stage-wise ablations and a coding-agent baseline (new Sections 4.4 and 4.5, Tables 3 and 4, Appendix D), now also stated in the Abstract, Introduction, and Conclusion.
Out of the 21 Cryptids already found by the community, we evaluate on the 20 Cryptids whose pattern is captured by the pattern detection stage (at most 17 can be retrieved, as 3 have a growing number of symbol blocks rather than a fixed number of exponents), checking each conjectured Python rule against our C++ simulator across all initial-simulation transitions, a 200,000-step forward orbit, and up to 500 boundary and modular perturbations of the block exponents (Table 3):
Accuracy on observed data is a poor signal of correctness; conjectures must be tested where data is missing.
Filtering candidates via informal proving before formal verification. Across 14 BB(6) candidates, running our generator-evaluator loop (M=4) on 214 conjectures that passed 100% empirical accuracy and 100,000-step orbit closure rejects 185 conjectures (86.4%) and retains 29 across 6 machines (a 7.4x reduction), all 6 of which (100%) succeed in Lean 4. Since formal verification takes up to 72 hours per run, this lightweight informal critique stage saves substantial compute.
Structuring discovery into dedicated stages. Compared against a Gemini 3.1 Pro coding agent given the machine, detected pattern, initial transitions, and a Python REPL with our C++ simulator and all our compression tools at the same 1.70M-token budget (pass@2), our pipeline retrieves 17 vs. 11 Cryptids (Table 3). Inspecting failed coding-agent runs shows that it stops early on recursive rules (4 machines) and runs out of context budget on systems with four or more coupled counters (2 machines; Appendix D). Giving a model tools is not enough without structuring how it tests and refines hypotheses.
Transfer beyond Turing machines (new Appendix D.4, Tables 5-6). To evaluate generality, we also show that these takeaways transfer to a benchmark of 18 multi-variable integer recurrences (recovering loop-free O(1) closed forms from 24-30 restricted inputs by actively probing unseen boundaries, residues, and thresholds on a black-box evaluator): initial-sample fits match 100% of initial points on all 18 at pass@16 yet recover 0/18 globally, whereas interactive refinement recovers 17/18 (climbing from 7 at N=0 to 17 at N=5) vs. 13/18 for the REPL coding agent (with 36% fewer tokens), proving 6 in Lean 4.
W2. Overstated significance (Lines 086-089).
We removed the sentence on Lines 086-089 (Section 1, third paragraph) noting that major conjectures (Fermat's Last Theorem, the Riemann hypothesis, Goldbach's conjecture, ZF consistency) can be encoded as the non-halting of specific Turing machines. We also replaced the claim that our approach applies to any Turing-complete language (Section 1, sixth paragraph) with an explicit distinction between our Turing-machine-specific tape compression and our domain-general discovery stages (code-based rule synthesis, interactive edge-case testing, informal proof filtering, and Lean 4 formalization), and replaced the broad remarks in the Conclusion (Section 5) with a grounded summary of our verified BB(6) and BB(2,5) accelerations, concrete next steps (richer configuration representations, broader dynamical systems, and automated halting or non-halting proofs), and our three empirical takeaways.