- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Tiny Prover, a simple and conventional baseline that combines an off-the-shelf LLM with a Lean REPL in a standard iterative agent loop. The authors demonstrate that this straightforward setup, using GPT-5.4 and Claude Opus 4.6, can nearly saturate or fully saturate two existing benchmarks, VERINA and the Aristotle subset of VeriSoftBench, with GPT-5.4 achieving 100% on both. Overall, the methodological and experimental setup is highly conventional, relying entirely on existing models, tools, and benchmarks.
Critical strengths
To the best of my knowledge, this is the first work to report that a conventional iterative agent equipped with a standard Lean REPL and powerful frontier LLMs (GPT 5.4, Claude Opus 4.6) is sufficient to saturate both VERINA and the Aristotle subset of VeriSoftBench.
Critical weaknesses
Limited Novelty and Contribution. My primary concern is the limited novelty of this work. The entire novelty of the paper can essentially be summarized in the single observation I wrote in the Strengths section above. However, the strong capabilities of modern LLMs in interactive theorem proving have already been well demonstrated in prior work (e.g., APOLLO, A Minimal Agent for Automated Theorem Proving, and LEAP). Beyond reporting this particular empirical result, the paper offers neither a new methodology nor a sufficiently distinct scientific insight. I therefore remain unconvinced that the novelty and overall contribution of this work meet the bar for ICLR.
Questions for authors
The paper concludes that "a simple architecture (Tiny Prover) of an LLM with a REPL is a strong baseline solution that saturates VERINA and VeriSoftBench." This benchmark-saturation claim is based solely on experiments with GPT-5.4 and Claude Opus 4.6. Have the authors considered the possibility of training-data contamination in these frontier LLMs, given that both benchmarks were publicly available before the models were released? If so, how might potential contamination affect the validity of the paper's central conclusion?
Weakness
While it is true that strong capabilities of modern LLMs in theorem proving have been established elsewhere, we argue that it is not the correct characterization of our work. To elicit these capabilities, these works typically propose an elaborate harness that captures aspects such as recursive decomposition, iterative refinement, search, etc. We ask if these harness complexity is necessary for modern LLM, instead of just allowing the LLM to use Lean's feature. For example, in the "A Minimal Agent for Automated Theorem Proving" work as suggested by the reviewer, the authors explicitly forbid the usage of
sorryor suggestion tactics likeapply?, even during the code-proof development process. We impose no such restriction when the model uses the REPL, and only forbid shortcuts (e.g.,sorry,admit,axiom, etc.) in the final proof, which allow the model to freely use all available Lean features that the model finds useful, such as#find,#print,apply?, etc. Any harness design that requires the model to submit a complete proof without direct interactive Lean access will typically prevent the usage of these interactive features.As explicitly stated in the title, Tiny Prover is not meant to be a state-of-the-art solver, but rather a baseline that more complex vericoding agentic provers should outperform to justify the additional complexity (lines 456-459). It makes little sense to just design another complex agentic prover for the sake of a novel harness. We first set up Tiny Prover to study the bottleneck of Lean vericoding that requires custom harnesses before designing our own. To our surprise, Tiny Prover effectively solves existing benchmarks like VERINA and VeriSoftBench (Aristotle). Furthermore, we chose GPT-5.4 and Claude Opus 4.6 as these models have been used in prior works but provide no evidence of saturating benchmarks like VERINA. We hypothesise that the direct, unconstrained access to Lean's features is what helps elicit the LLM's inherent performance and study them correspondingly.
Benchmark leakage
We have several observations that suggest the current performance is not due to benchmark leakage: