- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper asks whether hand-designed proving workflows (planning, decomposition, multi-agent orchestration) are still needed with frontier models, or whether a simple loop given enough test-time compute does as well. It introduces h1, a minimal Lean harness: the model edits a persistent proof file over a multi-turn ReAct block with immediate compiler feedback and optional search/REPL/LSP tools, submits to an external validator (SafeVerify), and on failure a reflection LLM compacts the transcript into progress and lessons before the next iteration, continuing until acceptance or a cost cap; no proof strategy is prescribed. Evaluating solve rate against API-equivalent cost, h1 reports state-of-the-art results (e.g., 670/672 PutnamBench with GPT-5.5 in one run; 90/100 FATE-X, 78/100 FC100-solved, 14/23 FormalQualBench, and 6/6 IMO 2026 with GPT-5.6 Sol), matches or surpasses specialized systems (Goedel-Architect, Numina, MerLean, OpenGauss, AlphaProof Nexus) at matched models and comparable cost in most cases, proves nine open Formal Conjectures targets and exposes several misformalizations, and uses same-model comparisons with AxProverBase and Claude Code plus ablations to identify which harness properties (efficient editing and feedback, persistence, context reuse and compaction, reasoning effort) drive efficient scaling.
Critical strengths
-
A timely, cost-aware evaluation of harness design. The paper makes a point the field needs: "pass@1" now means very different amounts of compute across systems (e.g., 16 blueprint refinements in Goedel-Architect vs. 50 iterations in AxProverBase), so solve rates alone cannot separate harness design from compute. Holding the model fixed and plotting solve rate against inference cost is the right framing, and the results — strong across PutnamBench, FATE-X, FC100-solved, FormalQualBench, and open problems, with new formal proofs and misformalizations found and fixed upstream — make a convincing case that a simple loop is a strong baseline that specialized workflows should be compared against.
-
Controlled analysis of what a simple harness must provide. The same-model reruns of AxProverBase and Claude Code and the ablations (tools, prompts, reflection, turns between reflections, reasoning effort; several with bootstrap intervals) yield concrete, transferable lessons: efficient incremental editing with immediate feedback, robust handling of predictable failures, a loop that enforces persistence, cache reuse with periodic compaction, and more thinking per turn. The paper is also candid about its exceptions (MerLean on FormalQualBench, full APN on Erdős 125).
Critical weaknesses
1. The cost comparisons need clearer accounting. The specialized baselines are not rerun; their scores and costs are taken from prior work, while h1's costs are API-equivalent estimates from its own token logs (for DeepSeek V4 Flash, with cache reuse reconstructed rather than measured; App. D.5). The quoted results also come from differing benchmark revisions and Lean/Mathlib versions; for example, PutnamBench corrected formalizations during the campaign, and the 672/672 result combines the original run with corrected statements (App. D.1, D.4). Since the paper's central claims are about matching or beating these systems at comparable cost, it should state clearly, for each comparison, how each side's cost was computed (price card, cache accounting, whether failed attempts are included, campaign total vs. per-problem vs. per-success), which benchmark revision and Lean/Mathlib version each side used, and how sensitive the conclusion is to these choices. I recognize that some baselines have no released code and cannot be rerun.
2. Framing, limited methodological novelty, and reproducibility.
- The "simple loop vs. specialized workflow" framing understates how much h1's own designed components matter. Reflection in particular plays a central role: without it, h1 with DeepSeek V4 Flash needs $0.78 per problem to match Goedel-Architect's pass@1 score (75.6%), against Goedel-Architect's $0.44, and plateaus at 82.4% at $1.96, below Goedel-Architect's 88.8% at $1.47 (Table 11). Likewise, the detailed prompts that tell the prover not to give up and the reflection LLM not to endorse giving up drive much of the open-problem performance (Fig. 5). h1 prescribes no proof strategy, but its persistence and reflection machinery is itself a hand-designed workflow, so the harness is specialized in its own way; the paper should frame its contribution accordingly.
- The loop builds directly on APN basic, AxProverBase, Reflexion, and the Ralph loop (Sec. 2.2), and APN already observed that its basic agent can Pareto-dominate its full system with repetition (Sec. 3.3). The contribution is primarily careful engineering and a large empirical study rather than a new method; this is valuable, but the paper's claims should be read accordingly.
- Because the conclusions hinge on interface details, reproducibility matters, yet the code is not released ("we hope to release the harness and analysis code at the paper's final decision"; p. 10). Contamination is also not measured (acknowledged in Sec. 5): PutnamBench has been public since 2024 and FC100-solved has known informal solutions.
Questions for authors
- For the Goedel-Architect comparison, how sensitive is the cost conclusion to the reconstructed cache reuse? Could you report the DeepSeek cost with no cache discount (or a range), and, where possible, rerun Goedel-Architect (or another specialized baseline) under your own cost accounting?
- Can you report repeated runs (or bootstrap over problems) for the main comparative claims, especially FormalQualBench, the Putnam 2025 comparisons, and the Goedel-Architect PutnamBench curve, and report cost including failed attempts throughout?
- Can you report wall-clock time alongside cost for the matched-model comparisons with specialized systems?
- For Green 25 and Written on the Wall II Conjectures 100 and 314, who repaired the statements, and how was it verified that the repaired statements express the intended conjectures?
- Will the harness, prompts, tool wrappers, and full trajectories (including failed attempts) be released? This seems essential for a paper whose conclusions depend on interface details.
- Can you say anything about contamination, e.g., results restricted to problems or statements released after each model's cutoff (beyond Putnam 2025 and IMO 2026)?
4 comments
C1, Q1, Q2 (cost accounting). "The cost comparisons need clearer accounting." "how sensitive is the cost conclusion to the reconstructed cache reuse?" "report cost including failed attempts throughout"
A. Each comparison is like for like: both sides use the same model at the same public API prices, the same benchmark statements and Lean version, and the same reporting convention, with failed attempts included.
It is true that we use the reported numbers of most systems. But the costs they publish are computed like ours: both are ultimately what the API of the same model charges at its public prices (Table 9). In particular, the "reconstructed" cache reuse of our DeepSeek run is not a noisy estimate: it is a direct count, and its small margin of error leaves h1 well below Goedel-Architect's cost. We also ran two baselines ourselves, AxProverBase and Claude Code, so that for these we can compare whole cost curves and not single points (Section 4). Similarly, the benchmark statements and Lean versions are the same on both sides wherever the baseline reports them. Finally, we report each cost the way the baseline does, and in most comparisons this already includes the failed attempts. The details of each point are in the appendix comment below, if needed.
C2 (framing). "The 'simple loop vs. specialized workflow' framing understates how much h1's own designed components matter."
A. Our claim is that a prescribed proof workflow is not needed to reach these results, and it holds as stated: h1 reaches them, and h1 prescribes no proof workflow. However, effective and efficient scaling is indeed enabled by reflection and the prompts. We agree with your feedback: we now emphasize them better and frame the claim around them. Yet we point out that they are generic, and not a design specific to proving.
Neither mechanism is specific to Lean, or even to proving. Reflection compacts the context so that a long run can continue, and the prompts keep the model from giving up: both address problems of long agentic runs in general, and only their wording mentions Lean. h1 is, in a sense, a recipe for agentic test-time scaling. In the revision, we now write "test-time scaling in a simple agentic loop", and Section 3 opens by saying that the tools and reflection are what turn the added compute into progress.
C2 (novelty). "The contribution is primarily careful engineering and a large empirical study rather than a new method."
A. We thank you for recognizing the careful engineering and the large empirical study, and we agree that h1 builds on these loops. However, although the combination is simple, it is new: none of these loops forms such a tight and efficient loop, with an inner multi-turn loop with Lean, external validation, and reflection that closes the loop. This combination realizes a recipe for effective and efficient test-time scaling whose results the previous loops did not achieve, and we ablate each of its ingredients.
APN basic is the closest to h1, but it overtakes the full APN system only through repetition, at pass@k: it does not show that a single attempt (pass@1), given more budget, can do the same, which is what h1 shows. AxProverBase regenerates the whole proof at each iteration, which is where it loses efficiency, resulting in it not being state of the art in terms of results. And the Ralph loop is only an abstract concept: it specifies a soft verification, under a completion signal that can be noisy, whereas h1's signal is a formal check by Lean.
C2, Q5 (reproducibility). "reproducibility matters, yet the code is not released."
A. We did everything we could to give as many details as possible in the paper itself. We will not be able to release the code before the end of the discussion period; however, we hope to release it in case of a positive decision, so that others can corroborate our results. The appendix gives the tools (Table 3), both system prompts in full (App. B.2), the error handling (App. B.3), the validation pipeline (App. C.2), and the ablation settings (App. F).
C2, Q6 (contamination). "Contamination is also not measured."
A. Contamination is not a challenge specific to our paper: since both sides use the same model, it would affect the specialized workflows just as much as the simple loop.
It is true that we mention in the limitations that contamination could help a simple loop close the gap with specialized workflows. However, we use the same models as the previous methods, so we do not introduce any new source of contamination, and the same contamination would also call into question every algorithmic contribution evaluated on these benchmarks. We acknowledge it as a limitation, but we believe that solving it is outside the scope of this paper. We recognize that this is not a fully satisfying answer. What we did do is make the timing visible: Fig. 21 (App. E.10) places the release dates of the models and of the benchmarks on one timeline.
Q2 (repeated runs). "Can you report repeated runs (or bootstrap over problems) [...]?"
A. Many of our experiments are single runs, but they consistently give the same conclusion, which makes them stronger together: in five comparisons with the same model on both sides, across four benchmarks and four model providers, h1 matches the specialized system or comes within one problem of it. Repeating runs on full benchmarks is very expensive, so we repeated them where we wanted controlled experiments, on subsets or small benchmarks (Figs. 4, 22). For the three comparisons you name:
- Goedel-Architect PutnamBench curve. As you suggest, we computed a bootstrap over the 672 problems: h1 solves 598 problems (95% interval: 582–614) at \0.94–\1.47 is outside the interval. Note, however, that this bootstrap measures the variation over problems, and not over reruns, so these intervals are wider than the run-to-run uncertainty.
- Putnam 2025 and FormalQualBench. Numina and OpenGauss did not report repeated runs either, so it is hard to know whether their result is a typical run or their best one. On Putnam 2025, we did not repeat the runs, as the first ones already solved all 12 problems with GPT-5.2 and were well above Numina with Opus 4.5 (7 against 4). On FormalQualBench, h1 with Opus 4.6 solves 7, 6, 8, and 6 problems over four runs, against 8 for OpenGauss. Table 2 now lists all four.
Q3 (time). "Can you report wall-clock time alongside cost [...]?"
A. Yes, Table 15 reports time next to cost for every row of Table 2. Where both systems solve the same number of problems, h1 is faster than Humanfia, on par with OpenGauss, and slower than Numina. There is a trade-off here: h1 is a single agent, with no subagents working in parallel. We believe that parallelism is an orthogonal dimension, which could be added to a simple loop as well.
Q4 (statement repairs). "For Green 25 and Written on the Wall II Conjectures 100 and 314, who repaired the statements, and how was it verified [...]?"
A. The corrected statements have been reviewed by the maintainers of Formal Conjectures and are now in the repository, marked as solved. Some corrections and proofs came from us and others from other contributors, working concurrently, and ours match those now in the repository. Note that the revision now reports eight open problems instead of nine (Table 1): Green 25 has since been proved in the repository by another contributor, in a stronger form, so we removed it from our list.
Your feedback made the paper clearer. We hope these clarifications and changes address your concerns and can be reflected in your final assessment, and we are happy to clarify anything that remains.
appendix on cost accounting (C1, Q1, Q2)
This comment gives the details behind our answer on cost accounting. Line, figure, and table numbers refer to the revised PDF.
- Cache sensitivity. Our server did not report cache hits, so we count them: in each request, the part of the prompt that repeats the previous request. This gives 92.3% (App. C.5), close to what other providers reported for the same loop (91.5% for Anthropic and 93.8–96.2% for OpenAI, Table 12). Implementation details, such as the size of the cache blocks, are worth about 1% of the cost, and h1 would reach Goedel-Architect's $1.47 only if the cache rate fell to 87.4%. In any case, the cache changes only the cost, and not the solve rates.
- Benchmark revisions and Lean versions (Table 7). On PutnamBench, h1, Goedel-Architect, and Humanfia all use the same Lean/Mathlib 4.27.0, and only one of the 672 statements changed between the release of DeepSeek V4 Flash and the revision we use. Only the last two problems of our 672/672 result use newer statements, precisely to match Aleph and Humanfia (App. C.4). FormalQualBench has had a single release, and on Putnam 2025 we use Numina's own statements and Lean version.
- Failed attempts. In the average spend per problem, an unsolved problem counts at its full budget (App. C.3, l.1728–1732). The only two exceptions are FormalQualBench and the open Erdős problems, where the benchmark's reporting protocol counts the solved problems only. Even there, for the Erdős problems, the revision also reports h1's spend with failed attempts included (Fig. 13). Lastly, the best@6 row of Table 2 is labelled "cost excludes failures": it shows how cheaply a favourable run reaches 12/12, and the pass@4 row above it gives the cost with failed attempts included.
We thank you for your review and your time, and for the careful reading, appendices included.
Your two main concerns are the clarity of the cost accounting and the framing of h1's own design. We did account for costs carefully in the submission; perhaps spreading the details over several appendices made it unclear, and we now gather them in one place. We also reworded "simple test-time scaling" to credit the tools and reflection that make it possible. More generally, the revision improves the overall presentation and framing of the paper, with all changes marked in blue.
We answer your points below, numbered as in your review (C for weaknesses, Q for questions). Line, figure, and table numbers refer to the revised PDF.