acceptodds
Under review as a conference paper at ICLR 2027

h1: A Taste of the Bitter Lesson in Automated Theorem Proving

Abstract

Advances in automated theorem proving (ATP) have largely been driven by systems built on large language models (LLMs). Formal proof systems often wrap an LLM in a complex harness with pre-designed planning, retrieval, and multi-agent orchestration. However, it is unclear whether benchmark advances come from these inductive biases, test-time scaling, or more capable models. We build `h1`, a simple harness of generic components that allows for unbounded test-time scaling for theorem proving in the Lean proof assistant, and study how far it can go with only Lean tools, verifier feedback, and self-reflection sustaining progress. `h1` achieves state-of-the-art results and advances the cost–accuracy frontier on PutnamBench, FATE-X, and FormalConjectures100-Solved, and resolves 9 open problems in FormalConjectures. It matches or outperforms more specialized systems, including AlphaProof, Goedel-Architect, Numina-Lean-Agent, OpenGauss, Aleph, and simpler alternatives like Humanfia, AxProverBase, and Claude Code, at matched models where available. On open Erdős problems, scaling `h1`'s pass@1 can be more cost-effective than repetition within `h1` or AlphaProof Nexus (APN) basic, and surpass the full APN system. Our ablations show how tool use, prompting, self-reflection, thinking budget, and turns per iteration shape scaling efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, automated theorem proving, large language models, agentic harnesses, Lean

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_P6gPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether hand-designed proving workflows (planning, decomposition, multi-agent orchestration) are still needed with frontier models, or whether a simple loop given enough test-time compute does as well. It introduces h1, a minimal Lean harness: the model edits a persistent proof file over a multi-turn ReAct block with immediate compiler feedback and optional search/REPL/LSP tools, submits to an external validator (SafeVerify), and on failure a reflection LLM compacts the transcript into progress and lessons before the next iteration, continuing until acceptance or a cost cap; no proof strategy is prescribed. Evaluating solve rate against API-equivalent cost, h1 reports state-of-the-art results (e.g., 670/672 PutnamBench with GPT-5.5 in one run; 90/100 FATE-X, 78/100 FC100-solved, 14/23 FormalQualBench, and 6/6 IMO 2026 with GPT-5.6 Sol), matches or surpasses specialized systems (Goedel-Architect, Numina, MerLean, OpenGauss, AlphaProof Nexus) at matched models and comparable cost in most cases, proves nine open Formal Conjectures targets and exposes several misformalizations, and uses same-model comparisons with AxProverBase and Claude Code plus ablations to identify which harness properties (efficient editing and feedback, persistence, context reuse and compaction, reasoning effort) drive efficient scaling.

Critical strengths
  1. A timely, cost-aware evaluation of harness design. The paper makes a point the field needs: "pass@1" now means very different amounts of compute across systems (e.g., 16 blueprint refinements in Goedel-Architect vs. 50 iterations in AxProverBase), so solve rates alone cannot separate harness design from compute. Holding the model fixed and plotting solve rate against inference cost is the right framing, and the results — strong across PutnamBench, FATE-X, FC100-solved, FormalQualBench, and open problems, with new formal proofs and misformalizations found and fixed upstream — make a convincing case that a simple loop is a strong baseline that specialized workflows should be compared against.

  2. Controlled analysis of what a simple harness must provide. The same-model reruns of AxProverBase and Claude Code and the ablations (tools, prompts, reflection, turns between reflections, reasoning effort; several with bootstrap intervals) yield concrete, transferable lessons: efficient incremental editing with immediate feedback, robust handling of predictable failures, a loop that enforces persistence, cache reuse with periodic compaction, and more thinking per turn. The paper is also candid about its exceptions (MerLean on FormalQualBench, full APN on Erdős 125).

Critical weaknesses

1. The cost comparisons need clearer accounting. The specialized baselines are not rerun; their scores and costs are taken from prior work, while h1's costs are API-equivalent estimates from its own token logs (for DeepSeek V4 Flash, with cache reuse reconstructed rather than measured; App. D.5). The quoted results also come from differing benchmark revisions and Lean/Mathlib versions; for example, PutnamBench corrected formalizations during the campaign, and the 672/672 result combines the original run with corrected statements (App. D.1, D.4). Since the paper's central claims are about matching or beating these systems at comparable cost, it should state clearly, for each comparison, how each side's cost was computed (price card, cache accounting, whether failed attempts are included, campaign total vs. per-problem vs. per-success), which benchmark revision and Lean/Mathlib version each side used, and how sensitive the conclusion is to these choices. I recognize that some baselines have no released code and cannot be rerun.

2. Framing, limited methodological novelty, and reproducibility.

  • The "simple loop vs. specialized workflow" framing understates how much h1's own designed components matter. Reflection in particular plays a central role: without it, h1 with DeepSeek V4 Flash needs $0.78 per problem to match Goedel-Architect's pass@1 score (75.6%), against Goedel-Architect's $0.44, and plateaus at 82.4% at $1.96, below Goedel-Architect's 88.8% at $1.47 (Table 11). Likewise, the detailed prompts that tell the prover not to give up and the reflection LLM not to endorse giving up drive much of the open-problem performance (Fig. 5). h1 prescribes no proof strategy, but its persistence and reflection machinery is itself a hand-designed workflow, so the harness is specialized in its own way; the paper should frame its contribution accordingly.
  • The loop builds directly on APN basic, AxProverBase, Reflexion, and the Ralph loop (Sec. 2.2), and APN already observed that its basic agent can Pareto-dominate its full system with repetition (Sec. 3.3). The contribution is primarily careful engineering and a large empirical study rather than a new method; this is valuable, but the paper's claims should be read accordingly.
  • Because the conclusions hinge on interface details, reproducibility matters, yet the code is not released ("we hope to release the harness and analysis code at the paper's final decision"; p. 10). Contamination is also not measured (acknowledged in Sec. 5): PutnamBench has been public since 2024 and FC100-solved has known informal solutions.
Questions for authors
  1. For the Goedel-Architect comparison, how sensitive is the cost conclusion to the reconstructed cache reuse? Could you report the DeepSeek cost with no cache discount (or a range), and, where possible, rerun Goedel-Architect (or another specialized baseline) under your own cost accounting?
  2. Can you report repeated runs (or bootstrap over problems) for the main comparative claims, especially FormalQualBench, the Putnam 2025 comparisons, and the Goedel-Architect PutnamBench curve, and report cost including failed attempts throughout?
  3. Can you report wall-clock time alongside cost for the matched-model comparisons with specialized systems?
  4. For Green 25 and Written on the Wall II Conjectures 100 and 314, who repaired the statements, and how was it verified that the repaired statements express the intended conjectures?
  5. Will the harness, prompts, tool wrappers, and full trajectories (including failed attempts) be released? This seems essential for a paper whose conclusions depend on interface details.
  6. Can you say anything about contamination, e.g., results restricted to problems or statements released after each model's cutoff (beyond Putnam 2025 and IMO 2026)?
4 comments

We thank you for your review and your time, and for the careful reading, appendices included.

Your two main concerns are the clarity of the cost accounting and the framing of h1's own design. We did account for costs carefully in the submission; perhaps spreading the details over several appendices made it unclear, and we now gather them in one place. We also reworded "simple test-time scaling" to credit the tools and reflection that make it possible. More generally, the revision improves the overall presentation and framing of the paper, with all changes marked in blue.

We answer your points below, numbered as in your review (C for weaknesses, Q for questions). Line, figure, and table numbers refer to the revised PDF.

C1, Q1, Q2 (cost accounting). "The cost comparisons need clearer accounting." "how sensitive is the cost conclusion to the reconstructed cache reuse?" "report cost including failed attempts throughout"

A. Each comparison is like for like: both sides use the same model at the same public API prices, the same benchmark statements and Lean version, and the same reporting convention, with failed attempts included.

It is true that we use the reported numbers of most systems. But the costs they publish are computed like ours: both are ultimately what the API of the same model charges at its public prices (Table 9). In particular, the "reconstructed" cache reuse of our DeepSeek run is not a noisy estimate: it is a direct count, and its small margin of error leaves h1 well below Goedel-Architect's cost. We also ran two baselines ourselves, AxProverBase and Claude Code, so that for these we can compare whole cost curves and not single points (Section 4). Similarly, the benchmark statements and Lean versions are the same on both sides wherever the baseline reports them. Finally, we report each cost the way the baseline does, and in most comparisons this already includes the failed attempts. The details of each point are in the appendix comment below, if needed.

C2 (framing). "The 'simple loop vs. specialized workflow' framing understates how much h1's own designed components matter."

A. Our claim is that a prescribed proof workflow is not needed to reach these results, and it holds as stated: h1 reaches them, and h1 prescribes no proof workflow. However, effective and efficient scaling is indeed enabled by reflection and the prompts. We agree with your feedback: we now emphasize them better and frame the claim around them. Yet we point out that they are generic, and not a design specific to proving.

Neither mechanism is specific to Lean, or even to proving. Reflection compacts the context so that a long run can continue, and the prompts keep the model from giving up: both address problems of long agentic runs in general, and only their wording mentions Lean. h1 is, in a sense, a recipe for agentic test-time scaling. In the revision, we now write "test-time scaling in a simple agentic loop", and Section 3 opens by saying that the tools and reflection are what turn the added compute into progress.

C2 (novelty). "The contribution is primarily careful engineering and a large empirical study rather than a new method."

A. We thank you for recognizing the careful engineering and the large empirical study, and we agree that h1 builds on these loops. However, although the combination is simple, it is new: none of these loops forms such a tight and efficient loop, with an inner multi-turn loop with Lean, external validation, and reflection that closes the loop. This combination realizes a recipe for effective and efficient test-time scaling whose results the previous loops did not achieve, and we ablate each of its ingredients.

APN basic is the closest to h1, but it overtakes the full APN system only through repetition, at pass@k: it does not show that a single attempt (pass@1), given more budget, can do the same, which is what h1 shows. AxProverBase regenerates the whole proof at each iteration, which is where it loses efficiency, resulting in it not being state of the art in terms of results. And the Ralph loop is only an abstract concept: it specifies a soft verification, under a completion signal that can be noisy, whereas h1's signal is a formal check by Lean.

C2, Q5 (reproducibility). "reproducibility matters, yet the code is not released."

A. We did everything we could to give as many details as possible in the paper itself. We will not be able to release the code before the end of the discussion period; however, we hope to release it in case of a positive decision, so that others can corroborate our results. The appendix gives the tools (Table 3), both system prompts in full (App. B.2), the error handling (App. B.3), the validation pipeline (App. C.2), and the ablation settings (App. F).

C2, Q6 (contamination). "Contamination is also not measured."

A. Contamination is not a challenge specific to our paper: since both sides use the same model, it would affect the specialized workflows just as much as the simple loop.

It is true that we mention in the limitations that contamination could help a simple loop close the gap with specialized workflows. However, we use the same models as the previous methods, so we do not introduce any new source of contamination, and the same contamination would also call into question every algorithmic contribution evaluated on these benchmarks. We acknowledge it as a limitation, but we believe that solving it is outside the scope of this paper. We recognize that this is not a fully satisfying answer. What we did do is make the timing visible: Fig. 21 (App. E.10) places the release dates of the models and of the benchmarks on one timeline.

Q2 (repeated runs). "Can you report repeated runs (or bootstrap over problems) [...]?"

A. Many of our experiments are single runs, but they consistently give the same conclusion, which makes them stronger together: in five comparisons with the same model on both sides, across four benchmarks and four model providers, h1 matches the specialized system or comes within one problem of it. Repeating runs on full benchmarks is very expensive, so we repeated them where we wanted controlled experiments, on subsets or small benchmarks (Figs. 4, 22). For the three comparisons you name:

  • Goedel-Architect PutnamBench curve. As you suggest, we computed a bootstrap over the 672 problems: h1 solves 598 problems (95% interval: 582–614) at \1.08perproblem(1.08 per problem (\\0.94–\1.23).Sothetwosystemssolveasmanyproblems,andh1doesitatalowercost,sinceGoedel−Architect′s1.23). So the two systems solve as many problems, and h1 does it at a lower cost, since Goedel-Architect's \\1.47 is outside the interval. Note, however, that this bootstrap measures the variation over problems, and not over reruns, so these intervals are wider than the run-to-run uncertainty.
  • Putnam 2025 and FormalQualBench. Numina and OpenGauss did not report repeated runs either, so it is hard to know whether their result is a typical run or their best one. On Putnam 2025, we did not repeat the runs, as the first ones already solved all 12 problems with GPT-5.2 and were well above Numina with Opus 4.5 (7 against 4). On FormalQualBench, h1 with Opus 4.6 solves 7, 6, 8, and 6 problems over four runs, against 8 for OpenGauss. Table 2 now lists all four.

Q3 (time). "Can you report wall-clock time alongside cost [...]?"

A. Yes, Table 15 reports time next to cost for every row of Table 2. Where both systems solve the same number of problems, h1 is faster than Humanfia, on par with OpenGauss, and slower than Numina. There is a trade-off here: h1 is a single agent, with no subagents working in parallel. We believe that parallelism is an orthogonal dimension, which could be added to a simple loop as well.

Q4 (statement repairs). "For Green 25 and Written on the Wall II Conjectures 100 and 314, who repaired the statements, and how was it verified [...]?"

A. The corrected statements have been reviewed by the maintainers of Formal Conjectures and are now in the repository, marked as solved. Some corrections and proofs came from us and others from other contributors, working concurrently, and ours match those now in the repository. Note that the revision now reports eight open problems instead of nine (Table 1): Green 25 has since been proved in the repository by another contributor, in a stronger form, so we removed it from our list.

Your feedback made the paper clearer. We hope these clarifications and changes address your concerns and can be reflected in your final assessment, and we are happy to clarify anything that remains.

AuthorsOpenReview ↗

appendix on cost accounting (C1, Q1, Q2)

This comment gives the details behind our answer on cost accounting. Line, figure, and table numbers refer to the revised PDF.

  • Cache sensitivity. Our server did not report cache hits, so we count them: in each request, the part of the prompt that repeats the previous request. This gives 92.3% (App. C.5), close to what other providers reported for the same loop (91.5% for Anthropic and 93.8–96.2% for OpenAI, Table 12). Implementation details, such as the size of the cache blocks, are worth about 1% of the cost, and h1 would reach Goedel-Architect's $1.47 only if the cache rate fell to 87.4%. In any case, the cache changes only the cost, and not the solve rates.
  • Benchmark revisions and Lean versions (Table 7). On PutnamBench, h1, Goedel-Architect, and Humanfia all use the same Lean/Mathlib 4.27.0, and only one of the 672 statements changed between the release of DeepSeek V4 Flash and the revision we use. Only the last two problems of our 672/672 result use newer statements, precisely to match Aleph and Humanfia (App. C.4). FormalQualBench has had a single release, and on Putnam 2025 we use Numina's own statements and Lean version.
  • Failed attempts. In the average spend per problem, an unsolved problem counts at its full budget (App. C.3, l.1728–1732). The only two exceptions are FormalQualBench and the open Erdős problems, where the benchmark's reporting protocol counts the solved problems only. Even there, for the Erdős problems, the revision also reports h1's spend with failed attempts included (Fig. 13). Lastly, the best@6 row of Table 2 is labelled "cost excludes failures": it shows how cheaply a favourable run reaches 12/12, and the pass@4 row above it gives the cost with failed attempts included.
Reviewer_X2ifOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether a simple harness without specialized workflow is necessary to achieve state-of-the-art performance in formal proof writing. The authors design a simple harness called h1 that provides efficient interfaces and is easy to scale the test-time computation. They report three main findings. First, simple test-time scaling with the same model can reach state-of-the-art results from specialized systems. Second, efficient tools to interact with files and Lean and continued work after failures are also necessary. Third, continuing an attempt after failure is better than starting several independent attempts. Their work encourages reconsideration of how formal proof harnesses should be designed.

Critical strengths
  • The paper shows that a simple harness with test-time scaling can match state-of-the-art results from more complex, specialized systems.
  • The paper provides extensive evaluation across benchmarks, models, and harnesses, while considering benchmark exposure relative to model release dates.
Critical weaknesses

The following concern is addressed by the authors' reply, including revising the conclusion of the paper and making the contributions from different parts of their pipeline clearer. I increased my score from 3 to 4.

The paper does not fully decouple the benefits of test-time scaling and other special designs in h1. This makes the contributions of components difficult to understand and may overstate the role of test-time scaling. This is the main concern that prevent me to give a higher score.

The title of Section 3 is 'Simple test-time scaling delivers frontier results' and the introduction states that 'a simple loop often matches or Pareto-dominates specialized systems.' The contribution of the efficient interface in their harness is not shown in the results in Section 3. h1 and those specialized harnesses differ in both workflow and tools, so the result does not show whether the gains come from scaling without prescribed workflows or from other harness designs.

Section 4 shows that interface and tool design contribute a lot to h1’s advantage over other simple-loop harnesses. Figures 20 and 23 in the appendix show that removing all optional tools reduces the solved probems from 69 to 49 on Putnam100 and from 63 to 49 on FATE-X. These results demonstrate that the gains from test-time scaling depends on the available tools.

I suggest the authors either provide more evidence that test-time scaling alone can reach SOTA performance, or revise their takeaways and claims to better reflect the contribution of their hand-designed tools and interface.

Questions for authors

I don't have additional questions. Please refer to the Weaknesses section.

3 comments

We thank you for your review and your time, and for recognizing the breadth of the evaluation.

Your main concern is how much of h1's result comes from test-time scaling and how much from its tools and interface. We do not see these as competing explanations: the tools and reflection are what make test-time scaling possible and efficient. Our claim in Section 3 is that a prescribed proof workflow is not needed, and Section 4 measures what the interface contributes and shows that it is generic. Our phrase "simple test-time scaling" may have read as playing down the role of the tools, and we reworded it, as part of a revision that improves the overall presentation and framing of the paper.

We go through your points below. Line, figure, and table numbers refer to the revised PDF.

Q1. "The title of Section 3 is 'Simple test-time scaling delivers frontier results' and the introduction states that 'a simple loop often matches or Pareto-dominates specialized systems.'"

A1. We are happy to clarify what we mean. By test-time scaling, we mean scaling the test-time compute of an agentic reasoning model in a simple loop, and not compute alone (as in a naive single-turn pass@k, for example). You can see the changes in the revised PDF, where they are marked in blue: overall, we replaced "simple test-time scaling" by "test-time scaling in a simple agentic loop", so that "simple" describes the loop, and "test-time compute alone" by "scaling the test-time compute of an agentic reasoning model in this loop". Section 3 now also opens by saying that the tools and reflection are what turn the added compute into progress, and points to Section 4, where we ablate each of them.

Q2. "The contribution of the efficient interface in their harness is not shown in the results in Section 3." "h1 and those specialized harnesses differ in both workflow and tools, so the result does not show whether the gains come from scaling without prescribed workflows or from other harness designs." "removing all optional tools reduces the solved problems from 69 to 49 on Putnam100 and from 63 to 49 on FATE-X. These results demonstrate that the gains from test-time scaling depends on the available tools."

A2. Section 3 claims that a prescribed proof workflow is not needed to reach these results. This claim holds as stated and does not require measuring the contribution of the interface: h1 reaches these results, and h1 prescribes no workflow. However, your concern about the contribution of the interface is warranted, and it is what Section 4 addresses: it measures this contribution and shows that what the interface provides is generic: the ability to search the library and to keep working, neither of which is a design specific to proving. Here is how we read those ablations:

  • Search is a generic capability, and a generic shell would do. The ablation shows that search is necessary in a system like Lean, where a proof has to reuse results from Mathlib to scale efficiently. Its implementation is interchangeable: text search (grep), semantic search, typed search, or the Lean REPL each recover most of the drop you cite (Fig. 22). A plain bash tool could therefore stand in for our compiler and search tools, and Claude Code confirms it: with only its shell and file tools and the same Opus 4.8, it stays within four solves of h1 on FATE-X up to $20 per problem (48 against 52).
  • Reflection is what keeps the scaling going, and it is not specific to Lean either. It compacts the context so that a long run can continue, and it keeps the model from giving up: both address problems of long agentic runs in general. Its effect shows as the budget grows. Claude Code keeps up with h1 at small budgets and then plateaus, while h1 continues to scale (Fig. 5 right). Likewise, without reflection, h1 with DeepSeek V4 Flash starts stalling and does not catch up with Goedel-Architect (Fig. 7).

Q3. "I suggest the authors either provide more evidence that test-time scaling alone can reach SOTA performance, or revise their takeaways and claims to better reflect the contribution of their hand-designed tools and interface."

A3. We did both, and we hope the previous answers have convinced you of this.

  • Claims. We no longer write "simple test-time scaling" or "test-time compute alone", and Section 3 now points to Section 4 for the contribution of the tools and reflection (A1).
  • Evidence. Beyond the reading in A2, two results show how far one gets without a workflow and with very little interface. With the compiler as its only tool, h1 with Gemini 3.1 Pro solves 49 FATE-X problems, above the best previously published score of 34. And Claude Code, a general coding agent, also exceeds that score, with another model, Opus 4.8: it solves 63 FATE-X problems, where h1 solves 83 with the same Opus 4.8 (Fig. 5 right). On FormalQualBench with Opus 4.6, Claude Code solves 7 of 23 problems in our run, where the benchmark's own Claude Code baseline solves only 4 (Table 2). This gain is purely scaling: it is the same agent and the same model, and the only change we make is to let it keep working until the proof is complete. It brings Claude Code within one problem of OpenGauss's 8.

Your feedback made the paper clearer. We hope these clarifications and changes address your concern and can be reflected in your final assessment, and we are happy to clarify anything that remains.

Thank you for the reply. This addresses my concern and I will raise my score.

Reviewer_igoiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies Automated Theorem Proving and it is concerned about lack of clarity about benchmark evaluation and advances, i.e., it mentions that the evaluation systems built around LLMs are complicated and the source of advances in benchmark accuracies remain unclear. The paper proposes a system called h1 which is limited to calling the tools in a ReAct loop with a limited budget. The paper evaluates h1 with a few LLMs comparing it with other alternatives on PutnamBench (12 problems?) , IMO 2026, FormalQualBench, and a few other problems. It reports the $ costs of computations and it claims some cost advantages for the h1 system.

Critical strengths
  • The topic is interesting.

  • I found the motivation and the defined aims of the paper strong but implementation is weak.

Critical weaknesses
  • Experiments do not appear coherent and presentation is weak. Most importantly, accuracies and success rates are reported pass @1, @3, @6 which are uncommon in the literature.

  • While the ${\}\ cost of computations are reported, it is not clear to me what are the token budgets, and overall computational costs. Different companies may charge differently for the same token budget and prices change over time. While providing the $ cost can be informative, for a paper, reporting the token budgets are more essential.

  • It is not clear to me whether the experiments are on entire benchmarks or on a subset of them. For example, paper describes PutnamBench as having 672 problems, correctly, but in the table 2 indicate 12 problems. I found this very confusing.

  • For IMO 2026, all settings solve all the 6 problems, so I guess the point is to cut the computational cost. But again, the reporting is not clear. The cost is reported $8.33 / $8.07/prob for h1+ GPT-5.6 Sol while only one number is reported for the other models.

  • For some of the models, they solve more problems that h1 at a higher cost yet it is not clear whether h1 has tried solving those extra problems or it had not attempted those extra ones. In such cases, the cost savings of h1 might be coming from giving up early or not attempting.

  • Experiments seem to be on 12 + 12 + 23 + 6 problems in total. This is not adequate to evaluate the contribution of the method.

  • Overall, h1 seems like an interesting idea, but it seems to me that it is not well developed yet as a method.

  • It would have been informative to report results on open source models, too, and on full benchmarks.

Questions for authors

Please see weaknesses. I'd be happy to see any additional results or clarifications.

8 comments

We thank you for your review and your time, and for being open to clarifications.

We believe there is a misunderstanding about the thoroughness and exhaustiveness of our experiments, and most of the concerns follow from it.

  • Scale. You read Table 2 (53 problems) as the whole evaluation; Table 2 is one piece of evidence among several. We ran our experiments on full benchmarks: 901 distinct problems, with 13 models across the paper.
  • Accounting. We attempt every problem, and we report solve rates against dollar cost per problem, with failed problems included in the cost. We compare methods with the same model and at the same prices.

The two other reviewers read the same experiments as "extensive evaluation across benchmarks, models, and harnesses" (X2if) and as "strong across PutnamBench, FATE-X, FC100-solved, FormalQualBench, and open problems", making "a convincing case that a simple loop is a strong baseline" (P6gP), and both report no substantive concerns about the paper's soundness.

With this picture in mind, we go through each weakness and question below and point to where the paper supports the two points above. Line, figure, and table numbers refer to the revised PDF.

Q1. "Experiments seem to be on 12 + 12 + 23 + 6 problems in total." "It would have been informative to report results [...] on full benchmarks."

A1. Our main results are on entire benchmarks (l.127–140): all 672 PutnamBench problems (Fig. 1 left), all 100 FATE-X problems, and all 100 FC100-solved problems (Fig. 3). With FormalQualBench (23) and IMO 2026 (6), that is 901 distinct problems. We also use a 100-problem subset of PutnamBench for ablations. The number of problems is stated in the benchmarks paragraph, on the y-axis of the plots ("672 (100%)" in Fig. 1, "Solved / 100" in Fig. 5), and in the denominators of Table 2.

Overall, here are our claims and the number of problems used as evidence for each:

ClaimEvidenceProblemsModels
1. A simple loop reaches state-of-the-art resultsPutnamBench, FATE-X, FC100-solved (Figs. 1, 3); FormalQualBench, IMO 2026 (Table 2)9015
2. At the same model, it matches specialized workflowsGoedel-Architect on PutnamBench (Fig. 1 left); AlphaProof Nexus on Erdős problems (Fig. 1 top right); Numina, MerLean, OpenGauss (Table 2)7075
3. A longer attempt can beat repeated attemptsOpen Erdős problems, 16 runs each at up to $400 (Fig. 1 top right)61
4. Simplicity alone is not enoughAxProverBase and Claude Code reruns, ablations (Figs. 4, 5)2006

Q2. "paper describes PutnamBench as having 672 problems, correctly, but in the table 2 indicate 12 problems."

A2. Those rows are Putnam 2025, the 2025 competition, which has 12 problems. The table labels them "Putnam 2025" and "PutnamBench's 2025 slice", with scores out of 12. The revised benchmarks paragraph now introduces Putnam 2025 and IMO 2026.

Q3. "pass@1, @3, @6 [...] are uncommon in the literature."

A3. pass@1 is the fraction of problems solved with a single attempt: it is exactly the success rate. pass@k is the fraction solved when at least one of k attempts succeeds. We believe there is a misunderstanding here too: these are the standard metrics of the field, used by AlphaProof Nexus (pass@1, @3, @6, @10), Goedel-Architect (pass@1, @4, @32), AxProverBase (pass@1), Seed-Prover 1.5, Leanstral 1.5, and others. We report pass@1 for all benchmark results. The specific values 3 and 6 come from AlphaProof Nexus, which reports pass@1, @3, @6, and @10 on the open Erdős problems: we use the same values so that our results can be compared with theirs directly (l.363–372). We also use pass@k to show when repetition alone closes a gap (l.311–319). We added their definition and how we compute them to Section 2.1.

Q4. "it is not clear to me what are the token budgets, and overall computational costs. Different companies may charge differently [...] and prices change over time."

A4. We argue that dollar cost at the public API prices of each model is the more faithful measure. An output token costs up to 100× a cached input token (l.183–184), so equal token totals can hide several-fold differences in compute. FLOPs would be ideal, but providers do not disclose them and they are hard to compute (l.1700–1703). Any single axis that combines cached input, input, and output tokens needs weights, and the API prices are that weighting, so such an axis is equivalent to reporting dollar cost. We price our runs at the public API rates of each model, frozen at one date (Table 9), so our numbers do not change over time. These are the rates that applied when the methods we compare with were evaluated. Moreover, while we use dollar cost to compare methods, we also use token counts by type to dive deeper into why two methods differ. We report them in our in-depth comparison with AxProverBase, for example, where they show how h1 makes better use of cached tokens (Fig. 18). Reviewer P6gP agrees that "plotting solve rate against inference cost is the right framing".

Q5. "it is not clear whether h1 has tried solving those extra problems [...] the cost savings of h1 might be coming from giving up early or not attempting."

A5. Yes, h1 attempted those extra problems, and the costs we report include them. The cost we report is the average over all problems of a benchmark, solved or not, and a problem h1 does not solve counts at its full budget (l.187–189). The savings therefore do not come from leaving out failed or unattempted problems. There is one exception, which may have caused the misunderstanding: FormalQualBench. Its leaderboard reports the cost of the solved problems only, excluding failures, and MerLean and OpenGauss publish their costs that way, so we follow the same convention to compare with them (l.1738–1741).

Q6. "For IMO 2026 [...] I guess the point is to cut the computational cost. [...] \8.33/8.33 / \\8.07/prob for h1 + GPT-5.6 Sol while only one number is reported for the other models."

A6. This row has two numbers because it reports two h1 settings, with 50 and with 200 turns per iteration (\8.33and8.33 and \\8.07), as the caption of Table 2 states (l.326). Every other row reports a single setting, hence a single number.

Q7. "results on open source models"

A7. We do, and this is important to us. Our main comparison with Goedel-Architect uses the open-weight DeepSeek V4 Flash: 598/672 on PutnamBench (Fig. 1 left, l.291–295). We also evaluate the open-weight Leanstral 1.5 in our comparison with AxProverBase (Fig. 5, left).

Q8. "Experiments do not appear coherent and presentation is weak." "it is not well developed yet as a method."

A8. We understand these overall assessments as following from the concerns above about the experiments. The experiments follow the four claims in the table above, one section each (Secs. 3.1, 3.2, 3.3, 4), and we hope our answers show that h1 is evaluated thoroughly: on full benchmarks (901 problems), with open-weight and frontier models, against the strongest published systems at the same model, and with costs that include failed problems. We also took your remark on presentation to heart: the revision improves the overall presentation and framing of the paper, with all changes marked in blue. If you meant something else, please help us address it by clarifying. The main idea of the paper is that a simple agentic loop, given efficient execution feedback from Lean and a larger test-time budget, matches the specialized proof workflows that formal theorem proving has come to rely on, and our thorough and exhaustive evidence backs this up.

If this clarifies the scope of the evaluation, we kindly ask you to reconsider the soundness score and the rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal outcome and status

Dear Area Chair and Reviewers,

We thank you for your time throughout this process. We briefly summarize the outcome of the discussion.

Strengths noted: "the paper makes a point the field needs", making "a convincing case that a simple loop is a strong baseline" and yielding "concrete, transferable lessons" (P6gP); "a simple harness with test-time scaling can match state-of-the-art results from more complex, specialized systems", with "extensive evaluation across benchmarks, models, and harnesses" (X2if); "the motivation and the defined aims of the paper [are] strong" and "I find the contribution of the paper interesting" (igoi).

Concerns and status

  • Reviewer igoi (rating 1, update pending): evaluation scale, pass@k metrics, token budgets, cost accounting → clarified (full benchmarks, 901 problems), with a claims table and token counts added (Tables 19, 16). The reviewer stated "I don't have any major concerns remaining" and that they would raise their score, which has not been updated yet.
  • Reviewer X2if (rating 3 → 4): contribution of test-time scaling vs. tools and interface → clarified: the two are not competing explanations, as the tools and reflection are generic and are what make test-time scaling effective, which the ablations in Section 4 measure. Wording sharpened accordingly. Addressed; score raised.
  • Reviewer P6gP (rating 3): cost accounting, framing, repeated runs, wall-clock time → clarified: the cost accounting was already in the submission and is now gathered in one place, complemented by bootstrap intervals and repeated runs. On framing, we clarified that h1's components are generic enablers of scaling rather than a proof workflow, consistent with our claim, and sharpened the wording accordingly. We are awaiting the reviewer's response.

Remaining actions: we kindly ask Reviewer igoi to update their scores, and Reviewer P6gP to let us know whether our response addresses their concerns.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.