- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Goedel-Architect is a Lean 4 theorem-proving pipeline that maintains a global blueprint (dependency graph) iteratively refined using proof, disproof, and forfeit feedback. With open-weight DeepSeek-V4-Flash, the 75.6% PutnamBench run costs 0.44 USD/problem (294 USD total). Reported baselines: ~5.8 USD/problem (Goedel-Architect + Gemini at 95.5%) vs 54–68 USD/problem (Aleph).
Critical strengths
-
Open, cost-efficient pipeline at academic scale. 75.6% PutnamBench with 294 USD total spend (~100× cheaper than Hilbert's 3.9k USD on same metric). Open backbone (DeepSeek-V4-Flash) and fully reproducible budgets/prompts in appendix enable verification.
-
Blueprint design beats top-down recursion on same backbone. Section 4.1 (Table 3): Hilbert reproduced on DeepSeek-V4-Flash drops to 83.6% vs Goedel-Architect's 99.2% on MiniF2F. Direct evidence that global blueprint + refinement loop outperforms recursive decomposition when backbone is held constant.
Critical weaknesses
-
Reliance on proprietary models for frontier performance. While the open DeepSeek-V4-Flash pipeline achieves 75.6% on PutnamBench at low cost (0.44 USD/problem), reaching the headline 95.5% solve rate requires seeding with proprietary Gemini 3.1 Pro natural language proofs. This reliance on closed frontier models weakens the claim of a purely open-source solution at the state-of-the-art tier, where the cost advantage over peers like Aleph (54–68 USD) is ~10× rather than the advertised 500×.
-
Incomplete ablation of blueprint mechanisms. The central claim that global blueprints outperform recursive decomposition lacks systematic causal ablation. Disproof and forfeit mechanisms are evaluated only via anecdotal case studies (Appendix B) rather than quantitative ablations disabling these channels, and the backbone comparison against Hilbert is reported only on the easier MiniF2F-test, leaving the harder PutnamBench row blank in Table 3.
Questions for authors
-
Could the authors evaluate Hilbert on DeepSeek-V4-Flash on the 200-problem PutnamBench subset to confirm whether the pipeline advantage over recursive decomposition holds on harder problems under matched backbone and compute?
-
Could the authors provide a quantitative ablation on PutnamBench disabling the disproof and forfeit channels to verify that these specific blueprint refinement mechanisms, rather than simply repeated sampling, drive the gains?
Response to Reviewer fx5Z
We thank the reviewer for the careful reading and concrete suggestions. We address each point below.
We agree that the open-weight and mixed-model results should be distinguished. Our fully open-weight settings achieve 75.6% on PutnamBench in the default run and 88.8% with additional runs and informal-proof guidance. The 95.5% result uses Gemini in both informal and formal stages; we will correct the description accordingly. The reported 500× cost reduction refers to the default setting versus Hilbert, not the 95.5% setting. We will clarify that these cross-system cost comparisons involve different models and budgets and do not establish an advantage under matched compute. It is also worth noting that even with closed-weight models, our system is still publicly accessible, in contrast to AlephProver, which cannot even be accessed via API.
To isolate each component's contribution to solve rate, we ran ablations that remove global graph revision, formal disproofs and structured failure diagnoses in turn. To stay within an academic budget, we used a pre-registered 48-problem PutnamBench subset, drawn at random from problems the default run solved within 1–8 iterations, with quotas so each feedback type is represented. Every arm, including the full system, was re-run from scratch with DeepSeek-V4-Flash-0423 and six refinement iterations.
With the same iteration budget, graph revision using only proved/unresolved feedback achieves a 77.1% solve rate, compared with 50.0% for repeated sampling on a fixed blueprint. Adding disproof and forfeit feedback further increases the solve rate to 89.6%. These results support graph revision's contribution beyond repeated sampling, with informative feedback providing an additional observed gain on this subset.
We have not run the same-backbone Hilbert comparison on PutnamBench. Our existing comparison is limited to MiniF2F, and we acknowledge that it does not automatically establish an advantage over Hilbert on harder problems under matched compute, but we believe it does provide a signal as to the effectiveness of each method.
We thank the reviewer again and are happy to address any further questions.