- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces Magenta, an agentic loop for math reasoning that combines natural language (NL) with formal language (FL) to achieve the best performance with verifiable compiler signals.
Critical strengths
Strengths
-
Straightforward idea. The paper presents a sensible, straightforward idea that couples informal inference with Lean verification.
-
Implementation transparency. The authors included sample proofs along with all the templates of Magenta.
-
Open-source experiment. I commend authors for using open-weight models in the experiments for better transparency.
Critical weaknesses
Weaknesses
-
Novelty. After reading the manuscript twice, I am unsure of the novelty of this work. Described methods are just prompted LLMs one after another with mild decision making conducted by an LLM. This seems more like an implementation project rather than a genuine novelty to the theorem proving community. NL-FL integration was explored in numerous works, such as DSP [1], DSP+ [2], Hermes [3], and Numina Lean Agent [4]; and I do not see substantial novelty over these works. Moreover, I would argue that this agent lacks relatively standard features explored in early auto formalization works such as backtranslation [5] for semantic alignment. Right now, the work presents a neat implementation inside existing harnesses for theorem proving. I cannot recommend this work for publishing, since it does not add substantial new knowledge to the community.
-
No human verification. Since the work operates on auto-formalization territory, I would expect to see some human ablations/verifications of results. However, every single experiment is conducted by prompting more LLMs. Authors also mention that final answers were also verified by another LLMs. I understand that checking every single output is tough and unrealistic, but verifying at least IMO proofs should be included with the work, since it is just six problems. Auto-formalization was also not verified, and instead was left at the mercy of the prompted LLM. It is bare minimum to provide human evaluation in the work of this nature. I would like to see human expert evaluation of the outputs and comparison with either real student answers or released golden solutions. Since the work mentions IMO, I believe the IMO statements were also released, so more comparisons and case studies could be conducted.
-
Unfair setting/experimental results. Correct me if I am wrong, but the setting of Table 1 seems to be extremely unfair. Authors mention:
Baselines. We compare against state-of-the-art baselines: Claude Opus 5, Gemini 3.7 Flash, and Kimi K3. We let each baseline only generate one answer (i.e., pass@1 rate).
While Magenta has budgets of:
For all the experiments, we use T = 32 reasoner attempts, M = 512 formal-statement samples per candidate answer, and K = 4096 proof attempts per accepted statement.
First, the crucial hyperparams were buried in the Appendix, which is not a good look. Second, these settings are just unfair. How can you compare running an LLM once against 32 reasoner attempts with 4096 proof attempts? Moreover, Table 1 omits base model running settings; is it also pass@1? If so, then it raises questions on settings, budgets, tokens, etc. This is just brute forcing to get better results. Could the authors clarify this to me?
References
[1] Jiang et al. 2023. Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs. ICLR
[2] Cao et al. 2025. Reviving DSP for Advanced Theorem Proving in the Era of Reasoning Models. NeurIPS
[3] Ospanov et al. 2026. HERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMs. ICML
[4] Liu et al. 2026. Numina-Lean-Agent: An Open and General Agentic Reasoning System for Formal Mathematics. ICML
[5] Gao et al. 2025 Herald: A Natural Language Annotated Lean 4 Dataset. ICLR
Questions for authors
- What are the full Table 1 experimental settings?
- What are the token budgets and expenditure of Magenta? How does it stack up against pass@1 inference methods in terms of spent budget/tokens/time?
- What is the clear novelty of this work compared to Numina Lean Agent, Hermes, DSP, etc.?
7 comments
Responses to Reviewer ANcM (Part 2)
[W2] No human verification. Since the work operates on auto-formalization territory, I would expect to see some human ablations/verifications of results. However, every single experiment is conducted by prompting more LLMs. Authors also mention that final answers were also verified by another LLMs. I understand that checking every single output is tough and unrealistic, but verifying at least IMO proofs should be included with the work, since it is just six problems. Auto-formalization was also not verified, and instead was left at the mercy of the prompted LLM. It is bare minimum to provide human evaluation in the work of this nature. I would like to see human expert evaluation of the outputs and comparison with either real student answers or released golden solutions. Since the work mentions IMO, I believe the IMO statements were also released, so more comparisons and case studies could be conducted.
We would like to clarify that human verification is part of our evaluation (Appendix F.2 and the AI Use Statement). Below, we separate what is checked by the Lean kernel, by humans and by LLM judges.
-
Correctness is checked by Lean, not by an LLM. Every AIME and HMMT answer Magenta returns comes with a proof accepted by the Lean 4 kernel under SafeVerify, which is deterministic and independent of any language model (shown in Table 1). LLM judges make only two alignment decisions: whether the formal statement matches the problem, and whether a failure is mathematical or syntactic. Neither can make an invalid proof pass. The statement judge is also measured against a non-LLM ground truth: in Table 3, VerCor and the false certification rate compare certified answers with the human reference answers on all 30 AIME 2026 problems. Without the judge, 45.5% of certificates prove a statement whose answer is wrong; with it, 0% do.
-
Human verification of the IMO 2026 proofs. A former IMO medallist independently marked all six solutions, awarding 7/7 on each (42/42). Codex, given the official marking scheme and public reference solutions from the Art of Problem Solving forums and Kimi K3, also awarded 7/7 on each, and a third annotator with a mathematics degree confirmed the Codex marking. All generated solutions are printed in full in Appendix F.4.
-
Human verification of the formal statements. For the proof problems (P1, P2, P6), the formaliser encodes the claim to be proved as the theorem, which the statement judge checks as usual. We also manually checked all six accepted IMO formal statements during the rebuttal, and confirmed that each faithfully represents the original problem.
These clarifications are in Appendix F.2 of the revised manuscript.
Responses to Reviewer ANcM (Part 3)
[W3] Unfair setting/experimental results. Correct me if I am wrong, but the setting of Table 1 seems to be extremely unfair. Authors mention: Baselines. We compare against state-of-the-art baselines: Claude Opus 5, Gemini 3.7 Flash, and Kimi K3. We let each baseline only generate one answer (i.e., pass@1 rate). While Magenta has budgets of: For all the experiments, we use T = 32 reasoner attempts, M = 512 formal-statement samples per candidate answer, and K = 4096 proof attempts per accepted statement. First, the crucial hyperparams were buried in the Appendix, which is not a good look. Second, these settings are just unfair. How can you compare running an LLM once against 32 reasoner attempts with 4096 proof attempts? Moreover, Table 1 omits base model running settings; is it also pass@1? If so, then it raises questions on settings, budgets, tokens, etc. This is just brute forcing to get better results. Could the authors clarify this to me?
[Q1] What are the full Table 1 experimental settings?
Thank you for raising this question. We give the full settings of Table 1, clarify the budgets T, M, and K, and show that the gain is not obtained by brute force.
Full settings of Table 1. All settings are in Appendix B.2, referenced from §3.1; we have moved them, as well as the configurations of budgets T, M, and K, to §3.1 in the revision. Specifically,
- Standalone reasoners (K2-Horizon-7B, K2-Horizon-375B, Qwen3.8-27B, GPT-5.6-Sol): one sample per problem (pass@1) with temperature 0.6, top-p 0.95, a 64k-token output limit, and seed 42. These are the same decoding settings as the reasoner inside Magenta.
- Magenta: the same reasoner settings, with Goedel-Formaliser-32B as formaliser, DeepSeek-V4-Flash as judge, and Leanstral-1.5 as prover.
Budgets. T, M and K are upper bounds, and the procedure stops at the first verified proof. In practice, K2-Horizon-7B uses on average 1, 2 and 3 reasoning rounds on AIME 2025, AIME 2026 and HMMT February 2026 (Table 5). An accepted statement is reached within 6 formaliser calls with Codex and about 120 with Goedel (Figure 3). The proof budget K cannot improve the answer: the prover only proves a fixed statement asserting the reasoner’s committed answer, so a wrong answer stays unprovable however many attempts are made. Only the reasoner can change the answer, and it runs 1 to 3 times on average rather than 4096.
The gain is not brute force. Independent resampling with Lean as a sound selector, i.e. brute force with a perfect filter, verifies only 1/6 IMO 2026 problems despite 8–16 generations per problem, against 6/6 for verification-guided correction, and 29/30 against 30/30 on AIME 2026 (§4.3). More standalone samples do not close the gap either: K2-Horizon-7B attains 86.67% at pass@2 and 93.33% at pass@32 on AIME 2026, both below Magenta, even though pass@k uses the reference answer that Magenta never sees. Since maj@k ≤ pass@k, maj@32 is also at most 93.33%.
Baselines. Magenta reaches 100% with every reasoner evaluated, from 7B to 375B across open-weight and proprietary families, and the 7B configuration outperforms every closed-source baseline in Table 1, including Kimi K3 (2.8T total parameters, approximately 400× the 7B reasoner). We would be glad to include any additional baseline the reviewer believes could achieve comparable verified accuracy from natural-language input.
[Q2] What are the token budgets and expenditure of Magenta? How does it stack up against pass@1 inference methods in terms of spent budget/tokens/time?
The budgets are the caps T = 32, M = 512 and K = 4096 discussed above; the expenditure is measured in §4.1, which reports cumulative verified accuracy against formaliser calls, external verification rounds, reasoner tokens and proof length (Figures 3–5), rather than at a single operating point. In practice, the expenditure is modest. On AIME 2026 with the Leanstral backend, every problem is verified within approximately 6 × 10⁴ cumulative reasoner tokens (Figure 5c), which is within the 64k-token output limit of a single pass@1 generation of the standalone reasoner; 42% of problems are verified with no failed external round (Figure 5b), and the verified proofs are approximately 1.1 × 10⁴ tokens long (Figure 5d).
Formalisation adds at most 6 calls with Codex and approximately 120 with Goedel per problem (Figure 3), each producing a single statement rather than a proof. The additional cost of Magenta relative to pass@1 therefore lies in formalisation and proof search, both of which run on open-weight models served locally, with Leanstral's internal calls (approximately 3 × 10² at median coverage, Figure 4) paid in local compute rather than metered requests.
Thank you again for the feedback. We remain available to address any further concerns/questions
Responses to Reviewer ANcM (Part 1)
We thank reviewer ANcM for the constructive feedback.
Thank you for bringing this to our attention. We respectfully disagree that Magenta is an implementation inside existing harnesses: its contribution is a closed-loop control structure with two adjudication points. We clarify the differences below.
Problem setting. DSP, DSP+ and Numina-Lean-Agent either take a formal statement as input (e.g. miniF2F, PutnamBench) and search for its proof, or heavily rely on closed-source models. They assume away what the natural-language problem asks and whether a proposed answer is correct. Magenta starts from the natural-language problem alone, with no known answer, and must produce the reasoning and answer, a faithful Lean 4 formalisation of the problem and answer, and a machine-checked proof. The autoformalisation gap (§2.3), absent in the cited settings, is the central reliability risk in ours.
Closed-loop error attribution. Prior NL–FL systems use formal verification as a selection or certification signal only (§5). Magenta instead uses failure as information: the error judge attributes each failed verification to the mathematics or the Lean implementation, and routes it to re-derivation or local repair. §4.3 isolates the effect: with the same reasoner and verifier, independent resampling verifies 1/6 IMO 2026 problems, while diagnostic-conditioned correction verifies 6/6.
Semantic alignment. Faithful formalisation has always been a central issue within theorem proving [1,2]. Our statement judge performs the alignment check that back-translation targets, but directly. It compares the Lean statement with the original problem and candidate answer before any proof effort, rejecting omitted hypotheses, altered constants and trivialising assumptions. Without it, 45.5% of returned certificates prove something other than the problem asked; with it, 0% do (Table 3). We would gladly address any concerns within the judge if the reviewer finds it informative.
Training-free design. Using prompted models is a design choice that makes Magenta training-free and model-agnostic: the same pipeline reaches 100% with four reasoners, from a 7B open-weight model to a frontier proprietary one (Table 1). The contribution lies in the loop’s two decisions, and removing either breaks the system. Without statement adjudication, verified-and-correct accuracy falls from 63.3% to 20.0% (Table 3). Without error attribution, 5 of 6 IMO problems are lost (§4.3).
Results. To our knowledge, Magenta is the first pipeline to attain 100% verified accuracy on AIME 2025, AIME 2026, and HMMT February 2026 from natural-language input with a fully open-weight configuration using a 7B reasoner. K2-Horizon-7B is also the smallest reported reasoner to solve all six IMO 2026 problems; it is much smaller than existing models such as Kimi-K3, GPT, Claude, and Gemini. We would welcome pointers to any prior NL–FL pipeline with comparable results and would gladly include a comparison.
We have added the contribution to the Appendix as a new Section G. Thank you again for raising this point.
[1] Leang, Joshua Ong Jun, et al. "Theorem prover as a judge for synthetic data generation." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
[2] Cornish, Rob, et al. "Faithformbench: Benchmarking faithfulness of mathematical chain-of-thought autoformalisation." arXiv preprint arXiv:2608.10916 (2026).