acceptodds
Under review as a conference paper at ICLR 2027

Magenta: Closing the Loop Between Mathematical Reasoning and Formal Verification

20 views

Abstract

Most of mathematical knowledge has been communicated through so-called informal use of mathematics and natural language. With large language models (LLMs) being highly adept in using natural language, they achieve strong performance, yet not perfect, in informal mathematical reasoning. Restraining LLMs to informal reasoning misses out on the opportunity to use the discrete verification abilities that machines offer through machine-checkable proofs. In this paper, we bridge the gap between informal and formal reasoning by integrating Lean signals into the informal reasoning process. We introduce Magenta , a training-free agentic pipeline that, given only a natural-language problem, produces an answer, expresses it as a Lean 4 statement, and constructs a machine-checked proof. A statement judge verifies whether the formalisation preserves the original problem, while an error-attribution judge routes failed attempts either to mathematical re-derivation or local Lean repair. Magenta achieves 100% accuracy across all evaluated olympiad benchmarks, including AIME 2025, AIME 2026, and HMMT February 2026. When paired with the open-weight K2-Horizon-7B reasoner, it solves all six IMO 2026 problems. Our analysis shows that statement adjudication is essential for preventing false certificates and that feedback-guided correction outperforms independent resampling on difficult problems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Theorem Proving, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ANcMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Magenta, an agentic loop for math reasoning that combines natural language (NL) with formal language (FL) to achieve the best performance with verifiable compiler signals.

Critical strengths

Strengths

  • Straightforward idea. The paper presents a sensible, straightforward idea that couples informal inference with Lean verification.

  • Implementation transparency. The authors included sample proofs along with all the templates of Magenta.

  • Open-source experiment. I commend authors for using open-weight models in the experiments for better transparency.

Critical weaknesses

Weaknesses

  • Novelty. After reading the manuscript twice, I am unsure of the novelty of this work. Described methods are just prompted LLMs one after another with mild decision making conducted by an LLM. This seems more like an implementation project rather than a genuine novelty to the theorem proving community. NL-FL integration was explored in numerous works, such as DSP [1], DSP+ [2], Hermes [3], and Numina Lean Agent [4]; and I do not see substantial novelty over these works. Moreover, I would argue that this agent lacks relatively standard features explored in early auto formalization works such as backtranslation [5] for semantic alignment. Right now, the work presents a neat implementation inside existing harnesses for theorem proving. I cannot recommend this work for publishing, since it does not add substantial new knowledge to the community.

  • No human verification. Since the work operates on auto-formalization territory, I would expect to see some human ablations/verifications of results. However, every single experiment is conducted by prompting more LLMs. Authors also mention that final answers were also verified by another LLMs. I understand that checking every single output is tough and unrealistic, but verifying at least IMO proofs should be included with the work, since it is just six problems. Auto-formalization was also not verified, and instead was left at the mercy of the prompted LLM. It is bare minimum to provide human evaluation in the work of this nature. I would like to see human expert evaluation of the outputs and comparison with either real student answers or released golden solutions. Since the work mentions IMO, I believe the IMO statements were also released, so more comparisons and case studies could be conducted.

  • Unfair setting/experimental results. Correct me if I am wrong, but the setting of Table 1 seems to be extremely unfair. Authors mention:

Baselines. We compare against state-of-the-art baselines: Claude Opus 5, Gemini 3.7 Flash, and Kimi K3. We let each baseline only generate one answer (i.e., pass@1 rate).

While Magenta has budgets of:

For all the experiments, we use T = 32 reasoner attempts, M = 512 formal-statement samples per candidate answer, and K = 4096 proof attempts per accepted statement.

First, the crucial hyperparams were buried in the Appendix, which is not a good look. Second, these settings are just unfair. How can you compare running an LLM once against 32 reasoner attempts with 4096 proof attempts? Moreover, Table 1 omits base model running settings; is it also pass@1? If so, then it raises questions on settings, budgets, tokens, etc. This is just brute forcing to get better results. Could the authors clarify this to me?


References

[1] Jiang et al. 2023. Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs. ICLR

[2] Cao et al. 2025. Reviving DSP for Advanced Theorem Proving in the Era of Reasoning Models. NeurIPS

[3] Ospanov et al. 2026. HERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMs. ICML

[4] Liu et al. 2026. Numina-Lean-Agent: An Open and General Agentic Reasoning System for Formal Mathematics. ICML

[5] Gao et al. 2025 Herald: A Natural Language Annotated Lean 4 Dataset. ICLR

Questions for authors
  1. What are the full Table 1 experimental settings?
  2. What are the token budgets and expenditure of Magenta? How does it stack up against pass@1 inference methods in terms of spent budget/tokens/time?
  3. What is the clear novelty of this work compared to Numina Lean Agent, Hermes, DSP, etc.?
7 comments
AuthorsOpenReview ↗

Responses to Reviewer ANcM (Part 1)

We thank reviewer ANcM for the constructive feedback.

[W1] Novelty. After reading the manuscript twice, I am unsure of the novelty of this work. Described methods are just prompted LLMs one after another with mild decision making conducted by an LLM. This seems more like an implementation project rather than a genuine novelty to the theorem proving community. NL-FL integration was explored in numerous works, such as DSP [1], DSP+ [2], Hermes [3], and Numina Lean Agent [4]; and I do not see substantial novelty over these works. Moreover, I would argue that this agent lacks relatively standard features explored in early auto formalization works such as backtranslation [5] for semantic alignment. Right now, the work presents a neat implementation inside existing harnesses for theorem proving. I cannot recommend this work for publishing, since it does not add substantial new knowledge to the community.

[Q3] What is the clear novelty of this work compared to Numina Lean Agent, Hermes, DSP, etc.?

Thank you for bringing this to our attention. We respectfully disagree that Magenta is an implementation inside existing harnesses: its contribution is a closed-loop control structure with two adjudication points. We clarify the differences below.

  1. Problem setting. DSP, DSP+ and Numina-Lean-Agent either take a formal statement as input (e.g. miniF2F, PutnamBench) and search for its proof, or heavily rely on closed-source models. They assume away what the natural-language problem asks and whether a proposed answer is correct. Magenta starts from the natural-language problem alone, with no known answer, and must produce the reasoning and answer, a faithful Lean 4 formalisation of the problem and answer, and a machine-checked proof. The autoformalisation gap (§2.3), absent in the cited settings, is the central reliability risk in ours.

  2. Closed-loop error attribution. Prior NL–FL systems use formal verification as a selection or certification signal only (§5). Magenta instead uses failure as information: the error judge attributes each failed verification to the mathematics or the Lean implementation, and routes it to re-derivation or local repair. §4.3 isolates the effect: with the same reasoner and verifier, independent resampling verifies 1/6 IMO 2026 problems, while diagnostic-conditioned correction verifies 6/6.

  3. Semantic alignment. Faithful formalisation has always been a central issue within theorem proving [1,2]. Our statement judge performs the alignment check that back-translation targets, but directly. It compares the Lean statement with the original problem and candidate answer before any proof effort, rejecting omitted hypotheses, altered constants and trivialising assumptions. Without it, 45.5% of returned certificates prove something other than the problem asked; with it, 0% do (Table 3). We would gladly address any concerns within the judge if the reviewer finds it informative.

  4. Training-free design. Using prompted models is a design choice that makes Magenta training-free and model-agnostic: the same pipeline reaches 100% with four reasoners, from a 7B open-weight model to a frontier proprietary one (Table 1). The contribution lies in the loop’s two decisions, and removing either breaks the system. Without statement adjudication, verified-and-correct accuracy falls from 63.3% to 20.0% (Table 3). Without error attribution, 5 of 6 IMO problems are lost (§4.3).

  5. Results. To our knowledge, Magenta is the first pipeline to attain 100% verified accuracy on AIME 2025, AIME 2026, and HMMT February 2026 from natural-language input with a fully open-weight configuration using a 7B reasoner. K2-Horizon-7B is also the smallest reported reasoner to solve all six IMO 2026 problems; it is much smaller than existing models such as Kimi-K3, GPT, Claude, and Gemini. We would welcome pointers to any prior NL–FL pipeline with comparable results and would gladly include a comparison.

We have added the contribution to the Appendix as a new Section G. Thank you again for raising this point.

[1] Leang, Joshua Ong Jun, et al. "Theorem prover as a judge for synthetic data generation." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

[2] Cornish, Rob, et al. "Faithformbench: Benchmarking faithfulness of mathematical chain-of-thought autoformalisation." arXiv preprint arXiv:2608.10916 (2026).

AuthorsOpenReview ↗

Responses to Reviewer ANcM (Part 2)

[W2] No human verification. Since the work operates on auto-formalization territory, I would expect to see some human ablations/verifications of results. However, every single experiment is conducted by prompting more LLMs. Authors also mention that final answers were also verified by another LLMs. I understand that checking every single output is tough and unrealistic, but verifying at least IMO proofs should be included with the work, since it is just six problems. Auto-formalization was also not verified, and instead was left at the mercy of the prompted LLM. It is bare minimum to provide human evaluation in the work of this nature. I would like to see human expert evaluation of the outputs and comparison with either real student answers or released golden solutions. Since the work mentions IMO, I believe the IMO statements were also released, so more comparisons and case studies could be conducted.

We would like to clarify that human verification is part of our evaluation (Appendix F.2 and the AI Use Statement). Below, we separate what is checked by the Lean kernel, by humans and by LLM judges.

  1. Correctness is checked by Lean, not by an LLM. Every AIME and HMMT answer Magenta returns comes with a proof accepted by the Lean 4 kernel under SafeVerify, which is deterministic and independent of any language model (shown in Table 1). LLM judges make only two alignment decisions: whether the formal statement matches the problem, and whether a failure is mathematical or syntactic. Neither can make an invalid proof pass. The statement judge is also measured against a non-LLM ground truth: in Table 3, VerCor and the false certification rate compare certified answers with the human reference answers on all 30 AIME 2026 problems. Without the judge, 45.5% of certificates prove a statement whose answer is wrong; with it, 0% do.

  2. Human verification of the IMO 2026 proofs. A former IMO medallist independently marked all six solutions, awarding 7/7 on each (42/42). Codex, given the official marking scheme and public reference solutions from the Art of Problem Solving forums and Kimi K3, also awarded 7/7 on each, and a third annotator with a mathematics degree confirmed the Codex marking. All generated solutions are printed in full in Appendix F.4.

  3. Human verification of the formal statements. For the proof problems (P1, P2, P6), the formaliser encodes the claim to be proved as the theorem, which the statement judge checks as usual. We also manually checked all six accepted IMO formal statements during the rebuttal, and confirmed that each faithfully represents the original problem.

These clarifications are in Appendix F.2 of the revised manuscript.

AuthorsOpenReview ↗

Responses to Reviewer ANcM (Part 3)

[W3] Unfair setting/experimental results. Correct me if I am wrong, but the setting of Table 1 seems to be extremely unfair. Authors mention: Baselines. We compare against state-of-the-art baselines: Claude Opus 5, Gemini 3.7 Flash, and Kimi K3. We let each baseline only generate one answer (i.e., pass@1 rate). While Magenta has budgets of: For all the experiments, we use T = 32 reasoner attempts, M = 512 formal-statement samples per candidate answer, and K = 4096 proof attempts per accepted statement. First, the crucial hyperparams were buried in the Appendix, which is not a good look. Second, these settings are just unfair. How can you compare running an LLM once against 32 reasoner attempts with 4096 proof attempts? Moreover, Table 1 omits base model running settings; is it also pass@1? If so, then it raises questions on settings, budgets, tokens, etc. This is just brute forcing to get better results. Could the authors clarify this to me?

[Q1] What are the full Table 1 experimental settings?

Thank you for raising this question. We give the full settings of Table 1, clarify the budgets T, M, and K, and show that the gain is not obtained by brute force.

Full settings of Table 1. All settings are in Appendix B.2, referenced from §3.1; we have moved them, as well as the configurations of budgets T, M, and K, to §3.1 in the revision. Specifically,

  • Standalone reasoners (K2-Horizon-7B, K2-Horizon-375B, Qwen3.8-27B, GPT-5.6-Sol): one sample per problem (pass@1) with temperature 0.6, top-p 0.95, a 64k-token output limit, and seed 42. These are the same decoding settings as the reasoner inside Magenta.
  • Magenta: the same reasoner settings, with Goedel-Formaliser-32B as formaliser, DeepSeek-V4-Flash as judge, and Leanstral-1.5 as prover.

Budgets. T, M and K are upper bounds, and the procedure stops at the first verified proof. In practice, K2-Horizon-7B uses on average 1, 2 and 3 reasoning rounds on AIME 2025, AIME 2026 and HMMT February 2026 (Table 5). An accepted statement is reached within 6 formaliser calls with Codex and about 120 with Goedel (Figure 3). The proof budget K cannot improve the answer: the prover only proves a fixed statement asserting the reasoner’s committed answer, so a wrong answer stays unprovable however many attempts are made. Only the reasoner can change the answer, and it runs 1 to 3 times on average rather than 4096.

The gain is not brute force. Independent resampling with Lean as a sound selector, i.e. brute force with a perfect filter, verifies only 1/6 IMO 2026 problems despite 8–16 generations per problem, against 6/6 for verification-guided correction, and 29/30 against 30/30 on AIME 2026 (§4.3). More standalone samples do not close the gap either: K2-Horizon-7B attains 86.67% at pass@2 and 93.33% at pass@32 on AIME 2026, both below Magenta, even though pass@k uses the reference answer that Magenta never sees. Since maj@k ≤ pass@k, maj@32 is also at most 93.33%.

Baselines. Magenta reaches 100% with every reasoner evaluated, from 7B to 375B across open-weight and proprietary families, and the 7B configuration outperforms every closed-source baseline in Table 1, including Kimi K3 (2.8T total parameters, approximately 400× the 7B reasoner). We would be glad to include any additional baseline the reviewer believes could achieve comparable verified accuracy from natural-language input.

[Q2] What are the token budgets and expenditure of Magenta? How does it stack up against pass@1 inference methods in terms of spent budget/tokens/time?

The budgets are the caps T = 32, M = 512 and K = 4096 discussed above; the expenditure is measured in §4.1, which reports cumulative verified accuracy against formaliser calls, external verification rounds, reasoner tokens and proof length (Figures 3–5), rather than at a single operating point. In practice, the expenditure is modest. On AIME 2026 with the Leanstral backend, every problem is verified within approximately 6 × 10⁴ cumulative reasoner tokens (Figure 5c), which is within the 64k-token output limit of a single pass@1 generation of the standalone reasoner; 42% of problems are verified with no failed external round (Figure 5b), and the verified proofs are approximately 1.1 × 10⁴ tokens long (Figure 5d).

Formalisation adds at most 6 calls with Codex and approximately 120 with Goedel per problem (Figure 3), each producing a single statement rather than a proof. The additional cost of Magenta relative to pass@1 therefore lies in formalisation and proof search, both of which run on open-weight models served locally, with Leanstral's internal calls (approximately 3 × 10² at median coverage, Figure 4) paid in local compute rather than metered requests.

Thank you again for the feedback. We remain available to address any further concerns/questions

Reviewer_HiUSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MAGENTA, a model-agnostic and training-free agentic pipeline that takes a competition problem in natural language format, and uses an informal reasoner that produces a solution and candidate answer, autoformalizes a Lean 4 statement asserting that answer, and then uses an LLM judge to check that the statement correctly encodes the problem and the proposed answer. Then, a Lean prover searches for a proof, which is checked under a SafeVerify policy. If verification fails, an LLM error judge attributes the error to either the Lean implementation or the informal solution. Across four configurations, the pipeline reaches 100% verified accuracy on AIME 2025, AIME 2026 and HMMT Feb 2026 for all four reasoning LLMs tested, and K2-Horizon-7B is reported to solve all six IMO 2026 problems. The paper also reports ablations on statement adjudication and the self-correction piece of the pipeline.

Critical strengths
  1. The problem framing is precise and the statement judge turns the autoformalization gap into a measurable quantity.

  2. The pipeline is fully specified, and open-weight configurations are provided, which makes the system reproducible in principle.

  3. The IMO solutions are printed in Appendix F.4, which allows independent checking.

Critical weaknesses
  1. The baseline of pass@1 for an LLM does not match MAGENTA, which draws up to T=32 iterations. The comparison is not compute-matched and thus unfair.

  2. The reasoners used in Table 1 were released after the competition problems were publicly available, which makes contamination possible.

  3. The IMO 2026 claim lacks sufficient evidence. Problems P1, P2 and P6 are pure proof problems, and the paper never explains how the answer-asserting formalization applies to them or what statement was verified. There are no scores from either the Codex grader or the human medallist. Additionally, no Lean statement or proof for any IMO problem is shown. Although an extra judge is added for the IMO claim, no prompt or judge feedback is shown.

Questions for authors
  1. The Reproducibility Statement refers to a "script" yet no code, supplementary material, or link is included in the submission. Could you provide, via an anonymized repository, the pipeline code with a minimal runnable configuration for the 7B experiments in Table 1, in case one attempts to reproduce parts of the empirical results? Additionally, Lean statements and proofs for all benchmark and IMO problems with SafeVerify outputs will further solidify the utility of the formalization half of the MAGENTA pipeline.

  2. Can you report baselines that match the iterative improvement that MAGENTA does, e.g. pass@k or maj@k for k=1,...,32, or giving the LLM T rounds of iteratively refining its solution?

  3. Can you provide further information on possible training data contamination for models where the training data is known? Additionally, can you report additional results on a problem set published after 3 September 2026 (K2-Horizon's release date)? One such example is MathArena's ArXivMath-0926, or potentially HMMT November 2026. Again, the bare reasoner baseline should be reported with pass@k/maj@k.

  4. Can you provide additional information on the K2-Horizon-7B evaluation on IMO 2026 problems? In particular, what are the model's scores per problem, as well as feedback from the Codex grader and the medallist? Additionally, can you provide the formalized problem statements?

5 comments
AuthorsOpenReview ↗

Responses to Reviewer HiUS (Part 1)

We sincerely thank Reviewer HiUS for noting that Magenta’s framing is precise and that the statement judge turns the autoformalisation gap into a measurable quantity, for recognising the transparency of its pipeline and its 100% verified accuracy across various models and mathematical benchmarks.

[W1] The baseline of pass@1 for an LLM does not match Magenta, which draws up to T=32 iterations. The comparison is not compute-matched and thus unfair.

Thank you for bringing this to our attention. Regarding the Iterations actually drawn, the budgets (T=32), (M=512), and (K=4096) are upper bounds, not actual usage, as Magenta terminates upon finding the first verified proof. The observed expenditure is substantially lower:

  • Reasoner: K2-Horizon-7B uses on average 1, 2 and 3 reasoning rounds on AIME 2025, AIME 2026 and HMMT February 2026, and K2-Horizon-375B a single round (Table 5).
  • Formaliser: Accepted statements require at most 6 Codex calls or approximately 120 Goedel calls in the worst case (Figure 3), well below (M=512). Each call generates only a theorem statement (~1,024 tokens), rather than a proof, limiting formalisation overhead.
  • Prover: With Leanstral, 42% of AIME 2026 problems are verified without any failed external proof round (Figure 5b).

Further token-usage statistics are provided in Figures 5 and 6. We have clarified the computational budgets and evaluation protocol in Section 3.1 and Appendix C.

[W2] The reasoners used in Table 1 were released after the competition problems were publicly available, which makes contamination possible.

[Q3] Can you provide further information on possible training data contamination for models where the training data is known? Additionally, can you report additional results on a problem set published after 3 September 2026 (K2-Horizon's release date)? One such example is MathArena's ArXivMath-0926, or potentially HMMT November 2026. Again, the bare reasoner baseline should be reported with pass@k/maj@k.

Thank you for raising this concern. While we acknowledge that paraphrasing alone cannot exclude memorisation and that the model-specific training cutoffs for AIME 2025/2026 and HMMT February 2026 have not been established, we provide additional evidence against memorisation through verification-guided reasoning and two new experiments.

  • Arguing against memorisation. First, memorisation can supply a candidate answer but not a certificate reasoning [1]: every Magenta reasoning and answer must be proved in Lean, so misremembered or incomplete answers yield an unprovable statement, not a false result. Second, many problems are verified only after verification-guided self-correction (Figure 2, Table 5). For example, the AIME 2026 trajectory in Appendix D starts from an incorrect answer (1) that is corrected to 396 only after Lean feedback.
  • Post-release competition. In addition to evidence against memorisation, we evaluated K2-Horizon-7B on HMMT November 2026, held on 7 November 2026, more than two months after the model’s release and hence after its training cutoff. Magenta again attains 100%. This demonstrates the strength of Magenta rather than memorisation and data contamination.
  • Numerical perturbation with changed answers. Following GSM-Symbolic [2], we perturbed the numerical quantities of 15 AIME 2026 problems so that the reference answer changes, selecting all problems that remained well-posed and checking each manually. Magenta with K2-Horizon-7B achieves 100% accuracy, whereas the standalone reasoner attains 73.33%. Together, these results indicate that Magenta’s performance is not attributable to contamination. We have added this to Appendix E.3 of the revised manuscript. Thank you again for raising this concern.

[1] Li, Huihan, et al. "Diagnosing memorization in chain-of-thought reasoning, one token at a time." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

[2] Mirzadeh, Iman, et al. "GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models." ICLR 2025.

AuthorsOpenReview ↗

Responses to Reviewer HiUS (Part 2)

[W3] The IMO 2026 claim lacks sufficient evidence. Problems P1, P2 and P6 are pure proof problems, and the paper never explains how the answer-asserting formalization applies to them or what statement was verified. There are no scores from either the Codex grader or the human medallist. Additionally, no Lean statement or proof for any IMO problem is shown. Although an extra judge is added for the IMO claim, no prompt or judge feedback is shown.

[Q4] Can you provide additional information on the K2-Horizon-7B evaluation on IMO 2026 problems? In particular, what are the model's scores per problem, as well as feedback from the Codex grader and the medallist? Additionally, can you provide the formalized problem statements?

We address this concern through (1) proof problems, (2) the completeness judge, (3) the per-problem scores and (4) grading protocol in turn.

Proof problems (P1, P2, P6). For answer problems, the formaliser encodes the problem together with the reasoner’s candidate answer. For proof problems, there is no answer to assert, so the formaliser encodes the claim to be proved as the theorem, and the statement judge checks it against the original problem as usual. The Lean certificate for P1, P2 and P6 is therefore a proof of the formalised claim, with the reasoner’s informal solution as the proof plan (§2.1). It is the same procedure with the answer slot empty.

Completeness judge. The final IMO judge (Appendix F.1) runs only after a Lean certificate is obtained. It compares the informal solution with the formal statement and proof, and returns incomplete proof if the certificate omits or misaligns a substantive part of the argument. It is an extra acceptance filter and does not change Lean verification or feedback routing (Appendix A.2).

Per-problem scores. Both Codex and the former IMO gold medalist independently awarded 7/7 on each of P1–P6, achieving 42/42 for both markers (100% agreement). This shows that there is no problem on which the scores differ, and neither marker assigned partial credit on any of the problems.

Protocol. The assessments were conducted independently under different conditions:

  • Human marking. The medallist marked the six solutions without access to the Codex scores. **Given the scarcity of such expertise, the medallist provided scores rather than written feedback.

  • Automated marking. Codex scored each solution against the scheme using public reference solutions, including those from the Art of Problem Solving forums and Kimi K3 (Appendix F.2).

  • Additional check. A third annotator with a mathematics degree reviewed and confirmed the Codex gradings, as a check on the automated grader rather than an independent marker.

  • Formal statements. During the rebuttal, we further manually checked all six accepted IMO formal statements and confirmed that each faithfully represents the original problem.

Given the scarcity of IMO gold medallist-level expertise, we consider the complete agreement between the independent human and automated assessments, supported by additional mathematical verification, to provide strong evidence for the reliability of our grading protocol. These clarifications have been incorporated into Appendix F.2.

[Q1] The Reproducibility Statement refers to a "script" yet no code, supplementary material, or link is included in the submission. Could you provide, via an anonymized repository, the pipeline code with a minimal runnable configuration for the 7B experiments in Table 1, in case one attempts to reproduce parts of the empirical results? Additionally, Lean statements and proofs for all benchmark and IMO problems with SafeVerify outputs will further solidify the utility of the formalization half of the Magenta pipeline.

Thank you for bringing this to our attention. To clarify, by “script” we were referring to the submitted manuscript itself, which describes the methodology in full and lists all hyperparameters, providing all the information needed to replicate our results. We will release the complete code upon acceptance.

AuthorsOpenReview ↗

Responses to Reviewer HiUS (Part 3)

[Q2] Can you report baselines that match the iterative improvement that Magenta does, e.g. pass@k or maj@k for k=1,...,32, or giving the LLM T rounds of iteratively refining its solution?

Thank you again for bringing this to our attention. We address this concern through three points: (1) pass@k and maj@k, (2) iterative refinement, and (3) robustness to the choice of baseline.

  • pass@k and maj@k. pass@k is a more generous baseline than Magenta, since it uses the reference answer to pick a correct sample, while Magenta never attempts to access the ground truth. Even so, K2-Horizon-7B reaches only 86.67% at pass@2 and 93.33% at pass@32 on AIME 2026, below Magenta’s 100%. Since maj@k ≤ pass@k, maj@32 is also at most 93.33%. Magenta solves the two problems left unsolved at pass@32 through verification-guided correction.

  • Iterative refinement. The closest controlled comparison is the ablation in §4.3, which keeps Lean as a sound selector but removes the feedback: resampling verifies 29/30 AIME 2026 and 1/6 IMO 2026 problems, against 30/30 and 6/6 for Magenta. Self-refinement without a verifier lacks both the feedback and the selector, leaving the model with no reliable signal that its answer is wrong. This suggests the gain comes from the verification signal rather than the number of rounds.

  • Robustness to the choice of baseline. Magenta achieves 100% with every reasoner evaluated, across model families and scales from 7B to 375B, and the 7B configuration outperforms every closed-source baseline in Table 1, including Kimi K3 (2.8T total and 104B active parameters). Hence, the conclusion does not depend on which baseline is chosen.

These findings highlight the contribution of verification-guided correction beyond increased sampling. We have clarified the computational budgets and evaluation protocol in Section 3.1 and Appendix C.

We hope our responses address your concerns. If you have any further questions or suggestions, please don't hesitate to let us know!

Reviewer_ATYyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MAGENTA, a training-free pipeline that couples natural-language mathematical reasoning with Lean 4 formal verification. Given a competition problem, a reasoner produces a chain and a candidate answer; a formaliser turns the problem and answer into a Lean 4 theorem statement; a statement judge checks that the statement faithfully represents the problem and the proposed answer; a prover writes a Lean proof, with the informal chain used as a proof plan; and the Lean kernel plus SafeVerify checks the proof. Crucially, when verification fails, an error judge attributes the failure to either mathematics or code, and routes the attempt accordingly: a MATH label regenerates the reasoning and answer, while a SYNTAX label repairs only the proof at fixed statement.

The main results are that MAGENTA reaches 100 percent on AIME 2025, AIME 2026, and HMMT February 2026 with four different reasoners, from a 7B open model to proprietary systems, and that the 7B reasoner solves all six IMO 2026 problems. The paper also reports robustness to answer-preserving paraphrase, ablations of the statement judge and of diagnostic-conditioned correction, and an analysis of how test-time compute is distributed across formalisation and proof search.

Critical strengths
  1. The problem is well posed. The paper clearly identifies the autoformalisation gap, namely that Lean certifies a proof relative to the generated statement and not the original problem, and it treats this as the central limitation rather than hiding it.
  2. The error-attribution mechanism is well presented. Distinguishing MATH from SYNTAX failures, and routing to different components, is a principled way to close the loop, and the ablation supports it: resampling verifies 1 of 6 IMO problems while diagnostic-conditioned correction verifies all six.
  3. The statement judge ablation is clean and quantitative. Without the judge, verified certificates drop from 63.3 to 36.7 percent, and the false certification rate reaches 45.5 percent. This is a direct measurement of the residual gap and is exactly the right experiment.
  4. The paraphrase robustness test is a thoughtful control for contamination. Standalone reasoners lose 3.3 and 10.0 points under paraphrase, while MAGENTA retains 100 percent. The authors correctly note that this is necessary but not sufficient evidence.
  5. The scaling analysis is unusually careful. Reporting cumulative pass rate against budget rather than at a single operating point, and distinguishing external from internal cost, is the right way to compare an agentic prover against a CLI prover, and the paper is explicit that the ordering reverses once internal calls are counted.
Critical weaknesses
  1. The benchmarks are small and possibly saturated. AIME 2025, AIME 2026, and HMMT February 2026 together contain 93 problems, and several baselines already score near 100 percent on AIME. A 100 percent result on 30 problems has wide confidence intervals, and the paper reports no variance or repeated runs. The IMO 2026 result is striking but rests on six problems.
  2. The IMO judging protocol relies on Codex as an automated grader, with a single human expert marking independently. The agreement between the two is not reported. Given that the headline claim is a 7B model solving all of IMO 2026, the grading protocol deserves more detail.
  3. The contamination concern is only partially addressed. Paraphrasing preserves the answer, and the authors acknowledge that perturbing the numerical quantities would be a stronger test. Since the reasoners are evaluated on 2025 and 2026 competitions that may be near or after training cutoffs, and since the paper reports no cutoff analysis, the possibility of memorization remains open.
  4. The budgets are very large and are not reported as a headline cost. T is 32 reasoner attempts, M is 512 statement samples, and K is 4096 proof attempts. The paper reports token usage for IMO in Figure 6, with problem 3 consuming 858K tokens, but does not report the aggregate cost per benchmark or compare it to the baselines, which are run at pass@1. This makes the accuracy comparison hard to interpret as a capability statement rather than a compute statement.
Questions for authors
  1. What is the agreement between the Codex grader and the human IMO medallist on the six IMO 2026 solutions? Please report per-problem scores from both.
  2. Have you checked training cutoffs for the reasoners relative to AIME 2025, AIME 2026, and HMMT February 2026? Can you run the stronger perturbation test in which the numerical quantities are changed so the reference answer changes?
  3. Can you report the aggregate token and dollar cost per benchmark for MAGENTA and for each baseline? The baselines are run at pass@1 while MAGENTA uses large T, M, K budgets, and the accuracy comparison is difficult to interpret without this.
  4. How sensitive are the results to the error judge's MATH versus SYNTAX decision? The prompt instructs the judge to return code_error when evidence is ambiguous, which biases toward proof repair. What happens if this bias is reversed?
4 comments
AuthorsOpenReview ↗

Responses to Reviewer ATYy (Part 1)

We sincerely thank Reviewer ATYy for recognising our well-posed problem formulation and principled error-attribution mechanism, and for acknowledging that MAGENTA achieves 100% on various benchmarks and models with thoughtful control for contamination. We would like to offer clarifications and additional results that we hope will address the reviewer’s concerns.

[W1] The benchmarks are small and possibly saturated. AIME 2025, AIME 2026, and HMMT February 2026 together contain 93 problems, and several baselines already score near 100 percent on AIME. A 100 percent result on 30 problems has wide confidence intervals, and the paper reports no variance or repeated runs. The IMO 2026 result is striking but rests on six problems.

Thank you for raising this concern. While we agree that AIME may be near-saturated for frontier models, we further address this in two ways:

(1) What the benchmark shows. Our claim is that a carefully curated pipeline allows each model to reach reliable performance, as shown across all the baselines evaluated: the gain over the same reasoner without the pipeline is significant under an exact paired McNemar test for every reasoner (p ≈ 1.2×10⁻⁷ for K2-Horizon-7B, 1.2×10⁻⁴ for K2-Horizon-375B and GPT-5.6-Sol, and 0.008 for Qwen3.8-27B). Likewise, we present the IMO 2026 results not as a statistical claim but as a qualitative demonstration that the pipeline extends to multi-step olympiad proofs. We have revised §1 and §3.2 accordingly.

(2) Repeated runs and held-out problems. We added three seeds for AIME 2025 and AIME 2026, and MAGENTA reaches 100% on every seed. We also evaluated K2-Horizon-7B with Magenta on HMMT November 2026, held on 7 November 2026 after the model’s release, and again obtained 100%. Together with the answer-changing numeric perturbations of 15 AIME 2026 problems (§4.2), this shows the result holds on problems the reasoner cannot have seen.

[W2] The IMO judging protocol relies on Codex as an automated grader, with a single human expert marking independently. The agreement between the two is not reported. Given that the headline claim is a 7B model solving all of IMO 2026, the grading protocol deserves more detail.

[Q1] What is the agreement between the Codex grader and the human IMO medallist on the six IMO 2026 solutions? Please report per-problem scores from both.

Thank you for bringing this to our attention. We now report the per-problem scores and clarify the grading protocol in Appendix F.2 of the manuscript.

Per-problem scores and agreement. Both Codex and the former IMO gold medalist independently awarded 7/7 on each of P1–P6, achieving 42/42 for both markers (100% agreement). This shows that there is no problem on which the scores differ, and neither marker assigned partial credit on any of the problems.

Protocol. The assessments were conducted independently under different conditions: Human marking. The medallist marked the six solutions without access to the Codex scores. Given the scarcity of such expertise, the medallist provided scores rather than written feedback.

  • Automated marking. Codex scored each solution against the scheme using public reference solutions, including those from the Art of Problem Solving forums and Kimi K3 (Appendix F.2).
  • Additional check. A third annotator with a mathematics degree reviewed and confirmed the Codex gradings, as a check on the automated grader rather than an independent marker.
  • Formal statements. During the rebuttal, we further manually checked all six accepted IMO formal statements and confirmed that each faithfully represents the original problem.

Given the scarcity of IMO gold medallist-level expertise, we consider the complete agreement between the independent human and automated assessments, supported by additional mathematical verification, to provide strong evidence for the reliability of our grading protocol. These clarifications have been incorporated into Appendix F.2.

AuthorsOpenReview ↗

Responses to Reviewer ATYy (Part 2)

[W3] The contamination concern is only partially addressed. Paraphrasing preserves the answer, and the authors acknowledge that perturbing the numerical quantities would be a stronger test. Since the reasoners are evaluated on 2025 and 2026 competitions that may be near or after training cutoffs, and since the paper reports no cutoff analysis, the possibility of memorization remains open.

[Q2] Have you checked training cutoffs for the reasoners relative to AIME 2025, AIME 2026, and HMMT February 2026? Can you run the stronger perturbation test in which the numerical quantities are changed so the reference answer changes?

Thank you for this insightful question! We acknowledge that paraphrasing alone cannot exclude memorisation. While model-specific training cutoffs for AIME 2025/2026 and HMMT February 2026 have not been established, we provide additional evidence against memorisation through verification-guided reasoning and two new experiments.

  • Arguing against memorisation. First, memorisation can supply a candidate answer but not a certificate reasoning [1]: every Magenta reasoning and answer must be proved in Lean, so misremembered or incomplete answers or reasoning yield an unprovable statement. Second, many problems are verified only after verification-guided self-correction (Figure 2, Table 5). For example, the AIME 2026 trajectory in Appendix D starts from an incorrect answer (1) that is corrected to 396 only after Lean feedback.
  • Post-release competition. In addition to evidence against memorisation, we evaluated K2-Horizon-7B on HMMT November 2026, held on 7 November 2026, more than two months after the model’s release and hence after its training cutoff. Magenta again attains 100%. This demonstrates the strength of Magenta rather than memorisation and data contamination.
  • Numerical perturbation with changed answers. Following GSM-Symbolic [2], we perturbed the numerical quantities of 15 AIME 2026 problems so that the reference answer changes, selecting all problems that remained well-posed and checking each manually. Magenta with K2-Horizon-7B achieves 100% accuracy, whereas the standalone reasoner attains 73.33%.

Together, these results indicate that Magenta’s performance is not attributable to contamination. We have added this to Appendix E.3 of the revised manuscript.

[1] Li, Huihan, et al. "Diagnosing memorization in chain-of-thought reasoning, one token at a time.” EMNLP 2025.

[2] Mirzadeh, Iman, et al. "GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models." ICLR 2025.

[W4] The budgets are very large and are not reported as a headline cost. T is 32 reasoner attempts, M is 512 statement samples, and K is 4096 proof attempts. The paper reports token usage for IMO in Figure 6, with problem 3 consuming 858K tokens, but does not report the aggregate cost per benchmark or compare it to the baselines, which are run at pass@1. This makes the accuracy comparison hard to interpret as a capability statement rather than a compute statement.

[Q3.1] Can you report the aggregate token and dollar cost per benchmark for Magenta and for each baseline? The baselines are run at pass@1 while MAGENTA uses large T, M, K budgets, and the accuracy comparison is difficult to interpret without this.

Thank you for raising this point. We clarify the computational cost, distinguish MAGENTA from pass@1 baselines, and provide evidence that its gains arise from verification-guided correction rather than additional sampling alone.

Compute actually spent. The budgets (T=32), (M=512), and (K=4096) are upper bounds, not actual usage, as Magenta terminates upon finding the first verified proof. The observed expenditure is substantially lower:

  • Reasoner: K2-Horizon-7B requires only 1, 2, and 3 reasoning rounds on average for AIME 2025, AIME 2026, and HMMT February 2026, respectively; K2-Horizon-375B requires 1–2 rounds across all three benchmarks (Table 5).
  • Formaliser: Accepted statements require at most 6 Codex calls or approximately 120 Goedel calls in the worst case (Figure 3), well below (M=512). Each call generates only a theorem statement (~1,024 tokens), rather than a proof, limiting formalisation overhead.
  • Prover: With Leanstral, 42% of AIME 2026 problems are verified without any failed external proof round (Figure 5b).

Further token-usage statistics are provided in Figures 5 and 6.

AuthorsOpenReview ↗

Responses to Reviewer ATYy (Part 3)

[Q3.2] The baselines are run at pass@1 while MAGENTA uses large T, M, K budgets, and the accuracy comparison is difficult to interpret without this.

Comparison with pass@n. Unlike pass@n, which relies on reference answers to identify correct samples, MAGENTA requires no ground-truth answers and instead establishes correctness through machine-checked proofs. Moreover, two experiments demonstrate that additional sampling alone does not explain the gains:

  • Under matched sampling budgets, independent resampling with Lean-based proof selection verifies only 1/6 IMO 2026 problems using 8–16 generations per problem, compared with 6/6 using verification-guided correction (§4.3).
  • Increasing standalone K2-Horizon-7B sampling yields only 86.67% at pass@2 and 93.33% at pass@32 on AIME 2025, both below MAGENTA's 100%.

Furthermore, MAGENTA achieves 100% accuracy across all evaluated reasoners, spanning model families and scales from 7B to 375B, including Kimi K3 (2.8T total parameters, approximately 400× the 7B reasoner). Hence, the conclusion does not depend on which baseline is chosen. These findings highlight the contribution of verification-guided correction beyond increased sampling. We have clarified the computational budgets and evaluation protocol in Section 3.1 and Appendix C.

[Q4] How sensitive are the results to the error judge's MATH versus SYNTAX decision? The prompt instructs the judge to return code_error when evidence is ambiguous, which biases toward proof repair. What happens if this bias is reversed?

Thank you again for bringing this to our attention. We clarify that error-judge misclassification affects efficiency rather than soundness, as all accepted solutions must pass Lean and SafeVerify against an adjudicated formal statement:

The two misclassification scenarios have distinct consequences:

  • A MATH error labelled SYNTAX. The prover attempts to repair the proof without correcting the underlying mathematical error, potentially requiring additional proof attempts until the error is correctly classified and control returns to the reasoner.
  • A SYNTAX error labelled MATH. The reasoner unnecessarily re-derives the solution using Lean's diagnostic feedback, incurring additional reasoning rounds despite the underlying mathematics being correct.

Thus, reversing the default under ambiguity from code_error to math_error may affect computational efficiency and convergence, but cannot compromise the soundness of accepted proofs, which remain subject to formal verification. We have clarified this distinction in §2.3 of the revised manuscript.

We are happy to address any further concerns. Thank you again for the insightful feedback!

AuthorsOpenReview ↗

Following up on our response to your concerns

Dear Reviewer ATYy,

We understand the discussion timeline may be inconvenient for your busy schedule, but we would love to continue our discussion before the discussion period closes.

We hope we've been able to address your questions and concerns so far! Please don't hesitate to let us know if there's anything else we can help clarify or discuss.

Thank you again for the feedback and comments, it was really helpful!

Warm Regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank Reviewers ANcM, HiUS, and ATYy for their valuable feedback. We are truly grateful for their constructive comments and their recognition of our Magenta framework.

We are pleased that the reviewers found our presentation clear (HiUS), our problem formulation precise and well posed (ATYy, HiUS), and our approach transparent (HiUS, ANcM). The reviewers further noted that our approach effectively closes the loop between natural-language reasoning and formal reasoning, and makes the gap between formal correctness and fidelity to the original problem explicit and measurable (ANcM, ATYy, HiUS). Finally, we also appreciate Reviewer ANcM’s recognition that the human verification and additional budget information strengthen the submission, and their subsequent decision to increase the rating.

Main contributions of the paper

  1. We introduce Magenta, an agentic, training-free pipeline that integrates natural-language mathematical reasoning with Lean 4 formal verification and uses verifiable compiler signals for self-correction, achieving a perfect score (100%) on competitive mathematics benchmarks (ATYy, HiUS, ANcM).
  2. Our core contribution is to close the loop between informal and formal reasoning. To this end, we develop a pipeline consisting of a reasoner that produces a reasoning chain and candidate answer, an autoformalizer that translates it into Lean 4, and a prover that writes the proof, verified by LLM judges and finally by the Lean kernel with SafeVerify (ATYy, HiUS, ANcM).
  3. Within Magenta, we introduce a statement judge that verifies the faithfulness of the autoformalized statement, and an error-attribution mechanism, routing MATH and SYNTAX failures to the appropriate component (ATYy, HiUS), closing the loop between formal and informal reasoning (ATYy, HiUS, ANcM).

Experimental Results and Presentation

  1. Magenta’s main contributions are its robustness and strong empirical results (ATYy, HiUS, ANcM), achieved while relying solely on open-source models (HiUS, ANcM, ATYy). Magenta achieves a perfect score on various benchmarks, including the recently released HMMT November 2026 (ATYy).
  2. When paired with a 7B reasoner model, Magenta further achieves 100% on IMO 2026 (HiUS), with results annotated by two IMO medalists, as well as by GPT together with a mathematics graduate annotator (ANcM).
  3. We further include thorough additional analyses, including a scaling analysis that reports cumulative pass rate against budget (ATYy), and ablations on statement adjudication and the self-correction component of the pipeline (HiUS, ATYy).

Changes in the revision

Following the reviewers’ feedback, we have incorporated the additional experimental results and clarifications provided during the rebuttal into our revised paper:

  1. In response to feedback from Reviewers ATYy, HiUS, and ANcM regarding potential contamination, we have added additional experiments evaluating K2-Horizon-7B with Magenta on HMMT November 2026, achieving 100%. We have also added experiments to perturb the numerical values of 15 AIME 2026 problems (manually checked), again achieving 100%. Both sets of results are included in the revised paper.
  2. In response to feedback from Reviewers ATYy, HiUS, and ANcM regarding the IMO grading annotators, we added a second IMO medalist and a second mathematics graduate as annotators. All awarded 7/7 on every problem (42/42). As suggested by Reviewer ANcM, full annotator details will be included in the final version.
  3. In response to concerns from Reviewers ATYy, HiUS, and ANcM, we have clarified that the budgets (T, M, K) are upper bounds, as Magenta stops at the first verified proof, averaging 1–2 generations. We also clarified that Magenta doesn’t have access to ground truth, and resampling alone leads to lower performance (e.g., only ⅙ of IMO problems solved). Since Magenta scores perfectly, its conclusion and robustness do not depend on the baseline.
  4. Finally, we have also addressed further clarifications raised by the reviewers in the main manuscript.

Most concerns were shared across reviewers and have been acknowledged by Reviewer ANcM, who raised their score. We sincerely thank the reviewers again for their recognition and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.