acceptodds
Under review as a conference paper at ICLR 2027

Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification

Abstract

Can AI make progress on important, unsolved mathematical problems? Large language models are now capable of sophisticated mathematical and scientific reasoning, but whether they can perform novel research is still widely debated and underexplored. We introduce HorizonMath, a benchmark of 113 predominantly unsolved problems spanning eight domains in mathematics and the mathematical sciences, paired with an open-source evaluation framework for automated verification. Our benchmark targets the generator-verifier gap: problems where discovery is hard and requires meaningful mathematical insight, but verification is computationally straightforward. This contrasts with most existing research-level benchmarks, which instead rely on formal proof verification or manual review, both of which are expensive to scale. Because these solutions are unknown, HorizonMath is resistant to data contamination, and most state-of-the-art models score under 10%. Using this framework, we identify five novel solutions to research problems that either resolve previously open questions or improve on the best-known published results, with GPT-5.4 Pro discovering two and GPT 5.6 Sol discovering three of these solutions. Across seven frontier model families, reasoning efficiency and behavior also vary substantially. We release HorizonMath as an open challenge and a growing community resource, where each verified solution is a candidate contribution to the mathematical literature.

Primary area:
datasets and benchmarks
Keywords:
AI for math, evaluation, benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_hiw4OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark HorizonMath with an automatic verification framework for evaluate mathematical discovery ability of LLM. The method is demonstrated and validated through the process of verifying the novel solutions of GPT. The work is significant for AI applications in mathematic research fields.

Critical strengths
  1. The benchmark consists of 113 predominantly unsolved problems which are selected carefully and resistant to memorization so that the mathematical reasoning ability of an LLM can be evaluated strictly and correctly.
  2. The automated verification pipeline and the associated verification scripts using the generator-verifier gap are provided in open-source so that the verification can be performed efficiently and conveniently.
Critical weaknesses

None

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Thank you!

Dear reviewer, thank you for your positive review! We're glad you found the problem selection, resistance to memorization, and open-source verification pipeline valuable. These were our main goals in creating HorizonMath.

Since submission, we have added a new set of 15 unsolved existence problems that bring the eval to 128 problems in total. The problem statements and their verifiers are in the anonymized repo. We are continuously growing HorizonMath as a community resource and unsaturated evaluation for frontier models.

Thank you again!

Reviewer_DNujOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces a new benchmark called HorizonMath which is made up of 113 unsolved advanced maths problems. The paper selected these problems for their ease to verify possible solutions. The paper presents the selection method, criteria for a solution to be valid, the level of difficulty and the mathematical domain. They also present a pipeline and formulation for evaluating potential solutions. They test these problems on nine LLMs and find that three of the GPT models were already able to solve up to 5 of the unsolved problems.

Critical strengths
  • the paper clearly outlines the methodology for selecting the problems
  • the paper discussed some major limatations transparently
  • the paper shows that some of the frontier model are already capable of solving or improving on existing results.
Critical weaknesses
  • Except for a brief mention as future work, the paper fails to mention why their evaluation method and model testing paradigm is different, complementary or more suitable that using formal-reasoning approaches.
  • The paper suggest that their automatic evaluation strategy is robust and reliable, but there is no testing of whether the evaluation system is robust and reliable.
  • There is little statistics in the paper about the dataset in more detail, for example the distribution of different types of outputs.
Questions for authors
  • Why do you not considering doing this for formal mathematics where similar datasets might already exist?
  • How robust is your evaluation strategy?
4 comments
AuthorsOpenReview ↗

Clarifying design principles and evaluation details

Dear reviewer, thank you for this review! We in fact discuss two of your three concerns (on formal methods and the evaluation's robustness) in the paper already. We hope to provide more clarity in the points below (and dataset statistics).

W1 / Q1: Why not formal mathematics?

Excluding formal proofs is one of the paper's central design choices: we motivate it in Sections 1, 3.1, 4, and 8 (as well as the abstract). There are two primary reasons.

First, verification is much easier. In the abstract we discuss how most research-level benchmarks rely on formal proof verification or manual review, both of which are expensive to scale. Formalization in Lean can be especially costly here since Mathlib currently has limited coverage of many of the relevant subfields; we go into more detail again in Sections 4 and 8. We deliberately chose problems whose answers can be checked by deterministic, verifiable computation (Section 3.1, Figure 2). And as Table 1 shows, most proof-based research benchmarks are small, and many are graded by hand.

Second, it lets us cover applied mathematics and the mathematical sciences, where a proof is often not the goal. (Physics is one such example.) The benchmark spans applied and computational mathematics, as well as branches of physics like statistical mechanics, electromagnetics, and continuum physics (Section 3.2, Figure 1). In these fields, progress usually means finding an explicit closed form, a construction or a better bound.

Finally, a practical problem with formalizing these questions is that a Lean statement of "find a closed form for I0I_0" must already contain the answer; however in our problems, finding that answer is the task itself. We see these as complementary: HorizonMath finds candidates closed-forms while proofs then certify them. All four closed forms the models found have since been proven (Section 6), with complete solutions provided in the codebase (Section 4.2). For the 42 problems that ask for a better bound or a construction, the verifier already checks every required property deterministically.

W2 / Q2: How robust is the evaluation?

For bound-improvement and existence problems, our verifiers are exact and deterministic. They programmatically check all required properties of the solution without requiring Lean, which is a unique aspect of these problems.

For closed-form problems, we directly test our evaluation in Table 5 of Appendix B.7. We carefully constructed 25 candidate solutions with hidden ground-truth labels to match possible failure modes. (Eight of them were invalid, including some written specifically to trick the checker.) The checker agreed with the labels on 123/125 cases and on all 25 majority votes, never accepting an invalid candidate.

Furthermore,

  • Section 7 reports 147 candidates that matched the numerical reference but were rejected for forbidden shortcuts, like fitting the numerical value or using numerical integration. A purely numerical check would have accepted all of them.
  • Experts review every accepted answer (Section 4.1). We verified all five new results, meaning the pipeline did not wrongly accept any of the novel answers.

W3: More detailed statistics.

Figure 1 and Section 3.2 give the breakdown by difficulty level and by domain. The full breakdown including output types, which is already provided in the readme file of the linked anonymous GitHub, is:

Evaluation modeL0L1L2L3Total
Closed-form (56 constants, 15 functions)10747771
Bound-improvement (30 constructions, 3 formulas)01517133
Existence (9 constructions)07209
Total1029668113

Along with showing accuracy for each model, we give analyses of per-model token efficiency (average output tokens, tokens per solve), and reasoning-strategy switches per 1k tokens for the open models in Appendix B.

Informativeness.

As far as we know, there are no other benchmarks of unsolved math research problems that are public, automatically verifiable, and have already produced novel results. Given the rapid pace of AI development in mathematical research, we believe each of these criteria (open problems, public availability, quick verification, and demonstration of robustness) are crucial, making HorizonMath particularly timely. We already found five new mathematical results in our evaluations of models; one improving on AlphaEvolve's Kakeya construction, and four other closed forms that have since been proven and were directly posed without closed forms in the published literature.

We also added 15 new unsolved Existence-type problems, bringing the dataset up to 128 total problems. The problems and their verifiers are now in the anonymized GitHub repo, and no solutions have been found for them yet.

We hope this clears up the concerns, and we would be very grateful if you would reconsider your score!

AuthorsOpenReview ↗

Following up on improvements and clarifications

Dear Reviewer,

We appreciate your feedback and have done our best to thoroughly address all of your comments. Please let us know if you have any additional questions as today is the end of the discussion period, and we've made several clarifications and improvements listed above.

We've also uploaded a revised PDF highlighting clarifications to your questions and comments on formal mathematics, novelty of our evaluation framework, and statistics of the dataset and model responses. These are colored in blue in:

  • Section 1, Fig. 1
  • Section 3.2, output types and justification of evaluation framework (no Lean)
  • Section 6, distribution of model performance in Fig. 4
  • Section 7, moved analysis & statistics of model reasoning efficiency and Table 2 into main text from the Appendix
  • Appendix B, analysis & statistics of model reasoning behavior and Tables 3 and 4

We would be very grateful for your consideration in updating our score given these clarifications. Thank you!

Thank you for the more detailed explanation especially of the first two points. Could you please use the extra page to make this clearer as I did not understand that from reviewing the paper. The example you give about mathematical science particularly resonates with me. I will change my score accordingly.

Thank you! Yes, we will certainly add more clarification of our evaluation framework and its differentiation from proof-based verifiers in the future manuscript. We appreciate your feedback.

Reviewer_fK26OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces HorizonMath, a benchmark of 113 mostly unsolved problems in mathematics and the mathematical sciences. Each problem is hard to solve but easy to check: an answer is verified against a high precision numerical reference, a published baseline, or a list of required properties. Across nine frontier models, the best one solves 12 problems, and the evaluation produces five new results that improve on or go beyond the published literature.

Critical strengths
  1. The benchmark is built on a clear idea. By choosing problems where discovery is hard but verification is easy, the authors measure progress on unsolved problems without relying on formal proofs or expert grading. Meanwhile, the evaluation is open and reproducible. Problem definitions, reference values, and verifiers are all public.
  2. The benchmark has already produced real results. Five new solutions came out of the evaluation, including a Kakeya construction that improves on AlphaEvolve. The authors proved the four closed forms instead of relying on numerical agreement alone, which makes these results convincing.
Critical weaknesses

None

Questions for authors

Several baselines, such as the Kakeya construction, come from search systems like AlphaEvolve. Can such systems be evaluated on HorizonMath under the same protocol, and how would their results compare with single model attempts?

1 comment

Dear reviewer, thank you for the positive review! We appreciate your recognition of why we designed the evaluation this way and how formal proofs/expert grading are not required. Below we answer your questions.

Can search systems like AlphaEvolve be evaluated under the same protocol?

Yes. The verifiers don't depend on what produced the solution, so any system that returns a solution in our proposed_solution() form is scored the same way—regardless of whether it's a single call to a model, an agent, or an evolutionary search loop as in AlphaEvolve.

We also rank bound-improvement problems by their relative improvement over the published baseline (discussed in section 4.2), which is the same type of objective that AlphaEvolve and other evolutionary frameworks look to optimize. So these can be plugged in directly.

How would they compare with single-model attempts?

The main different would likely be compute. Our results are pass@1 (for reasons we discuss in Section 6), but a search system might call the verifier many times. A more apples-to-apples comparison would report results with the inference compute used (perhaps tokens or turns). That would allow a more direct comparison with our token-efficiency analysis (in Appendix B.4) of each model.


We have also added 15 new unsolved existence-type problems to bring the benchmark to 128 total problems. Their problem statements and verifiers are now in the anonymized repo as well.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our responses to reviews, including clarifications and 15 new unsolved problems

Dear reviewers and ACs,

We thank all three reviewers for their time and feedback. We wanted to summarize the reviews and our responses.

Overall: Reviewers hiw4 (4, Clear Accept) and fK26 (3, Weak Accept) both listed no critical weaknesses and gave a 3/3 rating for soundness. Reviewer DNuj raised three concerns, which we believe our paper already addresses and justifies in depth. We also address them again in detail in our rebuttal. Below, we reiterate our main points to clarify all of Reviewer DNuj’s concerns.

Why not formal mathematics (W1/Q1). We exclude formal proofs as a central design choice, motivated in the abstract and in Sections 1, 3.1, 4 and 8. Specifically, Mathlib coverage can be limited for many of our subfields, and most models are not as natively strong at writing Lean as they are at regular Python code. Furthermore, writing proofs is a different type of mathematical reasoning than what we test, where only a correct result matters, and a formal derivation of all steps is unnecessary. (And by avoiding Lean verification, we preclude the possibility of a model producing a correct, novel discovery/answer but formalizing one step wrong in its proof—perhaps purely a Lean syntax error rather than a logical error—which makes our evaluation much more robust.)

Additionally, Lean proofs for problems asking for a closed form must already contain the answer, which is the very thing we ask the models to produce (which is unknown for our problems). And finally, the benchmark also covers applied mathematics and mathematical physics, where proofs are often not the goal. HorizonMath is complementary to formal approaches: it finds candidates, and proofs certify them.

Robustness of the evaluation (W2/Q2). We test this directly in Appendix B.7 / Table 5. We built 25 candidates, with eight of them invalid and some specifically written adversarially to fool our checker. The checker matched the labels on 123/125 cases and on all 25 majority votes, and did not accept invalid candidates. We also report in Section 7 147 candidates that matched the numerical reference but were correctly rejected for disallowed shortcuts or hacks like fitting numerical values or numerical integration. A purely numerical check would have accepted all of them.

And the other two classes of problems (bound and existence) all have verifiers that are exact and deterministic.

Dataset statistics (W3). The breakdown by domain and difficulty is in Figure 1 and Section 3.2. The full breakdown by model performance and difficulty levels is in Figure 4. Further analysis of evaluation mode, difficulty, model reasoning tokens, and trajectories are in Appendix B, and further detailed in our rebuttal and in the repository README.

On informativeness, we report five novel verified results, including an improvement on AlphaEvolve’s Kakeya construction, as well as several other novel solutions to previously unsolved open problems published in the literature. We appreciate Reviewers hiw4 for rating informativeness as a 3/3.

And in response to reviewer fK26's question about search systems such as AlphaEvolve: the verifiers are agnostic to how a solution is generated, so any system that can return a solution in our proposed_solution() format can be scored with our framework.

New problems. Since submission, we added 15 new unsolved existence-type problems, bringing the benchmark to 128. Their statements and verifiers are in the anonymized repository, and none has been solved yet.

We welcome any further questions from the reviewers or ACs and thank you again for your consideration. Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.