acceptodds
Desk rejected at ICLR 2027

From Rediscovery to Discovery: Verifiable Evaluation on Open Mathematical Problems

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent progress in AI for mathematics has pushed frontier models from competition-style problem solving toward research-level reasoning, yet how to measure verifiable progress on genuinely open mathematical problems remains largely unresolved. Most mathematical benchmarks evaluate reasoning against known answers, while existing open-problem settings do not clearly distinguish recovering established results, making measurable progress, and resolving open questions. We introduce **OpenProblemBench**, a result-verifiable benchmark of 65 mathematical research tasks spanning **L1 Rediscovery**, **L2 Progressive Discovery**, and **L3 Decisive Discovery**. We systematically evaluate frontier reasoning models, long-horizon agents, and discovery systems under this common setting. Our results reveal markedly different capability profiles across the three levels: frontier models can already rediscover multiple recently established results; long-horizon interaction substantially improves feasible-solution coverage and task-native metrics on L2 Progressive Discovery; yet no evaluated configuration reaches L3 Decisive Discovery. To better convert additional test-time computation into mathematical progress, we introduce **MathMap**, a simple yet effective verifier-grounded harness that combines diverse exploration, external verification, and verified selection. MathMap builds a persistent Verified Discovery Tree by expanding parallel tool-using branches, externally verifying candidate solutions, and continuing from the strongest verified checkpoint. Under matched backbone and time budgets, MathMap achieves rank 1 on 14 of 20 L2 tasks among the four same-backbone discovery systems, while using substantially fewer completion tokens and solution submissions. Together, OpenProblemBench and MathMap provide an open foundation for measuring AI progress from rediscovery through progressive discovery to decisive mathematical discovery.

Primary area:
datasets and benchmarks
Keywords:
Open Problem, Mathematical Discovery, Verifiable Evaluation, Agentic Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NDYgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper constructs OpenProblemBench, a benchmark for evaluating AI's research abilities on result rediscovery, making progress, and solving open problems. In contrast to many frontier researches that are not easy to verify, this benchmark underscores the verifiability that every task can be checked programmatically. This paper also introduces a simple workflow called MathMap. Experiments show that MathMap with DeepSeek can achieve comparable performance with Codex with GPT, beating some other baselines based on DeepSeek Harness.

Critical strengths

The three-layer benchmark level design is plausible. In order to verify whether generative AI can make progress in research-level problems, I agree that one should first test its ability on rediscovery, then making progress and targeting open problems.

Critical weaknesses
  1. The major weakness from my side is the aim for this benchmark. In the introduction, the authors list several AI's progress on famous open problems and say that an important question is verifiability. However, their benchmark only consider problems that are themselves easy to verify. That said, building this benchmark does not help with verifying problems that previously have such issues, and could not address the issue of "growing gap between what AI can now attempt and what current evaluations can reliably measure". What's more, the paper merely uses a short subsection to discuss about their design on result evaluation, which makes me further confused whether they really care about the verifiability issue they propose in the introduction. For example, as a human reader, I'm quite interested in the specific verification approaches but find it difficult to understand what does "symbolic coverage verification" mean from only a few words of definition: "symbolic coverage verification checks finite symbolic coverings that establish broader claims".
  2. The proposed workflow MathMap is not novel either. Workflows for automatic scientific research are here and there, and I see nothing new from this workflow. The experiments settings are not fair either, as the compared Codex and DSH are general-purposed and not curated for solving such problems. That said, the experiments should not only compare with these general-purpose frameworks, but at least with several other related baselines that are really designed for this purpose. Besides, the authors did not clearly consider how to deal with information leakage. If the AI agent directly gets the answer from the internet, or find other agents' answer in your computer, how could you tell whether they achieved the result mainly by themselves or through cheating? Moreover, due to the recent progress of OpenAI, I am concerned whether AI systems like GPT-6 Pro can already achieve perfect scores on the proposed bench. The compared systems are a little bit old and cannot reflect the real power of current AI systems.
Questions for authors

I notice that another paper has already used the name OpenProblemBench: https://arxiv.org/html/2610.11118v1. I suggest the authors modify their benchmark's name to an unused one that better describes their benchmark.

Reviewer_3Pe4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark of 65 research-level mathematics tasks, all checked by executable verifiers. The tasks are organized into three levels: L1 Rediscovery (10 results resolved in 2026), L2 Progressive Discovery (20 optimization tasks, each with a native objective), and L3 Decisive Discovery (35 constructions or counterexamples that would each resolve an open question). The authors evaluate reasoning-only models, agents running in harnesses, and discovery systems. The best configurations pass 60% of L1, long-horizon agents substantially improve L2 feasibility (51% → 82%) and objective values, and no configuration solves any L3 task. The paper also proposes MathMap, which repeatedly expands the best verified solution into three parallel tool-using branches. With DeepSeek-V4.1-Flash and a two-hour budget, MathMap ranks first (including ties) on 14 of 20 L2 tasks against SimpleTES, OpenEvolve and ShinkaEvolve, while using fewer tokens.

Critical strengths
  1. Every task is checked by an exact executable verifier, and the paper states explicitly what acceptance would mean mathematically for each task. Outcomes are therefore objective and reproducible. Beyond witness checks, some tasks are verified through symbolic coverage or RUP/LRAT certificates, which makes the benchmark auditable.

  2. The three-level framing is a useful way to separate reconstruction, measurable progress and resolution. In addition, the controlled comparison of discovery systems reports tokens, submissions and time-stamped progress curves, which is more informative than endpoint scores alone.

Critical weaknesses
  1. The evidence for MathMap is weak, and the comparisons are not clearly matched. Counting ties as Rank-1 hides the head-to-head picture. Against SimpleTES, MathMap is 8W/6L/6T in single runs, winning mostly by tiny margins on OR instances and going 1W/4L/2T on the "Substantial Advance" tier (e.g., W(2,7): 796 vs. 3,703). There are no ablations and no repeated seeds. The best-of-three selection is also degenerate on the binary levels L1 and L3. Several comparisons may not be matched. On L1, MathMap queries the verifier until it accepts, which amounts to an answer oracle on exact-answer tasks, and the other configurations' verifier access is not stated. Its fewer submissions partly reflect local self-testing. ShinkaEvolve runs without embeddings. The budget, tools and internet access of the Codex configuration are not specified. Finally, Table 3 and Fig. 4 disagree on Codex's MaxDet-23 result (0.9726 vs. 1.0).

  2. Each level has a validity problem, and the benchmark is not positioned against prior work. L1 uses public 2026 results, but model cutoffs are not reported, so rediscovery cannot be separated from recall. L1 also saturates, since the top three configurations solve the same 6 tasks, and it confounds format and tool limits with ability; for example, Hadamard-668 requires emitting the full 668×668 matrix. L2 reports no best-known values, so it is unclear whether any result advances the frontier. Its defect objectives are not validated as measures of progress toward L3, and its OR tasks lack classical-solver baselines. L3 gives zero signal, mostly because it asks for counterexamples to conjectures that are widely believed to be true. Finally, there is no related-work section: HorizonMath and FrontierMath: Open Problems are mentioned only in passing, although Hadamard-668 itself is taken from the latter.

  3. Each configuration is run only once, and there is no aggregate score. Theorem-type problems are excluded by design, because no formal proofs are used. The verifier code was written with AI assistance and has not been independently audited, which matters because a bug in an L3 verifier could certify a false resolution.

Questions for authors
  1. Fairness of the comparisons. Which configurations could query the verifier during search on L1 and L3, and how many verifier calls did MathMap make on the exact-answer L1 tasks? What were the time budget, tools and internet access for GPT-5.6-Sol + Codex? Why was ShinkaEvolve run without embeddings? Could you report multiple seeds for Table 4, together with MathMap ablations over branch count, selection rule, repair and verifier access? Could you also explain the MaxDet-23 discrepancy between Table 3 and Fig. 4?

  2. Validity of the three levels. What are the evaluated models' knowledge cutoffs relative to the L1 resolution dates? Did you run any contamination probe, and how do you separate format or tool limits (e.g., Hadamard-668) from mathematical ability? For each L2 task, what is the best-known literature value, and does any result exceed it (e.g., is P20's 3,703 equal to the known Rabung–Lotts bound)? Is there evidence that lower defect scores mean being closer to the L3 resolution? Why are classical solvers not included for P01–P06, and what is the rationale for L3 tasks whose counterexamples are believed not to exist? How will the benchmark be versioned as problems get solved?

  3. Positioning and verifier reliability. How does this benchmark differ from HorizonMath, from FrontierMath: Open Problems (from which R06 is drawn), and from AlphaEvolve-style construction problem sets? Could you add a related-work section? How were the verifiers validated, for example with mutation tests or independent second implementations? Do you plan to support formal proofs (e.g., in Lean) so that theorem-type open problems can be included?

Reviewer_jDcvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a deterministically judged benchmark for mathematical problems that asks the model to output a verifiable result rather than solution steps.

Critical strengths
  • It uses JSON output to do a deterministic judge and avoids an uncontrollable LLM-based judge.
Critical weaknesses
  • MathMap and SimpleTES's results are very close, so it is hard to say which one is better based on such a small number of tasks.
  • The judge only checks the JSON output and does not check the text solution explaining how the problem was solved. In addition, not all math problems can be converted into this format. Can the authors explain what kinds of problems fit this format and what kinds do not?
  • All systems failed on L3, and many conjectures in L3 are likely true, which means the required counterexamples may not exist and the tasks may not solvable. And L3 makes up about half of the 65-task dataset, making the effective dataset smaller.
  • Some models fail a lot on the format. Does this mean the format may be too difficult for models to follow? It may also cause models with sufficient math capability to fail because of format issues.
Questions for authors
  • How easy is it to scale this dataset to more tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.