- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper constructs OpenProblemBench, a benchmark for evaluating AI's research abilities on result rediscovery, making progress, and solving open problems. In contrast to many frontier researches that are not easy to verify, this benchmark underscores the verifiability that every task can be checked programmatically. This paper also introduces a simple workflow called MathMap. Experiments show that MathMap with DeepSeek can achieve comparable performance with Codex with GPT, beating some other baselines based on DeepSeek Harness.
Critical strengths
The three-layer benchmark level design is plausible. In order to verify whether generative AI can make progress in research-level problems, I agree that one should first test its ability on rediscovery, then making progress and targeting open problems.
Critical weaknesses
- The major weakness from my side is the aim for this benchmark. In the introduction, the authors list several AI's progress on famous open problems and say that an important question is verifiability. However, their benchmark only consider problems that are themselves easy to verify. That said, building this benchmark does not help with verifying problems that previously have such issues, and could not address the issue of "growing gap between what AI can now attempt and what current evaluations can reliably measure". What's more, the paper merely uses a short subsection to discuss about their design on result evaluation, which makes me further confused whether they really care about the verifiability issue they propose in the introduction. For example, as a human reader, I'm quite interested in the specific verification approaches but find it difficult to understand what does "symbolic coverage verification" mean from only a few words of definition: "symbolic coverage verification checks finite symbolic coverings that establish broader claims".
- The proposed workflow MathMap is not novel either. Workflows for automatic scientific research are here and there, and I see nothing new from this workflow. The experiments settings are not fair either, as the compared Codex and DSH are general-purposed and not curated for solving such problems. That said, the experiments should not only compare with these general-purpose frameworks, but at least with several other related baselines that are really designed for this purpose. Besides, the authors did not clearly consider how to deal with information leakage. If the AI agent directly gets the answer from the internet, or find other agents' answer in your computer, how could you tell whether they achieved the result mainly by themselves or through cheating? Moreover, due to the recent progress of OpenAI, I am concerned whether AI systems like GPT-6 Pro can already achieve perfect scores on the proposed bench. The compared systems are a little bit old and cannot reflect the real power of current AI systems.
Questions for authors
I notice that another paper has already used the name OpenProblemBench: https://arxiv.org/html/2610.11118v1. I suggest the authors modify their benchmark's name to an unused one that better describes their benchmark.