- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents OpenProblemBench, a benchmark of 82 unresolved problems from the mathematics (46) and theoretical physics (36) literature. Each problem comes with background, known progress and the conditions a solution must meet. Seven model–harness configurations each submit one solution per problem, working offline for up to 12 hours. Since no reference answers exist, four evaluator models (GPT-5.6-Sol, Kimi-K3, GLM-5.3, Qwen3.8-Max) independently judge each submission as solved, partial (trivial, nontrivial or breakthrough) or unsolved, and also flag overclaims. GPT-6-Astra has the highest mean judged solve rate (14.0%), and all other configurations are below 8%. Ten problems are judged solved by all four evaluators, eight of them from Astra. Case studies compare how models approach the same problems.
Critical strengths
- The goal is useful. Most science benchmarks have a known answer or a published result to reproduce. Testing models on questions that are still open in the research literature complements these benchmarks.
- The graded outcomes show differences that a solve rate hides. Qwen and Kimi solve fewer problems than GPT-5.6-Sol, but receive solved or substantive partial judgments more often (73.5% and 74.1% vs. 63.1%). GLM-5.3-Flash makes substantive progress at about the same rate as GLM-5.3 (63.7% vs. 62.8%) despite fewer solves.
Critical weaknesses
- The main metric is not validated. Every "solved" verdict is a model judgment. No solution was checked by a domain expert or formalized, and the judges were not tested on proofs with known flaws or on problems with known answers. The judges are also among the models being tested. Section 6 admits that agreement between judges cannot estimate false positive or false negative rates. A correct solution to any of these problems would be a research result on its own, so the solve rates are hard to interpret until at least some verdicts are checked.
- The mathematics half gives almost no signal. Under the Qwen evaluator, only one of 46 mathematics problems is solved by any model (Section 4.5). The authors explain that requiring checkable solutions selected broad, long-standing conjectures. As a result, comparisons between models rely mostly on the physics subset, which weakens the framing as a benchmark for both fields.
Questions for authors
- Have any of the ten unanimously solved problems been checked by domain experts, the original authors, or a formal proof? If not, could you verify at least one or two?
- For the eight problems that receive solved judgments from only one to three evaluators, did you analyze why the evaluators disagree? For example, which proof obligations cause the disagreement, whether some evaluators are systematically more lenient, and which evaluator is correct in at least one case?
Thank you for pressing on validation of the headline metric and what it measures in mathematics.
Checking positive judgments. A statistical physics specialist confirmed the Pauling asymptotic in GPT-6-Astra's ORB-PHYS-62 submission, one of the ten cases highlighted in the original paper. Appendix E also describes targeted checks of claims in the revised submissions. We are organizing further expert checks to measure the reliability of closure judgments.
Why evaluators disagree. Across the 318 matched submissions, GLM-5.3, Qwen3.8-Max, and Kimi-K3 give 35, 29, and 46 closure votes, respectively. Kimi-K3 also gives the most closure votes when its own submissions are excluded from all three judges' totals. We inspected all nine cases in which only Kimi-K3 voted closed (Section 4.3 and Appendix C.2). On ORB-PHYS-60 it treated an asserted identity as routine, leaving a required derivation unchecked. On ORB-PHYS-69 it credited finite-field computations and a proposed mechanism as establishing a classification over general fields. On ORB-PHYS-57 it accepted a boundary counterexample at which the split-bond parameter is undefined. The reports also identify errors caught by Kimi-K3. For ORB-PHYS-76, direct enumeration gives zero flows on a graph where DeepSeek-Flash's recurrence gives six, supporting the Qwen3.8-Max and Kimi-K3 decisions to withhold closure.
Mathematical reasoning and task difficulty. We added several mathematics problems in the revision. The new tasks remain challenging for the evaluated solvers, and none received a closure vote. Many physics-labeled tasks also require substantial mathematical reasoning. Examples include classifying the integrality of singular values of a generalized modular lambda function (ORB-PHYS-82), deriving deletion–contraction recurrences for local flows and tensions on nonorientable surfaces (ORB-PHYS-76), and proving the Yang–Baxter equation for an elliptic R-matrix (ORB-PHYS-88). Each of these tasks had at least one submission that all three judges marked closed. We plan to add mathematics problems across a broader range of difficulties to better distinguish model performance and to test newer frontier models on the hard problems already in the benchmark.