acceptodds
Under review as a conference paper at ICLR 2027

OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences

Abstract

Progress in the foundational theoretical sciences requires resolving questions beyond established knowledge. We introduce OpenProblemBench, a benchmark of 82 unresolved problems drawn from the mathematics and theoretical physics literature. Each problem supplies the research context, assumptions, and prior progress needed to investigate the question. We select problems whose proposed solutions admit comparatively clear checks of their decisive mathematical or computational claims. Four evaluator models independently assess the correctness, completeness, and degree of progress of each submission without reference solutions. Across seven evaluated configurations, GPT-6-Astra attains the highest mean judged solve rate of 14.02%. GLM-5.3-Flash also receives solved judgments and a rate of substantive partial progress comparable to several larger systems. Case comparisons connect stronger outcomes to changes in problem representation, general arguments that extend beyond finite evidence, and proofs of the steps needed to complete a solution. By grounding evaluation in questions arising from the research literature, OpenProblemBench provides a setting for investigating the capabilities and limitations of AI as a contributor to foundational theoretical science.

Primary area:
datasets and benchmarks
Keywords:
AI for Science, LLM Agents, Open Research Problems, Scientific Reasoning, Benchmarking, Automated Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fWibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents OpenProblemBench, a benchmark of 82 unresolved problems from the mathematics (46) and theoretical physics (36) literature. Each problem comes with background, known progress and the conditions a solution must meet. Seven model–harness configurations each submit one solution per problem, working offline for up to 12 hours. Since no reference answers exist, four evaluator models (GPT-5.6-Sol, Kimi-K3, GLM-5.3, Qwen3.8-Max) independently judge each submission as solved, partial (trivial, nontrivial or breakthrough) or unsolved, and also flag overclaims. GPT-6-Astra has the highest mean judged solve rate (14.0%), and all other configurations are below 8%. Ten problems are judged solved by all four evaluators, eight of them from Astra. Case studies compare how models approach the same problems.

Critical strengths
  1. The goal is useful. Most science benchmarks have a known answer or a published result to reproduce. Testing models on questions that are still open in the research literature complements these benchmarks.
  2. The graded outcomes show differences that a solve rate hides. Qwen and Kimi solve fewer problems than GPT-5.6-Sol, but receive solved or substantive partial judgments more often (73.5% and 74.1% vs. 63.1%). GLM-5.3-Flash makes substantive progress at about the same rate as GLM-5.3 (63.7% vs. 62.8%) despite fewer solves.
Critical weaknesses
  1. The main metric is not validated. Every "solved" verdict is a model judgment. No solution was checked by a domain expert or formalized, and the judges were not tested on proofs with known flaws or on problems with known answers. The judges are also among the models being tested. Section 6 admits that agreement between judges cannot estimate false positive or false negative rates. A correct solution to any of these problems would be a research result on its own, so the solve rates are hard to interpret until at least some verdicts are checked.
  2. The mathematics half gives almost no signal. Under the Qwen evaluator, only one of 46 mathematics problems is solved by any model (Section 4.5). The authors explain that requiring checkable solutions selected broad, long-standing conjectures. As a result, comparisons between models rely mostly on the physics subset, which weakens the framing as a benchmark for both fields.
Questions for authors
  1. Have any of the ten unanimously solved problems been checked by domain experts, the original authors, or a formal proof? If not, could you verify at least one or two?
  2. For the eight problems that receive solved judgments from only one to three evaluators, did you analyze why the evaluators disagree? For example, which proof obligations cause the disagreement, whether some evaluators are systematically more lenient, and which evaluator is correct in at least one case?
1 comment

Thank you for pressing on validation of the headline metric and what it measures in mathematics.

Checking positive judgments. A statistical physics specialist confirmed the Pauling asymptotic in GPT-6-Astra's ORB-PHYS-62 submission, one of the ten cases highlighted in the original paper. Appendix E also describes targeted checks of claims in the revised submissions. We are organizing further expert checks to measure the reliability of closure judgments.

Why evaluators disagree. Across the 318 matched submissions, GLM-5.3, Qwen3.8-Max, and Kimi-K3 give 35, 29, and 46 closure votes, respectively. Kimi-K3 also gives the most closure votes when its own submissions are excluded from all three judges' totals. We inspected all nine cases in which only Kimi-K3 voted closed (Section 4.3 and Appendix C.2). On ORB-PHYS-60 it treated an asserted identity as routine, leaving a required derivation unchecked. On ORB-PHYS-69 it credited finite-field computations and a proposed mechanism as establishing a classification over general fields. On ORB-PHYS-57 it accepted a boundary counterexample at which the split-bond parameter is undefined. The reports also identify errors caught by Kimi-K3. For ORB-PHYS-76, direct enumeration gives zero flows on a graph where DeepSeek-Flash's recurrence gives six, supporting the Qwen3.8-Max and Kimi-K3 decisions to withhold closure.

Mathematical reasoning and task difficulty. We added several mathematics problems in the revision. The new tasks remain challenging for the evaluated solvers, and none received a closure vote. Many physics-labeled tasks also require substantial mathematical reasoning. Examples include classifying the integrality of singular values of a generalized modular lambda function (ORB-PHYS-82), deriving deletion–contraction recurrences for local flows and tensions on nonorientable surfaces (ORB-PHYS-76), and proving the Yang–Baxter equation for an elliptic R-matrix (ORB-PHYS-88). Each of these tasks had at least one submission that all three judges marked closed. We plan to add mathematics problems across a broader range of difficulties to better distinguish model performance and to test newer frontier models on the hard problems already in the benchmark.

Reviewer_zHahOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OpenProblemBench, a benchmark of 82 open problems in mathematics and theoretical physics curated from the literature. Seven agent configurations each attempt every problem once, offline, for up to 12 hours. Four LLM judges then grade each attempt without reference solutions as solved, partial (trivial / nontrivial / breakthrough) or unsolved, and also flag overclaiming. GPT-6-Astra leads with a 14.0% mean judged solve rate (2.4–7.0% for the other configurations), and ten problems are judged solved by all four judges. The paper also includes a small harness/web-access ablation and qualitative case studies.

Critical strengths
  1. The paper provides a curated, well-documented set of open research questions spanning both mathematics and mathematical physics. This is broader than existing open-problem suites, which mostly cover mathematics.

  2. The outcome taxonomy is sensible. Separating full solutions from graded partial progress, together with an overclaim diagnostic, targets the right failure modes for research-level work, and the case studies illustrate these failure modes well.

Critical weaknesses
  1. The evaluation is LLM-only, under-specified and unvalidated. There are no reference solutions and no human-expert check, not even for the 10 unanimously "solved" problems. The judge prompts are not disclosed, the partial-progress levels rest on one-sentence definitions with no problem-specific rubric or weighting, and each judge grades each attempt only once.

    The paper's own data suggest low reliability. Only 8 of Astra's 15 solved judgments are unanimous, the same GLM-Flash submissions receive 67 vs. 33 "nontrivial" labels from two judges, and over half of all reviews are "nontrivial" on a set that includes Goldbach and Collatz. There are also signs of same-family bias: all four judges are themselves solvers, and the GPT judge rates Astra at 18.3% versus 11.0% from the Qwen and GLM judges.

  2. Solvers and judges are fully offline, and nothing checks claims against the literature. Disabling web access for solvers is not justified, even though the paper's own ablation shows that web access changes outcomes, and Astra is never run with web access. The judges' access policy is undefined, so they cannot tell whether a "partial" or "solved" result is already known. The literature cutoff, and who verified that each problem is open, are also not stated.

  3. Evaluation Confidence Interval Each configuration is run once, and different configurations use different harnesses; the harness alone flips 4 solved outcomes. Compute usage is not reported, solutions and judge reports are not promised for release, and the case studies were selected for contrasting outcomes.

Questions for authors
  1. Expert validation Has any domain expert checked the 10 unanimously "solved" submissions in Appendix C, and if so, how many were confirmed correct? If none have been checked, could you verify at least a few of them (e.g., ORB-PHYS-62, -57, -69) during the rebuttal? Will the solutions and judge reports be released so that others can check them?

  2. Judge protocol and reliability. Could you release the judge prompts? Do they contain any problem-specific criteria, for example each problem's central obstacle, or which known results count only as "trivial"? Were the judges calibrated on known-correct and deliberately flawed proofs? How consistent are they across repeated runs (e.g., 3–5 runs on the 18 problems with any "solved" judgment), and how do the results change when self- and same-family judgments are removed? What evidence shows that the partial levels reflect progress toward a full proof, and that a "nontrivial" result can be counted together with "solved" in the combined metric (84.45% for Astra)?

  3. Offline design. What is the judges' access policy in the primary setting? If the judges are offline, how are new partial results distinguished from known ones? Were the "solved" problems re-checked against the literature, and what is the curation cutoff? Why is offline the primary setting for solvers? Could you report web-enabled runs for Astra, graded by a judge that can consult the literature?

1 comment

Thank you for asking for expert checks, repeatability, disclosure of the prompts, and an online comparison.

Q1. Expert validation and access. During the rebuttal period, we completed one expert check among the ten cases highlighted in the original paper. A specialist in statistical physics reviewed the earlier GPT-6-Astra submission for ORB-PHYS-62 and confirmed its Pauling asymptotic. The revised GPT-6.1-Sol submission gives the same asymptotic through a different argument for the final uniform bound. We are organizing expert review of the remaining cases. The revised paper describes targeted checks of the counterexample in ORB-PHYS-57 and the tensor calculations in ORB-PHYS-69. Expert review of those submissions is still pending.

The anonymous supplement provides all 81 deployed problem statements, the solver and review instructions, the review prompt, and the referee skill. To limit answer leakage into future benchmark runs while supporting independent scrutiny, we welcome domain experts to contact us through OpenReview for the complete solutions and judge reports for the questions they would like to verify.

Q2. Judge protocol and scoring. Appendix D describes the workflow. The accompanying supplement includes the exact judge prompt, review instruction, and referee skill. Judges derive a checklist from the requirements of each task, trace decisive proof steps, check assumptions and exceptional cases, and can run computations. For example, Qwen3.8-Max verified the negative covariance of the submitted ORB-PHYS-64 counterexample by exact enumeration on a finite torus. Finding an admissible witness can be difficult within the solver's budget. A judge can verify a supplied witness directly.

We agree that the original progress categories were underspecified. The revised resolution rubric drops the trivial, nontrivial, and breakthrough labels. When a task explicitly asks several questions, the judge identifies them from the statement and can give partial if the submission resolves some of them. The headline metric counts a task as solved for a given solver only when the submission resolves all required parts according to every available judge. Partial judgments receive no closure credit. Appendix G reports the research value of supported contributions separately.

Repeatability and self-review. On nine fixed submissions from three problems, we compared four GLM-5.3 reviews per submission (Section 4.3 and Appendix C.3). Across 54 within-submission pairs, 40 agree on closure (74.1%). Five submissions receive the same closure decision in all four reviews. The follow-up checks show that instances of disagreement arise from variation in mathematical verification and task interpretation. Calibration on known-correct and deliberately flawed proofs remains open. Figure 3 gives the per-judge votes and coverage. Recomputing the counts after removing GLM-5.3 and Kimi-K3 self-votes on the fixed set of 318 submissions with three reports gives 13, 8, 2, and 1. The solver ranking is unchanged. The revised panel has no OpenAI-family judge.

Q3. Online evaluation and literature access. We agree that both solving and reviewing should allow literature retrieval. All four solver configurations and all three judge models in the revised main evaluation ran with Internet access. Judges can use web search and LKM to check whether an apparently new result is already known, and the protocol asks them to examine primary sources for decisive comparisons. The November 7, 2026, literature cutoff applies to all retained questions, including those later judged closed. Appendix H documents the curation decisions.

Reviewer_Eg9QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenProblemBench collects 82 literature-grounded questions, comprising 46 mathematics and 36 theoretical-physics problems. Seven model-harness configurations each produce one submission per problem. Four evaluator models assess whether each submission solves the task or makes trivial, nontrivial, or breakthrough partial progress, without reference solutions. This produces 574 submissions and 2,296 primary reviews. GPT-6-Astra has the highest mean judged solve rate at 14.02%, with eight submissions unanimously judged solved.

Critical strengths
  1. Evaluating open problems is meaningful. The paper evaluates the reasoning and problem-solving abilities of models on unresolved problems from the literature. This addresses an important limitation of existing benchmarks, which mainly focus on problems with known answers.

  2. The evaluation results are relatively transparent. The paper provides the full scoring matrix of models and evaluators, separately reports partially solved problems, and analyzes cases where models incorrectly claim to have solved a problem. It also lists the problems that all evaluators agree are solved. These results help show disagreements between evaluators rather than hiding them behind an average score.

Critical weaknesses
  1. The main evaluation lacks independent verification. All correctness and progress scores are given by LLM-based judges, without expert review or human-verified samples to estimate evaluation errors. This is especially concerning for open problems, where an incorrect but convincing proof may fool multiple judges. Appendix C lists 10 problems with solutions accepted by all judges, but their agreement does not guarantee correctness. Appendix E only provides summaries rather than complete proofs, making it difficult to verify the key steps. Also, claims of nontrivial progress or breakthroughs should be checked against existing literature to confirm their novelty.

  2. The problem selection and experimental protocols are not sufficiently clear or reproducible. Section 3.1 mentions reviewing the literature to confirm that problems remain open, but does not provide search dates, detailed records, or clear selection criteria. Table 9 only lists problem titles without complete statements, making it difficult to verify what was actually solved. In addition, different models use different harnesses and computing budgets, up to 12 hours with no fixed token budget, and varying realized resources, but detailed configurations and resource usage are missing.

Questions for authors

Q1. The reliability of using agreement among four judge models to evaluate open problems is unclear. Since these models may not be able to solve the open problems themselves, are they reliable enough to judge correctness? Also, how are cases handled when the judges disagree?

Q2. What is the literature search cutoff date for each open problem? What evidence confirms that these problems remain unsolved? Are the problem statements and allowed counterexamples given to the models consistent with the original conjectures?

Q3. Can the authors provide the complete solver and evaluator prompts, available tools, timeout settings, actual running times, and resource usage? Which model comparisons remain valid under the same compute budget?

Q4. How do the judges distinguish genuinely new research progress from known theorems, simple consequences, or results missing from the provided context? Has this evaluation been validated by domain experts rather than relying only on other LLMs?

1 comment

Thank you for asking for independent verification and an auditable evaluation protocol.

Q1. Correctness and judge agreement. We use unanimous closure as a consistent, auditable outcome for comparing solvers under the same protocol. A submission counts as solved only when every available judge votes closed. This prevents a single positive vote from deciding the result when several reports are available. Among the 318 submissions with all three reports, 25 had split votes (7.9%). The available-report rule gives 14, 8, 3, and 1 closures for GPT-6.1-Sol, DeepSeek-Flash, GLM-5.3, and Kimi-K3.

Each judge independently reviews the exact task and submission, checks decisive proof steps, assumptions, and exceptional cases, and runs symbolic or numerical checks where appropriate. In ORB-PHYS-82, judges followed the integrality proof to its modular equation. Qwen3.8-Max and Kimi-K3 independently recomputed its constant term for selected inputs. In ORB-PHYS-64, Qwen3.8-Max enumerated configurations on the submitted 3×33\times3 torus and verified its negative covariance. Finding an admissible counterexample can be difficult within the solver's budget. Once it is supplied, a judge can check that it meets the assumptions and violates the claimed inequality.

We agree that several judges can miss the same error. In ORB-PHYS-77, an interactive follow-up with GPT-6.1-Sol helped us refute a key bound in DeepSeek-Flash's submission, which all three judges had marked closed. We are organizing domain-expert checks to assess and improve the closure metric. A specialist in statistical physics has confirmed the Pauling asymptotic in the earlier GPT-6-Astra submission for ORB-PHYS-62. The revised GPT-6.1-Sol submission establishes the same result using a different argument for the final uniform bound.

On nine fixed submissions, four GLM-5.3 reviews per submission agreed on closure in 40 of 54 within-submission pairs (74.1%). Follow-up checks trace disagreements to differences in mathematical verification and task interpretation (Section 4.3 and Appendix C.3). We will use expert judgments to evaluate newer judges and improve how the protocol verifies decisive proof steps, including through targeted, lower-cost checks (Section 6).

Q2. Openness and fidelity to the source. The literature cutoff for the revised problem set was November 7, 2026. We checked the retained tasks against their source papers during curation and again during revision, with particular attention to assumptions, admissible counterexamples, and what counts as a resolution. The deployed tasks organize the source material into a consistent, self-contained format. Appendix H shows two cases where the earlier task needed clarification to match the intended scope. To check openness, we searched the web and used LKM's arXiv-wide reasoning-graph index, then inspected source passages to see whether a result covered the stated scope. During the rebuttal period, we retired several tasks after finding public results that resolved them within their stated scope. Appendix H documents examples. We found no public resolution of the retained questions by the cutoff. The web-enabled solver traces give us an additional way to follow up on any prior result a solver finds. All 81 deployed statements and references are in the supplement. A complete per-question query log is unavailable. LKM, Section 4.1

Q3. Prompts, tools, and compute. Section 3.3 and Appendix D describe the setup, and the supplement gives the full instructions and prompts. The four solvers use the same tasks and web-enabled Codex/Harbor harness under a 12-hour wall-clock limit and standard requests of two CPU cores and 8 GiB of memory. Review jobs request two cores and 4 GiB under the same time limit. The comparison is valid for the reported model–execution configurations under those resource limits. Reasoning profiles differ, and we lack complete realized resource accounting. For 18 exact-hash matches in one review wave, median review times were 59, 148, and 55 minutes for GLM-5.3, Qwen3.8-Max, and Kimi-K3.

Q4. Research progress and novelty. We agree that the original progress categories were too broad for a combined headline score. The revised main evaluation asks whether the stated problem was resolved. Appendix G separately rates the originality and scientific value of the contribution the judge could verify, using a five-point scale anchored by examples from relevant journal standards. Judges can use LKM and web search, then inspect the decisive passages of the primary literature to distinguish a new finding from a known result or a routine consequence. A proved special case can earn research-value credit while the main question remains open. Expert validation of these novelty judgments is part of the planned follow-up.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Updated Evaluation

In response to the reviews, we reran the evaluation on a revised release of OpenProblemBench. The research question remains how agents make verifiable progress on literature-grounded open problems. The revision makes four changes.

  • Problem set. We rechecked openness against the literature through November 7, 2026. We retained 74 of the original 82 source questions and added seven, bringing the total to 81. We repaired formatting, removed distracting guidance, and clarified assumptions, admissible counterexamples, and completion conditions. Section 3 and Appendix H describe the curation and give examples.
  • Solver setup. Four solver configurations now use a common web-enabled Codex/Harbor harness. The revised solver instruction asks for a self-contained submission with citations and a record of any remaining gaps. Section 3 and Appendix D describe the configuration, and the supplement contains the exact instruction.
  • Review protocol. Three judges independently check the revised statements and exact submissions. The updated prompt and referee skill make each judge record task-specific proof obligations, decisive checks, and grounds for its verdict. The supplement provides the review instructions, prompt, and referee skill.
  • Reported results. The revised manuscript reports the rerun as its main experiment. At the snapshot cutoff, Qwen3.8-Max had provided reports for 318 of the 324 submissions, leaving six reports pending. GLM-5.3 and Kimi-K3 had reports for all 324 submissions. Figure 3 gives closure votes and the number of available reports for each solver–judge pair. The three-judge agreement analysis uses the 318 submissions with all three reports. Section 4.3 and Appendix C also examine Kimi-K3's disputed acceptances and report a targeted GLM-5.3 repeatability study. Appendix E documents checks of specific claims, and Appendix G reports research value separately from closure.

The problem set, models, harness, and assessment protocol changed together. Each release's results apply to its own configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.