- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The reviewer uses "human" to denote my hand-written opinions in the review and the reviewer uses "Opus" to denote Opus 5.5. The reviewer uses the incognito mode so no data will be shared to Claude.
Human: This paper proposes a new benchmark about undergraduate-level research math problems, which could be less attended so avoid future leakage. Experiments show that the state-of-the-art models still are not doing well in this benchmark, where the improvement space could be large.
Critical strengths
Human:
-
This design could be insightful for future benchmarking operations. And it seems that this is a real advance over Formal Conjectures.
-
Experimental results cover a wide range of models, and it reveals that Astra cannot do well in those problems.
Critical weaknesses
Human:
-
The authors should compare to a few agentic approaches beyond pure LLMs.
-
Without human reviewers, the proposed Lean solutions may be flawed.
-
The number of tasks is only 244, which isn't large. The abstract logic is not sound because even though the domain is rare, the leackage could still be large because the size of the tasks.
-
Some sample problems should be provided to the reviewer as long as the Lean solutions. This is a standard practice for a benchmark paper.
Opus (verified by the reviewer):
-
The resolved problems are not audited, and the refutation-heavy results raise doubts about formalization fidelity. This is the central issue for a benchmark of open problems. Thirty-eight of the 56 resolutions are refutations, and GPT-6 Astra alone refutes 15.6% of the benchmark. That rate is hard to square with the claim that every problem is a genuine, unresolved research conjecture vetted by experts.
-
Dataset documentation is thin for a dataset paper. The benchmark is the main contribution, but the paper provides almost no descriptive statistics. Readers need: the number of source papers and programs, and their distribution over years and institutions; how many problems per paper; statement length; how many tasks introduce custom definitions versus pure Mathlib statements; dependence on specific Mathlib areas; the conjecture type (universal statements, existence, bounds, classifications).
The reviewer might raise his score if the rebuttal is convincing.
Questions for authors
From Opus:
For each of the 56 resolved problems, was the result a genuine resolution of the source conjecture, a misformalization artifact, or already known? How many refutations exploit generality or edge cases absent from the source?
Did agents use the Internet during runs? Can you rerun without network access, or report web-search usage in trajectories?
How many candidates were filtered at each stage of quality control, and what were the reasons? Can you report an independent expert audit of a random sample?
Can you provide dataset statistics (sources, years, custom definitions, statement types) and several worked examples in the main text?
What is the breakdown of failure modes, and how often did Comparator or nanoda reject patches that passed lake build?
6 comments
Thanks for the rebuttal. It is hard to convince the reviewer if the paper isn't revised significantly. And the reviewer's concerns are not addressed. For example, the reviewer questioned the human validation of the solutions, but the authors replied the human provided questions.
Thank you for the follow-up. We would like to clarify that the human validation of the solutions is addressed in our response to Opus W1 / Q1, and we apologize for not cross-referencing it under Human W2.
For each of the 56 resolved problems, we sent the accepted solution back to the human data contributor who formalized that problem, to validate it against the source paper. In addition, we ran independent agentic checks with Claude Fable 5.2 and GPT-6 Astra on all 56 solutions. Neither check found a formalization problem, and no human response so far has reported an issue.
We also note that every accepted solution is a complete Lean proof verified by Comparator with two independent kernels (Section 4.1), so its correctness with respect to the Lean statement does not rely on human judgment.
We hope this clarifies the point, and we would be glad to address any remaining concerns.
We thank the reviewer for the careful reading and for recognizing the design as an advance over Formal Conjectures.
Human W1. Agentic approaches. All results come from agents: every model runs in Leanimum-agent with filesystem access, shell commands, Mathlib search, and Lean feedback for up to 1,000 steps and 6 hours (Sections 4.2, 5.1). With general bash access, an agent can itself choose to use other tools, such as SMT solvers, neural provers, or neural tactics, as part of its policy. A common harness avoids confounding models with scaffolding (Section 4, issue 3). We welcome the community to use the dataset in other settings, e.g., with dedicated proving systems, but our benchmark targets the capability of models as coding agents.
Human W2. Human review. Every problem is reviewed by humans (Section 3.3). Formalizers on our team, including several active Mathlib contributors, formalize problems in their own areas. Independent checks with Codex and Claude Code then flag issues of origin, open status, centrality, and faithfulness, which go back to the formalizer for revision or replacement, and a Mathlib contributor gives a final review before inclusion. Every accepted solution is verified by Comparator with two kernels (Section 4.1).
Human W3. Size and leakage. 244 tasks is in line with widely used benchmarks: miniF2F (244), ProofNet (371), ProgramBench (200), and Terminal-Bench 2.0 (89). One pass over the 244 problems costs hundreds to thousands of US dollars per model and reasoning-effort setting. On leakage, all problems were open when collected, so no solution could be in the models' training data.
Human W4. Sample problems. All 244 tasks are available at https://anonymous.4open.science/r/reuF2F-021F.
Opus W1. Audit of resolved problems. Beyond the pipeline above, we ran independent agentic checks with Claude Fable 5.2 and GPT-6 Astra on each of the 56 resolved problems (including all 38 refutations), comparing each with its source paper. We also sent each solution back to the data contributor who formalized it. Neither check found a formalization problem (some contributor responses are still pending). Discrepancies among unresolved problems cannot be fully ruled out, but as discussed in Section 6, in the formal-to-formal setting a misformalized statement is still a precise question that requires a verified proof or refutation, so it remains a valid signal. On the share of refutations: we formalize each conjecture faithfully as posed in its source, without filtering by difficulty or expected truth value, so the benchmark reflects the actual difficulty of the collected problems. A genuinely open conjecture can be false, and refutations are more accessible to current agents (in Table 3, five of seven models only refute). Many of this year's AI contributions to well-known conjectures have likewise come through refutations, e.g., the disproof of the unit distance conjecture (Section 1).
Opus W2. Documentation. The full benchmark is at the link above, and we will open-source everything: the repository, the Leanimum-agent harness, the Harbor integration, and detailed per-problem metadata. Statistics are in Q4.
Q1. See Opus W1.
Q2. Internet use. Search is part of the general agentic capability we evaluate, so we consider Internet access reasonable. We checked all successful and failed trajectories and found no case of retrieving a solution online, so we regard these resolutions as new. Agents do search for formal resources, e.g., solutions reused Lean formalizations of the Borsuk–Ulam and Kneser theorems from GitHub. This is expected; submissions are still fully verified.
Q3. Quality control. Quality control is carried out by the formalizers and reviewers as described in Human W2. We do not collect information on candidates that were not selected, as it is not part of the benchmark we deliver.
Q4. Statistics. 244 distinct source papers (one per paper) from 64 programs (table below); 144 on arXiv; years 1979–2026, 85% from 2013 or later; 45 problems in each of five areas plus 19 in Others; median challenge file 57 lines (25–445); 205 problems introduce custom definitions (median 3), 39 use Mathlib only.
Q5. Failure modes and rejections. As discussed in Section 4 (issue 2) and Table 2, a patch can pass
lake buildwithout being a valid proof, e.g., by leavingsorry, adding axioms, changing the statement, or exploiting kernel bugs. Our protocol rules these out by design: Comparator and nanoda reject any such patch, which then counts as a failure, so no reported resolution relies on them. The rejected patches we observed were incomplete proofs that still containedsorry.We hope these answers merit a higher score.