acceptodds
Under review as a conference paper at ICLR 2027

REUF2F: Benchmarking coding agents on formalized undergraduate research problems

Abstract

As AI systems begin to resolve open research problems, reliable verification becomes increasingly important. Formal mathematics provides a trusted framework for mathematical reasoning, enabling agents to produce machine-checkable proofs while conducting mathematical exploration. Existing formal mathematics benchmarks often focus on known results or a small set of prominent open problems, which can become difficult to evaluate reliably once solutions are discovered and widely disseminated. Existing evaluation protocols are also not always aligned with how modern coding agents operate. We therefore introduce REUF2F, a Lean 4 benchmark of 244 open problems drawn from Research Experiences for Undergraduates (REU), spanning entry-level open research problems across a broad range of mathematical subfields. We expect these less prominent problems to attract less concentrated attention and their eventual solutions to circulate less widely, reducing the risk of future solution leakage. We adopt a standard coding-agent evaluation setting, placing each problem and its logical negation into a Lean project in a Docker container. We also build Leanimum-agent on mini-SWE-agent to provide a unified harness for fair comparisons across models. Our evaluation finds that even the best-performing model resolves only  20% of the problems, leaving substantial room for improvement on formalized open research problems.

Primary area:
datasets and benchmarks
Keywords:
Formal Mathematics, AI for Mathematics, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wMvVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The reviewer uses "human" to denote my hand-written opinions in the review and the reviewer uses "Opus" to denote Opus 5.5. The reviewer uses the incognito mode so no data will be shared to Claude.

Human: This paper proposes a new benchmark about undergraduate-level research math problems, which could be less attended so avoid future leakage. Experiments show that the state-of-the-art models still are not doing well in this benchmark, where the improvement space could be large.

Critical strengths

Human:

  1. This design could be insightful for future benchmarking operations. And it seems that this is a real advance over Formal Conjectures.

  2. Experimental results cover a wide range of models, and it reveals that Astra cannot do well in those problems.

Critical weaknesses

Human:

  1. The authors should compare to a few agentic approaches beyond pure LLMs.

  2. Without human reviewers, the proposed Lean solutions may be flawed.

  3. The number of tasks is only 244, which isn't large. The abstract logic is not sound because even though the domain is rare, the leackage could still be large because the size of the tasks.

  4. Some sample problems should be provided to the reviewer as long as the Lean solutions. This is a standard practice for a benchmark paper.

Opus (verified by the reviewer):

  1. The resolved problems are not audited, and the refutation-heavy results raise doubts about formalization fidelity. This is the central issue for a benchmark of open problems. Thirty-eight of the 56 resolutions are refutations, and GPT-6 Astra alone refutes 15.6% of the benchmark. That rate is hard to square with the claim that every problem is a genuine, unresolved research conjecture vetted by experts.

  2. Dataset documentation is thin for a dataset paper. The benchmark is the main contribution, but the paper provides almost no descriptive statistics. Readers need: the number of source papers and programs, and their distribution over years and institutions; how many problems per paper; statement length; how many tasks introduce custom definitions versus pure Mathlib statements; dependence on specific Mathlib areas; the conjecture type (universal statements, existence, bounds, classifications).

The reviewer might raise his score if the rebuttal is convincing.

Questions for authors

From Opus:

For each of the 56 resolved problems, was the result a genuine resolution of the source conjecture, a misformalization artifact, or already known? How many refutations exploit generality or edge cases absent from the source?

Did agents use the Internet during runs? Can you rerun without network access, or report web-search usage in trajectories?

How many candidates were filtered at each stage of quality control, and what were the reasons? Can you report an independent expert audit of a random sample?

Can you provide dataset statistics (sources, years, custom definitions, statement types) and several worked examples in the main text?

What is the breakdown of failure modes, and how often did Comparator or nanoda reject patches that passed lake build?

6 comments

We thank the reviewer for the careful reading and for recognizing the design as an advance over Formal Conjectures.

Human W1. Agentic approaches. All results come from agents: every model runs in Leanimum-agent with filesystem access, shell commands, Mathlib search, and Lean feedback for up to 1,000 steps and 6 hours (Sections 4.2, 5.1). With general bash access, an agent can itself choose to use other tools, such as SMT solvers, neural provers, or neural tactics, as part of its policy. A common harness avoids confounding models with scaffolding (Section 4, issue 3). We welcome the community to use the dataset in other settings, e.g., with dedicated proving systems, but our benchmark targets the capability of models as coding agents.

Human W2. Human review. Every problem is reviewed by humans (Section 3.3). Formalizers on our team, including several active Mathlib contributors, formalize problems in their own areas. Independent checks with Codex and Claude Code then flag issues of origin, open status, centrality, and faithfulness, which go back to the formalizer for revision or replacement, and a Mathlib contributor gives a final review before inclusion. Every accepted solution is verified by Comparator with two kernels (Section 4.1).

Human W3. Size and leakage. 244 tasks is in line with widely used benchmarks: miniF2F (244), ProofNet (371), ProgramBench (200), and Terminal-Bench 2.0 (89). One pass over the 244 problems costs hundreds to thousands of US dollars per model and reasoning-effort setting. On leakage, all problems were open when collected, so no solution could be in the models' training data.

Human W4. Sample problems. All 244 tasks are available at https://anonymous.4open.science/r/reuF2F-021F.

Opus W1. Audit of resolved problems. Beyond the pipeline above, we ran independent agentic checks with Claude Fable 5.2 and GPT-6 Astra on each of the 56 resolved problems (including all 38 refutations), comparing each with its source paper. We also sent each solution back to the data contributor who formalized it. Neither check found a formalization problem (some contributor responses are still pending). Discrepancies among unresolved problems cannot be fully ruled out, but as discussed in Section 6, in the formal-to-formal setting a misformalized statement is still a precise question that requires a verified proof or refutation, so it remains a valid signal. On the share of refutations: we formalize each conjecture faithfully as posed in its source, without filtering by difficulty or expected truth value, so the benchmark reflects the actual difficulty of the collected problems. A genuinely open conjecture can be false, and refutations are more accessible to current agents (in Table 3, five of seven models only refute). Many of this year's AI contributions to well-known conjectures have likewise come through refutations, e.g., the disproof of the unit distance conjecture (Section 1).

Opus W2. Documentation. The full benchmark is at the link above, and we will open-source everything: the repository, the Leanimum-agent harness, the Harbor integration, and detailed per-problem metadata. Statistics are in Q4.

Q1. See Opus W1.

Q2. Internet use. Search is part of the general agentic capability we evaluate, so we consider Internet access reasonable. We checked all successful and failed trajectories and found no case of retrieving a solution online, so we regard these resolutions as new. Agents do search for formal resources, e.g., solutions reused Lean formalizations of the Borsuk–Ulam and Kneser theorems from GitHub. This is expected; submissions are still fully verified.

Q3. Quality control. Quality control is carried out by the formalizers and reviewers as described in Human W2. We do not collect information on candidates that were not selected, as it is not part of the benchmark we deliver.

Q4. Statistics. 244 distinct source papers (one per paper) from 64 programs (table below); 144 on arXiv; years 1979–2026, 85% from 2013 or later; 45 problems in each of five areas plus 19 in Others; median challenge file 57 lines (25–445); 205 problems introduce custom definitions (median 3), 39 use Mathlib only.

Program#
Minnesota Duluth REU34
Williams SMALL24
Minnesota Twin Cities REU16
Polymath Jr.14
MIT SPUR11
Yale SUMRY10
Other (58 programs)135

Q5. Failure modes and rejections. As discussed in Section 4 (issue 2) and Table 2, a patch can pass lake build without being a valid proof, e.g., by leaving sorry, adding axioms, changing the statement, or exploiting kernel bugs. Our protocol rules these out by design: Comparator and nanoda reject any such patch, which then counts as a failure, so no reported resolution relies on them. The rejected patches we observed were incomplete proofs that still contained sorry.

We hope these answers merit a higher score.

Thanks for the rebuttal. It is hard to convince the reviewer if the paper isn't revised significantly. And the reviewer's concerns are not addressed. For example, the reviewer questioned the human validation of the solutions, but the authors replied the human provided questions.

Thank you for the follow-up. We would like to clarify that the human validation of the solutions is addressed in our response to Opus W1 / Q1, and we apologize for not cross-referencing it under Human W2.

For each of the 56 resolved problems, we sent the accepted solution back to the human data contributor who formalized that problem, to validate it against the source paper. In addition, we ran independent agentic checks with Claude Fable 5.2 and GPT-6 Astra on all 56 solutions. Neither check found a formalization problem, and no human response so far has reported an issue.

We also note that every accepted solution is a complete Lean proof verified by Comparator with two independent kernels (Section 4.1), so its correctness with respect to the Lean statement does not rely on human judgment.

We hope this clarifies the point, and we would be glad to address any remaining concerns.

Reviewer_G9RSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper collects 244 conjectures from undergraduate research and formalizes them in Lean. Agents can prove or refute them, using a shared environment and independent proof checks. The strongest model reaches about 20% pass@1.

Critical strengths

It makes sense to let agents disprove conjectures as well as prove them. Using the same setup for every model and checking proofs independently are also good choices. These make the benchmark easier to use and the results easier to compare.

Critical weaknesses
  1. The formalizations go through several checks, but I still wanted to know what happened in the 56 solved cases, especially the 38 refutations. Were they all checked against the original conjectures? The paper does not report whether any successes came from differences in the Lean statements.
  2. The authors are upfront that the lower-leakage argument is qualitative. But the agents have internet access, and we do not know what the successful runs found online. That makes it hard to judge how much of the result is new problem solving.
Questions for authors
  1. Were all 56 solved cases checked against the original conjectures? Were any formalization errors found?
  2. Did successful runs retrieve existing solutions online?
2 comments

We thank the reviewer for the careful reading and for recognizing the two-way setup, the common harness, and the independent proof checks. We respond to the weaknesses and questions below.

W1 / Q1. Were the 56 solved cases checked against the original conjectures? Yes. First, every problem already goes through a rigorous quality-control pipeline before inclusion (Section 3.3): independent checks with Codex and Claude Code for faithfulness, revision by the formalizer, and a final review by a Mathlib contributor.

Second, to address this concern directly, we additionally audited the resolved problems, where misformalizations that make a problem easier are most likely to surface (Section 6). We ran independent agentic checks using Codex and Claude Code, on each of the 56 problems claimed as resolved, comparing the accepted proofs and the Lean statement with the source paper, and we also sent each solution back to the human data contributor who formalized the problem. Neither check found a formalization problem. (Not all contributors have responded yet, but no response so far has reported an issue.)

We cannot fully rule out formalization discrepancies among the unresolved problems. As discussed in Section 6, however, this is not critical: in the formal-to-formal setting, even a slightly misformalized statement is a precise and nontrivial mathematical question, and resolving it still requires a correct proof or refutation that passes our verification. Such problems therefore still provide a valid evaluation signal.

We will report the audit in the revision. An anonymized version of the benchmark is available at https://anonymous.4open.science/r/reuF2F-021F, and the benchmark will be fully open: we will release the complete benchmark repository, the Leanimum-agent harness, the Harbor integration, and detailed metadata for every problem (source program, year, source URL, and source label), so that anyone can check each formalization against its source.

W2 / Q2. Did successful runs retrieve existing solutions online? Agents had Internet access during the runs (Section 5.1). We checked all trajectories, both successful and failed, and found no case where an agent retrieved a solution to the target problem online. We therefore regard these resolutions as new solutions to the problems. Agents do search the web for formal resources. For example, GPT-6 Astra found a Lean formalization of the Borsuk–Ulam theorem on GitHub, cloned it, and reused part of it in its final submission, and another solution similarly used a formal proof of Kneser's theorem from GitHub. This is expected behavior, as reusing existing formal libraries is part of how a researcher works, and the submission is still verified in full by Comparator. We will report this analysis in the revision.

We thank the reviewer again for the constructive comments. We hope these answers address the reviewer's concerns, and that the reviewer will reconsider the evaluation.

The response addressed several of my concerns, so I raised my score to 3.

Reviewer_at53OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces REUF2F, a benchmark of 244 open problems drawn from undergraduate mathematics research, to evaluate coding agents’ mathematical reasoning capabilities. The problems are formalized in Lean 4, enabling machine-checkable proofs or refutations and reliable verification of submitted solutions. Under the tested evaluation settings, the best-performing model resolves approximately 20% of the problems, leaving substantial room for improvement on formalized open research problems.

Critical strengths
  1. The paper raises an important issue: the risk of solution leakage in existing mathematical benchmarks.
  2. It develops a unified evaluation protocol that provides a controlled environment for comparing models and guards against invalid submissions through independent proof verification.
Critical weaknesses
  1. The low resolution rates warrant further analysis. Can the authors distinguish failures in mathematical reasoning from difficulties in constructing Lean proofs? For example, can models produce valid natural-language proofs or refutations for problems they fail to resolve formally? Such an analysis would clarify whether the main bottleneck is mathematical problem solving or formal proof implementation.
  2. The data collection and selection process needs more detail. Could the authors provide statistics on the source papers, including their publication years and research programs, and explain how these papers were identified and selected? How was each conjecture checked to ensure that it remained unresolved ? These details are important for assessing the dataset’s coverage and the reproducibility of its construction.
  3. The claim of lower solution leakage risk lacks empirical support. The paper argues that conjectures from undergraduate research are less likely to suffer from future solution leakage, but provides no empirical comparison with existing benchmarks to substantiate this claim. An analysis of solution availability or retrievability would help assess whether the proposed problem source actually reduces leakage risk.
Questions for authors

Please refer to the weaknesses discussed above.

4 comments

We thank the reviewer for the careful reading. We respond to the weaknesses below.

W1. Mathematical reasoning vs. Lean implementation. We agree that this distinction is important and ran a supplementary analysis. Several newer frontier models were released after the submission. Because they resolve more problems, their remaining failures show more clearly where the bottleneck lies. Using the same harness and settings, we sampled 60 failed attempts, 20 each from Gemini 4 Argon (high), Claude Opus 5.5 (max), and GPT-6.1 Sol (max), stratified by subject area. Each trajectory was analyzed with GPT-6 Astra (xhigh) in Codex, with manual inspection of selected cases.

  • Mathematical gap (53/60). The attempt lacked a substantive argument or valid counterexample, e.g., a special case (k = 1) without the general argument, a weaker bound (n+1 instead of n), or a candidate counterexample missing a required hypothesis. These partial results already compiled in Lean, so the missing piece was mathematical.
  • Boundary cases (3/60), e.g., a credible geometric construction whose key inequality was checked in Lean, but whose remaining conditions were not established.
  • Ineffective search (4/60). Repeated no-progress checking or early stopping.

We found no clear case where a complete mathematical solution was blocked solely by Lean. While preliminary, this suggests that mathematical reasoning remains the main bottleneck. Formalization can matter for problems needing theory not yet in Mathlib, and we will discuss this as a limitation in Section 6.

Finding a route that can be formalized is also part of the research task: the most natural informal proof is often not the easiest to formalize, and strong agents often find alternative strategies. In return, the formal-to-formal setting gives an objectively verifiable signal, whereas grading informal proofs of open problems is uncertain and requires expert time.

W2. Data collection and selection. Collection. As described in Section 3.2, we maintain a shared list of REU programs and other undergraduate research initiatives, contributed by all data contributors, and collect papers and reports from these programs. We select a question only if it was first proposed in the REU paper and is central to its research contribution.

Checking that a conjecture remains open. We verify this for each question (Sections 3.2 and 3.3). Independent checks with Codex and Claude Code then assess, among other things, whether the problem remains unresolved, and a Mathlib contributor reviews every problem before inclusion.

Statistics. We will add these to the revision. The 244 problems come from 244 distinct papers (one per paper) across 64 programs; 144 are on arXiv, the rest are journal articles or program reports. Publication years range from 1979 to 2026, with 85% from 2013 or later.

Source program# Problems
Minnesota Duluth REU34
Williams College SMALL24
Minnesota Twin Cities REU16
Polymath Jr.14
MIT SPUR11
Yale SUMRY10
Oregon State REU9
Cornell REU8
Michigan REU7
Budapest Semesters in Mathematics7
Other (54 programs)104

An anonymized version of the benchmark is available at https://anonymous.4open.science/r/reuF2F-021F. The benchmark will be fully open: we will release the complete repository, the Leanimum-agent harness, the Harbor integration, and detailed per-problem metadata (program, year, source URL, source label).

W3. Empirical support for lower leakage risk. We agree that this claim is qualitative (Section 6), but it rests on the domain expertise of the mathematicians on our team. For example, Erdős problems, used in FrontierMath Erdős and Formal Conjectures (Table 1), attract clearly more attention: they are tracked on a dedicated website, and AI resolutions of them have been widely publicized. As a quantitative indication, on Semantic Scholar our source papers have a median of 3 and a mean of 6.3 citations, with 80.7% below 10. The papers in which the FrontierMath Erdős problems were first posed have a median of 89 and a mean of 194.7. Our conjectures therefore receive far less follow-up attention, supporting our expectation that their solutions will circulate less widely. We will add this analysis to the revision.

We thank the reviewer again for the constructive comments and hope the reviewer will reconsider the evaluation.

Reviewer_at53OpenReview ↗

Official Comment by Reviewer at53

Thank you for the response. The citation comparison suggests that the REU source papers receive less scholarly attention, but I remain unconvinced that this translates into a lower risk of solution leakage. Even when a source paper is rarely cited, the mathematical techniques and closely related results needed to solve its conjecture may be well represented in model training data.

Thank you for the thoughtful follow-up. This is a fair point, and we realize that we did not define leakage precisely enough in the paper. We clarify it below and will make it explicit in the revision.

What we mean by leakage. By solution leakage we mean that a solution to the specific benchmark problem becomes available to the model, either in its training data or, for an agent with Internet access, online at inference time, so that the agent can reuse an existing solution instead of solving the problem. Since all problems were open when collected, no solution could have been in the training data. The remaining risk is that a solution is published during the benchmark's lifetime and then found online by an agent at inference time. Citation counts are a natural proxy for this risk: they reflect how much attention and follow-up work a problem attracts, and hence how likely it is to be solved, published, and become retrievable within the benchmark's lifetime. We also checked all trajectories, both successful and failed, and found no case in which an agent retrieved an existing solution to its target problem (Section 5.2).

Techniques in the training data are expected, not leakage. The techniques and related results needed for our problems may well be represented in the training data of frontier models. We agree that this is likely, and we consider it normal and expected. We view it as distinct from leakage, and the same holds for other benchmarks. For example, the programming languages, libraries, and coding patterns in SWE-bench (Jimenez et al., 2024) and ProgramBench (Yang et al., 2026) are all in the training data. What these benchmarks guard against is access to the specific solution. SWE-Bench+ (Aleithan et al., 2024) makes this precise: it defines solution leakage as the fix appearing in the issue report or its comments. Similarly, each year's new AIME problems are widely used to evaluate mathematical reasoning, e.g., in MathArena (Dekoninck et al., 2026), even though the competition techniques they require are abundant in the training data. They are considered uncontaminated because the specific problems are new. For our problems, if an existing result were close enough to resolve a conjecture directly, the conjecture would no longer be open. We verify for each problem at inclusion that it remains unresolved (Sections 3.2 and 3.3). If a closely related result only provides useful techniques, applying them to a new open problem is exactly part of the capability we aim to evaluate.

We thank the reviewer for raising this point, which helps us state the scope of our leakage claim more precisely. We hope this clarification addresses the concern, and we would be grateful if the reviewer would reconsider the evaluation.

Reviewer_at53OpenReview ↗

Official Comment by Reviewer at53

Thank you for your response. My concern about solution leakage remains, so I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.