acceptodds
Under review as a conference paper at ICLR 2027

CAM-Bench: A Benchmark for Computational and Applied Mathematics in Lean

Abstract

Formal theorem-proving benchmarks enable mechanically verifiable evaluation of mathematical reasoning in large language models. However, existing benchmarks mainly focus on Olympiad-style problems and algebraic domains, leaving computational and applied mathematics underrepresented. We introduce CAM-Bench, a Lean 4 theorem-proving benchmark of 500 designated sorry proof obligations. These problems cover optimization, numerical linear algebra, numerical analysis, and closely related topics in applied mathematics, and depend on locally introduced definitions, notation, algorithms, and elementary results. To construct CAM-Bench, we develop a dependency-recovery pipeline that reconstructs the local textbook context needed to state each problem faithfully. It then normalizes each problem into a standalone informal theorem and translates it into a Lean target. We validate the resulting formal problems through Lean compilation and semantic review, checking Lean well-formedness and semantic alignment with the original exercises. For each problem, we release the raw exercise, recovered context, normalized informal theorem, and final Lean target. CAM-Bench complements existing formal mathematics benchmarks by targeting applied mathematics problems that rely on textbook concepts and elementary theorems, many of which are not directly available as standard Mathlib4 lemmas. We evaluate large language models and formalization agents on CAM-Bench and analyze common failure modes in tracking local assumptions, applying elementary results, decomposing proofs, and maintaining long-horizon control in Lean.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Benchmark, Evaluation, Applied Mathematics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qHktOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CAM-Bench, a Lean 4 benchmark comprising 500 proof obligations in computational and applied mathematics. It proposes a pipeline to recover textbook context and formalize exercises into machine-checkable Lean targets, and evaluates several LLM-based theorem provers, analyzing their performance and failure modes.

Critical strengths

This paper makes a useful contribution to Lean 4 infrastructure by providing a curated benchmark for computational and applied mathematics, which could support future training and evaluation of AI theorem provers. The benchmark construction pipeline is carefully designed, and the experiments provide a clear and systematic evaluation of model performance and failure modes.

Critical weaknesses

A key weakness is the limited novelty relative to existing work, (for example, https://arxiv.org/abs/2602.01291) which overlaps substantially with CAM-Bench in its focus on applied mathematics, including optimization and numerical algebra. The paper does not sufficiently clarify or empirically establish how its contributions differ from this closely related benchmark, which weakens the perceived novelty and significance of the work.

Questions for authors
  1. How does CAM-Bench differ from closely related benchmarks such as AMBER in terms of task design, problem coverage, and construction methodology? Can the authors provide a more explicit comparison to clarify the novel contributions?

  2. Since benchmark construction involves LLM-assisted formalization and semantic review, how do the authors ensure that the final Lean statements faithfully represent the original textbook exercises? Was the validation independently performed by human experts? How much does the recovered textbook context contribute to proof success? Would providing the complete context, or removing parts of the recovered context, significantly change model performance?

  3. The paper evaluates different proving configurations under a fixed token budget. How sensitive are the results to the choice of budget and proof-search strategy, and how should the reported success rates be interpreted across different systems?

1 comment
AuthorsOpenReview ↗

Response to Reviewer qHkt

W1/Q1. Relationship to AMBER.

AMBER has 118 theorem-proving problems and 82 construction-verification problems. The latter ask the model to define a solution, an algorithmic update or a reformulated problem and then prove it correct. CAM-Bench provides fixed theorem statements and local declarations. In our CLI evaluation, the model must complete all proof placeholders in each file. This format overlaps with AMBER's theorem-proving subset.

The construction procedures differ as well. AMBER describes expert-led formalization and dependency analysis of elaborated Lean declarations. Our pipeline recovers the textbook context before translation. It follows references recursively, restores the required definitions and assumptions, and separates the material into blocks with different mathematical roles. Translation and repair operate on the current block while earlier accepted declarations stay fixed. We retain the original exercise, recovered context, normalized statement and Lean target so that readers can trace how each task was constructed.

Both benchmarks cover optimization, convexity and matrix reasoning. AMBER also includes high-dimensional probability. CAM-Bench concentrates on optimization, numerical linear algebra and numerical analysis.

Q2. Semantic validation and recovered context.

We audited all 500 benchmark files. Each file was checked by one undergraduate reviewer with Lean experience and relevant mathematical background, alongside a GPT-5.5 Pro review under a fixed protocol. Reviewers compared the source exercise, recovered context and final Lean declarations, checking definitions, variables, assumptions, quantifiers and conclusions. Uncertain or problematic cases received a final joint review.

Before correction, 24 files (4.8%) contained confirmed semantic errors. We corrected and rechecked all confirmed issues. For example, a convex-set separation statement was missing the assumptions that both sets are nonempty. We added these hypotheses and revalidated the target.

Recovering textbook context and giving the prover extra explanation are different steps. An exercise's references to earlier definitions, equations or algorithms must be resolved to obtain a faithful target. Once that target and its required Lean declarations are fixed, the accompanying informal text can be varied without changing the formal task. Comparing a shorter explanation with the recovered context or a broader textbook excerpt would measure the effect of that guidance. Our experiments use the supplied context throughout, so they do not isolate the effect of varying the informal text.

Q3. Budget and proof-search strategy.

The reported full-suite protocol sets a cumulative 2M-token ceiling, with up to 32 independent candidates for Codex and Claude and up to 40 feedback-driven turns for the agentic configuration. These procedures spend the budget differently. Independent sampling generates separate candidates, while the agentic procedure uses compiler diagnostics, theorem search and repair to guide later attempts. Actual token use depends on when a run succeeds or reaches a stopping condition.

The archived CLI selection contains 500 files with 560 active proof placeholders. Scoring is at the file level. All placeholders in a file must be completed for credit.

M2F and Aristotle are compared separately on the same 200-instance sample, with an eight-hour limit per instance. Their reported success rates are 71.5% and 62.5%. Figure 6 shows cumulative success over time, including how many tasks finish by earlier cutoffs. The service APIs do not expose the token accounting needed to match these runs to a 2M-token budget. The eight-hour limit is the common resource control available for this comparison.

Reviewer_qAjGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CAM-Bench, a Lean 4 benchmark containing 500 proof obligations in computational and applied mathematics, primarily covering optimization, numerical linear algebra, and numerical analysis.

The authors propose a dependency-aware construction pipeline that recovers textbook context, normalizes exercises, and translates them into Lean through semantic decomposition, compiler-guided repair, and semantic review.

Experiments evaluate direct and agentic theorem-proving approaches. Codex achieves 48.4% verified accuracy, Claude 39.2%, and agentic Codex 30.2% on the full benchmark. On 200 sampled problems, M2F and Aristotle achieve 71.5% and 62.5%, respectively. The paper also analyzes common proof-generation failures.

Critical strengths
  1. CAM-Bench addresses an underexplored area of formal mathematical reasoning. Unlike existing benchmarks focused on Olympiad problems or isolated theorem statements, it evaluates applied mathematics involving contextual dependencies, optimization formulations, and algorithmic constructions. This provides a useful complement to existing evaluations.

  2. The dependency-recovery and modularization pipeline preserves intermediate representations, improving transparency and facilitating error diagnosis. The evaluation includes multiple language models and interactive theorem-proving systems, demonstrating that the benchmark presents meaningful challenges.

Critical weaknesses
  1. Lean compilation with sorry verifies that statements are well-formed but does not establish their mathematical correctness or equivalence to the original exercises. Although the paper employs semantic review, its quantitative validation primarily relies on LLM judgments without a systematic independent human audit.

Some appendix examples also raise concerns. Appendix F.9 claims a storage reduction when replacing Sk,YkS_k,Y_k with Sk,QkS_k,Q_k, although both representations appear to have identical dimensions. Appendix F.10 uses MonotoneMap for gradient monotonicity, whose mathematical interpretation requires clarification.

Without stronger validation, potential formalization errors could undermine benchmark reliability.

  1. The paper does not sufficiently isolate the effects of dependency recovery, semantic decomposition, and compiler-guided repair. In particular, comparing proof success with and without recovered informal context would help establish whether the benchmark specifically measures contextual reasoning.

Furthermore, different evaluation budgets and protocols limit direct comparisons between baseline models and external theorem-proving systems. The reported failure categories also provide limited quantitative evidence about the underlying causes of difficulty.

Questions for authors
  1. Have the authors independently audited a random subset of the 500 Lean statements? Can they estimate the proportion that faithfully represents mathematically valid source problems?

  2. Could the authors clarify the storage-reduction claim in Appendix F.9 and the MonotoneMap formulation in Appendix F.10? Are these formulations present in the final benchmark?

  3. Appendix G.6 allows other sorry placeholders to remain in the surrounding context. How does the verifier ensure successful proofs do not depend on unproved auxiliary declarations?

  4. Can the authors report performance with and without recovered context, and analyze success rates by dependency depth or complexity?

1 comment
AuthorsOpenReview ↗

Response to Reviewer qAjG

W1/Q1. Human validation and semantic fidelity.

We completed a human semantic audit of all 500 benchmark files. Each file was assigned to one undergraduate reviewer with Lean experience and relevant mathematical background, alongside a GPT-5.5 Pro review under a fixed protocol. Reviewers compared the source exercise, recovered context and Lean declarations, checking definitions, variables, assumptions, quantifiers and conclusions. Uncertain or problematic cases received a final joint review.

Before correction, 476/500 files (95.2%) were judged faithful and 24/500 (4.8%) contained confirmed semantic errors. All confirmed issues were corrected and rechecked. For example, missing nonemptiness hypotheses in a convex-set separation statement were restored. These are human assessments of source fidelity, separate from formal proof verification.

Q2. The appendix examples.

CAM-136's archived theorem lower_triangular_entry_recovered_from_Qk establishes the entry-reconstruction identity for LkL_k. Its verified content is this identity. The storage reduction requires the compact SR1 representation Bk=B0+QkKk−1Qk⊤B_k=B_0+Q_kK_k^{-1}Q_k^\top, retaining Qk=Yk−B0SkQ_k=Y_k-B_0S_k and the p×pp\times p coefficient matrix KkK_k. With fixed symmetric B0B_0 and well-defined updates, the coefficients can be recorded as each new pair arrives, so historical Sk,YkS_k,Y_k need not be retained. Vector storage falls from 2np2np to npnp, with O(p2)O(p^2) additional storage. Keeping both SkS_k and QkQ_k still requires 2np2np entries; the reconstruction identity alone does not establish the storage bound.

JSON2LEAN-165 defines MonotoneMap by ∀x,y,  0≤⟨ψ(x)−ψ(y),x−y⟩\forall x,y,\;0\le\langle\psi(x)-\psi(y),x-y\rangle. The target therefore uses operator monotonicity, including the intended gradient inequality. Appendix F.10 records a failed proof invoking the invalid monotoneOn_gradient API. This is a proof/API failure; the local predicate has the intended mathematical meaning.

Q3. Proof placeholders and dependency checks.

Appendix G.6 shows a single-placeholder prompt. The archived Codex/Claude CLI scorer evaluates complete files: it reconstructs the original file from ordered proof replacements and requires all active placeholders to be filled. It preserves the original declarations and types and applies Lean's collectAxioms to every local declaration, including helpers. Its permitted axioms are propext, Classical.choice, Quot.sound and explicit axioms from the original input. Direct and transitive dependencies on sorryAx are rejected.

All 415 replayed successful CLI submissions, covering 298 distinct files across multiple runs, passed compilation, declaration and axiom checks. A completed proof citing an unfinished auxiliary was also rejected through its transitive sorryAx dependency. These checks establish validity of the archived proofs relative to the permitted input axioms.

The 415 submissions are not distinct benchmark instances or a reproduction of every reported score. This transitive-dependency audit covers the archived CLI path. The separate ProofBench agentic verifier currently checks submission text and compilation only.

W2/Q4. Construction, context and evaluation.

Dependency recovery specifies the mathematical task before formalization. A textbook reference to an algorithm or equation must be resolved to obtain a faithful target. Semantic decomposition then separates mathematical roles, and constrained repair operates on individual blocks. Removing required source dependencies can leave the task underspecified; that is a construction question, distinct from proof success on an unchanged target.

At proof time, the Lean target and required declarations remain fixed. Varying the accompanying informal explanation tests the benefit of additional guidance. The reported scores characterize proving with the supplied context; they do not separately estimate that benefit or performance by dependency depth.

Section 5 reports a cumulative 2M-token ceiling for the full-suite configurations, with up to 32 independent baseline candidates or 40 feedback-driven agentic turns. The archived CLI selection contains 500 files and 560 active proof placeholders, scored by complete-file success. M2F and Aristotle share a separate 200-instance sample and an eight-hour limit. Each comparison describes performance under its stated search procedure and resource control.

Table 1 also provides quantitative failure diagnostics. Library/infrastructure issues occur in 31/57 unsolved M2F instances and 38/75 Aristotle instances; proof-decomposition issues occur in 39/57 and 52/75. These labels can overlap. The Claude/Codex percentages summarize terminal runner statuses. The counts identify recurring obstacles in the observed runs, rather than causal effects of individual construction components.

Reviewer_8v5kOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CAM-Bench, a Lean 4 benchmark containing 500 proof obligations from computational and applied mathematics textbooks. It proposes an LLM-assisted pipeline that recovers contextual dependencies, constructs self-contained statements, and translates them into Lean using semantic decomposition, compiler-guided repair, and semantic review. The resulting benchmarks are checked for Lean well-formedness and semantic alignment with the original exercises. The authors evaluate several LLMs and theorem-proving agents on CAM-Bench and analyze their common failure modes.

Critical strengths
  • The benchmark extends Lean theorem-proving evaluation beyond competition mathematics to computational and applied mathematics, with 500 proof obligations.

  • The dataset preserves intermediate artifacts from textbook exercises to Lean statements, supporting inspection and reproducibility.

Critical weaknesses

Major Weaknesses

  • The construction pipeline primarily orchestrates existing LLM-based techniques for retrieval, translation, and repair. Its novelty beyond combining these components seems limited and its generalizability across different LLMs remains unclear. Also, more capable models might also reduce the need for such a complex pipeline.

  • Semantic fidelity relies heavily on LLM-based reviewers, without a human-expert audit. Lean compilation establishes well-formedness but not equivalence to the original exercises, which raises concerns about benchmark reliability.

  • The construction process filters out a substantial fraction of candidate exercises (45.7%), potentially favoring problems that are easier to formalize. The paper provides limited discussion of how this selection affects the benchmark's coverage, difficulty, and representativeness.

  • Important details are deferred to the appendices or remain unclear, e.g., implicit-condition recovery, semantic block decomposition, repair budgets, and how the frozen-context strategy handles errors in previously accepted blocks.

Minor

  • The paper does not convincingly explain why this benchmark matters beyond testing LLMs on textbook exercises.

  • Although the appendices provide examples of individual construction stages, the main paper lacks a representative end-to-end example connecting an original textbook exercise to its final Lean target.

Questions for authors
  • Q1: How do the semantic decomposition and frozen-context strategies differ substantively from existing LLM-based formalization and repair techniques?

  • Q2: How reliable is LLM-based semantic validation, given that Lean compilation does not guarantee semantic fidelity?

  • Q3: Given that only 54.3% of candidates become Lean-ready, how might this filtering affect benchmark representativeness?

  • Q4: If an earlier block contains a semantic error, does the frozen-context strategy allow revisiting it, or is the entire exercise discarded?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8v5k

W1/Q1. Semantic decomposition and repair.

Repairing an entire formalization can fix one error while changing a definition or assumption elsewhere. We split each exercise into definitions, theorem statements, algorithms and optimization problems, then review and repair these blocks one at a time. Each block is checked against its informal counterpart. A repair can change only the current block, and the result must pass compilation and semantic review before acceptance. For example, fixing a theorem cannot rewrite an earlier definition that has already been checked.

We used GPT-5.4 for construction, without an additional agentic harness. A stronger model may need fewer repairs, but the editing rules still prevent a local fix from changing earlier accepted content. The benchmark asks provers to work with definitions, assumptions and algorithms from the textbook and connect them to the results available in Mathlib.

W2/Q2. Human validation and semantic fidelity.

We have completed a human audit of all 500 benchmark files. Each file was checked by one undergraduate reviewer with Lean experience and relevant mathematical background, alongside a GPT-5.5 Pro review under a fixed protocol. Reviewers compared the original exercise, recovered context and Lean declarations, checking definitions, variables, assumptions, quantifiers and conclusions. Uncertain or problematic cases received a final joint review.

Before correction, 24 files (4.8%) contained confirmed semantic errors. We corrected and rechecked all confirmed issues. For example, a convex-set separation statement was missing the assumptions that both sets are nonempty. We added these hypotheses and revalidated the target.

W3/Q3. Filtering and representativeness.

The 54.3% figure comes from source review, before Lean translation. Screening checks whether the exercise is faithfully specified, its dependencies are complete and its proof objective is precise. Candidates that initially fail these checks can be repaired. Appendix B reports 410 immediate acceptances and another 431 after repair, raising the yield from 26.5% to 54.3%. More than half of the 841 usable sources were recovered through repair, with the intended assumptions, conclusion and proof objective preserved.

We also examined what the retained problems cover. Section 4.1 and Figure 3 report 69 targets in modeling and duality, 163 in convexity, 46 in spectral and matrix reasoning, 60 in applied modeling, 88 in algorithms and convergence, and 74 in optimality or constrained-quadratic reasoning. Each category has at least 46 targets, giving substantial coverage of all six kinds of reasoning, including algorithms and optimization problems that depend on textbook context.

The final CAM selection also uses prior Lean-verified proof runs as evidence of solvability. These problems still pose a substantial challenge: Section 5 reports full-suite success rates of 48.4% for Codex and 39.2% for Claude under the 2M-token ceiling. The coverage analysis and these results show that source screening leaves a broad range of mathematical tasks with considerable proving difficulty.

W4/Q4. Frozen context and semantic repair.

A block frozen after compilation can still be changed during semantic review. Algorithm 2 freezes compilable blocks during translation. Algorithm 3 then reviews them in order and can rewrite, recompile and re-review the current block.

Once a block has passed semantic review, later local repairs keep it fixed. The automatic loop does not reopen earlier accepted blocks. Our human audit also checked earlier definitions and the later statements that use them, and confirmed errors were corrected and rechecked.

The release retains only targets that pass source review, compilation and final Lean-level semantic review (Sections 3.2 and 4.2). Appendices A.3-A.4 explain how implicit conditions are recovered from the source material, and C.1 defines the block roles. Algorithms 2-3 give the compilation and semantic repair loops with their respective bounds R and S.

Thanks for the clarification. However, several concerns remain unaddressed.

First, for the construction pipeline, the response describes again how block-wise process works, but still does not show how it differs substantively from existing LLM-based formalization and repair techniques. There is no comparison with prior methods, and no ablation against whole-file repair to show the necessity of semantic block decomposition. In addition, the pipeline was tested only w/ GPT-5.x, so generalizability wrt different LLMs is unclear.

Second, the response does not explain why the benchmark matters beyond testing LLMs on txtbook exercises. Finally, response does not specify if the self-contained main text will include a representative end-to-end example connecting an original txtbook exercise to its final Lean target.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.