- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CAM-Bench, a Lean 4 benchmark comprising 500 proof obligations in computational and applied mathematics. It proposes a pipeline to recover textbook context and formalize exercises into machine-checkable Lean targets, and evaluates several LLM-based theorem provers, analyzing their performance and failure modes.
Critical strengths
This paper makes a useful contribution to Lean 4 infrastructure by providing a curated benchmark for computational and applied mathematics, which could support future training and evaluation of AI theorem provers. The benchmark construction pipeline is carefully designed, and the experiments provide a clear and systematic evaluation of model performance and failure modes.
Critical weaknesses
A key weakness is the limited novelty relative to existing work, (for example, https://arxiv.org/abs/2602.01291) which overlaps substantially with CAM-Bench in its focus on applied mathematics, including optimization and numerical algebra. The paper does not sufficiently clarify or empirically establish how its contributions differ from this closely related benchmark, which weakens the perceived novelty and significance of the work.
Questions for authors
-
How does CAM-Bench differ from closely related benchmarks such as AMBER in terms of task design, problem coverage, and construction methodology? Can the authors provide a more explicit comparison to clarify the novel contributions?
-
Since benchmark construction involves LLM-assisted formalization and semantic review, how do the authors ensure that the final Lean statements faithfully represent the original textbook exercises? Was the validation independently performed by human experts? How much does the recovered textbook context contribute to proof success? Would providing the complete context, or removing parts of the recovered context, significantly change model performance?
-
The paper evaluates different proving configurations under a fixed token budget. How sensitive are the results to the choice of budget and proof-search strategy, and how should the reported success rates be interpreted across different systems?
Response to Reviewer qHkt
W1/Q1. Relationship to AMBER.
AMBER has 118 theorem-proving problems and 82 construction-verification problems. The latter ask the model to define a solution, an algorithmic update or a reformulated problem and then prove it correct. CAM-Bench provides fixed theorem statements and local declarations. In our CLI evaluation, the model must complete all proof placeholders in each file. This format overlaps with AMBER's theorem-proving subset.
The construction procedures differ as well. AMBER describes expert-led formalization and dependency analysis of elaborated Lean declarations. Our pipeline recovers the textbook context before translation. It follows references recursively, restores the required definitions and assumptions, and separates the material into blocks with different mathematical roles. Translation and repair operate on the current block while earlier accepted declarations stay fixed. We retain the original exercise, recovered context, normalized statement and Lean target so that readers can trace how each task was constructed.
Both benchmarks cover optimization, convexity and matrix reasoning. AMBER also includes high-dimensional probability. CAM-Bench concentrates on optimization, numerical linear algebra and numerical analysis.
Q2. Semantic validation and recovered context.
We audited all 500 benchmark files. Each file was checked by one undergraduate reviewer with Lean experience and relevant mathematical background, alongside a GPT-5.5 Pro review under a fixed protocol. Reviewers compared the source exercise, recovered context and final Lean declarations, checking definitions, variables, assumptions, quantifiers and conclusions. Uncertain or problematic cases received a final joint review.
Before correction, 24 files (4.8%) contained confirmed semantic errors. We corrected and rechecked all confirmed issues. For example, a convex-set separation statement was missing the assumptions that both sets are nonempty. We added these hypotheses and revalidated the target.
Recovering textbook context and giving the prover extra explanation are different steps. An exercise's references to earlier definitions, equations or algorithms must be resolved to obtain a faithful target. Once that target and its required Lean declarations are fixed, the accompanying informal text can be varied without changing the formal task. Comparing a shorter explanation with the recovered context or a broader textbook excerpt would measure the effect of that guidance. Our experiments use the supplied context throughout, so they do not isolate the effect of varying the informal text.
Q3. Budget and proof-search strategy.
The reported full-suite protocol sets a cumulative 2M-token ceiling, with up to 32 independent candidates for Codex and Claude and up to 40 feedback-driven turns for the agentic configuration. These procedures spend the budget differently. Independent sampling generates separate candidates, while the agentic procedure uses compiler diagnostics, theorem search and repair to guide later attempts. Actual token use depends on when a run succeeds or reaches a stopping condition.
The archived CLI selection contains 500 files with 560 active proof placeholders. Scoring is at the file level. All placeholders in a file must be completed for credit.
M2F and Aristotle are compared separately on the same 200-instance sample, with an eight-hour limit per instance. Their reported success rates are 71.5% and 62.5%. Figure 6 shows cumulative success over time, including how many tasks finish by earlier cutoffs. The service APIs do not expose the token accounting needed to match these runs to a 2M-token budget. The eight-hour limit is the common resource control available for this comparison.