acceptodds
Under review as a conference paper at ICLR 2027

Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs

Abstract

The Ramanujan Challenge put AI-assisted mathematics proofs to the test, inviting the community to prove previously unpublished formulae involving mathematical constants using AI. The number, mathematical level, and diversity of submissions presented a substantial challenge: assessing the correctness of complex AI-assisted proofs at scale. Treating the Challenge as a research-level math benchmark, we introduce an AI-assisted pipeline to measure submission correctness, solution similarity, and the success of different AI models, Computer Algebra Systems (CAS), and formal tools in mathematical proofs. We introduce an AI-assisted evaluation pipeline for diverse mathematical evidence, including natural language proofs, Computer Algebra code, and Lean formalizations. Beyond simple execution or compilation, it evaluates semantic and logical correctness by verifying assumptions, bounds, and completeness of statements for Computer Algebra and Lean code. Comparing to a one shot prompting judgment setup, we show a reduction in false positive verdicts (31.25% to 3.75%) for natural language proofs, and an improvement in judgment accuracy (37.8% to 56.8%) for supporting code. Using this pipeline, we find that participants - some without research-level math experience - were able to generate correct submissions for every problem, including solutions conceptually distinct from the official proofs, and solve one of the published open conjectures. Together, the benchmark and evaluation framework provide a systematic methodology for assessing AI-assisted research mathematics at scale.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Large Language Models, Symbolic Mathematics, CAS, Tool Use, AI for Math, Benchmark, Reasoning, Mathematical Capabilities, Evaluation, Formal Proofs, Proof Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HKVnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes submissions to the Ramanujan Challenge on research-level number theory problems and proposes an LLM-assisted grading workflow for natural-language proofs, computer algebra system (CAS) code, and Lean formalizations. The pipeline runs submitted code to log basic execution info, prompts LLM agents to summarize the proof text and code separately, compares the two summaries with another LLM call, and passes the output to a human grader. On the challenge submissions, it lowers false-positive Correct labels on natural-language proofs while keeping overall human agreement at 60.0% (Table 1a), and reaches 56.8% agreement with human grades on the 74 submissions containing CAS or Lean code (Table 2).

Critical strengths
  1. The target problem is practical: grading research-level mathematical submissions where arguments span natural-language text, symbolic CAS scripts, and Lean formalizations.
  2. The evaluation is grounded in real Ramanujan Challenge submissions and documents how participants combine LLMs, CAS tools, and formal provers on open number theory problems.
Critical weaknesses
  1. The pipeline is a plain LLM summarization wrapper, and its core cross-format step is never isolated. Running code to collect execution output, prompting an LLM to summarize the proof and code separately, and prompting another LLM to compare the summaries is standard workflow engineering rather than a new verification mechanism. Crucially, the paper never tests what errors the cross-format comparison catches beyond a single-pass evaluation of the full submission, or a multi-pass check of the proof or code alone under the same inference budget.

  2. The accuracy numbers reflect conservative abstention rather than better proof verification. On natural-language proofs, overall agreement with human grades stays flat at 60.0% (Table 1a), indicating that the drop in false-positive Correct labels comes from shifting predictions toward Partial or Incorrect rather than identifying valid proofs, and the paper reports no confusion matrices or class-wise recall for Correct. On CAS and Lean submissions, the pipeline never predicts Correct at all (Section 7.3); in Table 2, 41 of the 74 code submissions are human-labeled Partial, so a trivial classifier that outputs Partial on every item gets 41/74 = 55.4% agreement, virtually tied with the pipeline's 56.8%.

  3. The baselines are not matched for compute, and the human reference grades are neither independent nor LLM-free. The pipeline uses multiple LLM summarization and comparison calls plus code execution, yet it is not compared against inference-budget-matched baselines or a simple conservative-grading prompt. Moreover, each reference grade comes from a single human reviewer whose grading was itself assisted by Codex (Section 10), with no independent multi-expert adjudication on disputed items.

Questions for authors

See weaknesses.

2 comments

To address various weaknesses raised by the reviewers, we ran the pipeline with three models, Claude Haiku 5.5, Claude Sonnet 5.5, and GPT-6 Luna, each giving verdicts based on its own summary files (Subsections 5.1 and 7.3). These runs included budget-matching between verdicts via the pipeline and without it. The old Opus 5 verdicts were dropped from the analysis because they were not budget-matched and because the summaries used by them were seen by human reviewers.

Following are answers regarding the listed weaknesses.

  1. The pipeline is more than a summarization wrapper: every script is executed and every Lean project built, and these results feed the summaries and verdicts (Step 1, Subsection 5.1). To our knowledge, no previous work evaluates research-level proofs this way (Section 2). The comparison with a grader without the pipeline is now budget-matched (Subsection 7.3): that grader is an agent with the same total cost cap, which can run the submission itself. We could not isolate the summary-comparison step within the discussion period, and this is now listed as a limitation in Section 8.

  2. We agree regarding natural-language proofs, and the paper now explicitly states this (in the abstract and Subsection 7.3). With the updated data, agreement goes down for all models, and there is a small reduction in false positives. For code, however, agreement rises from 54.2% to 63.3%. This is now compared explicitly in Table 1(a) to the 54.2% of the “Partial” predictor, with all 3 graders achieving higher agreement than this baseline, and the pipeline now grades most code judged Correct by human reviewers as Correct. Confusion matrices are included in Appendix D.

  3. As described in the first answer and in Subsection 7.3, the comparison is now budget-matched. The Opus 5 verdicts had no cost cap and we could not rerun them within the discussion period. They were replaced by verdicts from three different models, each budget-matches with its own summaries, which the human reviewers never saw (Subsection 5.1). Human reviewers used the AI summaries and Codex to cross-check, not to replace, their own full review; separate reviewers evaluated different parts and discussed verdicts for submissions with Lean code (Step 4, Subsection 5.1). A conservative-grading prompt baseline was not run. For natural-language proofs, we agree the pipeline acts as a strictness control, as the paper now states. For code, however, it also accepts more submissions that human reviewers judged Correct (76% to 81%), which stricter grading alone would not explain.

Thank you for the detailed response and the substantial revisions. The budget-matched comparison, the confusion matrices (App. D), the released prompts (App. C), and the explicit majority-class baseline make the evaluation considerably more transparent, and I appreciate that the abstract and Section 7.3 now describe the natural-language results candidly.

However, my main concerns remain:

  1. The cross-format comparison step, which is the core of the proposed pipeline, is still not isolated, so it remains unclear which errors it catches beyond code execution and a budget-matched agent.
  2. On natural-language proofs, the revised results show no improvement: agreement decreases for all three models, the false-positive reduction is small (7.1% to 4.9%), and more correct proofs are rejected (28.4% to 35.8%). As the authors now acknowledge, the pipeline mainly acts as a strictness control.
  3. On code, the increase from 54.2% to 63.3% is encouraging, but it is reported without confidence intervals or a paired significance test, and no conservative-grading prompt baseline is included. The reference labels still come from a single reviewer per submission who used AI summaries from the same model families and Codex assistance, without independent adjudication. Please also clarify the number of code submissions in the revised Table 1(a), since the majority-class baseline changed from 55.4% to 54.2%.

Given these points, I will keep my overall rating, although the revisions improve the soundness of the evaluation.

Reviewer_6fxLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper discusses the results of the Ramanujan Project, a public competition of research problems in math. The primary contributions are a set of unsolved (or privately solved) problems across math, an LLM-based automatic grader for submissions, and an analysis of the human-AI grading workflow that followed. They use a combination of GPT 5.6 Sol and Claude Opus 5 in the AI-powered grading part of the evaluation, which is then given to a human who provides the final grade (with further assistance from Codex). They find that when Claude is given a GPT-written summary of a proposed solution along with the solution versus just the solution only, Claude gives fewer false positives and rejects more proofs that the human grader considered correct.

Critical strengths
  • The ten problems are good research problems. 8 of them are solved but the authors withhold the proofs, so they're not public. The other two are conjectures

  • One open conjecture received a complete Lean proof that was verified

  • The dataset of submissions is quite large/comprehensive, as there are 228 submissions, 4.8k pages of proofs, and ~1m lines of code (Python and Lean)

  • They find that the false "correct" grades of proofs fall from 25/80 to 3/80 when the summaries are included

Critical weaknesses
  • Agreement with the human grader stays at 60% with and without the pipeline. The gain seems to be a stricter judge but not necessarily a more accurate one

  • The ground-truths were human grades submitted by only one reader, which could be quite variable or noisy. And it's unclear what the Codex assistance means—e.g., why were they given Codex, what model was used, what did Codex have access to?

  • The AI component of this research seems minimal. It's not an evaluation of LLMs but instead is a minor analysis of how Claude evaluates proposed solutions given different pieces of information from GPT 5.6 Sol. The fact that only Claude and GPT were used is bizarre to me, with no ablations of other models.

  • The prompts used by the models are not given in the paper either

  • It seems like there were 33 entries that never specify what model combination was used for grading, and only 139 of the 228 submissions were graded.

Questions for authors
  • Did GPT acting as the summarizers, Claude as the grader, or the human grader have access to the organizers' official solutions? How were these official solutions created? I.e., did multiple human experts jointly produce them or just one expert? And how were the official solutions created for conjectures?

  • What were the exact prompts given to the models? Did the pipeline prompt add any instructions to be stricter?

  • Please elaborate on the comparison of "conceptually distinct from the official solutions" that's mentioned in the abstract. What does "conceptually distinct" mean and how do you define that?

  • Minor question, but what is nn in figure 7? Is it submissions or people?

  • Was there any follow-on review of any submission? It seems limited to have just one human expert review per submission.

  • Can you have a second mathematician grade a subset without seeing the models' judgements? If multiple humans get to a ~60% agreement rate, then perhaps the pipeline has already reached the agreement ceiling.

2 comments

To address various weaknesses raised by the reviewers, we ran the pipeline with three models, Claude Haiku 5.5, Claude Sonnet 5.5, and GPT-6 Luna, each giving verdicts based on its own summary files (Subsections 5.1 and 7.3). These runs included budget-matching between verdicts via the pipeline and without it. The old Opus 5 verdicts were dropped from the analysis because they were not budget-matched and because the summaries used by them were seen by human reviewers.

Following are answers to the listed questions.

  1. The organizers’ official solutions were available to the human graders but not to any LLM. These solutions were each produced by multiple experts among the authors of the challenge and were provided to the authors of this paper. No proofs of the conjectures are known.

  2. The prompts are now included in Appendix C (C.1 for summaries, C.2 for verdicts). The pipeline prompts did not add any instructions to be stricter.

  3. Discussion with the authors of the challenge questions highlighted that in their judgment, certain solutions included proofs significantly different from the official solutions, in that they demonstrate a new approach to the proof rather than rephrasing the same ideas of the official proof.

  4. The figure shows the share of submissions with Correct or Partial proofs and code, given each tool combination. The number of submissions using each tool combination is listed below it. This is now stated in the caption for the figure (which is now Figure 9).

  5. There was no follow-on review of any submission. However, separate human reviewers did evaluate and discuss different parts of the solutions, as now described in Step 4: Verdict of Subsection 5.1. This includes a separate expert reviewer for Conjecture 2 from that of the challenge questions, a Lean expert for all Lean code, discussion between the different reviewers regarding the verdicts for submissions with Lean code, and consultation with the authors of the questions as needed. The lack of separate independent reviews is acknowledged in Section 8: Limitations.

  6. Unfortunately, we do not have the ability to get reviews from a second mathematician, as manual evaluation of such proofs is very labor intensive. Human evaluation of the challenge questions took about two months, and a similar independent effort for the evaluation of a subset would require a significant amount of time and human effort. Given that the reference verdicts are the same with and without the pipeline, random noise in them limits absolute agreement in both conditions, but would tend to reduce rather than create a difference between them. Effects of the pipeline (such as the increase in agreement on code and decrease in false positives) are consistent across three different models, each using its own summaries, which the human reviewers never saw. The limitation of no independent human verdicts is acknowledged in Section 8: Limitations.

I appreciate the authors for making these clarifications. In particular it is helpful to know that the solutions were available to human graders but not LLMs, what the prompts to the LLMs were, and that the differences in solutions were sometimes a novel approach.

However, my fundamental concern still remains, which is that there does not seem to be much novelty or core involvement of AI in a way that is new to me. My understanding is that the models are summarizing solutions and making judgments/evaluations of the quality and novelty of these solutions that are expressed in Lean, natural language, or CAS. The results are somewhat expected to me, and the results don't feel particularly strong (not a major jump in agreement with human experts, and furthermore the finding that correct proofs get rejected more often). So although I appreciate the time in making these revisions, I don't feel that the overall results are strong enough to warrant me increasing my score.

Reviewer_vN6iOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies mathematical proofs submitted to the Ramanujan Challenge, which contains ten research-level number theory problems. The submissions include natural language proofs, computer algebra code, and Lean formal proofs. The authors develop an AI-assisted review pipeline that checks the written proofs and code separately, then compares their reasoning and conclusions to help experts judge correctness and completeness. In the experiments, the pipeline reduces the proportion of natural language proofs incorrectly accepted from about 31% to 4%. Agreement with human reviewers on code increases from about 38% to 57%. In addition, one open conjecture in the challenge receives a correct proof.

Critical strengths
  1. The setting has practical value. The paper studies real submissions to research-level mathematics problems, including natural language proofs, CAS code, and Lean proofs. This provides useful experience for evaluating AI-assisted mathematical research.

  2. The pipeline reduces incorrect acceptance. Among 80 natural language proofs, incorrect acceptances decrease from 25 to 3. Although overall agreement does not improve, this result supports using the pipeline to help experts find flawed proofs.

Critical weaknesses
  1. The results do not clearly show better overall evaluation. Agreement on natural language proofs stays at 60%. Code agreement reaches 56.8%, only slightly above always predicting “Partial” (55.4%), and no code is judged fully correct. These results suggest stricter grading, but do not sufficiently support reliable proof evaluation.

  2. The reference labels lack independent checking. Each submission is evaluated by only one human reviewer, using pipeline summaries and additional AI assistance. The reference labels may therefore share biases with the evaluated system, weakening confidence in the main results.

Questions for authors
  1. Result clarification: Can you provide full confusion matrices for both methods, showing how many correct proofs are wrongly rejected? For code evaluation, why does the pipeline accept no submission as fully correct and perform only slightly better than always predicting “Partial”?

  2. Review independence: Which AI outputs did human reviewers see before assigning reference labels? Were any submissions independently checked by experts without seeing these outputs? If yes, do these checks support the main conclusions?

2 comments

To address various weaknesses raised by the reviewers, we ran the pipeline with three models, Claude Haiku 5.5, Claude Sonnet 5.5, and GPT-6 Luna, each giving verdicts based on its own summary files (Subsections 5.1 and 7.3). These runs included budget-matching between verdicts via the pipeline and without it. The old Opus 5 verdicts were dropped from the analysis because they were not budget-matched and because the summaries used by them were seen by human reviewers.

Following are answers to the listed questions.

  1. Confusion matrices are now included in Figures 10-13 of Appendix D. For natural-language proofs, 28.4% of proofs judged Correct by human evaluation were wrongly rejected without the pipeline, with 35.8% with it, but for code there is a reduction from 24% to 19%. The pipeline was not instructed to not accept a submission as fully correct. The relevant prompts are in Appendix C.2. Moreover, with the new data from different models, it is no longer the case that no code is judged Correct with the pipeline. Comparison to a majority-class baseline predictor which in this case predicts only “Partial” is added to Table 1(a): given the new data, there is 63.3% agreement with the pipeline compared to 54.2% of this baseline predictor.

  2. Human reviewers saw AI summary files produced by GPT-5.6-Sol and Claude Opus 5, and utilized Codex as described in Step 4 of Subsection 5.1. As discussed at the start of Subsection 7.3, the new summaries used in the analyzed pipeline runs were not seen by human experts during their evaluation. Evaluation of Lean proofs did not rely on AI whatsoever. However, no independent AI-free evaluation of the CAS and natural-language proofs was carried out.

Thank you for the detailed rebuttal. Most of my concerns have been resolved, rasing the score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.