- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper analyzes submissions to the Ramanujan Challenge on research-level number theory problems and proposes an LLM-assisted grading workflow for natural-language proofs, computer algebra system (CAS) code, and Lean formalizations. The pipeline runs submitted code to log basic execution info, prompts LLM agents to summarize the proof text and code separately, compares the two summaries with another LLM call, and passes the output to a human grader. On the challenge submissions, it lowers false-positive Correct labels on natural-language proofs while keeping overall human agreement at 60.0% (Table 1a), and reaches 56.8% agreement with human grades on the 74 submissions containing CAS or Lean code (Table 2).
Critical strengths
- The target problem is practical: grading research-level mathematical submissions where arguments span natural-language text, symbolic CAS scripts, and Lean formalizations.
- The evaluation is grounded in real Ramanujan Challenge submissions and documents how participants combine LLMs, CAS tools, and formal provers on open number theory problems.
Critical weaknesses
-
The pipeline is a plain LLM summarization wrapper, and its core cross-format step is never isolated. Running code to collect execution output, prompting an LLM to summarize the proof and code separately, and prompting another LLM to compare the summaries is standard workflow engineering rather than a new verification mechanism. Crucially, the paper never tests what errors the cross-format comparison catches beyond a single-pass evaluation of the full submission, or a multi-pass check of the proof or code alone under the same inference budget.
-
The accuracy numbers reflect conservative abstention rather than better proof verification. On natural-language proofs, overall agreement with human grades stays flat at 60.0% (Table 1a), indicating that the drop in false-positive Correct labels comes from shifting predictions toward Partial or Incorrect rather than identifying valid proofs, and the paper reports no confusion matrices or class-wise recall for Correct. On CAS and Lean submissions, the pipeline never predicts Correct at all (Section 7.3); in Table 2, 41 of the 74 code submissions are human-labeled Partial, so a trivial classifier that outputs Partial on every item gets 41/74 = 55.4% agreement, virtually tied with the pipeline's 56.8%.
-
The baselines are not matched for compute, and the human reference grades are neither independent nor LLM-free. The pipeline uses multiple LLM summarization and comparison calls plus code execution, yet it is not compared against inference-budget-matched baselines or a simple conservative-grading prompt. Moreover, each reference grade comes from a single human reviewer whose grading was itself assisted by Codex (Section 10), with no independent multi-expert adjudication on disputed items.
Questions for authors
See weaknesses.
2 comments
Thank you for the detailed response and the substantial revisions. The budget-matched comparison, the confusion matrices (App. D), the released prompts (App. C), and the explicit majority-class baseline make the evaluation considerably more transparent, and I appreciate that the abstract and Section 7.3 now describe the natural-language results candidly.
However, my main concerns remain:
- The cross-format comparison step, which is the core of the proposed pipeline, is still not isolated, so it remains unclear which errors it catches beyond code execution and a budget-matched agent.
- On natural-language proofs, the revised results show no improvement: agreement decreases for all three models, the false-positive reduction is small (7.1% to 4.9%), and more correct proofs are rejected (28.4% to 35.8%). As the authors now acknowledge, the pipeline mainly acts as a strictness control.
- On code, the increase from 54.2% to 63.3% is encouraging, but it is reported without confidence intervals or a paired significance test, and no conservative-grading prompt baseline is included. The reference labels still come from a single reviewer per submission who used AI summaries from the same model families and Codex assistance, without independent adjudication. Please also clarify the number of code submissions in the revised Table 1(a), since the majority-class baseline changed from 55.4% to 54.2%.
Given these points, I will keep my overall rating, although the revisions improve the soundness of the evaluation.
To address various weaknesses raised by the reviewers, we ran the pipeline with three models, Claude Haiku 5.5, Claude Sonnet 5.5, and GPT-6 Luna, each giving verdicts based on its own summary files (Subsections 5.1 and 7.3). These runs included budget-matching between verdicts via the pipeline and without it. The old Opus 5 verdicts were dropped from the analysis because they were not budget-matched and because the summaries used by them were seen by human reviewers.
Following are answers regarding the listed weaknesses.
The pipeline is more than a summarization wrapper: every script is executed and every Lean project built, and these results feed the summaries and verdicts (Step 1, Subsection 5.1). To our knowledge, no previous work evaluates research-level proofs this way (Section 2). The comparison with a grader without the pipeline is now budget-matched (Subsection 7.3): that grader is an agent with the same total cost cap, which can run the submission itself. We could not isolate the summary-comparison step within the discussion period, and this is now listed as a limitation in Section 8.
We agree regarding natural-language proofs, and the paper now explicitly states this (in the abstract and Subsection 7.3). With the updated data, agreement goes down for all models, and there is a small reduction in false positives. For code, however, agreement rises from 54.2% to 63.3%. This is now compared explicitly in Table 1(a) to the 54.2% of the “Partial” predictor, with all 3 graders achieving higher agreement than this baseline, and the pipeline now grades most code judged Correct by human reviewers as Correct. Confusion matrices are included in Appendix D.
As described in the first answer and in Subsection 7.3, the comparison is now budget-matched. The Opus 5 verdicts had no cost cap and we could not rerun them within the discussion period. They were replaced by verdicts from three different models, each budget-matches with its own summaries, which the human reviewers never saw (Subsection 5.1). Human reviewers used the AI summaries and Codex to cross-check, not to replace, their own full review; separate reviewers evaluated different parts and discussed verdicts for submissions with Lean code (Step 4, Subsection 5.1). A conservative-grading prompt baseline was not run. For natural-language proofs, we agree the pipeline acts as a strictness control, as the paper now states. For code, however, it also accepts more submissions that human reviewers judged Correct (76% to 81%), which stricter grading alone would not explain.