acceptodds
Under review as a conference paper at ICLR 2027

PACE: Research Proving Agent for Adaptive Conjecture Exploration

Abstract

Many mathematical research agents focus on solving problems with fixed statements. However, conjecture research, especially in applied mathematics, may require revising candidate questions or conclusions as investigation proceeds, coupling problem formulation with proof development. We present PACE, a research-proof agent that supports this process through a structured research loop that selects mathematical objectives and develops or challenges candidate mechanisms. The whole prover constructs proofs from these mechanisms, and an independent verifier checks them against the original target. Evidence-tracking memory stores assumptions and dependencies arising during research and marks evidence for revalidation when a candidate or condition changes. This memory supports task-specific context composition that keeps checked premises, exploratory material and unresolved obligations distinct. To evaluate proof construction and conjecture research, we introduce ResearchProofBench, an applied-mathematics benchmark focused on optimization, comprising 51 known-proof tasks with 331 hidden checkpoints under historical abstract access and six open-problem cases. The known-proof tasks assess reconstruction of published results, while the open-problem cases assess mathematical investigation and proof development. On the 51 known-proof tasks, PACE achieves an aggregate Proof Score of 75.16 and completes 20 proofs. In conjecture research, PACE develops conditional results, with its submissions receiving higher review scores than the compared baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mathematical research agents, Natural-language theorem proving, Conjecture research, Structured evidence memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4b1FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

PACE is an agent framework designed for mathematical theorem proving and conjecture exploration where problem statements, hypotheses, and proof strategies co-evolve. It decouples an intermediate result's mathematical validity from its applicability to a modified candidate target via an evidence-tracking memory graph and task-specific prompt composition. When a conjecture or assumption changes, affected dependencies are systematically reopened for revalidation without discarding valid lemmas. The authors also introduce ResearchProofBench, an applied mathematics and optimization benchmark consisting of 51 known-proof reconstruction tasks (with 331 hidden checkpoints under historical abstract access) and 6 open-problem conjecture cases. PACE achieves a 75.16 Proof Score and 39.22% complete proof rate on known-proof reconstruction, outperforming Iteris and QED, and obtains higher human/LLM review scores on the open-problem cases.

Critical strengths
  1. Principled formulation for evolving conjectures and proof obligations. Addressing the joint evolution of conjectures and proof mechanisms reflects mathematical research workflows much better than rigid, fixed-target theorem-proving setups. The dependency-tracking memory graph that reopens only invalidated application bridges rather than discarding valid lemmas is conceptually sound and directly addresses premise obsolescence during candidate revision.

  2. Comprehensive benchmark design in computational optimization. ResearchProofBench offers a multi-stage evaluation framework for applied mathematics, combining 331 hidden checkpoints on 51 known proofs under controlled historical abstract access with an open-ended 0–7 rubric that measures partial progress, conditional theorems, and counterexamples on open problems.

Critical weaknesses

While evaluating research-level, open-ended mathematical discovery is inherently challenging, the current evaluation leaves significant concerns regarding both mathematical soundness and empirical coverage:

  1. Absence of formal verification and compounding unreliability of natural language checks. Because PACE operates entirely in natural language without interactive theorem provers (e.g., Lean, Isabelle) or symbolic solvers, both internal proof acceptance and external benchmark evaluation rely on LLM reviewers (gpt-6-astra xhigh). Natural language mathematical reasoning is prone to subtle gaps, edge-case oversights, and circularity that LLM evaluators frequently miss. Without formal verification or rigorous human-LLM agreement studies on the 51 known-proof tasks (331 checkpoints), reported metrics like Complete Proof Rate (CPR) may conceal hallucinated or non-rigorous steps.

  2. Limited scope of open-problem validation and workflow ablations. Conjecture research is evaluated on only six problem cases, and the ablation study isolating individual components (such as revision-triggered revalidation and task-specific prompt composition) is conducted exclusively on a single case (C1). This narrow sample makes it difficult to establish whether the reported advantages generalize across diverse mathematical domains or reflect heuristics tuned to optimization problems.

Questions for authors
  1. Can the authors formalize and verify at least some of the proved theorems?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4b1F(1/2)

Thank you for the comments on proof reliability and experimental coverage. We describe the specific objects of formal verification, the existing independent-review evidence, and the additional cross-case ablation results.

W1 / Q1. Formal verification and the credibility of natural-language evaluation

We have verified the five known-proof tasks below and C1's accelerated upper bound under the strong/global assumption in Lean 4, covering six theorems and two corollaries. All proofs pass compilation and axiom checks, without using sorry or introducing additional axioms to bypass proofs.

Known-proof tasks. We conducted an AI-assisted comparison of the five original submissions with the formal code, covering 47 steps. We assign 2 to a retained step, 1 to partial correspondence, and 0 to a replaced or unmatched step. Averaging within each task and normalizing to 100 yields route similarity SrouteS_{\mathrm{route}}, which records how the submitted argument corresponds to the formal proof. The accompanying Lean derivations and targeted counterexamples check the specific mathematical obligations.

TaskNatural-language Proof ScoreRoute similarity
P21-T01100.00100.00
P26-T02100.0093.75
P23-T0282.0081.25
P16-T0252.3345.83
P23-T0352.0040.91

Lean checks the central arguments of both full-score submissions, with one changed decomposition and explicit boundary checks in P26-T02. It confirms P16-T02's threshold mismatch and supplies P23-T03's omitted root derivation using the existing nonzero-factor premise. Completing these two formal proofs also uses a replacement minorant and radial-certificate reasoning, respectively. For P23-T02, Lean validates the original Gram-matrix route, including zero radius; its component-score deductions concern registered scalar-factor steps bypassed by this valid alternative. These machine-checked derivations provide evidence for the retained arguments, boundary cases, and specific local grading findings.

Open problem C1. The human reviewer assigned 6/7 and requested further justification of the local Lipschitz/curvature estimate. Lean establishes this estimate from the preceding invariant and proximal optimality before invoking it in the energy recurrence, completing the local justification identified by the reviewer. Under the strong/global subgradient assumption, with R>0R>0, L1>0L_1>0, and Leff=L0+L1ℓ>0L_{\mathrm{eff}}=L_0+L_1\ell>0, the formalization establishes F(xN)−F∗≤2e1+L1RLeffR2(N+1)2,N≥1,F(x_N)-F^*\le\frac{2e^{1+L_1R}L_{\mathrm{eff}}R^2}{(N+1)^2},\qquad N\ge1, and verifies the gradient/proximal-call counts and accelerated complexity when L1RL_1R is bounded by an absolute constant. This upper bound and its two corollaries have route similarity 96.88, with 15 of the 16 comparison steps retained. The partially corresponding step derives the quadratic-model estimate through a gradient Lipschitz bound and derivative comparison in place of the integral Taylor remainder, preserving the verified dependency order.

The formalization package for the five known-proof tasks and C1 is publicly available in our anonymous repository. It contains the task-to-theorem mapping, frozen natural-language submissions, formal proofs, targeted deduction checks, and stepwise correspondences. Its reproduction script builds the proofs and audits their axiom dependencies under a pinned Lean/mathlib toolchain.

As reported in Appendix C (“Paired completion counts and expert re-examination”), the original evaluation included expert review of selected PACE–Iteris submissions. With method identities hidden, experts independently checked assumptions, derivations, conclusions, and the basis for credit and deductions against the original tasks and expert-endorsed criteria. No checked LLM evaluation was overturned and no Proof Score changed. The six calibration pairs in Appendix C also retain identical scores under equivalent transformations, including the gap in an incomplete argument.

AuthorsOpenReview ↗

Response to Reviewer 4b1F(2/2)

W2. Coverage of open problems and ablations

We extend A2–A5 to C2–C6 using the paper's task-specific generation budgets and 0–7 rubrics. A3–A5 retain the paper's configurations; the supplementary A2 runs use compressed fixed role prompts and simplified Director instructions.

CaseFull PACE (LLM/human)A2A3 (LLM)A4 (LLM)A5 (LLM)
C25/544†54†
C34/543†4†2
C46/655†63†
C53/432†2†2†
C67/764†4†2†

† Scored from the last mathematical argument saved before the budget cutoff.

Full PACE scores higher than A3 and A5 on all five tasks. It matches A4 on C2–C4 and scores 1 and 3 points higher on C5 and C6, respectively. On C4, A2's initialization still needs to satisfy the requirement for updates at every round. A5 lacks the argument connecting its aggregation-error lower bound to a stationarity lower bound for the optimization algorithm. Full PACE establishes an O(1/T)O(1/T) stationarity guarantee under the stated step-size and output conventions. On C6, A3 and A4 prove ρn≥1\rho_n\ge1, and A3 additionally provides a conditional construction. Full PACE and A2 both construct BFGS trajectories satisfying the smoothness and global exact-line-search conditions, proving attainability of the lower bound.

Thank you again for your constructive feedback. We hope that our responses have adequately addressed your concerns and would greatly appreciate your consideration of our clarifications in your overall assessment. We would be happy to address any further questions.

Reviewer_FYUvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses conjecture research in applied mathematics, mainly optimization. In this setting the problem statement, assumptions or target conclusion can change while the proof is being developed. The paper makes two contributions.

PACE (method). A multi-agent research–proof system with three main components:

  1. Evidence-tracking memory and task-specific prompt composition. A dependency graph keeps two questions apart: whether a result is valid, and whether it applies to the current candidate statement. It also separates checked premises, exploratory material and unresolved obligations. When a candidate statement or condition changes, the evidence that depends on it is marked for revalidation.
  2. Refined Conjecture Investigation (RCI). A sub-problem is a pair: a candidate statement and its supporting mechanism. Proof probes and refutation probes tell a failed intermediate lemma apart from a candidate that has been refuted. This decides the next step: repair the mechanism, revise the candidate, or split the sub-problem.
  3. Director-led loop with original-target review. A Director schedules the work but does no proving. The reasoning roles are Survey, RCI, Local Workbench and Whole Prover. A Scope Auditor and a Mathematical Reviewer check the final result against the original task.

ResearchProofBench (benchmark).

  • Known-proof: 51 tasks from 24 recent optimization and computational-mathematics papers, with 331 hidden checkpoints (993 authored claims). Evaluation uses a frozen clean-web protocol: an MCP broker allows only arXiv versions released before each paper's cutoff, with historical abstracts. Claims are graded 0–4 and scores propagate along dependencies. Reported metrics are Proof Score, Weighted Obligation Completion and Complete Proof Rate (CPR).
  • Conjecture research: six open problems (C1–C6; C2–C6 come from the OptBound collection). Each is scored on a 0–7 scheme by an independent LLM reviewer and separately by human experts.

Main results.

  • Known-proof: Proof Score 75.16 vs. 68.05 for the best baseline (Iteris). CPR is 39.22% (20/51) vs. 33.33% (17/51).
  • Conjecture research: PACE scores higher than direct and iterative solving on all six cases.
  • C6: a single infinite exact-line-search BFGS run attains Q-order one on a globally C^∞, strongly convex objective in every dimension n ≥ 2. This shows that the infimum established by Yuan (1984) is attained.
Critical strengths
  1. Targets a real gap: reusing evidence when the statement changes.

    • Most proof agents (QED, Aletheia, Iteris) assume a fixed target statement. Some allow revision, but none explicitly tracks whether earlier lemmas still apply under the revised hypotheses.
    • PACE separates a lemma's validity from its applicability to the revised candidate. This idea is clear and well motivated.
    • The C1 case (composite (L0, L1)-smooth optimization) and the C6 BFGS construction show that the mechanism produces real mathematical content, not only engineering scaffolding.
  2. Leakage-resistant and fine-grained evaluation.

    • Leakage control: the frozen clean-web MCP broker uses per-task private tokens, cutoff enforcement and audit logs. This takes data contamination and retrieval leakage seriously, a common weakness of LLM proof benchmarks.
    • Search ablation (Table 5): No-web 70.28, Frozen clean-web 75.16, Live full-web 80.82. This separates the value of historical literature from possible answer retrieval. CPR jumps to 56.86% under live web, which supports the need for the frozen protocol.
    • Scoring design: dependency-propagated checkpoint scores, explicit acceptance of valid alternative routes, and six frozen calibration pairs for representation invariance. Together these reduce the risk of template matching penalizing correct alternative proofs.
  3. Strong results at comparable cost, with an informative ablation.

    • Known-proof: PACE leads all six systems on all three metrics at a similar token budget to the agent baselines. Per task, PACE uses 5.32M tokens vs. 5.59M (Iteris), 4.59M (QED) and 4.44M (Rethlas) (Table 22).
    • Ablation (Table 4, C1): removing revision-triggered revalidation (A5) drops the score from 7 to 3 (LLM) and from 6 to 2 (human). This directly supports the paper's central claim.
  4. Thorough documentation. The appendices give task inventories, rubrics, review protocols, role responsibilities, token accounting and complete mathematical write-ups for the open cases.

Critical weaknesses
  1. Conjecture-research evidence is thin, and the baselines are weak.

    • Few cases: there are only six open problems, and five of them (C2–C6) come from one source (OptBound). That is narrow evidence for general conjecture-exploration ability.
    • Single-case ablation: all five ablations in Table 4 run on C1 only, the paper's running example, with one submission per configuration. There is no variance estimate and no check that the effects hold on other cases.
    • Asymmetric baselines: the known-proof comparison includes QED, Rethlas and Iteris. The conjecture comparison (Table 3) only uses Direct Solving and Generic Iterative Solving (four critique–revision rounds). Table 1 itself places Iteris and QED as the closest related systems, so leaving them out of the main open-problem comparison weakens the claim.
    • Compute mismatch: on C2–C6, Direct Solving uses 0.02–0.07M tokens, while PACE uses 5.5–11.6M (Table 22). The iterative baseline is roughly matched in tokens, but the direct baseline is not a meaningful comparison.
  2. Modest margin on known-proof theorem completion.

    • In the paired PACE–Iteris comparison, both complete 14 tasks, PACE alone completes 6 and Iteris alone completes 3. The net difference is 3 of 51 tasks (+5.88 pp CPR).
    • With a single run per system, a 6-vs-3 discordant split is unlikely to be statistically significant (an exact McNemar test gives p ≈ 0.5).
    • The Proof Score gap (+7 points) is more convincing. Still, the paper should report confidence intervals or results over multiple seeds.
  3. The formal notation hides that every check is an LLM judgment.

    • Sections 2–3 use logical notation: entailment, state transitions and a dependency graph. A reader may assume a symbolic or formal verification layer, such as Lean.
    • Table 19 and Appendix E make clear that the host only handles bookkeeping (versions, roles, sources). Applicability, valid inference, scope and correctness are all model judgments in natural language.
    • The internal reviewer and the generator use the same model family (gpt-5.6-sol), so correlated errors are a concern. The paper does not report how often the internal reviewer accepts a proof that the independent reviewer later rejects.
  4. Evaluation relies heavily on LLMs, with limited human calibration.

    • The 306 known-proof reviews are graded by gpt-6-astra. Expert re-examination is reported only for the PACE–Iteris paired submissions that were checked, and no agreement statistics are given.
    • Rubric drafted by the generator model: the conjecture-research scoring schemes are drafted by gpt-5.6-sol, the same model PACE uses for generation (Appendix D). The paper notes that no candidate outputs were seen when drafting, but this still risks bias toward that model's style of solutions.
  5. Reproducibility and generality.

    • All generation uses one proprietary model (gpt-5.6-sol xhigh). There is no experiment with other or open-weight models, so the gain from the scaffolding cannot be separated from the base model's strength.
    • Every configuration is run once, with no seeds and no pass@k.
    • Code and benchmark release plans should be stated explicitly.
  6. Presentation density.

    • The main text is abstract and uses many terms that are only defined in Appendices E–F (e.g., seed scout, frontier explorer, prompt composer). Some of these, such as seed scout and frontier explorer, are never introduced in the main text at all.
    • A short worked example in the main text would help readers follow the method. It should show how the prompt composer turns graph nodes into a concrete prompt, and how one revision triggers revalidation.
Questions for authors
  1. Agent baselines on open problems: why were Iteris and QED left out of the conjecture-research comparison (Table 3)? How do they perform on C1–C6 under a matched budget of about 5–18M tokens?
  2. Ablation coverage: can the most important ablations, A4 (proof critique and revision instead of RCI) and A5 (no revision-triggered revalidation), be run on C2–C6 or on the S3 known-proof subset? Can you report variance across repeated runs?
  3. LLM–human agreement: for known-proof grading, can you report agreement between gpt-6-astra and human experts on a stratified sample (e.g., Cohen's κ or rank correlation)? Please also describe how the expert-checked subset was selected.
  4. Internal-verifier error analysis: PACE does not complete 31 of the 51 known-proof tasks. How many of these stopped at budget without internal acceptance? How many were internally accepted but rejected by the independent reviewer?
  5. Statistical significance: please give confidence intervals (e.g., bootstrap over tasks) for Proof Score and CPR, and a paired test for the PACE–Iteris comparison.
  6. Rubric independence: how sensitive are the C1–C6 scores to a scoring scheme drafted by a different model or by human experts?
3 comments
AuthorsOpenReview ↗

Response to Reviewer FYUv(1/2)

Thank you for identifying the need for closer agent baselines on open problems and for the specific suggestions on proof evaluation and statistical uncertainty.

Q1 / W1. QED, Iteris, and computational budgets on open problems

We added QED and Iteris comparisons on C1–C6 under the same task-specific nominal generation budgets: 20, 8, 6, 12, 10, and 12 million tokens, respectively. The table scores the proofs retained at each task's original budget cutoff; continuation beyond that budget is reported separately below.

CasePACE (LLM/human)QED (LLM)Iteris (LLM)
C17/65—
C25/504
C34/5—5
C46/666
C53/413
C67/776

“—” denotes an unavailable original-budget score. After continuing the native workflow with additional budget, QED scored 5 and 3 on C2 and C3, respectively. On C1, Iteris obtained an accelerated upper bound conditional on the trajectory satisfying specified model conditions and scored 4.

On the comparable cases, PACE matches or exceeds QED within the original budgets, including 7 versus 5 on C1. The comparison with Iteris depends on the problem. On C3, Iteris gives a hypercube instance refuting the candidate eigengap bound and scores 5. Guided by the propagation lower bound, PACE uses the graph diameter and proves an accelerated upper bound under a model allowing tree routing, scoring 4. On C6, PACE, QED, and Iteris establish attainment.

Q2 / W1. Cross-case ablations and repeated runs

We extend A4 (Proof critique and revision) and A5 (Without the revision-triggered revalidation module) to C2–C6, retaining the paper's configurations and each task's original generation budget and 0–7 rubric.

CaseFull (LLM/human)A4 scoreA5 score
C25/554†
C34/54†2
C46/663†
C53/42†2†
C67/74†2†

† Scored from the last mathematical argument saved before the run stopped at the budget limit.

Full PACE scores higher than A5 on all five tasks. On C4, A5 establishes an aggregation-error lower bound without connecting it to an optimization stationarity lower bound; Full PACE proves an O(1/T)O(1/T) stationarity guarantee under explicit step-size and output conventions. Full PACE matches A4 on C2–C4 and scores 1 and 3 points higher on C5 and C6. On C6, A4 proves ρn≥1\rho_n\ge1, while Full PACE constructs a BFGS trajectory satisfying the smoothness and global exact-line-search conditions and attaining the bound.

Q3 / W4. LLM–human agreement on known-proof tasks

As reported in Appendix C (“Paired completion counts and expert re-examination”), the original evaluation included expert review of selected PACE–Iteris submissions. With method identities hidden, experts independently checked assumptions, derivations, conclusions, and the basis for credit and deductions against the original tasks and expert-endorsed criteria. No checked LLM evaluation was overturned and no Proof Score changed.

Q4 / W3. Natural-language verification and internal-reviewer error analysis

PACE returned submissions within the allotted budgets; 0 tasks stopped at budget exhaustion without internal acceptance. 31 internally accepted final submissions received less than full credit (Proof Score < 100) in independent evaluation. Generation and internal review use gpt-5.6-sol. Independent evaluation uses gpt-6-astra (xhigh) in a fresh context with method identities and internal acceptance labels hidden. The evaluator checks the original assumptions, actual derivations, and target conclusions against the expert-endorsed criteria, awarding credit for justified steps and identifying specific gaps. Valid equivalent formulations and alternative arguments receive credit according to their mathematical content.

AuthorsOpenReview ↗

Response to Reviewer FYUv(2/2)

Q5 / W2. Performance differences and statistical uncertainty on known-proof tasks

On the same 51 tasks, paired resampling over the 24 source-paper clusters gives PACE a mean Proof Score advantage of 7.12 points over Iteris, with a 95% confidence interval of [0.65, 13.57]. The table also reports paired resampling over tasks and equal weighting of source papers. All three procedures preserve the pairing between the two methods on the same task and use 50,000 bootstrap resamples. Differences are PACE minus Iteris. CPR (Complete Proof Rate) is calculated using the completion labels reported in the paper.

Statistical procedureProof Score difference (95% CI)CPR difference (percentage points, 95% CI)
Source-paper cluster resampling, equal task weights7.12 [0.65, 13.57]5.88 [-7.14, 18.37]
Paired task resampling7.12 [0.77, 13.34]5.88 [-5.88, 17.65]
Equal source-paper weights8.12 [2.84, 13.56]7.64 [-2.08, 17.36]

An exact two-sided paired test that jointly swaps the two method labels within each source paper gives p = 0.0367 for Proof Score. After leaving out each source paper in turn, the mean score difference remains positive in all 24 comparisons, ranging from 5.69 to 9.57 points.

Under the paper's reported completion labels, PACE completes 3 more tasks than Iteris (20/51 versus 17/51), with 6 tasks completed only by PACE and 3 only by Iteris. Source-paper cluster resampling gives a CPR difference of 5.88 percentage points, with a 95% confidence interval of [−7.14, 18.37]. Leaving out each source paper in turn gives differences of 4.08 to 10.64 percentage points. All three confidence intervals for the completion-rate difference include 0. The exact McNemar test and the paired source-paper label-swap test give p = 0.508 and p = 0.5625, respectively.

Q6 / W4. Independence of open-problem scoring criteria

The 0–7 rubrics for C1–C6 were drafted by the generation model, gpt-5.6-sol, from the original tasks and permitted background material, without access to the proofs to be evaluated. Domain experts reviewed and endorsed the rubrics against the original tasks.

The rubric assesses progress toward the original task by checking each submission's assumptions, quantifiers, and actual proof route. A conditional theorem can receive full credit when the original task permits that endpoint and its derivation is valid and complete.

Under these criteria, expert and LLM scores for PACE agree exactly on C2, C4, and C6 and differ by 1 point on C1, C3, and C5. Both evaluations rank PACE above Direct Solving and Iterative Solving on all six tasks.

W5. Models and reproducibility

PACE and all baselines use gpt-5.6-sol. Holding the generation model fixed provides a controlled comparison of proof workflows on the same tasks.

We will release the PACE code repository in the near future. ResearchProofBench task statements, experimental configurations, prompts, final scoring records, and offline table-generation scripts have been included in the supplementary material.

W6. Method presentation and concrete examples

We will introduce these roles at their first appearance in the main text: the seed scout proposes research starting points within the original task's scope; the frontier explorer develops candidate statements, refutation targets, and supporting arguments; and the prompt composer organizes task instructions from the selected objective and relevant mathematical records.

We will illustrate this with C1. The revised candidate requires a complexity bound uniform over all admissible instances and dimensions. The prompt composer combines this target with the retained lemma Lˉρ=eL1ρLeff\bar L_\rho=e^{L_1\rho}L_{\mathrm{eff}} and its outstanding applicability conditions to formulate the next task: prove that query points, proximal outputs, and connecting model segments lie in the controlled region, and identify conditions ensuring L1ρ=O(1)L_1\rho=O(1).

Thank you again for your constructive feedback. We hope that our responses have adequately addressed your concerns and would greatly appreciate your consideration of our clarifications in your overall assessment. We would be happy to address any further questions.

Reviewer_BjbBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PACE, an agent framework for mathematical proof construction and conjecture exploration. The system combines adaptive research planning, candidate refinement, evidence-tracking memory, and task-specific context composition. Its central design distinguishes the validity of a mathematical result from its applicability to a revised objective: changes to assumptions or conclusions trigger renewed checks of affected dependencies, while unresolved obligations guide subsequent research.

The paper also introduces ResearchProofBench, containing 51 known-proof tasks from 24 papers, with 331 checkpoints and 993 atomic grading claims, alongside six open-problem cases. PACE achieves the highest reported aggregate scores on the known-proof benchmark and completes 20 proofs, compared with 17 for Iteris at similar average token consumption. The open-problem evaluation reports both LLM and human assessments, while component ablations and recorded investigations examine the proposed workflow.

Critical strengths

A useful benchmark with supporting component analyses. ResearchProofBench provides a valuable evaluation setting for research-level mathematical reasoning, combining 51 known-proof tasks with six open-problem cases and explicit criteria for assessing mathematical progress. The accompanying workflow ablations and detailed C1 case study offer additional insight into the roles and behavior of individual components.

Critical weaknesses
  1. The central methodological contribution is insufficiently established. The paper emphasizes adaptive conjecture exploration, but does not clearly establish its distinction from the flexible reasoning and revision already available to existing agent workflows. Most quantitative evidence concerns fixed-statement proof tasks.

  2. Benchmark construction and item-level validation need clearer documentation. The respective contributions of humans and models to the 51 task statements and 993 atomic grading claims are not sufficiently specified. Although experts reviewed the scoring criteria, the extent of validation of individual statements remains unclear.

3.The main proof evaluation lacks sufficiently quantified human calibration. Known-proof results rely on a single LLM review per submission, while the reported expert re-examination does not specify its sample size, selection procedure, or agreement statistics.

Questions for authors
  1. What distinguishes PACE’s adaptive conjecture exploration from ordinary proof search and iterative revision? Could the authors evaluate candidate refinement across multiple open problems, and show which revisions lead to substantive mathematical progress?

  2. How were the task statements and atomic grading claims produced, and which stages involved expert validation? An independent audit of a representative sample—for example, 100 atomic claims across multiple tasks—would help quantify construction errors and their effect on the reported results.

  3. Could the authors report blinded expert assessments of final submissions across methods, including both accepted and rejected proofs, together with score agreement and false-acceptance/false-rejection rates?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BjbB(1/2)

Thank you for recognizing the value of ResearchProofBench and for your suggestions on methodological clarity, benchmark construction, and expert validation.

W1 / Q1. Adaptive conjecture exploration versus ordinary proof search

We systematically define refined conjecture investigation (RCI) as a procedure in which the model formulates and revises a candidate statement separately from its supporting claims. PACE constrains this procedure through distinct support and coverage reviews, followed by proof and refutation probes that receive the same assumptions and conclusion without the proposed argument. The reviews check both whether the support is justified without circular reliance on the candidate and whether it establishes the full conclusion. Their findings guide explicit transitions: repair the supporting argument, revise the candidate, or split it into subproblems within the original task's scope. A failed supporting lemma leaves the candidate open; a counterexample challenges the candidate only if it satisfies its assumptions and contradicts its conclusion. Candidate revision clears the associated support and coverage review records, requiring affected lemma applications to be checked again. Established lemmas remain valid under their original conditions, and missing applicability conditions become subsequent proof tasks. The distinction from ordinary iterative revision is this explicit specification of what is revised, which evidence supports that decision, and which checks must be repeated.

The 51 known-proof tasks from 24 computational mathematics papers require research-level proof construction, with retrieval restricted to abstracts published before the target result became public. Even with a specified target theorem, agents must propose and revise intermediate claims and establish how they support the original objective. This approximates research using the literature available before the result. The six open problems C1–C6 additionally permit revision of final claims within each task's scope; C1 and C3 illustrate the resulting mathematical progress.

C1 retains the acceleration target but revises its supporting argument. In the positive-parameter branch under the strong/global subgradient assumption, PACE specifies that the same algorithm and bound must cover all admissible instances and dimensions. A counterexample refutes the proposed global curvature estimate but leaves the local estimate valid. PACE proves that query points, proximal outputs, and model segments stay in a controlled region, yielding e1+L1R(L0+L1ℓ)e^{1+L_1R}(L_0+L_1\ell). For L1RL_1R bounded by an absolute constant, this gives the required accelerated leading term. Here, RR bounds the initial distance to a minimizer and ℓ\ell globally bounds the composite term's subgradient norms.

C3 revises the candidate complexity bound, which requires every node to achieve prescribed accuracy relative to the initial objective gap with network factor 1/γ1/\sqrt{\gamma}, where γ\gamma is the normalized eigengap. On the mm-dimensional hypercube, two instances differ only at a remote node. Before that information arrives, the opposite node cannot distinguish them, yet no common output meets both accuracy guarantees. Fixed relative accuracy and unit communication latency thus require Ω(m)\Omega(m) time, contradicting the candidate's O(m)O(\sqrt m) bound. PACE retains acceleration using the global condition number but makes propagation cost explicit through the graph diameter DGD_G. PACE proves a diameter-dependent accelerated upper bound using edge-specific neighbor messages and a shared routing tree, recovering the candidate's eigengap dependence when DG=O(1/γ)D_G=O(1/\sqrt{\gamma}).

We compare PACE with QED and Iteris, which also supports conjecture revision, and extend A4/A5 using the paper's configurations. On C1, PACE, A4, and A5 score 7/6, 5/3, and 3/2 (LLM/human), respectively. A4 must derive its backtracking-count bound from the task assumptions. A5's evaluated submission establishes an upper bound in the positive-parameter, strong/global subgradient branch. The table reports C2–C6 scores out of 7 within the original task budgets.

CaseFull PACE (LLM/human)QED (LLM)Iteris (LLM)A4 (LLM)A5 (LLM)
C25/50454†
C34/5—54†2
C46/66663†
C53/4132†2†
C67/7764†2†

† Scored from the last mathematical argument saved before the budget cutoff. “—” denotes an unavailable original-budget score. With additional budget to continue the native workflow, QED scored 5 and 3 on C2 and C3, respectively.

Full PACE receives higher LLM scores than A5 on all five tasks. A4 matches the full system on C2–C4 and scores 1 and 3 points lower on C5 and C6, respectively. On C6, A4 proves ρn≥1\rho_n\ge 1, while Full PACE constructs a BFGS trajectory satisfying the smoothness and global exact-line-search conditions and attaining the bound.

AuthorsOpenReview ↗

Response to Reviewer BjbB(2/2)

W2 / Q2. Construction and validation of tasks and atomic scoring items

We derived the task statements from the source theorems, preserving their assumptions, quantifiers, parameter ranges, and conclusions. We decomposed the reference proofs into 331 checkpoints, each containing two supporting claims and one central claim, yielding 993 atomic scoring items with explicit premises and dependencies.

An LLM drafts equivalent formulations of these claims, which domain experts check against the source proofs. An independent audit of sampled items checked statement accuracy, premises, dependencies, and preservation of the source theorems' assumptions and objectives in the associated tasks. The experts endorsed the final LLM-drafted scoring criteria on this basis. Equivalent formulations receive credit for their mathematical content; completeness of alternative proofs is assessed against the original theorem and assumptions.

W3 / Q3. Human calibration of known-proof evaluation

As reported in Appendix C (“Paired completion counts and expert re-examination”), the original evaluation included expert review of selected PACE–Iteris submissions. With method identities hidden, experts independently checked assumptions, derivations, conclusions, and the basis for credit and deductions against the original tasks and expert-endorsed criteria. No checked LLM evaluation was overturned and no Proof Score changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.