acceptodds
Under review as a conference paper at ICLR 2027

Learning to Prove, Not Just to Answer: Reinforcement Learning from Formal Verification for Natural-Language Logical Reasoning

29 views

Abstract

Large language models (LLMs) are increasingly deployed for natural-language logical reasoning, where the final answer is easy to check but the proof behind it is not. In natural-language logical reasoning, intermediate conclusion should follow from its premises, and the resulting derivation should support the final answer. Existing methods lack machine-checkable verification of intermediate conclusions and answer-supporting proof dependencies, so they may assign credit to invalid or answer-irrelevant steps. We propose Proof-R1, a RL framework from formal verification that trains LLMs to construct verifiable proofs for natural-language logical reasoning. Proof-R1 admits a generated conclusion into the verified proof state only when the corresponding reasoning action satisfie proof obligations through UNSAT-based machine-checkable formal verification. Proof-R1 also recovers the answer-supporting dependency closure to trace the proof structure of the final answer and align outcome credit with the proof dependencies. Experiments demonstrate that Proof-R1 improves answer accuracy across three logical reasoning benchmarks and four backbone models, and outperforms training-free agents and training-based methods in terms of reasoning-process verifiability.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Formal Verification, Reinforcement Learning, Large Language Model, Natural-Language Logical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_X4YTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Proof-R1 extends fine-grained credit assignment in reinforcement learning to natural language logical reasoning and introduces a reward allocation mechanism tailored to the formal logic. Specifically, MCFV evaluates the correctness of individual reasoning steps, while ASDC identifies redundant steps. Experimental results demonstrate that Proof-R1 improves both reasoning accuracy and the quality of intermediate reasoning steps across multiple datasets.

Critical strengths
  1. The proposed Proof-R1 is ingeniously designed.
  2. The experiments are well-structured and demonstrate Proof-R1's consistent superiority over the baselines.
Critical weaknesses
  1. My primary concern is that the authors do not seem to adequately acknowledge several contributions that were originally introduced in PRoSFI. In particular, Proof-R1's structured proof generation and MCFV builds upon PRoSFI's

    • Structured Formal Intermediates,
    • Structured Output Protocol,
    • Reward Design.

The current presentation does not sufficiently distinguish these inherited design elements from the novel contributions of Proof-R1. As a result, readers may be left with the impression that these shared components were originally developed as part of Proof-R1.

Overall, I believe that Proof-R1 introduces reasonable improvements over PRoSFI. My concern is primarily about the presentation of contributions rather than technical merit.

I would be willing to recommend acceptance if the authors could revise their contribution claims to more clearly distinguish the novel components from those inherited from prior work, particularly in the Introduction, Related Work, and Methodology sections.

Questions for authors
  1. Why does Proof-R1 adopt a PPO-style algorithm rather than a GRPO-style algorithm?

  2. How well do proprietary LLMs, such as GPT-5.6 and Claude Opus, perform when given the prompt designed to generate ata_t?

4 comments
AuthorsOpenReview ↗

Response to Reviewer X4YT (Part 1/2)

Dear Reviewer X4YT,

Many thanks for your valuable time and positive feedback! We are pleased that you appreciate the ingenious design of Proof-R1 and find our experiments well-structured. We also highly value your acknowledgment of the reasonable improvements over PRoSFI.

Following your suggestions, we have carefully revised the manuscript accordingly, with the changes highlighted in purple.


W1: The current presentation does not sufficiently distinguish these inherited design elements from the novel contributions of Proof-R1.

We thank the reviewer for the valuable comments and for acknowledging the reasonable improvements over PRoSFI. However, our contribution does not lie in the use of intermediate structures or the output format. It lies in using formal verification to determine which conclusions become trusted proof facts and which actions receive outcome credit during policy optimization.

Following your valuable advice, we have revised the Introduction, Related Work, and Methodology sections with the changes highlighted in purple. Specifically, these revisions are summarized as follows:

  • Introduction: We have made revisions in the Introduction to clearly distinguish our original contributions. First, we explicitly acknowledge that PRoSFI introduced RL post-training and formal verification feedback based on the SMT proof format. Second, we point out the limitation of this design: verification-as-reward alone cannot gate invalid intermediate conclusions, which therefore propagate as false dependencies to later steps. Finally, we clearly define the uniqueness of PROOF-R1: it lies not merely in structuring proof traces, but in introducing verification-aligned optimization: via MCFV and ASDC, formal verification serves as a substantive control over the proof-construction process ("determining which conclusions become trusted proof facts and which actions receive outcome credit"), serving as the core methodological basis that distinguishes our work from prior work.
  • Related Work: We have made corresponding revisions in the Related Work section to further clarify the foundation of prior work. First, in the "Natural-language logical reasoning task" part, we explicitly attribute the structured formal intermediates based on the SMT proof format to prior work [1-3], and clearly state that this structure makes intermediate conclusions explicitly checkable against their declared premises. Second, in the "Formal verification-augmented LLM reasoning" part, we specifically point out the limitation of existing paradigms such as PRoSFI: formal verification only scores the whole-trace, which allows invalid conclusions to be used by subsequent actions. Based on this, we clearly define the fundamental difference of PROOF-R1: while prior methods use formal verification merely as a reward, our method ensures that formal verification serves as a substantive control over the proof-construction process.
  • Methodology: In the Methodology, we explicitly state that the representation of ata_t follows SMT proof structures.

To clarify this issue, we provide further details on the following three aspects: Structured Formal Intermediates, Structured Output Protocol, and Reward Design.

  • Structured Formal Intermediates & Structured Output Protocol: The design of structured formal intermediates is derived from SMT proof structures in formal verification and is not our original contribution [1-3]. We utilize formal intermediates to satisfy the structural requirements of SMT solvers. We have revised the paper to avoid misunderstanding regarding our contributions.
  • Reward Design: Both Proof-R1 and PRoSFI adopt the reward values of {0, 0.1, 0.3, 1}, commonly used reward values in LLM reinforcement learning [4-6]. However, the semantic meanings of the rewards are different. We differentiate rewards across five dimensions (Schema, Semantics, Rules, Proof progress, and Correctness), whereas PRoSFI only uses Format and Correctness.

We detail the comparison between the Proof-R1 and PRoSFI below:

ComponentPRoSFIProof-R1
Structured Formal IntermediatesSMT proof structureSMT proof structure
Reward DesignFormat, CorrectnessSchema (Format), Semantics, Rules, Proof progress, Correctness
Formal-Verification Supervision GranularityScores the whole-traceAction-level machine-checkable formal verification
Answer-Supporting OptimizationIndiscriminate supervisionAnswer-supporting dependency closure
AuthorsOpenReview ↗

Response to Reviewer X4YT (Part 2/2)

Q1: Why does Proof-R1 adopt a PPO-style algorithm rather than a GRPO-style algorithm?

We would like to first clarify a terminology point: GRPO itself follows "PPO-style" clipping, but it estimates the advantage via group-relative normalization.

The reason we use "PPO-style" is that the standard GRPO advantage definition cannot fully capture Proof-R1: while ASDC follows a "GRPO-style" formulation, MCFV does not. The main difference in Proof-R1 is that its advantage function consists of two components ( Eq. 21 ):

Ak,t=λvAk,tv+λoAk,to=λv(Rk,tv−bρtk(u−1))⏟MCFV+λo(mk,tRko−μxσx+ϵ)⏟ASDC.\begin{aligned} A_{k,t} &= \lambda_v A^v_{k,t} + \lambda_o A^o_{k,t} \\\\ &= \lambda_v \underbrace{\left(R^v_{k,t}-b_{\rho_t^k}^{(u-1)}\right)}_{\text{MCFV}} + \lambda_o \underbrace{\left(m_{k,t}\frac{R_k^o-\mu_x}{\sigma_x+\epsilon}\right)}_{\text{ASDC}}. \end{aligned}

The first term is the action-verification advantage provided by MCFV without group reward normalization. Here, bρtk(u−1)b_{\rho_t^k}^{(u-1)} is estimated from the historical verification results of actions that adopt the same reasoning rule, and it performs rule-centering on the current verification signal rather than within-problem group reward normalization.

The second term for ASDC employs a group-normalized answer-outcome advantage ( GRPO-style ) and acts only on actions within the final-answer dependency closure through the ASDC closure mask mk,tm_{k,t}. This component retains the GRPO-style construction of group-relative advantages.

Therefore, the standard GRPO advantage definition cannot fully characterize Proof-R1. We use “PPO-style” to describe its clipped optimization objective while adopting the aforementioned hybrid action advantage. As shown in Appendix A, we explicitly define this clipping mechanism and the hybrid advantage to make the distinction clear.

Q2: How well do proprietary LLMs, such as GPT-5.6 and Claude Opus, perform when given the prompt designed to generate ata_t?

We thank the reviewer for the suggestion. We evaluated proprietary LLMs (gpt-5.6-luna, gpt-5.6-sol, and claude-opus-5-5) on the ProverQA-Hard test split used in our main experiments. The results are shown below.

ModelAvg@3 ↑Pass@3 ↑FVR ↑RVR ↑RGD ↓
Proof-R1 (Qwen3-8B)92.63 ± 1.1098.99 ± 0.7193.55 ± 0.3290.22 ± 0.260.3554 ± 0.002
gpt-5.6-luna (Medium)43.72 ± 0.2953.27 ± 3.5490.61 ± 0.0589.47 ± 0.221.0244 ± 0.007
gpt-5.6-sol (Medium)83.92 ± 0.5890.45 ± 2.0898.26 ± 0.0798.02 ± 0.100.5307 ± 0.015
gpt-5.6-sol (High)88.61 ± 0.6093.47 ± 1.7598.82 ± 0.0898.77 ± 0.130.4944 ± 0.011
claude-opus-5-599.50 ± 0.29100.00 ± 0.0099.97 ± 0.0299.69 ± 0.080.3973 ± 0.004

Proof-R1 achieves the lowest RGD (0.3554), outperforming GPT-5.6 and Claude Opus-5.5. This indicates that Proof-R1 generates proofs without excessive or redundant reasoning steps.

While Claude Opus-5.5 achieves high reasoning accuracy (Avg@3 and Pass@3), Proof-R1, built upon a small 8B open-source model, achieves the comparable performance to these proprietary LLMs.


We hope these revisions and our detailed clarifications help to clearly distinguish our novel contributions from PRoSFI. We appreciate your valuable suggestions, which have enhanced the clarity of our manuscript.

Thanks again for your constructive comments. We hope the revised version meets your expectations for acceptance!

Best regards, Authors of Paper 31925.

References:

[1] Thomas Bouton et al. veriT: An Open, Trustable and Efficient SMT-Solver. In Automated Deduction—CADE-22, volume 5663 of Lecture Notes in Computer Science, pp. 151–156. Springer, 2009.

[2] Frédéric Besson et al. A Flexible Proof Format for SMT: A Proposal. In Proceedings of the First International Workshop on Proof eXchange for Theorem Proving (PxTP), pp. 15–26, 2011.

[3] Hans-Jörg Schurr et al. Alethe: Towards a Generic SMT Proof Format (Extended Abstract). In Proceedings of the Seventh Workshop on Proof eXchange for Theorem Proving (PxTP), volume 336 of Electronic Proceedings in Theoretical Computer Science, pp. 49–54, 2021.

[4] Wachiravit Modecrua et al. Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration. arXiv preprint arXiv:2604.02869, 2026.

[5] Eric Spencer et al. Thiruvathukal, Konstantin Läufer, and Mohammed Abuhamad. TLA-Prover: Verifiable TLA+ Specification Synthesis via Preference-Optimized Low-Rank Adaptation. In Proceedings of the 21st International Conference on Software Technologies (ICSOFT 2026), pp. 627–636, 2026.

[6] Wenqiao Zhu et al. CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 5922–5937, 2025.

I appreciate the authors' efforts to clarify the contributions of Proof-R1 relative to PRoSFI. As indicated in my initial review, I have raised my score accordingly.

AuthorsOpenReview ↗

Thank you!

Dear Reviewer X4YT:

We sincerely thank you for raising your score. We are happy that our responses have convinced you. Thank you again for strongly supporting our paper!

Best regards,

Authors of Paper 31925

Reviewer_tfrZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PROOF-R1, a reinforcement learning framework based on formal verification for natural-language logical reasoning. It aims to shift the focus from final answers to intermediate steps, and from entire reasoning trajectories to individual reasoning actions. The framework consists of three main components: (1) MCFV, which determines which conclusions can become trusted proof facts; (2) ASDC, which determines which actions receive outcome-based credit during policy optimization; and (3) verification-aligned optimization, which combines the advantages obtained from the first two components. Experiments across multiple backbone models and datasets demonstrate strong performance in both answer accuracy and reasoning-process verifiability relative to the compared methods.

Critical strengths
  1. In contrast to much of the research in this area that emphasizes final-answer correctness, this paper focuses on intermediate reasoning steps. It considers both the correctness of individual reasoning actions and the contribution of intermediate conclusions to the final answer. This is a meaningful research direction with substantial value to the field.
  2. The SMT-based MCFV verifier does not simply replace the model by solving the problem directly. Instead, it participates in the reasoning process by assessing reasoning actions, and its feedback is combined with the advantages provided by ASDC for aligned optimization. This is a novel technical approach with promising algorithmic value.
  3. The ablation study is well designed, and its results are consistent with the intended roles of the components. The explanation for the reversal in the relative answer accuracy of the two ablated variants across backbone models is also reasonable.
  4. The paper provides extensive analyses of performance across backbone models and inference systems, training dynamics, and several aspects of the proposed mechanism. These complementary perspectives provide substantial evidence for the effectiveness of the method.
  5. The appendix contains extensive additional results across difficulty levels and backbone models. It also reports that the verification gains of PROOF-R1 are preserved across different solver backends. These supplementary experiments provide rich detail and broaden the empirical evaluation.
Critical weaknesses
  1. The method assumes that natural-language premises and candidate answers are already formalized, limiting its applicability to real-world reasoning scenarios.

  2. ASDC primarily relies on standard dependency-graph traversal and masking. Its novelty beyond existing structured credit assignment methods remains unclear.

  3. Improvements in formal verification metrics do not necessarily demonstrate stronger logical reasoning abilities, particularly on unseen rules and complex reasoning structures.

  4. The training rewards and evaluation metrics rely on similar formal verification mechanisms, potentially favoring models optimized for the specific verification framework.

  5. Training baselines use formalized inputs and structured proof outputs. It remains unclear whether formal-proof-based training outperforms conventional natural-language CoT fine-tuning or reinforcement learning.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Response to Reviewer tfrZ (Part 1/3)

Dear Reviewer tfrZ,

We sincerely thank you for your review, particularly for acknowledging the meaningful direction of focusing on intermediate reasoning steps, as well as the novel technical approach of integrating MCFV and ASDC.

In the following, we provide a point-by-point response to your comments. We have carefully revised the manuscript accordingly, with the changes highlighted in green.


W1: Natural-language premises and candidate answers are already formalized

The formalization of natural-language logical reasoning is well established, spanning from the FraCaS inference test suite [1] to more recent classic benchmarks such as FOLIO [2] and ProofWriter [3].

We acknowledge the limitation on applicability to real-world reasoning scenarios brought by the prerequisite “autoformalization” procedure, but “autoformalization” is not within the scope of our manuscript.

Within this scope, our task is not natural language autoformalization, but rather logical reasoning proof generation which aims to improve the logical correctness and verifiability of the reasoning process.

Related surveys also treat autoformalization and logical reasoning proof generation as distinct research tasks, as they have different task objectives and can be studied independently [2, 4].

[1] Robin Cooper et al. Using the Framework. FraCaS Deliverable D16, 1996.

[2] Simeng Han et al. FOLIO: Natural language reasoning with first-order logic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22017–22031. Association for Computational Linguistics, 2024.

[3] Oyvind Tafjord et al. ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3621–3634. Association for Computational Linguistics, 2021.

[4] Zhaoyu Li et al. A survey on deep learning for theorem proving. In First Conference on Language Modeling, 2024.

W2: ASDC primarily relies on standard dependency-graph traversal and masking.

It should be clarified that ASDC is tailored for verifiable logical reasoning rather than a general-purpose reinforcement learning algorithm. Accordingly, the novelty of ASDC does not lie in proposing a new graph algorithm, but rather in the following two aspects:

  • First, ASDC naturally fits the credit assignment problem in formal verification training, since a proof certificate is expected to provide a complete derivation basis for the target conclusion [5-6]. ASDC is inherently suited to this requirement: it recovers the proof dependency closure from the final answer and uses this closure to restrict the scope of outcome supervision.

  • Second, ASDC naturally addresses a practical issue specific to formal verification training, by explicitly distinguishing between "answer-supporting actions" and "unused reasoning branches" (line 101-102). During our empirical investigation, we observed a failure mode that motivated the ASDC design: models tend to generate reasoning steps that pass formal verification but are unused in the actual proof of the final answer. Our experimental results further confirm that ASDC leverages the answer dependency structure to determine which proof actions the outcome credit should be assigned to (Section 3.4; ablation in Table 2).

[5] Haniel Barbosa et al. Generating and Exploiting Automated Reasoning Proof Certificates. Communications of the ACM, 66(10):86–95, 2023.

[6] Hanna Lachnitt et al. Improving the SMT Proof Reconstruction Pipeline in Isabelle/HOL. In 16th International Conference on Interactive Theorem Proving (ITP 2025), volume 352 of Leibniz International Proceedings in Informatics, pp. 26:1–26:22. Schloss Dagstuhl–Leibniz-Zentrum für Informatik, 2025.

AuthorsOpenReview ↗

Response to Reviewer tfrZ (Part 2/3)

W3: Improvements in formal verification metrics do not necessarily demonstrate stronger logical reasoning abilities, particularly on unseen rules and complex reasoning structures.

Same view as you. Beyond formal verification metrics, our original paper also reports standard reasoning metrics (Avg@3 and Pass@3). The six metrics are relatively complete. Our original paper DOES experiments across unseen rules and complex reasoning structures.

  • Unseen Rules (Table 4, Appendix G): FOLIO is a human-annotated logic reasoning benchmark differs from the prover-generated pipeline of ProverQA, and it involves reasoning over a broader inference system with unseen rules [1].

  • Complex Reasoning Structures (Table 1, Table 6): In our original paper, we evaluated our method across different difficulty ("reasoning structures") subsets of ProverQA. To further clarify this, we analyzed the reasoning structures of the subsets (see table below). Specifically, ProverQA-Hard requires significantly deeper reasoning chains (5–6 steps) and integrates a much larger number of premises (9–12) and logical rules (3–4), making its reasoning structures substantially more complex.

SubsetsExperimentSizeDepthWidthRulesPremises
ProverQA-EasyTable 6[1, 2][1, 2][1, 1][1, 2][2, 4]
ProverQA-MediumTable 6[3, 5][3, 5][1, 2][2, 3][5, 8]
ProverQA-HardTable 1[6, 8][5, 6][2, 3][3, 4][9, 12]

We have added this analysis to Appendix C.0 (Table 5) of the revised manuscript with the changes highlighted in green.

W4: The training rewards and evaluation metrics rely on similar formal verification mechanisms, potentially favoring models optimized for the specific verification framework.

We thank the reviewer for this insightful observation. Actually, Our original paper DOES experiments examining the dependence on the verification framework across data sources, formalization protocols, and solver backends with "not similar" settings.

  • Data Sources (Table 4): We trained our models on the generated dataset ProverQA and evaluated them on the human-curated FOLIO [2], indicating our model's generalization capability to a different data source.

  • Formalization Protocols (Appendix E): Differing from the training setup, we conducted an independent semantic review of 2,000 local steps using GPT-5.5, achieving a weighted agreement rate of 99.73%. This serves as a supplementary cross-check for formalization protocols.

  • Solver Backends (Appendix E): Differing from the training setup of Z3, we recomputed FVR/RVR with cvc5, veriT, and Isabelle. The consistent results indicate that Proof-R1 is not tied to a specific solver-backend verification framework.

Additionally, we do understand that the "similar" you may be concerned about likely pertains to Training-Inference Consistency [7]. In fact, Using training feedback aligned with task correctness has clear precedents in verifiable reasoning research. For example, DeepSeek-R1 uses answer correctness to construct accuracy rewards [8], and DeepSeekMath designs its training signals and evaluation metrics around the same task objective [9].

[7] AgentFly Team. Training / Inference Consistency. AgentFly Documentation, 2026.

[8] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.

[9] Zhihong Shao et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.

AuthorsOpenReview ↗

Response to Reviewer tfrZ (Part 3/3)

W5: Training baselines use formalized inputs and structured proof outputs. It remains unclear whether formal-proof-based training outperforms conventional natural-language CoT fine-tuning or reinforcement learning.

We appreciate the suggestion and we agree that this could be an open question. The task and settings of dataset are built around structured formal verification. Structured reasoning is verifiable because it relies on formal verification representations that can be mechanically checked [10]. Our core focus is on verifiable reasoning based on structured formal intermediate. However, natural-language CoT fine-tuning or reinforcement learning cannot be directly formally verify, the quality of their reasoning processes cannot be assessed by formal metrics (FVR, RVR, RGD).

An unified experimental protocol is controlled for confounding variables and enable a fair comparison among training methods for structured proof generation (SFT, GRPO, PRoSFI).

[10] John Harrison. Formal proof—Theory and practice. Notices of the American Mathematical Society, 55:1395–1406, 2008.


We hope these detailed responses, the additional analyses on reasoning structures, and the clarifications help to clarify your concerns. Thanks again for your appreciation and constructive comments on our work!

Best regards, Authors of Paper 31925.

AuthorsOpenReview ↗

Follow-up of Rebuttal

Dear Reviewer tfrZ,

As the discussion period is about to close, we would like to kindly draw your attention to our response above and the updated manuscript (with revisions highlighted in green). In response to your reviews, we have clarified that our work focuses on verifiable proof generation rather than autoformalization, and explained that ASDC is tailored to filter out unused reasoning branches in formal verification training. We have also added an analysis of reasoning structures in Appendix C.0, suggesting that Proof-R1 maintains its performance on unseen rules (FOLIO) as well as more complex reasoning structures. Furthermore, cross-protocol validation using GPT-5.5 (achieving a 99.73% agreement rate) and re-evaluations across different verification backends (cvc5, veriT, and Isabelle) yield consistent results, suggesting that our performance gains do not depend on the specific verification framework.

Could you please let us know whether these additional analyses and explanations have addressed your concerns? If so, we would greatly appreciate it if you could consider updating your score.

If any concerns remain, we would be glad to clarify them further while the discussion period is still open.

Best regards,

Authors of Paper 31925.

Reviewer_EpudOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Proof-R1, a reinforcement-learning framework for generating structured logical proofs from problems supplied with natural-language premises and formal representations. Each generated action specifies dependencies, a conclusion, and an inference rule. A verifier checks local entailment, rule applicability, and proof progress, while an answer-rooted dependency closure determines which actions receive outcome credit. The resulting action-level advantages are used for policy optimization. Experiments on ProverQA and transfer evaluations on FOLIO and ProofWriter report improvements in answer accuracy and proof-step verification rates across several backbone models.

Critical strengths

S1: Verification feedback is evaluated beyond answer accuracy. The paper explicitly separates answer correctness from semantic entailment and rule verification, and reports improvements over answer-reward GRPO and a formal-verification-based RL baseline. The component ablations provide evidence that both verification feedback and dependency-based credit assignment contribute to performance.

S2: Substantial validation of local entailment judgments. Appendix E rechecks frozen outputs using alternative SMT solvers and native Isabelle proofs, supplemented by blinded semantic review. These checks strengthen confidence that the reported local-entailment improvements are not artifacts of a particular solver backend.

Critical weaknesses

W1: The evaluation does not fully establish the claimed improvement in answer-supporting proof construction. The central contribution concerns constructing valid proofs and assigning credit according to their answer-supporting dependencies. However, the main comparisons primarily report micro-averaged verification rates over individual steps, which do not directly establish how often an answer is accompanied by a complete valid proof. Figure 1 provides useful trace-level evidence, but does not include PRoSFI, the closest formal-verification-based RL baseline. Consequently, the paper demonstrates improvements in local verification more clearly than the advantage of the full method over the closest baseline in producing complete answer-supporting proofs.

W2: The comparative evidence for generalization is limited. The FOLIO and ProofWriter evaluations compare PROOF-R1 only with the backbone and SFT, omitting GRPO and PRoSFI. These results support transfer of the trained system, but do not establish whether PROOF-R1 offers stronger cross-dataset performance than existing reinforcement-learning approaches. This distinction is particularly important because the main training benchmark provides most of the comparative evidence, while the human-authored FOLIO evaluation contains only 51 problems.

Questions for authors

Q1: Does the implemented RuleCheck impose restrictions beyond Eq. 34 that require principal rule premises to appear explicitly among the dependencies? For example, would a single action deriving C from {A, A → B, B → C}, labeled IMPLICATION_ELIMINATION, be accepted without an explicit intermediate step deriving B? How should RVR be interpreted if such compressed deductions are accepted?

Q2: Figure 1 includes a trace-level verification statistic, whereas the main tables emphasize micro-averaged step rates. Can the authors report the proportion of outputs with a correct answer and a completely verified answer-supporting certificate across the main backbones and datasets, including malformed or missing certificates as failures?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Epud (Part 1/3)

Dear Reviewer Epud:

Many thanks for your feedback! We sincerely thank you for recognizing our separation of answer correctness from semantic entailment and rule verification, as well as the rigorous validation of local entailment judgments via alternative SMT solvers, native Isabelle proofs, and blinded semantic review.

Here we will carefully response to your concerns and questions in detail. Following your suggestions, we have revised the manuscript accordingly, with the changes highlighted in blue.


W1: The evaluation does not fully establish the claimed improvement in answer-supporting proof construction.

We thank the reviewer for the very helpful suggestions! We accept your suggestion and it will make our paper better. Following your suggestions, We have added the comparison with the PRoSFI method in Figure 1 and added statistics for the VPCR metric in Tables 1–4 and 6–9, with the changes highlighted in blue.

VPCR measures the output of methods with both a correct answer and a fully verified answer-supporting proof certificate. It reflects the model's ability to provide correct answers backed by complete and valid proofs. The formula is as follows:

VPCR=1NK∑i=1N∑k=1Kri,k1[vi,kcert].\mathrm{VPCR} = \frac{1}{NK}\sum_{i=1}^{N}\sum_{k=1}^{K} r_{i,k}\mathbf{1}\left[v_{i,k}^{\mathrm{cert}}\right].

By incorporating the VPCR metric, we draw the following findings from the experimental results:

Main Table (Table 1, Table 6): Proof-R1 achieves the highest VPCR in most settings, suggesting "the advantage of the full method over the closest baseline in producing complete answer-supporting proofs".

  • Qwen2.5-7B-Instruct on ProverQA (VPCR metric)
MethodOverallEasyMediumHard
Backbone2.30 ± 0.395.87 ± 0.970.15 ± 0.150.00 ± 0.00
LogicAgent7.78 ± 0.2917.88 ± 1.022.41 ± 0.300.17 ± 0.00
GoV4.41 ± 0.2911.11 ± 0.590.45 ± 0.260.00 ± 0.00
SFT2.94 ± 0.157.54 ± 0.510.15 ± 0.150.00 ± 0.00
GRPO1.81 ± 0.134.60 ± 0.440.15 ± 0.150.00 ± 0.00
PRoSFI5.58 ± 0.1513.41 ± 0.511.36 ± 0.150.00 ± 0.00
Proof-R113.80 ± 0.1534.55 ± 0.261.36 ± 0.450.50 ± 0.17
  • Qwen3-8B on ProverQA (VPCR metric)
MethodOverallEasyMediumHard
Backbone29.07 ± 0.7553.13 ± 0.8919.76 ± 1.967.87 ± 1.02
LogicAgent43.95 ± 0.3972.92 ± 1.0937.41 ± 1.5313.23 ± 1.17
GoV35.58 ± 0.6962.20 ± 0.7127.60 ± 1.889.55 ± 1.05
SFT31.38 ± 0.5157.60 ± 0.5621.27 ± 0.268.21 ± 0.89
GRPO48.95 ± 0.9579.44 ± 0.7842.99 ± 2.5715.58 ± 1.33
PRoSFI53.16 ± 0.6787.36 ± 0.6138.91 ± 1.5824.12 ± 1.24
Proof-R175.92 ± 0.6191.83 ± 1.0081.60 ± 0.8448.74 ± 1.16

Ablation (Table 2): We find that MCFV contributes more to VPCR. By constraining the model to generate verifiable steps, MCFV makes it easier to obtain complete proof certificates.

  • Ablation results on ProverQA-Hard (VPCR metric).
VariantQwen2.5-7B-InstructQwen3-8B
w/o MCFV0.178.21
w/o ASDC0.1734.51
Proof-R10.5048.74
AuthorsOpenReview ↗

Response to Reviewer Epud (Part 2/3)

Transferability (Table 3, Table 4, Table 7, Table 8): In both cross-backbone and cross-dataset experiments, Proof-R1 consistently attains high VPCR, indicating that "the advantage of the full method over the closest baseline in producing complete answer-supporting proofs" is transferable.

  • Results on ProverQA-Overall for additional model families (VPCR metric).
MethodLlama-3.1-8BGLM-Z1-9B-0414
Backbone0.539.10
SFT0.629.78
Proof-R15.6445.28
  • Results on ProverQA-Hard for additional model families (VPCR metric).
MethodLlama-3.1-8BGLM-Z1-9B-0414
Backbone0.006.16
SFT0.007.03
Proof-R10.0024.46
  • Cross-dataset generalization for Qwen3-8B (VPCR metric).
MethodFOLIO-ValidProofWriter
Backbone19.6123.11
SFT20.9221.78
Proof-R129.4161.11
  • Cross-dataset generalization for Qwen2.5-7B-Instruct (VPCR metric).
MethodFOLIO-ValidProofWriter
Backbone0.001.78
SFT0.002.22
Proof-R111.765.33

W2: The comparative evidence for generalization is limited.

We thank the reviewer for this suggestion! In Appendix C.4, we have added the performance of baselines on the FOLIO dataset, including GRPO and PRoSFI (Table 9). In addition, we have included FOLIO-Test to supplement the results on FOLIO-Valid (FOLIO in the pre-revision manuscript).

  • Results on FOLIO-Valid for Qwen3-8B
MethodAvg@3 ↑Pass@3 ↑FVR ↑RVR ↑VPCR ↑
Backbone88.8998.0464.4638.8719.61
GoV88.8996.0867.6244.4125.49
SFT85.6298.0463.2241.0320.92
GRPO88.8996.0873.5255.9625.49
PRoSFI88.8998.0473.1755.8227.45
Proof-R190.8598.0482.5859.7029.41
  • Results on FOLIO-Test for Qwen3-8B
MethodAvg@3 ↑Pass@3 ↑FVR ↑RVR ↑VPCR ↑
Backbone91.8497.9669.3445.7923.13
GoV87.76100.0065.3342.9024.49
SFT89.8095.9267.2044.6723.81
GRPO91.1697.9668.4848.2729.25
PRoSFI91.1695.9269.0442.3630.61
Proof-R191.8497.9678.1855.4834.69

On the FOLIO dataset, Proof-R1 trained on ProverQA achieves the highest formal verification metrics, together with competitive accuracy. This may serve as evidence that our method generalizes better than "existing reinforcement-learning approaches"(GRPO and PRoSFI).

AuthorsOpenReview ↗

Response to Reviewer Epud (Part 3/3)

Q1: Does the implemented RuleCheck impose restrictions beyond Eq. 34 that require principal rule premises to appear explicitly among the dependencies?

We thank the reviewer for this excellent question! The implemented RuleCheck does not exceed the constraints of Eq. 34. However, our implementation allows for simple merges of inferences under the same rule, as it is not strictly limited to single-step atomic inference. In this case, RVR still makes the correct judgment regarding the rules. The original intent is to ensure that the generated proofs are readable and verifiable, rather than enforcing strictly atomic inference.

In the example you provided, {A,A→B,B→C}⊢C\{A, A\to B, B\to C\} \vdash C, we select the implication elimination template B,B→C⊢CB, B\to C \vdash C. The declared dependencies can derive BB and B→CB\to C via IMPLICATION_ELIMINATION, thereby satisfying the premise entailment condition of the formula.

We provide another example: {A→B, B→C, ¬C}⊢¬A\{A\rightarrow B,\ B\rightarrow C,\ \neg C\}\vdash\neg A. For the declared rule MODUS_TOLLENS, we select the template A→B, ¬B⊢¬AA\rightarrow B,\ \neg B\vdash\neg A. The declared dependencies contain A→BA\rightarrow B and entail ¬B\neg B through B→CB\rightarrow C and ¬C\neg C, thereby satisfying the premise entailment condition of the formula.

To avoid misunderstanding, we have supplemented the description in Appendix F with the changes highlighted in blue.

Q2: Can the authors report the proportion of outputs with a correct answer and a completely verified answer-supporting certificate across the main backbones and datasets?

We thank the reviewer again for the very helpful suggestions! We accept your suggestion and it will make our paper better. We have made the corresponding revisions in Figure 1, Tables 1–4, and Tables 6–9.


We hope these revisions and the additional experiments help to clarify clarify your concerns. Thanks again for your appreciation and constructive comments on our work!

Best regards, Authors of Paper 31925.

Comments on the paper

2 comments
Area_Chair_w3opOpenReview ↗

Reminder: Please Respond to the Authors' Rebuttal

Dear Reviewer tfrZ and Epud,

Please take a moment to review and respond to the authors' rebuttal to facilitate further discussion, as the author-reviewer discussion period ends on November 18 at 11:59 PM AoE.

Best,
AC

Dear Area Chairs, Program Chairs, and Reviewers,

We thank all reviewers for their valuable feedback and helpful suggestions. We are pleased that all reviewers have acknowledged the contributions of our work and recognized its value. Furthermore, we are glad to see that the paper is considered to have the following strengths:

  • Meaningful research direction & substantial value (Reviewer tfrZ)
  • Ingeniously designed & novel technical approach with promising algorithmic value (Reviewer tfrZ, X4YT)
  • Soundness & informativeness (Reviewer tfrZ, Epud, X4YT)
  • Clear and easy to follow (Reviewer tfrZ)
  • Baseline superiority & strong performance across multiple datasets and backbone models (Reviewer Epud, tfrZ, X4YT)
  • Well-structured experiments (Reviewer X4YT)
  • Well-designed ablations & reasonable analyses (Reviewer Epud, tfrZ)
  • Substantial validation, broadened empirical evaluation & strengthened confidence (Reviewer Epud, tfrZ, X4YT)

We again thank all the reviewers for their comments, especially Reviewer X4YT and Epud for recognizing our novel contributions and raising the score accordingly. Here, we summarize our discussions with the reviewers during the discussion period.

  • Additional Experiments: Following the reviewers' suggestions, we introduced the complete proof verification rate (VPCR) metric (added to Tables 1–4 and 6–9), expanded cross-dataset generalization comparisons against baselines such as GRPO and PRoSFI on FOLIO (Table 9), and evaluated proprietary LLMs (GPT-5.6 and Claude Opus 5.5) on ProverQA-Hard. These additions were acknowledged by Reviewer Epud ("appreciate the additional experiments"), who raised the score accordingly.

  • Clarifications: In response to the reviewers' questions, we further clarified Proof-R1's task scope, technical details, and innovations over prior work. These clarifications were explicitly recognized by Reviewer X4YT ("clarify the contributions of Proof-R1"), who raised the score accordingly.

We responded to the individual comments of each reviewer. Additionally, we highlight the key updates of the revised version (Reviewer Epud: blue; Reviewer tfrZ: green; Reviewer X4YT: purple). The key updates are listed below:

  • In Figure 1 and Tables 1–4 and 6–9: We included VPCR metrics across the main tables, ablations, backbones, and cross-dataset evaluations.
  • In Introduction, Related Work, and Methodology sections: We refined the presentation to sharpen the positioning of Proof-R1 and highlight our core innovations.
  • In Appendices C.0, C.4, and F: We added reasoning structure complexity analyses (Table 5), extended FOLIO baseline evaluations (Table 9), and supplementary RuleCheck implementation details.

Finally, we thank all reviewers again for their thoughtful reviews and for spending time writing the reviews. Please let us know if you have any recommendations for further improving the paper.

Best regards,

Authors of Paper 31925.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.