- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Proof-R1 extends fine-grained credit assignment in reinforcement learning to natural language logical reasoning and introduces a reward allocation mechanism tailored to the formal logic. Specifically, MCFV evaluates the correctness of individual reasoning steps, while ASDC identifies redundant steps. Experimental results demonstrate that Proof-R1 improves both reasoning accuracy and the quality of intermediate reasoning steps across multiple datasets.
Critical strengths
- The proposed Proof-R1 is ingeniously designed.
- The experiments are well-structured and demonstrate Proof-R1's consistent superiority over the baselines.
Critical weaknesses
-
My primary concern is that the authors do not seem to adequately acknowledge several contributions that were originally introduced in PRoSFI. In particular, Proof-R1's structured proof generation and MCFV builds upon PRoSFI's
- Structured Formal Intermediates,
- Structured Output Protocol,
- Reward Design.
The current presentation does not sufficiently distinguish these inherited design elements from the novel contributions of Proof-R1. As a result, readers may be left with the impression that these shared components were originally developed as part of Proof-R1.
Overall, I believe that Proof-R1 introduces reasonable improvements over PRoSFI. My concern is primarily about the presentation of contributions rather than technical merit.
I would be willing to recommend acceptance if the authors could revise their contribution claims to more clearly distinguish the novel components from those inherited from prior work, particularly in the Introduction, Related Work, and Methodology sections.
Questions for authors
-
Why does Proof-R1 adopt a PPO-style algorithm rather than a GRPO-style algorithm?
-
How well do proprietary LLMs, such as GPT-5.6 and Claude Opus, perform when given the prompt designed to generate ?
4 comments
Response to Reviewer X4YT (Part 2/2)
Q1: Why does Proof-R1 adopt a PPO-style algorithm rather than a GRPO-style algorithm?
We would like to first clarify a terminology point: GRPO itself follows "PPO-style" clipping, but it estimates the advantage via group-relative normalization.
The reason we use "PPO-style" is that the standard GRPO advantage definition cannot fully capture Proof-R1: while ASDC follows a "GRPO-style" formulation, MCFV does not. The main difference in Proof-R1 is that its advantage function consists of two components ( Eq. 21 ):
The first term is the action-verification advantage provided by MCFV without group reward normalization. Here, is estimated from the historical verification results of actions that adopt the same reasoning rule, and it performs rule-centering on the current verification signal rather than within-problem group reward normalization.
The second term for ASDC employs a group-normalized answer-outcome advantage ( GRPO-style ) and acts only on actions within the final-answer dependency closure through the ASDC closure mask . This component retains the GRPO-style construction of group-relative advantages.
Therefore, the standard GRPO advantage definition cannot fully characterize Proof-R1. We use “PPO-style” to describe its clipped optimization objective while adopting the aforementioned hybrid action advantage. As shown in Appendix A, we explicitly define this clipping mechanism and the hybrid advantage to make the distinction clear.
Q2: How well do proprietary LLMs, such as GPT-5.6 and Claude Opus, perform when given the prompt designed to generate ?
We thank the reviewer for the suggestion. We evaluated proprietary LLMs (gpt-5.6-luna, gpt-5.6-sol, and claude-opus-5-5) on the ProverQA-Hard test split used in our main experiments. The results are shown below.
| Model | Avg@3 ↑ | Pass@3 ↑ | FVR ↑ | RVR ↑ | RGD ↓ |
|---|---|---|---|---|---|
| Proof-R1 (Qwen3-8B) | 92.63 ± 1.10 | 98.99 ± 0.71 | 93.55 ± 0.32 | 90.22 ± 0.26 | 0.3554 ± 0.002 |
| gpt-5.6-luna (Medium) | 43.72 ± 0.29 | 53.27 ± 3.54 | 90.61 ± 0.05 | 89.47 ± 0.22 | 1.0244 ± 0.007 |
| gpt-5.6-sol (Medium) | 83.92 ± 0.58 | 90.45 ± 2.08 | 98.26 ± 0.07 | 98.02 ± 0.10 | 0.5307 ± 0.015 |
| gpt-5.6-sol (High) | 88.61 ± 0.60 | 93.47 ± 1.75 | 98.82 ± 0.08 | 98.77 ± 0.13 | 0.4944 ± 0.011 |
| claude-opus-5-5 | 99.50 ± 0.29 | 100.00 ± 0.00 | 99.97 ± 0.02 | 99.69 ± 0.08 | 0.3973 ± 0.004 |
Proof-R1 achieves the lowest RGD (0.3554), outperforming GPT-5.6 and Claude Opus-5.5. This indicates that Proof-R1 generates proofs without excessive or redundant reasoning steps.
While Claude Opus-5.5 achieves high reasoning accuracy (Avg@3 and Pass@3), Proof-R1, built upon a small 8B open-source model, achieves the comparable performance to these proprietary LLMs.
We hope these revisions and our detailed clarifications help to clearly distinguish our novel contributions from PRoSFI. We appreciate your valuable suggestions, which have enhanced the clarity of our manuscript.
Thanks again for your constructive comments. We hope the revised version meets your expectations for acceptance!
Best regards, Authors of Paper 31925.
References:
[1] Thomas Bouton et al. veriT: An Open, Trustable and Efficient SMT-Solver. In Automated Deduction—CADE-22, volume 5663 of Lecture Notes in Computer Science, pp. 151–156. Springer, 2009.
[2] Frédéric Besson et al. A Flexible Proof Format for SMT: A Proposal. In Proceedings of the First International Workshop on Proof eXchange for Theorem Proving (PxTP), pp. 15–26, 2011.
[3] Hans-Jörg Schurr et al. Alethe: Towards a Generic SMT Proof Format (Extended Abstract). In Proceedings of the Seventh Workshop on Proof eXchange for Theorem Proving (PxTP), volume 336 of Electronic Proceedings in Theoretical Computer Science, pp. 49–54, 2021.
[4] Wachiravit Modecrua et al. Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration. arXiv preprint arXiv:2604.02869, 2026.
[5] Eric Spencer et al. Thiruvathukal, Konstantin Läufer, and Mohammed Abuhamad. TLA-Prover: Verifiable TLA+ Specification Synthesis via Preference-Optimized Low-Rank Adaptation. In Proceedings of the 21st International Conference on Software Technologies (ICSOFT 2026), pp. 627–636, 2026.
[6] Wenqiao Zhu et al. CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 5922–5937, 2025.
I appreciate the authors' efforts to clarify the contributions of Proof-R1 relative to PRoSFI. As indicated in my initial review, I have raised my score accordingly.
Thank you!
Dear Reviewer X4YT:
We sincerely thank you for raising your score. We are happy that our responses have convinced you. Thank you again for strongly supporting our paper!
Best regards,
Authors of Paper 31925
Response to Reviewer X4YT (Part 1/2)
Dear Reviewer X4YT,
Many thanks for your valuable time and positive feedback! We are pleased that you appreciate the ingenious design of Proof-R1 and find our experiments well-structured. We also highly value your acknowledgment of the reasonable improvements over PRoSFI.
Following your suggestions, we have carefully revised the manuscript accordingly, with the changes highlighted in purple.
We thank the reviewer for the valuable comments and for acknowledging the reasonable improvements over PRoSFI. However, our contribution does not lie in the use of intermediate structures or the output format. It lies in using formal verification to determine which conclusions become trusted proof facts and which actions receive outcome credit during policy optimization.
Following your valuable advice, we have revised the Introduction, Related Work, and Methodology sections with the changes highlighted in purple. Specifically, these revisions are summarized as follows:
To clarify this issue, we provide further details on the following three aspects: Structured Formal Intermediates, Structured Output Protocol, and Reward Design.
We detail the comparison between the Proof-R1 and PRoSFI below: