- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PRoSFI (Process Reward over Structured Formal Intermediates), which is a RL based on the structured reasoning. The authors argue that previous RLs (which is based on outcome-based) can improve final-answer accuracy without ensuring the validity of intermediate reasoning steps. Meanwhile, directly generating complete formal proofs remains challenging for relatively small language models. To address this issue, PRoSFI requires the model to generate structured formal intermediates alongside natural-language reasoning. These intermeidate reasoning steps are then checked by a formal prover, and the verification results are incorporated into a reward function for GRPO training (e.g., if the answer is correct but not all steps are verified, 0.3). An important distinction is that using intermediate reasoning steps to provide reward signals for model training has already been extensively explored in prior work, as acknowledged by the authors.
Critical strengths
The proposed structured formal intermediates provide a lightweight alternative to generating complete formal proofs, which can be particularly challenging for relatively small language models.
Critical weaknesses
-
The reward design is largely heuristic and insufficiently justified. The proposed reward function assigns scores of 1.0, 0.3, 0.1, and 0.0 depending on answer correctness, formal verification success, and output format validity. Although the authors explicitly acknowledge that these values are heuristically chosen, the paper does not provide a sensitivity analysis to justify these choices or clarify whether they were tuned empirically. Furthermore, the reward function raises several questions about its design. Why does the method assign the maximum reward only when all formal steps are successfully verified? A more fine-grained reward based on the proportion of verified steps might provide a more informative learning signal, particularly when only a small fraction of the reasoning steps fail verification (like previous step-wise PRM works). Overall, the reward function appears to rely on several heuristic design choices without sufficient empirical justification.
-
While the performance improvement is interesting, the ablation studies are not complete. PRoSFI introduces several changes simultaneously: (a) providing formalised premises as part of the input, (b) instructing the model to generate structured intermediate reasoning, and (c) performing GRPO training using rewards derived from formal verification. However, the existing baselines do not adequately isolate the contributions of these components. Outcome-CoT does not incorporate these three components in the same way as PRoSFI. Although Direct Lean Generation is described in Section 5.2 as using the same input information and reward schedule, it generates complete Lean proofs rather than structured intermediates. Therefore, to better understand the source of the improvements, the paper should include ablations that isolate (a) formalised premises only, (b) structured reasoning instructions only, and (c) the combination of formalised premises and structured reasoning instructions without formal-verification-based GRPO training. In particular, comparing the last configuration with the full PRoSFI framework would help determine the additional benefit of RL training with formal verification rewards.
Questions for authors
-
Could the authors provide the complete prompt used for Direct Lean Generation?
-
Which formal prover was used in each experiment?
-
As discussed in the weaknesses above, could the authors provide (a) a reward sensitivity analysis, (b) experiments with alternative reward designs, and (c) component-wise ablation studies?
3 comments
Response to Reviewer mBBs
Q2 (which prover). All reported PRoSFI results use Z3. For each structured step, Z3 checks whether the step's declared dependencies entail its conclusion. For the first-order logic tasks in this paper, this entailment can be decided directly by the solver. Direct Lean Generation is checked by compiling the generated proof with Lean 4. Section 4.1 and Appendix H also mention Lean tactics, Prover9 and DeepSeek-Prover-V1.5. These are back-ends the framework supports for harder tasks, where a stronger solver or LLM assistance may be needed; none was used in the reported results. We will state this explicitly.
Q3. (a) We have no controlled reward-sensitivity analysis; the informal observation above is all we have. (b) We have not trained with alternative reward designs, such as a proportional reward. (c) See W2 above and C1.
Thank you for your response. However, I still believe that the ablation studies are insufficient without (c) formal premises plus structured output, without the verification reward. And while I appreciate the authors' willingness to add the missing details, I still have some concerns about the completeness of the original submission, as the exact prompts and the provers used in the experiments were not clearly reported.
Therefore, I will maintain my current score.
Best,
Reviewer mBBs
Response to Reviewer mBBs
Thanks for the constructive reviews. Here are our point-by-point responses.
On prior work using intermediate steps as reward. Agreed. Section 2 discusses this work, including LogicReward, which also uses a theorem prover. Our claim is narrower: formal process rewards do not require the model to write complete formal proofs. At 7B, with the same input and reward, Direct Lean Generation reaches a Reward Hit Rate of only 13.77% (Hard) and 4.35% (Extra). Structured intermediates are what make the prover signal usable.
W1 (reward design). We agree the design is heuristic. The constants 1.0 / 0.3 / 0.1 / 0.0 were fixed at the outset to impose the ordinal preference stated in Section 4.1. They were not tuned on the reported results. In early internal runs we also tried 0.5 and 0.1 for the intermediate value and did not observe sensitivity to this choice. Training and evaluation cost kept us from running this as a controlled ablation, so those runs are not in the paper and we offer this only as an observation, not as evidence.
You also ask why full reward requires every step to verify, rather than scaling with the fraction of verified steps. Our reasons are design arguments, not empirical results:
We have not compared against a proportional reward, so we cannot claim that all-or-nothing is better in practice. Limitations already notes that "binary verification provides limited feedback on partial progress", and Algorithm 1 leaves the partial score as a configurable term.
W2 (ablations). See C1. Your three configurations map onto the paper as follows:
Q1 (Direct Lean prompt). The full prompt is below; we will add it to the appendix. The premises are given as a Lean 4 context, and the theorem statement for each option is fixed. The system prompt explicitly asks for
havesteps that follow the natural-language reasoning step by step. Despite this, the proofs that compiled were typically a singlesimpcall (Section 5.2).System prompt:
User prompt (
{context},{question}and{conclusion_fol}are filled per instance):