- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses an important problem in logical reasoning: an LLM may produce the correct answer while its reasoning process remains difficult to verify. To address this issue, the paper introduces ReaLogic-FT, a progressive post-training framework for generating formally verifiable reasoning traces. Experiments on ProofWriter and ProntoQA show that structured proof supervision improves both answer accuracy and proof quality, while the additional gains from verifier-guided reinforcement learning are relatively modest.
Critical strengths
- The paper targets an important problem: LLM reasoning needs to be verifiable.
- The methodology is written clearly and rigorously, with good reproducibility.
Critical weaknesses
- The experiments are relatively weak. The evaluation is limited to two well-defined, highly structured logical reasoning benchmarks and includes only a small set of baselines. Moreover, the gains from verifier-guided RL are marginal, which raises questions about the effectiveness and necessity of this component. Overall, the empirical significance and generalizability of the proposed method remain unclear.
- The novelty is also not sufficiently established. Prior work, such as LogicReward and VeriCoT, already uses formal verifiers together with reinforcement learning to improve intermediate reasoning. It is therefore unclear what new insight ReaLogic-FT provides beyond adopting a different formal representation and incorporating verifier feedback into GRPO. Furthermore, if the main novelty lies in generating structured proofs that can be formally verified, it is unclear why directly generating executable formal representations, such as Lean or Prolog, would not provide a simpler or stronger alternative. The paper would benefit from a clearer justification of why its specific formal representation and training design are necessary, and from additional experiments comparing against prior approaches such as LogicReward and VeriCoT, as well as alternatives that directly generate executable formal proofs, such as Lean.
Questions for authors
Could you add experiments on more benchmarks, such as Multi-LogiEval, FOLIO, and LogiQA, and include comparisons with LogicReward, VeriCoT, and Lean? It would also be helpful to provide a clearer justification for the paper’s positioning relative to these approaches.
Reference • Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models — EMNLP 2024. • FOLIO: Natural Language Reasoning with First-Order Logic — EMNLP 2024. • LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning — IJCAI 2020.
3 comments
Response to Reviewer CCHx (Part 2 of 2)
Response to Reviewer CCHx (Part 2 of 2)
W2 (second part). "... it is unclear why directly generating executable formal representations, such as Lean or Prolog, would not provide a simpler or stronger alternative. The paper would benefit from a clearer justification of why its specific formal representation and training design are necessary ..."
Answer: our proofs are executable formal objects (every step is an OWL assertion that HermiT checks). We chose a description logic because of the Unknown class.
- Lean → failing to prove a statement and its negation does not show that neither holds, so Unknown cannot be certified.
- Prolog-style engines → negation as failure gives a closed-world reading, and forward rules with explicit negation miss classical consequences such as contrapositives. This is precisely the source of the 37 label mismatches in ProofWriter (T5).
- ALC with HermiT → decidable, sound, complete, and terminating, so the three-way verdict is returned directly.
- Published evidence at this model scale → PRoSFI reports that a 7B model trained to emit Lean directly obtains low verified reward, and Leang et al. (ACL 2025) need iterative autoformalization to raise the Lean execution rate. For this reason, and given our compute, we did not run a Lean baseline.
- Training design → proof supervision is essential (T1: at best and G-Joint without it, and with it); assertion-level alignment helps where formalization is hard (iLVy, Q1); RL then improves proofs on the problems that require one (T3).
Q. "Could you add experiments on more benchmarks, such as Multi-LogiEval, FOLIO, and LogiQA, and include comparisons with LogicReward, VeriCoT, and Lean? It would also be helpful to provide a clearer justification for the paper's positioning relative to these approaches."
We appreciate these suggestions. We could not add them within this period, and beyond compute there is a specific reason for each: the method needs, per problem, a formalization against which every step can be checked, and a three-way label.
| Requested | Status |
|---|---|
| FOLIO | satisfies both conditions and is our next benchmark; it first requires identifying which of its theories fall within ALC |
| Multi-LogiEval | binary labels without Unknown, and it includes non-monotonic reasoning, which a monotonic logic does not express |
| LogiQA | four-option reading comprehension that relies on implicit and commonsense premises; solver-based methods struggle there in general (Aristotle reports against for chain-of-thought) |
| LogicReward, VeriCoT | their training data are built with GPT-4o and Claude 3.5 Sonnet, so a faithful reproduction with an 8B model is not possible, and their published numbers use different samples and label sets |
| Lean | please see W2 (second part) |
- Positioning → T6, Sec. 1, and App. G.
- Presentation → following your rating, we revised the text so that every abbreviation and symbol is defined at first use and each metric is defined before the results.
Dear Reviewer CCHx,
Thank you again for checking T2 against the original papers. Your question led us to add the comparison of settings and the breakdowns by label and depth (App. F.5), which we think makes the baseline comparison much easier to trust. We would be glad to clarify anything further, on T2 or on the positioning against LogicReward and VeriCoT. If the revision addresses your concerns, we would be grateful if your final assessment could take it into account. Either way, we sincerely appreciate the care you put into this review!
Response to Reviewer CCHx (Part 1 of 2)
Response to Reviewer CCHx (Part 1 of 2)
Thank you for the detailed comments on novelty and on the choice of formalism. They helped us make both explicit in the revision.
W1. "The experiments are relatively weak. The evaluation is limited to two well-defined, highly structured logical reasoning benchmarks and includes only a small set of baselines. Moreover, the gains from verifier-guided RL are marginal, which raises questions about the effectiveness and necessity of this component. Overall, the empirical significance and generalizability of the proposed method remain unclear."
W2 (first part). "Prior work, such as LogicReward and VeriCoT, already uses formal verifiers together with reinforcement learning to improve intermediate reasoning. It is therefore unclear what new insight ReaLogic-FT provides beyond adopting a different formal representation and incorporating verifier feedback into GRPO."
Answer: the difference lies in what the trained model outputs and in what the verifier can certify. We agree that these are closely related works, and Sec. 1 now discusses them explicitly.
T6. Positioning (to the best of our reading of each paper)