acceptodds
Under review as a conference paper at ICLR 2027

VeriLogic: Reinforcement Learning over Verifiable Logical Reasoning Traces

10 views

Abstract

Logical reasoning derives conclusions from premises through explicit intermediate inference steps, providing a natural basis for process-level verification and supervision. However, existing learning methods often rely on response-level outcome signals, which overlook heterogeneous correctness within the reasoning trace: a correct answer may contain invalid reasoning, while valid intermediate steps may also appear in an incorrect response. To address this issue, we introduce Reinforcement Learning on Verifiable Logical Reasoning (VeriLogic), which turns internal reasoning traces into explicit and verifiable learning objects. Rather than treating a response as an indivisible learning unit, VeriLogic decomposes logical reasoning into explicit intermediate judgments, proofs, and final decisions, and organizes them into a unified “Grounding - Optimization - Refinement” pipeline: (1) Verification-Grounded Reasoning Environment Construction grounds logical reasoning into explicit intermediate states and formally checkable proof obligations, making the internal reasoning process accessible to verification and learning; (2) Logical Structured Policy Optimization learns from verifier feedback through complementary global and local rewards, jointly optimizing the overall reasoning outcome while providing targeted optimization signals to different intermediate reasoning paths; and (3) Verifier-Guided Iterative Refinement acts on verifier feedback by leveraging proof states and error messages to iteratively diagnose and repair failed reasoning. Experiments on six logical reasoning benchmarks show consistent gains over strong baselines, with average improvements of 6.39% and 5.00% on Qwen3 and Llama-3.1. These results highlight the value of making logical reasoning traces verifiable, learnable, and refinable beyond final-answer supervision.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Reinforcement Learning, Logical Reasoning, Neuro-Symbolic Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_q538OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VeriLogic for three-way logical reasoning under a Lean4-formalized setting. It represents each response using Positive, Negative, and Unknown paths, with formal proofs for supported positive or negative claims. Its main training component, PSAR-GRPO, supplements a response-level advantage with separately normalized path-level advantages, routing local feedback only to the corresponding token spans. Local rewards combine gold-label-based path verdict correctness, proof verification, and consistency checks. At inference time, the system allows up to three additional verifier-guided repairs before falling back to a prediction from the current reasoning paths. Experiments use three backbone models and processed subsets of six benchmarks. The full system reports macro-average improvements of 1.83, 6.39, and 5.00 percentage points over the strongest available corresponding baselines for Qwen3-8B, Qwen3-4B, and Llama-3.1-8B, respectively. Ablations examine formalization, policy optimization, advantage routing, and iterative refinement.

Critical strengths
  1. A concrete approach to credit assignment within structured reasoning responses. PSAR-GRPO specifies how to normalize local rewards within each path type and apply the resulting advantages to aligned token spans, while retaining global supervision over the complete response. This is a meaningful technical contribution beyond simply adding another response-level reward: it allows a response with heterogeneous path correctness to receive differentiated updates. The appendix also correctly distinguishes proof failure from non-derivability, explicitly avoiding the invalid inference that two failed proof attempts establish Unknown.
  2. Encouraging empirical coverage across model families and benchmark sources. The full system improves average accuracy on all three backbones, and Table 1 reports improvements over the available baselines across all six processed benchmark subsets. This breadth makes the method more promising than a single-model or single-dataset demonstration. The component and routing ablations are also useful attempts to investigate the source of improvement, although their interpretation requires the clarifications below.
Critical weaknesses
  1. The validity of the transformed evaluation instances is insufficiently established. Appendix B.1 describes model-assisted label normalization and applies verification-based retention to both training and evaluation data: every retained example must pass formalization checks, and True/False examples additionally require an accepted proof, whereas Unknown examples do not. This is a legitimate way to construct a restricted benchmark, but it makes the translation, relabeling, and retention pipeline central to the validity of the evaluation. The supplied PDF does not report the identity of the stronger model, the frequency of label changes, pre-filtering counts and retention rates, or a semantic-fidelity audit. The split description also does not establish the relationship to the original dataset splits or the treatment of shared source contexts.
    An accepted proof establishes a proposition under the supplied formal premises; it does not establish that those premises faithfully preserve the original natural-language problem. The paper itself acknowledges this distinction, but does not provide sufficient evidence that the distinction is controlled in dataset construction. In particular, it is unclear how premise consistency and the continued validity of Unknown labels are checked after translation. My concern is not that only VeriLogic receives formalized inputs but that the semantic validity and selection effects of the shared evaluation set remain insufficiently characterized. This limits how confidently the results support the paper’s natural-language logical-reasoning claims.

  2. The contribution of the proposed training method is not cleanly isolated from inference-time refinement and auxiliary label supervision. Figure 3 reports 92.70% for Formalization + GRPO and 93.67% for Formalization + PSAR-GRPO, a 0.97-point improvement. The full system reaches 96.53%, adding another 2.86 points with refinement. However, the corresponding GRPO + the same verifier-guided refinement control is missing. Moreover, Table 2 reports 96.53% for “Correct routing,” rather than the 93.67% training-only result in Figure 3, without clearly stating the refinement configuration for every routing alternative. Matching structured outputs, rewards, and training budgets does not establish that test-time conditions are also matched. Consequently, the current evidence does not isolate the contribution of the proposed routing mechanism within the full system. There is a related ambiguity about what provides the useful local supervision. Appendix A.2 shows that path correctness is assessed through explicit verdicts against the gold label, while consistency uses verdicts, proof outcomes, and the final answer. It explicitly states that individual natural-language reasoning steps are not independently verified. Thus, the evidence supports a narrower mechanism rather than a general claim of verified intermediate reasoning. A control retaining gold path-label supervision while removing proof-derived feedback would help establish the importance of formal verification itself. The absence of reported run-to-run variability also matters when interpreting the 0.97-point training-only improvement.

Questions for authors
  1. Please specify the model and protocol used for translation and label normalization, and provide per-dataset retention rates, label-change counts, and class distributions. How are semantic fidelity, premise consistency, and Unknown-label validity checked after translation? Please also clarify the original split/source-instance mapping and whether reference proofs or label-derived information are excluded from evaluation inputs. Representative before/after examples and an independent audit would materially strengthen confidence in the evaluation.
  2. Does every configuration in Table 2 use the same verifier-guided refinement procedure and budget? Please reconcile its 96.53% “Correct routing” result with the 93.67% PSAR-GRPO result in Figure 3. What are the results for GRPO and PSAR-GRPO both with and without the same refinement procedure, under matched decoding and verifier-call budgets? Please report uncertainty for the key training-method comparison.
  3. Appendix B.1 states that retained evaluation examples already pass formalization checks, whereas Figure 5 starts from a 58.0% formalization pass rate. What exactly is being formalized in that experiment, by which model, and over which denominator? Are the premise context and target proposition fixed during repair, or can the model modify them? How are the restrictions against additional axioms, sorry, and admit enforced by the verification implementation, beyond their inclusion in the simplified prompt?
  4. How much improvement remains when local gold path-verdict supervision is retained but all proof-derived reward information is removed, including the verification-dependent portion of consistency? Please provide the numerical reward weights used in these comparisons and report both answer accuracy and verified-proof success under identical inference conditions. This would help distinguish learning from auxiliary label supervision from learning specifically enabled by formal verification.
5 comments
AuthorsOpenReview ↗

Response to Reviewer q538 – [1/2]

Thank you for your detailed feedback! We appreciate your recognition of PSAR-GRPO’s contribution to credit assignment within structured reasoning responses and of the empirical coverage across model families and benchmark sources. We address the weaknesses and questions you raised below.

W1 & Q1

How were the examples translated, relabeled, retained, and split, and what checks establish semantic fidelity, premise consistency, Unknown-label validity, and the exclusion of reference proofs or label-derived information from evaluation inputs?

(1) Formalization and label normalization.

We use DeepSeek-V3.2 to formalize problems and normalize their labels. The structured template spells out the entities, predicates, facts, rules, and queries so we can compare them directly with the original problems.

For LogiQA and MultiLogic, we separate refutable False cases from underdetermined Unknown cases to make the three-way labels more consistent. Table 1 reports the class distributions, and Table 2 reports the label reassignments.

During label normalization, DeepSeek-V3.2 uses the original natural-language premises to decide whether the answer is underdetermined. We do not use failed Lean4 proof attempts to assign Unknown.

(2) Dataset construction and fair evaluation.

We apply the same retention criteria to training and evaluation. Every retained instance passes Lean4 formalization checks. True/False instances must also have an accepted proof of the corresponding conclusion.

We evaluate all compared methods on exactly the same retained set. We will clarify preprocessing and retention in Appendix B.1. Table 1 also lists the candidate counts, retention rates, and final evaluation sizes for each dataset.

(3) Semantic fidelity.

During dataset preparation, we checked semantic fidelity by comparing formalizations with the original natural-language problems. The original paper did not explicitly report these human checks.

Of the 180 audited formalizations, 175 (97.2%) were semantically faithful. All 97 with an accepted proof (53.9% of the audit) were faithful, and all five unfaithful samples lacked an accepted proof. The human checks provide empirical evidence of semantic alignment beyond Lean4 verification.

(4) Split separation and evaluation inputs.

Where official splits exist, we keep the dataset authors' original training and evaluation partitions. For datasets without official splits, the problems are constructed independently, with no shared contexts or templates.

We split these datasets at the source-problem level into disjoint training and evaluation subsets. Instances that share a source context stay in the same partition.

At evaluation time, the model receives only the formalized premises and two unproved targets. The targets are the query and its negation. We exclude reference labels and completed proofs to prevent direct answer leakage. The representative example below illustrates this setup.

Table 1. Dataset composition and retention.

DatasetTrain T/F/UTrain retained/candidates (%)Final test T/F/UTest retained after filtering/candidates (%)Final test n
e-SNLI298/121/39458/1,000 (45.8%)127/118/55331/700 (47.3%)300
FOLIO211/119/21351/1,000 (35.1%)83/50/67204/204 (100.0%)200
LogiQA383/101/43527/1,000 (52.7%)122/131/47354/700 (50.6%)300
MultiLogic319/42/19380/1,000 (38.0%)129/115/56315/800 (39.4%)300
ProofWriter870/857/01,727/2,000 (86.4%)173/127/0598/700 (85.4%)300
ProverQA1,243/1,189/1012,533/3,500 (72.4%)111/109/80517/700 (73.9%)300
Total3,324/2,429/2235,976/9,500 (62.9%)745/650/3052,319/3,804 (61.0%)1,700

Table 2. Label reassignments.

DatasetTrain changed/original F (%)Test changed/original F (%)
LogiQA43/144 (29.86%)47/178 (26.40%)
MultiLogic19/61 (31.15%)56/171 (32.75%)
Total62/205 (30.24%)103/349 (29.51%)

Representative formalization example

Penelope is either adventurous or values stability, but not both. Anyone who settles down values stability. Penelope settles down.

Question: Is Penelope adventurous? Reference label: False (excluded from the model input).

Lean4 premises (excerpt)

axiom PenelopeEitherOr :
  (Adventurous Penelope ∨ ValuesStability Penelope) ∧
  ¬ (Adventurous Penelope ∧ ValuesStability Penelope)
axiom SettlesDownImplies :
  ∀ (x : Entity), SettlesDown x → ValuesStability x
axiom PenelopeSettles : SettlesDown Penelope

Proof targets: Adventurous Penelope and ¬ Adventurous Penelope.

AuthorsOpenReview ↗

Response to Reviewer q538 – [2/2]

W2 & Q2 & Q4

Under matched inference conditions, how much do routing, refinement, and proof-derived supervision each contribute to answer accuracy and verified-proof success, what reward weights were used, and how much do the training results vary across runs?

(1) Separating training and refinement contributions.

With the same formalization and zero repair, PSAR-GRPO improves answer accuracy and initial proof acceptance over Global GRPO.

Table 3 reports both metrics at zero and three repair rounds. We match decoding settings and verifier-call budgets at each setting. Higher proof acceptance after three rounds is consistent with stronger initial proof construction.

The Correct Routing (Ours) entry in VeriLogic Table 2 used test-time refinement. The other routing controls used zero repair. We will update that entry from 96.53% to 93.67% so that all routing controls are compared without repair.

Table 3. Answer accuracy and proof acceptance (%).

MethodAcc. (0)Proof (0)Acc. (3)Proof (3)
Global GRPO92.7025.393.8658.7
PSAR-GRPO93.6742.396.5376.4

Numbers in parentheses are repair rounds.

(2) Isolating proof-derived supervision.

PSAR-GRPO (Gold Label Only) keeps the three-path structure, gold path-verdict supervision, path-specific advantage normalization, and token routing. We remove all proof-derived rewards, including verification-dependent consistency signals. Other settings stay fixed.

Gold labels score verdicts, and Lean4 checks proofs. Under the same zero-repair setting, Gold Label Only achieves 93.47% macro-average answer accuracy and 27.1% proof acceptance.

In the global reward, we weight answer correctness at 0.8 and format correctness at 0.2. The local reward assigns 0.4 to path-verdict correctness, 0.4 to Lean4 proof verification, and 0.2 to path consistency. The local advantage coefficient is 0.1.

(3) Verification of intermediate formal reasoning steps.

Lean4 checks the formal derivation and final conclusion in proofs for the Positive and Negative paths. An invalid inference invalidates the proof. PSAR-GRPO uses verification outcomes as path-specific learning signals for formal reasoning and proof generation. Independent verification of individual natural-language reasoning steps is outside the scope of this work.

(4) Consistency of training improvements.

Without test-time refinement, PSAR-GRPO outperforms the reported Global GRPO baseline at each of the three independent training seeds (Table 4). The small variation across runs suggests that the training-only gain holds beyond a single favorable run.

Table 4. Zero-repair accuracy by training seed.

SeedMacro-average (%)Gain vs. reported GRPO (pp)
127398173993.69+0.99
183704926193.67+0.97
206581374993.64+0.94

Q3

What was formalized in Figure 5, by which model and over which denominator, and how does the verifier keep the premises and target fixed during proof repair and enforce the restrictions on additional axioms, sorry, and admit?

(1) Formalization pass rate and dataset filtering.

Figure 5 measures formalization before filtering. DeepSeek-V3.2 translates natural-language problems into Lean4 and uses Lean4 feedback to repair failed formalizations over multiple rounds. The denominator is all 13,304 candidates (9,500 training and 3,804 evaluation). The Lean4 pass rate rises from 58.0% on the first attempt to 99.6% after repair. Figure 5 reports proof acceptance separately.

Appendix B.1 describes later filtering. All retained instances pass formalization checks. Checks on proof validity and suitability of proof strategies for training further reduce retention.

(2) Constraints during iterative refinement.

During formalization repair, the external model may revise the Lean4 representation using verifier feedback. During proof repair, the premise context (Γ\Gamma) and target proposition (PP) stay fixed. The policy can revise only the proof attempt.

(3) Enforcement of proof validity.

During both RL training and inference, we check extracted proofs in code before sending them to Lean4. We reject proofs that contain prohibited constructs such as sorry or admit. These attempts receive negative rewards in training and count as invalid proofs in evaluation.

The verifier also rejects new axiom declarations. It checks proof dependencies against an allowlist of the fixed premises and permitted standard Lean axioms. We will describe the checks in the revised appendix.

Thank you again for your detailed comments. We appreciate the opportunity to clarify these points and would be glad to answer any further questions.

Dear Reviewer,

We hope our response has addressed the concerns raised in the review. As the discussion period is approaching its end, we wanted to check whether there are any remaining questions or points we could further clarify. We would be glad to provide additional detail if helpful.

Best regards, The Authors

Reviewer_K4c4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VeriLogic addresses a limitation of answer-level reasoning rewards, i.e., a correct answer can hide an invalid proof. It constructs Positive, Negative, and Unknown paths, uses Lean4 verification to supply path-specific GRPO advantages, and repairs failed proofs at test time. Tests use formalized subsets of six datasets and three backbones.

Critical strengths

Assigning credit to the relevant reasoning path is a sensible idea when different parts of one response have different correctness states. The routing controls directly test this alignment. The method also correctly states that failed proof attempts do not establish that neither the query nor its negation follows from the premises. Evaluation across model families gives the results some breadth.

Critical weaknesses
  1. The central novelty needs stronger support relative to [1], which already provides theorem-prover process supervision. The specific advance is three-path advantage routing. On Qwen3-8B, PSAR raises global GRPO from 92.70 to 93.67, with refinement then reaching 96.53. The modest training-only gain is not inherently inadequate, but its consistency and attribution matter. Rewording the “new paradigm” claim alone would not establish the contribution.

  2. The evaluation covers filtered, formalized, partly relabeled tasks. Missing coverage counts or unclear repair budgets may be resolved by reporting existing information. However, if semantic fidelity was not checked or baselines received materially less inference compute, the evidence for the main comparison would need additional validation. Unfiltered benchmark evaluation would broaden significance, but is not automatically required for this more restricted task.

[1] LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervision

Questions for authors
  1. Under the same formalized inputs and repair budget, what evidence isolates the benefit of path-specific routing over global GRPO and LogicReward-style supervision? Please clarify whether all Table 2 routing controls receive identical refinement, and whether the training-only gain is consistent across the reported runs.

  2. What fraction of each benchmark was retained or relabeled, and what checks establish semantic fidelity and split separation? Please distinguish information already available but omitted from validation that has not yet been performed.

6 comments
AuthorsOpenReview ↗

Response to Reviewer K4c4 – [1/2]

Thank you for your valuable feedback. We appreciate your recognition of the rationale for assigning credit to the relevant reasoning path and the breadth of the evaluation across model families. We address the weaknesses and questions you identified below.

W1 & Q1

Under the same formalized inputs and repair budget, what evidence isolates the benefit of path-specific routing over Global GRPO and LogicReward-style supervision, and how consistent is the training-only gain across runs?

Our contribution is how PSAR-GRPO uses verifier feedback to optimize the policy.

(1) Path-specific credit assignment.

LogicReward aggregates step-level verification results into response scores for SFT selection and DPO pairing.

During on-policy training, PSAR-GRPO normalizes each path’s reward separately. It applies the resulting advantage to that path’s tokens. Differences in path correctness can drive more targeted local updates even when the global rewards are identical.

(2) Consistency and benefits beyond answer accuracy.

Without test-time refinement, PSAR-GRPO outperforms the reported Global GRPO baseline across all three training seeds (Table 1). Reliable logical reasoning requires both correct conclusions and verifiable supporting evidence.

Table 2 shows that PSAR-GRPO improves initial proof acceptance. This gain goes beyond final-answer accuracy and reduces reliance on later repair.

Table 1. Accuracy across training seeds.

Training seedMultiLogicProofWriterFOLIOProverQALogiQAe-SNLIMacro-average (%)Gain over reported global GRPO (pp)
127398173996.0089.0091.5092.3396.3397.0093.69+0.99
183704926195.3389.3391.0092.6797.0096.6793.67+0.97
206581374995.6789.6790.5092.0096.6797.3393.64+0.94

Table 2. Proof acceptance before and after refinement.

Training methodBefore repairAfter 3 repair rounds
Global GRPO25.3%58.7%
PSAR-GRPO42.3%76.4%

(3) Internalizing complementary reasoning capabilities.

To our knowledge, VeriLogic is the first framework to internalize three-way logical reasoning and formal proof generation through verifier-guided, path-specific reinforcement learning.

The model can explore multiple reasoning directions at inference time. During training, PSAR-GRPO gives complementary paths independent learning signals so the model can learn and combine valid reasoning patterns across logical hypotheses.

(4) Clarification of refinement settings (VeriLogic Table 2).

The reported result for Correct Routing (Ours) included test-time refinement. All other routing controls used zero repair. We will report the matched zero-repair results in the revised PDF. Table 3 shows the routing comparison without test-time repair.

Table 3. Routing ablation without test-time repair.

RoutingAvg. (%)Δ vs. GRPO (pp)
Base (GRPO)92.70—
Swapped72.75−19.95
Full-response90.76−1.94
Random-region86.86−5.84
Correct Routing (Ours)93.67+0.97
AuthorsOpenReview ↗

Response to Reviewer K4c4 – [2/2]

W2 & Q2

What fraction of each benchmark was retained or relabeled, which checks were performed to establish semantic fidelity and split separation, and how comparable were the inference budgets?

(1) Dataset composition and split separation.

As described in Appendix B.1, we build training and evaluation subsets from six benchmarks and normalize the labels to True, False, and Unknown. For LogiQA and MultiLogic, we use a stronger model to help identify underdetermined examples among those originally labeled False. We relabel those examples as Unknown.

Table 4 gives the class distributions of the retained examples. Table 5 summarizes the label reassignments.

When a dataset has official splits, we sample training and evaluation examples separately. For the other datasets, we build disjoint subsets so that no source problem appears in both. We will clarify the original dataset sizes and retention rates in the revised manuscript.

Table 4. Class distributions.

DatasetTrain TTrain FTrain UTrain totalTest TTest FTest UTest total
e-SNLI2981213945812711855300
FOLIO21111921351835067200
LogiQA3831014352712213147300
MultiLogic319421938012911556300
ProofWriter87085701,7271731270300
ProverQA1,2431,1891012,53311110980300
Total3,3242,4292235,9767456503051,700

Table 5. Label reassignments.

DatasetTrain original False (F+U)Train changed to Unknown (U)Train U/(F+U) (%)Test original False (F+U)Test changed to Unknown (U)Test U/(F+U) (%)
LogiQA1444329.861784726.40
MultiLogic611931.151715632.75
Total2056230.2434910329.51

(2) Semantic fidelity of formalization.

Semantic fidelity matters when building an environment for verifiable reasoning. The fixed template in §3.1 and Appendix A.1 explicitly represents entities, predicates, facts, rules, and queries, making them easier to check against the original problem.

We checked semantic consistency by hand during dataset preparation. The original submission omitted a separate account of these checks.

Our audit of randomly sampled formalizations from all six datasets found high semantic fidelity (Table 6). We confirmed that every sample with an accepted proof was semantically faithful. None of the samples we identified as unfaithful had an accepted proof.

The audit provides strong empirical evidence for the semantic reliability of our formalization process and the quality of the verifiable reasoning environment we use for policy optimization. We will describe this existing validation procedure in the revised appendix.

Table 6. Semantic fidelity and proof acceptance.

Audited samplesSemantically faithfulAccepted-proof coverageFidelity among proof-accepted samplesUnfaithful samples without accepted proofs
180175/180 (97.2%)97/180 (53.9%)97/97 (100.0%)5/5 (100.0%)

(3) Fair inference comparison.

We compare PSAR-GRPO with Global GRPO to isolate the contribution of PSAR-GRPO. Both methods use the same formalized inputs, backbone models, three-path reasoning formats, maximum generation lengths, and decoding settings.

We evaluate both methods with zero test-time repair, ruling out extra repair steps as the source of the gains. This controlled comparison provides direct evidence for the gains from path-specific optimization beyond formalization alone (Figure 3).

Thanks again for your helpful feedback. We would be happy to clarify any remaining points during the discussion.

Thanks for clarifying the repair budgets and reporting the matched comparisons. My main remaining question is how much the proposed path-specific routing adds over an alternative that uses the same verifier feedback under the same budget. The rebuttal makes the case for the full training method stronger, but leaves that comparison open. I’m keeping my score at 2 (weak reject).

Reviewer_uXrDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VeriLogic, a reinforcement learning framework that addresses the limitations of outcome-based RL in logical reasoning, where final-answer rewards fail to distinguish the correctness of intermediate reasoning processes.

Specifically, VeriLogic formalizes natural-language problems into Lean4 and decomposes reasoning into three paths: Positive, Negative, and Unknown. It introduces PSAR-GRPO, which combines global outcome rewards with path-specific local rewards derived from logical correctness and formal verification, assigning optimization signals to the corresponding reasoning regions. Additionally, verifier-guided iterative refinement uses Lean4 proof states and error feedback to repair failed proofs at inference time.

Experiments across six logical reasoning benchmarks and three backbone models demonstrate consistent improvements over existing baselines

Critical strengths

1.Fine-grained RL supervision through formal verification: The paper addresses an important limitation of outcome-based RL, where a correct final answer does not necessarily imply valid intermediate reasoning. Leveraging Lean4 verification to provide path-specific learning signals is a meaningful approach to improving logical reasoning.

2.Comprehensive experiments and ablations: The proposed framework demonstrates consistent improvements across three backbone models and six logical reasoning benchmarks. In particular, the routing ablations show that correctly assigning local advantages to their corresponding reasoning regions outperforms swapped, random-region, and full-response routing, supporting the importance of properly aligned credit assignment.

Critical weaknesses

1.Limited novelty compared to prior work: Formalizing natural-language reasoning and using formal verifiers to provide intermediate reasoning supervision have already been explored in prior work, such as LogicReward [1]. The primary methodological novelty of VeriLogic appears to lie in PSAR-GRPO, which routes path-specific advantages to predefined reasoning regions. However, the distinction from existing verifier-guided process supervision methods and the additional benefits of this design are not sufficiently established.

2.Insufficient justification for the three-path decomposition: VeriLogic requires the model to generate Positive, Negative, and Unknown reasoning paths for every problem. However, many logical reasoning problems can be solved using a single valid reasoning path. While the motivation concerns heterogeneous correctness within reasoning traces, it remains unclear why addressing this issue requires explicitly generating all three paths. The paper does not sufficiently justify why this predefined decomposition is preferable to directly verifying intermediate steps in naturally generated reasoning trajectories.

3.Relatively limited contribution of PSAR-GRPO: According to the ablation study, formalization alone substantially improves accuracy from 80.70% to 90.00%, whereas adding PSAR-GRPO to standard GRPO yields a relatively modest improvement from 92.70% to 93.67%. Since the full VeriLogic framework also incorporates test-time refinement, it is important to better distinguish the contribution of the proposed RL optimization method from that of formalization and inference-time verification.

4.Unclear semantic correctness of formalization: The framework relies on translating natural-language problems into Lean4 representations, but it is unclear how semantic equivalence between the original problems and their formalized counterparts is ensured. Successful Lean4 verification guarantees the validity of a proof with respect to the formalized proposition, but does not necessarily guarantee that the proposition accurately reflects the original natural-language problem. Since verifier feedback is directly used for RL supervision, incorrect formalization could introduce misleading learning signals.

Xu, Jundong, et al. "Logicreward: Incentivizing llm reasoning via step-wise logical supervision." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

All the questions I want to raise are relevant to the weakness.

1.Semantic alignment of formalization: How do the authors verify or ensure semantic consistency between the original natural-language problems and their Lean4 formalizations? Specifically, does the formalization pass rate reported in Figure 5 (58.0% → 99.6%) measure only syntactic and type validity in Lean4, or does it also account for semantic correctness relative to the original natural-language problems? If semantic correctness is not explicitly verified, have the authors investigated how incorrect formalizations affect verifier rewards and subsequent policy optimization?

2.Necessity of the three reasoning paths: Why is it necessary to generate all three reasoning paths (Positive, Negative, and Unknown) for every problem? Is there empirical evidence that this predefined decomposition is more effective than generating a single natural reasoning trajectory and directly verifying its intermediate steps?

3.Contribution of PSAR-GRPO: Given the relatively modest improvement from adding PSAR-GRPO to standard GRPO under the same formalization setting, how do the authors interpret the contribution of path-specific optimization to the overall performance gains? What advantages does PSAR-GRPO offer over existing verifier-guided step-level supervision approaches?

5 comments
AuthorsOpenReview ↗

Response to Reviewer uXrD – [1/2]

Thank you for your positive feedback on fine-grained RL supervision through formal verification and on the experiments and ablations across models and benchmarks. We address the weaknesses and questions you raised below.

W1 & Q3-1

How does PSAR-GRPO differ from LogicReward and other verifier-guided step-level supervision methods, and what does path-specific advantage routing add?

Our main contribution is how we use verifier feedback in training to improve and internalize logical reasoning.

(1) Path-specific credit assignment.

Step-level supervision targets individual steps. LogicReward aggregates step-level verification results into response scores for SFT selection and DPO pairing.

During on-policy training, PSAR-GRPO normalizes each path’s reward separately. Within a single policy objective, it applies the resulting advantage to the tokens in that path’s reasoning region. Differences in path correctness can therefore drive more targeted local updates even when the global rewards are identical.

(2) Improvements beyond answer accuracy.

A correct answer can still come with invalid reasoning or a proof that fails verification. We consider both a correct conclusion and verifiable supporting evidence necessary for reliable logical reasoning.

With the same formalization and zero-repair setting, PSAR-GRPO improves both answer accuracy and initial proof acceptance over Global GRPO (Figure 3 and Table 1). The gains before repair show that path-specific optimization helps the model generate verifiable proofs directly.

(3) Internalizing complementary reasoning capabilities.

To our knowledge, VeriLogic is the first framework to internalize three-way logical reasoning and formal proof generation through verifier-guided, path-specific reinforcement learning.

The model can explore multiple reasoning directions at inference time. During training, PSAR-GRPO gives complementary paths independent learning signals. These signals help the model learn and combine valid reasoning patterns across logical hypotheses.

Table 1. Proof acceptance before and after refinement.

Training methodBefore repairAfter 3 repair rounds
Global GRPO25.3%58.7%
PSAR-GRPO42.3%76.4%

W2 & Q2

Why generate Positive, Negative, and Unknown paths for every problem, and how does this compare with verifying intermediate steps in a single naturally generated reasoning trajectory?

(1) Exploring complementary logical hypotheses.

Aristotle [1], LogicAgent [2], and HexLogicAgent [3] show that exploring complementary reasoning directions can improve logical reasoning performance.

A single valid path can solve many problems, but a naturally generated trajectory can remain stuck on an early mistake.

Given the same premises, our three paths check for support for PP, support for ¬P\neg P, and whether the information is insufficient. Stepwise verification checks the trajectory the model has generated. The three paths also prompt the model to explore alternatives and reconsider a potentially incorrect conclusion.

(2) Internalizing multi-path reasoning through reinforcement learning.

Prior training-free multi-perspective methods also explore different directions.

VeriLogic gives each path a defined reasoning region for verification and credit assignment, allowing PSAR-GRPO to optimize the complementary paths and formal proof generation separately.

Table 2 compares three-path VeriLogic with three single-direction variants. It also includes Global GRPO as an optimization baseline. All methods use the same zero-repair setting. VeriLogic scores higher than all four baselines on all six benchmarks, showing the benefits of complementary reasoning and path-specific optimization.

Table 2. Accuracy of each path.

MethodMultiLogicProofWriterFOLIOProverQALogiQAe-SNLIAvg. ↑
Pos91.3384.6787.0088.3392.0092.6789.33
Neg75.6767.3370.0071.6774.3377.0072.67
Unknown85.3378.6781.0082.6786.0086.3383.33
Global GRPO94.6788.0090.5091.0095.6796.3392.70
Ours95.6789.3391.0092.3396.6797.0093.67

[1] Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework
[2] From Ambiguity to Verdict: A Semiotic-Grounded Multi-Perspective Agent for LLM Logical Reasoning
[3] Semiotic logical hexagon theory for LLM logical reasoning

AuthorsOpenReview ↗

Response to Reviewer uXrD – [2/2]

W3 & Q3-2

With formalization held fixed, what does PSAR-GRPO add to standard GRPO, and what further gains come from test-time refinement?

PSAR-GRPO contributes to proof generation as well as final-answer accuracy.

(1) Distinct contributions of each component.

Formalization puts reasoning in a form the verifier can check. Test-time refinement repairs proofs that fail verification during inference. PSAR-GRPO uses verifier feedback to train the model.

Its gains over Global GRPO with the same formalization and no repair show what path-specific optimization contributes on its own.

(2) Beyond answer accuracy.

A correct answer can still come with invalid reasoning or a proof that fails verification. We consider both a correct conclusion and verifiable supporting evidence necessary for reliable logical reasoning.

In the same zero-repair setting, PSAR-GRPO yields a modest gain in answer accuracy over Global GRPO and substantially higher initial proof acceptance (Table 1). The higher acceptance rate shows that PSAR-GRPO helps the model generate verifiable proofs and provides a better starting point for verifier-guided iterative repair.

(3) Finer-grained credit assignment.

Some approaches combine verifier feedback into a single score for the response. PSAR-GRPO preserves differences in correctness across paths and applies each path's learning signal to the corresponding tokens.

The model can reinforce valid reasoning and correct failed paths even when the final-answer reward stays the same. Table 3 shows the routing comparison without test-time repair.

Table 3. Routing ablation without test-time repair.

RoutingAvg. (%)Δ vs. GRPO (pp)
Base (GRPO)92.70—
Swapped72.75−19.95
Full-response90.76−1.94
Random-region86.86−5.84
Correct Routing (Ours)93.67+0.97

W4 & Q1

What does the formalization pass rate in Figure 5 measure, how was semantic consistency checked, and was the effect of incorrect formalizations on verifier rewards and policy optimization investigated?

The accuracy of formalization directly affects both verification and learning.

(1) Structured formalization.

We use the fixed template described in §3.1 and Appendix A.1 to represent entities, predicates, facts, rules, and queries explicitly. The template makes it easier to check how the natural-language premises correspond to the formal propositions.

Figure 5 reports the proportion of formalizations that pass Lean4 syntax and type checks. We assess semantic fidelity separately through human evaluation.

(2) Human evaluation of semantic fidelity.

We randomly sampled 30 formalizations from each of the six datasets. We manually checked their premises and queries against the original natural-language problems.

The audit found high semantic fidelity (Table 4). All samples with an accepted proof were semantically faithful. None of the samples we identified as unfaithful had an accepted proof.

The audit provides further evidence that our formalization process produces semantically faithful representations. We will include the evaluation in the revised appendix.

(3) Impact of incorrect formalizations.

Incorrect formalizations can distort verifier rewards and policy updates, potentially leading to reward hacking. Our work focuses on path-specific optimization, so we prioritize structured formalization and semantic validation to reduce this risk.

A controlled study of how incorrect formalizations affect RL training falls outside the scope of this paper. We will explore this question in future work.

Table 4. Semantic fidelity and proof acceptance.

Audited samplesFaithful before proof checkingAccepted-proof coverageFidelity among proof-accepted samplesUnfaithful samples without accepted proofs
180175/180 (97.2%)97/180 (53.9%)97/97 (100.0%)5/5 (100.0%)

Thank you again for your helpful comments. We appreciate the opportunity to improve the paper and are happy to address any further questions.

Dear Reviewer,

We hope our response has addressed the concerns raised in the review. As the discussion period is approaching its end, we wanted to check whether there are any remaining questions or points we could further clarify. We would be glad to provide additional detail if helpful.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Chairs and Reviewers,

We sincerely appreciate the chairs and reviewers for their time and effort in handling the large volume of ICLR 2027 submissions, and we are deeply grateful for all your feedback on our paper. So far, we have received ratings of 3, 3, and 2.

We introduce VeriLogic, a new RL paradigm for logical reasoning and a unified verifiable reasoning framework. All reviewers responded positively to our core contributions.

  1. Reviewer uXrD (Rating 3) strongly supported (1) our design for fine-grained RL supervision through formal verification and recognized (2) the novelty of path-specific credit assignment. The reviewer also appreciated (3) our extensive experiments across models and benchmarks and highlighted (4) our method's gains in formal proof capabilities.

  2. Reviewer q538 (Rating 3) recognized (1) the novelty and effectiveness of path-specific credit assignment and the rationale for our multi-path reasoning design. The reviewer appreciated (2) the breadth of our experiments and acknowledged (3) the independent contributions of path-specific optimization and iterative refinement. The reviewer further recognized (4) the gains in proof capabilities from formal verification feedback and the reliability of our verifiable reasoning environment.

  3. Reviewer K4c4 (Rating 2) found (1) our path-specific credit assignment design sensible. The reviewer also gave positive feedback on (2) the design of our path-routing controls and (3) our evaluation across different model families.

Reviewer K4c4 (Rating 2): A misunderstanding in our discussion of the existing control experiments. We have fully addressed all of Reviewer K4c4's concerns. In the reviewer's last response, only one remaining question was raised:

Does path-specific credit assignment make an independent contribution given the same verifier feedback and training budget?

  • Full-response broadcast control: The original paper already reports this control with the same verifier feedback and path advantages. We further matched the inference budgets in our rebuttal.
  • Response-level verifier feedback control: We added a standard GRPO control experiment with the same verifier feedback, training budget, and inference budget.

Both experiments clearly show the independent contribution of path-specific credit assignment. They directly address the reviewer's concern. We understand that the reviewer may not yet have had time to consider our follow-up response or revisit the rating.

It is with great passion that we submit this work. It reflects our careful design and bold rethinking of how to integrate RL with logical reasoning. We also devoted significant effort during the rebuttal to clarifying and strengthening our case. We hope our rebuttal will help inform your final assessment, and we deeply thank you for taking the time to consider our work.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.