acceptodds
Withdrawn from ICLR 2027

Learning Formal-Informal routing for stepwise COT verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) often solve complex reasoning tasks by generating chain of thought (CoT), making step-level verification critical. Existing approaches rely on either _formal verification_, whose effectiveness depends on whether a CoT can be faithfully formalized and whether the resulting formal representation preserves its meaning, or _informal LLM-based verification_, which is prone to inconsistent judgements. We propose an adaptive routing policy framework that learns to select the appropriate verification tool (_formal_ or _informal_) for each CoT verification instance. The policy is initialized through supervised finetuning (SFT) and subsequently optimized with reinforcement learning (RL) using two latent faithfulness scores, which encourage it to choose formal verifier only when symbolic formal translation is sufficiently faithful and semantically consistent. We evaluate the framework on six reasoning benchmarks, including _Sieve_, a new step-level verification dataset introduced in this work to assess generalization. Across all benchmarks, our RL routing policy outperforms 1) neuro-symbolic pipelines (_Vanessa_, _Parc-Graph_) and the formal-only verifier, is competitive with the 2) informal-only verifier, and rivals or exceeds substantially larger 3) proprietary verifiers such as GPT-4.1 and GPT-5.4. The improvement reaches ∼36 Macro-F1 points over the formal-only verifier and up to ∼11 points over the informal-only verifier. Its edge over the larger proprietary verifiers is clearest on the challenging _Sieve_ and _PRM_ benchmarks, where the RL routing policy attains the best results among comparable systems and surpasses even GPT-4.1 and GPT-5.4 on _Sieve_ (e.g., vs. / Macro-F1 on _Sieve-MCQ_) and exceeds GPT-4.1 on _PRM_ ( vs. ). These results demonstrate that adaptive verification tool selection is an effective strategy for improving step-level reasoning verification.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Verification, Formal tools, Informal tools, Chain-of-thought, Neuro-symbolic, Reinforcement Learning, Faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_SWxPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies stepwise Chain-of-Thought (CoT) verification. It proposes a routing framework that chooses, for each reasoning step, between (1) formal verification, where the step is translated into SMT-LIB and checked by Z3, and (2) informal verification by an LLM. The router is trained first with supervision derived from which verifier happens to be correct on labeled data, then refined with reinforcement learning that incorporates accuracy, cost, formalizability, and semantic-consistency rewards.

Critical strengths

1.The paper addresses a meaningful practical problem. Formal solvers are precise but cannot faithfully represent many natural-language phenomena, while LLM-based verification is flexible but unreliable. Per-step routing is a sensible framing for combining the two.

2.The paper evaluates across several verification settings, including EBank, ProofWriter, FOLIO, ProntoQA, PRM benchmarks, and the proposed SIEVE benchmark. The results on SIEVE suggest that routing can be useful for some realistic and difficult step-verification cases.

3.The authors make a concrete effort to account for formalization failures, using formalizability and semantic-consistency signals rather than treating every successful SMT translation as faithful.

Critical weaknesses

1.The main motivation is not sufficiently supported. Under the matched GPT-4.1-mini setting, Informal-only consistently outperforms Formal-only on all reported benchmarks. If direct LLM verification is already better overall, the paper needs to show that the router can reliably identify a subset of instances where SMT-LIB verification is actually better than the LLM. However, the paper does not report route distributions for each benchmark, nor the conditional accuracy of formal and informal verification on the routed subsets. It is therefore unclear why routing is necessary instead of simply using the LLM verifier for all instances.

2.The methodological novelty is limited. The method combines LLM-to-SMT translation, Z3-based checking, LLM judging, learned routing, SFT, and RL. The contribution is mainly the combination of these existing components. The paper would be more convincing if it showed that the routing mechanism itself produces a clear and well-understood benefit.

Questions for authors

1.For each benchmark, what percentage of instances is routed to formal, informal, and hybrid verification? What is the accuracy of each route on the instances assigned to it?

2.On the instances routed to the formal verifier, how does formal verification compare with the informal verifier on exactly the same instances?

3.Please provide the missing GPT-OSS-20B Informal-only baseline, as well as ablations for the formalizability score, semantic-consistency score, and RL training stage.

Reviewer_xVC4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an adaptive routing framework for step-wise CoT verification, selecting between formal symbolic verification and informal LLM-based verification. The routing policy is trained using SFT and reinforcement learning with two faithfulness scores. The authors also introduce the SIEVE benchmark and evaluate the method across several reasoning datasets.

Critical strengths

The paper addresses a relevant problem in reliable LLM reasoning. The motivation for combining formal and informal verification is reasonable, and the evaluation covers diverse reasoning benchmarks. The proposed method also achieves promising improvements on SIEVE.

Critical weaknesses

My main concern is that the paper offers limited technical novelty and insufficient experimental evidence to support its contributions. The proposed framework largely combines existing components, including LLM-based verification, symbolic solvers, supervised routing, and standard RL techniques, without clearly establishing what fundamentally new capability this combination provides. The two proposed faithfulness scores are also not sufficiently justified. The formalizability score relies on predefined linguistic categories and heuristic penalties, while the semantic consistency score uses back-translation and embedding similarity, which may not reliably capture logical equivalence. More importantly, the paper lacks direct validation of these scores and critical ablations to demonstrate the necessity of the proposed design. Comparisons with stronger and more closely related verification methods, such as VeriCoT, and simpler routing alternatives would also strengthen the evaluation. Although the method performs well on SIEVE, its improvements over informal-only verification are inconsistent, with substantial degradation on some benchmarks. I would encourage the authors to better identify the specific limitations of existing approaches that their method addresses, validate the reliability of the faithfulness scores, and conduct controlled experiments isolating the contributions of SFT, RL, and each constraint. These analyses would help establish whether the proposed framework provides a meaningful advance beyond combining existing techniques.

Questions for authors

What is the key advantage of the proposed faithfulness-constrained RL over simpler supervised or heuristic routing strategies, and can this advantage be demonstrated through controlled ablations?

Reviewer_tJkSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper frames step-by-step chain-of-thought verification as a choice between formal and informal verifiers. It trains a router through supervised fine-tuning followed by reinforcement learning constrained by two proxy scores for faithfulness, and introduces the SIEVE benchmark to evaluate the value of selecting a verification method for each instance.

Critical strengths
  1. The paper addresses a practical and well-motivated question. The reliability of formal verification depends on the quality of the translation from natural language into symbolic representations, making it useful to study when formal verification should be used. Rather than treating solver correctness as a guarantee for the entire verification pipeline, the paper learns when each verification approach is appropriate. This direction could help build more reliable hybrid verification systems.

  2. Several results provide clear evidence of the approach’s potential. With GPT-4.1-mini as the verifier, routing increases Macro-F1 from 58.19 to 65.66 on SIEVE-QA, from 48.06 to 59.51 on SIEVE-MCQ, and from 90.71 to 98.40 on ProofWriter-Neg. These gains suggest that selective use of a formal verifier can be beneficial even when its overall performance is weak, warranting further investigation.

  3. SIEVE-QA broadens the range of inputs covered by existing verification benchmarks. Compared with short, templated logical tasks, it includes longer reasoning contexts with greater linguistic diversity, along with annotations from two annotators and inter-annotator agreement statistics. With a more complete annotation protocol and clearer data release documentation, it could provide a useful additional benchmark for assessing where verifiers work in practice.

Critical weaknesses
  1. Much of the content appears to be AI-generated and lacks adequate human review.

  2. The paper is difficult to follow because the exposition is disjointed and often unclear. It also lacks explanatory figures to help readers understand the approach.

  3. The appendix figures do not look like plots derived from experimental results. They show apparent signs of AI generation and seem not to have been checked carefully by the authors.

Questions for authors

I could not find the code or data needed to reproduce the results.

Reviewer_3bVYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a learning-based routing framework for step-level chain-of-thought verification. Given a reasoning step and its supporting premises, the framework selects either a formal verification tool, which translates natural language into SMT-LIB and checks entailment using Z3, or an informal LLM-based verifier. The routing policy is initialized through supervised fine-tuning and subsequently optimized using reinforcement learning with two faithfulness-related scores: formalizability and semantic consistency. The authors evaluate the framework on several reasoning benchmarks and introduce SIEVE, a step-level verification dataset derived from existing reasoning resources. The reported results show improvements in Macro-F1 on several benchmarks, particularly SIEVE, although the gains relative to the informal-only verifier vary substantially across tasks.

Critical strengths

1. A practically relevant verification problem.

The paper addresses an important limitation of fixed verification strategies: symbolic verification can fail due to inaccurate formalization, while informal LLM-based verification may struggle with complex logical reasoning. Selecting an appropriate tool for each reasoning instance is a meaningful research direction.

2. Evaluation across diverse reasoning benchmarks.

The experiments cover multiple reasoning datasets and include comparisons with formal-only, informal-only, and neuro-symbolic verification systems. The improvements on some challenging benchmarks suggest that adaptive tool selection can be useful in practice.

Critical weaknesses

1. Insufficient contribution from formal-informal verification.

Although the motivation is meaningful, the proposed framework essentially learns to select between two existing verification tools rather than introducing a substantive mechanism for combining their complementary strengths. In the GPT-4.1-mini setting, only approximately 3.5% of instances are routed to formal verification, and the paper does not establish the independent contribution of these formal calls. Moreover, improvements over the informal-only verifier are inconsistent, with substantial performance degradation on several benchmarks. The empirical evidence therefore does not convincingly demonstrate the benefit of the proposed formal-informal routing framework.

2. Unreliable faithfulness measures and methodological inconsistencies.

The two proposed faithfulness scores are heuristic estimates without sufficient validation of their ability to identify faithful formalizations. In particular, backtranslation and Sentence-BERT similarity cannot establish logical equivalence between natural-language statements and their SMT representations. Furthermore, the constraint statistics in the main formulation differ from those implemented in Algorithm 1, and the routing action space is inconsistently defined. These issues directly undermine the reliability of the proposed constrained reinforcement learning framework.

Questions for authors

Q1. How is the faithfulness of formal verification established?

The semantic consistency score relies on LLM backtranslation and Sentence-BERT similarity, neither of which guarantees logical equivalence. Have the authors independently evaluated whether these scores actually predict semantic preservation between natural-language reasoning steps and their SMT representations? Without such validation, what reliability claim can be made about the formal route?

Q2. What is the independent contribution of formal verification and the faithfulness constraints?

With GPT-4.1-mini, only approximately 3.5% of instances are routed to the formal verifier. How much of the reported improvement is directly attributable to these formal verification calls? Could the authors provide ablations comparing SFT-only routing, RL routing without faithfulness constraints, and the complete method? A comparison with a simpler confidence-based or classification-based router would also be useful.

Q3. Which routing objective and action space are actually implemented?

Eqs. (3)-(4) define formal-tool misuse using an average over all routing trajectories, while Algorithm 1 computes conditional averages over formal/hybrid selections. These lead to different dual updates. Additionally, the main formulation uses two actions, but the appendix introduces a hybrid route. Could the authors clarify the exact objective, constraint estimators, and action space used in the reported experiments?

Q4. How consistent are the improvements over informal-only verification?

The routing policy improves performance on SIEVE but substantially underperforms the informal-only verifier on several other datasets, including EntailmentBank-Hallucination and ProntoQA-Mixtral. What explains these degradations? Could the authors provide per-benchmark routing distributions and error analyses showing when formal routing actually helps or harms verification?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewers and Summary of the Paper

Dear Reviewers, AC, and PCs,

We thank the reviewers for their time. While we plan to withdraw the paper, we believe a summary comment has been made necessary by certain reviews.
First we summarise our work:
Summary

  • We study step-level chain-of-thought verification as a tool selection problem. A small router (Qwen-2.5-7B-Instruct) decides, for each step, whether to use a formal verifier (LLM translation to SMT-LIB, checked with Z3) or an informal LLM verifier.
  • The router is trained with SFT and then RL (RLOO). The reward combines accuracy and cost, plus Lagrangian penalties from two faithfulness scores: formalizability and semantic consistency.
  • We introduce SIEVE (SIEVE-QA with 1,228 instances and SIEVE-MCQ with 950). It has labels from two annotators (Cohen's κ of 0.70 and 0.64) and longer, more varied contexts than templated benchmarks.
  • The router outperforms the formal-only verifier, VANESSA, and PARC-GRAPH. With GPT-4.1-mini, Macro-F1 improves from 58.19 to 65.66 on SIEVE-QA and from 48.06 to 59.51 on SIEVE-MCQ, and it beats GPT-4.1 and GPT-5.4 on SIEVE-MCQ.
  • The router adapts to the verifier. It sends about 3.5% of steps to the formal tool with GPT-4.1-mini and about 23.8% with GPT-OSS-20B.

The reviewers noted that the problem is practically relevant (all), the evaluation is diverse (3bVY, SWxP, xVC4), the gains on SIEVE are promising (all), and SIEVE covers more realistic inputs than templated benchmarks (tJkS).

Revised Tables 1 and 2 (showing only those results that were missed earlier)

In the revised paper we added three things for GPT-OSS-20B. First, the missing Informal-only baseline requested by SWxP. Second, the missing RL Policy result on SIEVE-MCQ, and Third, a variant called RL Policy + NLI, which replaces the Sentence-BERT score with the entailment score of an NLI model (we use only the entailment score).

Table 1 (GPT-OSS-20B):

MethodEBank NegEBank HalluSIEVE QASIEVE MCQPRMFOLIO LLaMA2FOLIO LLaMA3FOLIO MixtralProntoQA LLaMA2ProntoQA LLaMA3ProntoQA Mixtral
Informal Only87.7183.27––50.5787.8781.6485.2696.4696.5193.68
RL Policy56.03
RL Policy + NLI85.7171.4757.3957.3560.7084.9683.1180.5492.8993.9485.24

Table 2 (GPT-OSS-20B):

MethodProofWriter NegProofWriter HalluProofWriter Remove
Informal Only100.0095.36100.00
RL Policy + NLI93.3488.4696.29

What these results show:

  • Comparison with informal-only (SWxP, 3bVY Q4): With GPT-OSS-20B, the informal-only verifier is stronger than routing on synthetic datasets like ProofWriter and ProntoQA. Routing is better on PRM (+7.4, or +10.1 with NLI) and on FOLIO-LLaMA3 (+0.5, or +1.0 with NLI).
  • Faithfulness score (3bVY Q1, SWxP Q3, xVC4): NLI entailment is better than Sentence-BERT on 10 of 14 datasets, with an average gain of about 1.3 points. The largest gains are on ProofWriter-Neg (+6.0), ProofWriter-Hallu (+3.9), PRM (+2.7), and ProntoQA-LLaMA2 (+2.4). It is slightly worse on EBank Neg, EBank Hallu, SIEVE-QA, and FOLIO-LLaMA2 (by 0.1 to 1.2 points). So the choice of consistency measure matters, and our framework does not depend on Sentence-BERT. This is still not a direct validation against logical equivalence, and it is not an ablation that removes the score.

Also, reviewer 3bVY mentioned that the main text uses two actions while the appendix adds a hybrid route. Yes, this is true but we clarified in appendix A about the hybrid route: "We also experimented with a Hybrid verifier/route, which determines the final prediction by taking a majority vote over the outputs of the formal and informal verifiers after running each of them (K) times. However, this route was selected only rarely by the router."

We will work on the following points raised by the reviewers:

  • Use of Backtranslation and Sentence-BERT similarity scores to be used to establish logical equivalence between natural language statements.
  • Remaining ablations and baselines (3bVY, SWxP, xVC4): SFT-only routing, RL without each faithfulness constraint, a simple confidence-based or classifier-based router, and a comparison with VeriCoT.
  • Improvement of proposed Routing framework over Informal-Only route.

On Reviewer tJkS's claim of AI-generated content

  • We respectfully disagree with the reviewer tJkS that the paper is purely written by AI and lacks any human oversight. We have put 3 months into experimentations.
  • The appendix figures are dataset-statistics plots (label counts, token-length and sentence-count histograms, boxplots, etc.), not result plots as wrongly pointed out by the reviewer.

Thanks,
Authors of Submission 56813

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.