- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies stepwise Chain-of-Thought (CoT) verification. It proposes a routing framework that chooses, for each reasoning step, between (1) formal verification, where the step is translated into SMT-LIB and checked by Z3, and (2) informal verification by an LLM. The router is trained first with supervision derived from which verifier happens to be correct on labeled data, then refined with reinforcement learning that incorporates accuracy, cost, formalizability, and semantic-consistency rewards.
Critical strengths
1.The paper addresses a meaningful practical problem. Formal solvers are precise but cannot faithfully represent many natural-language phenomena, while LLM-based verification is flexible but unreliable. Per-step routing is a sensible framing for combining the two.
2.The paper evaluates across several verification settings, including EBank, ProofWriter, FOLIO, ProntoQA, PRM benchmarks, and the proposed SIEVE benchmark. The results on SIEVE suggest that routing can be useful for some realistic and difficult step-verification cases.
3.The authors make a concrete effort to account for formalization failures, using formalizability and semantic-consistency signals rather than treating every successful SMT translation as faithful.
Critical weaknesses
1.The main motivation is not sufficiently supported. Under the matched GPT-4.1-mini setting, Informal-only consistently outperforms Formal-only on all reported benchmarks. If direct LLM verification is already better overall, the paper needs to show that the router can reliably identify a subset of instances where SMT-LIB verification is actually better than the LLM. However, the paper does not report route distributions for each benchmark, nor the conditional accuracy of formal and informal verification on the routed subsets. It is therefore unclear why routing is necessary instead of simply using the LLM verifier for all instances.
2.The methodological novelty is limited. The method combines LLM-to-SMT translation, Z3-based checking, LLM judging, learned routing, SFT, and RL. The contribution is mainly the combination of these existing components. The paper would be more convincing if it showed that the routing mechanism itself produces a clear and well-understood benefit.
Questions for authors
1.For each benchmark, what percentage of instances is routed to formal, informal, and hybrid verification? What is the accuracy of each route on the instances assigned to it?
2.On the instances routed to the formal verifier, how does formal verification compare with the informal verifier on exactly the same instances?
3.Please provide the missing GPT-OSS-20B Informal-only baseline, as well as ablations for the formalizability score, semantic-consistency score, and RL training stage.
Response to Reviewers and Summary of the Paper
Dear Reviewers, AC, and PCs,
We thank the reviewers for their time. While we plan to withdraw the paper, we believe a summary comment has been made necessary by certain reviews.
First we summarise our work:
Summary
The reviewers noted that the problem is practically relevant (all), the evaluation is diverse (3bVY, SWxP, xVC4), the gains on SIEVE are promising (all), and SIEVE covers more realistic inputs than templated benchmarks (tJkS).
Revised Tables 1 and 2 (showing only those results that were missed earlier)
In the revised paper we added three things for GPT-OSS-20B. First, the missing Informal-only baseline requested by SWxP. Second, the missing RL Policy result on SIEVE-MCQ, and Third, a variant called RL Policy + NLI, which replaces the Sentence-BERT score with the entailment score of an NLI model (we use only the entailment score).
Table 1 (GPT-OSS-20B):
Table 2 (GPT-OSS-20B):
What these results show:
Also, reviewer 3bVY mentioned that the main text uses two actions while the appendix adds a hybrid route. Yes, this is true but we clarified in appendix A about the hybrid route: "We also experimented with a Hybrid verifier/route, which determines the final prediction by taking a majority vote over the outputs of the formal and informal verifiers after running each of them (K) times. However, this route was selected only rarely by the router."
We will work on the following points raised by the reviewers:
On Reviewer tJkS's claim of AI-generated content
Thanks,
Authors of Submission 56813