- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Type-6 logic for checking the reasoning process in LLM CoTs. The authors first convert a CoT into propositions, operators, and states, then build a graph and apply a set of rules to detect contradictions, unsupported revisions, missing derivations, and unresolved doubt. The method is evaluated on 996 CoTs and compared with LLM-as-a-judge, ROSCOE, LINC, and PRMs. The results show large disagreement across existing verifiers. Type-6 further finds that derived contradiction is the most common hard failure, and that only about 3% of propositions in a trace directly contribute to the final derivation.
Critical strengths
- The question raised in this paper about whether a CoT is internally coherent is important and worth studying.
- Type-6 gives more useful feedback than a single score by identifying concrete failure types such as contradiction, reasoning avoidance, and unresolved doubt.
Critical weaknesses
- Type-6 lacks a truly independent validation. The paper mainly compares agreement between different verifiers, but agreement does not mean correctness. More importantly, Type-6 defines what counts as contradiction, revision, or reasoning avoidance, and then uses these same rules to report how often such problems occur. This feels somewhat circular. Without an independent human gold standard, it is hard to know whether Type-6 is actually more reliable.
- The experimental setup itself may explain much of the disagreement. The evaluation uses only 996 CoTs from a few datasets, generated by two Qwen3 models, so the coverage is limited. The baselines also target different settings (eg. PRMs are mainly designed for mathematical reasoning, while ROSCOE almost always hard-fails here), so disagreement is not very surprising. In addition, Type-6 relies on GPT-5 to extract propositions, operators, and states before the symbolic verifier runs, making it unclear how much of the final result comes from Type-6 itself versus GPT-5 parsing.
Questions for authors
- The paper template also seems to have some formatting issues (eg. the lack of a running header looks like an oversight).
- The coverage of the most relevant work is clearly insufficient. The paper uses LINC as its main neurosymbolic baseline but does not discuss closely related methods (eg. Logic-LM, SymbCoT, and LogicAgent). In particular, SymbCoT already includes a verifier for symbolic translation and reasoning chains. What fundamentally distinguishes Type-6 from these methods? Why do the experiments only compare against LINC and leave out these closer alternatives?