acceptodds
Withdrawn from ICLR 2027

What Was Said, Not What Was 'Thought': Type-6 Logic for CoT Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce Type-6 logic, a variant of dynamic epistemic logic augmented with two operators (uncertainty and recurrence), designed to model the inferential dynamics of contemporary large language model (LLM) chain-of-thought (CoT) reasoning. Type-6 accounts for common LLM reasoning pathologies such as unlicensed revision, enthymemes, loopbacks, and unverifiable/incorrect claims. We propose a verifier based on Type-6 logic that builds a graph out the trace, and checks it against Type-6's axioms and inference rules. We evaluate our framework on LLM-generated CoTs four splits spanning formal and informal reasoning. Our verifier detects structurally unsound reasoning steps that surface-level heuristics miss, and allows for easy visualisation of the model's reasoning process. In our corpus, our verifier shows that derived contradiction is the most common hard-fail category in CoT, and that only about 3% of the propositions of a trace have impact on the final derivation. Ablation studies show that other verification methods (LLMs-as-judges, other neurosymbolic approaches, etc.) cannot be considered interchangeable: for example, agreement between LLMs-as-judges and LINC is , and this persists within a method across underlying models. Type-6, however, is the most agreed-with method amongst the ones we tested. We prove our verifier runs on average-case linear time; and release our logic specification and artefacts.

Primary area:
interpretability and explainable AI
Keywords:
dynamic epistemic logic, chain-of-thought verification, model auditability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3MMtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Type-6 logic for checking the reasoning process in LLM CoTs. The authors first convert a CoT into propositions, operators, and states, then build a graph and apply a set of rules to detect contradictions, unsupported revisions, missing derivations, and unresolved doubt. The method is evaluated on 996 CoTs and compared with LLM-as-a-judge, ROSCOE, LINC, and PRMs. The results show large disagreement across existing verifiers. Type-6 further finds that derived contradiction is the most common hard failure, and that only about 3% of propositions in a trace directly contribute to the final derivation.

Critical strengths
  • The question raised in this paper about whether a CoT is internally coherent is important and worth studying.
  • Type-6 gives more useful feedback than a single score by identifying concrete failure types such as contradiction, reasoning avoidance, and unresolved doubt.
Critical weaknesses
  • Type-6 lacks a truly independent validation. The paper mainly compares agreement between different verifiers, but agreement does not mean correctness. More importantly, Type-6 defines what counts as contradiction, revision, or reasoning avoidance, and then uses these same rules to report how often such problems occur. This feels somewhat circular. Without an independent human gold standard, it is hard to know whether Type-6 is actually more reliable.
  • The experimental setup itself may explain much of the disagreement. The evaluation uses only 996 CoTs from a few datasets, generated by two Qwen3 models, so the coverage is limited. The baselines also target different settings (eg. PRMs are mainly designed for mathematical reasoning, while ROSCOE almost always hard-fails here), so disagreement is not very surprising. In addition, Type-6 relies on GPT-5 to extract propositions, operators, and states before the symbolic verifier runs, making it unclear how much of the final result comes from Type-6 itself versus GPT-5 parsing.
Questions for authors
  • The paper template also seems to have some formatting issues (eg. the lack of a running header looks like an oversight).
  • The coverage of the most relevant work is clearly insufficient. The paper uses LINC as its main neurosymbolic baseline but does not discuss closely related methods (eg. Logic-LM, SymbCoT, and LogicAgent). In particular, SymbCoT already includes a verifier for symbolic translation and reasoning chains. What fundamentally distinguishes Type-6 from these methods? Why do the experiments only compare against LINC and leave out these closer alternatives?
Reviewer_WimsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Language models have chains of thought that can be long and wandering. Sometimes the model will assert something, walk back the assertion, circle around it, and might reach a conclusion it never actually argued for. The paper's idea is to treat the trace of these arguments as a set of propositions, each tagged with a truth value, a commitment strength, and whether it is being doubted. They call this Type-6 logic. It then walks through the trace and flags problems, such as if the trace said X and then later not X with no particular reasoning in between, or if doubts were left hanging toward the end of the chain of thought. There is a neurosymbolic verifier that builds a graph of this chain-of-thought trace, and you can, ideally, find where the reasoning went astray. The authors report that the other baselines do not agree with each other, whereas this method is the one others agree with the most.

Critical strengths

The problem is definitely interesting. The authors seem to have put some thought into the vocabulary of failure modes. I thought the aspect of reasoning avoidance (that is, the model stating a conclusion without ever attempting to derive it) was interesting. Generally, this area of research is interesting: finding, from a chain of thought, a graph-level or statement-level audit trail of where the model might have gone wrong, instead of just a binary signal that it is wrong.

Critical weaknesses

I do not think there is sufficient scientific evidence supporting the main claims of the paper, especially empirical evidence. There is an elaborate formal framework, but the empirical evidence showing that this proposed method actually reliably finds these reasoning failures is missing. There are no results to directly measure the accuracy, precision, recall, or practical utility of the verifier's flags. The authors rely instead on agreement and correlation between different verification methods. There is no independent ground truth for reasoning coherence, but one could have been collected: the authors already had trained annotators review all 996 traces, so obtaining human judgements of whether the flagged errors are genuine was feasible. Without such an evaluation, it is unclear whether the errors identified are genuine reasoning failures.

The writing is poor and extremely dense, to the point where the main text has only one table with quantitative results. All the rest of the empirical evidence tables are delegated to the appendix. The reader is left trying to piece together all of this seemingly unrelated evidence from different parts of the paper.

On the theoretical front, I struggle to understand why we need an entirely new logic, given that formal methods have very well-developed systems for representing all of the primary concepts we are looking at in this paper (for example, knowledge, contradictions, and uncertainty).

The authors say they build on dynamic epistemic logic and a few more things, but why do we really need a new logic? What exactly is fundamentally new here in terms of a new logical capability, or is it just combining existing concepts into a new set of rules and notation? Could some other formal system, for example, do the same job? Why is a new formalism necessary?

Questions for authors

I would really like to see a genuine reasoning error that cannot be represented in existing frameworks or informal methods. Again, the baselines are mostly existing chain-of-thought scoring methods, but if the theory is actually contributing mainly a new form of logic, wouldn't it be more informative to compare it with the existing logical formalisms that can already handle uncertainty?

If you're fundamentally proposing a new logic, you have to demonstrate the theoretical properties that distinguish it from established frameworks, and then this paper would be more suited for a venue in formal methods. If you want to do a useful application of existing ideas from logic to chain-of-thought verification, then you need much stronger experimental evidence that this works better than the alternatives.

Reviewer_davKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Type-6 Logic, a formal framework for evaluating the structural coherence of LLM-generated chain-of-thought (CoT) reasoning. Drawing on epistemic, many-valued, and paraconsistent logic, the authors define proposition states, revision rules, and a taxonomy of reasoning failures. A graph-based verifier is developed to identify contradictions, unsupported derivations, and unresolved commitments.

The framework is evaluated on 996 CoT traces across formal and informal reasoning tasks. The authors report substantial disagreement among existing verification methods and use Type-6 to identify structural issues in reasoning traces.

Critical strengths
  1. Relevant research problem. The paper addresses an important distinction between final-answer correctness and reasoning coherence. Evaluating the logical structure of CoT traces is a meaningful research direction.

  2. Structured and interpretable framework. Type-6 provides an explicit representation of proposition states, logical conflicts, and reasoning paths. Its fine-grained diagnostic reports offer potentially useful information beyond aggregate coherence scores.

  3. Implementation and empirical evaluation. The paper presents a formal framework alongside a working verifier and experiments across different reasoning tasks. The released code and data provide a useful basis for further investigation and reproducibility.

Critical weaknesses

W1. Insufficient independent validation of the diagnostic rules and concerns about the path-level implementation.

Type-6's diagnostic decisions depend on predefined state variables, revision conditions, and failure categories. Although these choices are motivated by existing logical theories, the paper does not sufficiently establish their necessity or demonstrate that they reliably distinguish legitimate reasoning from actual reasoning errors. For example, a revision is considered licensed only when specific structural and derivational conditions are satisfied, but the paper does not independently validate whether revisions failing these conditions are genuinely erroneous.

The human annotation study evaluates the correctness of converting natural-language statements into Type-6 representations rather than the accuracy of the final error judgments. Table 5 reports an 85.3% hard-fail rate, while ground-truth labels are unavailable for most splits. It is therefore unclear how many detected failures correspond to genuine reasoning errors.

Further inspection of the released code raises an additional concern. While Section 3.4 describes the Direct Implication Chain as being computed per path, the implementation appears to use a globally computed chain and generate path-level reports by filtering global verification events. It is unclear whether this approach fully preserves the path-specific semantics described in the paper. Overall, both the validity of the diagnostic rules and the correctness of their path-level implementation require stronger evidence.

W2. Mathematical issues in the complexity analysis and insufficient support from the implementation.

In Appendix E, Lemma E.17 concludes that μ∣V∣2B\mu^{|V|}2^B is polynomial under the assumptions μ=O(1)\mu=O(1) and B=O(log⁡∣V∣)B=O(\log |V|). This inference is mathematically incorrect: setting μ=2\mu=2 and B=0B=0 still yields exponential growth. This does not imply that actual CoT graphs necessarily contain exponentially many paths, but the stated argument does not establish the claimed bound.

The O(n)O(n) verification-time claim in Lemma E.20 is also insufficiently justified. The paper explicitly states that existing constraints are reevaluated at each step. Inspection of the released code confirms that _run_backprop iterates over all accumulated constraints without an apparent incremental mechanism for avoiding unnecessary evaluations. When the number of constraints grows with trace length, the total evaluation cost may be quadratic.

Furthermore, the abstract claims "average-case linear time", while Section 4.1 describes the algorithm as "quasilinear", without a clearly specified input distribution supporting the average-case claim. The current theoretical analysis and implementation therefore do not adequately support the stated complexity guarantees.

W3. The experimental comparisons do not sufficiently support the claimed comparative advantage of Type-6.

The evaluation relies primarily on agreement between verification methods rather than independently established diagnostic accuracy. Several baselines also produce nearly constant hard-fail predictions, limiting the interpretability of these comparisons. Moreover, Table 3 reports a best-of Spearman correlation of 0.363 between LLM-Judge and PRM, exceeding their respective correlations with Type-6 (0.248 and 0.312), contrary to the stronger claim in the conclusion.

Inspection of the provided code archive raises a further concern. In scoring/hard_fail.py (_pick_best_of_type6, hard_fail_kappa_matrix_aggregated) and scoring/scalar_corr.py (_pick_best_of_type6_scalar, scalar_rho_matrix_aggregated), the best-of model for each baseline appears to be selected by maximizing agreement with Type-6, with the selected model then reused for all method pairs. This introduces an asymmetric comparison: pairs involving Type-6 are optimized over baseline model choices, whereas baseline–baseline pairs are not independently optimized. The authors should clarify whether this procedure was used to generate the reported results and how it supports the claimed comparative advantage.

Questions for authors

Q1. Validity of the diagnostic rules

Could the authors evaluate the final diagnostic decisions against independent human annotations? In particular, how accurately does Type-6 distinguish genuine reasoning errors from legitimate revisions? Could targeted ablations also establish the necessity of the predefined revision conditions and failure categories?

Q2. Path-level verification

Section 3.4 describes path-specific Direct Implication Chains, whereas the released code appears to use a global implication chain and filtered global verification events. How does this implementation preserve path-specific reasoning states, and what guarantees that path-specific contradictions or unresolved dependencies are not missed?

Q3. Complexity analysis and implementation

Could the authors correct the arguments in Lemmas E.17 and E.20 and clarify how the implementation achieves the claimed complexity despite repeatedly scanning accumulated constraints? What input distribution or assumptions support the "average-case linear time" claim?

Q4. Baseline selection and comparative claims

Does the reported best-of comparison select each baseline model by maximizing agreement with Type-6 and then reuse that selection across all method pairs, as suggested by the released code? If so, how is this asymmetric comparison justified? Could the authors also clarify the definition of “most agreed-with” and reconcile it with Table 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.