acceptodds
Under review as a conference paper at ICLR 2027

Why Implicit CoT Fails at Complex Logic: Theory, Benchmark, and Solution

18 views

Abstract

Chain-of-Thought (CoT) has unlocked advanced reasoning abilities of Large Language Models (LLMs) with intermediate steps, yet incurs prohibitive computational costs due to generation of extra tokens. Recent studies empirically show that compressing reasoning steps into latent states, or implicit CoT compression, offers a token-efficient alternative. However, the mechanism behind CoT compression remains unclear. In this paper, we provide the first theoretical analysis of the difficulty of learning to internalize intermediate reasoning steps. We introduce *Order- Interaction* and show that, under uniform attention initialization, the gradient signal identifying relevant inputs decays exponentially with interaction order. Separating this signal from finite-sample fluctuations is ensured by a sample size that grows exponentially with the order. Motivated by the semantic shortcuts that can obscure the evaluation of implicit CoT, we introduce NatBool-DAG, a benchmark of Boolean reasoning expressed in natural language and designed to reduce reliance on such shortcuts. Guided by our theory, we propose ALiCoT (**Al**igned **I**mplicit **CoT**), which aligns latent and explicit reasoning states to enable lower-order computation through latent tokens. Experimental results demonstrate that ALiCoT maintains strong reasoning performance under extreme token compression.

Primary area:
interpretability and explainable AI
Keywords:
Implicit CoT, Transformer, Chain of Thought

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ydzpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why implicit chain-of-thought (CoT) reasoning is hard to learn on complex logical tasks and proposes ALiCoT to address the problem. The authors start with parity problems, studying how interaction order affects learning. They find that, under uniform attention initialization, the gradient signal for identifying relevant inputs gets exponentially weaker as interaction order increases. Meanwhile, the sufficient sample-size bound for distinguishing this signal from sampling noise grows. This finding suggests that intermediate computational states should be preserved when compressing explicit reasoning. The authors also introduce NatBool-DAG, a benchmark that describes Boolean computational graphs in natural language and aims to reduce reliance on semantic shortcuts. For natural-language reasoning, ALiCoT supplements the task loss with a step-wise alignment objective between latent hidden states and pooled representations of corresponding explicit reasoning steps from the same model. Experiments examine the predicted gradient scaling and report fewer training iterations with intermediate-state supervision. The GSM8K results also show improved mathematical reasoning accuracy within the training budgets.

Critical strengths
  1. NatBool-DAG provides a useful setting for studying compressed logical reasoning. Its symbolic construction supplies verifiable answers and explicit derivations.
  2. The empirical benefits extend across multiple model families and scales.
Critical weaknesses
  1. The novelty is limited or insufficiently clarified. ALiCoT is closely related to existing representation distillation and step-level latent supervision methods. Similarly, the theoretical analysis builds on existing parity-learning and scratchpad frameworks [1][2]. While the theory motivates preserving intermediate computational states (meanwhile, the benefit of representation alignment and step-level latent supervision has already been demonstrated by CODI and SimCOT), it does not directly justify the specific alignment mechanism. The choices of max pooling and cosine similarity remain largely heuristic, and the paper lacks comparisons with alternative aggregation and alignment objectives to establish the benefits of these particular choices.
  2. The experimental evaluation is incomplete. The main NatBool-DAG results primarily compare ALiCoT with Imp.Base-1, leaving its effectiveness relative to the closest baselines (CODI and SIM-CoT) insufficiently evaluated. Although these baselines are included in the GSM8K experiments, their low accuracies and the reported non-convergence of Coconut make the comparison difficult to interpret.

[1] Kim, Juno, and Taiji Suzuki. "Transformers provably solve parity efficiently with chain of thought." ICML 2025.

[2] Abbe, Emmanuel, et al. "How far can transformers reason? the globality barrier and inductive scratchpad." Neurips 2024.

Questions for authors

/

Reviewer_3uAEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines why compressing CoT into latent states can make logical reasoning harder. Using a parity task, the authors define Order-r Interaction and show that, under uniform-attention initialization and other assumptions, the initial gradient signal for identifying relevant inputs weakens as the interaction order grows. They then introduce NatBool-DAG, a synthetic Boolean reasoning benchmark written in natural language, and ALiCoT, which aligns latent hidden states with max-pooled representations of gold CoT steps. ALiCoT needs fewer training iterations on 16-bit parity and improves accuracy over Imp. Base-1 on NatBool-DAG across several models. On GSM8K, its accuracy under full CoT compression is 12.74%.

Critical strengths

S1. Clear theoretical setup. The paper uses interaction order to study the effect of removing intermediate reasoning steps. The parity analysis gives a concrete account of how the initial gradient signal changes with interaction order.

S2. Gains on synthetic tasks. ALiCoT improves over Imp. Base-1 on NatBool-DAG, from 77.88% to 95.01% on Qwen3-4B and from 71.36% to 86.85% on Llama-3.2-3B. The 16-bit parity experiment also reports fewer training iterations under strong compression.

Critical weaknesses

W1. Limited results beyond synthetic Boolean reasoning.

Most of the evidence comes from parity and NatBool-DAG, which are both synthetic Boolean tasks. They do not tell us much about whether the method works on other kinds of multi-step reasoning. GSM8K is the only non-synthetic benchmark used to evaluate ALiCoT, where it reaches 12.74% accuracy under full compression. This is higher than the pause-token baseline's 9.40%, but the absolute performance is still very low. Results on reasoning tasks from a wider range of domains, along with direct-answer and full-CoT reference scores, are needed to judge how useful the method is.

W2. Key design choices are not ablated.

The authors use max pooling to summarize each gold CoT step, but do not compare it with mean pooling or using the last token's hidden state. The latent length is fixed at 20 without a sensitivity study. Gold steps are also matched to latent positions in order, yet the paper does not clearly explain how steps are segmented on natural-language datasets such as GSM8K, or test whether this particular matching matters.

W3. The proposed mechanism is not directly tested.

The theory suggests that preserving intermediate states should reduce the effective interaction order and make the useful gradient signal easier to learn. The experiments show better accuracy and faster convergence, but not that ALiCoT actually performs lower-order intermediate computations. The gains could also come from the extra supervision. Probing the latent states for intermediate logical values, or intervening on them and checking the effect on later computations, would help distinguish these explanations.

W4. NatBool-DAG's irreducibility is not established.

NatBool-DAG is designed to test irreducible logical reasoning, but it is unclear whether the generated problems actually satisfy this property. For example, an AND gate can be resolved as soon as one input is false, without evaluating the other branch. As a result, a problem with a deep computation graph may not require all of its reasoning steps. The paper does not examine how often such shortcuts occur or whether the effective interaction order matches the graph depth.

Questions for authors

Q1. The labels in Figures 2–5 are hard to read at normal size. Could the authors make them larger?

Reviewer_NYWuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why compressing intermediate reasoning steps can make implicit CoT harder to learn. It analyzes how higher-order dependencies weaken the initial gradient signal, introduces a natural-language Boolean reasoning benchmark, and proposes ALiCoT to align latent states with explicit reasoning steps. Experiments show improved accuracy under strong compression, mainly on the proposed benchmark.

Critical strengths
  1. The paper offers a useful perspective on why CoT compression can make learning harder: removing intermediate steps may force the model to learn higher-order dependencies directly. The gradient analysis gives a concrete explanation for this effect under the studied assumptions.
  2. The alignment method shows substantial gains on NatBool-DAG across several model families and sizes. These results provide meaningful evidence that supervising latent intermediate states helps under strong compression, rather than expecting them to emerge from answer supervision alone.
Critical weaknesses
  1. My main concern is the gap between the theoretical analysis and the broader claims. The theory studies gradients near uniform-attention initialization, but the paper does not establish that this mechanism explains failures in pretrained LLMs or that alignment improves performance by reducing interaction order.
  2. The strongest results are on the proposed benchmark (which is kinda toy), while the GSM8K comparison is less convincing. Also, Coconut does not converge in the experiments, which is concerning that I'm not sure if the baselines are set up correctly.
Questions for authors
  1. In practice, the attentions are not uniform. How does it impact your theoretical analysis? Is it one of the reasons that the performance on realistic benchmarks + pretrained LMs is not ideal?
  2. The analysis assumes the intermediate CoTs are oracle step-by-step analysis. But in realistic rollouts, CoTs contain incorrect info and self-correction, how does that impact the conclusions here?
  3. Any thoughts on why the baselines didn't converge in your setup, are the findings aligned with the original paper? The current results make it difficult to assess whether ALiCoT improves over existing methods beyond NatBool-DAG.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.