acceptodds
Withdrawn from ICLR 2027

Robustness Is Not Reasoning: A Double Dissociation in Latent Chain-of-Thought

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent chain-of-thought methods such as Coconut and CODI replace token-level reasoning with continuous "thoughts," and a growing literature probes whether these thoughts are robust to perturbation and whether they carry the reasoning, often collapsing both questions into a single perturbation test. We show that these are two distinct properties that can be dissociated. We apply three interventions to individual latent thoughts: random perturbation, counterfactual patching, and content-shuffle (replacing a thought with a real thought computed for a different input). We find that a thought's robustness to random noise is largely independent of whether its content is necessary for the answer. On a controlled k-hop modular-arithmetic task (Pythia-160m/410m, multi-seed, accuracy-matched), the fraction of thoughts that are noise-inert rises monotonically with recurrence depth (from 0.00 at depth 3 to 0.87 at depth 9), while the content of those same thoughts remains necessary at moderate depth and only degrades at excess depth, so the "robust and load-bearing" regime is non-monotonic. On a released CODI-Llama-1B model on GSM8K, latent thoughts are simultaneously noise-robust (answers rarely flip under perturbations of 16–64× the thought norm), individually redundant (single-thought ablation costs at most 4.3% accuracy), collectively essential (ablating all thoughts returns accuracy to near the no-thought floor), and content-load-bearing (shuffling content collapses accuracy from 0.436 to 0.165 ± 0.004), together operating as a distributed, error-correcting code that is individually redundant yet collectively essential. Robustness is therefore not evidence of reasoning, and robustness-only diagnostics can misrepresent what latent thoughts do.

Primary area:
interpretability and explainable AI
Keywords:
latent chain-of-thought, continuous chain-of-thought, interpretability, mechanistic interpretability, LLM reasoning, activation patching, robustness, chain-of-thought monitoring

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_o1B8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper intends to distinguish robustness to random noise from content necessity in latent chain-of-thought models. It combines random perturbation, counterfactual patching, and content ablation or replacement to assess these properties separately. Experiments on synthetic modular arithmetic with Pythia-based Coconut and CODI models test how they vary with latent reasoning depth. An additional evaluation of CODI-Llama-1B on GSM8K compares individual and joint thought interventions. The reported results demonstrate that noise robustness alone is insufficient to determine whether latent thought content contributes to the output answer.

Critical strengths

A meaningful distinction between noise robustness and content necessity. The paper highlights that a latent thought’s resistance to random perturbations does not necessarily establish whether its content leads to the answer. Evaluating these properties separately can help distinguish representations whose content has little effect from those that carry useful information while tolerating noise. This is a meaningful diagnostic perspective

Critical weaknesses
  1. The core research gap and contributions are not fully established. It's reasonable to distinguish noise robustness from content necessity. However, the paper motivates this distinction by arguing that prior work often conflates the two through a single perturbation test. This is not adequately supported by the cited studies. [1] already conducts cross-example latent-token swapping in addition to targeted steering and shortcut experiments. [2] uses activation patching alongside decoding and probing to assess the causal role of intermediate states. The paper does not explain which specific inference in these studies is corrected by its analysis, or establish that their different findings arise from diagnostic choices rather than differences in models, tasks, or training. Since the interventions build on existing techniques, clearly identifying the new empirical insight is critical to assessing the contribution.
  2. The key experiments do not fully support the stronger interpretations of the results. The central GSM8K comparison in Figure 1 contrasts random perturbation of individual thoughts with content shuffling of all thoughts. This changes both intervention type and scope, still mixing robustness up with content necessity. Furthermore, single-thought ablation changes 17–31% of answers despite a small net accuracy drop. Cancellation between improvements and errors does not establish that the original computation is preserved through redundancy or error correction. These observations therefore do not substantiate the claimed distributed, error-correcting mechanism without further controls. Although the synthetic experiments provide useful evidence, the real-task analysis covers only one CODI checkpoint on GSM8K. Together, these limitations leave the paper’s broader interpretation of latent reasoning insufficiently supported.

[1] Zhang, Yuyi, et al. "Do latent tokens think? a causal and adversarial analysis of chain-of-continuous-thought." arXiv preprint arXiv:2512.21711 (2025).

[2] Liang, Jia, and Liangming Pan. "Do latent-cot models think step-by-step? a mechanistic study on sequential reasoning tasks." arXiv preprint arXiv:2602.00449 (2026).

Questions for authors
  1. Clarification of contributions. Please see weakness 1. Considering that existing works have already applied cross-example token swapping and activation patching, what specific limitation of these analyses does your study resolve? Which finding establishes a previously unrecognized distinction, and what evidence shows that the apparent disagreement between prior studies reflects their diagnostic choices?
  2. More evidence is needed. Do noise perturbation and content replacement still have different effects when both modify the same thought positions? Figure 1 compares noise added to one thought with content replacement across all thoughts. Thus, the number of affected thoughts may explain the difference. Also, what supports the claim that the model corrects errors? A small accuracy drop does not show error correction when 17–31% of answers change. In addition, the real-task evaluation uses only CODI-Llama-1B on GSM8K. Can you provide results on another model or dataset to test whether noise robustness and content necessity remain distinct? This would help assess whether the main finding extends beyond the current setting.
Reviewer_q7RKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether continuous “thoughts” in latent-CoT models actually contribute to answering. It distinguishes three tests: adding random noise, replacing a thought with one from a minimally changed input, and shuffling thoughts between examples.

On synthetic arithmetic, latent thoughts can tolerate large random perturbations while remaining sensitive to counterfactual replacement. Increasing latent depth increases noise resistance, while dependence on specific thought content eventually declines. On CODI-Llama-1B/GSM8K, shuffling all thoughts reduces accuracy from 43.6% to 16.5%, although ablating individual thoughts has a much smaller net effect. The authors interpret this as a distributed, error-correcting representation.

Critical strengths

The distinction between noise robustness and causal relevance is useful. A representation can ignore random directions while depending critically on particular directions.

Multiple interventions and training seeds improve the analysis. Accuracy matching also reduces the possibility that the depth results simply compare successful models with poorly trained ones.

Critical weaknesses

Shuffle sensitivity does not specifically establish reasoning content. A donor thought is drawn from the marginal activation distribution, but pairing it with another question and trajectory can create an incompatible internal state. Performance loss could reflect disrupted conditioning, answer information, or trajectory consistency.

The contraction explanation remains suggestive. Average attenuation of sampled noise does not establish that the full computation is contractive or follows the proposed exponential decay. The analysis also needs to account for information retained in attention/KV history.

Questions for authors

Does replacing thoughts with those from another question that has the same final answer still hurt accuracy?

Reviewer_TvECOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper mainly studies whether latent CoT can be robust to noise means the thoughts are not actually used for reasoning, it used random noise., and other things to test this, mainly on mod-7 and codi-1B.

Critical strengths

S1. It makes good point that noise robustiness doesnt necessarily mean the latent thoughts are not doing reasoning. S2. The CODI-1B result is interesting to learn that remove one thought barely changes accuracy, but removing all the thoughts causes a much larger drop (but within expectation)

Critical weaknesses
  1. bigges concern is the reliability of depth experiments. The paper claims the training setup is fixed for T=3,6, and 9 , but the scripts actually sue different training steps and learning rates when I read the supplment.. Also, a T=3 run has the same setup as T=6/9. I dont think current results are sufficent to support the depth claim
  2. For COID-1B exp, I think the comparsion is not really fair. The noise test chagnes on thought, but the shuffle changes all six thoughts, not apple to apple comparison.
  3. I also have some concerns on the novelty.
Questions for authors

Q1. can the authors rerun the depth experiments using exact same training setup for T=3,6,9? Q2. For CODI-1B exp, can you compare single-thought noise with single-though shuffle? Q3. Why the clean accuracy is 43.6% instead of 55.65? Q4. how is this work different from the previous latent swaplping and intervention studies?

Reviewer_M9srOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper study whether latent reasoning are robust to perturbation and whether they carry the reasoning. The paper considers three interventions: random perturbation, counterfactual patching, and content-shuffle. The paper shows robustness is not evidence of reasoning.

Critical strengths
  1. The paper dissociate robustness and reasoning in latent chain-of-thouht, they are two aspects rather than corrolated. The paper shows that even if the latent chain-of-thouht is robust, it can be without carrying reasoning content.

  2. The paper applies three intervention to probe the latent CoT, random perturbation for robustness; counterfactual patching and content-shuffle for whether the CoT is necessary for reasoning.

Critical weaknesses
  1. The papers need to further examine more models and real-world datasets.
Questions for authors

Do we figure out the boundary when reasoning and robustness are correlated, and when they are not?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.