acceptodds
Under review as a conference paper at ICLR 2027

From Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought Faithfulness

Abstract

Chain-of-thought (CoT) can sound plausible yet be unfaithful to the model's underlying reasoning. Most prior work probes CoT faithfulness through input–output behavior or input attributions, leaving internal computation largely underexplored. We instead cast faithfulness as an internal concept grounding: Does a large language model's (LLM) CoT reasoning engage the same internal concepts that support the LLM's direct prediction, and do the shared concepts causally drive its answer? Concretely, we encode the prediction and CoT passes with a single shared sparse autoencoder (SAE), treating its latent features as concepts and thereby mapping both passes into a shared concept space. Within this space, we design three correlational metrics of concept-level alignment and a causal metric, , which measures the drop in answer probability when the shared concepts are ablated. Across five LLMs and four datasets, concept alignment is generally high, as indicated by the correlational metrics; yet these only identify which concepts are shared, not how much they causally contribute. fills this gap: causal faithfulness varies substantially with model depth, peaking at mid-to-late layers, and model scale reshapes the layer-wise profile. Moreover, causally important shared concepts are not always verbalized in the CoT. These dissociations suggest that faithfulness cannot be reliably assessed from surface-level or representational correspondence alone; assessing it requires causal tests of whether the internal concepts underlying a CoT actually drive the model's prediction.

Primary area:
interpretability and explainable AI
Keywords:
Chain-of-Thought, explanation faithfulness, sparse autoencoder

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ss4UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the faithfulness of chain-of-thought reasoning in large language models using an evaluation framework based on sparse autoencoders. The authors compare residual representations at the answer position for the same question under direct answering and CoT-conditioned answering. At each layer, a shared SAE (sparse autoencoder)maps both representations into a common feature space. CC-SAE, Jaccard similarity, and Prediction-Grounded Recall measure feature alignment. They then ablate the shared active features S_cap and use the drop in the probability of the original prediction, Delta p, to measure their causal contribution. The experiments report weak correlations between feature alignment and ablation effects, differences in layer profiles across models and tasks, and some stability under changes to SAE configurations.

Critical strengths
  1. The paper asks whether features that appear under both answering conditions actually support the prediction. Rather than relying entirely on representational similarity, the authors intervene on internal states and measure how removing shared features changes the output. This provides more direct evidence of dependence than feature overlap or cosine similarity alone and offers a practical way to study the relationship between CoT and answer prediction.

  2. Using the same SAE for both conditions at a given layer allows feature indices to correspond directly, avoiding the lack of a natural alignment between separately trained dictionaries. This gives the shared set S_cap a clear definition and makes selective ablation straightforward. Relative to the prior method, which trains separate SAEs and injects features, the main contribution is the selection of shared features within one dictionary followed by an ablation test of their contribution.

  3. The evaluation spans multiple model families, sizes, and tasks. It includes random feature ablations, norm matching, interventions on both answering paths, sufficiency tests, controls for answer identity and ubiquitous features, and changes to SAE configurations. The weak correlations between alignment and ablation effects, together with the differences in layer profiles, are also useful descriptive findings.

Critical weaknesses
  1. The metric measures how much SAE features shared by the two answering conditions affect the original prediction at the answer position. It does not establish a causal link between these features and the reasoning steps expressed in the CoT. The choice of the final answer position rests on the assumption that relevant concepts have already been aggregated there, but this assumption has not been adequately tested through interventions on individual reasoning steps.

  2. The feasibility and fairness of the random and norm-matched controls are unclear. Conditions B and D in Section 5.2.1 require drawing as many prediction-only features as there are shared features. However, Table 5 reports an average of 61 shared features and only 39 prediction-only features for Qwen3-8B on LogiQA, and 13.4 versus 4.2 for Llama-3.1-8B. The specified sampling procedure therefore cannot be carried out without replacement for at least some examples. Sampling with replacement and removing duplicates, subtracting the same feature contribution repeatedly, and reducing the sample size define different interventions and cannot be treated as equivalent controls for feature count.

  3. The paper interprets the small additional effect of removing non-shared features after ablating the shared set as evidence that non-shared features contribute little. It also uses a high effect ratio to support completeness and dismiss floor effects. This conclusion does not follow in a nonlinear model. Suppose the answer requires two groups of information, and removing either group is enough to bring the answer probability close to zero. Removing the second group after the first would then have little additional effect, even though the second group remains necessary. A small conditional effect or a ratio close to one therefore does not establish that non-shared features contribute little or that probability floor effects are negligible.

  4. The probability drop is bounded by the original answer probability. Comparisons across models and tasks therefore require control for baseline confidence and consistent definitions of answer probability and tokenization. For GSM8K, it is unclear whether the metric uses the first answer token or the full answer sequence. The interpretation of layer profiles also goes beyond the evidence. Large effects at many layers may reflect the same information passing through successive computations rather than distributed redundancy or robustness.

Questions for authors
  1. Does Delta p respond to the reasoning content of the CoT rather than shared question or answer representations? With the question fixed and the final answer held constant where possible, does the metric remain high when the original CoT is replaced by unrelated reasoning, incorrect reasoning, or text containing only the conclusion?

  2. What explains the discrepancies between the ablation definitions and Tables 12 and 14? Under Eq. (8), removing CoT-only features on the direct answering path should leave the residual unchanged. Under Eq. (15), removing S_pred and removing S_cap on the CoT path should be equivalent. Please specify the intervention location, the activation values used, and whether the residual is replaced by an SAE reconstruction.

  3. Do the main findings hold under feasible and properly matched control interventions? How is sampling performed when fewer than k non-shared features are available, and how are failed norm matches handled?

Reviewer_QYmXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an SAE-based framework for evaluating CoT faithfulness by comparing internal features used in direct prediction and CoT-based prediction. It introduces (\Delta p), which measures the drop in answer probability after ablating shared features. Experiments across five models and four datasets show that representational alignment is weakly correlated with causal effects, with substantial variation across layers and architectures.

Critical strengths
  1. Extensive and systematic experiments. The paper evaluates five models across four datasets and includes a range of control experiments and SAE configuration ablations. This breadth makes the empirical analysis relatively thorough and helps assess whether the observed patterns are robust across models, tasks, and experimental settings.

  2. A useful causal perspective on internal representations. The paper goes beyond measuring representational similarity between direct prediction and CoT passes by intervening on their shared SAE features. The finding that concept alignment does not necessarily imply a strong causal effect is interesting and provides a useful perspective for studying the relationship between internal representations and model predictions.

Critical weaknesses
  1. The proposed metric does not sufficiently establish CoT faithfulness. The proposed (\Delta p) measures how much the answer probability drops when shared SAE features are ablated. While the experiments provide evidence that these features causally contribute to the answer, this does not establish that they reflect the reasoning steps expressed in the CoT. The paper lacks direct evidence linking the identified features to the reasoning described in the generated text, which weakens the central claim that the metric measures CoT faithfulness.

  2. The evaluation is limited to cases where direct prediction and CoT prediction agree. Although this choice avoids confounding faithfulness with changes in the predicted answer, it limits the scope of the conclusions. It remains unclear how the proposed metric behaves when the two predictions disagree and whether the findings generalize beyond the selected subset.

Questions for authors
  1. What evidence supports interpreting (\Delta p) as a measure of CoT faithfulness, rather than simply the causal importance of features shared between the two prediction paths? Can the authors establish a more direct connection between these features and the reasoning steps expressed in the CoT?

  2. What fraction of examples is excluded because direct prediction and CoT prediction disagree? Have the authors examined whether the proposed metric remains meaningful for these cases, or can they clarify the scope of the conclusions accordingly?

Reviewer_ScapOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper aims to study the faithfulness of chain-of-thought by casting it as internal concept grounding. They extract concepts associated with no-CoT and CoT predictions from a shared SAE on their residual streams and ask whether the shared concepts explain the prediction and cause the answer. The study defines both correlational and causal metrics and measure them across five LLMs and four datasets and give insight into how they measure and cause the faithfulness behavior in the model.

Critical strengths
  1. A single pretrained SAE per layer makes cross-pass feature overlap well-defined, the ablation in Eq. 8–9 yields a per-instance, training-free quantity at constant cost per layer, and the §5.2 controls address the obvious confounds for an ablation metric. This matters because my objections concern what the quantity measures, not the machinery.

  2. §5.1 shows high correlational alignment (CC-SAE ≈ 0.9) with negligible association to the causal effect (lift 1.08), and the layer-wise framing in §5.3 is a plausible reframing of a quantity usually reported as one number. This would survive even if the faithfulness interpretation of Δp does not, which is why I consider the work worth resubmitting.

Critical weaknesses
  1. Δp (Eqs. 8–9) invervens on the residual stream of the no-CoT prediction and reads out the no-CoT answer probability. The CoT enters only as the index set S_cot that selects which of the prediction pass's own features are removed. Since both passes contain the same question, S∩ can be driven by the question alone and the paper never intervenes on the CoT itself, so nothing establishes that S∩ or Δp depend on the CoT at all. The interpretation in lines 209-213 thus seem unjustified to me: large Δp means the CoT-conditioned pass re-activates the decision features the model uses without any CoT, which is the signature of a CoT that did not change the computation, a post-hoc rationalization could score high. Also, restricting evaluation to items where CoT and no-CoT answers agree selects the cases where CoT demonstrably did not change the answer. Moreover, S_cot is read from a single position, the answer-position residual stream. The claim in §3.3 that concepts surfaced along the reasoning path are “already aggregated” at that position is asserted rather than tested. That position needs only to support a one-token decision, so the computations that distinguish a faithful chain from a rationalized one, which happen at the CoT token positions, need leave no trace there This is decisive for my recommendation because the metric is the contribution: every headline finding is a statement about the quantity in Eq. 9, and if that quantity reflects where the no-CoT answer is computed and intervenable rather than anything about the CoT, the paper does not support its title or abstract.

  2. Δp is an absolute probability difference, bounded by the base answer probability and by how much of the answer the SAE reconstruction carries, yet neither base probabilities nor direct-answer accuracies are reported, and every cross-layer, cross-scale, and cross-architecture comparison uses the unnormalized value, which approaches 1 on ARC-Easy. The one normalized comparison in the text (§5.3) shrinks the Qwen3-1.7B vs 8B gap from roughly 0.15 to 0.90 vs 0.96, so “larger models are more faithful” is largely a confidence effect. The §5.2 controls are underspecified: by footnote 5, S∩ is 33–75% of S_pred, so whenever overlap exceeds 50% the pool S_pred \ S∩ is smaller than k and Conditions B–D cannot be sampled as described, and the overlap between S∩ and the top-k magnitude set, the one number that would separate conceptual reuse from “S∩ is simply the high-magnitude features,” is never reported. No known-unfaithful manipulation (e.g., hint injection) is used to show that Δp moves in the expected direction. This is consequential because the paper’s value proposition is a better faithfulness measure; without discriminative range after normalization, without validation, and with the comparative findings confounded, there is no evidence it improves on existing measures.

Questions for authors
  1. If the chain c is replaced by filler tokens, a shuffled chain, or a chain from a different item with the same answer letter, how much do |S∩|, Jaccard, and Δp change? Could you design experiments that capture the role of CoT computations more effectively?

  2. How does Δp behave on instances with a more reliable faithfulness signal than the agreement set? Specifically: in an injected-hint setting (Turpin et al., 2023; Chen et al., 2025), does Δp separate items where the answer follows the hint and the chain does not mention it from items where the chain verbalizes the hint, or from unhinted controls, and what is the AUROC? And on a test bed with ground-truth labels, such as the meta-evaluation of Gur-Arieh et al. (2026) that the paper cites, or the question-pair setting of Arcuschin et al. (2025) where one member of a pair is rationalized, does Δp assign lower values to the unfaithful cases?

  3. What evidence supports the §3.3 claim that reasoning-path concepts are aggregated at the answer position, e.g., how much of S∩ is also active at CoT token positions versus at question tokens?

  4. Under Eq. 9, a model that answers through the same shortcut with and without a CoT, with the chain purely decorative, could re-activate its no-CoT decision features in the CoT pass and therefore score high Δp. How do the authors reconcile this with §3.5’s reading that large Δp indicates faithfulness and low Δp indicates post-hoc rationalization?

  5. What are the base answer probabilities P(ŷ | r_pred), direct-answer accuracies, agreement rates under footnote 3, and item counts per model and dataset, and what does the per-layer ratio Δp(S∩)/Δp(S_pred) look like? Do the depth, scale, and architecture findings in §5.3 survive normalization, given that the one normalized comparison reported (0.90 vs 0.96) is far smaller than the absolute gap in Figure 4?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.