- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the faithfulness of chain-of-thought reasoning in large language models using an evaluation framework based on sparse autoencoders. The authors compare residual representations at the answer position for the same question under direct answering and CoT-conditioned answering. At each layer, a shared SAE (sparse autoencoder)maps both representations into a common feature space. CC-SAE, Jaccard similarity, and Prediction-Grounded Recall measure feature alignment. They then ablate the shared active features S_cap and use the drop in the probability of the original prediction, Delta p, to measure their causal contribution. The experiments report weak correlations between feature alignment and ablation effects, differences in layer profiles across models and tasks, and some stability under changes to SAE configurations.
Critical strengths
-
The paper asks whether features that appear under both answering conditions actually support the prediction. Rather than relying entirely on representational similarity, the authors intervene on internal states and measure how removing shared features changes the output. This provides more direct evidence of dependence than feature overlap or cosine similarity alone and offers a practical way to study the relationship between CoT and answer prediction.
-
Using the same SAE for both conditions at a given layer allows feature indices to correspond directly, avoiding the lack of a natural alignment between separately trained dictionaries. This gives the shared set S_cap a clear definition and makes selective ablation straightforward. Relative to the prior method, which trains separate SAEs and injects features, the main contribution is the selection of shared features within one dictionary followed by an ablation test of their contribution.
-
The evaluation spans multiple model families, sizes, and tasks. It includes random feature ablations, norm matching, interventions on both answering paths, sufficiency tests, controls for answer identity and ubiquitous features, and changes to SAE configurations. The weak correlations between alignment and ablation effects, together with the differences in layer profiles, are also useful descriptive findings.
Critical weaknesses
-
The metric measures how much SAE features shared by the two answering conditions affect the original prediction at the answer position. It does not establish a causal link between these features and the reasoning steps expressed in the CoT. The choice of the final answer position rests on the assumption that relevant concepts have already been aggregated there, but this assumption has not been adequately tested through interventions on individual reasoning steps.
-
The feasibility and fairness of the random and norm-matched controls are unclear. Conditions B and D in Section 5.2.1 require drawing as many prediction-only features as there are shared features. However, Table 5 reports an average of 61 shared features and only 39 prediction-only features for Qwen3-8B on LogiQA, and 13.4 versus 4.2 for Llama-3.1-8B. The specified sampling procedure therefore cannot be carried out without replacement for at least some examples. Sampling with replacement and removing duplicates, subtracting the same feature contribution repeatedly, and reducing the sample size define different interventions and cannot be treated as equivalent controls for feature count.
-
The paper interprets the small additional effect of removing non-shared features after ablating the shared set as evidence that non-shared features contribute little. It also uses a high effect ratio to support completeness and dismiss floor effects. This conclusion does not follow in a nonlinear model. Suppose the answer requires two groups of information, and removing either group is enough to bring the answer probability close to zero. Removing the second group after the first would then have little additional effect, even though the second group remains necessary. A small conditional effect or a ratio close to one therefore does not establish that non-shared features contribute little or that probability floor effects are negligible.
-
The probability drop is bounded by the original answer probability. Comparisons across models and tasks therefore require control for baseline confidence and consistent definitions of answer probability and tokenization. For GSM8K, it is unclear whether the metric uses the first answer token or the full answer sequence. The interpretation of layer profiles also goes beyond the evidence. Large effects at many layers may reflect the same information passing through successive computations rather than distributed redundancy or robustness.
Questions for authors
-
Does Delta p respond to the reasoning content of the CoT rather than shared question or answer representations? With the question fixed and the final answer held constant where possible, does the metric remain high when the original CoT is replaced by unrelated reasoning, incorrect reasoning, or text containing only the conclusion?
-
What explains the discrepancies between the ablation definitions and Tables 12 and 14? Under Eq. (8), removing CoT-only features on the direct answering path should leave the residual unchanged. Under Eq. (15), removing S_pred and removing S_cap on the CoT path should be equivalent. Please specify the intervention location, the activation values used, and whether the residual is replaced by an SAE reconstruction.
-
Do the main findings hold under feasible and properly matched control interventions? How is sampling performed when fewer than k non-shared features are available, and how are failed norm matches handled?