acceptodds
Withdrawn from ICLR 2027

Thought Latents: Which SAE Features Matter for Reasoning?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A reasoning state can lead to different future outcomes, yet a recorded trajectory reveals only one of them. How do internal features reflect these possibilities as reasoning unfolds? We study thought latents: sparse-autoencoder (SAE) features examined through their relationship to future outcomes. We pair boundary activations with repeated continuations and use changes in activation and outcome probability to identify candidates. Our evaluation separates discovery associations, generalization to unseen sources, and decoder-direction interventions. We evaluate these relationships in blackmail and whistleblowing scenarios, including independent-source tests. On held-out whistleblowing histories, two selected directions increase external-reporting decisions by 12.5 and 11.5 percentage points relative to contemporaneous zero, surviving correction over 16 candidates. These measurements connect internal representations to possible futures while distinguishing natural readouts from the effects of applying decoder directions.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Transformers, Language Models, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nQXGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies SAE features whose activation changes across reasoning boundaries correlate with changes in future outcome probabilities estimated by repeated continuations. It selects features using conservative scores across several correlation variants, evaluates their tracking on unseen sources, and tests sustained interventions along the corresponding decoder directions. The main experiments concern blackmail and whistleblowing decisions, with an exploratory kernel-generation extension. Two whistleblowing interventions increase external-reporting frequency after multiplicity correction, while natural tracking generalization and the controlled blackmail intervention results are less conclusive.

Critical strengths
  1. The question of which SAE features matter for reasoning is relevant, and combining repeated continuations with feature interventions is a reasonable way to investigate it.
  2. The paper is careful about distinguishing correlation, generalization, and steering effects. I also appreciate the explicit discussion of the exploratory nature of some experiments.
  3. The whistleblowing interventions provide a positive result on unseen histories, with correction for multiple comparisons. This is a useful finding even if its scope is limited.
Critical weaknesses
  1. The advantage of the proposed feature-selection method is not sufficiently established. Showing that selected features can change behavior is useful, but does not demonstrate that the relatively involved discovery procedure is better than simpler selection rules. The baseline comparisons are not strong enough to make that distinction.
  2. Generalization is a significant weakness. The held-out tracking results are weak, and the informative events come from a small number of independent histories. The distinction between tracking and steering is acknowledged, but the evidence still falls short of showing that the method reliably discovers features that track future outcomes.
  3. The empirical contribution feels narrow and somewhat fragmented. The strongest evidence comes from whistleblowing, the controlled blackmail results are less convincing, and the kernel study remains exploratory. Taken together, these are interesting case studies, but I do not yet see a broadly validated procedure for identifying features that matter for reasoning.
Questions for authors
  1. How does feature selection compare with simpler outcome-correlation methods and matched random features under the same budget?
  2. Are the tracking results consistent across independent histories, or driven by a few particular contexts?
  3. How stable are the selected features when accounting for uncertainty in the continuation-based outcome estimates?
Reviewer_Zg2eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between sparse autoencoder (SAE) features and future outcomes in language model reasoning. The authors propose a framework that combines SAE activation measurements with repeated reasoning continuations to identify features associated with changes in future outcome probabilities. The framework further evaluates whether these relationships generalize to unseen sources and whether interventions along selected SAE decoder directions can influence model behavior. Experiments on blackmail, whistleblowing, and exploratory code generation scenarios provide insights into the relationship between internal representations and future reasoning outcomes.

Critical strengths

1.The paper addresses an interesting research question by exploring how internal SAE features relate to possible future outcomes during language model reasoning.

2.The proposed framework is interesting, combining repeated continuations with SAE feature analysis to investigate the relationship between internal representations and future decisions.

3.The separation of feature discovery, generalization evaluation, and intervention analysis provides a systematic perspective on understanding reasoning-related representations.

4.The paper includes intervention experiments and quantitative analyses, providing empirical evidence that certain SAE directions can influence model behavior.

Critical weaknesses

1.The novelty of the proposed framework is somewhat limited, as it builds on existing work on reasoning resampling and SAE-based feature steering. The authors should more clearly distinguish their contributions from related approaches.

2.The correlations between SAE activations and changes in future outcome probabilities are relatively weak. Several selected features also show limited predictive performance on unseen sources, raising questions about their reliability.

3.The experimental evaluation focuses mainly on a small number of behavioral scenarios. More diverse reasoning benchmarks and model architectures would strengthen the generalizability of the findings.

4.Although decoder-direction interventions can influence model outputs, the experiments do not establish whether the corresponding SAE features play a causal role in natural reasoning. More targeted interventions would strengthen the mechanistic conclusions.

5.The intervention results are not consistently significant across tasks. In particular, the blackmail results do not remain significant after multiple-comparison correction, while the kernel-generation experiments lack independent held-out validation.

Questions for authors

see weakness

Reviewer_2NaCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the relationship between SAE features and the LLM reasoning processes. The main experiments aim to 1) analyze the mapping from certain SAE features to fixed reasoning patterns, 2) if the mapping generalizes across generations, 3) if modifications on the features could lead to different reasoning on later generations. The paper studies on 3 tasks, Whistleblowing, Blackmail and Kernel generation.

Critical strengths
  1. Figure 1 and Figure 2 are easy to follow and explain main idea of the paper.
Critical weaknesses
  1. The experimental settings are not well organized. There are numbers everywhere in Section 2. For example, much details of the task kernel generation are missing, even in the appendix. What's the relationship between operators and trajectories? Do you eval on the speedup or solely correctness?
  2. Since this is an analysis paper if I understand correctly, I would expect more in-depth findings or conclusions. The current experiments seem to simply replay well-known results without some interesting ones.
  3. The paper uses Qwen3.5-base SAE and applies to Qwen3.6-35B-A3B (not base model). This is indeed a mismatch. The paper says they should have similar weights and only differ from the post training. But it's still confusing that not directly using the matching checkpoint.
Questions for authors

The kernel generation uses a new benchmark recently released, instead of the well-known KernelBench. May I know the reason for this choice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.