- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies SAE features whose activation changes across reasoning boundaries correlate with changes in future outcome probabilities estimated by repeated continuations. It selects features using conservative scores across several correlation variants, evaluates their tracking on unseen sources, and tests sustained interventions along the corresponding decoder directions. The main experiments concern blackmail and whistleblowing decisions, with an exploratory kernel-generation extension. Two whistleblowing interventions increase external-reporting frequency after multiplicity correction, while natural tracking generalization and the controlled blackmail intervention results are less conclusive.
Critical strengths
- The question of which SAE features matter for reasoning is relevant, and combining repeated continuations with feature interventions is a reasonable way to investigate it.
- The paper is careful about distinguishing correlation, generalization, and steering effects. I also appreciate the explicit discussion of the exploratory nature of some experiments.
- The whistleblowing interventions provide a positive result on unseen histories, with correction for multiple comparisons. This is a useful finding even if its scope is limited.
Critical weaknesses
- The advantage of the proposed feature-selection method is not sufficiently established. Showing that selected features can change behavior is useful, but does not demonstrate that the relatively involved discovery procedure is better than simpler selection rules. The baseline comparisons are not strong enough to make that distinction.
- Generalization is a significant weakness. The held-out tracking results are weak, and the informative events come from a small number of independent histories. The distinction between tracking and steering is acknowledged, but the evidence still falls short of showing that the method reliably discovers features that track future outcomes.
- The empirical contribution feels narrow and somewhat fragmented. The strongest evidence comes from whistleblowing, the controlled blackmail results are less convincing, and the kernel study remains exploratory. Taken together, these are interesting case studies, but I do not yet see a broadly validated procedure for identifying features that matter for reasoning.
Questions for authors
- How does feature selection compare with simpler outcome-correlation methods and matched random features under the same budget?
- Are the tracking results consistent across independent histories, or driven by a few particular contexts?
- How stable are the selected features when accounting for uncertainty in the continuation-based outcome estimates?