- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks whether SAE features learned independently in different language models show statistically meaningful agreement in their activations. The authors argue that existing cross-model matching methods are statistically vulnerable: choosing each feature’s highest-correlating partner inflates similarity, while comparisons with randomly paired features fail to account for this.
To address these issues, the authors introduce a stricter testing procedure that accounts for the matching process, evaluates feature pairs on unseen documents, and controls for shared input properties. They find significant coactivation across nine pairs of models from different families. Their conclusions are limited to coactivation, rather than shared feature identity or computation.
Critical strengths
The paper addresses a methodological weakness in some previous attempts to match SAE features across language models. Selecting the most highly correlated features can exaggerate their apparent similarity. Pairing each feature with its most highly correlated counterpart can exaggerate apparent similarity, and a random-pairing baseline does not account for this. The authors correct for this by repeating the matching process when estimating chance similarity and by testing the resulting pairs on unseen documents. Under these stricter tests, feature coactivation persists across all nine cross-family model pairs on natural text.
Critical weaknesses
First and foremost, the paper's (presumed) over-reliance on AI for writing makes it close to impossible to read.
Besides that, the authors argue that shared document properties, such as length, topic and vocabulary, can cause features to coactivate "with no shared feature behind them". It is unclear why these properties should be treated as confounds. If two independently trained models learn similar topic or vocabulary detectors, that could itself be evidence of feature universality. Appendix L explicitly names “two models’ topic detectors co-firing” as the alternative being excluded. The controls test whether coactivation extends beyond these properties, but they may also discount genuine shared representations. The paper needs a clear definition of “shared feature” and an explanation of why each excluded form of coactivation should not count towards universality.
Questions for authors
What additional evidence would establish feature identity rather than mere coactivation? Both ablation experiments in Appendix K were inconclusive. Could the authors develop a more reliable intervention method, first validating it on features known to have similar causal effects, and then testing whether matched features across models produce similar changes in predictions?
The paper treats coactivation driven by shared document properties, such as topic or vocabulary, as a potential confound. But if two models independently learn features that detect the same topic or words, wouldn't this itself be evidence of feature universality? Could the authors clarify what these controls are intended to rule out, and why such agreement should not count as genuine feature sharing?