- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Prior work on SAE has largely relied on model-internal metrics to evaluate features, often underperforming simple supervised baselines. To address this gap, the authors train vanilla and JumpReLU SAEs on Gemma-2-2B activations from MIMIC-IV discharge summaries and evaluate their associations with 46 selected ICD-9 codes. They conduct a comprehensive comparison against baselines including GemmaScope, covariance-matched random directions, PCA, and TF-IDF n-grams. Their findings reveal a disconnect between prediction and description: while TF-IDF achieves the strongest predictive performance, blind judges rate SAE features as more interpretable at matched correlation levels.
Critical strengths
- This paper effectively fills the a gap in previous intern metrics scoring by scoring features agains labels the model never saw
- The baselines and statistics are carefully chosen to rule out alternative explanations.
- The authors honestly reported the evaluated results on TF-IDF beating them on prediction.
Critical weaknesses
- The strong TF-IDF baseline is never tested on the judge task.
- There are some inconsistencies with the documented approach: section 4.3 says the judge chooses from the true code, seven same-chapter distractors and 'none'. But Appendix D says the slate is the feature's top-5 codes by correlation, three low-correlation negatives and 'none'.
- Some of the baselines may not be accurate to reflect the contribution of the paper: for example, the difference-in-means are fitted on note-level max-pooled activations. And then they are projected to token-level and then max-pooled. This causes inconsistency in evaluation and might introduce the low correlation. A more calibrated baseline can solve this concern.
- In appendix C, GemmaScope has an explained variance (held-out) of -4.21. If the reconstruction error is about 5 times the data's variance, simply outputting activation average would even do better. Although Appendix G investigates recentering, I think a further verification of this checkpoint might make the baseline result more convincing.
Questions for authors
- How does TF-IDF do on the judge task?
- What approach produces the result in table 3 (corresponding to weakness 2)
- Can you calibrate the baselines of GemmaScope and other methods such as difference-in-means?