- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents a controllable study of how recovery of ground-truth features by sparse autoencoders is controlled by handling dead latents, and how it corresponds with the optimal sparsity; additional point of scrutiny is what happens when SAE dictionary size is lower than the number of ground-truth features, and therefore cannot represent all of them, and how it connects to the cross-seed instability problem. Authors subsequently present the method to estimate number of ground-truth features from data and trained SAEs. Alongside other findings, authors found that dictionary size is also a component of optimal sparsity choice.
Critical strengths
S1. The result that recovery can be controlled by handling the presence of dead latents in SAE seem novel to me.
S2. Authors test a method to choose sparsity from cross-seed feature agreement and found that it comes apart with the recovery, they drive away from each other outside recovery-optimal conditions. I think this is nice result, but see W2.
Critical weaknesses
W1. Writing is heavily convoluted due to bad practices in using AI for writing assistance. It is very hard to get to the substance of results and contributions due to it and distinguish it from my interpretation. Most empirical sections are just number and formula dumps, and most connections to prior literature establish what was done in it, but not how the current paper differs or build upon; many terms are used before they introduced. This also hurts reproducibility - although there are many details stated, I'm not sure that a reader would be able to follow the procedures.
W2. Main results seem to be too toy; I found conceptual results to be little explained from first principles, and empirical results to be built on narrow setups with unsatisfactory design. Experiments are conducted on one configuration of toy model generator of feature activations, only number of features is controlled, which is currently below standard (see SynthSAEBench for example). Unclear whether any claim breaks when going beyond setup in the paper, which is important for transfer to real applications. It is unclear whether trained SAEs converged and were trained well enough. On related note, the connection to stability problem is strangely handled: authors use only two runs to establish which features reproduce and which are not. This does not tell us much and is also below the standard of experimental design for any seed-related problem (see Paulo & Belrose and recent Unstable Features, Reproducible Subspaces papers).
Questions for authors
Q1. You propose an approach to count the GT features. Have you compared your approach accuracy with the prior literature on that matter?
Q2. Can you extend your study on different data configurations to test generalization of your findings? I'm mostly interested on whether a broad class of toy models supports your conclusions, and how different parts of it change them (the coactivation structure, for example).
Q3. Is there a way to theoretically explain why your conclusions hold, and use it for prediction? How practical are your conclusions, how do you see them extending towards real-world SAEs?
Q4. Can you please state what is the delta between your work and prior work in this field? What exactly your paper introduces that has never existed and could never be derived from prior work, and what you build upon?
Q5. Can you please explain in plain terms what exactly one atom per feature result means and what its consequences, in your view? I have not understood it well enough.
I support this direction of study because it gives interpretability proper ground, but I'm unwilling to support the paper in its current form. I would like to see substantial rewrite for proper communication of results (which I do not expect during rebuttal) and extension of empirical material, with proper handling of any deltas.
I believe there is no rebuttal from the authors to my review or to any review... Sadly, I have nothing to comment on.. thank you authors for the paper, nonetheless!