- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the formation of language model features over the course of training. The authors train sparse autoencoders (SAEs) separately on several LLM checkpoints across pretraining, and then analyze the latents in these SAEs. They have a procedure for pairing latents together between SAEs at different checkpoints, to study how these latents change over training. Most of their analysis focuses on the contexts/tokens that these latents respond to, and how these tokens/contexts change over training. Broadly, they find that the tokens that the latents activate on remains somewhat stable for most of training, but some other properties of the latents, like their decoder directions, does change throughout training.
Critical strengths
- The paper is motivated, very broadly, to understand the formation of language model features/representations over the course of training. I think this is an interesting topic at the intersection of a few important research agendas.
- The core idea of the experiments seems sensible to me: train SAEs on several LLM pretraining checkpoints and then pair together latents between these SAEs to see how the features evolve over training.
Critical weaknesses
- I find the paper to be extremely difficult to read. It has all the markings of being almost entirely AI written. It introduces excessive new terminology that add nothing to the analysis ("clocks", "wiring clock", "carriers", "scaffold", "life history"). Its abstract is almost incomprehensible to me, and reports a variety of metrics that will mean nothing to readers. Overall I am not able to understand much of the paper.
Questions for authors
- Do I understand correctly that your SAEs are trained on 100k-500k tokens/activations total? In my experience, this is probably 10-1000x too few, and unsurprisingly it seems like your SAEs are a poor model of the network activations here, since replacing activation vectors with the SAE reconstruction increases cross-entropy by at least 1 nat (Table 5). That is a lot!
- Would you please comment more on the writing process you used for the paper? What fraction of the main text was primarily written by AI? If it was a large fraction, was there a particular reason why you had to rely on AI for this?