acceptodds
Withdrawn from ICLR 2027

The Two Clocks Of Representation Learning In Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Representation learning runs on two clocks. The token clock, which tokens a feature answers to, is largely done within the first percent of pretraining. The wiring clock, how it answers within those tokens, calibrates to the end of the run, in step with the direction the feature writes. We track sparse-autoencoder features across checkpoints of three Pythia scales and OLMo-2-1B, splitting each activation profile exactly into token-set and context parts. By step 1k the token set reads r = 0.71–0.78 against its final form, while context selectivity and direction stand at 0.22–0.42 and move together to the end. The window the two clocks open is where a feature’s mature role becomes readable. Containment, the nesting of one feature’s firing set inside another’s, ranks at step 1k which persisting features will organise the mature network, at AUC 0.74–0.89 with nothing fitted, while which features persist stays near chance. Those persistent, high-containment features are the model’s carriers. They cost 1.7–1.9× as much to remove as populations matched on layer, breadth and activation energy, and ablating a containment parent cuts its child’s firing by 11% at the two smaller Pythia scales.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation learning, function before geometry, scaffolding behavior in LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_FHGJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the formation of language model features over the course of training. The authors train sparse autoencoders (SAEs) separately on several LLM checkpoints across pretraining, and then analyze the latents in these SAEs. They have a procedure for pairing latents together between SAEs at different checkpoints, to study how these latents change over training. Most of their analysis focuses on the contexts/tokens that these latents respond to, and how these tokens/contexts change over training. Broadly, they find that the tokens that the latents activate on remains somewhat stable for most of training, but some other properties of the latents, like their decoder directions, does change throughout training.

Critical strengths
  • The paper is motivated, very broadly, to understand the formation of language model features/representations over the course of training. I think this is an interesting topic at the intersection of a few important research agendas.
  • The core idea of the experiments seems sensible to me: train SAEs on several LLM pretraining checkpoints and then pair together latents between these SAEs to see how the features evolve over training.
Critical weaknesses
  • I find the paper to be extremely difficult to read. It has all the markings of being almost entirely AI written. It introduces excessive new terminology that add nothing to the analysis ("clocks", "wiring clock", "carriers", "scaffold", "life history"). Its abstract is almost incomprehensible to me, and reports a variety of metrics that will mean nothing to readers. Overall I am not able to understand much of the paper.
Questions for authors
  • Do I understand correctly that your SAEs are trained on 100k-500k tokens/activations total? In my experience, this is probably 10-1000x too few, and unsurprisingly it seems like your SAEs are a poor model of the network activations here, since replacing activation vectors with the SAE reconstruction increases cross-entropy by at least 1 nat (Table 5). That is a lot!
  • Would you please comment more on the writing process you used for the paper? What fraction of the main text was primarily written by AI? If it was a large fraction, was there a particular reason why you had to rely on AI for this?
Reviewer_KMXbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I have an hard time summarizing the paper, as did the abstract have a hard time doing it. From my understanding, the authors train sparse autoencoders (which are either severely undertrained or badly trained)

Critical strengths

The research idea seems interesting, and the paper results could have been good if presented in a better way.

Critical weaknesses

It is unclear how the proposed results are novel or what they add to our understanding of models. Why should we care about the 'token' set? Do we know if these SAE features are at all related to any underlying fact about the studied model?

The introduced 'containment' concept seems to be related to feature absorption but this second one is never mentioned in the main text and only on the appendix

The paper uses a lot of jargon and does not meaninfully interact with the SAE literature, such that it is hard to understand what the message is, e.g 'Containment, (...), is the relational readout that makes structural position legible inside that early window', what is a relational readout? What is a structural position, what does it mean to be legible?

The reported loss recovered scores are much lower than the >0.95 reported in reports like SAEBench. Given that loss recovered is such a easy metric, its hard to take the results of these SAEs as being correct.

It is unclear if 'Carriers are disproportionately load-bearing'. I agree that a 1.7-1.9x increase in next token loss is important but 2x as much is not 'disproportionately load-bearing'.

Am I misunderstanding that the fraction of alive features trained on the model at step 1k is less than 20%? What is the point of the reported numbers, why should we care about this SAEs?

I believe these problems to be enough to merit rejection and won't comment on results further along the paper. This is not an endorsement of those results.

Questions for authors

How many tokens did each batch old when training the SAEs? Batch size 64 and 800 steps is not enough information to infer the training conditions of the SAE.

Why use unconventional terms in the literature to mean things that already exist in the literature? What do you mean by 'measure load'? What does it mean for a feature answering to a token?

Why matching on the pearson correlation? Why a threshold of 0.5?

Reviewer_m3TzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to study how features are formed and learned during language model pretraining using pythia and Olmo models. Their main finding is that token set features converged early during the training, while the "wiring clock," i.e, context selectivity and direction, continued changing during the training. The authors train multiple SAE per layer and checkpoint pair, and monitor the feature across training evolution, building a "containtment graph".

Critical strengths
  1. The paper aims to study how the features are learned and emerges during training, which is an interesting and timely problem.

  2. The analysis to study how token set and context selectivity are learned during training is interesting.

Critical weaknesses
  1. The paper is difficult to follow -- many parts are too dense to read and understand, and terms like "load-bearing" (line 172) do not add technical meaning, instead making the text convoluted.

  2. I believe the paper and the analysis could be presented in much simpler way --- containtment graphs, stratum, churn rate and other jargons should be avoided.

  3. The scaffold is load-bearing. (line 323)

I am really not sure what the authors mean to convey here; this is one such example among many in the manuscript.

  1. Context direction and selectivity is observed to move/learned together. However, their causal link is not established in the paper .

  2. Only four layers in the models are studied. More information how four layers were selected from all the layers.

  3. I think the same problem can be studied using a simpler setting with synthetic compositonal dataset could provide more tractable and controlled setting.

  4. How is the containment graph is being learned is not clear. the authors should explain how candidate edges are selected etc, and should provide an algorithm to implement it.

  5. There is not explanation or hypothesis provided for the oberserved phenemenon.

Questions for authors

How do you select the four layers from all the layers in the models?

Can you explain in simpler term the main finding of the paper and why it could be useful for the broader community?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.