The Clocks of Composition: Built by the Signal, Lost to Weight Decay, Kept by Rehearsal
Abstract
A network can learn a skill it will not keep. In a looped network, a signal on the intermediate state builds two-hop composition over facts seen only alone; when the signal stops, with the training data unchanged, the composition fades, and plain training does not build it (OOD 0.42). We measure the three clocks that govern its life. It is built on the signal's clock: at dense signal rates the time to compose barely depends on weight decay. It is lost on the weight-decay clock: the time to lose it scales as across a tenfold range of learning rates and a 33-fold range of weight decays, and without weight decay no seed loses it. It is kept only by rehearsal above a threshold; with a short second-moment memory, which removes the damage that rare rehearsal steps do under Adam's default, the threshold rises as . The law travels: registered in advance, it predicts the rehearsal threshold of a transformer's fact memory, a different architecture, task and quantity, with no parameter fitted (exponent 2.3). In a transformer with chain-of-thought, composition fails at high weight decay because no training row exercises the out-of-distribution facts where chain-of-thought reads them; presenting every fact once in that position, with no knowledge of the split, restores it from 0.38 to 0.95.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.