- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors investigates the memory capacity of Transformers beyond the storage of independent, random facts. Using a hierarchical random model, it explores the shared structures and latent knowledge that can emerge across diverse sequences of tokens. The authors quantify stored knowledge by measuring the reduction in expected encoding length relative to the full generative distribution. The experiments reveal that the model first undergoes a phase of syntactic learning, after which its capacity scales roughly linearly. The number of bits stored per parameter decreases as the level of abstraction rises, but increases with the number of training exposures. And the high-frequency key-value associations suppress low-frequency ones.
Critical strengths
-
The latent-knowledge setup is a useful extension of earlier capacity studies. Testing on new realizations of the same latent pieces allows the authors to measure something beyond recall of training strings. The code-length definition also makes the connection to verbatim memorization explicit.
-
The structured-data results are supported by a substantial set of experiments. The paper varies model size, dataset size, training exposure, and grammar configuration. The additional grammars, precision checks, and keyed-versus-keyless comparison help show that the main result is not confined to one setting. The separation of training, ID, and OOD losses in the latent experiment is particularly informative.
-
The associative-memory analysis gives a concrete explanation for rare-item suppression. It shows how the frequency-weighted objective favors larger weights for frequent associations, which then interfere with rare ones. This adds useful analysis to the Transformer observations rather than leaving the frequency result purely descriptive.
Critical weaknesses
-
The claim that abstract knowledge costs more to store is not fully isolated by the experiments. Changing the abstraction level also changes the information per piece, the dataset ceiling, and the difficulty of identifying a piece from its realization. The latent experiments use only one grammar. The probe results suggest that identification overhead explains much of the slope difference, but they do not show that later layers are where the knowledge is stored. I would like to see comparisons with matched information budgets before drawing a general conclusion about the cost of abstraction.
-
The frequency experiment leaves a training-schedule confound. Increasing exposure to group A while fixing exposure to group B changes both total training time and the spacing of B examples. Appendix H acknowledges this, but the main discussion still attributes the effect mainly to interference. The associative-memory model shows that cross-talk can produce the pattern; it does not establish that this is the mechanism in the Transformers. Keeping the presentation schedule fixed and varying loss weights would make the comparison cleaner.
-
The fitted scaling laws need more evidence about uncertainty.
Questions for authors
refer to the weakness
3 comments
Part 2: Loss-weight control and confidence intervals
W2(a)
"... changes both total training time and the spacing of B examples ..."
A loss-weight control with a fixed presentation schedule reproduces the suppression.
In Sec. 6.1, each sequence of the rare group is shown 1,000 times and each sequence of the frequent group is shown times. As suggested, we ran a control that keeps the presentation schedule fixed and varies only the loss weights (App. A.6.1). Every sequence of both groups is shown 1,000 times, so the number of training steps and the spacing of are the same as in the uniform run (). We train on the weighted loss
where is the batch, is the cross-entropy of sequence and is its group. The ratio matches the run of Sec. 6.1, and the weights average to 1, so the overall scale of the loss is the same as in the uniform run. Between M and M, where the suppression occurs, the control reproduces the run. The suppression therefore comes from the frequency weighting of the loss, not from the longer training or the spacing of . In larger models, where recovers, the control run stores less about than the run, since it trains for 5.5 times fewer steps.
W2(b)
"... it does not establish that this is the mechanism in the Transformers."
We agree that the associative memory does not establish the mechanism in Transformers.
The revised abstract, introduction and Sec. 6 state that the associative memory (Sec. 6.2, App. G) reproduces the suppression through cross-talk between stored associations, and no longer claim that cross-talk is the mechanism in Transformers.
W3
"The fitted scaling laws need more evidence about uncertainty."
We added 95% BCa wild-bootstrap confidence intervals and seed replicates.
For Fig. 2, bits per parameter at and at . For the abstraction levels in Fig. 4(b), , and at , and , and the intervals do not overlap. The intervals of all other fits are in Table 3 (App. B.2). We also repeat the main measurements with three seeds (App. B.3). The seed changes both the model initialization and the training data. The relative standard deviation of the stored information across seeds averages 1.2% for verbatim memorization after 1000 epochs and 7% for latent memorization.
Presentation
We clarified the definitions in the main text, redrew several figures to make them clearer, revised the main-text captions and reorganized the appendix by topic.
Thank you for the detailed response and the additional experiments. The matched-information comparison directly addresses my concern about differences in information per piece and dataset ceilings. I appreciate the revised distinction between the associative-memory explanation and the mechanism operating in Transformers, as well as the added confidence intervals and seed replicates. Overall, the response substantially addresses my original concerns and strengthens the empirical support for the paper. I maintain my Weak Accept recommendation.
Part 1: Matched information per piece and freeze-and-retrain experiment
We thank the reviewer for the careful review and constructive comments. We revised the paper and uploaded the new version. All section, figure, table and appendix numbers below refer to the revised PDF. The former Appendix H is now App. A.6 (Exposure settings for the Transformer frequency sweep).
W1(a)
Abstract knowledge still costs more to store when the information per piece and the dataset ceiling are matched (App. E.7).
We use three RHMs with (v,c,m,s)=(16,16,4,2) that differ only in depth, L=6, 7 and 8. We set ℓmem=1, 2 and 3, respectively, so that one piece carries the same 130 bits in all three grammars. The three knowledge sets therefore have the same ceiling. The capacity ratio still falls with abstraction, from 2.44 to 1.49 to 0.69 bits per parameter. The ratios r2/r1=0.61 and r3/r1=0.28 are nearly the same as the values 0.66 and 0.28 in Fig. 4.
In App. E.6, we also repeat the latent experiments on a second grammar, (v,c,m,s,L)=(16,16,4,2,8), which has two more levels than the grammar of Fig. 4. At the same ℓmem, its pieces carry four times more bits. Yet both grammars give r=1.6 at ℓmem=2, and 0.67 and 0.69 at ℓmem=3. The capacity ratio therefore follows ℓmem, not the number of bits per piece.
W1(b)
We added a freeze-and-retrain experiment. It directly shows that the early blocks do not store the knowledge and the later blocks do.
In Sec. 5.3, a linear probe classifies the piece from the residual stream after each Transformer sublayer. We define the identification sublayer j∗ as the first sublayer at which the probe accuracy reaches 0.9.
In the new experiment, we take a trained model and freeze the embedding and the blocks up to a chosen block. We reinitialize the later blocks and train them on a new knowledge set K′ from the same grammar. K′ has the same size n as K and shares no piece with it. At ℓmem=2, j∗ lies in block 2. Freezing up to the block that contains j∗ changes the stored information by less than 1.5% (Fig. 6b, App. F.2). The change is 0.6M of 44M bits at nlayer=8 and 0.2M of 61M bits at nlayer=10. By contrast, each further frozen block removes substantial stored information: 5–8M bits for block 3 and about 10M bits for each later block. The blocks up to j∗ therefore store nothing specific to K. Although trained on K, they work for K′ as well as blocks trained on K′ from scratch.
Let N>j∗ be the number of parameters in the blocks after j∗. We fit the stored information against N>j∗ at ℓmem=2. The slopes are 2.0, 2.1 and 2.1 bits per parameter for nlayer=6, 8 and 10. Against the total parameter count N, the capacity ratios are 1.3, 1.6 and 1.7. Counting only N>j∗ therefore removes the dependence of the capacity ratio on Transformer depth. Identification difficulty accounts for part of the dependence on ℓmem. Against N>j∗, the slope falls from 2.7 to 2.1 to 1.2 bits per parameter for ℓmem=1, 2 and 3. Against N, the capacity ratio falls from 2.4 to 1.6 to 0.7 (Sec. 5.3, App. F.3).