acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Capacity Beyond Independent Facts: Shared Structure, Abstraction and Frequency

Abstract

Empirical studies find that Transformers store – bits per parameter when trained on uniform random data, but this linear capacity scaling has been established only for independent facts that are spelled out token by token in the training sequences. We ask how knowledge capacity changes when data share compositional structure, when knowledge is latent and expressed through many different surface sequences, and when items occur with unequal frequencies. Using the Random Hierarchy Model, we introduce a measure of stored knowledge that credits a model for what it has learned about a latent knowledge piece rather than about any particular sequence. For verbatim memorization of structured sequences, we find a transition from learning the shared grammar to a regime in which stored information grows linearly with parameter count, with an offset reflecting incomplete grammar learning. The capacity ratio reaches – bits per parameter after epochs and scales with the total number of training exposures as . For latent knowledge, stored information still grows linearly with parameter count even though no training sequence is ever repeated, showing that linear scaling is a property of memorization itself rather than a result of repeated training data. At the same exposure budget, the capacity ratio falls from to bits per parameter as the knowledge becomes more abstract. For unequal frequencies, capacity is allocated to frequent items first, rare items are suppressed even at fixed exposure count, and total stored information departs from a single linear law. An associative memory optimized under frequency-weighted cross-entropy reproduces this suppression and traces it to interference between stored associations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
memorization, knowledge capacity, neural scaling laws, transformers, representation learning, data distribution, associative memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Jsx4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigates the memory capacity of Transformers beyond the storage of independent, random facts. Using a hierarchical random model, it explores the shared structures and latent knowledge that can emerge across diverse sequences of tokens. The authors quantify stored knowledge by measuring the reduction in expected encoding length relative to the full generative distribution. The experiments reveal that the model first undergoes a phase of syntactic learning, after which its capacity scales roughly linearly. The number of bits stored per parameter decreases as the level of abstraction rises, but increases with the number of training exposures. And the high-frequency key-value associations suppress low-frequency ones.

Critical strengths
  1. The latent-knowledge setup is a useful extension of earlier capacity studies. Testing on new realizations of the same latent pieces allows the authors to measure something beyond recall of training strings. The code-length definition also makes the connection to verbatim memorization explicit.

  2. The structured-data results are supported by a substantial set of experiments. The paper varies model size, dataset size, training exposure, and grammar configuration. The additional grammars, precision checks, and keyed-versus-keyless comparison help show that the main result is not confined to one setting. The separation of training, ID, and OOD losses in the latent experiment is particularly informative.

  3. The associative-memory analysis gives a concrete explanation for rare-item suppression. It shows how the frequency-weighted objective favors larger weights for frequent associations, which then interfere with rare ones. This adds useful analysis to the Transformer observations rather than leaving the frequency result purely descriptive.

Critical weaknesses
  1. The claim that abstract knowledge costs more to store is not fully isolated by the experiments. Changing the abstraction level also changes the information per piece, the dataset ceiling, and the difficulty of identifying a piece from its realization. The latent experiments use only one grammar. The probe results suggest that identification overhead explains much of the slope difference, but they do not show that later layers are where the knowledge is stored. I would like to see comparisons with matched information budgets before drawing a general conclusion about the cost of abstraction.

  2. The frequency experiment leaves a training-schedule confound. Increasing exposure to group A while fixing exposure to group B changes both total training time and the spacing of B examples. Appendix H acknowledges this, but the main discussion still attributes the effect mainly to interference. The associative-memory model shows that cross-talk can produce the pattern; it does not establish that this is the mechanism in the Transformers. Keeping the presentation schedule fixed and varying loss weights would make the comparison cleaner.

  3. The fitted scaling laws need more evidence about uncertainty.

Questions for authors

refer to the weakness

3 comments
AuthorsOpenReview ↗

Part 1: Matched information per piece and freeze-and-retrain experiment

We thank the reviewer for the careful review and constructive comments. We revised the paper and uploaded the new version. All section, figure, table and appendix numbers below refer to the revised PDF. The former Appendix H is now App. A.6 (Exposure settings for the Transformer frequency sweep).


W1(a)

"The claim that abstract knowledge costs more to store is not fully isolated ... only one grammar."

Abstract knowledge still costs more to store when the information per piece and the dataset ceiling are matched (App. E.7).

We use three RHMs with (v,c,m,s)=(16,16,4,2)(v, c, m, s) = (16, 16, 4, 2) that differ only in depth, L=6L = 6, 77 and 88. We set ℓmem=1\ell_{\mathrm{mem}} = 1, 22 and 33, respectively, so that one piece carries the same 130 bits in all three grammars. The three knowledge sets therefore have the same ceiling. The capacity ratio still falls with abstraction, from 2.442.44 to 1.491.49 to 0.690.69 bits per parameter. The ratios r2/r1=0.61r_2/r_1 = 0.61 and r3/r1=0.28r_3/r_1 = 0.28 are nearly the same as the values 0.660.66 and 0.280.28 in Fig. 4.

In App. E.6, we also repeat the latent experiments on a second grammar, (v,c,m,s,L)=(16,16,4,2,8)(v, c, m, s, L) = (16, 16, 4, 2, 8), which has two more levels than the grammar of Fig. 4. At the same ℓmem\ell_{\mathrm{mem}}, its pieces carry four times more bits. Yet both grammars give r=1.6r = 1.6 at ℓmem=2\ell_{\mathrm{mem}} = 2, and 0.670.67 and 0.690.69 at ℓmem=3\ell_{\mathrm{mem}} = 3. The capacity ratio therefore follows ℓmem\ell_{\mathrm{mem}}, not the number of bits per piece.


W1(b)

"... [the probe results] do not show that later layers are where the knowledge is stored."

We added a freeze-and-retrain experiment. It directly shows that the early blocks do not store the knowledge and the later blocks do.

In Sec. 5.3, a linear probe classifies the piece from the residual stream after each Transformer sublayer. We define the identification sublayer j∗j^\ast as the first sublayer at which the probe accuracy reaches 0.9.

In the new experiment, we take a trained model and freeze the embedding and the blocks up to a chosen block. We reinitialize the later blocks and train them on a new knowledge set K′K' from the same grammar. K′K' has the same size nn as KK and shares no piece with it. At ℓmem=2\ell_{\mathrm{mem}} = 2, j∗j^\ast lies in block 2. Freezing up to the block that contains j∗j^\ast changes the stored information by less than 1.5% (Fig. 6b, App. F.2). The change is 0.6M of 44M bits at nlayer=8n_{\mathrm{layer}} = 8 and 0.2M of 61M bits at nlayer=10n_{\mathrm{layer}} = 10. By contrast, each further frozen block removes substantial stored information: 5–8M bits for block 3 and about 10M bits for each later block. The blocks up to j∗j^\ast therefore store nothing specific to KK. Although trained on KK, they work for K′K' as well as blocks trained on K′K' from scratch.

Let N>j∗N_{>j^\ast} be the number of parameters in the blocks after j∗j^\ast. We fit the stored information against N>j∗N_{>j^\ast} at ℓmem=2\ell_{\mathrm{mem}} = 2. The slopes are 2.02.0, 2.12.1 and 2.12.1 bits per parameter for nlayer=6n_{\mathrm{layer}} = 6, 88 and 1010. Against the total parameter count NN, the capacity ratios are 1.31.3, 1.61.6 and 1.71.7. Counting only N>j∗N_{>j^\ast} therefore removes the dependence of the capacity ratio on Transformer depth. Identification difficulty accounts for part of the dependence on ℓmem\ell_{\mathrm{mem}}. Against N>j∗N_{>j^\ast}, the slope falls from 2.72.7 to 2.12.1 to 1.21.2 bits per parameter for ℓmem=1\ell_{\mathrm{mem}} = 1, 22 and 33. Against NN, the capacity ratio falls from 2.42.4 to 1.61.6 to 0.70.7 (Sec. 5.3, App. F.3).

AuthorsOpenReview ↗

Part 2: Loss-weight control and confidence intervals

W2(a)

"... changes both total training time and the spacing of B examples ..."

A loss-weight control with a fixed presentation schedule reproduces the suppression.

In Sec. 6.1, each sequence of the rare group BB is shown 1,000 times and each sequence of the frequent group AA is shown 1,000ρ1{,}000\rho times. As suggested, we ran a control that keeps the presentation schedule fixed and varies only the loss weights (App. A.6.1). Every sequence of both groups is shown 1,000 times, so the number of training steps and the spacing of BB are the same as in the uniform run (ρ=1\rho = 1). We train on the weighted loss

L=1∣B∣∑i∈Bwg(i)CEi,wA=2011,wB=211,\mathcal{L} = \frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}} w_{g(i)}\mathrm{CE}_i, \qquad w_A = \frac{20}{11}, \quad w_B = \frac{2}{11},

where B\mathcal{B} is the batch, CEi\mathrm{CE}_i is the cross-entropy of sequence ii and g(i)g(i) is its group. The ratio wA/wB=10w_A/w_B = 10 matches the ρ=10\rho = 10 run of Sec. 6.1, and the weights average to 1, so the overall scale of the loss is the same as in the uniform run. Between N=0.66N = 0.66M and 2.92.9M, where the suppression occurs, the control reproduces the ρ=10\rho = 10 run. The suppression therefore comes from the frequency weighting of the loss, not from the longer training or the spacing of BB. In larger models, where BB recovers, the control run stores less about BB than the ρ=10\rho = 10 run, since it trains for 5.5 times fewer steps.


W2(b)

"... it does not establish that this is the mechanism in the Transformers."

We agree that the associative memory does not establish the mechanism in Transformers.

The revised abstract, introduction and Sec. 6 state that the associative memory (Sec. 6.2, App. G) reproduces the suppression through cross-talk between stored associations, and no longer claim that cross-talk is the mechanism in Transformers.


W3

"The fitted scaling laws need more evidence about uncertainty."

We added 95% BCa wild-bootstrap confidence intervals and seed replicates.

For Fig. 2, r=2.99[2.66,3.32]r = 2.99\enspace[2.66, 3.32] bits per parameter at m=2m = 2 and 2.84[2.77,2.90]2.84\enspace[2.77, 2.90] at m=128m = 128. For the abstraction levels in Fig. 4(b), r=2.38[2.15,2.57]r = 2.38\enspace[2.15, 2.57], 1.58[1.47,1.68]1.58\enspace[1.47, 1.68] and 0.67[0.62,0.71]0.67\enspace[0.62, 0.71] at ℓmem=1\ell_{\mathrm{mem}} = 1, 22 and 33, and the intervals do not overlap. The intervals of all other fits are in Table 3 (App. B.2). We also repeat the main measurements with three seeds (App. B.3). The seed changes both the model initialization and the training data. The relative standard deviation of the stored information across seeds averages 1.2% for verbatim memorization after 1000 epochs and 7% for latent memorization.


Presentation

We clarified the definitions in the main text, redrew several figures to make them clearer, revised the main-text captions and reorganized the appendix by topic.

Thank you for the detailed response and the additional experiments. The matched-information comparison directly addresses my concern about differences in information per piece and dataset ceilings. I appreciate the revised distinction between the associative-memory explanation and the mechanism operating in Transformers, as well as the added confidence intervals and seed replicates. Overall, the response substantially addresses my original concerns and strengthens the empirical support for the paper. I maintain my Weak Accept recommendation.

Reviewer_NH7FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work approaches the question of the memorization capacity of a transformer from the perspective of paraphrasing. Rather than viewing the training tokens as iid independent facts, they use the random hierarchy model to construct a latent knowledge memorization task, where a single knowledge piece is represented by many sequences. For this latent knowledge the bit-rate per parameter is still linear. More abstract knowledge is more expensive to store so the bit-rate is smaller. This paper improves the theoretical understanding of a more realistic model of how knowledge is represented and saved in transformers.

Critical strengths
  • S1: Considering a specific aspect of memorized knowledge, the fact that it may be "paraphrasable" according to some grammatical rule, is new, and viewing the random hierarchy model as a model of this is a smart framing which allows for an analysis.
  • S2: The model is further interesting because it allows for understanding at different abstraction levels, which emerge naturally from the model and to my knowledge have not been done before, but seem relevant for our understanding of language.
  • S3: The writing with examples combined with clear notation is very instructive.
Critical weaknesses
  • W1: The linear capacity is for abstract knowledge is perhaps not such a surprise, and it is unclear how the results translate to real models in practice. There, the per token bit-rate in the parameters can be measured, but we do know how to measure the latent knowledge content in the data. It is unclear how much of the input data of real models are concerned by the type of paraphrasing that is modeled here.
  • W2: The lines between generalizing (the grammar) and memorizing become slightly blurry in this model. While the onset of pure memorization is tracked via test-train discrepancy, it is not clear that mechanistically the model first generalizes the grammar, and then memorized the knowledge. There are beginnings of an investigation on this in the paper, but incomplete (as admitted in the limitation section).
  • W3: The generality of the model is unclear.
  • W4: The code for reproducing the results is not available, and no intentions to make it public have been declared.
Questions for authors
  • Q1: Can you expand on the observation in L356 "abstract knowledge is more expensive to store per bit"? I would like to understand better if this is something that could be explained by an alternative hypothesis that assumes that lower levels are being memorized instead.
  • Q2: (Re:W1) There are some attempts to create measurable multi-hop facts as a version of more abstract knowledge representations (perhaps in the style of https://arxiv.org/pdf/2504.03635). I would be good to clarify the connections to this work.
  • Q3: Can Table 1 be extended predictably to larger levels (level 2) when m>128?

Further work:

  • I think it could also be interesting to understand how different levels of abstraction interact when being memorized jointly.

Minor Comments:

  • L23 reads weirdly specific
  • Re:S3: Figure 1 could perhaps mark what a piece of knowledge is in this setting for different abstraction levels. Define (b) NTP in the caption.
  • I find the comment on identifying and completing knowledge being separated in the network (L368) very interesting, and perhaps it could be expanded on a bit in the introduction, as the grammar part essentially introduces a second task and this is not overly emphasized earlier.
  • L314,318 fresh = random iid?
  • Fig. 5 rho is not defined in caption
  • L122 Save s ?
  • It could be useful to understand on what transformer architectures (depth, heads, embed dim). Sometimes it is indicated, but e.g. Figure 1 b misses explanations for the brackets next to the points
4 comments
AuthorsOpenReview ↗

Part 1: Implications of the findings and freeze-and-retrain experiment

We thank the reviewer for the careful review and constructive comments. We revised the paper and uploaded the new version. All section, figure, table and appendix numbers below refer to the revised PDF. The former Fig. 5 is now Fig. 7, the former Fig. 1(b) is now Fig. 8 in the appendix, and the former Table 1 is extended in Table 6 (App. D.1).


W1(a)

"The linear capacity is for abstract knowledge is perhaps not such a surprise ..."

Linear capacity holds even when no training sequence repeats, and abstract knowledge costs more parameters per stored bit. Neither follows from prior capacity measurements.

  1. The linear decrease of the training loss in the overfitting regime is general. Prior capacity measurements use unstructured data or facts with a fixed token form. To our knowledge, this scaling had not been measured on data with shared compositional structure. We find the same linear decrease with NN on such data once the model overfits (Sec. 4).
  2. Linear scaling does not rely on repeated token strings. For latent knowledge, stored information grows linearly with NN even though no training sequence is ever repeated (Sec. 5).
  3. The capacity ratio is not a universal constant. It falls from 2.42.4 to 0.70.7 bits per parameter as the knowledge becomes more abstract. It also grows with Transformer depth, which verbatim memorization does not show. Capacity estimates from facts with a fixed token form therefore do not directly apply to abstract knowledge. The capacity ratio grows with depth because the early blocks only identify the piece and do not store it. Counting only the parameters of the later blocks, which store the pieces, removes this depth dependence (Sec. 5.3, W2(a)).

W1(b)

"... it is unclear how the results translate to real models in practice."

We agree that this is an important open question, and we discuss it in the Limitations section. Our measure requires the marginal M\mathcal{M} of the generating process. This marginal is unknown for natural text, so the measure cannot yet be applied to real models.


W2(a)

"... mechanistically ... incomplete (as admitted in the limitation section)."

We added a freeze-and-retrain experiment. It directly shows that the early blocks do not store the knowledge and the later blocks do.

In Sec. 5.3, a linear probe classifies the piece from the residual stream after each Transformer sublayer. We define the identification sublayer j∗j^\ast as the first sublayer at which the probe accuracy reaches 0.9.

In the new experiment, we take a trained model and freeze the embedding and the blocks up to a chosen block. We reinitialize the later blocks and train them on a new knowledge set K′K' from the same grammar. K′K' has the same size nn as KK and shares no piece with it. In these models at ℓmem=2\ell_{\mathrm{mem}} = 2, j∗j^\ast is the attention sublayer of block 2. Freezing up to the block that contains j∗j^\ast changes the stored information by less than 1.5% (Fig. 6b, App. F.2). The change is 0.6M of 44M bits at nlayer=8n_{\mathrm{layer}} = 8 and 0.2M of 61M bits at nlayer=10n_{\mathrm{layer}} = 10. By contrast, each further frozen block removes substantial stored information: 5–8M bits for block 3 and about 10M bits for each later block. The blocks up to j∗j^\ast therefore store nothing specific to KK. Although trained on KK, they work for K′K' as well as blocks trained on K′K' from scratch.

As suggested in the minor comment on L368, the introduction now lists this separation as a contribution.

AuthorsOpenReview ↗

Part 2: Grammar-first learning, generality and code

W2(b)

"... it is not clear that ... the model first generalizes the grammar, and then memorized the knowledge."

In our setup, generalizing the grammar and memorizing the knowledge are two separate parts of the task. To predict fresh realizations of a piece, the model must learn the grammar below ℓmem\ell_{\mathrm{mem}}. This grammar is shared by all pieces, including those outside KK. Which pieces are in KK, in contrast, can only be memorized.

The model learns the grammar before it memorizes the knowledge, both across model sizes and during training.

  • Model size (Sec. 4, Sec. 5.1). In the verbatim setting, the training and test losses of small models agree, so these models store nothing specific to the training sequences. Memorization begins at NmemN_{\mathrm{mem}}, near the minimum of the test loss. At that size, the test offset matches the prediction for the grammar levels that the nn training sequences allow (Tables 1 and 6). In the latent setting, the training, ID and OOD losses of small models coincide at about 1.431.43 nats per token (Fig. 4(a)). This is close to the loss 1.411.41 of the reference M\mathcal{M}, which knows the grammar but no piece. These models have learned the grammar and nothing about KK. Above NmemN_{\mathrm{mem}}, the OOD loss rises while the training and ID losses keep falling, so the model starts to memorize KK.
  • Training time (App. E.1, Fig. 13). Throughout training, models too small to memorize keep the test loss within 4% of the training loss for latent memorization and within 10% for verbatim memorization. Larger models also learn the grammar first. Their test loss drops to within 3% of its minimum before memorization begins, in all latent runs and in most verbatim runs. On average, this minimum lies within 0.2% of the loss predicted for the learned grammar levels for latent memorization and within 5% for verbatim memorization.

W3

"The generality of the model is unclear."

We agree that the RHM is a simplified model of language. Bringing knowledge measurements closer to natural language is an important next step. The RHM captures two properties of knowledge that fixed-form facts lack. One piece appears in many surface forms, and pieces exist at several levels of abstraction. Prior capacity measurements use random sequences or facts with a fixed token form, which have neither property. The RHM is therefore a step from these settings toward natural language. Within this model, our results hold on eight additional verbatim grammars (App. E.4) and on two additional latent grammars (App. E.6, App. E.7).


W4

"The code for reproducing the results is not available ..."

The repository is at https://anonymous.4open.science/r/kcap-code-439E/

It contains the training script, the dataset generation, the evaluation script and a trained checkpoint. We will release the complete code, including the analysis scripts, publicly in the final version.

AuthorsOpenReview ↗

Part 3: Questions

Q1

"... could be explained by an alternative hypothesis that assumes that lower levels are being memorized instead."

In Sec. 5.2, the capacity ratio rr decreases as the knowledge becomes more abstract. A more abstract piece therefore requires more parameters per stored bit. This remains true when every piece carries the same 130 bits (App. E.7), so the effect is not due to abstract pieces carrying fewer bits.

Lower-level memorization cannot explain the higher cost, because the model stores nothing specific to the training strings. In latent memorization, the training loss and the in-distribution test loss are the same (Fig. 4(a), App. B.1), so the lower-level content of the training strings is not memorized. The lower-level structure shared by the training sequences and fresh realizations is the grammar. Learning the grammar changes the offset Δ\Delta and does not change the slope rr.


Q2

"... measurable multi-hop facts ... clarify the connections to this work."

Wang et al. (2026) pretrain models on synthetic knowledge graphs and look for the smallest model that can infer missing edges by multi-hop reasoning. They find that this minimal size scales with a graph search entropy. In their setting, the target facts can be inferred from the stored ones. Our work instead asks how much a model stores about pieces that cannot be inferred, since the grammar does not say which pieces belong to the knowledge set. We added this comparison to Related Work.


Q3

"Can Table 1 be extended predictably to larger levels (level 2) when m>128?"

The prediction extends to level 2 once the number of sequences nn exceeds the level-2 threshold n2n_2. For the m=128m = 128 grammar of Table 1, n2=1.7×107n_2 = 1.7\times10^{7} sequences, beyond our compute budget. We tested level 2 on a smaller m=64m=64. The grammar is (v,c,m,s,L)=(4,4,64,8,3)(v, c, m, s, L) = (4, 4, 64, 8, 3). We use n=400,000n = 400{,}000 sequences, which exceeds n2=131,586n_2 = 131{,}586. Its test offset is 8.88.8 bits per sequence, close to the predicted Δ2=8\Delta_2 = 8 (Table 6). We extended the former Table 1 into Table 6. It compares the predicted and measured offsets for thirteen runs.


Further work

"... how different levels of abstraction interact when being memorized jointly."

This is an interesting direction, and our setup supports it directly. We leave it to future work.


Minor comments

All minor comments are addressed in the revised version. We answer the two questions here.

L314, 318: fresh realizations are i.i.d. samples from Q(⋅∣x(i))\mathcal{Q}(\cdot \mid x^{(i)}), distinct from the training realizations.

Architectures: App. A.2 gives the depth, width and number of heads of all models.

Dear authors, Thank you for your detailed response and the updates and clarifications in the manuscript. I stand by my recommendation for acceptance.

Reviewer_iUTPOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how Transformer knowledge capacity depends on shared structure, abstraction, and frequency. Using the Random Hierarchy Model, the authors define stored knowledge through code-length reduction on fresh realizations of latent knowledge. They observe approximately linear capacity scaling within the memorization regime for both verbatim and latent knowledge, with lower capacity ratios at higher abstraction levels. Separate experiments on nonuniform key–value data show preferential allocation to frequent associations, supported by a simplified associative-memory analysis.

Critical strengths
  1. The framework distinguishes memorizing particular sequences from learning latent knowledge expressed through multiple realizations. Evaluating unseen realizations of previously encountered knowledge provides a useful extension of conventional memorization studies.

  2. The paper combines controlled experiments with interpretable analyses. Additional grammar configurations support the structured-data findings, while the associative-memory model offers a concrete explanation for low-frequency suppression.

Critical weaknesses
  1. The interaction between shared structure and unequal knowledge frequencies remains unexplored. The RHM experiments use uniformly sampled production rules and equal exposures per knowledge piece, whereas the frequency experiments consider only unstructured key–value associations. It therefore remains unclear whether the reported scaling behavior extends to structured or latent knowledge with unequal frequencies, where shared representations and competition between knowledge groups may affect capacity allocation.
Questions for authors
  1. Within the RHM, what happens when latent knowledge groups associated with different structural branches are sampled at unequal frequencies? Does approximate linear scaling persist for each group, or do low-frequency groups exhibit acquisition thresholds? This would connect the structured-knowledge results with the frequency-dependent allocation findings.

  2. Does the lower capacity ratio for more abstract knowledge persist with increased exposure? Could the authors evaluate multiple exposure budgets and intermediate checkpoints to distinguish slower learning from a persistent increase in storage cost?

2 comments
AuthorsOpenReview ↗

Frequency groups in latent memorization and exposure budgets

We thank the reviewer for the careful review and the positive assessment of our work. We revised the paper and uploaded the new version. All section, figure, table and appendix numbers below refer to the revised PDF.


W1

"The interaction between shared structure and unequal knowledge frequencies remains unexplored."

We added an experiment with unequal frequencies on structured, latent knowledge (App. E.9). Frequent knowledge suppresses rare knowledge there as well.

We give the details in Q1. The RHM still samples its production rules uniformly. Grammars with skewed rule frequencies are listed as future work in the Limitations.


Q1

"... Does approximate linear scaling persist for each group, or do low-frequency groups exhibit acquisition thresholds?"

Approximate linear scaling persists for each group, and the low-frequency group also shows an acquisition threshold (App. E.9, Fig. 23).

We use the RHM of Fig. 4 at ℓmem=2\ell_{\mathrm{mem}} = 2 with n=750,000n = 750{,}000 pieces. We split the pieces by root class into two groups of equal size, so the two groups lie in different branches of the grammar. Group AA is sampled five times as often as group BB, with the same total number of training steps as the uniform run. Group AA grows at about 1.0 bits per parameter up to N=14.2N = 14.2M and then saturates. The stored information about group BB is negative for the two smallest models. It then grows at 0.60 bits per parameter between N=14.2N = 14.2M and 31.931.9M. At N=39.4N = 39.4M, group AA reaches 99.6% of its ceiling and group BB reaches 89.4%. The total stored information lies below that of the uniform run at every model size. So far we have measured a single frequency ratio. We are running additional model sizes and will extend the experiment to more frequency ratios in App. E.9.


Q2

"Does the lower capacity ratio for more abstract knowledge persist with increased exposure? ..."

The lower capacity ratio persists with more exposure.

We trained the same model, (nlayer,dembd)=(8,384)(n_{\mathrm{layer}}, d_{\mathrm{embd}}) = (8, 384), at ℓmem=1\ell_{\mathrm{mem}} = 1, 22 and 33 for 1000 and 2000 epochs (App. E.3). Doubling the number of epochs raises the final rr by only 3–6% at every level. The ratio rℓ/r1r_\ell/r_1 agrees within 0.01 between the two budgets (Table 7). During the 2000-epoch runs, r1>r2>r3r_1 > r_2 > r_3 at every evaluation after the first 100 epochs (Fig. 16). The gap is therefore a persistent storage cost, not slower learning.


Presentation

We clarified the definitions in the main text, redrew several figures to make them clearer, revised the main-text captions and reorganized the appendix by topic.

Thank you for your detailed response and the updates and clarifications in the manuscript. I stand by my recommendation for clear acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for the positive reviews and constructive comments. We summarize the main contributions and the revisions below.

Main contributions

  1. We introduce a framework to measure how much a model stores about abstract knowledge without a fixed token form.
  2. We show that on data generated by a hierarchical grammar, models first learn the shared grammar and then memorize the training sequences. In the memorization phase, the stored information grows linearly with the number of parameters.
  3. We show that linear scaling persists at every level of abstraction. The capacity ratio decreases as the level of abstraction increases, from 2.4 bits per parameter for verbatim memorization to 0.7 for the most abstract knowledge we study.
  4. We find that the Transformer first identifies a knowledge piece and then stores it, and that the blocks up to identification do not contribute to memorization. The number of blocks needed for identification grows with the level of abstraction.
  5. With skewed frequencies, frequent knowledge suppresses rare knowledge even when the number of exposures to the rare knowledge is fixed. An associative memory reproduces this suppression through cross-talk between stored associations.

Revisions

  1. Direct evidence that early blocks identify and later blocks store. A freeze-and-retrain experiment shows that freezing the blocks up to identification leaves the stored information about a new knowledge set almost unchanged, while freezing later blocks removes it (Sec. 5.3, Fig. 6(b), App. F.2).
  2. Controls for the cost of abstraction: matched bits per piece, a second grammar and longer training (App. E.7, E.6, E.3).
  3. Learning curves showing grammar before memorization (App. E.1).
  4. Unequal frequencies on the RHM (App. E.9) and a loss-weight control for the key–value frequency experiment (App. A.6.1).
  5. A level-2 grammar (Table 6), confidence intervals (Table 3), seed replicates (App. B.3) and anonymized code (https://anonymous.4open.science/r/kcap-code-439E/).
  6. We revised the writing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.