acceptodds
Under review as a conference paper at ICLR 2027

How Much Information Can We Fit in a Transformer?

Abstract

How efficiently can transformers store factual knowledge? The answer to this question has implications for both model design and understanding the limits of transformer-based systems. Prior studies observe a linear relationship between the number of facts memorized and the number of model parameters required; substantially worse than what theoretical analyses predict is achievable. We re-examine empirical scaling: through rigorous and systematic experiments, including extensive hyperparameter search via Bayesian optimization, we determine the smallest model that can perfectly memorize a given set of facts, and measure how this minimum size scales with the number of facts. On synthetic datasets of random, independent (subject, relation, object) triples—e.g., (Paris, capital_of, France)—drawn at random with no exploitable structure, we find that memorization capacity, the size of the largest dataset a model memorizes completely, grows exponentially with the transformer dimension: the minimum transformer dimension required to memorize facts grows logarithmically in , and across 11 dataset sizes spanning to a lack-of-fit F-test does not reject the logarithmic fit (with an offset) but it does reject all other tested alternatives including power-law. Along the depth axis, however, at fixed width, the maximum memorizable dataset size grows sublinearly with the number of layers over the measured range. We further extend our exploration to structured data containing symmetric relations or two-hop composition. In this setting, we find that transformers exploit these regularities to further improve capacity. All code is available to reproduce the results at [redacted].

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Memorization Capacity, Scaling Law

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_izFwOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks how small a transformer can be and still memorize N random (subject, relation, object) facts, with every entity a single token. For one-layer GPT-2 blocks, the minimum width grows logarithmically in N, and a lack-of-fit F-test rejects power-law and linear fits but not this law. The authors read this as memorization capacity growing exponentially with width, unlike the linear scaling of earlier empirical work, which they attribute to limited tuning and training. Experiments on depth, structured and Zipfian data, and a second architecture (OLMo-3) extend the core measurement.

Critical strengths
  1. The measurements are done carefully. Each dataset size gets its own hyperparameter search over three seeds, and a lack-of-fit F-test, not R² alone, selects the functional form. Apart from the training budget (Q1), I find the widths credible, so my concerns are mostly about their interpretation.

  2. The parameter accounting is transparent. Table 4 and the "Where the bits live" paragraph (p.9) separate embedding from non-embedding parameters and report bits per parameter with and without embeddings. That makes the data a reusable capacity measurement for models with one embedding row per entity.

Critical weaknesses
  1. Counted in total parameters, the GPT-2 results match prior work rather than beat it. Each entity is a single token with its own embedding row, and for large datasets these rows hold nearly all the parameters (Table 4). Over all parameters, the paper's own analysis on p.9 finds close to 2 bits per parameter beyond the five smallest datasets, in line with Allen-Zhu & Li (2025), who also count embeddings. Yet the introduction claims substantially better scaling than that work, and this mismatch is the main reason for my rating. Per non-embedding parameter, the largest dataset would need several times what a 32-bit weight can hold, so the facts must live mainly in the embedding rows. A roughly constant rate per embedding coordinate then makes width grow logarithmically with the number of facts. The authors keep the embedding table uncapped to avoid an artificial bottleneck, but their vocabulary test (Table 8) still gives each entity its own token and, by my count, yields a similar rate. Lu et al. (2024) and Allen-Zhu & Li, like deployed LLMs, instead spell entity names with several tokens from a fixed vocabulary and have no per-entity rows. Since Lu et al. count only non-embedding parameters, the gain over their linear scaling shows up precisely in the unit that leaves these rows out. By my count, OLMo-3 stores more bits per total parameter than GPT-2 at every dataset size both were trained on, tied embeddings or not, but it has the same per-entity rows. Neither result therefore supports the suggestion that LLMs may hold more knowledge than estimated (p.3 l.108-110), and settling this needs new experiments that could change the main claim. Concretely, I would like to see the width search repeated at a few sizes with multi-token entity names over a fixed vocabulary, or with frozen random embeddings, reporting bits per trainable parameter.
  2. The case against a power law is weaker than it looks. Table 1's decisive rejection of the offset power law depends on a constraint that keeps the fitted width positive for every dataset size, however small, which pins the offset at zero. The paper exempts the pure logarithm from this constraint, and a free-offset power law turns negative only below the smallest dataset measured. My refit without the constraint, using per-seed widths reconstructed from Table 4 and pure error inferred from Table 1, rejects the free-offset form only narrowly, just under the 0.05 threshold. The pure error also comes from the seeds that selected each configuration (App. B, Eq. 3), which can bias it in either direction. The paper's claim that only the logarithm survives the test thus rests on a constrained comparison and a pure-error estimate whose calibration is unclear. I would ask for the free-offset test on the actual per-seed widths, ideally with pure error from fresh seeds.
Questions for authors
  1. In Fig. 4 (cosine), a run narrower than Table 4's minimum width for a similar N converges only after nearly twice the 40,000-iteration cap. What is the minimum width at twice the budget for the largest N? A binding cap would inflate the widths both weaknesses analyze.

  2. Are input and output embeddings tied in GPT-2 and OLMo-3? Table 4 counts one embedding table, so its totals assume they are.

  3. The theoretical constructions the paper contrasts with count parameters, not bits, and their parameter count grows only with the square root of the number of facts. Storing random facts, however, takes at least as many bits as the facts carry, and that amount grows with the number of facts. Even at 32-bit precision, the largest dataset would need about a hundred times more parameters than the square-root count, so any construction whose size grows more slowly than the number of facts must let the precision of each parameter grow with it. It would help to state the precision these bounds assume and to make the comparison in bits.

  4. The structured-data and OLMo-3 results (Fig. 1, Table 6) compare widths at equal N, but symmetric and two-hop data carry less information than uniform data. Bits per total parameter, computed from each dataset's entropy, would show how fully the model exploits structure and how the two architectures compare.

7 comments
AuthorsOpenReview ↗

Reply to Reviewer izFw (1/4)

We thank the reviewer for the careful and detailed review, and in particular for recomputing our results from the reported numbers. We checked the reviewer's computations on our per-seed data, and we report the results below together with the changes we made in the revised version. Please let us know if anything falls short or if any concerns remain. We would really appreciate the opportunity to continue to improve our paper and address additional concerns before the end of the discussion period. Changes in the revised PDF are highlighted in blue. Additional minor revisions are not highlighted, for clarity.

Weakness 1 (bits per parameter with one token per entity)

We thank the reviewer for this suggestion. As the reviewer requested, we added the Uniform-NL\texttt{Uniform-NL} experiment (Sections 3.6 and 4.5 and Appendix E of the revised version). In this experiment, the entities and relations are real-world phrases that each consist of several tokens, and the vocabulary of tokens is fixed.

Each fact of Uniform-NL\texttt{Uniform-NL} is written as a natural-language declarative sentence, for example "Shana Bald works at Tim Hortons", and tokenized with the GPT-2 tokenizer. The minimum transformer dimension needed to memorize a dataset of size NN behaves similarly to the Uniform\texttt{Uniform} experiment and grows sublinearly with NN. A lack-of-fit F-test rejects a linear fit (p=6.0×10−6p = 6.0\times10^{-6}, Table A). Since the revised version includes this experiment, we removed the sentence in the Limitations and future work paragraph that named natural-language facts as a next step.

Table A. Minimum transformer dimension on Uniform-NL\texttt{Uniform-NL} (L=1L = 1, m=3m = 3, averaged over 3 seeds), with the Uniform\texttt{Uniform} result of Table 4 for comparison.

NN45901803607801,5603,1206,240
Uniform-NL\texttt{Uniform-NL}4.004.335.677.3310.0013.6717.3326.67
Uniform\texttt{Uniform}3.334.005.006.679.0010.3312.6714.33

Moreover, we believe that we were not sufficiently clear in the submission version of our analysis of embedding and non-embedding parameters, and we rewrote this part of Section 5 in the revised version (starting with the paragraph "Embedding versus non-embedding parameters"). The rewritten part contains the following text.

"The number of embedding parameters does not have an effect on the transformer's information capacity. Consider the Uniform-NL\texttt{Uniform-NL} experiment, where the embedding table has the 50,257 rows of the GPT-2 vocabulary at every dataset size. The datasets in this experiment contain at most 1,764 distinct tokens, so at least 48,493 of these rows correspond to tokens that do not occur in any fact. Therefore, the number of bits divided by the total number of parameters is inappropriate for measuring information capacity. Furthermore, we define the capacity of a transformer as the largest dataset size that it can memorize completely (Section 3.1), and we control for how the facts are embedded, which is a confounding factor."

AuthorsOpenReview ↗

Reply to Reviewer izFw (2/4)

Weakness 2 (power-law with a free offset)

We thank the reviewer for the careful recomputation. We conduct additional analyses without the positivity constraint and find that our main claims remain supported. On our per-seed minimum dimensions, the power-law with a free offset is rejected on the 11 dataset sizes that we test in the submission (p=0.046p = 0.046), as the reviewer found. To provide more statistical evidence for the comparison of the power-law and logarithmic fits, we added the dataset size N=24N = 24 (8 entities) to Table 1, Figure 1 and the Abstract of the revised version. With this dataset size, the power-law with a free offset is rejected with p=0.0010p = 0.0010, and the logarithmic-with-offset fit is the only fit that is not rejected (p=0.53p = 0.53). The power-law with a free offset is also rejected (p=0.0096p = 0.0096) when capacity is measured as the maximum memorizable dataset size at each dimension (Appendix E.1).

To make the role of the positivity constraint clearer for future readers, we added the unconstrained fit to Tables 1 and 9 of the revised version and explained the constraint in Section 4.1 and Appendix B.2. In Section 4.1 (Lines 361–363), we added "We require the fitted dd to be positive for every N>0N > 0 (positivity constraint) in every fit where this constraint applies. Without this constraint, the fitted power-law with an offset can become indistinguishable from the logarithmic fit (see Appendix B.2)." In Appendix B.2, we added "The power-law with an offset, f(N)=a(Nb−1)/b+c0f(N) = a(N^b - 1)/b + c_0, approaches alog⁡N+c0a\log N + c_0 as b→0b \to 0, so it contains the logarithm as a limiting case. On Symmetric\texttt{Symmetric} and on OLMo-3, the unconstrained fit drives the exponent bb to the lower bound of the search (10−610^{-6}), and the fitted power-law is then indistinguishable from the logarithm. The positivity constraint excludes these degenerate cases."

In order to make the computation of the pure error clearer, we revised Line 843 in Appendix B to state that the pure error is computed from the seeds of the selected hyperparameter configuration.

Question 1 (training budget)

We thank the reviewer for the question, and we agree that a larger budget can lower the minimum dimension. We could not measure the minimum dimension at twice the budget for N=49,920N = 49{,}920 due to its computational cost. In order to make this limitation clearer for future readers, we added the following to the Limitations and future work paragraph of the revised version (Lines 527–534). "We measure every minimum dimension and every maximum memorizable dataset size under the same budget of 40,000 training iterations, and a larger budget may change these measurements. At N=6,000N = 6{,}000, a model of dimension 13 memorizes the dataset after about 75,000 iterations (Figure 4), while the average minimum dimension at N=6,240N = 6{,}240 under our budget is 14.33 (Table 4). We did not repeat the experiments with a larger budget due to their computational cost. Each measurement requires two 60-trial hyperparameter searches with three seeds. Under our budget, the two searches take 344 GPU-hours at N=24,960N = 24{,}960 and 779 GPU-hours at N=49,920N = 49{,}920 in the width experiment, and 88 GPU-hours at L=1L = 1 and 270 GPU-hours at L=6L = 6 in the depth experiment."

Question 2 (tied embeddings)

We thank the reviewer for the question. GPT-2 uses tied embeddings, so Table 4 counts its single embedding table once. In order to clarify this, we revised Line 247 in Section 3.2 with the additional phrase "shared between input and output". OLMo-3 uses untied embeddings, with a separate output layer of the same size. We revised Lines 802–806 in Appendix B.1 to state this difference, and Table C counts both OLMo-3 tables (Question 4).

AuthorsOpenReview ↗

Reply to Reviewer izFw (3/4)

Question 3 (theoretical bounds in bits)

We thank the reviewer for this point, and we agree that a construction with O~(N)\tilde{\mathcal{O}}(\sqrt{N}) parameters requires a number of bits per parameter that grows with NN. The theoretical results we cite study a setting that differs from our setting. In order to make this clearer, we made two changes. First, we removed the comparisons between these results and our measurements. Specifically, we removed the clause "substantially worse than what theoretical analyses predict is achievable" from the Abstract, the clause "and are much better than the linear scaling observed in empirical studies" from the Introduction, and the comparison of PP with N\sqrt{N} from Section 5. Second, in the Related Work section (Lines 146 and 156–158), we added the precision of the parameters to the assumptions that vary across settings, together with the following text. "The construction of Kajitsuka & Sato (2025) uses O~(N)\tilde{\mathcal{O}}(\sqrt{N}) bits per parameter, and Madden et al. (2025) treat parameters as real numbers with unbounded precision. The memorization bounds above therefore do not predict the capacity of a trained transformer with a fixed precision, such as our models, whose parameters are stored in 32 bits."

Reviewer_b48JOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the limit of transformer model storing facts with trainable embeddings. The authors consider the subject-relation-object triplet setting where the subjects and objects share the same domain and each relation is a permutation on it. They construct 4 types of synthetic dataset: 1. Uniform, each relation is a random permutation; 2. Symmetric, all relation r satisfy r(o)=s for any s,o such that r(s)=0; 3. 2Hop, there are 3 relations and the third is a composition of the first and the second; 4. Powerlaw, the frequency distribution fact triplets is a Zipf distribution. The authors discover that after largely hyperparameter optimizations, for these four types of data, the minimal embedding dimension d required is in log⁡(N)\log(N) where N is the number of fact triplets. Therefore the number of non-embedding parameters is O(log⁡2(N))O(\log^2(N)), which breaks the linear scaling capacity in previous works.

Critical strengths
  1. The writing is very clear and I appreciate authors' efforts on exhausted hyperparameter search.
  2. The research problem of memorization capacity is fundamental.
Critical weaknesses

I concern about how broad the logarithmic capacity scaling could hold across different datasets. This paper seems to attribute the previous discovered linear-scaling capacity to suboptimal hyperparameters, but the dataset of interest of challenged papers is different from the dataset in this paper. For example, in Section 4.3, Lu et al. 2024 use natural language corpus from Wiki and Morris et al., 2026 use random sequences, which are very different from the (s,r,o) data in this paper. Actually the (s,r,o) data in Nichani et al. 2025 is also a bit different -- they consider m=nm=n in their experiments and theory while this paper mostly fix m=3m=3 and consider the small relation regime, which is same as Ravfogel et al. 2026. I understand that this setting would favor trainable embeddings over fixed ones, but I guess this logarithmic capacity scaling is unlikely to hold when m=Ω(n)m=\Omega(n) and Table 8 also justifies this concern.

Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuanjing Huang, and Xipeng Qiu. Scaling laws for fact memorization of large language models.

John Xavier Morris, Chawin Sitawarin, Narine Kokhlikyan, Chuan Guo, G. Edward Suh, Alexander M Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. How much can language models memorize?

Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories.

Shauli Ravfogel, Gilad Yehudai, Joan Bruna, and Alberto Bietti. Geometric factual recall in transformers.

Questions for authors
  1. Can this logarithmic capacity scaling extend to broader datasets like natural language corpus in Lu et al. 2024?
  2. How would the capacity scaling look like with larger mm like m=nm=n?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer b48J (1/2)

We thank the reviewer for the clear review and for the kind words on the writing and the hyperparameter search. We answer both questions below, together with the changes we made in the revised version. Please let us know if anything falls short or if any concerns remain. We would really appreciate the opportunity to continue to improve our paper and address additional concerns before the end of the discussion period. Changes in the revised PDF are highlighted in blue. Additional minor revisions are not highlighted, for clarity.

Question 1 (natural-language datasets)

We thank the reviewer for the question. To test our results on natural-language facts, we added the Uniform-NL\texttt{Uniform-NL} experiment (Sections 3.6 and 4.5 and Appendix E of the revised version). In this experiment, the entities and relations are real-world phrases that each consist of several tokens, and the vocabulary of tokens is fixed.

Each fact of Uniform-NL\texttt{Uniform-NL} is written as a natural-language declarative sentence, for example "Shana Bald works at Tim Hortons", and tokenized with the GPT-2 tokenizer. The minimum transformer dimension needed to memorize a dataset of size NN behaves similarly to the Uniform\texttt{Uniform} experiment and grows sublinearly with NN. A lack-of-fit F-test rejects a linear fit (p=6.0×10−6p = 6.0\times10^{-6}, Table A). Since the revised version includes this experiment, we removed the sentence in the Limitations and future work paragraph that named natural-language facts as a next step.

Table A. Minimum transformer dimension on Uniform-NL\texttt{Uniform-NL} (L=1L = 1, m=3m = 3, averaged over 3 seeds), with the Uniform\texttt{Uniform} result of Table 4 for comparison.

NN45901803607801,5603,1206,240
Uniform-NL\texttt{Uniform-NL}4.004.335.677.3310.0013.6717.3326.67
Uniform\texttt{Uniform}3.334.005.006.679.0010.3312.6714.33

Moreover, we believe that we were not sufficiently clear in the submission version of our analysis of embedding and non-embedding parameters, and we rewrote this part of Section 5 in the revised version (starting with the paragraph "Embedding versus non-embedding parameters"). The rewritten part contains the following text.

"The number of embedding parameters does not have an effect on the transformer's information capacity. Consider the Uniform-NL\texttt{Uniform-NL} experiment, where the embedding table has the 50,257 rows of the GPT-2 vocabulary at every dataset size. The datasets in this experiment contain at most 1,764 distinct tokens, so at least 48,493 of these rows correspond to tokens that do not occur in any fact. Therefore, the number of bits divided by the total number of parameters is inappropriate for measuring information capacity. Furthermore, we define the capacity of a transformer as the largest dataset size that it can memorize completely (Section 3.1), and we control for how the facts are embedded, which is a confounding factor."

The reviewer also notes that our datasets differ from the datasets of prior work. The difference between the linear scaling of prior work and our findings could therefore come from the datasets and not from the hyperparameter optimization. Section 4.3 separates these two explanations by keeping our datasets fixed and changing only the rigor of the hyperparameter search. With the search restricted to the learning rate and the batch size, as in prior work, the number of non-embedding parameters grows superlinearly with the number of facts (as N1.46N^{1.46} to N1.54N^{1.54}). With the more systematic hyperparameter search, the number of non-embedding parameters grows sublinearly with the number of facts (as (log⁡N)2(\log N)^2).

AuthorsOpenReview ↗

Reply to Reviewer b48J (2/2)

Question 2 (number of relations)

We thank the reviewer for the question. Studying how memorization capacity scales when the number of relations grows with the number of entities is an interesting direction. However, it is outside the scope of this paper, where we study how memorization capacity scales with the transformer dimension and the number of layers. In order to make this limitation clearer for future readers, we added the following to the Limitations and future work paragraph of the revised version (Lines 534–537). "Our scaling experiments use a fixed number of relations. Ravfogel et al. (2026) prove that d=O(mlog⁡n)d = \mathcal{O}(m\log n) suffices to memorize mm relations over nn entities. Based on this result, we expect the minimum dimension to grow logarithmically in NN for other fixed numbers of relations as well." The dependence of the minimum dimension on the number of relations in Table 10 (Table 8 of the submission) is consistent with the bound of Ravfogel et al. (2026).

We hope these answers help to address the reviewer's concerns. Please let us know if anything falls short or if any concerns remain. We would really appreciate the opportunity to continue to improve our paper and address additional concerns before the end of the discussion period.

Thanks for the response and the revision. I appreciate the new experiments on natural language data. However, I'm still not convinced by the claimed logarithmic scaling. Here are the reasons.

  1. The claimed logarithmic scaling is essentially dmin∝log⁡Nd_{min}\propto \log N where NN is the number of facts. For the setting of subject-relation-object, we have N=n⋅mN=n\cdot m. Most of the logarithmic scaling is fitted by fixing mm and varying nn, which indicates dmin∝log⁡nd_{min}\propto \log n. This is weaker than the general dmin∝log⁡Nd_{min}\propto \log N claim throughout the paper, which feels overclaim to me.

  2. The new Uniform-NL experiments also admit the possibility of a power law scaling dmin∝Ncd_{min}\propto N^c, which could also break the sublinear scaling claim of P∝sublinear(N)P \propto \mathrm{sublinear}(N) since P∝dmin2P\propto d_{min}^2. Here PP is the non-embedding parameter count.

AuthorsOpenReview ↗

Reply to Reviewer b48J

We thank the reviewer for the follow-up. Below, we answer the two remaining concerns.

Number of relations

We thank the reviewer for the point on the number of relations. In our width and depth scaling experiments, the number of relations is fixed (m=3m = 3) and the number of entities nn varies, so N=nmN = nm and log⁡N=log⁡n+log⁡m\log N = \log n + \log m. At a fixed number of relations, a logarithmic functional form in NN is equivalent to a logarithmic functional form in nn modulo a constant. Our claim is that the minimum dimension grows logarithmically with respect to the number of facts at a fixed number of relations. We do not make claims about the scaling of the minimum dimension with respect to the number of relations. The Limitations paragraph states that our scaling experiments use a fixed number of relations. One experiment in the paper varies the number of relations (Appendix E.2). In this experiment, we hold the number of facts fixed at N=9,216N = 9{,}216 and vary the number of relations from m=1m = 1 to m=96m = 96. The minimum dimension increases from 6.006.00 at m=1m = 1 to 46.3346.33 at m=64m = 64. The minimum dimension therefore depends on the number of relations for a fixed number of facts. Ravfogel et al. (2026) prove that d=O(mlog⁡n)d = \mathcal{O}(m \log n) suffices to memorize mm relations over nn entities. This bound is logarithmic in nn when mm is fixed. Based on this bound, we expect the minimum dimension to also grow logarithmically in NN for other fixed numbers of relations, and the Limitations paragraph states this expectation.

Power-law scaling on Uniform-NL\texttt{Uniform-NL}

We agree that the Uniform-NL\texttt{Uniform-NL} experiment does not rule out a power-law scaling of the minimum dimension with respect to the dataset size. The Uniform-NL\texttt{Uniform-NL} experiment is currently running for N=12,480N = 12{,}480 and N=24,960N = 24{,}960, but these two experiments did not finish before the end of the discussion period, and we will add them to the paper once they finish. As of now, the experiment has completed for 8 dataset sizes. With these data points, a lack-of-fit F-test does not reject the logarithmic-with-offset functional form (p=0.17p = 0.17) or the power-law with an offset (p=0.43p = 0.43). The lack-of-fit F-test does not reject the functional form d=aN+cd = a\sqrt{N} + c either (p=0.37p = 0.37). Under this functional form, the number of non-embedding parameters grows linearly in NN. We have edited Section 4.5 to no longer state that the minimum dimension grows sublinearly with respect to the dataset size. On Uniform\texttt{Uniform}, the lack-of-fit F-test does reject the power-law functional form when considering all data points, but it does not reject the power-law when considering only the dataset sizes up to N=6,240N = 6{,}240. The lack-of-fit F-test rejected the power-law functional forms only after we included N=12,480N = 12{,}480 (Section 4.1).

We claim that the number of non-embedding parameters grows sublinearly with respect to the dataset size only for the experiments on symbolic facts that meet two conditions. First, the lack-of-fit F-test rejects the functional form d=aN+cd = a\sqrt{N} + c. Under this functional form, the number of non-embedding parameters P=Θ(d2)P = \Theta(d^2) grows linearly in NN, so this condition rejects linear growth. Second, the lack-of-fit F-test does not reject the logarithmic-with-offset functional form, under which the number of non-embedding parameters grows as (log⁡N)2(\log N)^2. These experiments are Uniform\texttt{Uniform}, Symmetric\texttt{Symmetric} and 2Hop\texttt{2Hop} with GPT-2, and Uniform\texttt{Uniform} with OLMo-3. For the functional form d=aN+cd = a\sqrt{N} + c, the pp-values of these four experiments are 1.1×10−121.1\times10^{-12}, 2.0×10−52.0\times10^{-5}, 6.8×10−56.8\times10^{-5} and 0.0100.010. For the logarithmic-with-offset functional form, the pp-values are 0.530.53, 0.800.80, 0.530.53 and 0.570.57. Table 14, which we added in the updated PDF, reports these pp-values for every experiment.

For Uniform-NL\texttt{Uniform-NL}, the updated PDF reports only the results of these lack-of-fit F-tests. The updated PDF does not claim, for Uniform-NL\texttt{Uniform-NL}, that the minimum dimension grows logarithmically with respect to the dataset size or that the number of non-embedding parameters grows sublinearly with respect to the dataset size.

We thank the reviewer again for the time and attention given to our paper. The reviewer's feedback helped us improve the paper and make the claims more precise.

Reviewer_tdLvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the minimum transformer width required to perfectly memorize synthetic subject–relation–object triples. Using extensive hyperparameter optimization and full-batch training, it reports approximately logarithmic width growth with dataset size. It also examines depth scaling, structured relations, skewed fact frequencies, and an OLMo-3 variant. The main claim is that sufficiently optimized training reveals more favorable memorization scaling than previous empirical studies.

Critical strengths

The controlled synthetic task and explicit complete-memorization criterion provide a clear experimental setting. The substantial hyperparameter search and training ablations offer useful evidence that measured memorization capacity depends strongly on the training procedure.

The paper also reports embedding-inclusive parameter counts and acknowledges the training-budget limitations of the depth experiments. These details help readers assess the scope of the results.

Critical weaknesses
  1. The claimed improvement over prior capacity scaling is not established. The embedding table grows with the dataset, while the main comparison emphasizes width and non-embedding parameters. Table 4 reports approximately 2.04–2.19 bits per total parameter for N≥1,560N\geq1,560, consistent with existing capacity estimates. Although Section 5 acknowledges this, the introduction and conclusion still present the findings as a substantially better scaling law explained by improved training.

  2. The contribution beyond closely related work is limited. Ravfogel et al. (2026) already provide theoretical and empirical results on logarithmic width requirements with learned embeddings. Testing random mappings rather than bijections, requiring 100% accuracy, and conducting more extensive tuning are useful extensions. However, the paper does not clearly establish what substantially new understanding follows from these changes.

  3. The evidence does not convincingly distinguish the logarithmic law from relevant alternatives. Table 1 requires the shifted power-law fit to remain positive for every N>0N>0. For an increasing function d=aNb+cd=aN^b+c, this excludes negative offsets, even when the fitted width would remain positive throughout the measured range. The reported shifted power law consequently reduces to the unshifted fit.

This restriction needs justification or a comparison allowing offsets valid over the observed range. Moreover, both logarithmic and power-law fits pass the test in the OLMo-3 experiment, so the cross-architecture results do not independently establish the claimed functional form.

  1. The depth result remains substantially dependent on the training budget. Deeper models take 77–91% of the available budget to converge, and training becomes less stable. The authors acknowledge this limitation, but it restricts the conclusion to capacity attained under the chosen training procedure. The experiment does not establish whether the observed sublinear trend persists when deeper models receive sufficient additional training.

Overall, the paper contains useful measurements, but the interpretation of the main result and the distinction from existing work are not strong enough to support acceptance.

Questions for authors
  1. How should the claimed improvement over prior capacity scaling be reconciled with the approximately constant bits per total parameter in Table 4?
  2. What new conclusion follows beyond Ravfogel et al. (2026) from testing random mappings and requiring 100% accuracy?
  3. Does the rejection of power-law scaling still hold when negative offsets are allowed, provided the fitted width remains positive over the measured range?
  4. Have you checked whether increasing the training budget changes the observed depth-scaling trend?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer tdLv (1/3)

We thank the reviewer for the careful review and the four precise questions. We answer each question below, together with the changes we made in the revised version. Please let us know if anything falls short or if any concerns remain. We would really appreciate the opportunity to continue to improve our paper and address additional concerns before the end of the discussion period. Changes in the revised PDF are highlighted in blue. Additional minor revisions are not highlighted, for clarity.

Question 1 (bits per parameter)

We thank the reviewer for raising this point. Our claim concerns the number of non-embedding parameters PP, which is also how Lu et al. (2024) measure model size when they report linear scaling (their Appendix D.4). To show that this claim does not depend on the assumption of one token per entity, we added the Uniform-NL\texttt{Uniform-NL} experiment (Sections 3.6 and 4.5 and Appendix E of the revised version). In this experiment, the entities and relations are real-world phrases that each consist of several tokens, and the vocabulary of tokens is fixed.

Each fact of Uniform-NL\texttt{Uniform-NL} is written as a natural-language declarative sentence, for example "Shana Bald works at Tim Hortons", and tokenized with the GPT-2 tokenizer. The minimum transformer dimension needed to memorize a dataset of size NN behaves similarly to the Uniform\texttt{Uniform} experiment and grows sublinearly with NN. A lack-of-fit F-test rejects a linear fit (p=6.0×10−6p = 6.0\times10^{-6}, Table A). Since the revised version includes this experiment, we removed the sentence in the Limitations and future work paragraph that named natural-language facts as a next step.

Table A. Minimum transformer dimension on Uniform-NL\texttt{Uniform-NL} (L=1L = 1, m=3m = 3, averaged over 3 seeds), with the Uniform\texttt{Uniform} result of Table 4 for comparison.

NN45901803607801,5603,1206,240
Uniform-NL\texttt{Uniform-NL}4.004.335.677.3310.0013.6717.3326.67
Uniform\texttt{Uniform}3.334.005.006.679.0010.3312.6714.33

Moreover, we believe that we were not sufficiently clear in the submission version of our analysis of embedding and non-embedding parameters, and we rewrote this part of Section 5 in the revised version (starting with the paragraph "Embedding versus non-embedding parameters"). The rewritten part contains the following text.

"The number of embedding parameters does not have an effect on the transformer's information capacity. Consider the Uniform-NL\texttt{Uniform-NL} experiment, where the embedding table has the 50,257 rows of the GPT-2 vocabulary at every dataset size. The datasets in this experiment contain at most 1,764 distinct tokens, so at least 48,493 of these rows correspond to tokens that do not occur in any fact. Therefore, the number of bits divided by the total number of parameters is inappropriate for measuring information capacity. Furthermore, we define the capacity of a transformer as the largest dataset size that it can memorize completely (Section 3.1), and we control for how the facts are embedded, which is a confounding factor."

The reviewer also questions whether the training procedure explains the difference between the linear scaling of prior work and our findings. Section 4.3 tests this by keeping our datasets fixed and changing only the rigor of the hyperparameter search. With the search restricted to the learning rate and the batch size, as in prior work, the number of non-embedding parameters grows superlinearly with the number of facts (as N1.46N^{1.46} to N1.54N^{1.54}). With the more systematic hyperparameter search, the number of non-embedding parameters grows sublinearly with the number of facts (as (log⁡N)2(\log N)^2).

AuthorsOpenReview ↗

Reply to Reviewer tdLv (2/3)

Question 2 (comparison with Ravfogel et al.)

We thank the reviewer for the question. We believe that we were not sufficiently clear in the submission version about how our study differs from that of Ravfogel et al. (2026), and we rewrote the corresponding Related Work paragraph in the revised version (Lines 178–190) as follows.

"Our results on memorization capacity scaling with respect to transformer dimension are consistent with concurrent work by Ravfogel et al. (2026). Ravfogel et al. (2026) study how transformers store facts, and prove that a single-layer transformer with learned embeddings can memorize mm random bijections over nn entities with d=O(mlog⁡n)d = \mathcal{O}(m\log n). They further show that such scaling is possible by fitting a logarithmic curve to the smallest embedding dimension that reaches 95% training accuracy under a single training configuration. However, they do not report a statistical test or compare the logarithmic fit with other functional forms such as a power-law. Our work, on the other hand, systematically studies how the maximum number of facts that a transformer can memorize scales with the transformer dimension and the number of layers. We control for confounding factors in the optimization, the exposure to each fact and the embedding of the facts. We optimize the training hyperparameters at every dataset size, full-batch training allows up to 40,000 exposures per fact at every dataset size, and the embedding table does not limit the number of facts that can be expressed. We then compare logarithmic, power-law and linear functional forms with a lack-of-fit F-test (Section 4.1). We further measure how fact memorization scales on structured datasets (Symmetric\texttt{Symmetric}, 2Hop\texttt{2Hop}, Section 4.1) and on a second architecture (Section 4.4)."

Question 3 (power-law with a free offset)

We thank the reviewer for the question, and we agree that the submission did not clearly motivate the positivity constraint. We conduct additional analyses without the positivity constraint and find that our main claims remain supported. On our per-seed minimum dimensions, the power-law with a free offset is rejected on the 11 dataset sizes that we test in the submission (p=0.046p = 0.046). To provide more statistical evidence for the comparison of the power-law and logarithmic fits, we added the dataset size N=24N = 24 (8 entities) to Table 1, Figure 1 and the Abstract of the revised version. With this dataset size, the power-law with a free offset is rejected with p=0.0010p = 0.0010, and the logarithmic-with-offset fit is the only fit that is not rejected (p=0.53p = 0.53). The power-law with a free offset is also rejected (p=0.0096p = 0.0096) when capacity is measured as the maximum memorizable dataset size at each dimension (Appendix E.1).

To make the role of the positivity constraint clearer for future readers, we added the unconstrained fit to Tables 1 and 9 of the revised version and explained the constraint in Section 4.1 and Appendix B.2. In Section 4.1 (Lines 361–363), we added "We require the fitted dd to be positive for every N>0N > 0 (positivity constraint) in every fit where this constraint applies. Without this constraint, the fitted power-law with an offset can become indistinguishable from the logarithmic fit (see Appendix B.2)." In Appendix B.2, we added "The power-law with an offset, f(N)=a(Nb−1)/b+c0f(N) = a(N^b - 1)/b + c_0, approaches alog⁡N+c0a\log N + c_0 as b→0b \to 0, so it contains the logarithm as a limiting case. On Symmetric\texttt{Symmetric} and on OLMo-3, the unconstrained fit drives the exponent bb to the lower bound of the search (10−610^{-6}), and the fitted power-law is then indistinguishable from the logarithm. The positivity constraint excludes these degenerate cases."

We agree that the OLMo-3 experiment does not distinguish between the power-law and logarithmic fits. The OLMo-3 experiment tests whether the scaling behavior is specific to the GPT-2 architecture, and it covers dataset sizes up to N=6,240N = 6{,}240. On GPT-2, the lack-of-fit F-test does not reject the power-law at these dataset sizes either (Section 4.1). We only extended the Uniform\texttt{Uniform} experiment on GPT-2 to larger dataset sizes, in order to distinguish between the power-law and logarithmic fits with statistical significance. We commit to obtaining these results for OLMo-3 for the camera-ready version, since this experiment will require more time than available in the discussion period.

AuthorsOpenReview ↗

Reply to Reviewer tdLv (3/3)

Question 4 (training budget)

We thank the reviewer for the question, and we agree that the depth result is a measurement of capacity under our training budget. We could not repeat the depth experiment with a larger budget due to its computational cost. In order to make this limitation clearer for future readers, we added the following to the Limitations and future work paragraph of the revised version (Lines 527–534). "We measure every minimum dimension and every maximum memorizable dataset size under the same budget of 40,000 training iterations, and a larger budget may change these measurements. At N=6,000N = 6{,}000, a model of dimension 13 memorizes the dataset after about 75,000 iterations (Figure 4), while the average minimum dimension at N=6,240N = 6{,}240 under our budget is 14.33 (Table 4). We did not repeat the experiments with a larger budget due to their computational cost. Each measurement requires two 60-trial hyperparameter searches with three seeds. Under our budget, the two searches take 344 GPU-hours at N=24,960N = 24{,}960 and 779 GPU-hours at N=49,920N = 49{,}920 in the width experiment, and 88 GPU-hours at L=1L = 1 and 270 GPU-hours at L=6L = 6 in the depth experiment."

We hope these answers help to address the reviewer's concerns. Please let us know if anything falls short or if any concerns remain. We would really appreciate the opportunity to continue to improve our paper and address additional concerns before the end of the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.