acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws for LLM Memory Sparsity

Abstract

Sparse memory architectures offer a route to expanding language-model capacity while accessing only a small fraction of parameters per token. Their finer access granularity promises greater sparsity than conventional mixture-of-experts (MoE), but how memory capacity should scale with active computation and training data remains insufficiently understood. We introduce and validate scaling laws for Product Key Memory (PKM) that capture how active parameters, memory capacity, and training tokens jointly determine loss. Our analysis spans 421 measured training endpoints from 44 configurations of dense Transformers, PKM, STEM, and MoE, with total parameter capacities reaching 21.9B and training budgets reaching 128B tokens. A joint PKMMoE fit with shared coefficients worsens held-out MoE predictions, motivating separate scaling relationships for the two architectures. Using these architecture-specific fits, we find that optimal memory allocation depends on the balance between capacity and compute: increasing capacity at fixed compute favors greater sparsity, whereas increasing compute at fixed capacity favors more active parameters. Among the studied variants, predicted loss favors PKM when capacity is abundant relative to compute, MoE in an intermediate regime, and dense Transformers when compute is abundant relative to capacity. A complementary roofline analysis estimates a PKM throughput advantage over MoE in the modeled mobile inference setting, where selective retrieval reduces flash-memory traffic. Together, these findings identify when sparse memory offers advantages in predictive performance and execution efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling, scalinglaws, MoE, memory, PKM, STEM

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_PRV4OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper fits scaling laws for dense Transformers, STEM, PKM, and MoE using 421 training endpoints from 44 configurations. It models loss as a function of active parameters, additional capacity, and training tokens, then predicts resource-optimal sparsity and architecture choices. A separate roofline model estimates mobile-inference throughput. The large-scale architecture rankings and throughput advantage are predictions rather than direct measurements.

Critical strengths
  1. The controlled experiments, lineage-aware fitting, and multiple holdout tests provide substantial evidence for the scaling behavior within the measured range (Sections 3–4; Appendix B).
  2. The paper connects fitted loss surfaces to explicit capacity–compute allocation decisions, making the analysis practically relevant (Section 5).
Critical weaknesses
  1. The headline architecture rankings rely heavily on extrapolation. Figure 4 starts at roughly 29B total parameters and reaches 1.88T, whereas the measured maxima are 21.9B for PKM and 6.4B for MoE. Several demanding MoE holdout families also yield invalid fits (Appendix Table 5). Without uncertainty or sensitivity analysis, it is unclear whether the predicted rankings remain stable.
  2. Figure 5 labels PKM as A3.3B and MoE as A3.9B, despite the text and Appendix E.6 describing a matched 3.93B-active-parameter comparison. Moreover, the reported 10.5× speedup is a bandwidth-based estimate without end-to-end device validation or an explicit random-flash-access latency analysis.
Questions for authors
  1. Using the existing fits, how stable are the Figure 1 boundaries and Figure 4 PKM–MoE ranking across alternative well-performing formulas or lineage-level resampling?
  2. Are the PKM and MoE models in Figure 5 matched in active parameters? Please reconcile the figure labels with Section 5.5 and clarify how random flash-access costs could affect the 10.5× estimate.
1 comment

We thank the reviewer for the careful reading and for asking how stable the predicted rankings are. This suggestion led to the new Appendix F (Figures 22 to 24), a stability analysis that we consider a valuable addition to our work.

Q1. How stable are the Figure 1 boundaries and the Figure 4 PKM–MoE ranking?

We answer this with a bootstrap. We resample the measured endpoints of each architecture (192 PKM, 124 MoE, 57 dense Transformer, 48 STEM), refit its selected law, and recompute the lowest-loss allocations for 100 joint draws, on a 41×41 grid of the Figure 1 window and along the Figure 4 path. As the reviewer suggested, we repeat this with lineages as the resampling unit, i.e. all endpoints of one (N,M)(N,M) configuration (20 PKM, 15 MoE, 5 dense Transformer, 4 STEM). With so few lineages, a lineage resample keeps on average about two-thirds of the configurations, whereas an endpoint resample keeps nearly all of them, so Appendix F reports endpoint resampling first and lineage resampling as a more conservative contrast.

Share of draws that keep the result of the fitted lawsEndpoints resampledLineages resampled
Figure 1: cell keeps its winner, mean over cells92%82%
Figure 1: MoE–PKM boundary within a quarter decade of capacity94%80%
Figure 1: upper edge of the dense region within a quarter decade96%64%
Figure 4: PKM below MoE at n=16n=16 / n=128n=12897% / 83%88% / 69%

The Figure 1 rows refer to the window above the dash-dotted line of Figure 22, where a dense Transformer would train on below 1000 tokens per parameter. Below that line the comparison of our sparse laws with the dense law is unreliable (Appendix F, "Extreme overtraining").

W1. Reliance on extrapolation and invalid MoE holdout fits.

We agree that the large-scale rankings are predictions of the fitted laws rather than measurements, and Section 5.4 presents the Figure 4 curves as extrapolations. Appendix F now quantifies their uncertainty (Q1). Agreement falls as the laws extrapolate further, but at n=16n=16, 32, 64 and 128 (the end of the Figure 4 path) PKM stays below MoE in 97%, 92%, 85% and 83% of endpoint draws (88%, 77%, 71% and 69% of lineage draws) and below the dense Transformer in 100%, 100%, 99% and 97% (100%, 99%, 88% and 69%).

Table 5 flags MoE P6F in some holdouts that remove whole lineages or capacity families, which is demanding for the smaller MoE dataset (124 endpoints against 192 for PKM). The revised Table 5 reports the errors of these fits in gray instead of omitting them. The full-data fit and all five LLLO fractions of P6F, the key evaluations and the ones most commonly reported in other scaling-law papers, remain valid. Their RMSE (5.80e-3 on the full data and 5.36e-3 to 9.79e-3 across the LLLO fractions, Table 1) is comparable to or lower than reported there (Section 4.2), for example 6.08e-3 against 19e-3 for MoE at LLLO20\mathrm{LLLO}_{20} (Ludziejewski et al., 2024) and 5.36e-3 against 3.9e-3 at LLLO10\mathrm{LLLO}_{10} for the joint MoE and dense Transformer law (Ludziejewski et al., 2025).

Q2. Are the PKM and MoE models in Figure 5 matched in active parameters? Please reconcile the figure labels with Section 5.5 and clarify how random flash-access costs could affect the 10.5× estimate.

You are correct that the PKM and MoE models are not exactly matched in active parameters: PKM has 3.3B active parameters, whereas MoE has 3.93B. This difference arises because, while keeping the backbone depth and width fixed, PKM replaces every fourth FFN with a smaller PKM projection. We revised Section 5.5 to qualify the comparison as "approximately active-parameter matched" and expanded Appendix E.6 to explain the architectural difference and reconcile the figure labels.

The reported throughput advantage is a roofline-model estimate, not an end-to-end device measurement. We thank the reviewer for highlighting that the original analysis did not explicitly model random flash-access latency. We have revised the model to include this cost, with the assumptions and parameters documented in Appendix E.6. Under the revised, conservative analysis, PKM achieves a 7.61x advantage at the representative operating point, increasing to 30.45x as model capacity grows. The principal scaling distinction therefore remains: PKM retrieval cost is independent of total capacity, whereas MoE must load more distinct experts as capacity increases. Analytical modeling is also commonly used to study inference and memory-offloading constraints, including in LLM in a Flash (ACL 2024), FlexGen (ICML 2023), and Pope et al. (MLSys 2023). We revised Section 5.5 to clearly qualify these results as model estimates rather than end-to-end measurements. End-to-end measurements require a considerable engineering effort left to future work and additionally depends on runtime, quantization, caching, and kernel implementations, which remain outside the scope of this work.

Reviewer_YtrDOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies scaling laws for sparse language models, focusing on PKM and MoE. The authors model how total parameter size, active parameter size, and training tokens jointly affect loss, and evaluate their extrapolation accuracy.

Critical strengths
  1. Extensive experiments. The paper includes substantial experiments across PKM, MoE, STEM, and dense Transformers. The proposed laws achieve low prediction errors in many validation settings.
  2. Analysis of model–data interactions. The authors study the interactions between model capacity and training data, and compare different formulas to identify those with better predictive performance.
  3. Practical guidance. The fitted laws can help researchers choose architectures and sparsity levels under different parameter and compute budgets.
Critical weaknesses
  1. Limited validation of longer-training extrapolation. In the largest-corner extrapolation tests, both PKM and MoE are evaluated at training-token-to-total-parameter ratios ((D/T)) below 20. Their loss-prediction accuracy at substantially longer training budgets and higher (D/T) ratios remains insufficiently validated.
  2. Although the appendix discusses monotonicity, it remains unclear whether the fitted laws remain non-increasing in N and M throughout the extrapolation domain used in the paper. Their limiting behavior also needs clarification.
Questions for authors
  1. Did any fits rejected for monotonicity violations achieve lower prediction errors than the retained alternatives?
  2. With the fitted coefficients, over what range are the final laws non-increasing in each of (N), (M), and (D), with the others fixed? What are their limiting behaviors?
2 comments

We thank the reviewer for the review and are glad that our experiments were found extensive and the fitted laws useful for choosing architectures and sparsity levels. Below we answer both questions and both weaknesses, using two additions to the revised PDF: every previously suppressed cell of the appendix error tables now reports its error (Tables 4, 5, 8, 10, 11 and 14), and a new Appendix G shows the training-token ratio of the optimal allocations along the Figure 4 path.

Q1. Did fits rejected for monotonicity achieve lower errors than the retained alternatives?

Rejected refits still produce finite held-out errors, so the tables now report them where they previously showed only a validity mark. Flagged values are set in gray with a subscript naming the failed check, average all declared folds including the invalid ones, and are never bolded.

  • Across the six RMSE and MAE tables (PKM, MoE, joint), none of the 86 cells flagged for monotonicity has a lower error than the best valid entry of its column.
  • Only four flagged cells fall below their column's best valid value, all flagged only because a refit did not converge (subscript i): MoE P7 in LOMO and joint P6F in CF, in both RMSE and MAE.

Q2 (and the second weakness). Over what range are the laws non-increasing, and what are their limits?

With the coefficients of Table 3, we find where the local responses EN\mathcal E_N, ED\mathcal E_D and EM\mathcal E_M of Appendix B.1.1 change sign, in closed form for DD and by numerical root-finding on log-spaced grids for NN and MM. The laws are non-increasing:

  • in DD wherever qD=β+ζlog⁡M+ξlog⁡N>0q_D=\beta+\zeta\log M+\xi\log N>0, i.e. for PKM up to N=336N=336B active parameters at M=0.3M=0.3B (5.2T at M=1M=1T) and for MoE up to N=461N=461T at M=0.3M=0.3B (1,350T at M=1M=1T),
  • in NN up to a boundary N∗(D,M)N^*(D,M), beyond which they predict rising loss at fixed data: 1.2B to 12.7B for PKM and 7.6B to 328B for MoE, for DD from 2B to 10T tokens and MM from 0.3B to 1T,
  • in MM everywhere for PKM trained on at least 45B tokens.

All observed endpoints and every optimal allocation plotted in Figures 1 and 4 and Appendix G lie where the laws are non-increasing in all variables, the closest is PKM at the largest scale, still 1.7× below N∗N^*. As D→∞D\to\infty the loss tends to AN−αM−κAN^{-\alpha}M^{-\kappa} (the laws have no irreducible floor), and as N→∞N\to\infty at fixed DD it diverges. As MM grows, the loss of PKM trained on fewer than 45B tokens and of MoE eventually rises, but for NN from 0.1B to 100B and DD from 2B to 10T tokens it starts rising below 100T memory parameters only with fewer than 16 tokens per active parameter for PKM and 90 for MoE (22 when D≥100D\ge100B). These limits lie far outside the range where the paper uses the laws.

W1. Longer-training extrapolation.

We agree that longer training runs are more informative for scaling laws, but ours were limited by our compute budget. However, sparse models have no fixed optimal ratio such as 20 tokens per parameter, the optimum depends on the regime. Appendix G shows that the optimal allocations along the Figure 4 path use D/TD/T from 9.7 to 52.3 for PKM and from 8.5 to 24.5 for MoE, with more GPUs per unit of nn the ratios are lower, and with 64 GPUs the MoE optimum stays below 20 throughout. Shorter runs are also informative for fitting, even when undertrained: the fitted curves follow each configuration's measured endpoints over training length and extend smoothly beyond them (Figures 16 and 17).

Thank you for your detailed response and for addressing my questions and concerns.

Reviewer_oFVMOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper produces a scaling law for the product key memory (PKM) architecture, which is a sparse parametric memory model.

Training on DCLM, the paper analyzes 421 models over 44 model configurations (192 PKM, 124 MoE, 57 dense, and 48 STEM), up to 21.9B total parameters, on up to 128B tokens.

PKM and MoE share a memory-sparsity scaling law famuly, whereas dense and STEM share a six-coefficient family.

After fitting the scaling laws, the paper uses them with constrained optimization to draw conclusions about relative efficiencies and effectiveness.

The paper shows that PKM is favored when parameter capacity is abundant relative to compute, MoE is favored in an intermediate regime, and dense Transformers are favored when compute is abundant relative to parameter capacity. Optimal PKM sparsity increases with the total parameter budget at fixed compute and decreases with compute at a fixed parameter budget.

The paper also prodvides some hardware estimates. In the setting studied, PKM sustains its throughput as memory capacity grows, whereas MoE slows as more experts are added.

Critical strengths

1. Clear problem definition with well-fitted scaling laws and solid use of extrapolation.

The scaling laws in the study fit and are tested to extrapolate well. The use of constrained optimization is then used to answer interesting questions about the optimality of the methods in practical scenarios of interest. The findings are generally useful to people designing and building models.

Critical weaknesses

None

Questions for authors

1. [Clarification] Why do you not include the irreducible error term EE in your scaling laws equations 2 and 3?

I understand you went through many iterations of the candidate scaling laws and eventually dropped EE. Can you expand a bit on why this was done, and whether this is a behavior you expected?

2 comments

We thank the reviewer for the review and are glad that the reviewer found our scaling laws well fitted, and the findings useful. Below we explain why we dropped the irreducible term E and whether this was expected.

Farseer Li et al. (2026) shows (Section 3.3 and Appendix C.4) that the model-dependent term G(N) is an estimate of "E + U(N)", with E "absorbed as a constant within the fit". Like Farseer, the selected law has no standalone floor (E): any constant is absorbed by the model-dependent term. When we fit P6F with a standalone floor E to the PKM data (candidate P6D), the optimizer drove E to numerically zero (about 1e-9). Dropping it left the full-data error unchanged (MAE differs by 4e-12) and the LLLO errors identical or better. The floor is not identifiable from these data: it trades off against the model-size-dependent data amplitude B*N^−λ. P6C, which lacks that amplitude, keeps E = 1.52, whereas P6B and P6D, which include it, drive E to zero. In hindsight this was expected: over the measured range the losses are far from saturating, so a constant cannot be separated from the slowly decaying model and data-dependent terms. E = 0 is therefore a parameterization decision for the fitted range, not a claim that the true irreducible loss is zero. We already have added this in the updated paper - Appendix B.3.

Thank you for the response and the new paragraph in Appendix B.3. The explanation is sensible. Over the measured range the losses are far away from saturating, so any constant floor is hard to separate from slowly decaying terms.

Since E=0E = 0 is a choice for the fitted range, it may be worth noting in Section 5.4 that the far end of Figure 4, where the PKM–MoE bands already overlap (Appendix F), also depends on it (more strongly than the observed region does).

My assessment is unchanged, and I am keeping my Rating at 4. Thank you for a clear and useful paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.