- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper fits scaling laws for dense Transformers, STEM, PKM, and MoE using 421 training endpoints from 44 configurations. It models loss as a function of active parameters, additional capacity, and training tokens, then predicts resource-optimal sparsity and architecture choices. A separate roofline model estimates mobile-inference throughput. The large-scale architecture rankings and throughput advantage are predictions rather than direct measurements.
Critical strengths
- The controlled experiments, lineage-aware fitting, and multiple holdout tests provide substantial evidence for the scaling behavior within the measured range (Sections 3–4; Appendix B).
- The paper connects fitted loss surfaces to explicit capacity–compute allocation decisions, making the analysis practically relevant (Section 5).
Critical weaknesses
- The headline architecture rankings rely heavily on extrapolation. Figure 4 starts at roughly 29B total parameters and reaches 1.88T, whereas the measured maxima are 21.9B for PKM and 6.4B for MoE. Several demanding MoE holdout families also yield invalid fits (Appendix Table 5). Without uncertainty or sensitivity analysis, it is unclear whether the predicted rankings remain stable.
- Figure 5 labels PKM as A3.3B and MoE as A3.9B, despite the text and Appendix E.6 describing a matched 3.93B-active-parameter comparison. Moreover, the reported 10.5× speedup is a bandwidth-based estimate without end-to-end device validation or an explicit random-flash-access latency analysis.
Questions for authors
- Using the existing fits, how stable are the Figure 1 boundaries and Figure 4 PKM–MoE ranking across alternative well-performing formulas or lineage-level resampling?
- Are the PKM and MoE models in Figure 5 matched in active parameters? Please reconcile the figure labels with Section 5.5 and clarify how random flash-access costs could affect the 10.5× estimate.
We thank the reviewer for the careful reading and for asking how stable the predicted rankings are. This suggestion led to the new Appendix F (Figures 22 to 24), a stability analysis that we consider a valuable addition to our work.
Q1. How stable are the Figure 1 boundaries and the Figure 4 PKM–MoE ranking?
We answer this with a bootstrap. We resample the measured endpoints of each architecture (192 PKM, 124 MoE, 57 dense Transformer, 48 STEM), refit its selected law, and recompute the lowest-loss allocations for 100 joint draws, on a 41×41 grid of the Figure 1 window and along the Figure 4 path. As the reviewer suggested, we repeat this with lineages as the resampling unit, i.e. all endpoints of one (N,M) configuration (20 PKM, 15 MoE, 5 dense Transformer, 4 STEM). With so few lineages, a lineage resample keeps on average about two-thirds of the configurations, whereas an endpoint resample keeps nearly all of them, so Appendix F reports endpoint resampling first and lineage resampling as a more conservative contrast.
The Figure 1 rows refer to the window above the dash-dotted line of Figure 22, where a dense Transformer would train on below 1000 tokens per parameter. Below that line the comparison of our sparse laws with the dense law is unreliable (Appendix F, "Extreme overtraining").
W1. Reliance on extrapolation and invalid MoE holdout fits.
We agree that the large-scale rankings are predictions of the fitted laws rather than measurements, and Section 5.4 presents the Figure 4 curves as extrapolations. Appendix F now quantifies their uncertainty (Q1). Agreement falls as the laws extrapolate further, but at n=16, 32, 64 and 128 (the end of the Figure 4 path) PKM stays below MoE in 97%, 92%, 85% and 83% of endpoint draws (88%, 77%, 71% and 69% of lineage draws) and below the dense Transformer in 100%, 100%, 99% and 97% (100%, 99%, 88% and 69%).
Table 5 flags MoE P6F in some holdouts that remove whole lineages or capacity families, which is demanding for the smaller MoE dataset (124 endpoints against 192 for PKM). The revised Table 5 reports the errors of these fits in gray instead of omitting them. The full-data fit and all five LLLO fractions of P6F, the key evaluations and the ones most commonly reported in other scaling-law papers, remain valid. Their RMSE (5.80e-3 on the full data and 5.36e-3 to 9.79e-3 across the LLLO fractions, Table 1) is comparable to or lower than reported there (Section 4.2), for example 6.08e-3 against 19e-3 for MoE at LLLO20 (Ludziejewski et al., 2024) and 5.36e-3 against 3.9e-3 at LLLO10 for the joint MoE and dense Transformer law (Ludziejewski et al., 2025).
Q2. Are the PKM and MoE models in Figure 5 matched in active parameters? Please reconcile the figure labels with Section 5.5 and clarify how random flash-access costs could affect the 10.5× estimate.
You are correct that the PKM and MoE models are not exactly matched in active parameters: PKM has 3.3B active parameters, whereas MoE has 3.93B. This difference arises because, while keeping the backbone depth and width fixed, PKM replaces every fourth FFN with a smaller PKM projection. We revised Section 5.5 to qualify the comparison as "approximately active-parameter matched" and expanded Appendix E.6 to explain the architectural difference and reconcile the figure labels.
The reported throughput advantage is a roofline-model estimate, not an end-to-end device measurement. We thank the reviewer for highlighting that the original analysis did not explicitly model random flash-access latency. We have revised the model to include this cost, with the assumptions and parameters documented in Appendix E.6. Under the revised, conservative analysis, PKM achieves a 7.61x advantage at the representative operating point, increasing to 30.45x as model capacity grows. The principal scaling distinction therefore remains: PKM retrieval cost is independent of total capacity, whereas MoE must load more distinct experts as capacity increases. Analytical modeling is also commonly used to study inference and memory-offloading constraints, including in LLM in a Flash (ACL 2024), FlexGen (ICML 2023), and Pope et al. (MLSys 2023). We revised Section 5.5 to clearly qualify these results as model estimates rather than end-to-end measurements. End-to-end measurements require a considerable engineering effort left to future work and additionally depends on runtime, quantization, caching, and kernel implementations, which remain outside the scope of this work.