acceptodds
Under review as a conference paper at ICLR 2027

STAPLE: Static Sparse Upcycling of Pretrained Checkpoints using Embedding Modules

Abstract

Sparse computation can scale the parametric capacity of large language models without proportionally increasing active computation, but pretraining sparse ar- chitectures remains expensive and data-hungry. Existing sparse upcycling meth- ods mainly target Mixture-of-Experts (MoE) models, where expert homogeneity, sample inefficiency, and system overheads can limit practical scalability. Static sparse memory can address the efficiency and expert-homogeneity challenges, but naively integrating it into pretrained computation yields poor returns. In this work, we introduce STAPLE, a training recipe and architecture design for successfully integrating static sparse memory into a pretrained model. STAPLE preserves the pretrained model’s computation and adds complementary token-specific knowl- edge in a scalable manner. On OLMo-2 1B, STAPLE improves over continued dense training by up to 2–3 points across math, code, discrete reasoning, and knowledge-intensive tasks. Average performance increases with static capacity, and improvements persist on the larger OLMo-2 7B backbone. STAPLE also outperforms static sparse baselines such as STEM and Engram, and achieves a higher average score than an MoE upcycling baseline while decoding 2× faster with less than half its GPU memory. Importantly, despite introducing billions of parameters late in training, our host-offloaded implementation keeps generation throughput within 1.5% and training throughput within 5% of the dense base- line, using a 1 GB GPU cache for inference. These results show that static sparse memory can provide a sample-efficient, scalable, and system-efficient path for expanding pretrained models without sparse pretraining from scratch.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse architecture, Model Upcycling, Static sparse architecture, Midtraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ySafOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STAPLE, a static sparse upcycling method that expands pretrained language models by introducing token-indexed embedding modules into Transformer FFNs. The method preserves pretrained computation through an additive architecture and zero initialization, while employing a warmup strategy to facilitate adaptation. Additionally, CPU offloading and caching techniques are introduced to reduce memory and computational overhead. Experiments on OLMo-2 1B and 7B demonstrate improvements across several downstream tasks while maintaining training and inference throughput close to the dense baseline.

Critical strengths

S1. Simple and effective architecture for preserving pretrained computation.

STAPLE preserves the original FFN computation while introducing additional token-specific memory through a residual pathway. Combined with zero initialization, this design allows the expanded model to retain the functionality of the pretrained checkpoint, providing a practical approach to sparse upcycling.

S2. Promising system efficiency with large-scale static memory.

The proposed CPU offloading and caching mechanisms enable substantial parameter expansion with limited additional GPU memory consumption. The reported training and inference throughput remains close to the dense baseline, demonstrating the practical potential of static sparse memory.

Critical weaknesses
  • W1. Limited methodological novelty and insufficient differentiation from existing static sparse memory methods.

STAPLE shares substantial conceptual similarities with existing static sparse memory architectures, particularly Engram. Its main contributions primarily involve preserving pretrained FFN computation, introducing additive token-indexed embeddings, and applying zero initialization and warmup. While these choices are reasonable, their methodological novelty appears relatively limited. Furthermore, the Engram-style baseline omits certain components of the original architecture, making it difficult to determine whether the reported improvements sufficiently establish STAPLE's advantages over existing methods.

  • W2. Limited performance improvements and insufficient baseline comparisons, particularly at the 7B scale.

The performance improvements reported in Table 1 and Table 2 are relatively modest considering the substantial increase in total model parameters, and several benchmarks exhibit performance degradation. More importantly, the OLMo-2 7B experiments compare STAPLE only against continued dense training, without including other sparse upcycling baselines evaluated in the 1B experiments, such as MoE, STEM, and Engram-style models. Consequently, the current results do not adequately establish whether STAPLE maintains its relative advantages at larger model scales.

  • W3. Insufficient analysis of additional memory-access overhead and system-level efficiency.

Although STAPLE reports promising throughput results, the method introduces substantial additional CPU memory requirements and relies heavily on embedding retrieval, caching, and CPU-GPU data transfers. The system evaluation is conducted under limited hardware and workload settings, leaving its performance under different memory bandwidths, context lengths, and deployment configurations unclear. Without a more comprehensive analysis of end-to-end computational costs and total system memory consumption, the practical efficiency and scalability advantages of STAPLE remain insufficiently established.

Questions for authors

Q1. What are the key methodological differences between STAPLE and existing static sparse memory architectures, particularly Engram? Given their conceptual similarities, could the authors further clarify the distinctive novelty of STAPLE?

Q2. Why are the OLMo-2 7B experiments limited to comparisons against continued dense training? Can the authors provide comparisons with other sparse upcycling baselines, such as MoE, STEM, and Engram-style models, to demonstrate whether STAPLE maintains its advantages at larger scales?

Q3. Given the relatively modest performance improvements reported in Table 1 and Table 2, can the authors provide additional evidence supporting the effectiveness and robustness of STAPLE, particularly across different training seeds and model configurations?

Q4. How significant are the additional memory-access and CPU-offloading overheads introduced by STAPLE? Can the authors provide a more comprehensive analysis of end-to-end efficiency, including CPU memory consumption, data-transfer costs, and performance under different hardware and workload settings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ySaf (Part 1/3)

We thank the reviewer for their thorough review and constructive suggestions. We are glad that they found our approach simple and effective, and appreciated the efficiency promise of our method. We tried to clarify the novelty of our approach, and performed additional experiments to explore a broader scope of STAPLE in expanding model's performance. We hope the reviewer will consider raising their score in light of our response.

(Q1) Limited novelty over existing static sparse memory methods such as Engram. STAPLE's changes (preserving the pretrained FFN, additive token-indexed embeddings, zero initialization and warmup) are reasonable but appear incremental. The Engram-style baseline also omits parts of the original architecture, so the advantage over existing methods is not established.

We thank the reviewer for highlighting the need to clarify both our contribution and the scope of the Engram comparison. We do not claim static sparse memory itself as novel. Our contribution is to highlight an underexplored opportunity for expanding pretrained models through static sparse upcycling and identify simple changes that make it effective under limited continued-training budgets.

STAPLE builds on STEM, not Engram. The two differ mainly in where the memory enters the model: STEM places a token-indexed table inside the FFN, where the FFN's own gate modulates the retrieved row, whereas Engram retrieves hashed n-gram embeddings and fuses them into the residual stream through a separate context-conditioned gating module. Applied directly to a pretrained model, STEM underperforms dense continuation. We make three changes to STEM for the upcycling setting:

  1. Add instead of replace. STEM replaces the FFN up-projection with the table; STAPLE keeps the pretrained up-projection and adds the table output to it, so the pretrained FFN computation is preserved.
  2. Zero initialization. The tables start at zero, so the expanded model initially computes exactly the seed model's function.
  3. Selective freezing during warmup. The up-projection is frozen in the augmented layers during warmup, so the tables must learn what the backbone does not already provide.

Each change is supported by ablations (Sec. 4.1\color{blue}{\text{Sec. 4.1}}), complemented by interpretability analyses of how the added memory interacts with contextual computation (Sec. 4.2\color{blue}{\text{Sec. 4.2}}). We view STAPLE as an empirically grounded upcycling recipe whose insights may also inform Engram and related static-memory architectures.

Regarding Engram, our baseline retains hashed n-gram memory, context-conditioned key/value projections, sigmoid gating, and depthwise causal convolution. It omits multi-branch hyper-connections to retain the same single-stream pretrained backbone used by STAPLE. This allows us to compare alternative memory-integration designs under a common backbone, matched embedding-memory capacity, and the same upcycling budget, without also changing the backbone’s residual architecture. The multi-branch component can enhance our Engram-style baseline as well as the other methods we study. But our broader aim is to illustrate the potential of static sparsity in enhancing a pretrained model, not just to establish superiority of our proposed upcycling architecture. We presented simple fixes that can help us realize the benefits of existing static sparse architecture like STEM, and we believe that these insights can potentially transfer to Engram as well as its stronger variants.

AuthorsOpenReview ↗

Response to Reviewer ySaf (Part 2/3)

(Q2) Modest gains and missing sparse baselines at 7B. The improvements in Tables 1 and 2 are small relative to the added parameters, and several benchmarks regress. At 7B, STAPLE is compared only with dense continuation, not with the MoE, STEM and Engram-style baselines used at 1B, so it is unclear whether its advantages hold at larger scale.

We thank the reviewer for this feedback. We agree that STAPLE substantially increases total parameters. However, static indexing enables CPU offloading and asynchronous prefetching, allowing this capacity to be used with limited training and serving overhead. Its resource requirements therefore differ from expanding GPU-resident dense weights or MoE experts. Nevertheless, we could explore orthogonal techniques such as vocabulary compression similar to Engram, which maps semantically similar tokens to the same embedding, thus reducing the vocab size substantially (35% in our case) and potentially reducing redundancy in parameters.

Another key consideration is jointly scaling through static and dynamic sparsity to achieve maximum performance benefits. Engram’s sparsity-allocation study finds that allocating approximately 20%–25% of the sparse parameter budget to static sparse memory and 75%–80% to experts gives the best performance in its evaluated settings under fixed parameters and compute budgets. Additionally, we find that more fine-grained MoE configurations can achieve better performance although at much larger efficiency cost. To maximize performance improvement, we considered a joint scaling of fine-grained MoE and STAPLE. Interestingly, our experiments in Sec. A.6\color{blue}{\text{Sec. A.6}} shows that joint scaling provides a more favorable performance-efficiency tradeoff.

We denote MoE configurations as ExGyTk-nL: each dense FFN is split into y shards and expanded x times, giving x·y experts with top-k routing, in n MoE layers. Thus E8G8T8-8L has 64 experts and E17G8T8-8L has 136, both activating 8 per token; the coarse top-1 MoE in our submission is E8G1T1-8L.

Under the same 200B-token budget, adding STAPLE-4L to E8G8T8-8L improves the ten-task average from 47.12 to 48.35, including gains of 4.7 points on GSM8K and 3.8 on MBPP. Full results are shown below:

ModelMMLUGSM8KMATHMBPPHumanEvalARC-EARC-CHellaSwagWinoGrandeDROPAverage
E8G8T8-8L51.745.58.831.631.772.946.969.265.447.547.12
E8G8T8-8L + STAPLE-4L52.150.28.935.430.577.747.668.166.946.148.35
E17G8T8-8L52.348.48.035.832.375.146.870.765.449.748.45

The corresponding serving measurements demonstrate the practical benefit of this allocation:

ModelTotal parametersHBM model memoryDecode tok/sPrefill tok/s
E8G8T8-8L4.31B8.62 GB9.61K214.44K
E8G8T8-8L + STAPLE-4L7.59B9.62 GB9.61K212.24K
E17G8T8-8L7.93B15.86 GB7.29K206.79K

Measurements use a single H200; decoding uses batch size 64 and context length 1024. HBM model memory estimates include BF16 weights and the joint model’s 1 GB embedding cache, excluding KV-cache and runtime allocations.

The joint model approaches the larger MoE’s quality with approximately 4% fewer total parameters and 1.32× higher decoding throughput. Relative to E8G8T8 alone, it preserves decoding throughput at the reported precision with approximately 1% lower prefill throughput. Thus, although task-level gains vary, static memory provides a useful capacity-scaling option alongside stronger MoE configurations.

Results at 7B. STAPLE improves nine of ten benchmarks, raising the average from 57.11 to 58.01, with gains of 2.34 points on GSM8K, 1.60 on MBPP, and 1.22 on HumanEval. The smaller MMLU gain may reflect the reduced warmup budget - 30B versus 100B tokens, consistent with our ablations. We concentrated the extensive sparse-baseline comparisons at 1B because of the higher training cost at 7B. The current 7B results establish improvements over matched-budget dense continuation; we agree that broader sparse-baseline comparisons would strengthen the evidence of relative advantages at this scale and plan to extend them in the revision.

Seed variance. We repeated the 1B dense continuation and STAPLE-12L experiments with three seeds. Individual task scores can vary by up to 0.5 points, particularly on benchmarks with few evaluation samples such as HumanEval, but the average downstream score stays within 0.1 points.

AuthorsOpenReview ↗

Response to Reviewer ySaf (Part 3/3)

(Q3) Insufficient analysis of memory-access overhead and system-level efficiency. STAPLE adds substantial CPU memory and relies on embedding retrieval, caching and CPU-GPU transfers, but is evaluated on limited hardware and workloads. Its end-to-end cost and total system memory under different memory bandwidths, context lengths and deployment configurations remain unclear.

We thank the reviewer for this concern. Below we (1) account for STAPLE's total system memory, (2) quantify its CPU-GPU traffic, and (3) show that its efficiency holds across hardware, context lengths, and deployment configurations.

Total system memory

GPU memory (weights + cache)CPU memory
Dense2.97 GB–
STAPLE-12L, serving2.97 GB + 1 GB cache19.73 GB tables
MoE baseline (E8G1T1-8L), serving8.61 GB–
STAPLE-12L, trainingSame as dense (+~3% peak for row buffers)92 GiB per node (tables + fp32 AdamW moments), shared by all ranks
MoE baseline, training2.9x dense–
  • STAPLE's extra capacity lives in commodity CPU memory. MoE needs extra capacity in scarce GPU memory: 2.9x dense in both serving and training.
  • The serving time and training time CPU memory requirement for STAPLE is small compared with the available CPU memory of a typical GPU server (e.g. 1.5 TB on our 8-GPU host).
  • The tables compress well. Post-hoc product quantization shrinks them 7.8x, with barely 0.2 pp drop in average downstream task performance. Learned quantization could be more beneficial.

Memory-access and transfer overhead

Each lookup fetches one 16 KiB row per layer, and the GPU cache serves the frequent tokens. Caching 5-6% of the vocabulary achieves a ~80% hit rate. Token deduplication across batched tokens further reduces the host-to-device traffic. With increasing batch size, the number of unique tokens only increases sublinearly. We explain how we optimize decode and prefill workload separately in Sec. A.4\color{blue}{\text{Sec. A.4}}

Robustness across hardware, context, and deployment

We measured STAPLE (with 12 tables at 1B scale and 16 tables at 7B scale) on three platforms with different GPU-CPU interconnects and host memory systems: B200, an NVLink-connected H200 SXM node, and a PCIe-only H200 server with AMD EPYC CPUs.

SettingHardwareSTAPLE vs. dense
Decode, ctx 256, B64B200-1.5%
Decode, ctx 1K, B1 / B64H200 SXM-1.3% / -0.6%
Decode, ctx 4K / 8K / 32K, B32H200-0.8% / -0.6% / -0.3%
Prefill, 1K x B64H200-1.6%
Prefill, 4K / 8K / 32K, B32H200-2.4% / -2.2% / -1.4%
4 replicas sharing one host's CPU memory, prefill B16-64H200-1.5%
8-GPU serving under p95 TTFT/TBT SLOs8x H200 SXMSame admitted requests (192 interactive, 512 batch)
7B backbone vs 7B STAPLE-16L, decode B64H200 SXM-0.2%
  • Context length. STAPLE's overhead shrinks as context grows. Table traffic is fixed per token, while KV-cache work grows with context.
  • Shared host bandwidth. During multi-GPU inference and training, the offloaded embedding tables and their optimizer states are shared across ranks on the same server node.

Training efficiency

We measured training on one node with 8x H200 SXM, 2x AMD EPYC 9224 and 1.5 TB DDR5. The workload is OLMo2-1B STAPLE-12L, data-parallel, at 262K tokens per step (8 x 4,096 tokens per GPU).

Training step (8 GPUs, 262K tokens)Step time (ms)Throughput vs. dense
Dense OLMo2-1B6111.00
STAPLE-12L, lazy AdamW6400.955
STAPLE-12L, dense AdamW6570.93
  • Host memory. The tables and their FP32 Adam moments take 91.9 GiB. They are stored once per node in shared memory, so this cost does not grow with the number of GPUs. It is about 6% of the node's DRAM we use.
  • Traffic grows sublinearly. A 262K-token step touches only 20.6K unique tokens, about 20% of the vocabulary. A GPU reduce-scatter sums each row's gradient across ranks and assigns each rank a disjoint set of rows. Each rank therefore transfers and updates only its slice of the unique rows, with no locks and no redundant updates.
  • Efficient host-side updates. Each rank updates only rows stored on its own NUMA node, so updates never cross the inter-socket link. A fused SIMD kernel reads and writes each element once, the minimum for FP32 Adam.
  • Overlap. Updates run on host threads as each table's gradient arrives during backward. With dense AdamW, rows without a gradient need only momentum decay, so they are updated early in the step, during the forward pass. The next step waits only for the rows it reads. Lazy AdamW updates only the touched rows, and we verified that it does not hurt model quality.
Reviewer_GC3zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STAPLE, which expands pretrained dense language models by adding token-indexed embedding tables to their feed-forward layers. The method combines continued training with CPU offloading to improve downstream performance while largely preserving training and inference throughput.

Strengths

The paper addresses an important problem: increasing the capacity of an existing pretrained model without training a larger model from scratch. The approach is relatively simple, and the experiments demonstrate improvements on both 1B and 7B backbones. The offloading implementation also achieves low additional GPU-memory and throughput overhead.

Weaknesses

  1. Substantial adaptation cost and insufficient evaluation of whether backbone updates are necessary.

This is my main concern. STAPLE does not simply train the newly added embeddings. Except for temporarily freezing selected FFN up-projections during warmup, the backbone also participates in training. The 1B and 7B experiments use approximately 200B and 100B training tokens, respectively, making the demonstrated approach a substantial continued-pretraining effort rather than a lightweight upgrade to an existing model.

The reported small throughput overhead establishes that adding sparse capacity is inexpensive relative to dense continued training, but it does not establish that the overall adaptation process is inexpensive. The paper should compare training only the new embeddings, updating a subset of the backbone, and full joint training, reporting both model quality and actual training costs. Without this comparison, it remains unclear whether extensive backbone updates are necessary or whether a substantially cheaper alternative could retain most of the gains.

  1. Limited evidence of practical benefits under smaller adaptation budgets.

Although the paper includes training-progress curves, the warmup stage itself uses approximately 100B tokens in the 1B setting. These curves do not provide a clear end-to-end comparison of performance against training cost with both stages accounted for. Experiments with smaller total budgets would help establish when the added memory begins to provide worthwhile improvements and whether the method is practical for users who are not already planning large-scale continued pretraining.

Overall Assessment

The paper presents an appealing model-expansion approach and a useful systems implementation. The matched-budget dense baseline supports the effectiveness of adding sparse capacity; my concern is not that the gains merely come from additional training. However, the necessity of the expensive joint-training recipe and its performance–cost tradeoff remain insufficiently examined.

Critical strengths

The paper addresses an important problem: increasing the capacity of an existing pretrained model without training a larger model from scratch.

Critical weaknesses

Substantial adaptation cost and insufficient evaluation of whether backbone updates are necessary.

Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer GC3z (Part 1/2)

We thank the reviewer for the insightful feedback. We are glad that they found our research direction important, our approach simple and effective, and our efficient implementation impactful. We have tried to address the concern about the adaptation cost and performed additional ablation studies requested by the reviewer. We hope the reviewer will consider raising their score in light of our response.

(Q1, main concern) High adaptation cost: are backbone updates necessary? STAPLE also trains the backbone, on ~200B (1B) and ~100B (7B) tokens, which makes it substantial continued pretraining rather than a lightweight upgrade. Low per-step overhead does not make the whole adaptation cheap. The paper should compare training only the embeddings, a subset of the backbone, and everything jointly, in both quality and training cost.

We thank the reviewer for suggesting this comparison. To directly test whether training only the added memory can retain the benefits of joint training, we evaluated frozen-backbone STAPLE-12L under the same 200B-token budget:

ModelMMLUGSM8KMATHMBPPHumanEvalARC-EARC-CHellaSwagWinoGrandeDROPAvg.
Baseline (initial checkpoint)25.62.80.94.27.965.938.267.565.726.430.5
Baseline (after 200B training tokens)50.337.767.6828.823.1775.4247.4465.1866.2241.9144.39
Frozen backbone + STAPLE (L=12)35.794.73.47.89.1573.8641.9868.6667.0128.9534.13
STAPLE (L=12)51.2140.037.4431.627.4475.0445.9968.2763.7743.5645.59

Updating only the embeddings improves the initial checkpoint, but does not retain the performance achieved by dense continuation or jointly trained STAPLE, particularly on mathematics and programming tasks. This supports joint learning of the backbone and added memory under our evaluated recipe. Selective backbone updates remain a promising intermediate option, especially from a system efficiency point of view. We will perform more experiments to explore this direction and would like to share more in our final version.

We also agree that 200B tokens represent substantial continued training. However, it is crucial to note that this budget is shared by the dense baseline and STAPLE, rather than being an additional training stage required only by STAPLE. Dense continuation itself substantially improves the starting checkpoint, as the table shows. Our central question is whether adding static sparse capacity produces further improvements within that same budget, with the broader objective of making those benefits emerge as early as possible. We have clarified this setting in the abstract and introduction, defining the upcycling token budget to include all post-expansion training, including warmup and midtraining.

The additional execution cost of the expanded architecture is an important consideration, and it directly contributes to the adaptation overhead. Our architecture selection and implementation particularly focus on this aspect. Fortunately, static sparsity allows us to maintain the training throughput within 3% of the dense training throughput with no extensive HBM memory footprint, unlike MoE alternatives (Sec. 2.4\color{blue}{\text{Sec. 2.4}}).

AuthorsOpenReview ↗

Response to Reviewer GC3z (Part 2/2)

(Q2) Little evidence of benefit under smaller adaptation budgets. Warmup alone uses ~100B tokens at 1B, and the training curves do not show quality against total training cost with warmup included. Smaller-budget experiments are needed to show when the added memory starts to pay off, and whether STAPLE is practical without large-scale continued pretraining.

We thank the reviewer for highlighting the importance of evaluating benefits against the complete training investment. We agree that warmup must be included when assessing the total upcycling budget.

Our existing STAPLE-4L ablation in Figure 2\color{blue}{\text{Figure 2}} provides evidence that the full warmup stage is not necessary for all benefits. The no-warmup variant achieves comparable math/code reasoning performance to the warmup variant across much of midtraining, using approximately 100B fewer total training tokens at corresponding midtraining checkpoints. Warmup provides clearer benefits for MMLU and commonsense reasoning, motivating its inclusion in our main recipe for broader capability improvements. It is an important consideration for every upcycling strategy that introduces a substantial amount of additional parameters, and we agree that it is desirable to minimize this cost as much as possible. Because of the limited rebuttal period, we have not been able to systematically study the cumulative effect of the warmup training budget on the final average downstream performance. We shall include this study in our final revision. Overall, warmup helps STAPLE mostly with knowledge-based tasks, while STAPLE can improve performance on the advanced tasks that subsequent training phases target even without warmup, as evident in our math/code ablation studies in Figure 2\color{blue}{\text{Figure 2}}.

Warmup is a one-time investment when introducing the additional capacity; the resulting model can proceed to subsequent training phases without repeating this initialization stage. At 1B, its approximately 100B tokens correspond to 2.5% of the seed model's 4T-token pretraining budget. Our motivation is to establish useful additional capacity that can support continued learning in later phases. We nevertheless include the full warmup cost in the upcycling budget when assessing when this investment becomes worthwhile.

Dear Reviewer GC3z,

Thank you for your thoughtful and constructive feedback on our work. We have carefully addressed your comments and revised the work and manuscript accordingly. As the discussion period nears its end, we would greatly appreciate any additional questions or points of clarification. If our responses have satisfactorily addressed your concerns, we kindly ask you to consider reflecting this in your score.

Thanks again for your time.

Reviewer_WecnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes STAPLE a training solution and architecture design to add sparse memory to a pre-trained model. STAPLE expands the capacity of an existing dense language model by adding token-indexed embedding tables to its feed-forward layers, without replacing the pretrained feed-forward computation. Their solution avoids the complications of converting models to MoE. The authors evaluated their solution on two models from OLMo family. The addition itself is simple, and the underlying techniques are largely known. The scientific contribution therefore depends heavily on the strength of the empirical evidence.

Critical strengths

1- Expanding existing pretrained models without restarting expensive pretraining is an important research direction. 2- The additive design is easy to explain and preserves the seed model's computation at initialization. 3- The conclusion of the paper is interesting about replacement-based static memory performs poorly during upcycling, while additive static memory can improve performance without destroying useful pretrained features. 4- The paper is a useful empirical study which suggest a potentially useful design rule: preserve pretrained computation and attach static memory as a complementary pathway.

Critical weaknesses

1- Only one model family is evaluated. This hits the generalizability of the solution to other model families. 2- Is static sparse upcycling better than allocating the same training and memory budget to alternative adaptation methods? 3- The novelty of the method is pretty limited. Some components of this work are already established in the literature such as static token-indexed sparse memory, CPU offloading, caching, asynchronous lookup or Low-rank memory factorization. 4- The results are not very significant. At 1B, STAPLE improves seven of ten tasks, but at 7B, it improves nine of ten tasks, but the average gain is smaller. Based on the scaling behavior reported in the paper, the gain of STAPLE reduces as the size of the model grows. It might give a hint that this solution is not scalable. There is evidence that STAPLE helps, but limited evidence that the gains are large and scalable. 5- Another concern is about missing baselines. The authors compare against relevant sparse-memory methods, but their experiments do not sufficiently isolate whether the improvements come from the proposed static memory mechanism. The missing baselines which will be good to add: {LoRA / DoRA or other efficient adaptation}{Frozen backbone + trainable STAPLE}{STAPLE without selective freezing, with matched training tokens} {A stronger Engram implementation}{A stronger or better-tuned MoE configuration}. Also we could add comparison to some papers mentioned in the related work section. Some of the shown baselines are not their best form implementation. For example: their Engram-style baseline explicitly omits Engram's multi-branch hyper-connections (Section 3, p. 6). Therefore, the results establish superiority over that implementation, not necessarily over a fully optimized Engram approach. Similarly, their MoE configuration is one specific top-1 design. It does not establish superiority over MoE upcycling generally.

6- The full-rank STAPLE-12L configuration adds approximately 9.87 billion embedding parameters. This requires about 20 GB of BF16 table storage, with approximately ~90 GiB of host memory occupied by tables and optimizer moments during training. This matters because CPU DRAM, data movement, bandwidth, NUMA configuration, and caching are not free. More importantly, the serving benchmarks use only context length 256, and the paper reports batch sizes 1 and 64. This is a limitation and would be great if the authors can report their performance results for 4K , 8K, 32K as well. 7- The study concentrates on pretrained model continuation. There is no clear demonstration that the gains survive supervised instruction tuning, preference optimization, or reasoning-oriented post-training.

Questions for authors

1- Can we add results for other model families as well? Does the approach generalize beyond the chosen backbone? 2- Can we evaluate this model on long sequence tasks such as RULER and LongBench? Does increased sparse capacity improve actual long-context reasoning or retrieval, as explored by earlier static-memory methods? 3- What happens when the dense backbone is frozen throughout training and only the added memory is optimized? How much improvement can memory produce independently?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Wecn (Part 1/4)

We thank the reviewer for their thorough review and constructive suggestions. We are glad that they found our research direction important, our approach simple and effective. Since the reviewer's main concerns are about the strength of the empirical evidence, we have added new experiments: a second model family (DCLM-1B), a frozen-backbone ablation, an ablation without selective freezing, stronger and more expensive MoE baselines with joint STAPLE-MoE scaling, and an exhaustive analysis of the system efficiency of our approach. We hope the reviewer will consider raising their score in light of our response.

(Q1) Generalization and scalability. The gains are modest and appear to shrink with scale: at 1B, STAPLE improves seven of ten tasks, and at 7B it improves nine of ten but with a smaller average gain. Does STAPLE generalize beyond the chosen backbone to other model families, and are its gains large and scalable?

We appreciate the concern and agree that a broader scaling study would strengthen the evidence. However, the 1B and 7B gains are not directly comparable. Because of compute constraints, the 7B run used about half the continuation budget of the 1B run: warmup was cut from 100B to ~30B tokens and midtraining from 100B to 70B. Each scale is compared with its own matched-budget dense baseline, so the difference between the two gains does not isolate the effect of backbone size.

Despite the smaller budget, STAPLE improves the 7B average from 57.11 to 58.01 and improves nine of ten tasks, including +2.34 on GSM8K, +1.60 on MBPP and +1.22 on HumanEval. Our 1B ablations show that warmup mainly benefits knowledge and commonsense tasks, so the shorter 7B warmup may limit those gains.

Within a fixed budget, STAPLE also scales with added memory capacity at 1B:

Model (1B, same 200B-token budget)Avg.
Dense continuation44.39
STAPLE-4L45.10
STAPLE-12L, low-rank tables (r = 512)44.97
STAPLE-12L, full-rank tables45.59

More augmented layers and wider tables both improve quality. Static memory also scales alongside MoE expert scaling: adding STAPLE-4L to an E8G8T8-8L MoE raises its average from 47.12 to 48.35 (Q3\color{blue}{\text{Q3}}).

Other model families. We considered OLMo model family for our experiments primarily because their open training data, training recipe, and intermediate checkpoints. However, to address the reviewer's concern we have additionally performed experiments with another model family with open training recipe, DCLM-1B-v0, which has been pretrained on DCLM baseline pretraining data. It differs from OLMo-2 1B in tokenizer (50,432 vs. 100,352 vocabulary), depth (24 vs. 16 layers), FFN width (5,632 vs. 8,192), normalization (pre-norm with gain-only LayerNorm vs. post-sublayer RMSNorm), context length and RoPE configuration, and training stack. We add STAPLE tables to layers 1-12. Because of limited time, we used a shortened budget: under 20B warmup tokens and 80B midtraining tokens on the same midtraining mixture. This setting is conservative, since our ablations show that a shorter warmup reduces STAPLE's gains, especially on knowledge tasks such as MMLU.

Model (DCLM-1B-v0)MMLUGSM8KMATHMBPPHumanEvalARC-EARC-CHellaSwagWinoGrandeDROPAvg.
Dense continuation47.4922.744.1619.2016.4676.2245.8270.5867.4832.5640.27
STAPLE-12L47.4625.634.6020.8017.0877.0246.5970.3868.4332.5241.05

STAPLE improves seven of ten benchmarks and raises the average from 40.27 to 41.05, including relative gains of 12.7% on GSM8K and 8.3% on MBPP. Its gains are therefore not specific to the OLMo-2 architecture. In the revision, we will report DCLM-1B with a full warmup and midtraining budget matching our OLMo-2 1B setup, and add a controlled study of backbone size, memory capacity and training budget.

AuthorsOpenReview ↗

Response to Reviewer Wecn (Part 2/4)

Attribution and alternative adaptation. The experiments do not isolate whether the gains come from the static memory itself, or show that STAPLE beats spending the same training and memory budget on alternatives. Missing comparisons: LoRA/DoRA; a frozen backbone with only STAPLE trained; and STAPLE without selective freezing at matched tokens. How much does the memory improve the model on its own?

We thank the reviewer for these suggestions. We added the two requested ablations (frozen backbone, no selective freezing) and discuss LoRA/DoRA below. The results share the OLMo2-1B backbone and the 200B-token budget.

How much the memory contributes on its own. We trained STAPLE-12L with the backbone frozen, so only the tables learn:

ModelMMLUGSM8KMATHMBPPHumanEvalARC-EARC-CHellaSwagWinoGrandeDROPAvg.
Initial checkpoint25.62.80.94.27.965.938.267.565.726.430.5
Dense continuation50.337.767.6828.823.1775.4247.4465.1866.2241.9144.39
Frozen backbone + STAPLE-12L35.794.73.47.89.1573.8641.9868.6667.0128.9534.13
STAPLE-12L51.2140.037.4431.627.4475.0445.9968.2763.7743.5645.59

The memory alone improves the initial checkpoint (30.5 → 34.13), most clearly on MMLU and commonsense tasks. It cannot replace backbone adaptation, especially for math and code, so STAPLE's gain comes from training the memory and the backbone together. The trained model also relies on the memory: removing all 12 tables from STAPLE-12L at evaluation time drops the average from 45.59 to 32.17 (Sec. 4.1\color{blue}{\text{Sec. 4.1}}).

Without selective freezing. In a STAPLE layer, the up-projection output and the table output are summed, w2(silu(w1 x) ⊙ (w3 x + y)), so w3 and the tables receive the same gradient. If w3 trains during warmup, the two compete for that update signal, and w3 has every advantage: it is pretrained, dense and updated on every token. Each table row starts at zero and is updated only when its token appears. Freezing w3 during warmup removes this competition, so the tables must learn what the backbone does not already provide.

We trained STAPLE-12L without selective freezing on the same token budget:

ModelMMLUGSM8KMATHMBPPHumanEvalARC-EARC-CHellaSwagWinoGrandeDROPAvg.
Dense continuation50.337.767.6828.823.1775.4247.4465.1866.2241.9144.39
STAPLE-12L, with selective freezing51.2140.037.4431.627.4475.0445.9968.2763.7743.5645.59
STAPLE-12L, without selective freezing50.5339.28.3230.426.8375.5146.0867.265.3542.0245.14

Its average is slightly lower (45.14 vs. 45.59), and its tables end up smaller and contribute little.

Smaller tables. Median row L2 norm of the trained tables:

Layer14812
With selective freezing4.94.84.66.0
Without selective freezing3.12.92.53.0
Ratio1.6x1.7x1.8x2.0x

Without freezing, rows are 1.6-2.0x smaller in every layer, consistent with the weaker update signal the tables receive during warmup.

Limited contribution. This also limits the tables' contribution to the final model output. Points lost when all 12 tables are removed at evaluation:

ModelGSM8KHumanEvalMBPPDROPARC-EMMLUAvg.
With selective freezing-31.6-23.2-25.6-10.8-9.7-6.5-13.0
Without selective freezing-2.5-2.8-1.6-1.80.0+0.1-0.92

With selective freezing, the model relies heavily on its tables, especially for math and code. Without it, removing the tables costs 0.92 points, and leaves the model roughly at the dense continuation's level. Selective freezing is what turns the added parameters into capacity the model actually uses.

LoRA/DoRA. This can address a different question: how to update existing weights cheaply, rather than how to add capacity for subsequent training stages. A merged LoRA model has the dense architecture. In continued pretraining with this many tokens, full dense continuation (our primary control) typically matches or exceeds low-rank updates on the same data. We agree that LoRA/DoRA comparisons would further characterize the quality-cost tradeoff, and we will discuss them in the revision.

AuthorsOpenReview ↗

Response to Reviewer Wecn (Part 3/4)

Stronger sparse baselines. The comparisons omit a full Engram implementation (with multi-branch hyper-connections) and stronger MoE configurations beyond one top-1 design, so STAPLE's advantage over existing sparse methods is not established.

We thank the reviewer for raising this concern. We clarify the Engram comparison and add stronger MoE baselines in Sec. A.6\color{blue}{\text{Sec. A.6}}. All quality comparisons use the OLMo2-1B backbone and the same 200B-token budget.

Engram. Our implementation retains hashed n-gram memory, context-conditioned key/value projections, sigmoid gating, and causal convolution. We omit multi-branch hyper-connections to preserve the pretrained backbone’s single-stream residual architecture and compare memory integration without simultaneously changing the backbone. Our goal is not merely to rank static-memory architectures, but to establish how static sparse memory can effectively expand pretrained models, why naive upcycling can underperform, and how a simple recipe makes it useful. We demonstrate this through STEM; these insights may also inform upcycling with Engram and related architectures.

Stronger MoE baselines. We originally selected coarse top-1 MoE to study capacity expansion with execution cost close to dense. Finer expert segmentation can improve quality [1], but increases memory traffic and lowers arithmetic intensity, potentially reducing hardware efficiency [2]. We observe the same trend on a single H200: prefill throughput falls 4.7% below dense for the coarse top-1 E8G1T1-8L (8 experts, top-1, 8 MoE layers), 10.7% for E8G8T8-8L (64 experts, top-8, 8 MoE layers). STAPLE-12L stays within 1.6% of dense. We agree that fine-grained MoEs can provide better performance at the cost of more system inefficiency. Static memory adds capacity very cheaply, and can further complement MoE. The Engram paper points the same way: its best models give roughly 20-25% of the sparse parameter budget to static memory and the rest to experts. Our core objective is to illustrate the usefulness of static sparsity in upcycling and fortunately it can be used alongside MoE scaling to achieve better performance under a more relaxed efficiency constraint.

Please refer to Sec. A.6\color{blue}{\text{Sec. A.6}} for the details of our joint scaling study. Our joint model adds STAPLE tables at MoE layers 1, 3, 5, and 7 of E8G8T8-8L model. Each token-indexed row is divided into shards shared across corresponding experts, keeping table size independent of expert count and enabling CPU prefetch before routing. Sec. A.6\color{blue}{\text{Sec. A.6}} provides the architecture figure (Figure 6\color{blue}{\text{Figure 6}}) and formulation.

Below, E8 denotes E8G8T8-8L (64 experts, top-8), Joint adds STAPLE-4L, and E17 denotes E17G8T8-8L (136 experts, top-8). All use 100B warmup and 100B midtraining tokens.

ModelParamsMMLUGSM8KMATHMBPPHumanEvalARC-EARC-CHellaSwagWinoGrandeDROPAvg.
E84.31B51.745.58.831.631.772.946.969.265.447.547.12
Joint7.59B52.150.28.935.430.577.747.668.166.946.148.35
E177.93B52.348.48.035.832.375.146.870.765.449.748.45

Serving throughput on one H200, batch 64 and context length 1,024; percentages are relative to dense:

ModelGPU model memoryDecode tok/sPrefill tok/s
Dense2.98 GB15.52K (100%)240.05K (100%)
E88.62 GB9.61K (62%)214.44K (89.3%)
Joint9.62 GB9.61K (62%)212.24K (88.4%)
E1715.86 GB7.29K (47%)206.79K (86.1%)

Memory estimates include BF16 weights and Joint’s 1 GB GPU embedding cache, excluding KV-cache and runtime allocations; its embedding tables reside in CPU memory.

Adding STAPLE improves E8’s average by 1.23 points, including +4.7 GSM8K and +3.8 MBPP, with unchanged decode throughput at the reported precision and approximately 1% lower prefill throughput. Joint approaches E17’s average (48.35 versus 48.45) with 4% fewer parameters, 39% less GPU model memory, and 1.32× decoding throughput. These results demonstrate that static memory remains a key consideration of upcycling pretrained models.

[1] DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. 2024.

[2] SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations. 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all the reviewers for their thoughtful reviews and constructive feedback, and for recognizing the value of expanding pretrained models with static sparse memory without restarting pretraining. We have prepared point-by-point responses for each reviewer and revised the manuscript in response to your suggestions. Below we summarize the main changes that address the reviewers' common concerns.

Contribution and positioning. We do not claim static sparse memory, or its systems components, as particularly novel. Our contribution is to identify static sparse upcycling as an underexplored way to expand pretrained models, and simple fixes to an existing static sparse architecture STEM that make the added capacity pay off early in continued training: adding the table to the pretrained FFN up-projection instead of replacing it, zero initialization, and selective freezing of the up-projection during warmup. Each is supported by ablations (Sec. 4.1\color{blue}{\text{Sec. 4.1}}). We present STAPLE as an empirically grounded recipe, not merely as a superior architecture, and expect its insights to benefit other static-memory designs, including Engram (R1 Q4, R3 Q1).

Stronger sparse baselines and complementarity with MoE. We added fine-grained MoE baselines. They improve downstream performance but add more overhead on top of our existing coarse top-1 MoE baseline: prefill throughput falls 4.7% below dense for the coarse MoE and 10.7% for E8G8T8-8L, while STAPLE-12L stays within 1.6%. More importantly, we do not position static memory as a replacement for experts. Adding STAPLE-4L to E8G8T8-8L raises its average from 47.12 to 48.35 with unchanged decode throughput, nearly matching E17G8T8-8L performance (48.45), which more than doubles the experts, with 39% less GPU model memory and 1.32x higher decode throughput (Appendix A.6\color{blue}{\text{Appendix A.6}}). Static sparsity therefore remains valuable even when stronger MoE configurations are available (R1 Q3, R3 Q2). On Engram, our baseline omits multi-branch hyper-connections because they modify the backbone's residual stream, not the memory mechanism; we clarified that the comparison does not claim superiority over the full Engram architecture (R1 Q3, R3 Q1).

System cost. We added a full accounting of memory and data movement. STAPLE-12L keeps 19.73 GB of tables in CPU memory with a 1 GB GPU cache, and training needs 91.9 GiB of host memory per node, shared by all GPUs. A miss moves one 16 KiB row on a side stream that overlaps with model compute, which is under 1% of PCIe bandwidth. Serving stays within 0.8% of dense decode throughput and 2.4% of dense prefill throughput at 4K, 8K and 32K context, and the overhead shrinks as context grows. Training throughput on 8x H200 is 0.955x dense (R1 Q5, R3 Q3).


Paper Revision Summary

All changes in the revised paper are highlighted in blue.

1. Additional experiments and analysis

  1. Added Appendix A.6\color{blue}{\text{Appendix A.6}}. Joint scaling of static memory and fine-grained MoE: architecture, downstream quality and serving efficiency against E8G8T8-8L and E17G8T8-8L. (R1 Q3 and Q4, R3 Q2)
  2. Updated Section 4.1\color{blue}{\text{Section 4.1}} and added Appendix A.7\color{blue}{\text{Appendix A.7}}. Ablation without selective freezing. The model reaches a similar average (45.14 vs. 45.59) but barely uses its tables: removing them costs 0.9 points, compared with about 13 points for STAPLE. (R1 Q2, R3 Q1)

2. Writing and clarifications

  1. Updated the Abstract, Introduction. Clarified the core objective and positioning of STAPLE: the added capacity should pay off early in continued training, and the continued-training budget includes warmup and is shared with dense continuation. (R1 Q4, R2 W1 and W2, R3 Q1)
  2. Updated the Conclusion. Added the joint-scaling result, which shows the potential of STAPLE memory as a complement to MoE, and clearly states future directions. (R1 Q3 and Q4, R3 Q2)
  3. Updated Sections 3 and 4. Clarified the scope of the Engram-style and MoE comparisons, and added a paragraph that summarizes the joint-scaling results with fine-grained MoE in the main text. (R1 Q3, R3 Q1 and Q2)

We thank the reviewers again for their constructive feedback. We hope the responses and revised manuscript address your concerns, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.