- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes STAPLE, a static sparse upcycling method that expands pretrained language models by introducing token-indexed embedding modules into Transformer FFNs. The method preserves pretrained computation through an additive architecture and zero initialization, while employing a warmup strategy to facilitate adaptation. Additionally, CPU offloading and caching techniques are introduced to reduce memory and computational overhead. Experiments on OLMo-2 1B and 7B demonstrate improvements across several downstream tasks while maintaining training and inference throughput close to the dense baseline.
Critical strengths
S1. Simple and effective architecture for preserving pretrained computation.
STAPLE preserves the original FFN computation while introducing additional token-specific memory through a residual pathway. Combined with zero initialization, this design allows the expanded model to retain the functionality of the pretrained checkpoint, providing a practical approach to sparse upcycling.
S2. Promising system efficiency with large-scale static memory.
The proposed CPU offloading and caching mechanisms enable substantial parameter expansion with limited additional GPU memory consumption. The reported training and inference throughput remains close to the dense baseline, demonstrating the practical potential of static sparse memory.
Critical weaknesses
- W1. Limited methodological novelty and insufficient differentiation from existing static sparse memory methods.
STAPLE shares substantial conceptual similarities with existing static sparse memory architectures, particularly Engram. Its main contributions primarily involve preserving pretrained FFN computation, introducing additive token-indexed embeddings, and applying zero initialization and warmup. While these choices are reasonable, their methodological novelty appears relatively limited. Furthermore, the Engram-style baseline omits certain components of the original architecture, making it difficult to determine whether the reported improvements sufficiently establish STAPLE's advantages over existing methods.
- W2. Limited performance improvements and insufficient baseline comparisons, particularly at the 7B scale.
The performance improvements reported in Table 1 and Table 2 are relatively modest considering the substantial increase in total model parameters, and several benchmarks exhibit performance degradation. More importantly, the OLMo-2 7B experiments compare STAPLE only against continued dense training, without including other sparse upcycling baselines evaluated in the 1B experiments, such as MoE, STEM, and Engram-style models. Consequently, the current results do not adequately establish whether STAPLE maintains its relative advantages at larger model scales.
- W3. Insufficient analysis of additional memory-access overhead and system-level efficiency.
Although STAPLE reports promising throughput results, the method introduces substantial additional CPU memory requirements and relies heavily on embedding retrieval, caching, and CPU-GPU data transfers. The system evaluation is conducted under limited hardware and workload settings, leaving its performance under different memory bandwidths, context lengths, and deployment configurations unclear. Without a more comprehensive analysis of end-to-end computational costs and total system memory consumption, the practical efficiency and scalability advantages of STAPLE remain insufficiently established.
Questions for authors
Q1. What are the key methodological differences between STAPLE and existing static sparse memory architectures, particularly Engram? Given their conceptual similarities, could the authors further clarify the distinctive novelty of STAPLE?
Q2. Why are the OLMo-2 7B experiments limited to comparisons against continued dense training? Can the authors provide comparisons with other sparse upcycling baselines, such as MoE, STEM, and Engram-style models, to demonstrate whether STAPLE maintains its advantages at larger scales?
Q3. Given the relatively modest performance improvements reported in Table 1 and Table 2, can the authors provide additional evidence supporting the effectiveness and robustness of STAPLE, particularly across different training seeds and model configurations?
Q4. How significant are the additional memory-access and CPU-offloading overheads introduced by STAPLE? Can the authors provide a more comprehensive analysis of end-to-end efficiency, including CPU memory consumption, data-transfer costs, and performance under different hardware and workload settings?
5 comments
Response to Reviewer ySaf (Part 2/3)
(Q2) Modest gains and missing sparse baselines at 7B. The improvements in Tables 1 and 2 are small relative to the added parameters, and several benchmarks regress. At 7B, STAPLE is compared only with dense continuation, not with the MoE, STEM and Engram-style baselines used at 1B, so it is unclear whether its advantages hold at larger scale.
We thank the reviewer for this feedback. We agree that STAPLE substantially increases total parameters. However, static indexing enables CPU offloading and asynchronous prefetching, allowing this capacity to be used with limited training and serving overhead. Its resource requirements therefore differ from expanding GPU-resident dense weights or MoE experts. Nevertheless, we could explore orthogonal techniques such as vocabulary compression similar to Engram, which maps semantically similar tokens to the same embedding, thus reducing the vocab size substantially (35% in our case) and potentially reducing redundancy in parameters.
Another key consideration is jointly scaling through static and dynamic sparsity to achieve maximum performance benefits. Engram’s sparsity-allocation study finds that allocating approximately 20%–25% of the sparse parameter budget to static sparse memory and 75%–80% to experts gives the best performance in its evaluated settings under fixed parameters and compute budgets. Additionally, we find that more fine-grained MoE configurations can achieve better performance although at much larger efficiency cost. To maximize performance improvement, we considered a joint scaling of fine-grained MoE and STAPLE. Interestingly, our experiments in shows that joint scaling provides a more favorable performance-efficiency tradeoff.
We denote MoE configurations as ExGyTk-nL: each dense FFN is split into y shards and expanded x times, giving x·y experts with top-k routing, in n MoE layers. Thus E8G8T8-8L has 64 experts and E17G8T8-8L has 136, both activating 8 per token; the coarse top-1 MoE in our submission is E8G1T1-8L.
Under the same 200B-token budget, adding STAPLE-4L to E8G8T8-8L improves the ten-task average from 47.12 to 48.35, including gains of 4.7 points on GSM8K and 3.8 on MBPP. Full results are shown below:
| Model | MMLU | GSM8K | MATH | MBPP | HumanEval | ARC-E | ARC-C | HellaSwag | WinoGrande | DROP | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| E8G8T8-8L | 51.7 | 45.5 | 8.8 | 31.6 | 31.7 | 72.9 | 46.9 | 69.2 | 65.4 | 47.5 | 47.12 |
| E8G8T8-8L + STAPLE-4L | 52.1 | 50.2 | 8.9 | 35.4 | 30.5 | 77.7 | 47.6 | 68.1 | 66.9 | 46.1 | 48.35 |
| E17G8T8-8L | 52.3 | 48.4 | 8.0 | 35.8 | 32.3 | 75.1 | 46.8 | 70.7 | 65.4 | 49.7 | 48.45 |
The corresponding serving measurements demonstrate the practical benefit of this allocation:
| Model | Total parameters | HBM model memory | Decode tok/s | Prefill tok/s |
|---|---|---|---|---|
| E8G8T8-8L | 4.31B | 8.62 GB | 9.61K | 214.44K |
| E8G8T8-8L + STAPLE-4L | 7.59B | 9.62 GB | 9.61K | 212.24K |
| E17G8T8-8L | 7.93B | 15.86 GB | 7.29K | 206.79K |
Measurements use a single H200; decoding uses batch size 64 and context length 1024. HBM model memory estimates include BF16 weights and the joint model’s 1 GB embedding cache, excluding KV-cache and runtime allocations.
The joint model approaches the larger MoE’s quality with approximately 4% fewer total parameters and 1.32× higher decoding throughput. Relative to E8G8T8 alone, it preserves decoding throughput at the reported precision with approximately 1% lower prefill throughput. Thus, although task-level gains vary, static memory provides a useful capacity-scaling option alongside stronger MoE configurations.
Results at 7B. STAPLE improves nine of ten benchmarks, raising the average from 57.11 to 58.01, with gains of 2.34 points on GSM8K, 1.60 on MBPP, and 1.22 on HumanEval. The smaller MMLU gain may reflect the reduced warmup budget - 30B versus 100B tokens, consistent with our ablations. We concentrated the extensive sparse-baseline comparisons at 1B because of the higher training cost at 7B. The current 7B results establish improvements over matched-budget dense continuation; we agree that broader sparse-baseline comparisons would strengthen the evidence of relative advantages at this scale and plan to extend them in the revision.
Seed variance. We repeated the 1B dense continuation and STAPLE-12L experiments with three seeds. Individual task scores can vary by up to 0.5 points, particularly on benchmarks with few evaluation samples such as HumanEval, but the average downstream score stays within 0.1 points.
Response to Reviewer ySaf (Part 3/3)
(Q3) Insufficient analysis of memory-access overhead and system-level efficiency. STAPLE adds substantial CPU memory and relies on embedding retrieval, caching and CPU-GPU transfers, but is evaluated on limited hardware and workloads. Its end-to-end cost and total system memory under different memory bandwidths, context lengths and deployment configurations remain unclear.
We thank the reviewer for this concern. Below we (1) account for STAPLE's total system memory, (2) quantify its CPU-GPU traffic, and (3) show that its efficiency holds across hardware, context lengths, and deployment configurations.
Total system memory
| GPU memory (weights + cache) | CPU memory | |
|---|---|---|
| Dense | 2.97 GB | – |
| STAPLE-12L, serving | 2.97 GB + 1 GB cache | 19.73 GB tables |
| MoE baseline (E8G1T1-8L), serving | 8.61 GB | – |
| STAPLE-12L, training | Same as dense (+~3% peak for row buffers) | 92 GiB per node (tables + fp32 AdamW moments), shared by all ranks |
| MoE baseline, training | 2.9x dense | – |
- STAPLE's extra capacity lives in commodity CPU memory. MoE needs extra capacity in scarce GPU memory: 2.9x dense in both serving and training.
- The serving time and training time CPU memory requirement for STAPLE is small compared with the available CPU memory of a typical GPU server (e.g. 1.5 TB on our 8-GPU host).
- The tables compress well. Post-hoc product quantization shrinks them 7.8x, with barely 0.2 pp drop in average downstream task performance. Learned quantization could be more beneficial.
Memory-access and transfer overhead
Each lookup fetches one 16 KiB row per layer, and the GPU cache serves the frequent tokens. Caching 5-6% of the vocabulary achieves a ~80% hit rate. Token deduplication across batched tokens further reduces the host-to-device traffic. With increasing batch size, the number of unique tokens only increases sublinearly. We explain how we optimize decode and prefill workload separately in
Robustness across hardware, context, and deployment
We measured STAPLE (with 12 tables at 1B scale and 16 tables at 7B scale) on three platforms with different GPU-CPU interconnects and host memory systems: B200, an NVLink-connected H200 SXM node, and a PCIe-only H200 server with AMD EPYC CPUs.
| Setting | Hardware | STAPLE vs. dense |
|---|---|---|
| Decode, ctx 256, B64 | B200 | -1.5% |
| Decode, ctx 1K, B1 / B64 | H200 SXM | -1.3% / -0.6% |
| Decode, ctx 4K / 8K / 32K, B32 | H200 | -0.8% / -0.6% / -0.3% |
| Prefill, 1K x B64 | H200 | -1.6% |
| Prefill, 4K / 8K / 32K, B32 | H200 | -2.4% / -2.2% / -1.4% |
| 4 replicas sharing one host's CPU memory, prefill B16-64 | H200 | -1.5% |
| 8-GPU serving under p95 TTFT/TBT SLOs | 8x H200 SXM | Same admitted requests (192 interactive, 512 batch) |
| 7B backbone vs 7B STAPLE-16L, decode B64 | H200 SXM | -0.2% |
- Context length. STAPLE's overhead shrinks as context grows. Table traffic is fixed per token, while KV-cache work grows with context.
- Shared host bandwidth. During multi-GPU inference and training, the offloaded embedding tables and their optimizer states are shared across ranks on the same server node.
Training efficiency
We measured training on one node with 8x H200 SXM, 2x AMD EPYC 9224 and 1.5 TB DDR5. The workload is OLMo2-1B STAPLE-12L, data-parallel, at 262K tokens per step (8 x 4,096 tokens per GPU).
| Training step (8 GPUs, 262K tokens) | Step time (ms) | Throughput vs. dense |
|---|---|---|
| Dense OLMo2-1B | 611 | 1.00 |
| STAPLE-12L, lazy AdamW | 640 | 0.955 |
| STAPLE-12L, dense AdamW | 657 | 0.93 |
- Host memory. The tables and their FP32 Adam moments take 91.9 GiB. They are stored once per node in shared memory, so this cost does not grow with the number of GPUs. It is about 6% of the node's DRAM we use.
- Traffic grows sublinearly. A 262K-token step touches only 20.6K unique tokens, about 20% of the vocabulary. A GPU reduce-scatter sums each row's gradient across ranks and assigns each rank a disjoint set of rows. Each rank therefore transfers and updates only its slice of the unique rows, with no locks and no redundant updates.
- Efficient host-side updates. Each rank updates only rows stored on its own NUMA node, so updates never cross the inter-socket link. A fused SIMD kernel reads and writes each element once, the minimum for FP32 Adam.
- Overlap. Updates run on host threads as each table's gradient arrives during backward. With dense AdamW, rows without a gradient need only momentum decay, so they are updated early in the step, during the forward pass. The next step waits only for the rows it reads. Lazy AdamW updates only the touched rows, and we verified that it does not hurt model quality.
Response to Reviewer ySaf (Part 1/3)
We thank the reviewer for their thorough review and constructive suggestions. We are glad that they found our approach simple and effective, and appreciated the efficiency promise of our method. We tried to clarify the novelty of our approach, and performed additional experiments to explore a broader scope of STAPLE in expanding model's performance. We hope the reviewer will consider raising their score in light of our response.
We thank the reviewer for highlighting the need to clarify both our contribution and the scope of the Engram comparison. We do not claim static sparse memory itself as novel. Our contribution is to highlight an underexplored opportunity for expanding pretrained models through static sparse upcycling and identify simple changes that make it effective under limited continued-training budgets.
STAPLE builds on STEM, not Engram. The two differ mainly in where the memory enters the model: STEM places a token-indexed table inside the FFN, where the FFN's own gate modulates the retrieved row, whereas Engram retrieves hashed n-gram embeddings and fuses them into the residual stream through a separate context-conditioned gating module. Applied directly to a pretrained model, STEM underperforms dense continuation. We make three changes to STEM for the upcycling setting:
Each change is supported by ablations (Sec. 4.1), complemented by interpretability analyses of how the added memory interacts with contextual computation (Sec. 4.2). We view STAPLE as an empirically grounded upcycling recipe whose insights may also inform Engram and related static-memory architectures.
Regarding Engram, our baseline retains hashed n-gram memory, context-conditioned key/value projections, sigmoid gating, and depthwise causal convolution. It omits multi-branch hyper-connections to retain the same single-stream pretrained backbone used by STAPLE. This allows us to compare alternative memory-integration designs under a common backbone, matched embedding-memory capacity, and the same upcycling budget, without also changing the backbone’s residual architecture. The multi-branch component can enhance our Engram-style baseline as well as the other methods we study. But our broader aim is to illustrate the potential of static sparsity in enhancing a pretrained model, not just to establish superiority of our proposed upcycling architecture. We presented simple fixes that can help us realize the benefits of existing static sparse architecture like STEM, and we believe that these insights can potentially transfer to Engram as well as its stronger variants.