- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces MIDUS, a depth up‑scaling (DUS) method that replaces full duplicated Transformer blocks with lightweight memory‑augmented blocks. Instead of heavy FFN‑based expansion, MIDUS leverages head‑wise product‑key sparse memory: each attention head uses its own key space, while value storage is shared across heads with head‑specific projections to retain per‑head expressiveness. Experiments across Llama‑3.2‑1B, Llama‑3.1‑8B and Qwen‑2.5‑3B demonstrate that MIDUS matches or outperforms strong DUS baselines (OpT‑DeUS, LESA, Llama‑Pro) on CPT, SFT, general, math, retrieval‑heavy and long‑context tasks, while greatly cutting trainable parameters, training GPU memory and boosting throughput. Extensive ablations and head‑importance analysis are also presented.
Critical strengths
-
Novel perspective for DUS: Prior DUS focuses on copying full Transformer blocks. This work explores sparse memory as an alternative added capacity, opening a new efficient up‑scaling direction.
-
Competitive performance‑efficiency trade‑off: MIDUS achieves comparable or better downstream results with 8.3× fewer trainable parameters and 42% lower peak training memory than OpT‑DeUS. Results transfer across model families without extra hyper‑parameter tuning.
-
Practical engineering considerations: Zero‑initialized value bank preserves pre‑trained behavior at initialization; customized implementations handle sparse‑memory pain‑points such as gradient atomic contention.
Critical weaknesses
- The theoretical analysis is built upon the fixed‑retrieval assumption, while product‑key parameters are jointly updated alongside other model weights in practical end‑to‑end training. Consequently, the presented theorems can only deliver partial structural insight for understanding value‑realization trade‑offs and do not fully capture the optimization dynamics of real‑world MIDUS training.
- Experimental results on Llama‑3.1‑8B and Qwen‑2.5‑3B are produced from single‑seed executions only, which makes it hard to estimate result variance and statistical significance. Furthermore, the paper lacks empirical evidence on 70B‑scale or larger language models, leaving it unvalidated whether the observed performance‑efficiency benefits of MIDUS can generalize to much higher model scales.
- MIDUS introduces a higher practical deployment barrier. It requires specialized custom operators to implement head‑wise product‑key retrieval, fused Cartesian top‑k selection, and deduplicated gradient aggregation. These customized components cannot be natively parsed or accelerated by mainstream off‑the‑shelf inference frameworks such as vLLM or TensorRT‑LLM.
Questions for authors
- Your core theoretical theorem relies on the fixed‑retrieval assumption, holding retrieval weights constant for analysis. In real continual pre‑training and fine‑tuning, however, product‑key banks are learnable and updated throughout training, resulting in dynamic retrieval patterns. Could you comment on the practical performance gap between your derived theoretical optimum and the real learned MIDUS model?
- Sparse memory systems often suffer from retrieval degeneracy, where some memory slots become dead or heavily under‑utilized during training. Did you observe such dead slots or imbalanced memory usage when training MIDUS? If these issues occur, what mitigation strategies could be adopted to improve memory utilization?
- All throughput numbers are measured under your bespoke PyTorch decoding loop with FlashAttention‑2, instead of production‑grade inference frameworks like vLLM. While baselines such as OpT‑DeUS consist of vanilla Transformer blocks and could fully benefit from vLLM’s paged‑attention optimizations, could you comment on how much the reported throughput gap would shrink or reverse if all baseline methods are benchmarked inside vLLM? Are there any practical paths to port MIDUS into production inference frameworks?
6 comments
For the shared value bank, the uniform-reference count million means that the and thresholds correspond to approximately and cumulative slot selections, respectively. In the observed training run, no shared value rows fall at or below the threshold, and of rows exceed the threshold. Meanwhile, the key addresses per head are virtual Cartesian pairs formed from just physical sub-keys, each of which can receive gradients through multiple selected pairs. Thus, infrequently selected Cartesian pairs do not necessarily imply unused physical sub-keys. These results show broad retrieval coverage of the shared value bank despite uneven usage of head-specific key pairs.
In the observed CPT run, our current training configuration achieved full cumulative retrieval coverage of the shared value bank, using shared storage across heads, zero weight decay, and a constant learning rate for memory parameters. Load-balancing losses or sub-key entropy regularization could further improve key-space utilization.
W2: Multi-Seed Evaluation and Scaling Beyond 8B
To address result variance on larger backbones, we have completed three-seed evaluations on Llama-3.1-8B using the same training data and training schedule across methods. The full results are provided in Table 6 of the revised manuscript and in our response to W2&Q2 of Reviewer vz6A. The reproduced results show low run-to-run variation while maintaining competitive performance across CPT and SFT. Additional three-seed results for Qwen-2.5-3B will be provided in a follow-up.
While CPT at the 70B scale exceeds our available computational resources, our experiments cover 1B, 3B, and 8B backbones across Llama and Qwen. These results support MIDUS's performance–efficiency trade-off across the evaluated model scales and families.
Q3&W3: Inference Framework Compatibility
We clarify that MIDUS is architecturally compatible with vLLM's PagedAttention. PagedAttention operates on attention and the KV cache, and it does not require the surrounding block to contain a dense FFN. MIDUS preserves the native QKV computation and KV-cache structure, and its memory branch consumes attention-head outputs. The omitted output projection lies outside the paged-attention computation and therefore does not affect this compatibility. Consequently, PagedAttention's attention and cache-management optimizations are applicable to MIDUS as well as OpT-DeUS. A vLLM port can reuse existing attention backends while adapting the model interfaces and integrating the memory branch.
The customized deduplicated gradient aggregation (Algorithm 1) is used only during training. At inference, cached head-specific values (, Appendix A.3) eliminate repeated value projections, leaving small sub-key multiplications (), Top- () selection, and weighted table lookup in the memory branch. OpT-DeUS retains dense FFN projections in its inserted blocks, whereas MIDUS avoids these projections and uses substantially fewer model parameters. PagedAttention does not eliminate this architectural difference.
As promised in our initial response, we have finished running the 3-seed experiments for SOLAR and LESA, both CPT and SFT, on Llama-3.1-8B.
| Stage | Method | PPL↓ | ARC | LogiQA | Wino | CSQA | BoolQ | PIQA | MMLU | Average↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| CPT | Base | 9.83±0.00 | 79.39±0.18 | 26.16±1.07 | 70.80±0.72 | 51.19±2.30 | 79.45±0.31 | 79.09±0.22 | 50.96±1.11 | 62.44±0.49 |
| SOLAR | 9.87±0.01 | 79.50±0.27 | 26.27±1.09 | 71.40±0.21 | 53.21±3.40 | 79.16±0.49 | 78.80±0.09 | 51.15±0.46 | 62.79±0.64 | |
| Llama Pro | 7.76±0.00 | 81.52±0.13 | 28.11±0.82 | 72.95±0.27 | 69.45±0.07 | 82.26±0.11 | 80.00±0.02 | 61.20±0.10 | 67.93±0.14 | |
| LESA | 8.06±0.13 | 81.19±0.40 | 28.78±0.98 | 73.51±0.10 | 68.14±2.57 | 82.51±0.26 | 79.98±0.12 | 60.28±2.01 | 67.77±0.65 | |
| OpT‑DeUS | 7.43±0.02 | 82.11±0.15 | 27.14±0.69 | 74.48±0.45 | 71.88±0.10 | 81.72±0.32 | 80.21±0.05 | 63.08±0.09 | 68.66±0.18 | |
| Avg‑DeUS | 7.83±0.00 | 82.12±0.02 | 26.68±0.08 | 73.14±0.21 | 71.94±0.10 | 81.72±0.07 | 80.00±0.02 | 62.66±0.04 | 68.32±0.04 | |
| MIDUS | 7.35±0.00 | 82.55±0.04 | 27.86±0.40 | 74.37±0.29 | 71.99±0.07 | 82.88±0.09 | 80.34±0.22 | 63.36±0.21 | 69.05±0.06 | |
| SFT | Base | 9.62±0.00 | 79.94±0.11 | 26.06±0.19 | 70.51±0.20 | 58.23±0.24 | 80.10±0.06 | 79.27±0.08 | 53.86±0.07 | 64.00±0.04 |
| SOLAR | 9.70±0.00 | 79.98±0.05 | 26.68±0.08 | 70.61±0.16 | 55.59±0.19 | 80.00±0.02 | 79.22±0.05 | 52.73±0.05 | 63.54±0.04 | |
| Llama Pro | 7.82±0.00 | 83.00±0.04 | 27.44±0.15 | 74.64±0.30 | 71.55±0.14 | 82.97±0.00 | 80.74±0.08 | 61.82±0.12 | 68.88±0.04 | |
| LESA | 7.97±0.00 | 83.85±0.04 | 26.37±0.26 | 75.19±0.21 | 72.48±0.18 | 83.42±0.04 | 80.61±0.09 | 64.15±0.01 | 69.44±0.09 | |
| OpT‑DeUS | 7.46±0.00 | 83.92±0.13 | 28.16±0.07 | 75.51±0.43 | 72.92±0.04 | 83.26±0.19 | 80.85±0.16 | 64.64±0.11 | 69.89±0.05 | |
| Avg‑DeUS | 7.90±0.00 | 83.80±0.04 | 26.58±0.33 | 75.43±0.26 | 72.32±0.13 | 83.02±0.11 | 80.54±0.10 | 64.10±0.09 | 69.40±0.06 | |
| MIDUS | 7.49±0.03 | 84.34±0.15 | 27.44±0.51 | 75.45±0.47 | 73.22±0.00 | 83.90±0.03 | 81.12±0.18 | 64.17±0.07 | 69.95±0.05 |
We also ran 3-seed evaluations on Qwen-2.5-3B to make the comparison more solid.
| Method | PPL | ARC | LogiQA | Wino | CSQA | BoolQ | PIQA | MMLU | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Base | 10.94±0.09 | 79.88±0.21 | 31.23±0.19 | 69.53±0.53 | 69.94±0.33 | 75.49±0.18 | 77.55±0.27 | 62.52±0.17 | 66.59±0.16 |
| OpT‑DeUS | 12.24±0.00 | 78.41±0.24 | 30.82±0.48 | 70.96±0.23 | 74.96±0.14 | 78.90±0.09 | 77.95±0.13 | 64.24±0.00 | 68.03±0.01 |
| MIDUS | 10.63±0.10 | 79.29±0.27 | 33.59±1.13 | 70.85±0.39 | 74.91±1.60 | 78.64±0.39 | 78.02±0.16 | 64.66±0.16 | 68.56±0.10 |
Even across multiple random seeds, MIDUS holds up well, matching strong baselines like OpT-DeUS and LESA in accuracy while offering much better efficiency. To briefly recap our earlier responses to your main points, we first noted that because MIDUS leaves the core attention mechanism untouched, existing inference optimizations in engines like vLLM carry over directly without custom kernel workarounds for attention structure. We also plotted the slot utilization distribution for the memory module and confirmed that sharing V across heads effectively prevents dead slots in the value space. Finally, we addressed your questions about the assumptions and implications of Theorem 1 by extending the theoretical bound and backing it up with direct empirical comparisons in practical training setups.
Thanks again for the time and thought you put into reviewing our paper. Please let us know if anything remains unclear or needs further discussion. If our new results and clarifications have resolved your concerns, we would really appreciate it if you could consider updating your score.
We sincerely thank you for recognizing the clarity of our presentation, our sparse-memory perspective on DUS, and practical engineering choices such as zero initialization and deduplicated gradient aggregation. Your insightful questions on dynamic end-to-end retrieval, memory-slot utilization, multi-seed reliability, and production inference deployment directly guided our new experiments and analyses below.
W1&Q1: Fixed retrieval and practical training
We agree that the fixed-retrieval analysis does not characterize the full optimization trajectory. Its purpose is to isolate the effect of value parameterization from changes in retrieval. In the revised manuscript, Theorem 3.1 (Section 3.3, Appendix A.12) covers arbitrary fixed head-specific retrieval matrices and approximate factorization:
LInd⋆≤LMIDUS⋆≤min(LShare⋆,LInd⋆+2ρ∑i>dhσi2(V⋆)), where V⋆ concatenates optimal independent banks and ρ=maxh∥Ah∥op2.
This relationship applies at any frozen retrieval snapshot produced by joint training, with the targets held fixed. It bounds the approximation cost of the value parameterization. It does not guarantee that the learned parameters attain the corresponding conditional optimum.
We do not directly measure the learned-to-optimal gap. The theorem compares fixed-target squared losses, whereas actual training minimizes the language-model objective. To assess whether factorized values introduce a practical performance penalty, we added a controlled comparison retaining head-wise queries and distinct key spaces while varying the value parameterization (Table 12):
Under joint product-key training, independent values recover much of the performance lost with shared values. MIDUS achieves comparable reported performance to independent values with 21.3 times fewer learned value parameters, including projections, and a 32 times smaller learned value table. These results show no observed performance degradation from factorization in this setting, without implying attainment of the theoretical optimum.
W1&Q2: Optimization dynamics of real‑world MIDUS and Memory Slot Utilization and Mitigation Strategies
We tracked cumulative Top-k (k=4) slot selections across all 12,104 CPT steps on Llama-3.2-1B (8 memory blocks, H=32, N=4,096, refer to Appendix A.11 and Figure 7 for visualization and more details). Because total token queries are large, uniform selection corresponds to μkey=1.55M calls per composite key pair and μvalue=32μkey=49.6M calls per shared value row in V. The table below reports both the absolute call cutoffs and the mean number of slots out of 4,096 receiving ≤a% of each bank's expected calls.
Permanently dead slots (a=0%) are negligible in keys (0.79%) and completely absent in V (0.00%). Because μvalue is scaled by 32×, both banks would show identical percentages if heads selected the same slots, so the drop from 65.47% to 6.06% at a≤1% confirms that heads form complementary retrieval patterns across V.