acceptodds
Under review as a conference paper at ICLR 2027

MIDUS: Memory-Infused Depth Up-Scaling

Abstract

Depth Up-Scaling (DUS) expands pre-trained language models by inserting additional Transformer blocks, but duplicating FFN-heavy structures increases both parameter count and dense per-token computation. Sparse retrieval offers an alternative by decoupling stored capacity from dense computation, but efficient scaling requires avoiding large memory overhead while preserving the heterogeneous representations of individual attention heads. We propose Memory-Infused Depth Up-Scaling (MIDUS), which organizes memory around attention heads using distinct product-key spaces and shared value storage with head-specific projections. Under common retrieval weights, we show that shared value realization incurs a loss gap when different heads require different outputs, while MIDUS recovers the independent optimum under a shared factorization. Empirically, MIDUS remains competitive with strong DUS baselines while substantially reducing cost. On Llama-3.1-8B, MIDUS uses 8.3 fewer trainable parameters and 42% less peak training GPU memory than OpT-DeUS, together with higher generation throughput. Head-importance analysis further shows a more concentrated distribution across attention heads, consistent with the proposed head-wise memory design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Scaling, Efficient Language Model, Memory-augmented Neural Network, Depth Up-scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wuTVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces MIDUS, a depth up‑scaling (DUS) method that replaces full duplicated Transformer blocks with lightweight memory‑augmented blocks. Instead of heavy FFN‑based expansion, MIDUS leverages head‑wise product‑key sparse memory: each attention head uses its own key space, while value storage is shared across heads with head‑specific projections to retain per‑head expressiveness. Experiments across Llama‑3.2‑1B, Llama‑3.1‑8B and Qwen‑2.5‑3B demonstrate that MIDUS matches or outperforms strong DUS baselines (OpT‑DeUS, LESA, Llama‑Pro) on CPT, SFT, general, math, retrieval‑heavy and long‑context tasks, while greatly cutting trainable parameters, training GPU memory and boosting throughput. Extensive ablations and head‑importance analysis are also presented.

Critical strengths
  1. Novel perspective for DUS: Prior DUS focuses on copying full Transformer blocks. This work explores sparse memory as an alternative added capacity, opening a new efficient up‑scaling direction.

  2. Competitive performance‑efficiency trade‑off: MIDUS achieves comparable or better downstream results with 8.3× fewer trainable parameters and 42% lower peak training memory than OpT‑DeUS. Results transfer across model families without extra hyper‑parameter tuning.

  3. Practical engineering considerations: Zero‑initialized value bank preserves pre‑trained behavior at initialization; customized implementations handle sparse‑memory pain‑points such as gradient atomic contention.

Critical weaknesses
  1. The theoretical analysis is built upon the fixed‑retrieval assumption, while product‑key parameters are jointly updated alongside other model weights in practical end‑to‑end training. Consequently, the presented theorems can only deliver partial structural insight for understanding value‑realization trade‑offs and do not fully capture the optimization dynamics of real‑world MIDUS training.
  2. Experimental results on Llama‑3.1‑8B and Qwen‑2.5‑3B are produced from single‑seed executions only, which makes it hard to estimate result variance and statistical significance. Furthermore, the paper lacks empirical evidence on 70B‑scale or larger language models, leaving it unvalidated whether the observed performance‑efficiency benefits of MIDUS can generalize to much higher model scales.
  3. MIDUS introduces a higher practical deployment barrier. It requires specialized custom operators to implement head‑wise product‑key retrieval, fused Cartesian top‑k selection, and deduplicated gradient aggregation. These customized components cannot be natively parsed or accelerated by mainstream off‑the‑shelf inference frameworks such as vLLM or TensorRT‑LLM.
Questions for authors
  1. Your core theoretical theorem relies on the fixed‑retrieval assumption, holding retrieval weights constant for analysis. In real continual pre‑training and fine‑tuning, however, product‑key banks are learnable and updated throughout training, resulting in dynamic retrieval patterns. Could you comment on the practical performance gap between your derived theoretical optimum and the real learned MIDUS model?
  2. Sparse memory systems often suffer from retrieval degeneracy, where some memory slots become dead or heavily under‑utilized during training. Did you observe such dead slots or imbalanced memory usage when training MIDUS? If these issues occur, what mitigation strategies could be adopted to improve memory utilization?
  3. All throughput numbers are measured under your bespoke PyTorch decoding loop with FlashAttention‑2, instead of production‑grade inference frameworks like vLLM. While baselines such as OpT‑DeUS consist of vanilla Transformer blocks and could fully benefit from vLLM’s paged‑attention optimizations, could you comment on how much the reported throughput gap would shrink or reverse if all baseline methods are benchmarked inside vLLM? Are there any practical paths to port MIDUS into production inference frameworks?
6 comments

We sincerely thank you for recognizing the clarity of our presentation, our sparse-memory perspective on DUS, and practical engineering choices such as zero initialization and deduplicated gradient aggregation. Your insightful questions on dynamic end-to-end retrieval, memory-slot utilization, multi-seed reliability, and production inference deployment directly guided our new experiments and analyses below.

W1&Q1: Fixed retrieval and practical training

We agree that the fixed-retrieval analysis does not characterize the full optimization trajectory. Its purpose is to isolate the effect of value parameterization from changes in retrieval. In the revised manuscript, Theorem 3.1 (Section 3.3, Appendix A.12) covers arbitrary fixed head-specific retrieval matrices and approximate factorization:

LInd⋆≤LMIDUS⋆≤min⁡(LShare⋆,LInd⋆+ρ2∑i>dhσi2(V⋆)),\mathcal{L}_{\mathrm{Ind}}^\star \le \mathcal{L}_{\mathrm{MIDUS}}^\star \le \min\left(\mathcal{L}_{\mathrm{Share}}^\star, \mathcal{L}_{\mathrm{Ind}}^\star + \frac{\rho}{2}\sum_{i > d_h}\sigma_i^2(V^\star)\right), where V⋆V^\star concatenates optimal independent banks and ρ=max⁡h∥Ah∥op2\rho=\max_h\|A_h\|_{\mathrm{op}}^2.

This relationship applies at any frozen retrieval snapshot produced by joint training, with the targets held fixed. It bounds the approximation cost of the value parameterization. It does not guarantee that the learned parameters attain the corresponding conditional optimum.

We do not directly measure the learned-to-optimal gap. The theorem compares fixed-target squared losses, whereas actual training minimizes the language-model objective. To assess whether factorized values introduce a practical performance penalty, we added a controlled comparison retaining head-wise queries and distinct key spaces while varying the value parameterization (Table 12):

Value parameterizationValue parameters per blockWikiText PPL ↓Average accuracy ↑
Shared Values0.26M11.8652.52
Independent Values8.39M11.6853.29
MIDUS0.39M11.6453.50

Under joint product-key training, independent values recover much of the performance lost with shared values. MIDUS achieves comparable reported performance to independent values with 21.3 times fewer learned value parameters, including projections, and a 32 times smaller learned value table. These results show no observed performance degradation from factorization in this setting, without implying attainment of the theoretical optimum.

W1&Q2: Optimization dynamics of real‑world MIDUS and Memory Slot Utilization and Mitigation Strategies

We tracked cumulative Top-kk (k=4k=4) slot selections across all 12,10412{,}104 CPT steps on Llama-3.2-1B (88 memory blocks, H=32H=32, N=4,096N=4{,}096, refer to Appendix A.11 and Figure 7 for visualization and more details). Because total token queries are large, uniform selection corresponds to μkey=1.55M\mu_{\mathrm{key}} = 1.55\text{M} calls per composite key pair and μvalue=32μkey=49.6M\mu_{\mathrm{value}} = 32\mu_{\mathrm{key}} = 49.6\text{M} calls per shared value row in V‾\overline{V}. The table below reports both the absolute call cutoffs and the mean number of slots out of 4,0964{,}096 receiving ≤a%\le a\% of each bank's expected calls.

Threshold aaKey Call CutoffHead‑Specific Key PairsValue Call CutoffShared Value V‾\overline{V}
0%0\% (Strict Dead)0032.54 (0.79%)000.00 (0.00%)
0.01%0.01\%≤155\le 155347.73 (8.49%)≤4.96K\le 4.96\text{K}0.00 (0.00%)
0.05%0.05\%≤775\le 775817.64 (19.96%)≤24.8K\le 24.8\text{K}0.00 (0.00%)
0.1%0.1\%≤1.55K\le 1.55\text{K}1,148.23 (28.03%)≤49.6K\le 49.6\text{K}0.00 (0.00%)
0.5%0.5\%≤7.75K\le 7.75\text{K}2,197.73 (53.66%)≤248K\le 248\text{K}47.38 (1.16%)
1%1\%≤15.5K\le 15.5\text{K}2,681.60 (65.47%)≤496K\le 496\text{K}248.12 (6.06%)

Permanently dead slots (a=0%a=0\%) are negligible in keys (0.79%) and completely absent in V‾\overline{V} (0.00%). Because μvalue\mu_{\mathrm{value}} is scaled by 32×32\times, both banks would show identical percentages if heads selected the same slots, so the drop from 65.47% to 6.06% at a≤1%a \le 1\% confirms that heads form complementary retrieval patterns across V‾\overline{V}.

For the shared value bank, the uniform-reference count μvalue≈49.6\mu_{\mathrm{value}} \approx 49.6 million means that the 0.1%0.1\% and 1%1\% thresholds correspond to approximately 49,60049{,}600 and 496,000496{,}000 cumulative slot selections, respectively. In the observed training run, no shared value rows fall at or below the 0.1%0.1\% threshold, and 93.94%93.94\% of rows exceed the 1%1\% threshold. Meanwhile, the 4,0964{,}096 key addresses per head are virtual Cartesian pairs formed from just 2N=1282\sqrt{N}=128 physical sub-keys, each of which can receive gradients through multiple selected pairs. Thus, infrequently selected Cartesian pairs do not necessarily imply unused physical sub-keys. These results show broad retrieval coverage of the shared value bank despite uneven usage of head-specific key pairs.

In the observed CPT run, our current training configuration achieved full cumulative retrieval coverage of the shared value bank, using shared storage across heads, zero weight decay, and a constant learning rate for memory parameters. Load-balancing losses or sub-key entropy regularization could further improve key-space utilization.

W2: Multi-Seed Evaluation and Scaling Beyond 8B

To address result variance on larger backbones, we have completed three-seed evaluations on Llama-3.1-8B using the same training data and training schedule across methods. The full results are provided in Table 6 of the revised manuscript and in our response to W2&Q2 of Reviewer vz6A. The reproduced results show low run-to-run variation while maintaining competitive performance across CPT and SFT. Additional three-seed results for Qwen-2.5-3B will be provided in a follow-up.

While CPT at the 70B scale exceeds our available computational resources, our experiments cover 1B, 3B, and 8B backbones across Llama and Qwen. These results support MIDUS's performance–efficiency trade-off across the evaluated model scales and families.

Q3&W3: Inference Framework Compatibility

We clarify that MIDUS is architecturally compatible with vLLM's PagedAttention. PagedAttention operates on attention and the KV cache, and it does not require the surrounding block to contain a dense FFN. MIDUS preserves the native QKV computation and KV-cache structure, and its memory branch consumes attention-head outputs. The omitted output projection lies outside the paged-attention computation and therefore does not affect this compatibility. Consequently, PagedAttention's attention and cache-management optimizations are applicable to MIDUS as well as OpT-DeUS. A vLLM port can reuse existing attention backends while adapting the model interfaces and integrating the memory branch.

The customized deduplicated gradient aggregation (Algorithm 1) is used only during training. At inference, cached head-specific values (Vhcache=V‾WhV_h^{\mathrm{cache}} = \overline{V}W_h, Appendix A.3) eliminate repeated value projections, leaving small sub-key multiplications (n=64n=64), Top-kk (k=4k=4) selection, and weighted table lookup in the memory branch. OpT-DeUS retains dense FFN projections in its inserted blocks, whereas MIDUS avoids these projections and uses substantially fewer model parameters. PagedAttention does not eliminate this architectural difference.

As promised in our initial response, we have finished running the 3-seed experiments for SOLAR and LESA, both CPT and SFT, on Llama-3.1-8B.

StageMethodPPL↓ARCLogiQAWinoCSQABoolQPIQAMMLUAverage↑
CPTBase9.83±0.0079.39±0.1826.16±1.0770.80±0.7251.19±2.3079.45±0.3179.09±0.2250.96±1.1162.44±0.49
SOLAR9.87±0.0179.50±0.2726.27±1.0971.40±0.2153.21±3.4079.16±0.4978.80±0.0951.15±0.4662.79±0.64
Llama Pro7.76±0.0081.52±0.1328.11±0.8272.95±0.2769.45±0.0782.26±0.1180.00±0.0261.20±0.1067.93±0.14
LESA8.06±0.1381.19±0.4028.78±0.9873.51±0.1068.14±2.5782.51±0.2679.98±0.1260.28±2.0167.77±0.65
OpT‑DeUS7.43±0.0282.11±0.1527.14±0.6974.48±0.4571.88±0.1081.72±0.3280.21±0.0563.08±0.0968.66±0.18
Avg‑DeUS7.83±0.0082.12±0.0226.68±0.0873.14±0.2171.94±0.1081.72±0.0780.00±0.0262.66±0.0468.32±0.04
MIDUS7.35±0.0082.55±0.0427.86±0.4074.37±0.2971.99±0.0782.88±0.0980.34±0.2263.36±0.2169.05±0.06
SFTBase9.62±0.0079.94±0.1126.06±0.1970.51±0.2058.23±0.2480.10±0.0679.27±0.0853.86±0.0764.00±0.04
SOLAR9.70±0.0079.98±0.0526.68±0.0870.61±0.1655.59±0.1980.00±0.0279.22±0.0552.73±0.0563.54±0.04
Llama Pro7.82±0.0083.00±0.0427.44±0.1574.64±0.3071.55±0.1482.97±0.0080.74±0.0861.82±0.1268.88±0.04
LESA7.97±0.0083.85±0.0426.37±0.2675.19±0.2172.48±0.1883.42±0.0480.61±0.0964.15±0.0169.44±0.09
OpT‑DeUS7.46±0.0083.92±0.1328.16±0.0775.51±0.4372.92±0.0483.26±0.1980.85±0.1664.64±0.1169.89±0.05
Avg‑DeUS7.90±0.0083.80±0.0426.58±0.3375.43±0.2672.32±0.1383.02±0.1180.54±0.1064.10±0.0969.40±0.06
MIDUS7.49±0.0384.34±0.1527.44±0.5175.45±0.4773.22±0.0083.90±0.0381.12±0.1864.17±0.0769.95±0.05

We also ran 3-seed evaluations on Qwen-2.5-3B to make the comparison more solid.

MethodPPLARCLogiQAWinoCSQABoolQPIQAMMLUAvg.
Base10.94±0.0979.88±0.2131.23±0.1969.53±0.5369.94±0.3375.49±0.1877.55±0.2762.52±0.1766.59±0.16
OpT‑DeUS12.24±0.0078.41±0.2430.82±0.4870.96±0.2374.96±0.1478.90±0.0977.95±0.1364.24±0.0068.03±0.01
MIDUS10.63±0.1079.29±0.2733.59±1.1370.85±0.3974.91±1.6078.64±0.3978.02±0.1664.66±0.1668.56±0.10

Even across multiple random seeds, MIDUS holds up well, matching strong baselines like OpT-DeUS and LESA in accuracy while offering much better efficiency. To briefly recap our earlier responses to your main points, we first noted that because MIDUS leaves the core attention mechanism untouched, existing inference optimizations in engines like vLLM carry over directly without custom kernel workarounds for attention structure. We also plotted the slot utilization distribution for the memory module and confirmed that sharing V across heads effectively prevents dead slots in the value space. Finally, we addressed your questions about the assumptions and implications of Theorem 1 by extending the theoretical bound and backing it up with direct empirical comparisons in practical training setups.

Thanks again for the time and thought you put into reviewing our paper. Please let us know if anything remains unclear or needs further discussion. If our new results and clarifications have resolved your concerns, we would really appreciate it if you could consider updating your score.

Reviewer_iPdQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a novel approach for depth up-scaling which organizes memory around attention heads, instead of adding compute-heavy transformer blocks. They remain comparable with existing approaches while having several efficiency benefits, in terms of training time, generation throughput, peak training memory as well as the number of trainable parameters. While the accuracy gains are limited, the efficiency and performance benefits make up for them. They also present theoretical as well as empirical analysis to support their claims.

Critical strengths
  1. The paper demonstrates significant improvements in terms of efficiency (training memory, training time, generation throughput) while maintaining competitive performance with existing methods.
  2. Comprehensive experiments are done to verify the design choices as well as experiments on multiple seeds. They are also supplemented by various helpful ablations including different datasets, different memory choices, DUS policies, etc. The component-wise analysis is also quite helpful to isolate the performance gains.
  3. Using sparse memory for architectural interventions is a new and interesting direction. The proposed architecture is also reasonably well-motivated and the accompanying theoretical analysis complements their claims.
Critical weaknesses
  1. The evidence for retrieval and long-context improvements is less convincing than the aggregate results since most of the improvement is concentrated in Code. This is peculiar since the architecture promotes a retrieval-based memory.
  2. The paper stresses the fact that the proposed memory mechanism actually exploits head specialization via increase in variance of head importance, but that does not necessarily indicate improved specialization. Claims about efficiency are also overstated, for e.g., the 8.3x fewer trainable parameters only happen in the CPT case, not the SFT case where all the model parameters are being trained.
Questions for authors
  1. While the head-importance analysis in Section 4.4 is interesting, an increase in variance of head importance does not necessarily indicate improved specialization. Can the authors provide more direct evidence of head specialization (for e.g., through head ablation)?
  2. Would the same mechanism extend to newer attention variants like grouped query attention?
  3. Appendix A.3 introduces Head-wise value caching, what is the size of this cache? Is this also included in the efficiency measurements?
3 comments

We sincerely thank you for recognizing our head-wise sparse memory design, efficiency gains, and comprehensive ablations. Your constructive feedback on head specialization, CPT versus SFT efficiency claims, and long-context evaluation helped us meaningfully strengthen the paper with new experiments and clearer exposition below.

W1: Retrieval and Long-Context Performance

Even on Llama-3.2-1B (Table 2), MIDUS achieves higher scores than OpT-DeUS not only on Code (54.64 vs. 47.57), but also across TriviaQA (39.75 vs. 39.02), NQ-Open (12.24 vs. 12.11), Multi-Doc QA (8.48 vs. 7.97), Summarization (23.23 vs. 22.94), and Few-shot learning (63.52 vs. 60.60), while using substantially fewer trainable parameters during CPT.

This trend becomes even clearer at the Llama-3.1-8B scale (please refer to the full tables in our response to Reviewer vz6A and Table 2 of the revised manuscript). On 8B, MIDUS achieves the highest retrieval-heavy average and leads in 5 out of 6 long-context categories. In particular, large gains appear well beyond Code, including Single-Doc QA (20.33 vs. 12.34 to 13.84 across DUS baselines) and Few-shot learning (70.46 vs. 64.59 to 67.82). Even excluding Code, MIDUS achieves the highest average across the remaining five long-context categories (27.67 vs. 24.07–25.22 for the DUS baselines), showing that its advantage extends beyond Code.

W2&Q1: Additional Evidence of Head Specialization

First, regarding the framing of Section 4.4 (Figure 4), our premise (Sections 1 and 3.1) is that pre-trained Base attention heads are already specialized. Because each specialized head draws on different information depending on the task, equipping each head with a distinct product-key space and a head-specific projection of the shared value bank in MIDUS supplies head-specific knowledge and reinforces reliance on task-relevant heads, yielding the higher IShIS_h mean and variance in Figure 4.

At the same time, we agree that single-task IShIS_h variance alone cannot distinguish task-specific specialization from concentration onto universal heads. To test this directly, we newly evaluate cross-task head selection across PIQA, ARC-Easy, and CSQA on Llama-3.2-1B. For any two tasks A≠BA \neq B, let TAT_A and TBT_B denote the sets of top-4 highest IShIS_h heads per layer. We measure their mean overlap ∣TA∩TB∣/∣TA∪TB∣|T_A \cap T_B| / |T_A \cup T_B|, where 00 indicates completely disjoint heads and 11 indicates identical heads across tasks.

Model & Evaluated LayersCross‑Task Top‑4 Overlap (↓\downarrow)
Pre-trained Base Block0.34
OpT-DeUS Inserted Block0.56
MIDUS Inserted Block0.29

Whereas OpT-DeUS shows greater cross-task reuse of the same high-importance heads, MIDUS exhibits lower overlap (0.29 vs. 0.56), providing additional evidence of task-dependent head specialization.

W2: Efficiency Claims

Regarding the efficiency claims, we have revised the Abstract, Section 4, and Section 4.2 (Table 3) to explicitly state that the 8.3×8.3\times reduction in trainable parameters (0.42B vs. 3.49B) and 42% lower peak training memory apply to the CPT stage. During full-parameter SFT where all weights are updated, MIDUS still updates and stores 26.6% fewer total parameters than OpT-DeUS (8.45B vs. 11.52B) while avoiding dense FFN computation in all inserted blocks.

Q2: Extension to GQA and Newer Attention Variants

Yes. In fact, all backbones evaluated in our paper (Llama-3.2-1B, Llama-3.1-8B, and Qwen-2.5-3B) already use GQA. In MIDUS, Attn′\mathrm{Attn}' directly inherits the anchor block's native attention structure and Q,K,VQ, K, V projections. While KK and VV heads remain grouped inside Attn′\mathrm{Attn}', GQA still produces HH distinct query-head outputs [a1(x)∣⋯∣aH(x)][a_1(x) \mid \cdots \mid a_H(x)] (H=32H=32 for Llama-3, H=16H=16 for Qwen-2.5) before the output projection, which serve directly as the memory queries qh=ah(x)q_h = a_h(x).

More broadly, MIDUS is agnostic to the internal attention formulation. Because it does not alter how attention scores or context aggregations are computed, requiring only per-head outputs ah(x)a_h(x) prior to output mixing. MIDUS extends directly to other modern attention variants.

Q3: Size and Accounting of the Head-Wise Value Cache

Each Memory block caches Vhcache=V‾Wh∈RN×dhV_h^{\mathrm{cache}} = \overline{V}W_h \in \mathbb{R}^{N \times d_h} per head (Eq. 15), requiring HNdh=NdH N d_h = N d elements per block.

BackbonePer Memory BlockMemory BlocksTotal Cache
Llama-3.2-1B16 MiB8128 MiB
Llama-3.1-8B32 MiB16512 MiB
Qwen-2.5-3B16 MiB18288 MiB

As noted in Appendix A.3, this cache is included in all reported inference GPU memory, prefill, and generation-throughput measurements. The resulting memory overhead is small relative to the overall model footprint. Even with the cache materialized, MIDUS uses 20.1 GB of inference memory on Llama-3.1-8B compared to 27.0 GB for OpT-DeUS (Table 3).

Dear Authors,

Thank you for your thoughtful response and detailed clarifications. Your rebuttal has addressed my concerns, and I appreciate the additional insights provided.

I have no further concerns at this stage. After carefully weighing their responses and the responses to other reviewers, I have decided to maintain my original rating.

Best, Reviewer iPdQ

Thank you very much for taking the time to read our responses and for confirming that your concerns have been addressed. We truly appreciate your constructive feedback and positive support, which helped us meaningfully improve the clarity and empirical depth of our manuscript.

Reviewer_vz6AOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether depth up-scaling has to add FFN-heavy Transformer blocks. It proposes MIDUS, which inserts memory blocks instead. Each attention head queries its own product-key space, and the retrieved values come from a shared bank with head-specific projections. The memory output is added residually, and the backbone stays frozen during continual pre-training. Experiments compare MIDUS with SOLAR, Llama Pro, LESA, OpT-DeUS and Avg-DeUS on Llama-3.2-1B and Llama-3.1-8B, and report large savings in trainable parameters and memory.

Critical strengths
  1. Large efficiency gain: On Llama-3.1-8B, MIDUS uses 0.42B trainable parameters and 34.8 GB peak training memory, against 3.49B and 59.8 GB for OpT-DeUS (Table 3). Throughput is 50.5 vs 41.3 tokens/s.

  2. Better accuracy: MIDUS has the best average in CPT and SFT on both the 1B and 8B backbones (Table 1), and the best LongBench average on the 1B model (Table 2).

Critical weaknesses
  1. Capacity mismatch: MIDUS adds 0.42B trainable parameters, while the DUS baselines add 3.49B. A DUS baseline with fewer inserted blocks, matched to the same trainable budget, is missing, so the accuracy comparison mixes the form and the amount of capacity.

  2. Copied 8B baselines and single runs: The 8B baseline numbers are copied from Cao et al. (2025), not reproduced, and the 8B/3B results are single runs. Long-context and retrieval results are only on 1B.

  3. Theory does not cover learned retrieval: Theorem 3.1 assumes identical retrieval weights across heads, so it does not explain the gains under the learned, non-identical retrieval used in practice. The shared factorization assumption is also not checked empirically.

Questions for authors
  1. Can you compare with a DUS baseline that has the same trainable parameter budget, for example by inserting fewer blocks?

  2. Were the 8B baselines retrained under the same data and compute, and can you add long-context results for 8B?

  3. How often does the shared factorization assumption in Theorem 3.1 hold for the learned memory?

4 comments

We sincerely thank you for highlighting the clarity of our presentation alongside the efficiency and accuracy gains of MIDUS. Your constructive suggestions on parameter-matched DUS baselines, reproduced multi-seed and 8B long-context evaluations, and learned retrieval analysis directly guided our new experiments and theoretical extensions below.

W1&Q1: Performance on Llama-3.1-8B Under Comparable Parameter Budgets

While our default Llama-3.1-8B setup matches the number of added blocks (16 blocks), we evaluate DUS baselines under a comparable trainable parameter budget by reducing the added DUS blocks to 1/81/8 (2 blocks), yielding 0.44B trainable parameters compared to 0.42B for MIDUS (Appendix A.6, Table 7,8 in the revised manuscript)

Llama-3.1-8B under comparable budget: CPT Results

MethodPPL ↓\downarrowARCLogiQAWinoCSQABoolQPIQAMMLUAvg ↑\uparrow
Llama Pro7.6982.5327.0473.6468.9682.5780.0362.6668.20
OpT‑DeUS7.3882.5327.5074.1171.9980.9580.3663.3368.68
Avg‑DeUS7.4382.4527.8073.8071.8381.6280.2562.9268.67
MIDUS7.3582.4928.4274.7471.9182.9780.3063.0669.13

Llama-3.1-8B under comparable budget: Retrieval-heavy Evaluation

MethodTriviaQANQ-OpenRACEAvg$\uparrow$
Llama Pro40.6625.0445.3637.02
OpT‑DeUS70.0123.5744.8846.15
Avg‑DeUS64.4526.0445.0745.19
MIDUS69.5625.8244.7846.72

Llama-3.1-8B under comparable budget: Long-context Evaluation

MethodSingle‑Doc QAMulti‑Doc QASummarizationFew‑shotSyntheticCodeAvg ↑\uparrow
Llama Pro13.6610.6125.7069.2713.8941.5929.12
OpT‑DeUS12.4410.5325.1567.5710.5959.0230.88
Avg‑DeUS12.9710.8225.0570.268.9362.2531.71
MIDUS20.3310.9426.0070.4610.6166.3134.11

Under comparable trainable parameter budgets, MIDUS maintains general-purpose and retrieval-heavy performance on par with the DUS baselines while continuing to outperform them on long-context evaluations. This indicates that our findings reflect the effectiveness of the MIDUS design rather than a capacity mismatch.

W2&Q2: Reproduced Multi-Seed 8B Results and 8B Retrieval/Long-Context Evaluation

In our initial submission, the Llama-3.1-8B baseline results were reported directly from Cao et al. (2025). We have retrained the 8B models in our own setup using the same training data and training schedule across three random seeds (Appendix A.5 in the revised manuscript). We've updated the 8B results in Table 1 of the main paper with these 3 seed means. Due to computational constraints, we first report 3 seed results for Base, Llama Pro, Avg-DeUS, OpT-DeUS, and MIDUS. SOLAR and LESA runs for Llama-3.1-8B and runs for Qwen-2.5-3B backbone are currently in progress and will be shared as soon as they complete.

Llama-3.1-8B: CPT (3 Seeds)

MethodPPL ↓\downarrowARCLogiQAWinoCSQABoolQPIQAMMLUAverage ↑\uparrow
Base9.83±0.009.83{\pm}0.0079.39±0.1879.39{\pm}0.1826.16±1.0726.16{\pm}1.0770.80±0.7270.80{\pm}0.7251.19±2.3051.19{\pm}2.3079.45±0.3179.45{\pm}0.3179.09±0.2279.09{\pm}0.2250.96±1.1150.96{\pm}1.1162.44±0.4962.44{\pm}0.49
Llama Pro7.76±0.007.76{\pm}0.0081.52±0.1381.52{\pm}0.1328.11±0.82\mathbf{28.11{\pm}0.82}72.95±0.2772.95{\pm}0.2769.45±0.0769.45{\pm}0.0782.26±0.1182.26{\pm}0.1180.00±0.0280.00{\pm}0.0261.20±0.1061.20{\pm}0.1067.93±0.1467.93{\pm}0.14
OpT‑DeUS7.43±0.027.43{\pm}0.0282.11±0.1582.11{\pm}0.1527.14±0.6927.14{\pm}0.6974.48±0.45\mathbf{74.48{\pm}0.45}71.88±0.1071.88{\pm}0.1081.72±0.3281.72{\pm}0.3280.21±0.0580.21{\pm}0.0563.08±0.0963.08{\pm}0.0968.66±0.1868.66{\pm}0.18
Avg‑DeUS7.83±0.007.83{\pm}0.0082.12±0.0282.12{\pm}0.0226.68±0.0826.68{\pm}0.0873.14±0.2173.14{\pm}0.2171.94±0.1071.94{\pm}0.1081.72±0.0781.72{\pm}0.0780.00±0.0280.00{\pm}0.0262.66±0.0462.66{\pm}0.0468.32±0.0468.32{\pm}0.04
MIDUS7.35±0.00\mathbf{7.35{\pm}0.00}82.55±0.04\mathbf{82.55{\pm}0.04}27.86±0.4027.86{\pm}0.4074.37±0.2974.37{\pm}0.2971.99±0.07\mathbf{71.99{\pm}0.07}82.88±0.09\mathbf{82.88{\pm}0.09}80.34±0.22\mathbf{80.34{\pm}0.22}63.36±0.21\mathbf{63.36{\pm}0.21}69.05±0.06\mathbf{69.05{\pm}0.06}

Llama-3.1-8B: SFT (3 Seeds)

MethodPPL ↓\downarrowARCLogiQAWinoCSQABoolQPIQAMMLUAverage ↑\uparrow
Base9.62±0.009.62{\pm}0.0079.94±0.1179.94{\pm}0.1126.06±0.1926.06{\pm}0.1970.51±0.2070.51{\pm}0.2058.23±0.2458.23{\pm}0.2480.10±0.0680.10{\pm}0.0679.27±0.0879.27{\pm}0.0853.86±0.0753.86{\pm}0.0764.00±0.0464.00{\pm}0.04
Llama Pro7.82±0.007.82{\pm}0.0083.00±0.0483.00{\pm}0.0427.44±0.1527.44{\pm}0.1574.64±0.3074.64{\pm}0.3071.55±0.1471.55{\pm}0.1482.97±0.0082.97{\pm}0.0080.74±0.0880.74{\pm}0.0861.82±0.1261.82{\pm}0.1268.88±0.0468.88{\pm}0.04
OpT‑DeUS7.46±0.00\mathbf{7.46{\pm}0.00}83.92±0.1383.92{\pm}0.1328.16±0.07\mathbf{28.16{\pm}0.07}75.51±0.43\mathbf{75.51{\pm}0.43}72.92±0.0472.92{\pm}0.0483.26±0.1983.26{\pm}0.1980.85±0.1680.85{\pm}0.1664.64±0.11\mathbf{64.64{\pm}0.11}69.89±0.0569.89{\pm}0.05
Avg‑DeUS7.90±0.007.90{\pm}0.0083.80±0.0483.80{\pm}0.0426.58±0.3326.58{\pm}0.3375.43±0.2675.43{\pm}0.2672.32±0.1372.32{\pm}0.1383.02±0.1183.02{\pm}0.1180.54±0.1080.54{\pm}0.1064.10±0.0964.10{\pm}0.0969.40±0.0669.40{\pm}0.06
MIDUS7.49±0.037.49{\pm}0.0384.34±0.15\mathbf{84.34{\pm}0.15}27.44±0.5127.44{\pm}0.5175.45±0.4775.45{\pm}0.4773.22±0.00\mathbf{73.22{\pm}0.00}83.90±0.03\mathbf{83.90{\pm}0.03}81.12±0.18\mathbf{81.12{\pm}0.18}64.17±0.0764.17{\pm}0.0769.95±0.05\mathbf{69.95{\pm}0.05}

Across three seeds under the same training conditions, MIDUS maintains comparable general-purpose performance to DUS baselines in CPT and SFT.

We also report 8B retrieval-heavy and long-context results below (revised Table 2):

Llama-3.1-8B: Retrieval-heavy Evaluation

MethodTriviaQANQ‑OpenRACERetrieval Avg. ↑\uparrow
Base44.6814.6341.9133.74
Llama Pro67.2020.3345.3644.30
OpT‑DeUS69.9525.4044.7846.71
Avg‑DeUS67.3320.3044.0243.89
MIDUS69.5625.8244.7846.72

Llama-3.1-8B: Long-context Evaluation

MethodSingle‑Doc QAMulti‑Doc QASummarizationFew‑shotSyntheticCodeLongBench Avg. ↑\uparrow
Base11.909.3725.5751.863.5827.2321.58
Llama Pro13.8410.7325.1564.596.0349.0428.23
OpT‑DeUS12.3410.2424.6067.8210.8055.8130.27
Avg‑DeUS13.1310.6925.4466.7110.1458.0330.69
MIDUS20.3310.9426.0070.4610.6166.3134.11

As on 1B, MIDUS remains competitive on retrieval-heavy benchmarks and performs strongly on long-context tasks at 8B.

W3&Q3: Non-identical retrieval and shared factorization

The revised Theorem 3.1 (Section 3.3; Appendix A.12) covers arbitrary fixed head-specific retrieval matrices {Ah}h=1H\{A_h\}_{h=1}^{H}. Identical retrieval gives the simpler variance expression.

For the same fixed retrieval matrices and targets, the revised theorem establishes

LInd⋆≤LMIDUS⋆≤min⁡(LShare⋆,LInd⋆+ρ2∑i>dhσi2(V⋆)),\mathcal{L}_{\mathrm{Ind}}^\star \le \mathcal{L}_{\mathrm{MIDUS}}^\star \le \min\left(\mathcal{L}_{\mathrm{Share}}^\star, \mathcal{L}_{\mathrm{Ind}}^\star + \frac{\rho}{2}\sum_{i > d_h}\sigma_i^2(V^\star)\right),

where V⋆=[V1⋆∣⋯∣VH⋆]V^\star=[V_1^\star \mid \cdots \mid V_H^\star] is an optimal independent solution and ρ=max⁡h∥Ah∥op2\rho=\max_h \|A_h\|_{\mathrm{op}}^2.

Thus, the result remains valid without exact shared factorization. The spectral tail quantifies an upper bound on the resulting approximation penalty. The shared-value gap is strictly positive when no common bank can simultaneously realize all projected head targets.

We have not measured how often independent optima satisfy rank⁡(V⋆)≤dh\operatorname{rank}(V^\star)\le d_h. This condition concerns an independent optimum for the same fixed-retrieval problem; MIDUS's learned banks are factorized by construction. The revised theorem treats exact factorization as a sufficient special case and bounds the approximation penalty.

To examine the practical effect of this restriction, we added MIDUS with Independent Values to the controlled end-to-end comparison in Appendix A.9.1, Table 12.

Value ParameterizationValue Params / BlockWikiText PPL ↓\downarrowAverage Accuracy ↑\uparrow
Shared Values (AhVshA_h V_{\mathrm{sh}})0.26M11.8652.52
Independent Values (AhVh)(A_h V_h)8.39M11.6853.29
MIDUS (AhV‾WhA_h \overline{V} W_h)0.39M11.6453.50

These results are consistent with the benefit of head-specific value realization. MIDUS achieves comparable reported performance to independent values with 21.3 times fewer learned value parameters, including the projections, and a 32 times smaller learned value table.

This comparison shows no observed performance degradation from the factorized parameterization in the evaluated setting. It does not directly estimate the spectral-tail term or establish the frequency of exact factorization. As discussed in Appendix A.12.4, independently trained product-key heads also admit different slot labelings, so an unaligned post-hoc concatenation of their banks is not an invariant diagnostic of the factorization condition.

I appreciate the authors' response, which has satisfactorily addressed my concerns. Additionally, having read the comments from the other two reviewers, my further concerns have also been addressed. Therefore, I decide to increase my score to 3 to reflect this.

Thank you very much for taking the time to review our responses across the discussion thread and for raising your score in light of our revisions. Your constructive feedback directly helped us make the manuscript much stronger and more rigorous.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.