- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper revisits RAM-Net, a prior architecture, and interprets it as a bridge between KV-cache quantization and linear attention. It shows that, under a restricted construction, soft assignments over discrete addresses can approximate softmax query-key similarity while enabling recurrent aggregation into fixed-size slot states. Building on this connection, the authors propose a staged Transformer-to-RAM-Net migration pipeline using a soft-quantized intermediate model followed by head-, block-, and full-model recovery. Experiments on nine pretrained Transformers ranging from 0.3B to 7B parameters show that the migrated models retain a substantial fraction of the teachers' downstream performance under a 500M-token recovery budget.
Critical strengths
-
Interesting theoretical connection between quantization and linear attention.
The paper provides a novel interpretation of RAM-Net by showing that soft assignments over discrete addresses can connect hard quantized matching with separable linear-attention-style aggregation. This offers a useful perspective for understanding how discrete routing and recurrent state compression can be unified. -
Practical migration from pretrained Transformers to RAM-Net.
Rather than requiring training from scratch, the paper develops a staged migration pipeline that first aligns a soft-quantized intermediate attention mechanism and then progressively recovers head-, block-, and model-level behavior. This makes the proposed connection practically relevant to existing pretrained models. -
Evaluation across diverse model families and useful ablations.
The method is tested on nine pretrained models spanning 0.3B--7B parameters, multiple model families, and both MHA and GQA architectures. The paper also includes ablations on the codebook, assignment concentration, and recovery pipeline, which help validate several key design choices.
Critical weaknesses
-
There is a substantial gap between the theoretical analysis and the practical RAM-Net evaluated in the paper.
The theoretical connection to softmax attention is established only under a restricted construction with a fixed circular codebook, two-dimensional factors, constrained projections, and full soft address distributions. In the actual migrated model, these projection constraints are removed, while Top-K routing, GSU forgetting, and the standard RAM-Net readout are used. These components can materially change the behavior of the model, making it unclear to what extent the theoretical approximation explains the performance of the practical system. This gap is particularly relevant because the original RAM-Net work motivates its architecture by reducing interference and improving fine-grained long-range retrieval [1], while the present theoretical analysis does not establish that these properties follow for the practical configuration used in the experiments. -
The paper does not establish whether the migration preserves the teacher's long-context capability.
RAM-Net compresses an increasing number of historical KV pairs into a fixed set of recurrent slots, so interference, slot contention, or forgetting may become increasingly significant as the context grows. However, neither the theoretical analysis nor the experiments characterize performance as a function of context length. Notably, some teachers evaluated in the paper already support substantially longer contexts (e.g., 32K for Qwen2.5), while the corresponding migrated RAM-Net models are evaluated with much shorter contexts. The reported LongBench results also do not provide controlled length scaling or retrieval-position analysis. Controlled evaluations such as NIAH/passkey retrieval or RULER [2], together with curves over 2K/4K/8K/16K/32K and beyond, would better reveal whether the fixed-slot representation preserves long-range information. This issue is increasingly important because recent small full-attention models such as Qwen3-4B-Instruct-2507 natively support 256K context [3], while Qwen3.5-4B adopts a hybrid architecture with three Gated DeltaNet linear-attention blocks for every full-attention block and also natively supports 256K context [4]. Comparing post-hoc RAM-Net migration against such natively trained long-context full/hybrid-attention models would help clarify whether the migration preserves competitive long-context capability. -
The practical memory, inference, and migration costs are insufficiently characterized.
The main motivation for RAM-Net is to replace a context-growing KV cache with fixed-size recurrent state, but the paper does not report the actual memory footprint of the RAM-Net slots or identify the context length at which this representation becomes more memory-efficient than standard full attention. In the relatively short-context regime used by most experiments, the fixed slot state could potentially incur comparable or even larger memory overhead, in which case the practical benefit should primarily emerge only at longer contexts through reduced KV-cache storage and memory traffic. It would therefore be useful to report memory usage versus context length and the corresponding break-even point. End-to-end serving measurements are also missing, including prefill latency, decoding TPOT, token throughput under different batch sizes, and an attention/RAM-Net runtime breakdown. Finally, the migration requires a 500M-token recovery budget per model, but there is limited analysis of how migration quality scales with the token/compute budget, whether recovery requires sufficiently diverse or task-matched data, and how well the resulting model generalizes to domains or tasks not covered by the recovery corpus. These measurements are important for assessing the practical cost-benefit trade-off of post-hoc migration compared with directly using or training an efficient-attention architecture.
[1] Kaicheng Xiao, Haotian Li, Liran Dong, and Guoliang Xing. RAM-Net: Expressive Linear Attention with Selectively Addressable Memory. arXiv:2602.11958, 2026.
[2] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, et al. RULER: What's the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654, 2024.
[3] Qwen Team. Qwen3-4B-Instruct-2507. Hugging Face model card, 2025. https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507
[4] Qwen Team. Qwen3.5: Towards Native Multimodal Agents. 2026. https://qwen.ai/blog?id=qwen3.5
Questions for authors
Could the authors characterize how RAM-Net's memory capacity scales with context length? As more historical tokens are aggregated into a fixed number of slots, it is unclear when slot contention or interference becomes significant and how many slots are required to maintain a given approximation or retrieval quality.
In particular, what is the relationship between the number of slots and supported context length ? Does maintaining a fixed error require , or can grow sublinearly with ? Since attention distributions are often highly concentrated or heavy-tailed, it would be interesting to know whether RAM-Net can exploit this sparsity to support much longer contexts with substantially fewer slots.
Empirically, the authors could sweep context length and slot capacity while reporting slot occupancy/collision statistics, approximation error, and controlled long-range retrieval results such as NIAH or passkey retrieval. This would clarify whether RAM-Net exhibits a predictable capacity-scaling law and where slot saturation begins to degrade long-context capability.
9 comments
Response to W1 (Part 1/2)
W1: Clarify the gap between the theoretical analysis and RAM-Net in practical weight migration.
Re: Thank you for this constructive comment. The restricted construction guides how we initialize RAM-Net from a Transformer. During recovery, RAM-Net can learn unrestricted read and write projections and use GSU to control how much historical information it retains. We examine how these extensions, sparse routing, and the choice of readout affect the initial approximation and subsequent recovery.
1. The rank-2 circular construction gives RAM-Net a closer initial match to the teacher’s attention. Our pipeline follows the same general strategy as HedgeMamba [1] and MOHAWK [2]. We first approximate the teacher’s attention and then refine the converted model through end-to-end training. Theorem 1 guides the first step. Under its conditions, the scaled overlap of the full circular read and write distributions locally approximates the teacher’s exponential dot-product similarity in the aligned Q/K space. Table 2 shows the practical benefit at initialization. The circular codebook yields lower Jensen–Shannon divergence from the teacher’s attention distribution than the alternative initializations.
We test how long the rank-2 constraint should remain in the migration pipeline. Within each model, all runs use the same circular initialization and 500M-token budget. “None” means we release the constraint immediately after initialization. The “Head” and “Head, block, and full” runs start from the same constrained head checkpoint and differ only in whether we retain the constraint during the remaining stages.
| Recovery stages with rank-2 constraint | Wiki PPL @ 2K ↓ | ARC-E | ARC-C | PIQA | SciQ | COPA | Hella. | Avg. ↑ |
|---|---|---|---|---|---|---|---|---|
| FLA-340M teacher | 30.05 | 57.41 | 24.06 | 65.56 | 82.50 | 73.00 | 32.05 | 55.76 |
| None | 35.14 | 54.71 | 23.72 | 64.69 | 80.20 | 65.00 | 31.02 | 53.22 |
| Head | 35.19 | 55.30 | 23.38 | 63.93 | 81.30 | 68.00 | 30.94 | 53.81 |
| Head, block, and full | 36.05 | 55.81 | 23.29 | 64.64 | 80.20 | 67.00 | 31.06 | 53.67 |
| Recovery stages with rank-2 constraint | Wiki PPL @ 4K ↓ | ARC-E | ARC-C | PIQA | SciQ | COPA | Hella. | Avg. ↑ |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-1.5B teacher | 12.60 | 75.21 | 41.13 | 75.63 | 94.20 | 83.00 | 50.19 | 69.89 |
| None | 16.33 | 75.34 | 41.55 | 74.32 | 89.80 | 82.00 | 45.61 | 68.10 |
| Head | 16.01 | 74.96 | 42.24 | 74.48 | 89.30 | 82.00 | 45.76 | 68.12 |
| Head, block, and full | 16.76 | 74.92 | 41.04 | 74.05 | 88.80 | 80.00 | 45.25 | 67.34 |
Releasing the constraint for the remaining stages lowers Wiki PPL on both FLA (36.05 to 35.19) and Qwen (16.76 to 16.01) and slightly improves six-task average accuracy on both. The reason is that the two-dimensional projections can still adapt to match teacher head outputs under a fixed codebook, while releasing the constraint lets each code’s read and write projections adjust independently.
2. Top-K reduces the cost of slot access, and its approximation improves as address weights concentrate. Theorem 1 concerns the overlap of the full soft read and write distributions. Top-K routing reduces computation by accessing only the selected slots at each step. Fig. 4 illustrates why the effect of truncation depends on the codeword norm . At small , the address softmax spreads weight over many slots, so truncation visibly changes the predicted value field. At larger , the softmax already concentrates on fewer slots. Across nine teachers, the top 32 of 1,024 slots retain 52.1% of the dense write-address mass at , rising to 87.7% at .
| Write mass retained (top 32 slots) | |
|---|---|
| 0.10 | 5.7% |
| 1.00 | 52.1% |
| 1.58 | 73.2% |
| 2.51 | 87.7% |
| 10.00 | 99.0% |
Fig. 8 in the paper shows the practical effect on FLA-2.7B. Top-K truncation causes the largest increase in attention KL during initialization. Yet head-level recovery reduces attention-output NMSE from about 0.82 to 0.13 while attention KL remains high. Thus, RAM-Net can recover much of the teacher’s head output even when sparse routing changes the attention distribution.
Response to W1 (Part 2/2)
3. GSU extends cumulative averaging by learning how much historical information to retain. Sec. 3.3 connects RAM-Net to linear attention through cumulative averaging, which GSU recovers in the no-forgetting limit. During migration, GSU can adjust how much of each slot’s previous state survives a new write. The theory therefore identifies a particular setting of the update rule, while the practical model can adapt retention during recovery.
4. Mass-aware readout gives a closer initial attention match, while standard RAM-Net readout generally yields lower perplexity after migration. The two readouts reflect different inductive biases. Transformer attention assigns weight to individual historical tokens. When several keys match a query, their contributions add up. In the address-kernel construction of Sec. 3.3, mass-aware readout captures this effect by weighting each slot by both its query read score and its accumulated write mass. Standard RAM-Net instead uses the query read score to weight the stored value directly.
This distinction matters early in migration. Transformer queries score historical keys directly, while RAM-Net must learn to read from the slots to which relevant keys write. Mass weighting can make that correspondence easier to recover initially. In Table 8, it lowers attention JS divergence before head-level recovery for six of nine teachers. After migration, however, standard readout gives lower Wiki PPL for eight of nine teachers, with comparable six-task accuracy overall (Table 7). The 340M scratch-trained model shows a similar trade-off. Mass weighting raises Wiki PPL from 26.89 to 27.10 while increasing six-task accuracy from 55.6% to 56.8% (Table 9).
One possible explanation is that mass weighting couples readout to forgetting. When GSU reduces a slot’s retained mass, mass-aware readout also reduces that slot’s influence, even if the query still assigns it a high read score. In addition, Top-K selects slots by read score alone, so it can omit a slot that would rank highly after mass weighting. Standard readout uses the same query scores for selection and readout.
We address interference and fine-grained long-range retrieval in W2/Q through slot-contention analysis and controlled NIAH experiments.
[1] Moudgil et al. Attention to Mamba: A Recipe for Cross-Architecture Distillation. 2026. ArXiv:2604.14191.
[2] Bick et al. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models. 2024. ArXiv:2408.10189.
Rebuttal by Authors
Dear Reviewer FkUG,
Thank you for the constructive feedback. Below, we explain how projection constraints, sparse routing, gating, and readout choices affect initialization and recovery. We also show that larger slot capacity reduces write overlap and improves retrieval, with the 4,096-slot model matching the teacher at 32K on the RULER NIAH-1 task. We also report state memory, inference latency and throughput, and migration cost. We address each point below.