acceptodds
Under review as a conference paper at ICLR 2027

Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration

Abstract

KV-cache quantization and linear attention are two representative approaches to tackling the storage and computational costs of Transformers. KV-cache quantization compresses individual KV entries into discrete codes but retains all entries, whereas linear attention recurrently aggregates multiple historical KV contributions into a fixed-size continuous state but can introduce interference. This contrast raises the question of whether per-KV compression and multi-KV aggregation can be bridged within a single mechanism for efficient attention. We identify RAM-Net as such a bridge through soft assignments over a discrete address space. These assignments determine recurrent updates to the continuous slot state associated with each address. Under a restricted RAM-Net construction, we prove that soft address assignments extend hard quantized matching to a separable read-write overlap that locally approximates full-attention similarity and supports recurrent aggregation. These connections further enable Transformer-to-RAM-Net weight migration through a new path based on a soft-quantized intermediate construction. Across nine pretrained Transformer models from 0.3B to 7B parameters, RAM-Net recovers an average of 87.1% of the teachers' accuracy gains over random guessing across six commonsense and knowledge tasks using only a 500M-token budget per model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Attention, Linear Attention, KV-cache Quantization, Pretrained Weight Transfer, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FkUGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper revisits RAM-Net, a prior architecture, and interprets it as a bridge between KV-cache quantization and linear attention. It shows that, under a restricted construction, soft assignments over discrete addresses can approximate softmax query-key similarity while enabling recurrent aggregation into fixed-size slot states. Building on this connection, the authors propose a staged Transformer-to-RAM-Net migration pipeline using a soft-quantized intermediate model followed by head-, block-, and full-model recovery. Experiments on nine pretrained Transformers ranging from 0.3B to 7B parameters show that the migrated models retain a substantial fraction of the teachers' downstream performance under a 500M-token recovery budget.

Critical strengths
  1. Interesting theoretical connection between quantization and linear attention.
    The paper provides a novel interpretation of RAM-Net by showing that soft assignments over discrete addresses can connect hard quantized matching with separable linear-attention-style aggregation. This offers a useful perspective for understanding how discrete routing and recurrent state compression can be unified.

  2. Practical migration from pretrained Transformers to RAM-Net.
    Rather than requiring training from scratch, the paper develops a staged migration pipeline that first aligns a soft-quantized intermediate attention mechanism and then progressively recovers head-, block-, and model-level behavior. This makes the proposed connection practically relevant to existing pretrained models.

  3. Evaluation across diverse model families and useful ablations.
    The method is tested on nine pretrained models spanning 0.3B--7B parameters, multiple model families, and both MHA and GQA architectures. The paper also includes ablations on the codebook, assignment concentration, and recovery pipeline, which help validate several key design choices.

Critical weaknesses
  1. There is a substantial gap between the theoretical analysis and the practical RAM-Net evaluated in the paper.
    The theoretical connection to softmax attention is established only under a restricted construction with a fixed circular codebook, two-dimensional factors, constrained projections, and full soft address distributions. In the actual migrated model, these projection constraints are removed, while Top-K routing, GSU forgetting, and the standard RAM-Net readout are used. These components can materially change the behavior of the model, making it unclear to what extent the theoretical approximation explains the performance of the practical system. This gap is particularly relevant because the original RAM-Net work motivates its architecture by reducing interference and improving fine-grained long-range retrieval [1], while the present theoretical analysis does not establish that these properties follow for the practical configuration used in the experiments.

  2. The paper does not establish whether the migration preserves the teacher's long-context capability.
    RAM-Net compresses an increasing number of historical KV pairs into a fixed set of recurrent slots, so interference, slot contention, or forgetting may become increasingly significant as the context grows. However, neither the theoretical analysis nor the experiments characterize performance as a function of context length. Notably, some teachers evaluated in the paper already support substantially longer contexts (e.g., 32K for Qwen2.5), while the corresponding migrated RAM-Net models are evaluated with much shorter contexts. The reported LongBench results also do not provide controlled length scaling or retrieval-position analysis. Controlled evaluations such as NIAH/passkey retrieval or RULER [2], together with curves over 2K/4K/8K/16K/32K and beyond, would better reveal whether the fixed-slot representation preserves long-range information. This issue is increasingly important because recent small full-attention models such as Qwen3-4B-Instruct-2507 natively support 256K context [3], while Qwen3.5-4B adopts a hybrid architecture with three Gated DeltaNet linear-attention blocks for every full-attention block and also natively supports 256K context [4]. Comparing post-hoc RAM-Net migration against such natively trained long-context full/hybrid-attention models would help clarify whether the migration preserves competitive long-context capability.

  3. The practical memory, inference, and migration costs are insufficiently characterized.
    The main motivation for RAM-Net is to replace a context-growing KV cache with fixed-size recurrent state, but the paper does not report the actual memory footprint of the RAM-Net slots or identify the context length at which this representation becomes more memory-efficient than standard full attention. In the relatively short-context regime used by most experiments, the fixed slot state could potentially incur comparable or even larger memory overhead, in which case the practical benefit should primarily emerge only at longer contexts through reduced KV-cache storage and memory traffic. It would therefore be useful to report memory usage versus context length and the corresponding break-even point. End-to-end serving measurements are also missing, including prefill latency, decoding TPOT, token throughput under different batch sizes, and an attention/RAM-Net runtime breakdown. Finally, the migration requires a 500M-token recovery budget per model, but there is limited analysis of how migration quality scales with the token/compute budget, whether recovery requires sufficiently diverse or task-matched data, and how well the resulting model generalizes to domains or tasks not covered by the recovery corpus. These measurements are important for assessing the practical cost-benefit trade-off of post-hoc migration compared with directly using or training an efficient-attention architecture.

[1] Kaicheng Xiao, Haotian Li, Liran Dong, and Guoliang Xing. RAM-Net: Expressive Linear Attention with Selectively Addressable Memory. arXiv:2602.11958, 2026.

[2] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, et al. RULER: What's the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654, 2024.

[3] Qwen Team. Qwen3-4B-Instruct-2507. Hugging Face model card, 2025. https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507

[4] Qwen Team. Qwen3.5: Towards Native Multimodal Agents. 2026. https://qwen.ai/blog?id=qwen3.5

Questions for authors

Could the authors characterize how RAM-Net's memory capacity scales with context length? As more historical tokens are aggregated into a fixed number of slots, it is unclear when slot contention or interference becomes significant and how many slots are required to maintain a given approximation or retrieval quality.

In particular, what is the relationship between the number of slots SS and supported context length NN? Does maintaining a fixed error require S=O(N)S=O(N), or can SS grow sublinearly with NN? Since attention distributions are often highly concentrated or heavy-tailed, it would be interesting to know whether RAM-Net can exploit this sparsity to support much longer contexts with substantially fewer slots.

Empirically, the authors could sweep context length and slot capacity while reporting slot occupancy/collision statistics, approximation error, and controlled long-range retrieval results such as NIAH or passkey retrieval. This would clarify whether RAM-Net exhibits a predictable capacity-scaling law and where slot saturation begins to degrade long-context capability.

9 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Dear Reviewer FkUG,

Thank you for the constructive feedback. Below, we explain how projection constraints, sparse routing, gating, and readout choices affect initialization and recovery. We also show that larger slot capacity reduces write overlap and improves retrieval, with the 4,096-slot model matching the teacher at 32K on the RULER NIAH-1 task. We also report state memory, inference latency and throughput, and migration cost. We address each point below.

AuthorsOpenReview ↗

Response to W1 (Part 1/2)

W1: Clarify the gap between the theoretical analysis and RAM-Net in practical weight migration.

Re: Thank you for this constructive comment. The restricted construction guides how we initialize RAM-Net from a Transformer. During recovery, RAM-Net can learn unrestricted read and write projections and use GSU to control how much historical information it retains. We examine how these extensions, sparse routing, and the choice of readout affect the initial approximation and subsequent recovery.

1. The rank-2 circular construction gives RAM-Net a closer initial match to the teacher’s attention. Our pipeline follows the same general strategy as HedgeMamba [1] and MOHAWK [2]. We first approximate the teacher’s attention and then refine the converted model through end-to-end training. Theorem 1 guides the first step. Under its conditions, the scaled overlap of the full circular read and write distributions locally approximates the teacher’s exponential dot-product similarity in the aligned Q/K space. Table 2 shows the practical benefit at initialization. The circular codebook yields lower Jensen–Shannon divergence from the teacher’s attention distribution than the alternative initializations.

We test how long the rank-2 constraint should remain in the migration pipeline. Within each model, all runs use the same circular initialization and 500M-token budget. “None” means we release the constraint immediately after initialization. The “Head” and “Head, block, and full” runs start from the same constrained head checkpoint and differ only in whether we retain the constraint during the remaining stages.

Recovery stages with rank-2 constraintWiki PPL @ 2K ↓ARC-EARC-CPIQASciQCOPAHella.Avg. ↑
FLA-340M teacher30.0557.4124.0665.5682.5073.0032.0555.76
None35.1454.7123.7264.6980.2065.0031.0253.22
Head35.1955.3023.3863.9381.3068.0030.9453.81
Head, block, and full36.0555.8123.2964.6480.2067.0031.0653.67
Recovery stages with rank-2 constraintWiki PPL @ 4K ↓ARC-EARC-CPIQASciQCOPAHella.Avg. ↑
Qwen2.5-1.5B teacher12.6075.2141.1375.6394.2083.0050.1969.89
None16.3375.3441.5574.3289.8082.0045.6168.10
Head16.0174.9642.2474.4889.3082.0045.7668.12
Head, block, and full16.7674.9241.0474.0588.8080.0045.2567.34

Releasing the constraint for the remaining stages lowers Wiki PPL on both FLA (36.05 to 35.19) and Qwen (16.76 to 16.01) and slightly improves six-task average accuracy on both. The reason is that the two-dimensional projections can still adapt to match teacher head outputs under a fixed codebook, while releasing the constraint lets each code’s read and write projections adjust independently.

2. Top-K reduces the cost of slot access, and its approximation improves as address weights concentrate. Theorem 1 concerns the overlap of the full soft read and write distributions. Top-K routing reduces computation by accessing only the selected slots at each step. Fig. 4 illustrates why the effect of truncation depends on the codeword norm β\beta. At small β\beta, the address softmax spreads weight over many slots, so truncation visibly changes the predicted value field. At larger β\beta, the softmax already concentrates on fewer slots. Across nine teachers, the top 32 of 1,024 slots retain 52.1% of the dense write-address mass at β⋆\beta^\star, rising to 87.7% at 2.51β⋆2.51\beta^\star.

β/β⋆\beta/\beta^\starWrite mass retained (top 32 slots)
0.105.7%
1.0052.1%
1.5873.2%
2.5187.7%
10.0099.0%

Fig. 8 in the paper shows the practical effect on FLA-2.7B. Top-K truncation causes the largest increase in attention KL during initialization. Yet head-level recovery reduces attention-output NMSE from about 0.82 to 0.13 while attention KL remains high. Thus, RAM-Net can recover much of the teacher’s head output even when sparse routing changes the attention distribution.

AuthorsOpenReview ↗

Response to W1 (Part 2/2)

3. GSU extends cumulative averaging by learning how much historical information to retain. Sec. 3.3 connects RAM-Net to linear attention through cumulative averaging, which GSU recovers in the no-forgetting limit. During migration, GSU can adjust how much of each slot’s previous state survives a new write. The theory therefore identifies a particular setting of the update rule, while the practical model can adapt retention during recovery.

4. Mass-aware readout gives a closer initial attention match, while standard RAM-Net readout generally yields lower perplexity after migration. The two readouts reflect different inductive biases. Transformer attention assigns weight to individual historical tokens. When several keys match a query, their contributions add up. In the address-kernel construction of Sec. 3.3, mass-aware readout captures this effect by weighting each slot by both its query read score and its accumulated write mass. Standard RAM-Net instead uses the query read score to weight the stored value directly.

This distinction matters early in migration. Transformer queries score historical keys directly, while RAM-Net must learn to read from the slots to which relevant keys write. Mass weighting can make that correspondence easier to recover initially. In Table 8, it lowers attention JS divergence before head-level recovery for six of nine teachers. After migration, however, standard readout gives lower Wiki PPL for eight of nine teachers, with comparable six-task accuracy overall (Table 7). The 340M scratch-trained model shows a similar trade-off. Mass weighting raises Wiki PPL from 26.89 to 27.10 while increasing six-task accuracy from 55.6% to 56.8% (Table 9).

One possible explanation is that mass weighting couples readout to forgetting. When GSU reduces a slot’s retained mass, mass-aware readout also reduces that slot’s influence, even if the query still assigns it a high read score. In addition, Top-K selects slots by read score alone, so it can omit a slot that would rank highly after mass weighting. Standard readout uses the same query scores for selection and readout.

We address interference and fine-grained long-range retrieval in W2/Q through slot-contention analysis and controlled NIAH experiments.

[1] Moudgil et al. Attention to Mamba: A Recipe for Cross-Architecture Distillation. 2026. ArXiv:2604.14191.

[2] Bick et al. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models. 2024. ArXiv:2408.10189.

Reviewer_diKVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper describes a well-grounded distillation process for converting transformer architectures into fixed-overhead linear RAMNet models. Authors map quantized radial-bin soft-assignments of channel pairs in attention to the combinatorial register routing mechanism of RAMNet, and in doing so provide a clear equivalence mechanism for distilling teacher model behavior into a student. With a very small distillation budget, student RAMNets are able to recover most of the transformer teacher performance.

Critical strengths

Approach is clever and conceptually elegant. The mapping is explained in detail and theoretically justified. Paper is well organized and material is presented straightforwardly. I like the idea overall and consider this strong proof of concept.

Critical weaknesses

I have two major concerns, plus a far distant third. In order of importance:

  1. Quality of results is difficult to interpret. On the one hand, performance on commonsense reasoning benchmarks lags the teacher by ~2-4 points, whereas the original RAMNet claims to outperform the same architectural baseline by 1-2 points when pretrained from scratch. That’s a significant lag. On the other hand, getting that close with a token budget of only 500M tokens is pretty remarkable – Wang et al 2024 (The Mamba in the Llama: Distilling and Accelerating Hybrid Models) use a full 20B tokens for distillation, for comparison. I would consider these results very promising if I had reason to believe that performance would continue to increase with a larger token budget. But as is, no ablation on token budget is provided. How does performance improve with larger token budgets? Or if that’s not feasible, then I would also accept ablations against smaller token budgets – I just need some evidence of positive signal against the scaling up of distillation.

  2. No discussion of memory overhead. What’s your relative memory compared to the teacher models? Codebook is 4^5=1024 entries, but it’s not clear how you’re handling head dimensionality and GQA. Section 5.1 says “each KV head is repeated over its query-head group” – is it repeated, or reused? If the latter, and if each codebook entry contains a value buffer at full original dimensionality, then that’s a pretty glaring potential issue. Many modern architectures use aggressive GQA, and if you have 8x as many stateful heads as the teacher model, and the teacher has a sequence length of 4k or less, then you’re actually inflating memory overhead at that point, not reducing it (here I assume v dimensionality overwhelms the reduced k dimensionality). Perhaps my understanding of the derived RAMNet architecture is wrong, but for a paper motivated in the very first paragraph by memory and overhead savings compared to full attention, I’m not convinced that that much memory or overhead is actually being saved here.

  3. A much smaller issue: the material is (in my opinion) a bit over-technically presented. Mathematical constructions are given from first principles, with the conceptual and intuitive payoff largely provided afterwards, demanding more forbearance on the reader’s part. As a couple of examples (these are somewhat nitpicky, but the comment applies more generally), it might be better to discuss the behavioral impact of retention gate gamma when it’s first introduced in sec2.3 (representative values of inf, neginf, and zero mapping to overwriting, symmetric updates, and accumulation respectively, for example) rather than when it’s revisited in sec3.3. And similarly, working backward from the softmax attention construction provides a more intuitive sketch of the conceptual bridge: e^(sum_u(x_u)) = prod_u(e^x_u) for 2-D subvectors x_u, where x_u is the dot product of q_u and k_u, which you are then approximating as weighted sums over discrete assignments to d_p radial positions in 2D space. Then Beta, and the magnitude of q_u and k_u, determine the strictness of the radial assignment. All of this is sensible and intuitive behavior that falls out of your construction, but spelling it out in these terms in advance would (I think) help to aid in understanding by the time we get to the formal construction.

Questions for authors

Feel free to gloss or ignore these, they have no bearing on my evaluation of the paper. I’m just curious.

  • Is Fig 1 accurate? The region boundaries appear to indicate Euclidean distance nearest neighbor boundaries, but attention uses the dot product. To my understanding, the decision boundaries should be radial, as on the right-hand side. Is this for illustration purposes, or am I missing something here?

  • Could you encode frequencies corresponding to rational fractions of d_p, and convert that to semiregular advancements in CAPE rather than regular advancements? i.e. advancing the CAPE offset 3 times every 5 steps to encode a frequency equal to 3/5 of 2pi/d_p? Or even extend to real numbers via irregular updates, tracking int(t*f/d_p) with f any real-valued frequency and t the token position? And would this provide any benefit? Or is this just the already-ablated pattern-free CAPE selection?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Dear Reviewer diKV,

Thank you for the constructive feedback. Below, we show gains from longer recovery consistent with scaling laws and clarify state allocation and memory efficiency. We also explain the angular coordinates in Fig. 1 and how rounded fractional CAPE shifts can introduce absolute-position dependence into relative offsets. We address each point below.

AuthorsOpenReview ↗

Response to W1

W1: Clarify the performance gap between migration and training from scratch, and show how migration performance varies with the training token budget.

Re: Thank you for raising these points.

1. The performance gap arises from the different training dynamics of migration and training from scratch. When RAM-Net trains from scratch, its attention layers and feed-forward networks (FFNs) learn jointly from the outset. During migration, RAM-Net inherits FFNs that are already adapted to the teacher’s attention outputs. This initialization helps the student recover much of the teacher’s performance with relatively few tokens, but it also limits the optimization from the one reached through training from scratch. As a result, the migrated model with limited token budget may not fully match the performance RAM-Net achieves when trained from scratch.

2. The token budget ablation shows that the trend is consistent with scaling laws that describe a power-law relationship between language-model loss and the training token budget. Following the ablation study at Table 12 in the paper, we continue full-model recovery of Qwen2.5-1.5B on FineWeb-Edu. We evaluate at total migration budgets of 750M, 1B, and 2B tokens. For reference, we also evaluate the 155M-token checkpoint before full-model recovery.

Token budgetWiki PPL ↓ARC-EARC-CPIQASciQCOPAHella.Avg. ↑Avg. gap (pp) ↓
Teacher12.6075.2141.1375.6394.2083.0050.1969.890.00
155M19.4472.1037.6373.6187.2079.0044.1265.614.28
500M16.3375.3441.5574.3289.8082.0045.6168.101.79
750M15.9975.1341.4774.3289.5080.0045.6067.672.22
1B15.7675.4641.2174.7689.6081.0045.9367.991.90
2B15.5375.3841.6474.9289.7081.0046.0068.111.79
  • Language modeling continues to improve through 2B tokens, with diminishing gains. From 500M to 1B, an additional 500M tokens reduce WikiText perplexity by 0.57 (16.33 to 15.76). From 1B to 2B, twice as many additional tokens reduce it by only 0.23 (15.76 to 15.53).
  • Six-task accuracy shows little change beyond 500M. The average accuracy gap to the teacher falls from 4.28 percentage points at 155M to 1.79 percentage points at 500M. Accuracy then fluctuates and reaches 68.11% at 2B, essentially unchanged from 68.10% at 500M. The longer run therefore improves language modeling while maintaining downstream average accuracy.
  • We choose 500M tokens as a balance between model quality and training cost. The full 500M-token migration uses 32.5 GPU-hours across two H200 GPUs. Extending that checkpoint to 1B and 2B total tokens costs an additional 29.0 and 75.8 GPU-hours, respectively. At 500M, the model already reaches essentially the same six-task average as at 2B, with a 0.80 higher WikiText PPL.
  • The choice of recovery data may help explain the remaining six-task gap. Qwen2.5 pretraining draws on a broad mixture of datasets [1], while our migration uses only FineWeb-Edu. Extending recovery to 2B tokens gives the student more context from the same distribution. If that distribution underrepresents the knowledge or reasoning patterns these tasks require, additional tokens can improve WikiText perplexity while six-task accuracy remains unchanged. This interpretation is consistent with prior work showing that downstream performance depends on the data used for distillation [2] [3].

Overall, this trend is consistent with the language-model scaling laws and suggests that continued training with more tokens yields further gains.

[1] Qwen Team. Qwen2.5 Technical Report. 2024. ArXiv:2412.15115.

[2] Peris et al. Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks. 2022. DOI:10.18653/v1/2022.emnlp-industry.12.

[3] Nguyen et al. Knowledge Distillation with Distribution Mismatch. 2021. DOI:10.1007/978-3-030-86520-7_16.

AuthorsOpenReview ↗

Response to W2

W2: Compare memory overhead to the teacher models.

Re: Thank you for this constructive comment. We agree that RAM-Net can use more memory than the Transformer teacher at short context lengths. However, as context length LL increases, RAM-Net’s recurrent state occupies O(1)O(1) memory, while the teacher’s KV cache occupies O(L)O(L) memory. RAM-Net uses less state memory than the teacher’s BF16 KV cache beyond 6,192-token context length for Qwen2.5-1.5B and 1,032 tokens for Llama-2-7B. We clarify each point below.

1. RAM-Net copies each teacher KV head’s projection weights at initialization and maintains an independent recurrent state per query head at inference. RAM-Net currently uses a multi-head architecture with an independent recurrent state per query head. To test migration from a GQA teacher to this architecture, we copy each teacher KV head’s projection weights to every student head in the corresponding query-head group at initialization. Thus, for example, the Qwen2.5-1.5B teacher shares 2 KV heads across 12 query heads per layer, whereas the RAM-Net student maintains 12 independent states. We agree that, compared to Qwen at 4K, maintaining one state per query head uses more memory than the teacher’s shared KV cache, but our focus of this paper is the feasibility of weight migration though achieving a GQA-style RAM-Net is possible. We leave the design and evaluation of the GQA variant to future work.

2. Each student head stores 1,024 slots, each with a 128-dimensional FP32 value vector and one FP32 mass scalar. The low-rank construction maps queries and keys to 2U=102U=10 dimensions for address selection. It does not reduce the value dimension. Each complete address indexes a slot, and each slot stores a 128-dimensional FP32 value vector and one FP32 mass scalar. Each head therefore maintains 45=1,0244^5=1{,}024 slots. Top-32 selection limits the number of slots accessed per step, but all 1,024 slots remain in memory.

3. RAM-Net uses less state memory than the teacher’s BF16 KV cache beyond 6,192 tokens for Qwen2.5-1.5B and 1,032 tokens for Llama-2-7B. We report the additional KV-cache memory per 1K input tokens and the context length at which the teacher’s cache equals the corresponding RAM-Net state. All sizes are per sequence, summed across layers. RAM-Net’s state grows by 0 MiB per additional 1,024 tokens and occupies a fixed 169.31 MiB for Qwen and 516 MiB for Llama.

Teacher cacheState growth (MiB/1K tokens)Break-even length (tokens)
Qwen2.5-1.5B (GQA, BF16 KV)286,192
Qwen2.5-1.5B (GQA, HQQ INT4 KV)≈7.88≈22,016
Llama-2-7B (MHA, BF16 KV)5121,032

The break-even length is the context length at which the teacher’s KV cache and RAM-Net’s recurrent state use the same amount of memory. Because the KV cache grows with context length while RAM-Net’s state remains fixed, RAM-Net uses less state memory at longer context lengths.

Reviewer_ektzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the connection between KV-cache quantization and linear attention, and uses this connection to convert pretrained Transformers into the existing RAM-Net architecture. The method first aligns the teacher's query and key projections and builds an intermediate attention model. It then trains the RAM-Net model in several stages to recover the teacher's behavior.

The experiments cover nine pretrained models from five model families, ranging from approximately 0.3B to 7B parameters, with 500M additional training tokens per model. On the six main evaluation tasks, the converted models lose 1.8–4.5 percentage points in average accuracy compared with their teachers. The paper also studies different initialization choices and training stages.

Critical strengths
  • The paper connects its theoretical analysis to a practical conversion method. The proposed connection between KV-cache quantization and linear attention provides a way to initialize RAM-Net from pretrained Transformer weights. This is a useful direction for reusing existing pretrained models.

  • The evaluation covers several model families and sizes. Testing nine models provides evidence that the conversion procedure can work across different pretrained checkpoints. The converted models retain much of the teachers' accuracy on the six main tasks under the same 500M-token budget.

  • The ablation studies provide some support for the design choices. The circular codebook gives better initial attention matching than the alternatives in Table 2. The staged training procedure also improves final results: on Qwen2.5-1.5B, the complete pipeline achieves an average accuracy of 68.1, compared with 66.6 for full-model-only training under the same token budget (Table 4).

Critical weaknesses
  • There is no experimental comparison with existing conversion methods. The paper discusses methods such as LoLCATs and MOHAWK, but the experiments compare only against the original teachers and variants of the proposed method. This makes it difficult to judge whether the proposed conversion achieves better model quality or requires less training than existing approaches.

  • The practical efficiency improvement is unclear. Although RAM-Net maintains a fixed-size recurrent state, the paper does not provide inference latency, throughput, or memory usage comparisons with the original Transformer models. A fixed-size state does not necessarily provide better efficiency at all context lengths, especially when the original model uses GQA or a quantized KV cache. Without these measurements, it is difficult to evaluate whether the performance loss after conversion is justified by the efficiency improvement.

  • There is no study of how the conversion training budget affects model quality. The paper uses a fixed 500M-token training budget for all models, but does not investigate whether additional training can further close the performance gap with the teacher. It is unclear whether the remaining performance loss comes from insufficient training or limitations of the converted architecture. A training-token scaling study would help clarify this and provide a better understanding of the compute needed for successful conversion.

  • The converted models show significant performance degradation on LongBench. For example, the LCC score drops from 70.0 to 33.4 for Mistral-7B and from 66.7 to 24.8 for Llama-2-7B (Table 10). These results suggest that the converted models may struggle to preserve some of the teachers' capabilities, particularly on longer-context tasks. More evaluations across different context lengths would help explain these performance gaps.

Questions for authors
  • Could the authors compare the proposed method with existing Transformer-to-linear-attention conversion methods using the same teacher models and similar training budgets?

  • Could the authors provide inference efficiency comparisons between RAM-Net and the original Transformer models, including latency, throughput, and memory usage at different context lengths and batch sizes? It would also be helpful to compare against KV-cache quantization baselines.

  • Could the authors provide a training-token scaling study by extending the full-model recovery beyond 500M tokens, such as 1B or 2B tokens? Does the converted model continue approaching the teacher's performance, or does the improvement start to plateau? How much training compute is needed for conversion?

  • Could the authors provide more comments and insights on the LongBench performance?

10 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Dear Reviewer ektz,

Thank you for the constructive feedback. Below, we present three additional conversion baselines and decoding benchmarks showing 2.8×2.8\times the Llama-2-7B teacher’s throughput at 8K and batch size 8. We also show gains from longer recovery consistent with scaling laws and discuss how the dataset for recovery may contribute to gaps on downstream and retrieval tasks. We address each point below.

AuthorsOpenReview ↗

Response to W1/Q1

W1/Q1: Compare the proposed method with existing Transformer-to-linear-attention conversion methods.

Re: Thank you for raising this point. We add the two baselines you suggested, LoLCATs [1] (Transformer to hybrid linear and sliding-window softmax attention) and MOHAWK [2] (Transformer to Mamba-2), along with Mistral-SUPRA [3] (Transformer to gated recurrent linear attention). All three comparisons use Mistral-2-7B-v0.1 as the teacher. Our migration achieves the lowest WikiText perplexity among the methods we compare, with downstream task accuracy close to that of MOHAWK and Mistral-SUPRA.

Notably, a fair comparison for this experiment is difficult. First, the token budget needed to recover teacher performance depends on both the student architecture and the conversion recipe. For example, a student that retains softmax attention within a sliding window may require fewer tokens to recover the teacher’s behavior than one that replaces softmax attention entirely. Even for a fixed architecture and total budget, as in the MOHAWK comparison, the allocation of tokens across recovery stages can affect final quality. Second, the compute cost per token differs across methods and training stages. For LoLCATs as an example, at a 4K context length, it computes a 4096×40964096 \times 4096 attention matrix per head to match the teacher’s attention outputs layer by layer. Based on throughput measured on a single H200, its attention distillation on the first 250M tokens alone would take about 53 GPU-hours, before LoRA fine-tuning. By comparison, RAM-Net’s full 500M-token migration from the same Mistral-7B teacher takes about 46 GPU-hours.

Based on these issues, we therefore use three complementary settings:

  • LoLCATs: original dataset and token budget. We retain LoLCATs’ two-stage procedure, 44.7M tokens from the Alpaca dataset and use a 4K training context to match RAM-Net.
  • MOHAWK: matched data, context length, and token budget. We use the same FineWeb-Edu dataset, 4K training context, and 500M-token budget as RAM-Net. We scale its token budget and allocate 13.3M tokens to matrix alignment, 26.7M to block alignment, and 460M to end-to-end distillation.
  • Mistral-SUPRA: released checkpoint. The authors report uptraining Mistral on RefinedWeb with budgets of 20B and 100B tokens. We evaluate their released 100B-token checkpoint.

The table reports migration runtime using GPU hours per 100M tokens, accuracy on the six tasks in Table 1 of our paper, and WikiText perplexity at context lengths of 1K, 2K, 4K, and 8K, respectively.

MethodTokensGPU-h/100M tok ↓Wiki PPL @ 1K ↓Wiki PPL @ 2K ↓Wiki PPL @ 4K ↓Wiki PPL @ 8K ↓ARC-EARC-CPIQASciQCOPAHella.Avg. ↑R (%) ↑
Teacher (8k ctx)——9.448.638.228.0980.350.980.996.492.061.577.0100.0
LoLCATs44.7M38.412.2711.8711.8168.2381.350.781.094.294.062.477.3100.9
Mistral-SUPRA100B—12.1911.4311.3611.4276.442.880.194.687.058.073.289.2
MOHAWK500M6.111.9711.3511.1011.0377.045.578.992.989.057.973.590.8
RAM-Net500M9.211.2110.4710.1410.0077.244.179.791.587.058.072.989.4

Key conclusions:

  1. RAM-Net achieves the lowest WikiText PPL among the converted models and retains its ability to extrapolate beyond the migration window. In the controlled comparison with MOHAWK, both methods use the same dataset and token budget. RAM-Net achieves lower PPL at all four context lengths while trailing MOHAWK by only 0.6 percentage points in six-task average accuracy. Compared with Mistral-SUPRA, RAM-Net achieves lower PPL and comparable retention (89.4% vs. 89.2%) with just 1/200 of the migration tokens.

  2. LoLCATs’ hybrid attention may ease migration by retaining part of the teacher’s attention mechanism. LoLCATs combines linear attention with sliding-window softmax attention, which may keep it architecturally close to the teacher. Under its original recipe, it comes closest to the teacher’s six-task average with only 44.7M tokens. Another evidence is the gap in language modeling beyond the migration window: both LoLCATs and RAM-Net use 4K sequences for migration from the same teacher pretrained at 8K. At 8K, LoLCATs’ PPL rises sharply, a pattern often observed when Transformers exceed their training length, whereas RAM-Net’s PPL remains stable and lower than LoLCATs.

[1] Zhang et al. LoLCATs: On Low-Rank Linearizing of Large Language Models. 2024. ArXiv:2410.10254.

[2] Bick et al. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models. 2024. ArXiv:2408.10189.

[3] Mercat et al. Linearizing Large Language Models. 2024. ArXiv:2405.06640.

AuthorsOpenReview ↗

Response to W2/Q2 (Part 1/2)

W2/Q2: Report inference latency, throughput, or memory usage comparisons with the original Transformer models.

Re: Thank you for raising this.

  1. At 8K sequence length and batch size 8, RAM-Net achieves 2.8×2.8\times the decode throughput of the Llama-2-7B teacher and 71% of the Qwen2.5-1.5B teacher’s throughput, with lower peak decode memory in both cases. We compare each migrated model with its original teacher on a single NVIDIA H200 GPU. For Qwen, we also evaluate an INT4 KV-cache baseline using the HQQ backend [1] in Hugging Face Transformers. All models use BF16 weights. Each sequence has 8,192 input tokens. We measure prefill latency for the entire batch and average decode latency over the next 128 cached steps. Each cell reports results for batch sizes 1 and 8, respectively, using the median of five runs after warm-up. Decode throughput counts tokens across the batch, and peak decode memory covers the whole model.
ModelPrefill (ms) ↓Decode (ms/step) ↓Decode (tokens/s) ↑Decode peak (GiB) ↓
Qwen2.5-1.5B teacher (GQA)74 / 53018.49 / 16.5454.1 / 483.73.13 / 4.72
Qwen teacher (GQA, INT4 KV)264 / 1,45325.12 / 26.0539.8 / 307.13.04 / 3.91
RAM-Net from Qwen108 / 69923.15 / 23.4243.2 / 341.63.20 / 4.35
Llama-2-7B teacher (MHA)272 / 2,23425.88 / 78.4238.6 / 102.016.73 / 45.59
RAM-Net from Llama328 / 2,42727.88 / 28.0035.9 / 285.712.41 / 15.94
  • For Llama-2-7B, RAM-Net’s decode latency stays near 28 ms as batch size increases from 1 to 8. As batch size increases from 1 to 8, RAM-Net’s decode latency stays near 28 ms per step, while the teacher’s rises from 25.88 to 78.42 ms. At 8K, the teacher caches 4,096 MiB of KV per sequence, while RAM-Net maintains 516 MiB of state size. At batch size 8, RAM-Net also reduces peak decode memory by 65%. Prefill remains slower at both batch sizes. The prefill latency gap likely reflects differences in kernel optimization, including FlashAttention-style tiling and GPU thread-block utilization.

  • Qwen2.5-1.5B: RAM-Net achieves 70.6% of native GQA’s decode throughput with 7.7% less peak decode memory at 8K and batch size 8. The teacher shares two KV heads across 12 query heads, reducing the amount of cached data it reads during decoding. At both batch sizes, the native GQA implementation has lower prefill and decode latency than RAM-Net. As with Llama-2-7B, RAM-Net’s slower prefill may reflect differences in kernel optimization. At batch size 8, it reaches 70.6% of native BF16 GQA throughput. Relative to the BF16 teacher, INT4 KV-cache quantization further reduces peak decode memory, but the HQQ implementation is slower in both prefill and decoding, consistent with the latency overhead noted by Hugging Face [2]. Relative to HQQ INT4, RAM-Net achieves 8.5% higher decode throughput at batch size 1 and 11.2% higher throughput at batch size 8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.