acceptodds
Under review as a conference paper at ICLR 2027

SALSA: Training-Consistent Sparse Caching with Sink Preservation for Linearized LLMs

Abstract

The full attention mechanism in standard Transformer architectures incurs quadratic computational complexity with respect to sequence length and linearly growing KV cache overhead, posing severe efficiency bottlenecks in long-sequence scenarios. Although linear attention reduces computational and cache overhead, its compression of history into a fixed-size hidden state prevents precise recall of distant information. Existing conversion methods, such as LoLA, attempt to introduce a sparse cache into hybrid architectures to retain hard-to-compress tokens, but add it only during inference. This causes the attention mechanism learned during conversion to differ from the one actually executed at deployment, leading to a training-inference computation graph mismatch. The core insight of this paper is that the sparse cache should not be merely an inference-time patch; rather, it should be incorporated into the conversion computation graph from the attention distillation stage onward, so that training and inference execute the same attention mechanism. Based on this, we propose SALSA (Sink-Aligned Linearization with Sparse Adaptive Caching), a training-inference-consistent sparse caching linearization framework. SALSA partitions history into three components—a sliding window, a sparse global cache, and a linear attention hidden state—and enables the sparse cache to participate in both attention distillation and fine-tuning, thereby eliminating training-inference distribution shift. On this basis, we further propose two supporting refinements: (1) retaining initial sink tokens in the sliding window to stabilize the attention distribution; and (2) offline layer-wise sparse cache budget allocation based on inter-layer heterogeneity. Experiments on Llama-3.1-8B show that SALSA substantially outperforms prior state-of-the-art methods such as LoLCATs and LoLA on S-NIAH, improving average accuracy by more than 20 absolute points over LoLA while closely matching the full-attention baseline on S-NIAH-1 at 0.5K–2K context lengths. On the efficiency side, SALSA keeps decode cost nearly constant as context grows and, at 32K, achieves higher maximum decode throughput than FA2 under memory constraints, with nearly flat decode memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid attention, long-sequence modeling, model conversion, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gBMVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SALSA, a training-inference-consistent sparse caching framework for converting pretrained full-attention LLMs into hybrid linear-attention models.

The key motivation is to address the training-inference mismatch in previous methods such as LoLA, where the sparse cache is introduced only during inference. SALSA incorporates the sparse cache into both attention distillation and fine-tuning, allowing the model to learn the same attention mechanism used during deployment. The authors further introduce sink token preservation and offline layer-wise cache budget allocation to improve model performance.

Experiments on Llama-3.1-8B, Qwen3-8B, and Mistral-7B demonstrate substantial improvements over existing linearization methods, particularly on associative retrieval tasks. SALSA also achieves higher decode throughput under memory-constrained long-context settings, although its prefill efficiency remains limited.

Critical strengths
  1. Meaningful motivation. Training-inference consistency is an important issue in model conversion. Incorporating the sparse cache into the training process, rather than introducing it only at inference time, is a reasonable and well-motivated approach.

  2. Significant improvements over existing linearization methods. SALSA substantially outperforms LoLCATs and LoLA on several retrieval tasks. The ablation studies also provide evidence that including the sparse cache during training can improve model performance.

  3. Practical memory advantages. By combining linear attention with a fixed-capacity sparse cache and sliding window, SALSA maintains nearly constant decode memory as context length increases, making it potentially useful for memory-constrained inference.

Critical weaknesses
  1. Limited long-context capability. Despite the improvements over previous linearization methods, SALSA still suffers from severe performance degradation as context length increases. For example, on Llama-3.1-8B, S-NIAH-1 accuracy drops from 90.2% at 4K to only 0.8% at 8K, while the full-attention baseline maintains 100% accuracy. Even with a larger inference cache, the accuracy at 8K is only 40.4% and drops to 0% at 16K. These results raise concerns about the practical effectiveness of SALSA for long-context inference, which is a central motivation of the paper.

  2. Limited novelty and unclear contribution attribution. Several components of SALSA, including SRE-based sparse caching, sink token preservation, and pyramid-shaped cache budget allocation, are adapted from existing work. The main novelty lies in incorporating sparse caching into the model conversion training process. However, the ablation results show that sink token preservation contributes substantially to the overall performance improvement. For instance, adding sink tokens improves S-NIAH-1 accuracy at 2K from 42.2% to 97.2% under the training-consistent framework. While the effectiveness of training consistency is supported by the experiments, the large contribution of an existing technique makes the independent significance of the proposed innovation less compelling.

  3. Limited end-to-end inference efficiency. Although SALSA achieves promising decode throughput under memory constraints, its prefill throughput is significantly lower than FlashAttention-2. At 32K context length, FA2 achieves approximately 8.3× higher prefill throughput. Since prefill is an important component of long-context inference, the lack of a comprehensive end-to-end latency and throughput comparison makes it difficult to assess the overall practical benefits of SALSA.

Questions for authors
  1. SALSA demonstrates significant improvements over LoLA, but its retrieval performance still degrades sharply at longer context lengths. What are the main causes of this degradation? Is it primarily due to the limited capacity of the sparse cache, or the representational limitations of the linear attention state? Could the authors provide more comprehensive quality evaluations at 8K, 16K, and 32K?

  2. The ablation studies indicate that sink token preservation contributes substantially to the final performance. Could the authors further clarify the independent contribution of training-inference consistency, particularly its interaction with sink token preservation?

  3. Could the authors provide end-to-end inference latency or throughput results that include both prefill and decoding? This would help establish whether the decode-stage benefits are sufficient to compensate for the significantly slower prefill stage under realistic workloads.

4 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Long-context degradation and its cause

Thank you for your specific, testable questions. Addressing your concerns, we have added the degradation-cause diagnosis, the enlarged-inference-cache and distillation-length-right-shift experiments, and the end-to-end efficiency table; all new data is in the revised manuscript (marked blue). We respond point by point.

W1 & Q1: Limited long context (90.2% @4K → 0.8% @8K) — is the main cause cache capacity or the linear-state limit?

Question summary: You point out that SALSA's long-context capability is limited — after 90.2% @4K, accuracy drops to 0.8% @8K — and ask us to confront this directly; you further ask for a causal attribution of the 8K+ degradation: insufficient cache capacity, or a representation/readout limit of the linear state.

Response: We confront the numbers directly. The main cause is the representation/readout limit of the linear state (constrained by the distillation length), with cache capacity only a secondary amplifier — established by a three-step evidence chain: enlarging the inference cache rules out the capacity hypothesis → needle-retention diagnosis localizes the failing component → distillation-length right-shift validates the root cause. The operating boundary shifts right with the training side (8K recovers to 40.6%); within the calibrated operating region (≤4K) SALSA remains the best conversion; 16K is the current hard boundary. The distillation-length right-shift experiment was in fact added in direct response to your question about the cause of the degradation.

Details:

  1. Step 1: ruling out the capacity hypothesis (enlarged-cache experiment; weights unchanged, inference configuration only). Scale the window and sparse budget proportionally (window sizes all include 4 sink tokens, matching the paper's convention; coverage matches the paper's appendix large-cache configuration — window 260, sparse 768 — item by item at 8K: 3.1% / 9.4%):
ConfigurationS-NIAH-1 @8K@16K
Default (window 68, sparse 64)1.0%0%
Large cache (window 260, sparse 768, paper appendix)40.4%0%
Proportional scale (window 516, sparse 1536, new this round)94.4%0%

Capacity is indeed an effective knob at 8K (threshold-style recovery to 94.4%); but a setting with identical coverage still yields 0% @16K — the 16K failure is not insufficient cache share. Table and boundary statement in revised Appendix B.4 (Table 14, marked blue).

  1. Step 2: needle-retention diagnosis (localizing the failing component). We audit the sparse-cache content per (layer, head) in the generation mechanism, record the actual cached content of every layer and head, compute the retention rate of needle tokens in the attention-visible path (sink ∪ window ∪ sparse), and jointly classify with answer correctness:
Metric4K (normal)8K16K
Accuracy0.880.000.00
Mean layer coverage of the needle0.350.260.16
Layers that fully lose the needle3.1%4.5%20.9%
Generation formnormalrepetition looprepetition loop

8K: the selection mechanism has not failed — the needle remains on the visible path most of the time (mean coverage only drops from 0.35 to 0.26), and there is no intermediate state of "visible but read out wrong"; generation degrades globally into repetition loops — the failure is in linear-state integration/readout. 16K: the selection mechanism also begins to degrade — coverage drops further to 0.16, fully-lost layers rise to 20.9%, with a depth gradient (needle coverage near 0 in deep layers); both information channels fail simultaneously. Diagnosis protocol and full metric table in revised Appendix B.5 "Diagnosing Accuracy Degradation at Longer Contexts" (Table 15; the four metric definitions are written into the main text, marked blue).

  1. Step 3: root-cause validation (distillation-length right-shift). The failure is consistent with the training distribution: ϕ\phi is distilled only on 1024-length sequences and extrapolates beyond. Doubling the distillation length to 2048 (data, total tokens, and inference configuration all unchanged) recovers 8K from 1.0% to 40.6%, with no degradation ≤2K (98% @4K) — the operating boundary is set by the distillation supervision distance and can be shifted right on the training side. See revised main text §4.5 (Table 9).

  2. Future directions: larger linear-state capacity (e.g., a larger feature dimension) and gated/decay-type linear attention are proposed here as future work; the combination experiment of training length and inference cache is ongoing, and results will be supplemented during the discussion period.

AuthorsOpenReview ↗

Supplement to Response to W1 & Q1

  1. The operating range is calibrated: ≤4K is the standard operating region (accuracy close to full attention, constant memory); 8K can be recovered to 94.4% by enlarging the inference cache, or to 40.6% by shifting the training side; 16K is the current hard boundary — the mechanism analysis and boundary statement are in the revision (Appendix B.5 Table 15 + B.4 Table 14).

  2. Within the calibrated region SALSA remains the best conversion: the combination of constant memory ≤4K and near-full-attention accuracy has no peer among LoLCATs/LoLA/HALO/LAWCAT — RULER @2K at matched budget: SALSA 70.23 vs. LoLA 22.44 / LoLCATs-516 5.81 (base 95, paper Table 2); S-NIAH @4K: SALSA 90.2 vs. LoLA 1 / LoLCATs 0.8 (paper Table 1).

AuthorsOpenReview ↗

Response to W2 & Q2: Novelty and component attribution

W2 & Q2: Limited novelty (sink is an existing technique yet presented as the largest contributor); the independent contribution of training consistency and its interaction with sink

Question summary: You question the novelty of this work — sink tokens are an existing technique (StreamingLLM), yet are presented as the largest source of contribution — and ask for an attribution of the independent contribution of training consistency and its interaction with sink.

Response: The attribution experiments directly answer both questions: how much sink itself contributes, and what training consistency independently adds. Sink alone (cache not in the training loop) reaches only 16.6% @4K, far below the full method's 91.2% (same inference configuration) — sink provides no retrieval capability; its contribution is making the three-path mechanism trainable; the independent gain from bringing the cache into the training loop is the 2×2 below. The novelty of this work therefore lies not in sink itself, but in three reusable principles and the adaptation design.

Details:

  1. Mechanism-consistency principle: the cache mechanism must participate in the training loop to be effective — the full 2×2: sink alone 16.6%, cache alone 2.4%, full combination 90.2% (@4K; paper Tables 6/7 with the revised §4.4 Table 8), either component alone near-useless. This is a general constraint on the conversion paradigm, not specific to this architecture. The attribution ablation and the new baseline row are in revised main text §4.4 (Table 8, marked blue).

  2. Capacity–competition principle: small-cache training > large-cache training (70.23 vs. 56.76) — a counterintuitive, reusable empirical rule. Paper Appendix B.3 (Table 13).

  3. Sink's three-path accounting adaptation: permanently retained during training and never entering the cache/linear state — different from StreamingLLM's inference-time usage; this is the design component that makes intra-layer three-path mixing trainable.

AuthorsOpenReview ↗

Response to W3 & Q3: End-to-end efficiency measurements

W3 & Q3: End-to-end results including prefill + decode

Question summary: You ask for complete end-to-end efficiency results covering both prefill and decode, rather than component-level measurements only.

Response: We provide the end-to-end table fully measured on H800 80 GiB / bf16. Conclusions: SALSA's TPOT and memory are nearly independent of length from 2K onward; every workload where FA2-vanilla OOMs runs under SALSA; even when FA2 uses chunked prefill to trade time for memory, its TPOT @32K is only on par with SALSA's, with 1.7× higher memory.

Details (the table is in revised Appendix B.8 "End-to-End Serving Measurements", Table 18, marked blue):

prompt × batchFA2 TTFTSALSA TTFTFA2 TPOTSALSA TPOTFA2 memorySALSA memory
2K × 80.50 s5.27 s13.8 ms55.1 ms17.5 GiB28.2 GiB
8K × 82.35 s22.2 s20.3 ms55.5 ms23.6 GiB28.2 GiB
16K × 85.49 s44.7 s29.0 ms55.8 ms31.8 GiB28.2 GiB
16K × 16OOM (11.2 s)88.4 sOOM (46.8 ms)60.4 msOOM (48.5 GiB)41.3 GiB
32K × 8OOM(14.6 s)89.8 sOOM (46.3 ms)55.8 msOOM (48.0 GiB)28.2 GiB
32K × 16OOM177.5 sOOM60.3 msOOM41.3 GiB

Note: values in parentheses are measured with FA2 chunked prefill (chunk 4096, trading time for memory); at batch 32 both methods OOM (not listed); 32K×16 is feasible only for SALSA.

  1. Feasibility boundary: every FA2-vanilla OOM setting (b8@32K, b16@16K/32K) runs under SALSA; linear extrapolation of the maximum batch @16K: 80 GiB card FA2 ≈31 vs. SALSA ≈39; 48 GiB card 15 vs. 20.

  2. TPOT/memory flatness: SALSA TPOT goes only 55.1→55.8 ms from 2K→32K (×1.01), with memory constant at 28.2 GiB; FA2 TPOT grows 13.8→29.0 ms (×2.1) with memory growing linearly to OOM — predictable decode latency has direct value for serving schedules.

  3. The "4.0×" accounting: aggregated decode throughput at each method's maximum feasible batch @32K is 66 vs. 266 tok/s ≈ 4.0×, consistent with paper §4.6; the accounting is now stated explicitly in revised Appendix B.8.

  4. Counting FA2's chunked-prefill variant: @32K b8 its TPOT is 46.3 ms, on par with SALSA's 55.8 ms (ratio 1.21), with memory 48.0 vs. 28.2 GiB (1.7×).

  5. Applicability: we do not claim per-request superiority (FA2 is faster within its feasible region; the prefill gap is a constant factor, with the narrowing path in Appendix B.7, Table 17); SALSA's end-to-end value is serving capability in memory-constrained regimes plus length-independent decode cost.

Reviewer_NSDMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel sequence-mixer architecture, SALSA, that consist of three parts: sliding window, sparse cache, and linear state, which is a mixture of sliding window attention, token sparse attention, and linear attention. Existing method integrates these three parts in a training free manner, resulting in training-inference mismatch. SALSA alleviates this problem by integrating the target architecture into continuous training, thereby eliminating the mismatch and improve long-context performance. The experiments of this paper show the possibility of finetuning an existing model to SALSA architecture and improve retrieval performance and commonsense reasoning.

Critical strengths
  1. This paper targets an important problem, that the intra-layer hybrid architecture of swa, linear attention and token sparse attention is often converted without training. Integrating this into training can largely alleviate this problem and improve the final performance.

  2. Experiments are conducted at large scale LLMs and included multiple benchmarks, which makes this work solid and sound.

  3. This paper is presented in a well-written manner, that the statement of methods and experiments are clear and easy to interpret.

Critical weaknesses
  1. There are several related works that should be compared with, but currently lacking in the experiment section. For example, SWA + full attention, SSM + full attention, SSM + sparse attention, should be included into comparison. Note that converting an existing model to those architectures might be hard and compute consuming, but you can conduct pretraining experiments to show the superiority of the proposed architecture rather than existing popular hybrid method.

  2. The necessity of hybridizing swa, linear, and token-sparse attention into one layer should be discussed with empirical results. Hybrid architectures can be designed in multiple ways. For example, you may have a layer of swa and token-sparse attention, hybridized with linear layers, while keeping the budget of linear state size and token cache same. Which one is better?

  3. Since the experiments are conducted at 7-8B level (Qwen3-8B and Mistral-7B), RULER-single 1-3 are considered too simple for these models, while they can obtain non-trival scores on multi-key/value/query retrieval tasks, as well as VT/CWE/FWE. What is the reason of not reporting these results? Also, retrieval is only one aspect of long-context performance. You may benchmark the proposed method against baselines on other long-context benchmarks, e.g., HELMET, to include long-doc QA, ICL, and other tasks.

Questions for authors

See above.

5 comments
AuthorsOpenReview ↗

Response to W1: Comparison with hybrid architectures and the conversion paradigm

Thank you for your comments on the architecture design space and evaluation coverage. We respond point by point.

W1: Missing comparisons with hybrid architectures such as SWA+full and SSM+full; suggests pretraining experiments to demonstrate architectural superiority

Question summary: You point out the lack of comparisons with hybrid architectures such as SWA+full and SSM+full, and suggest demonstrating architectural superiority via pretraining experiments.

Response: This work belongs to the model conversion paradigm — no re-pretraining is performed; 40M-token distillation + LoRA lets an existing 8B model inherit its 15T-token pretraining knowledge. Architecture comparisons from scratch via pretraining are beyond this paradigm and our compute budget, and we honestly state that they are infeasible for us. Moreover, the hybrid architectures you list (SWA+full / SSM+full) still keep a full KV cache in their attention/sparse layers, so overall memory still grows linearly with length — a different class from this paper's goal of constant per-layer memory, and not directly comparable. The comparable methods in the same class (conversion-based) are already fully covered in the paper, and SALSA leads among them.

Details:

  1. Scope of this work's paradigm: the conversion paradigm does not re-pretrain; training Jamba/Hymba-style models from scratch is beyond the paradigm and our compute budget, and we honestly state this is infeasible for us.

  2. The listed architectures are a different class from our goal: SWA+full / SSM+full (Jamba, Zamba, Hymba types) and inter-layer hybrids (NHA, MiniCPM-SALA) still keep a full KV cache in their attention/sparse layers — overall memory grows linearly with length. Our goal is an end-to-end conversion with constant per-layer memory; the optimization targets differ, and a direct comparison would conflate "hybrid architecture design" with "the conversion paradigm".

  3. Same-class comparators are fully covered: conversion-based methods (LoLCATs, LoLA, HALO, LAWCAT) are all compared in the paper, and SALSA leads at a matched cache budget — RULER @2K at matched budget: SALSA 70.23 vs. LoLA 22.44 / LoLCATs-516 5.81 (base 95). Full comparisons in paper Table 1 (S-NIAH) / Table 2 (RULER @2K) / Table 5 (commonsense reasoning).

AuthorsOpenReview ↗

Response to W2: Intra-layer three-path mixing vs. inter-layer mixing

W2: Intra-layer three-path mixing vs. inter-layer mixing (some layers swa+sparse, others linear) — which is better at the same budget?

Question summary: You propose inter-layer mixing (swa+sparse in some layers, linear in the rest) as an alternative design and question the necessity of intra-layer three-path mixing.

Response: In inter-layer mixing, the layers containing full attention/sparse must keep a full KV cache — total memory still grows linearly with length, forfeiting the linear layers' memory advantage. Intra-layer three-path mixing lets the three paths share one softmax normalization and compete token by token, and exact capacity is available in every layer. A rigorous same-budget comparison is valuable future work; we do not present unverified numbers during the rebuttal period.

Details:

  1. The inter-layer design forfeits the linear layers' core advantage: in inter-layer mixing, layers with full attention/sparse must keep full KV (the sparse layers in your proposal likewise) — total memory still grows with length, i.e., "the layers with exact retrieval lose the memory advantage, while the memory-owning layers cover only part of the depth" — worst of both. MiniCPM-SALA, cited in the paper, is of this type: its sparse layers must store full KV.

  2. Key properties of intra-layer three-path mixing: the three paths share a single softmax normalization, jointly calibrated and competing token by token (sink permanently retained + window covering recency + sparse cache covering selected distant tokens); cache capacity is available in every layer rather than concentrated in a few layers; in the inter-layer design, a linear layer's approximation error cannot be corrected by exact attention in the same layer. The design motivation is in paper §3.

  3. Same-budget comparison (future work): a rigorous "intra-layer three-path vs. inter-layer" comparison at the same budget requires training-level ablations; we list it as future work and do not present unverified numbers during the rebuttal period.

AuthorsOpenReview ↗

Response to W3: S-NIAH difficulty, full RULER coverage, and HELMET

W3: S-NIAH too simple; should report full RULER task types; should test HELMET

Question summary: You consider S-NIAH too simple, ask for the full set of RULER task types, and ask us to test the HELMET benchmark.

Response: The main Llama model already has the full 9-task RULER @2K (SALSA 70.23 vs. LoLA 22.44 / LoLCATs-516 5.81, with large leads on CWE/FWE/SQA: 86.9/75.67/63.33); this round we additionally ran full Qwen RULER and included it in the revised appendix. The main text uses S-NIAH to expose component contributions because component effects are most distinguishable there — S-NIAH is indeed easy for base models (100%), but it is precisely the hardest for converted models. HELMET is listed as future work.

Details:

  1. The main Llama model already has full RULER (paper Table 2, @2K, 9 tasks): SALSA mean 70.23 vs. LoLA 22.44 / LoLCATs-516 5.81 (base 95), with large leads on CWE/FWE/SQA (86.9/75.67/63.33).

  2. New evaluation: full Qwen RULER @2K (9 tasks × 4 models, protocol identical to the paper; full table in revised Appendix B.1, Table 12, marked blue): SALSA mean 49.0 is the best among converted models (LoLA 31.2 / LoLCATs 18.9, base 95.8); retrieval is close to base (MK1 99.4 / MV 96.1 / MQ 97.6 / VT 81.7, base 100); MK2 is the only non-zero conversion (7.0); SQA 49.5 (second among conversions, behind LoLCATs' 54.0). Weaknesses are presented as they are: on Qwen, all three conversions collapse on the aggregation tasks CWE/FWE (SALSA 0.06/9.8, unimproved and last; on Llama, SALSA leads on these two, 86.9/75.67 — stated per family). In addition, we ran limited RULER tests on Mistral; the results are indeed poor (all three converted methods in the failure regime), and we honestly acknowledge this as a family limitation — the Mistral boundary table in paper Table 4 already reported the collapse @2K, and this round's limited tests independently confirm it.

  3. HELMET: we were unable to complete the full HELMET suite within the rebuttal period. RULER already includes a QA task (ruler_qa_squad) and aggregation tasks similar to ICL, and SALSA's QA performance (Qwen SQA 49.5, second among conversions) partially addresses this; the full HELMET evaluation is listed as future work.

  4. On "S-NIAH is too simple": true for base models (100%), but it is precisely the hardest for converted models — the previous best conversion result was only 93.8% @0.5K (LoLA), and LoLCATs only 8.0% @0.5K (paper Table 1). Component contributions (training consistency / sink / budget) are most distinguishable on this task, which is why the ablation centers on it; breadth is ensured by full RULER (paper Table 2 plus this round's full Qwen tasks).

Reviewer_EFyQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SALSA proposes a framework for converting pretrained Transformers into hybrid linear-attention models by incorporating sparse caching into both attention distillation and parameter-efficient fine-tuning. It combines a sliding window, a sparse global cache, and a linear attention state, with sink token preservation and offline layer-wise cache budget allocation. SALSA improves retrieval and reasoning performance over LoLCATs and LoLA while maintaining bounded cache memory during decoding.

Critical strengths
  • Incorporating sparse caching into both conversion stages is a reasonable approach to addressing the architectural mismatch introduced by inference-only caching.
  • SALSA achieves substantial improvements over LoLA, increasing average accuracy from 22.44% to 70.23% on the evaluated RULER tasks at 2K context length. MMLU improvements across all three evaluated model families also support the applicability of the approach beyond a single model.
  • The sink token ablations and distillation-loss curves provide useful evidence that preserving initial tokens improves conversion quality. The comparison between small-cache and large-cache training also offers an interesting observation about the interaction between sparse caching and linear attention.
Critical weaknesses
  • Regarding the impact on the incremental contributions: Since the architecture and supporting refinements largely build on existing methods, the benefit of incorporating sparse caching into training is particularly important. However, Table 6 changes multiple training components simultaneously, while Table 7 shows that adding sink tokens alone raises 4K S-NIAH-1 accuracy from 2.4% to 75.4%. A baseline that preserves sinks during conversion but introduces sparse caching only at inference, with the same inference cache configuration, would help isolate the benefit of the proposed training consistency.
  • Limited effectiveness at longer contexts: Although SALSA substantially improves over the evaluated baselines, the broader RULER evaluation is conducted only at 2K context length. Even with the enlarged cache in Table 11, S-NIAH-1 accuracy drops to 40.4% at 8K and 0% at 16K. This raises concerns about its practical utility in the long-context regime where the paper reports its strongest efficiency benefit. Accuracy–efficiency comparisons at matching context lengths and cache configurations would help establish the useful operating range.
  • Incomplete practical efficiency assessment: The reported 4.0× maximum decode throughput improvement at 32K demonstrates an advantage under memory constraints, but FA2 achieves 8.3× higher prefill throughput at the same context length. Although the authors acknowledge this limitation, the overall benefit remains unclear without end-to-end measurements that include prefill. Efficiency comparisons against LoLA and LoLCATs under matched cache budgets would also help assess the additional cost of the proposed framework.
Questions for authors

Please address the concerns raised in Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to W1: Contribution attribution (2×2 ablation)

Thank you for your constructive comments. Addressing your concerns, we have added the attribution baseline (completing a full 2×2), the calibration and mechanism analysis of the operating range, and an end-to-end efficiency table; all new data has been written into the revised manuscript (marked in blue). We respond point by point.

W1: Contribution attribution unclear (Table 6 varies multiple components at once; the "sink in training, cache inference-only" baseline is missing)

Question summary: You point out that the ablation in Table 6 varies multiple components simultaneously and cannot separate their contributions, and that a key baseline is missing — sink included in training while the cache is introduced only at inference.

Response: The confound you identified is real; we have added that baseline and completed the full 2×2 ablation. The conclusion is clear: either component alone is near-useless (sink alone 16.6%, cache alone 2.4%), and only the full combination reaches 90.2% — training consistency (the mechanism must be used inside the training loop) is the main source of retrieval capability, not the mere presence of the mechanism.

Details:

  1. New baseline design (attributing training consistency): we train a set of weights with sink retained during distillation + LoRA but sparse_budget=0 (the sparse cache is not in the training loop), and at inference inject the identical pyramid cache as SALSA via configuration — different weights, identical inference configuration; the difference isolates the independent contribution of training consistency. S-NIAH-1 results:
Seq lengthSALSA (full method)Sink in training + cache injected at inference only (new)Same weights, no cache allocation
512100.0100.0100.0
1024100.099.899.6
204897.463.853.6
409691.216.616.2

The full 9-cell (task × length) table is in the revised main text §4.4, new attribution ablation (Table 8, marked blue).

  1. Training consistency is the dominant contributor: with the inference configuration completely identical (same sink, same pyramid allocation), whether the cache is inside the distillation loop makes a 91.2% vs. 16.6% difference @4K (~75 points). The benefit is not "the mechanism exists" but "the mechanism must be used consistently during training".
  2. Merging with the paper's existing rows into a full 2×2 (corresponding to paper Tables 6 and 7 respectively):
ConfigurationSink in trainingCache in trainingS-NIAH-1 @4K
w/o training (paper Table 6)✗✗2.4%
w/o sink & budget (paper Table 7, cache in training)✗✓2.4%
New baseline (added this round: sink in training, cache not in loop)✓✗16.6%
+Sink (paper Table 7, cache in training + sink, no pyramid)✓✓75.4%
SALSA (paper Table 7, +Sink & Budget)✓✓90.2%

Reading the roles off the table row by row: sink in training serves to stabilize distillation and make the three-path mechanism trainable (2.4%→16.6%) and does not itself provide retrieval capability; the bulk of retrieval capability comes from cache-in-training (16.6%→75.4%); the pyramid budget allocation contributes the remaining 75.4%→90.2%.

AuthorsOpenReview ↗

Response to W2: Operating range calibration

W2: Limited long context — please honestly calibrate the useful operating range

Question summary: You point out that SALSA's long-context capability is limited (sharp degradation after 4K) and ask the paper to honestly calibrate its useful operating range.

Response: We fully agree, and the operating range is now explicitly calibrated in the revision: ≤4K is the standard operating region (accuracy close to full attention; per-layer cache constant, not growing with length); 8K can be recovered to 94.4% by enlarging the inference cache, or to 40.6% by shifting the training side; 16K is the current hard boundary (mechanism analysis in revised Appendix B.5, Table 15). The training-side path was in fact added in direct response to this comment of yours; we are currently testing an enlarged inference cache on top of it for 16K, and results will be supplemented during the discussion period.

Details:

  1. Operating range summary table (S-NIAH-1):
ContextDefault cache (window 68 incl. 4 sinks, sparse 64)Enlarged inference cache (window 516 incl. 4 sinks, sparse 1536, weights unchanged)Distillation length 2048 (cache unchanged)
≤2K97.4–100%—99%
4K91.2%—98%
8K1.0%94.4%40.6%
16K0%0%in progress
  1. ≤4K is the standard operating region: accuracy is close to the full-attention baseline (full attention @4K = 100, paper Table 1), and the per-layer cache stays constant, not growing with context.
  2. 8K can be recovered by enlarging the inference cache: scaling the cache proportionally (coverage strictly matching the paper's appendix large-cache configuration — window 260 incl. 4 sinks, sparse 768) reaches 94.4% @8K. Note this is threshold-style recovery — a setting with identical coverage still yields 0% @16K. The enlarged-cache table and boundary statement are in revised Appendix B.4 "S-NIAH with Larger Inference Cache" (Table 14, marked blue; including the explicit statement that "every configuration evaluated fails at 16K").
  3. Training-side right-shift: your comment on the operating range prompted us to look for ways to shift the boundary from the training side: merely extending the distillation length 1024→2048 (data, total tokens, and inference configuration all unchanged) recovers 8K from 1.0% to 40.6% — the operating region shifts right with the distillation supervision distance, showing that long-context capability can be improved by extending the training-side context length. This experiment is in the revised main text §4.5, new "Training with Longer Contexts" (Table 9, marked blue).
  4. Ongoing extended tests: we are completing the missing "distillation 2048" cell @16K in the table above, and testing an enlarged inference cache on top of the 2048 training length (stacking the training-side and inference-side paths) — the latter may effectively improve 16K performance. We emphasize that these are ongoing explorations; based on all experiments completed so far, 16K remains the current performance boundary (every enlarged-inference-cache configuration fails @16K, Appendix B.4 Table 14). New results will be honestly supplemented during the discussion period.
AuthorsOpenReview ↗

Response to W3: End-to-end efficiency and matched-budget comparison

W3: Incomplete efficiency evaluation (no end-to-end measurement; no matched cache-budget comparison)

Question summary: You point out that the efficiency evaluation is incomplete: it lacks an end-to-end measurement covering both prefill and decode, and an efficiency comparison with a matched cache budget.

Response: We have completed the end-to-end table measured on H800 (TTFT/TPOT/memory/feasibility × length × batch). Core conclusions: SALSA's decode latency and memory are independent of context length from 2K onward (TPOT only ×1.01); every workload where FA2-vanilla OOMs runs under SALSA; the prefill gap is a constant factor and can be narrowed via chunk size (+30% throughput). We do not claim uniform per-request speedups — SALSA's value is long-context serving capability under memory constraints.

Details (full tables in revised Appendix B.7 "Effect of Prefill Chunk Size" (Table 17) and B.8 "End-to-End Serving Measurements" (Table 18), marked blue):

  1. Feasibility: every FA2-vanilla OOM setting (b8@32K, b16@16K/32K) runs under SALSA (full table in Appendix B.8, Table 18).
  2. TPOT and memory flatness: SALSA TPOT goes only 55.1→55.8 ms from 2K to 32K (×1.01), with memory constant at 28.2 GiB from 2K; over the same range FA2 TPOT grows 13.8→29.0 ms (×2.1) and its memory grows linearly to OOM.
  3. Applicability: within FA2's feasible region FA2 is faster; SALSA's value is serving capability under memory constraints (a higher max batch at matched memory; aggregated decode throughput at each method's maximum feasible batch @32K ≈4.0× — the accounting reported in §4.6 and now stated explicitly in revised Appendix B.8).
  4. The prefill gap is a constant factor: after raising the internal chunk size (a declared implementation parameter in the paper) from 64 to 512, the gap to FA2 narrows from 9.5× to 7.3× @8K b8 and from 8.2× to 6.3× @16K b8, with +30% throughput (2943→3819 tok/s @8K b8); and there is no accuracy loss throughout (the ablation includes accuracy controls, Table 17). The compute graph consists entirely of public kernels (FLA/Triton + FlashAttention); no kernels need to be developed for any new operator, and integrating dedicated fused kernels to further narrow the gap is explicit future work.
  5. Matched-budget comparison: on the quality side, the paper already includes LoLCATs-132/516 (same total cache, paper Tables 1/2); on the efficiency side, the LoLA paper self-reports vanilla PyTorch with decode throughput of the same order (~28 tok/s/seq), so SALSA's additional cost relative to LoLA is only sparse-cache maintenance (measured in revised Table 18).
Reviewer_Z1HxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes model conversion framework, called SALSA, designed to transform full-attention pretrained large language models into hybrid architectures comprising a sliding window, a linear attention hidden state, and a sparse global cache. Addressing the training-inference computation graph mismatch present in prior inference-time caching methods (such as LoLA), SALSA integrates the sparse cache directly into the attention distillation and fine-tuning pipeline. Additionally, the paper proposes incorporating initial sink tokens into the sliding window and utilizing an offline, pyramid-shaped layer-wise sparse cache budget allocation. The paper sounds theoretically right and its method is novel. However, while the method achieves strong long-context performance on Llama-3.1-8B, we observe inconsistent stability and occasional performance fluctuations when generalizing to Qwen3-8B and Mistral-7B-v0.1.

Critical strengths

The framework that the paper is proposing is well principled and solid because:

The paper correctly identifies that past inference-time sparse cache patches suffer from a mechanism mismatch rather than merely a capacity mismatch. Elevating the sparse cache into the training loop and formalizing the small-cache training + large-cache inference strategy provide useful insights into capacity-competition effects during distillation step.

On Llama-3.1-8B, the method definitely demonstrates impressive gains on specific long-context associative recall benchmarks by notably improving average RULER/S-NIAH performance against static baselines like LoLCATs and LoLA.

Critical weaknesses

Prefill Bottleneck and Impractical TTFT: One of the most critical limitations of SALSA is its prefill throughput, which lags roughly an order of magnitude behind FlashAttention-2 (up to 8.3 x slower). Because the sparse cache requires chunked or per-token evaluation steps during prefill, Time-to-First-Token (TTFT) increases drastically. Deferring hardware-optimized CUDA or Thunder Kittens kernels to future work leaves the method practically unsuited for real-world interactive long-context serving, where low TTFT is a strict requirement.

Generalizability Across Other Model Families: While results on Llama-3.1-8B are solid, the paper's generalization claims are undermined by noisy and occasionally regressive performance on other architectures, such as Qwen3-8B and Mistral-7B-v0.1. For instance, on commonsense reasoning tasks like PIQA and WinoGrande for Mistral-7B-v0.1, SALSA occasionally yields marginal or negative deltas compared to the base model and baseline configurations. This suggests that the distillation recipe and pyramid budget allocation overfit to Llama's specific attention topology rather than offering a truly universal linearization paradigm.

Hyperparameter Complexity and Fragility: The framework stacks multiple interacting hyperparameters including sliding window length, sink token count, sparse cache size, chunk size CC for prefill, and arithmetic progression slopes for layer-wise allocation, making the training and deployment pipeline overly complex and brittle to tune for new architectures

Questions for authors

Question 1: TTFT issue. Given that Figure 3(c) demonstrates an 8.3x reduction in prefill throughput compared to FlashAttention-2, which severely penalizes TTFT, do you have some explanation of how SALSA can claim practical efficiency benefits in interactive agentic or chat workloads, especially where prompt lengths frequently and dynamically fluctuate?

Question 2: Performance degradation or neglibable gain for certain models: Do you have a good explanation of why SALSA works that well for Llama-3.1-8B while SALSA does experience performance regressions or negligible gains on certain model families? For instance, Mistral-7B-v0.1 commonsense reasoning results in Table 9 are showing that baseline metrics occasionally outperform the proposed method, despite utilizing identical distillation pipelines? Is the Self-Recall Error (SRE) based selection function inherently coupled to Llama's specific hidden state geometry? Specifically, it remains unclear to me whether the method is truly generalizable in practice, or if the reported gains on Llama-3.1-8B are an artifact of heavy hyperparameter tuning tailored specifically to Llama's architecture.

3 comments
AuthorsOpenReview ↗

Response to W1&Q1: Prefill efficiency and interactive use

Thank you for your careful analysis of efficiency and generalization. Addressing your core concerns, we have added a prefill chunk-size ablation, a sensitivity sweep over inference hyperparameters, an end-to-end re-analysis, and full RULER evaluations on two additional model families; all new data has been written into the revised manuscript (marked in blue). We respond point by point.

W1 & Q1: Prefill bottleneck and practicality for interactive workloads

Question summary: You point out that SALSA's prefill is about an order of magnitude slower than full attention, that TTFT appears impractical, and that the compute path lacks optimized kernels; you further ask how SALSA could be used in interactive agentic/chat workloads where prompt lengths fluctuate dynamically, given such TTFT.

Response: We answer on two levels. (1) The prefill gap is an implementation-level constant factor, not an inherent complexity deficiency: it measures ~8–10× and is roughly constant with length, and it can be narrowed to 7.3× (@8K, b8) via a declared implementation parameter (internal sparse chunk size) with no accuracy cost. We do not claim that the current implementation has eliminated the prefill gap; integrating dedicated fused kernels is explicit future work. (2) We do not claim uniform per-request speedups — FA2 is faster whenever it fits in memory. SALSA's value for decode-dominated interactive workloads lies in two measured dimensions: predictable decode latency (TPOT and memory are nearly invariant to context length) and serving capability under memory constraints (every workload where FA2 OOMs runs under SALSA; at 32K, aggregate decode throughput at each method's maximum feasible batch is ≈4.0×).

Details (end-to-end measurements: revised Appendix B.8 "End-to-End Serving Measurements", Table 18):

  1. Magnitude and nature of the gap: the prefill throughput gap is ~8–10× and roughly constant with length — this constancy indicates an engineering gap in the constant factor (linear state update O(Ld2)O(Ld^2) versus softmax attention O(L2d)O(L^2 d)), not an asymptotic deficiency that worsens with length.
  2. Internal chunk size (new experiment): prefill throughput increases monotonically with the internal chunk size and saturates around 512: at 8K b8, throughput 2943→3819 tok/s (+30%), narrowing the gap to FA2 from 9.5× to 7.3×; at 16K b8, from 8.2× to 6.3×; TTFT at 32K b8 drops from 90.2 s to 70.1 s (−22%). Accuracy was verified in parallel: chunk sizes 16–256 are all stable (S-NIAH-1@4K 88.8–90.4%), so larger chunks carry no quality cost. The full throughput × accuracy ablation table is in revised Appendix B.7 "Effect of Prefill Chunk Size" (Table 17).
  3. Feasibility of kernel integration: the computation graph consists entirely of public kernels (chunked linear attention → FLA/Triton; the softmax path → FlashAttention); no kernels need to be developed for any new operator — this is integration engineering, not novel kernel research.
  4. Decode-dominated workload shapes: agentic workloads = one prefill + hundreds to thousands of decode steps. SALSA's TPOT and memory are nearly length-invariant from 2K onward (b8: 55.1→55.8 ms from 2K→32K, only ×1.01; memory flat at 28.2 GiB), whereas FA2's TPOT grows 13.8→29.0 ms (×2.1) over the same range and its memory grows linearly to OOM — predictable decode latency has direct value for SLA-sensitive serving schedules (Appendix B.8, Table 18).
  5. Memory feasibility: every FA2-vanilla OOM setting (b8@32K, b16@16K/32K; both OOM at batch 32) runs under SALSA. Linear extrapolation of the maximum feasible batch at 16K: on an 80 GiB card, FA2 ≈31 vs. SALSA ≈39; on a 48 GiB card, 15 vs. 20. At 32K, aggregate decode throughput at each method's maximum feasible batch is 66 vs. 266 tok/s ≈ 4.0× (the accounting reported in §4.6; now stated explicitly in Appendix B.8).
  6. Counting FA2's chunked-prefill variant: even if FA2 uses chunked prefill (chunk 4096, trading time for memory), its TPOT at 32K b8 is only on par with SALSA's (46.3 vs. 55.8 ms), with 1.7× higher memory (48.0 vs. 28.2 GiB); 16K@32K remains infeasible for FA2 even with chunking (Appendix B.8, Table 18).
  7. Remaining gap: the prefill constant-factor gap remains; the narrowing paths are as above (tunable chunk size + public-kernel integration), and dedicated fused kernels are future work.
AuthorsOpenReview ↗

Response to W2&Q2: Cross-model-family generalization

W2 & Q2: Cross-model-family generalization

Question summary: You point out that SALSA shows fluctuations and regressions on Qwen/Mistral (including Mistral commonsense reasoning), and question their cause: is the method overfit to Llama, is SRE coupled to Llama geometry, and are the gains artifacts of tuning.

Response: The three model families share the completely identical distillation recipe and inference configuration with zero per-family tuning — if the recipe were overfit to Llama, this zero-tuning cross-family transfer could not hold. SRE is architecturally decoupled from Llama geometry (ϕ\phi is distilled for each model, and SRE is computed within each model's own feature space). We honestly acknowledge that SALSA performs differently across families (Mistral collapses @2K; Qwen aggregation tasks unimproved), but the specific pattern is incompatible with "the recipe is overfit to Llama": Qwen's retrieval tasks all exceed Llama's (+2.0 to +12.4), and MMLU gains are large and consistent across all three families (vs. LoLA +8.6/+6.2/+7.2) — an overfit-to-Llama recipe cannot explain this pattern.

Details:

  1. Zero per-family tuning: the three families share the identical distillation recipe and inference configuration (window 64 / sink 4 / budget-2048 pyramid / the same PyramidKV slope formula), with no per-model tuning of any kind.
  2. SRE is decoupled from Llama geometry: ϕ\phi is distilled for each model and SRE is computed in each model's own feature space; the mechanism is a query-agnostic, generic design. If SRE were coupled to Llama geometry, it could not explain the large MMLU gains on Qwen/Mistral (vs. LoLA +6.2/+7.2) or Llama's Avg surpassing base (72.89 vs. 72.37).
  3. Comparison with LoLA (same pipeline, same inference cache): MMLU rises across all three families: Llama +8.6 (63.62 vs. 55.0, close to base 65.97), Qwen +6.2 (67.48 vs. 61.28), Mistral +7.2 (55.66 vs. 48.48) (Appendix B.1, Tables 10/11); Mistral Avg: SALSA 71.86 > LoLCATs 70.81 > LoLA 70.69; on Mistral S-NIAH, SALSA leads S-NIAH-1/2 by large margins (0.5K: 98.8/100 vs. LoLA 72/100; 1K: 96.2/83 vs. 51/9.2) (main text §4.2, Tables 3/4).
  4. New evaluation: full Qwen RULER @2K (9 tasks × 4 models, full 500, protocol identical to the paper): SALSA's mean 49.0 is the best among converted models (LoLA 31.2 / LoLCATs 18.9; base 95.8); retrieval is close to base (MK1 99.4 / MV 96.1 / MQ 97.6 / VT 81.7, base 100), and SALSA is the only non-zero conversion on MK2 (7.0). The full 9×4 table is in revised Appendix B.1 (Table 12, marked blue).
  5. Source of the Qwen–Llama mean gap: the mean gap (−21.2) between Qwen SALSA (49.0) and Llama SALSA (70.23) is contributed almost entirely by MK2/CWE/FWE (−51.4/−86.8/−65.9, partially offset by +27 summed over the four retrieval tasks). On these three tasks, Qwen's base is near-perfect (CWE/FWE 98.2/87.1) while all three conversions collapse — a family-wide conversion difficulty, not SALSA-specific degradation. On the four retrieval tasks, zero-tuned Qwen SALSA exceeds Llama SALSA across the board (MK1 +7.8 / MQ +4.8 / MV +2.0 / VT +12.4).
  6. Mistral RULER tests and family differences: we ran limited RULER tests on Mistral; the results are indeed poor, with all three converted methods in the failure regime. We honestly acknowledge this as a family limitation: this boundary is already reported in the paper (the Mistral boundary table in main text §4.2, Table 4, S-NIAH-1 @2K = 0), and this round's limited tests independently confirm it. The regression on S-NIAH-3 @0.5K (43.4 vs. LoLA 90.8, Table 4) is likewise honestly acknowledged — it is the hardest distractor-needle variant, while SALSA leads on every cell of S-NIAH-1/2. But "performs differently across families" does not equal "tuned for Llama": (a) zero per-family tuning (point 1); (b) the Qwen mean gap concentrates on the three tasks where all conversions collapse, while the four retrieval tasks reverse in SALSA's favor (points 4–5); (c) gains over the alignment target are consistent across all three families (point 3).
  7. PiQA/WinoGrande fluctuations: converted models surpassing base by ±1pt on short-context tasks is a known LoLCATs phenomenon (Mistral PiQA: LoLCATs 81.50 > base 80.58, Appendix B.1, Table 11); SALSA does not fall behind its alignment target LoLA (Qwen PiQA 74.21 vs. 73.61, WinoGrande 65.59 vs. 65.27, Table 10). ±1pt is noise and does not constitute evidence of regression.
AuthorsOpenReview ↗

Response to W3: Inference hyperparameter sensitivity

W3: Inference hyperparameters are numerous, fragile, and hard to tune

Question summary: You question whether SALSA's inference-side hyperparameters are too numerous and potentially fragile and hard to tune.

Response: In new experiments we fix the trained weights and sweep four groups of inference hyperparameters one at a time (full 500): two groups are nearly flat and two are monotonic and predictable — no hidden fragility exists. The training-side hyperparameters are identical to LoLCATs; deployment only requires avoiding the minimum settings of two explicit capacity parameters, with no per-model tuning.

Details (each point: 2 tasks × full 500; the full sweep table is in revised Appendix B.6 "Sensitivity to Inference Hyperparameters", Table 16, marked blue; the default row is repeated within each group for reference):

KnobRangeResult (S-NIAH-1@4K / S-NIAH-2@1K)Character
Number of sinks1→1687.6–90.0 / 67.4–77.0Flat, no cliff
Sparse chunk16→25688.8–90.4 / 70.2–74.4Flat (implementation-detail level)
Sparse cache size (per layer)16→256 (total budget 512→8192)24.0→98.0 / 8.0→100Monotonic capacity knob; predictable accuracy-for-capacity trade
Sliding window16→2567.8→96.6 / 56.4→92.0Most sensitive, but an explicit capacity parameter (hard lower bound)

Of the four groups, two are nearly flat and two are monotonic and predictable; the hidden sensitivity implied by "fragile and hard to tune" does not exist. The training side uses LoLCATs' default hyperparameters untouched; the budget slope is the PyramidKV formula, not a free parameter obtained by search; the hard lower bound of the window (16 collapses, 32 already recovers to 84.2) is explicitly given in the sweep — deployment only needs to avoid the minimum end of the capacity parameters, with no per-model search.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.