- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SALSA, a training-inference-consistent sparse caching framework for converting pretrained full-attention LLMs into hybrid linear-attention models.
The key motivation is to address the training-inference mismatch in previous methods such as LoLA, where the sparse cache is introduced only during inference. SALSA incorporates the sparse cache into both attention distillation and fine-tuning, allowing the model to learn the same attention mechanism used during deployment. The authors further introduce sink token preservation and offline layer-wise cache budget allocation to improve model performance.
Experiments on Llama-3.1-8B, Qwen3-8B, and Mistral-7B demonstrate substantial improvements over existing linearization methods, particularly on associative retrieval tasks. SALSA also achieves higher decode throughput under memory-constrained long-context settings, although its prefill efficiency remains limited.
Critical strengths
-
Meaningful motivation. Training-inference consistency is an important issue in model conversion. Incorporating the sparse cache into the training process, rather than introducing it only at inference time, is a reasonable and well-motivated approach.
-
Significant improvements over existing linearization methods. SALSA substantially outperforms LoLCATs and LoLA on several retrieval tasks. The ablation studies also provide evidence that including the sparse cache during training can improve model performance.
-
Practical memory advantages. By combining linear attention with a fixed-capacity sparse cache and sliding window, SALSA maintains nearly constant decode memory as context length increases, making it potentially useful for memory-constrained inference.
Critical weaknesses
-
Limited long-context capability. Despite the improvements over previous linearization methods, SALSA still suffers from severe performance degradation as context length increases. For example, on Llama-3.1-8B, S-NIAH-1 accuracy drops from 90.2% at 4K to only 0.8% at 8K, while the full-attention baseline maintains 100% accuracy. Even with a larger inference cache, the accuracy at 8K is only 40.4% and drops to 0% at 16K. These results raise concerns about the practical effectiveness of SALSA for long-context inference, which is a central motivation of the paper.
-
Limited novelty and unclear contribution attribution. Several components of SALSA, including SRE-based sparse caching, sink token preservation, and pyramid-shaped cache budget allocation, are adapted from existing work. The main novelty lies in incorporating sparse caching into the model conversion training process. However, the ablation results show that sink token preservation contributes substantially to the overall performance improvement. For instance, adding sink tokens improves S-NIAH-1 accuracy at 2K from 42.2% to 97.2% under the training-consistent framework. While the effectiveness of training consistency is supported by the experiments, the large contribution of an existing technique makes the independent significance of the proposed innovation less compelling.
-
Limited end-to-end inference efficiency. Although SALSA achieves promising decode throughput under memory constraints, its prefill throughput is significantly lower than FlashAttention-2. At 32K context length, FA2 achieves approximately 8.3× higher prefill throughput. Since prefill is an important component of long-context inference, the lack of a comprehensive end-to-end latency and throughput comparison makes it difficult to assess the overall practical benefits of SALSA.
Questions for authors
-
SALSA demonstrates significant improvements over LoLA, but its retrieval performance still degrades sharply at longer context lengths. What are the main causes of this degradation? Is it primarily due to the limited capacity of the sparse cache, or the representational limitations of the linear attention state? Could the authors provide more comprehensive quality evaluations at 8K, 16K, and 32K?
-
The ablation studies indicate that sink token preservation contributes substantially to the final performance. Could the authors further clarify the independent contribution of training-inference consistency, particularly its interaction with sink token preservation?
-
Could the authors provide end-to-end inference latency or throughput results that include both prefill and decoding? This would help establish whether the decode-stage benefits are sufficient to compensate for the significantly slower prefill stage under realistic workloads.
4 comments
Supplement to Response to W1 & Q1
-
The operating range is calibrated: ≤4K is the standard operating region (accuracy close to full attention, constant memory); 8K can be recovered to 94.4% by enlarging the inference cache, or to 40.6% by shifting the training side; 16K is the current hard boundary — the mechanism analysis and boundary statement are in the revision (Appendix B.5 Table 15 + B.4 Table 14).
-
Within the calibrated region SALSA remains the best conversion: the combination of constant memory ≤4K and near-full-attention accuracy has no peer among LoLCATs/LoLA/HALO/LAWCAT — RULER @2K at matched budget: SALSA 70.23 vs. LoLA 22.44 / LoLCATs-516 5.81 (base 95, paper Table 2); S-NIAH @4K: SALSA 90.2 vs. LoLA 1 / LoLCATs 0.8 (paper Table 1).
Response to W2 & Q2: Novelty and component attribution
W2 & Q2: Limited novelty (sink is an existing technique yet presented as the largest contributor); the independent contribution of training consistency and its interaction with sink
Question summary: You question the novelty of this work — sink tokens are an existing technique (StreamingLLM), yet are presented as the largest source of contribution — and ask for an attribution of the independent contribution of training consistency and its interaction with sink.
Response: The attribution experiments directly answer both questions: how much sink itself contributes, and what training consistency independently adds. Sink alone (cache not in the training loop) reaches only 16.6% @4K, far below the full method's 91.2% (same inference configuration) — sink provides no retrieval capability; its contribution is making the three-path mechanism trainable; the independent gain from bringing the cache into the training loop is the 2×2 below. The novelty of this work therefore lies not in sink itself, but in three reusable principles and the adaptation design.
Details:
-
Mechanism-consistency principle: the cache mechanism must participate in the training loop to be effective — the full 2×2: sink alone 16.6%, cache alone 2.4%, full combination 90.2% (@4K; paper Tables 6/7 with the revised §4.4 Table 8), either component alone near-useless. This is a general constraint on the conversion paradigm, not specific to this architecture. The attribution ablation and the new baseline row are in revised main text §4.4 (Table 8, marked blue).
-
Capacity–competition principle: small-cache training > large-cache training (70.23 vs. 56.76) — a counterintuitive, reusable empirical rule. Paper Appendix B.3 (Table 13).
-
Sink's three-path accounting adaptation: permanently retained during training and never entering the cache/linear state — different from StreamingLLM's inference-time usage; this is the design component that makes intra-layer three-path mixing trainable.
Response to W3 & Q3: End-to-end efficiency measurements
W3 & Q3: End-to-end results including prefill + decode
Question summary: You ask for complete end-to-end efficiency results covering both prefill and decode, rather than component-level measurements only.
Response: We provide the end-to-end table fully measured on H800 80 GiB / bf16. Conclusions: SALSA's TPOT and memory are nearly independent of length from 2K onward; every workload where FA2-vanilla OOMs runs under SALSA; even when FA2 uses chunked prefill to trade time for memory, its TPOT @32K is only on par with SALSA's, with 1.7× higher memory.
Details (the table is in revised Appendix B.8 "End-to-End Serving Measurements", Table 18, marked blue):
| prompt × batch | FA2 TTFT | SALSA TTFT | FA2 TPOT | SALSA TPOT | FA2 memory | SALSA memory |
|---|---|---|---|---|---|---|
| 2K × 8 | 0.50 s | 5.27 s | 13.8 ms | 55.1 ms | 17.5 GiB | 28.2 GiB |
| 8K × 8 | 2.35 s | 22.2 s | 20.3 ms | 55.5 ms | 23.6 GiB | 28.2 GiB |
| 16K × 8 | 5.49 s | 44.7 s | 29.0 ms | 55.8 ms | 31.8 GiB | 28.2 GiB |
| 16K × 16 | OOM (11.2 s) | 88.4 s | OOM (46.8 ms) | 60.4 ms | OOM (48.5 GiB) | 41.3 GiB |
| 32K × 8 | OOM(14.6 s) | 89.8 s | OOM (46.3 ms) | 55.8 ms | OOM (48.0 GiB) | 28.2 GiB |
| 32K × 16 | OOM | 177.5 s | OOM | 60.3 ms | OOM | 41.3 GiB |
Note: values in parentheses are measured with FA2 chunked prefill (chunk 4096, trading time for memory); at batch 32 both methods OOM (not listed); 32K×16 is feasible only for SALSA.
-
Feasibility boundary: every FA2-vanilla OOM setting (b8@32K, b16@16K/32K) runs under SALSA; linear extrapolation of the maximum batch @16K: 80 GiB card FA2 ≈31 vs. SALSA ≈39; 48 GiB card 15 vs. 20.
-
TPOT/memory flatness: SALSA TPOT goes only 55.1→55.8 ms from 2K→32K (×1.01), with memory constant at 28.2 GiB; FA2 TPOT grows 13.8→29.0 ms (×2.1) with memory growing linearly to OOM — predictable decode latency has direct value for serving schedules.
-
The "4.0×" accounting: aggregated decode throughput at each method's maximum feasible batch @32K is 66 vs. 266 tok/s ≈ 4.0×, consistent with paper §4.6; the accounting is now stated explicitly in revised Appendix B.8.
-
Counting FA2's chunked-prefill variant: @32K b8 its TPOT is 46.3 ms, on par with SALSA's 55.8 ms (ratio 1.21), with memory 48.0 vs. 28.2 GiB (1.7×).
-
Applicability: we do not claim per-request superiority (FA2 is faster within its feasible region; the prefill gap is a constant factor, with the narrowing path in Appendix B.7, Table 17); SALSA's end-to-end value is serving capability in memory-constrained regimes plus length-independent decode cost.
Response to W1 & Q1: Long-context degradation and its cause
Thank you for your specific, testable questions. Addressing your concerns, we have added the degradation-cause diagnosis, the enlarged-inference-cache and distillation-length-right-shift experiments, and the end-to-end efficiency table; all new data is in the revised manuscript (marked blue). We respond point by point.
W1 & Q1: Limited long context (90.2% @4K → 0.8% @8K) — is the main cause cache capacity or the linear-state limit?
Question summary: You point out that SALSA's long-context capability is limited — after 90.2% @4K, accuracy drops to 0.8% @8K — and ask us to confront this directly; you further ask for a causal attribution of the 8K+ degradation: insufficient cache capacity, or a representation/readout limit of the linear state.
Response: We confront the numbers directly. The main cause is the representation/readout limit of the linear state (constrained by the distillation length), with cache capacity only a secondary amplifier — established by a three-step evidence chain: enlarging the inference cache rules out the capacity hypothesis → needle-retention diagnosis localizes the failing component → distillation-length right-shift validates the root cause. The operating boundary shifts right with the training side (8K recovers to 40.6%); within the calibrated operating region (≤4K) SALSA remains the best conversion; 16K is the current hard boundary. The distillation-length right-shift experiment was in fact added in direct response to your question about the cause of the degradation.
Details:
Capacity is indeed an effective knob at 8K (threshold-style recovery to 94.4%); but a setting with identical coverage still yields 0% @16K — the 16K failure is not insufficient cache share. Table and boundary statement in revised Appendix B.4 (Table 14, marked blue).
8K: the selection mechanism has not failed — the needle remains on the visible path most of the time (mean coverage only drops from 0.35 to 0.26), and there is no intermediate state of "visible but read out wrong"; generation degrades globally into repetition loops — the failure is in linear-state integration/readout. 16K: the selection mechanism also begins to degrade — coverage drops further to 0.16, fully-lost layers rise to 20.9%, with a depth gradient (needle coverage near 0 in deep layers); both information channels fail simultaneously. Diagnosis protocol and full metric table in revised Appendix B.5 "Diagnosing Accuracy Degradation at Longer Contexts" (Table 15; the four metric definitions are written into the main text, marked blue).
Step 3: root-cause validation (distillation-length right-shift). The failure is consistent with the training distribution: ϕ is distilled only on 1024-length sequences and extrapolates beyond. Doubling the distillation length to 2048 (data, total tokens, and inference configuration all unchanged) recovers 8K from 1.0% to 40.6%, with no degradation ≤2K (98% @4K) — the operating boundary is set by the distillation supervision distance and can be shifted right on the training side. See revised main text §4.5 (Table 9).
Future directions: larger linear-state capacity (e.g., a larger feature dimension) and gated/decay-type linear attention are proposed here as future work; the combination experiment of training length and inference cache is ongoing, and results will be supplemented during the discussion period.