acceptodds
Withdrawn from ICLR 2027

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical efficiency-quality trade-off. Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top- routing over the full cache remains expensive. In this work, we propose cross-layer sparse attention (CLSA), which is built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes token-level top- selection once and reuses the resulting index across layers, thereby preserving the fine-grained selectivity of token sparse attention while amortizing the routing overhead. The resulting architecture improves all major inference bottlenecks jointly, including pre-filling, KV-cache storage, and long-context decoding. Experiments across short-context and long-context benchmarks show that is both accurate and efficient, achieving up to decoding speedup and overall throughput improvement at 128K context. These results suggest a more complete architectural solution for long-context LLMs that jointly advances model quality and inference efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Attention, Cross Attention, Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_52nVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Cross-Layer Sparse Attention (CLSA), which computes top-k routing once and shares the resulting indices across layers in the YOCO architecture, avoiding repeated per-layer routing computation. The indexer is adapted to the model through a two-stage training procedure. The method reduces the overhead of proxy score computation while preserving the architectural advantages of YOCO. Experimental results demonstrate substantial speedups with limited performance degradation.

Critical strengths
  1. The paper explores a novel and timely research setting. Given the growing interest in the YOCO architecture, improving its efficiency is a meaningful contribution.
Critical weaknesses
  1. The methodological novelty appears limited. Cross-layer reuse of routing indices has already been explored in Transformer-based sparse attention methods, such as IndexCache and HySparse. The paper primarily adapts this idea to the YOCO architecture, without sufficiently establishing the additional technical novelty beyond this architectural adaptation.

  2. The two-stage training procedure follows a conventional warmup-and-joint-adaptation recipe used in prior architectures. Moreover, the experiments do not isolate the contributions of S1 and S2, leaving the necessity of indexer warmup and the incremental benefit of joint adaptation unclear.

Questions for authors
  1. What additional technical novelty does CLSA introduce beyond adapting existing cross-layer routing reuse to YOCO? How does sharing one routing index across all cross-decoder layers compare with sharing an index every nn layers in terms of model quality and inference efficiency?

  2. Could the authors provide ablation studies to clarify the individual contributions of the S1 and S2 training stages?

Reviewer_frPjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Cross-Layer Sparse Attention (CLSA) for YOCO-style decoder–decoder architectures, in which all cross-decoder layers read a single KV cache produced by a sliding-window self-decoder. CLSA adds a lightweight single-head indexer that works on the self-decoder's output hidden states. The indexer computes token-level top-k routing once per query, and all 16 cross-decoder layers reuse that routing index. The indexer is trained by distillation: its routing distribution is matched (via KL) to the attention distribution of the dense cross-decoder, averaged over all layers and heads. Training has two stages: an indexer warmup with the backbone frozen, then joint adaptation with the language-modeling loss plus the KD loss.

Critical strengths

None

Critical weaknesses
  1. The authors claim "Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive." I'm wondering if the authors have ever checked the performance of block-wise sparse attention methods on long-context benchmarks such as training-free ones [1,2,3,4] or train ones such as [5,6] ? These methods are not only achieving accuracy almost the same as full attention, but also outperforming the full attention on some cases, which also makes me question what is the benefit of the proposed method while other classes are training-free while the proposed method needs training.
  2. Following up on (1), since the authors mention multiple places, I believe they should be aware how suitable these methods are as baselines, yet none are featured in experiment section. Moreover, the benchmarks featured in experiments section are very limited. I neither see traditional long-context benchmarks such as LongBench, nor newer one like agentic coding such as SWEBench, Tau-Bench, etc.
  3. I also believe CLSA essentially combines a DSA-style indexer with YOCO's shared KV cache and extends cross-layer index reuse, which IndexCache already does for groups of layers, to the full cross-decoder. HySparse already combines KV-cache sharing with token-level selection. The paper's discussion of related work does not clearly distinguish CLSA from it.

[1] Sparser Block-Sparse Attention via Token Permutation, ICML 2026

[2] Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inference, ICML 2026

[3] RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression, ICML 2025

[4] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML 2024

[5] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, ACL 2025

[6] DeepSeek Sparse Attention (DSA)

Questions for authors
  1. Can we compare with sparse attention methods?
  2. Can we feature more long-context benchmarks?
  3. The central ablation, the quality cost of sharing, is projected rather than measured. Table 14 projects latency for sharing granularities of 1, 2, 4, 8, and 16 layers, but the authors state they "did not train or benchmark separate grouped-sharing models". So the paper never shows how quality changes with sharing granularity. Can we actually have real ablation studies?
Reviewer_XrC4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Based on the YOCO architecture, this paper proposes CLSA, which employs a single-head indexer and multi-layer distillation to compute and cross-layer reuse top-kk routing indices. Its effectiveness in accelerating long-context inference while maintaining accuracy is validated on a 4B model.

Critical strengths
  • Addresses the top-kk latency bottleneck by distributing routing overhead across layers.
  • Employs a streamlined two-stage distillation process, maintaining high accuracy for both short and long texts.
Critical weaknesses
  • A hybrid of YOCO and sparse routing; limited novelty.
  • Coupled with non-mainstream YOCO, maybe making it difficult to migrate to mainstream architectures; impact is somehow constrained.
  • No code provided; lacks indexing dimensions and operator-level details. Performance drops sharply during 128K multi-key retrieval.
  • Tested only on a 4B model; ablation studies on granularity are based on theoretical derivation rather than empirical testing.
Questions for authors
  1. What are the indexer dimension (didxd_{idx}), positional encoding, and inference batch size? Is the code open-sourced?
  2. What is the reason for the sharp drop in performance for Multi-key-3 in Table 13?
  3. Could you provide actual training and measured latency data for different sharing granularities, rather than relying on the arithmetic estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.