acceptodds
Withdrawn from ICLR 2027

LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers reduces this overhead but constrains multiple layers to the same token set. We introduce LatentIndex, which extends the latent-sharing principle of Multi-head Latent Attention across indexer layers. Each layer group constructs a shared latent cache from its first layer’s hidden states, while layer-specific scoring enables independent token selection. Absorbing key decoders into queries enables direct scoring of the shared cache without reconstructing historical per-layer keys. We develop training-free calibration and investigate a training-aware instantiation of this principle. To balance quality and computation, a hierarchical selection (HS) variant lets followers independently refine a shared candidate set proposed by the anchor. With four-layer sharing, LatentIndex reduces logical indexer-cache storage by 61.1% on DeepSeek-V3.2. Across DeepSeek-V3.2 and GLM-5, training-free LatentIndex improves head-wise attention-mass recall over IndexCache by up to 3.28 percentage points while maintaining RULER and LongBench performance close to native DSA. HS further achieves 2.30–2.72× decode indexer speedups over DSA across 8K–128K contexts, retaining most of LatentIndex’s recall. LatentIndex offers a new perspective on cross-layer indexing: sharing continuous representations rather than discrete selections enables efficient reuse while preserving layer-specific token selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse attention, Efficient indexer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7A15OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LatentIndex to reduce indexing overhead in sparse attention while preserving layer-specific token selection.

  1. Method: Shares compressed indexer key representations across layers, while allowing each layer to independently select relevant tokens. A hierarchical selection variant further reduces computation.
  2. Results: Reduces logical indexer-cache storage by 61.1%, improves attention recall by up to 3.28 percentage points over IndexCache, and achieves 2.30–2.72× indexer-only decoding speedups over native DSA.
  3. Overall: A promising architectural improvement, but stronger benchmarks and fair, end-to-end efficiency comparisons are needed to establish its practical value.
Critical strengths

1、Effective cross-layer sharing: LatentIndex shares compressed key representations across layers while preserving layer-specific token selection. Experiments show consistently higher attention recall, especially at longer context lengths.

2、Practical and efficient design: The method supports training-free adaptation through closed-form regression. Query-side decoder absorption avoids reconstructing historical keys. The appendix provides clear mathematical derivations and implementation details, including RoPE compatibility and FP8 execution, making the approach technically sound and relatively easy to reproduce.

Critical weaknesses

1、Limited evaluation on challenging tasks: The experiments mainly rely on RULER and LongBench, which are relatively old benchmarks. RULER scores are already close to saturation, making it difficult to measure meaningful improvements. The paper lacks evaluations on more challenging, uncontaminated long-context tasks. Moreover, attention recall is measured only at the first token of the question, which may not reflect token selection quality during full generation.

2、Insufficient baseline comparison and efficiency evaluation: The paper compares LatentIndex with IndexCache using fixed layer groups, without considering IndexCache's optimized grouping strategy. This may underestimate the baseline's performance. Furthermore, the reported 2.30–2.72× speedups apply only to the indexer, not end-to-end inference. LatentIndex with hierarchical selection is also slower than IndexCache at the indexer level. The paper does not establish whether LatentIndex provides a better quality–latency trade-off under equal computational or memory budgets.

Questions for authors

Does the method still maintain performance comparable to native DSA on more challenging long-context reasoning and in-context learning tasks?

Reviewer_8MQHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LatentIndex reduces the storage overhead of sparse-attention indexers by sharing a latent key cache across consecutive layers while preserving layer-specific scoring and token selection. Unlike IndexCache, which reuses selected indices, each layer independently queries the shared representation through its own decoder, absorbed into the query to avoid reconstructing historical keys. The paper develops training-free calibration via reduced-rank regression, explores a training-aware variant, and introduces hierarchical selection (HS) for layer-specific refinement within anchor-proposed candidates. Experiments on DeepSeek-V3.2 and GLM-5 show better attention-mass recall than IndexCache and downstream performance close to native DSA. Four-layer sharing reduces logical indexer-cache storage by 61.1% on DeepSeek-V3.2; adding HS achieves 2.30–2.72× decode indexer speedups. DeepSeek-V2-Lite training experiments provide complementary evidence of improved selection quality and LM loss close to DSA.

Critical strengths
  1. A principled separation of representation sharing and routing decisions. The method exploits cross-layer redundancy without forcing layers to attend to identical token sets. Query-side decoder absorption makes this separation operational without materializing historical per-layer keys. This is consequential because it addresses a central limitation of direct index reuse: at 128K on DeepSeek-V3.2, average follower recall loss relative to native DSA decreases from 6.69 to 1.81 percentage points.

  2. Practical adaptation supported by downstream validation. The training-free approach applies to existing large sparse-attention models without updating pretrained weights. Evaluation on both DeepSeek-V3.2 and GLM-5 combines controlled attention-mass diagnostics with end-to-end RULER and LongBench results, showing that the storage reduction preserves downstream quality. HS additionally demonstrates a useful quality–latency trade-off, providing evidence that the approach can support efficient indexing while retaining layer-specific refinement.

Critical weaknesses
  1. The motivation for compressing the indexer cache is not clear. DSA already uses a single shared key head with a small dimension and low-precision fp4/8. The reported 61.1% reduction concerns only the indexer cache, so it is unclear how much this saves in total serving memory or improves serving capacity.

  2. The efficiency benefit of latent sharing itself remains unclear. Decoder absorption increases the scoring dimension, which could make indexer slower. Since the reported speedups include hierarchical selection (HS), the current results do not establish whether latent sharing improves runtime or whether the gains mainly come from searching fewer candidates.

Questions for authors
  1. What is the main practical bottleneck that indexer-key compression addresses? Given that DSA already uses a single shared key head and low-precision storage, what fraction of the total cache does the indexer cache occupy in your actual implementation? How much total memory does LatentIndex save at representative context lengths and batch sizes?

  2. Is the indexer compute-bound or memory-bandwidth-bound in your tested settings? If it is compute-bound, wouldn’t the larger scoring dimension after decoder absorption make LatentIndex slower without HS? Could you provide profiling results and a latency comparison with native DSA to clarify this?

  3. What is the latency of LatentIndex without HS, compared with native DSA? The scoring dimension increases from 128 to 192 on DeepSeek-V3.2 and to 256 on GLM-5. Does this increase computation and make indexing slower despite the smaller overall cache?

  4. How does LatentIndex compare with directly sharing the anchor’s indexer keys across layers while keeping each layer’s own queries and Top-K selection? Also, how much of the speedup can HS achieve using native per-layer keys, without latent compression? These comparisons would help establish what the latent representation adds.

Reviewer_f1fMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LatentIndex, a cross-layer sharing mechanism for sparse attention indexers. Unlike IndexCache, which directly reuses selected token indices across layers, LatentIndex shares a compressed latent key representation while allowing each layer to perform its own token selection. The authors introduce an offline calibration procedure for training-free adaptation and a training-aware implementation. They also propose Hierarchical Selection (HS), where an anchor layer generates a shared candidate set and subsequent layers independently refine their selections.

Critical strengths
  1. The paper evaluates both training-free and training-aware settings
  2. The writing is clear and is easy to follow.
Critical weaknesses
  1. Limited novelty. The overall motivation is related to IndexCache, and the calibration method is also mentioned as in a previous KV sharing paper HShare[1] (oh maybe you should also cite them). To clarify, I am not suggesting that limited novelty alone should preclude acceptance. My main concern is that the experimental results are not sufficiently compelling, which is weakness 2/3.

  2. Missing ablation on the source of improvement. The paper lacks an IndexCache + Top-B Reranking baseline. The gains of Hierarchical Selection may primarily come from allowing follower layers to independently rerank a larger candidate set, rather than from the proposed latent representation sharing. Without this comparison, it is difficult to isolate the contribution of the core proposed method.

  3. Insufficient efficiency evaluation under realistic serving workloads. The authors explicitly acknowledge that their efficiency evaluation is limited to batch size one. I consider this a significant limitation, since practical LLM serving systems often operate with multiple concurrent requests and substantially larger effective batch sizes.

[1] Wu H, Li L, Huang H, et al. Hshare: Fast llm decoding by hierarchical key-value sharing[C]//International Conference on Learning Representations. 2025

Questions for authors

Q1. In Table 4, IndexCache achieves 13.40 ms at 128K, compared to 20.48 ms for LatentIndex + HS. Would LatentIndex still outperform IndexCache under matched computational budgets, for example, by reducing IndexCache's group size?

Q2. Why do the training-aware and training-free implementations use different designs? Specifically, the training-aware version generates all queries and head weights from the anchor hidden states, while the training-free version retains layer-specific query inputs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.