- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes LatentIndex to reduce indexing overhead in sparse attention while preserving layer-specific token selection.
- Method: Shares compressed indexer key representations across layers, while allowing each layer to independently select relevant tokens. A hierarchical selection variant further reduces computation.
- Results: Reduces logical indexer-cache storage by 61.1%, improves attention recall by up to 3.28 percentage points over IndexCache, and achieves 2.30–2.72× indexer-only decoding speedups over native DSA.
- Overall: A promising architectural improvement, but stronger benchmarks and fair, end-to-end efficiency comparisons are needed to establish its practical value.
Critical strengths
1、Effective cross-layer sharing: LatentIndex shares compressed key representations across layers while preserving layer-specific token selection. Experiments show consistently higher attention recall, especially at longer context lengths.
2、Practical and efficient design: The method supports training-free adaptation through closed-form regression. Query-side decoder absorption avoids reconstructing historical keys. The appendix provides clear mathematical derivations and implementation details, including RoPE compatibility and FP8 execution, making the approach technically sound and relatively easy to reproduce.
Critical weaknesses
1、Limited evaluation on challenging tasks: The experiments mainly rely on RULER and LongBench, which are relatively old benchmarks. RULER scores are already close to saturation, making it difficult to measure meaningful improvements. The paper lacks evaluations on more challenging, uncontaminated long-context tasks. Moreover, attention recall is measured only at the first token of the question, which may not reflect token selection quality during full generation.
2、Insufficient baseline comparison and efficiency evaluation: The paper compares LatentIndex with IndexCache using fixed layer groups, without considering IndexCache's optimized grouping strategy. This may underestimate the baseline's performance. Furthermore, the reported 2.30–2.72× speedups apply only to the indexer, not end-to-end inference. LatentIndex with hierarchical selection is also slower than IndexCache at the indexer level. The paper does not establish whether LatentIndex provides a better quality–latency trade-off under equal computational or memory budgets.
Questions for authors
Does the method still maintain performance comparable to native DSA on more challenging long-context reasoning and in-context learning tasks?