- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes IndexServe, a shared service that separates index storage and token selection from sparse-attention decoders to enable larger batches. Graph-compatible communication, paged allocation, and length-aware batching support efficient remote selection. Experiments on two models report higher resource-normalized decoding throughput at the cost of increased latency.
Critical strengths
- The paper addresses a practical limitation of sparse-attention serving. KV cache offloading leaves a growing index on the decoder, which can still restrict batching and throughput.
- A weight-free indexer service is a useful architectural choice. It allows index capacity and selection work to be shared across decoders. The ablation shows that graph execution, communication, and memory management are important for making this separation effective.
Critical weaknesses
- The performance comparison remains incomplete. Memory-partition sensitivity is reported only for vLLM, not ECHO or HiSparse. The headline gains also compare peak-throughput points at different latencies, leaving the advantage under tuned baseline partitions and a common mean ITL budget unclear. 2.Selection equivalence is not directly validated. The paper reports no measured comparison of selected token IDs or model quality, leaving implementation-level preservation of the deployed selection rule unverified.
Questions for authors
- How do the gains change when ECHO and HiSparse’s memory partitions are varied, and is CPU-socket striping applied consistently across systems?
- Under identical inputs and numerical formats, do remote and colocated execution produce equivalent selected token IDs and decoder outputs during both page-boundary appends and batch changes?
- How does IndexServe’s resource-normalized throughput compare with the baselines under the same mean ITL limit?