acceptodds
Under review as a conference paper at ICLR 2027

IndexServe: Indexer as a Service for Efficient Sparse LLM Serving

Abstract

Sparse attention reduces long-context decoding costs, but the decoder-resident index still grows with context length, limiting batch sizes even after key–value (KV) cache offloading. We present INDEXSERVE, a shared indexer service that moves index storage and token selection out of decoders while preserving the model’s selection rule. The service retains historical indexes and exchanges only current-step inputs and selected token IDs during decoding. A graph-compatible runtime and direct device-to-device transfers over a superpod fabric support per- layer remote selection, while PagedIndex and length-aware batching pool index memory and selection work across decoders. INDEXSERVE is independent of KV offloading but complements it to enable larger decode batches. Evaluated with KV offloading on DeepSeek-V3.2 and GLM-5.1, INDEXSERVE achieves 2.46–8.32× the peak generation throughput of ECHO, the strongest evaluated baseline, after accounting for service accelerators, at the cost of higher inter-token latency

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving, Sparse Attention, AI Infra, Serving Engine

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_piMaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IndexServe, a shared service that separates index storage and token selection from sparse-attention decoders to enable larger batches. Graph-compatible communication, paged allocation, and length-aware batching support efficient remote selection. Experiments on two models report higher resource-normalized decoding throughput at the cost of increased latency.

Critical strengths
  1. The paper addresses a practical limitation of sparse-attention serving. KV cache offloading leaves a growing index on the decoder, which can still restrict batching and throughput.
  2. A weight-free indexer service is a useful architectural choice. It allows index capacity and selection work to be shared across decoders. The ablation shows that graph execution, communication, and memory management are important for making this separation effective.
Critical weaknesses
  1. The performance comparison remains incomplete. Memory-partition sensitivity is reported only for vLLM, not ECHO or HiSparse. The headline gains also compare peak-throughput points at different latencies, leaving the advantage under tuned baseline partitions and a common mean ITL budget unclear. 2.Selection equivalence is not directly validated. The paper reports no measured comparison of selected token IDs or model quality, leaving implementation-level preservation of the deployed selection rule unverified.
Questions for authors
  1. How do the gains change when ECHO and HiSparse’s memory partitions are varied, and is CPU-socket striping applied consistently across systems?
  2. Under identical inputs and numerical formats, do remote and colocated execution produce equivalent selected token IDs and decoder outputs during both page-boundary appends and batch changes?
  3. How does IndexServe’s resource-normalized throughput compare with the baselines under the same mean ITL limit?
Reviewer_jnC5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IndexServe. It disaggregate the indexer service in Native Sparse Attention. Its key observation is that indexes grow with content length and become new HBM capacity bottleneck. To address the limitation, they design IndexServe to move index storage and selection out of decoders without changing the model’s selection rule.

Critical strengths
  1. The motivation stands. Batching matters especially in sparse attention to increase the arithmetic intensity, while index grows with the context and therefore limits decoding batch. The motivation section is good.
  2. The systematical design is solid. IndexServe integrates many techniques, such as graph-compatible execution, memory-semantic communication, PagedIndex, and dynamic batching, to support the disaggregation of Indexer.
Critical weaknesses
  1. The paper measures peak-throughput which is under different settings of latency and request rate. However, the throughput gains might comes at the cost of higher latency. It would be helpful to further evaluate the throughput under matched latency constraints or tail-latency SLOs.
  2. The fixed service pool becomes saturated when more decoder servers are added. Also, under some configurations, the HBM becomes the capacity bottleneck. It would be valuable if authors evaluate more systematically how the Indexer Service would scale with the number of decoders.
Questions for authors

see weakness.

Reviewer_7upZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In sparse attention-based LLM inference, while KV caches can be offloaded from HBM/NPU memory, the index grows linearly with context length and still occupies substantial HBM space, becoming a hidden bottleneck that limits concurrent batch sizes. Through profiling, the authors observe that even with sparse offloading in vLLM Ascend, when the context exceeds 120K tokens on DeepSeek-V3.2, each DP replica can only accommodate two concurrent requests. The authors propose INDEXSERVE, a shared indexer service architecture that decouples index storage and top-k selection computation from individual decoders and places them into an independent accelerator service pool. The system is evaluated on an Ascend A3 superpod using DeepSeek-V3.2 INT8 and GLM-5.1 INT4 models, compared against SGLang HiSparse and vLLM sparse offloading baselines. The proposed method performs better on these tasks.

Critical strengths

1 The decision to disaggregate the indexer into a shared service is architecturally clean. The graph-compatible runtime with fixed communication buffers avoids the per-layer allocation overhead that would negate the benefits of offloading.

2 The evaluation is thorough, covering two model families, three workload patterns, and multiple tensor/sequence parallelism layouts.

3 The PagedIndex mechanism demonstrates thoughtful systems engineering. By dividing index memory into fixed-size pages with logical-to-physical mapping, the design handles variable-length sequences without fragmentation while allowing multiple decoders to share the same index pool.

Critical weaknesses

1 The entire system is implemented and evaluated on Ascend A3 superpod hardware with its proprietary Unified Bus fabric. While Appendix provides detailed hardware descriptions, the paper does not discuss how the design would port to NVIDIA-based clusters or other accelerator ecosystems. The graph replay mechanism and device-side synchronization primitives are likely vendor-specific, and the absence of a discussion on portability costs weakens the broader impact claim.

2 The paper treats the indexer service as a dedicated resource, but in a real serving cluster, multiple models or tenants may compete for the same indexer pool. The paper does not discuss admission control policies, priority scheduling across sessions, or isolation guarantees. Figure 10 shows that Workload C is capacity-unavailable for vLLM but INDEXSERVE handles it. However, the scaling limit of adding more service NPUs is not measured. A discussion of multi-tenant resource sharing would strengthen the practical relevance.

3 The per-layer remote selection requires sending query vectors and index keys across the fabric and receiving selected token IDs for every decode step. While the paper characterizes bandwidth in Appendix, it does not analyze how latency spikes or network jitter affect performance under bursty request arrivals.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.