acceptodds
Under review as a conference paper at ICLR 2027

ASIR: An Agent-Structure-Aware Indexer with Block-Local Reuse for Native Sparse Attention

Abstract

Long-context agents accumulate conversations, tool outputs, and files, increasing the cost of attention over the growing history. Native sparse attention uses an Indexer to select a fixed number of tokens for attention, but selection still scans the full context. Hierarchical Indexers narrow the search by selecting a prescribed number of blocks before scoring their tokens. However, their budgets overlook request-specific differences in evidence coverage, and they rescan every selected block at each decoding step. We present ASIR, an Agent-Structure-Aware Indexer with block-local candidate reuse. During prefill, ASIR combines native segment boundaries, request relevance, and spatial extent to allocate candidate capacity. During decoding, it retains historical candidates in some selected blocks and refreshes others, scoring all candidates with the current query. The two components reduce the spatial extent and temporal repetition of Indexer scoring, respectively. On LongBench and LongMemEval, ASIR achieves aggregate quality comparable to a hierarchical baseline searching one quarter of the context while reducing both candidate-search capacity and repeated token-level scoring. Across 16K–128K contexts, ASIR achieves – Indexer speedup over native sparse attention and – over this hierarchical baseline.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Long-context language models, Efficient LLM inference, Native sparse attention, Adaptive token selection, Agentic systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZjJuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ASIR, which optimizes Indexer by using context structure to set request-specific budgets and selectively reusing token positions within blocks. Historical candidates are rescored with the current query, while local refresh allows discovery of previously unselected tokens. On LongBench and LongMemEval, ASIR achieves upto 1.59× faster Indexer search with matched quality.

Critical strengths
  1. The motivation is well illustrated. As shown in Figure 1, growing context length makes the indexer a bottleneck and hence a desirable target to optimize.
  2. The reuse design is reasonable. Reusing positions rather than scores preserves current-query ranking.
  3. The latency measurement is detailed. Timing breakdown includes Token Scoring, Top-K, and history management, clearly showing where the gains are from.
Critical weaknesses
  1. The paper is not well-written. Figure 4 is blurred and hard to read. I suggest using PDF with high resolution. Also, Algorithm 1 should be shown in one page.
  2. The evaluation lacks end-to-end experiment. Full-model evaluation with real application configurations like batched serving is needed to validate the gain. ASIR can change hidden states and following execution so the efficiency gains can only be evaluated in end-to-end setting instead of just replay.
  3. Different models, including recent models like DeepSeek V4, need to be tested to validate genralization.
Questions for authors

I think the experiment scope is the main concern. See weaknesses.

Reviewer_PiegOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ASIR, a new indexer designed based on HISA. ASIR designs a dynamic block budget based on the concentration of each request and only refreshes a small portion of the index during decoding. It achieves 1.11–1.59 times faster performance compared to HISA, but its accuracy is lower than that of native sparse attention.

Critical strengths
  1. ASIR designs a dynamic candidate capacity to reduce unnecessary retrieval.
  2. Based on the high overlap of top-k indices, ASIR designs the reuse of historical indices to further reduce computational overhead.
Critical weaknesses
  1. The candidate capacity remains unchanged after prefill. However, as the length of the agent’s thought increases, more information is required to generate the next token. The fixed capacity limits the model’s retrieval ability.
  2. ASIR has a slight improvement in indexer latency compared to HISA, but the benefit to the overall model’s inference latency is minimal; however, its accuracy decreases to some extent compared to HISA.
  3. It needs to be considered whether it is compatible with the prefix cache, because even when the prefix hits, ASIR still requires an additional calculation of the candidate capacity C.
Questions for authors
  1. Dt=⋓p∈BtPpD_t=\Cup_{p\in\mathcal{B}_t}P_p, so Dt∩Pp=PpD_t\cap P_p=P_p. Why is Dt∩PpD_t\cap P_p used in the paper?
  2. Why choose to use blocks with alternating parity to refresh indices turn by turn, rather than other refresh methods?
Reviewer_FCWMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AI was used to rephrase/polish wording. The reviewer wrote the original assessment from scratch and reviewed the final wording.

Original reviewer-written assessment supplied as input:

Flag For Ethics Review

No.

Critical Weaknesses

  1. The current evidence does not separate the contribution of the structure-guided capacity controller from that of simply using a smaller search budget. In the main setting, capacity varies only within the narrow interval from 3L/16 to L/4. The rank-reversed control is informative: on the diagnostic cohort, it reduces LongBench quality by 1.33 points while doing slightly more candidate work. On LongMemEval, however, accuracy is unchanged, and the average capacity perturbation there amounts to only 1.8%. Nor is rank reversal the most practical alternative to the controller. What is centrally missing is a comparison against a request-independent capacity set to match the work ASIR actually performs, under the same reuse policy. Without it, the paper shows that ASIR is an effective configuration, but it shows less clearly that native structure is needed for its quality–efficiency advantage.

  2. The evidence on deployment and quality is narrower than the agent-oriented motivation. Quality is evaluated on only one checkpoint, and speed is obtained by replaying native query/key traces at batch size one on a single H20. This demonstrates an optimization of the Indexer, but not the latency benefit actually realized once sparse attention, prefill, and full-model execution are taken into account. In addition, similar aggregate quality masks large differences: with reuse added, DuReader drops from 34.03 to 24.45, whereas Qasper rises from 31.55 to 37.80. Appendix A.4 further indicates that the subsets used to choose the refresh interval are part of the full evaluation. None of this invalidates the operator measurements, but it does weaken the claim that the chosen policy reliably maintains quality on unseen agent workloads.

Questions For Authors

  1. How does ASIR compare, at matched mean candidate work, with a fixed-capacity HISA variant that uses the same block-local reuse policy? The answer would bear directly on my assessment of the structure-guided controller.
  2. For the same configurations evaluated for quality, could you report full-model time per output token, with capacity initialization included? What share of the Indexer gain persists outside the replay benchmark?
  3. Could the refresh policy be evaluated on examples held out from its development subsets, with paired uncertainty estimates and a breakdown of the large loss on DuReader?

Rating

2: Weak rejection.

I place this paper slightly below the acceptance threshold. The mechanism for maintaining block-local candidates is reasonable, and the Indexer gains are credibly measured. My reservations concern two points: the comparison against a simpler matched-work controller is incomplete, and there is limited evidence that the chosen configuration maintains quality and improves full-model latency beyond the development workload. Adding a matched-work baseline and a held-out end-to-end evaluation could change this assessment materially.

Soundness

2: I have minor or fixable concerns about the paper’s soundness.

Paper Summary

ASIR addresses the expense of building the token support used by native sparse attention. It pairs a controller that sets candidate capacity per request with reuse of candidates within local blocks. In the prefill stage, native segment boundaries and request-conditioned Indexer scores are used to estimate how widely the relevant evidence is spread, and a candidate capacity is then fixed for each layer. In the decoding stage, the current query chooses a block route: certain blocks expose every token, whereas others supply positions selected earlier. The current query then rescores every resulting candidate. Evaluated on Keye-VL-2.0-30B-A3B, ASIR reaches LongBench and LongMemEval aggregate scores near those of HISA-L/4, and it speeds up the full decoding-time Indexer by 2.23–4.39× relative to native selection and by 1.11–1.59× relative to HISA-L/4. What these results support is lowering the work spent on candidate discovery under a fixed final attention budget; they do not yet establish an end-to-end speedup for agents.

Presentation

3: Clear and easy to follow.

Critical Strengths

  1. Separating region routing from candidate renewal is a useful idea with a clear operational meaning. A block that remains relevant does not need to be fully scanned at each decoding step. ASIR keeps historical positions only within the current route; blocks that are new or lack history are opened at once, older regions are revisited periodically, and the current query reranks every candidate. This offers a concrete way to benefit from temporal stability without mistaking stale relevance scores for current evidence.

  2. The operator evaluation accounts for meaningful implementation costs and reveals boundary cases. Routing, candidate construction, selection, mapping, and history maintenance are all inside the timed Indexer. At 64K, ASIR reduces latency to 20.37 μs, compared with 31.25 μs for HISA-L/4. The authors further report that at 16K reuse runs slightly slower than capacity adaptation by itself, quantify the one-time cost of constructing the signal, and include a rank-reversed capacity control. Such details give the systems result more credibility than a speedup measured on token scoring alone.

Informativeness

2: I learned something interesting or useful.

Critical strengths

ASIR addresses the expense of building the token support used by native sparse attention. It pairs a controller that sets candidate capacity per request with reuse of candidates within local blocks. In the prefill stage, native segment boundaries and request-conditioned Indexer scores are used to estimate how widely the relevant evidence is spread, and a candidate capacity is then fixed for each layer. In the decoding stage, the current query chooses a block route: certain blocks expose every token, whereas others supply positions selected earlier. The current query then rescores every resulting candidate. Evaluated on Keye-VL-2.0-30B-A3B, ASIR reaches LongBench and LongMemEval aggregate scores near those of HISA-L/4, and it speeds up the full decoding-time Indexer by 2.23–4.39× relative to native selection and by 1.11–1.59× relative to HISA-L/4. What these results support is lowering the work spent on candidate discovery under a fixed final attention budget; they do not yet establish an end-to-end speedup for agents.

Critical weaknesses
  1. Separating region routing from candidate renewal is a useful idea with a clear operational meaning. A block that remains relevant does not need to be fully scanned at each decoding step. ASIR keeps historical positions only within the current route; blocks that are new or lack history are opened at once, older regions are revisited periodically, and the current query reranks every candidate. This offers a concrete way to benefit from temporal stability without mistaking stale relevance scores for current evidence.

  2. The operator evaluation accounts for meaningful implementation costs and reveals boundary cases. Routing, candidate construction, selection, mapping, and history maintenance are all inside the timed Indexer. At 64K, ASIR reduces latency to 20.37 μs, compared with 31.25 μs for HISA-L/4. The authors further report that at 16K reuse runs slightly slower than capacity adaptation by itself, quantify the one-time cost of constructing the signal, and include a rank-reversed capacity control. Such details give the systems result more credibility than a speedup measured on token scoring alone.

Questions for authors
  1. The current evidence does not separate the contribution of the structure-guided capacity controller from that of simply using a smaller search budget. In the main setting, capacity varies only within the narrow interval from 3L/16 to L/4. The rank-reversed control is informative: on the diagnostic cohort, it reduces LongBench quality by 1.33 points while doing slightly more candidate work. On LongMemEval, however, accuracy is unchanged, and the average capacity perturbation there amounts to only 1.8%. Nor is rank reversal the most practical alternative to the controller. What is centrally missing is a comparison against a request-independent capacity set to match the work ASIR actually performs, under the same reuse policy. Without it, the paper shows that ASIR is an effective configuration, but it shows less clearly that native structure is needed for its quality–efficiency advantage.

  2. The evidence on deployment and quality is narrower than the agent-oriented motivation. Quality is evaluated on only one checkpoint, and speed is obtained by replaying native query/key traces at batch size one on a single H20. This demonstrates an optimization of the Indexer, but not the latency benefit actually realized once sparse attention, prefill, and full-model execution are taken into account. In addition, similar aggregate quality masks large differences: with reuse added, DuReader drops from 34.03 to 24.45, whereas Qasper rises from 31.55 to 37.80. Appendix A.4 further indicates that the subsets used to choose the refresh interval are part of the full evaluation. None of this invalidates the operator measurements, but it does weaken the claim that the chosen policy reliably maintains quality on unseen agent workloads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.