acceptodds
Under review as a conference paper at ICLR 2027

SABER: State-Aware Budget Estimation and Routing for Sparse Attention

Abstract

Sparse attention accelerates long-context Transformer inference by selecting a subset of key–value (KV) entries for each query. Existing adaptive sparse-attention methods typically rely on posterior pruning, leaving the upstream pre-selection budget as a manual design choice and introducing irregular computation patterns. We propose State-Aware Budget Estimation and Routing (SABER), an internal-probe-guided framework that unifies pre-allocation and online control of sparse-attention budgets. SABER predicts, for each query or generation state, the sufficiency of candidate budgets using lightweight probes over intermediate hidden states of the language model, and allocates the smallest budget that preserves task performance. This mechanism applies consistently across generation: it selects sufficient budgets from prompt states and, for long reasoning, continuously refines them as the generation trajectory evolves, enabling both one-shot allocation and trajectory-aware dynamic adjustment in a single framework. By performing input- and state-dependent allocation before and during decoding, SABER avoids over-provisioned attention and reduces unnecessary computation. Across long-context and reasoning benchmarks, SABER preserves accuracy relative to strong fixed-budget, top-, and trainable sparse-attention baselines while reducing average selector budget by up to 67.6% and achieving up to 1.35 end-to-end speedup over state-of-the-art fixed-budget sparse attention methods.

Primary area:
generative models
Keywords:
Sparse Attention, Efficient LLM Inference, Long-Context Inference, Long-Reasoning Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EEX1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This is a good paper that brings together several findings in the field and empirically demonstrates their effectiveness in real applications. The paper shows that applying pre-determined yet dynamic budget control to sparse attention can improve inference efficiency while maintaining quality. The experiments provide good results. Although the overall experimental design does not analyze every step or component in an extremely complete and fine-grained manner, it is already sufficient to support the paper’s claims.

Critical strengths

As a paper that focuses more on methodological and empirical progress, it demonstrates solid improvements and shows a certain degree of generality. I believe some readers will find ideas in this paper that can be applied to their own work. Therefore, I lean toward acceptance.

Critical weaknesses

The main incremental contribution of this paper lies in the specific budget prediction and control scheme, together with the corresponding empirical validation, rather than in a new principle for computation allocation. I think the paper makes limited conceptual progress and does not bring enough new knowledge. This limits me from giving a rating of 4.

There are also some presentation issues. For example, the contrast in Figure 4(b) and Figure 5 seems somewhat too high/too low, and Table 3 itself is placed quite far from where it is actually discussed. However, I can see that the authors have already put effort into the design and layout, so this is acceptable if there is no clearly better way to present them. In addition, some terms in the paper are a little confusing. For example, “panel head” (854–855) and “panel distillation” (858) somewhat increase the cost of understanding. Also, “posterior pruning” (013) may be interpreted by readers with a theoretical background as something related to probability, so this could be clarified. This point will affect my Presentation score. Since these issues are easy to fix, I am giving a Presentation score of 3 in advance.

Questions for authors

I found that some works I know that are related to this paper in terms of method or idea were not included in the comparison. This is understandable, because under the mainstream ideas in this area, there can be many specific implementations, and it is difficult to compare against all of them. Therefore, I only hope that the authors can conceptually clarify the novelty of SABER relative to Fluxion [1], Elastic Attention [2], FBS [3], and ACM [4], as a supplement.

[1] Yao, F., Niu, Z., Li, X., Xiong, Y., Fang, J., & Wang, Q. (2026). An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference. arXiv preprint arXiv:2605.07719.

[2] Tang, Z., Qiu, Q., Yang, Y., Hong, Z., Xiang, H., Liu, K., Dang, Q., Li, J. & Zhang, M.. (2026). Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:119115-119142.

[3] Wang, T. (2026). Fbs: Modeling native parallel reading inside a transformer. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 4106-4137).

[4] Wójcik, B., Devoto, A., Pustelnik, K., Minervini, P., & Scardapane, S. (2025). Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference. Proceedings of the AAAI Conference on Artificial Intelligence, 39(20), 21510–21518.

2 comments

W1. “The main incremental contribution of this paper lies in the specific budget prediction and control scheme, together with the corresponding empirical validation, rather than in a new principle for computation allocation. I think the paper makes limited conceptual progress and does not bring enough new knowledge. This limits me from giving a rating of 4.”

We respectfully disagree: SABER introduces not only a budget prediction and control scheme, but also a new principle for computation allocation. Prior adaptive sparse-attention methods such as Twilight [1] and Double-P [2] determine budgets through top-p selection based on attention statistics, aiming to preserve a target amount of attention mass. SABER instead uses the model’s internal hidden states to predict whether a budget is sufficient to preserve the downstream task outcome and provides online budget control as decoding progresses.

This principle provides a deployment-friendly direction for inference optimization. Compared with training-free sparse-attention methods, SABER provides adaptive budget control while reusing existing backend kernels (as demonstrated in our main experiments and our response to Reviewer 2GZ1’s Q5), offering a practical deployment advantage on accelerators with restricted support for custom kernels, such as Intel NPUs. Compared with methods requiring training, such as Elastic Attention[3] and FBS[4], SABER uses separate supervised training of its predictors and controller, avoiding backpropagation through the LLM and reinforcement learning.

We therefore respectfully ask the reviewer to reconsider the assessment of limited conceptual progress.

Q1. “I found that some works I know that are related to this paper in terms of method or idea were not included in the comparison. This is understandable, because under the mainstream ideas in this area, there can be many specific implementations, and it is difficult to compare against all of them. Therefore, I only hope that the authors can conceptually clarify the novelty of SABER relative to Fluxion [1], Elastic Attention [2], FBS [3], and ACM [4], as a supplement.”

Fluxion: Fluxion’s novelty lies in a CPU–GPU co-designed sparse-attention system. Its predictor identifies whether an attention head requires retrieval from CPU-resident historical KV and estimates the budget–granularity relationship for retrieval heads. SABER’s predictor instead predicts whether the model can produce a correct answer under a given attention budget.

Elastic Attention: Elastic Attention’s novelty lies in a per-head router that switches between full and sparse attention, trained via backpropagation through the LLM while keeping the backbone frozen. SABER instead trains a separate predictor to estimate budget sufficiency from hidden states.

FBS: FBS’s novelty lies in controlling computation depth through layer skipping, whereas SABER controls the sparse attention backend’s budget. FBS modifies the model architecture and trains it through continual pretraining followed by PPO; SABER instead constructs supervision from model-generated answers and trajectories to train separate predictors and a controller, without modifying the backbone or sparse-attention backend.

ACM: ACM’s novelty lies in replacing models’ original modules with distilled learners and using a learnable gate to adapt the number of learners executed per token. ACM pursues efficient inference through distillation, whereas SABER focuses on budget allocation for existing sparse-attention backends.

W2. “There are also some presentation issues. For example, ... This point will affect my Presentation score. Since these issues are easy to fix, I am giving a Presentation score of 3 in advance.”

We thank the reviewer for these valuable suggestions. We will adjust the contrast in Figures 4(b) and 5, move Table 3 closer to its discussion, and clarify the terms “panel head,” “panel distillation,” and “posterior pruning” in the revised manuscript.

[1] Chaofan Lin et al. Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning. NeurIPS, 2025.

[2] Wentao Ni et al. Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs. arXiv:2602.05191, 2026.

[3] Zecheng Tang et al. Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers. ICML, 2026.

[4] T. Wang. FBS: Modeling Native Parallel Reading inside a Transformer. Findings of ACL, 2026.

Thank you for the authors’ response. My concerns were relatively limited to begin with, and I think the response has largely addressed them.

I believe there was a slight misunderstanding regarding what I meant by a “principle” of computation allocation. I was referring to a relatively high-level idea: how we understand the computation-allocation problem, how the overall design space is structured, or at least a broad family of methods for addressing it. From this perspective, I still view SABER primarily as a strong and useful methodological contribution. Its task-outcome-oriented budget prediction and online control are meaningful extensions for sparse attention, but they do not substantially change my broader understanding of how adaptive computation can be allocated based on model states or input-dependent signals.

This clarification does not negatively affect my overall assessment. I appreciate the authors’ detailed response, and I remain inclined to accept the paper.

In addition, I have read the other reviewers' comments and the authors' responses. Their concerns are somewhat different from mine, and I look forward to seeing more follow-up discussion.

Reviewer_2GZ1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tries to reduce the cost of top-k sparse attention by deciding how many KV tokens to select for each input, instead of using a fixed budget. SABER does not change which tokens are selected; that is left to an existing sparse attention backend. The premise is that the model's hidden states already carry enough information to tell whether a given budget will be sufficient.

For long-context inputs, a lightweight probe on the prompt's hidden states predicts whether each candidate budget is enough, and the smallest sufficient one is used for the whole decode. For long reasoning, the budget is updated periodically during generation: budget-specific predictors estimate whether the current trajectory will end correctly, and a small controller decides whether to raise or lower the budget, trading off predicted accuracy against attention cost.

The evaluation covers LongBench and math reasoning benchmarks (AIME24/25, MATH-500), using Quest and SeerAttention-R as backends. Reported outcomes include accuracy comparable to fixed large-budget baselines at roughly a third of the average budget, and moderate end-to-end decode speedups.

Critical strengths
  1. Deciding the sparse attention budget per input, and for reasoning per segment, is an interesting and timely problem. Most adaptive methods (Twilight, Tactic) still derive the budget from attention scores inside the kernel. Moving the decision upstream keeps the kernel a plain top-k and makes it independent of the backend. The online version, where the budget can go up or down during a long reasoning trace, is the part I find most novel, and I think it is a direction worth exploring.

  2. The method is practical and easy to adopt. It does not modify the backbone or the sparse attention kernel; it only adds a small predictor on top of an existing backend, and the paper shows it working with two different ones (Quest and SeerAttention-R). The training cost is modest (about 20 GPU-hours for the long-context estimator, Table 11), and the runtime overhead is negligible. It also performs well on downstream long-context and reasoning benchmarks with a clearly smaller average budget.

Critical weaknesses
  1. The reasoning results in Table 2 are hard to trust as they are. Full-attention Qwen3-4B gets 54.58 / 43.13 / 85.2 on AIME24 / AIME25 / MATH-500, while the Qwen3 report gives 73.8 / 65.6 / 97.0. My guess is that the 12K generation limit is the main cause: SeerAttention-R reports average AIME generation lengths of 11K-18K under a 32K limit. Several sparse settings also beat full attention (SeerAttention-4K/8K on both AIME sets, SABER on AIME24 by about 4 points). This goes against SeerAttention-R's own finding that less accurate sparse attention makes reasoning longer, which should hurt under a hard limit. Without output lengths and truncation rates, I can't tell whether the gains come from better budgets or simply from how often each method finishes within 12K. In addition, reasoning is trained and tested only on math, so it is unclear whether the router transfers to other tasks such as LiveCodeBench. Since the online router is the main new part of the paper, this weighs heavily in my score.

  2. The design of the modules is largely empirical, and the paper does not show that the predicted budget is actually sufficient. Many choices are not motivated or analyzed: using the last prompt token from layers 24/28, one predictor per budget, the 512-token check interval, and the utility p − λC(b) with a hand-picked λ. The controller is also trained on scores produced by the predictors rather than on the real outcome of switching budgets. All the evidence is end-task accuracy vs. average budget. There is no attention-level check, e.g., attention recall at the predicted K compared with fixed K, and no accuracy or calibration results for the predictors, even though the method thresholds their probabilities directly. As a result, it is hard to tell whether the probe learns "this budget is sufficient" or just "this question is hard". In the latter case, simple rules such as a per-task budget or a budget proportional to context length might work equally well, but they are not compared.

  3. The paper does not compare with decode-time sparse attention methods designed for long reasoning, such as TidalDecode [1] and LessIsMore [2]. These methods target the same setting as SABER's online router, and both report strong accuracy at small budgets on reasoning tasks. Without them, it is unclear whether SABER's adaptive budget actually improves over the current state of the art for reasoning, or whether a better token selection method alone would already give similar or better results. It would also be useful to know whether SABER's budget routing can be combined with these methods.

[1] Yang et al. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention. ICLR 2025.

[2] Yang et al. Less Is More: Training-Free Sparse Attention with Global Locality for Efficient Reasoning. ICML 2026.

Questions for authors
  1. Truncation under the 12K limit. Could you report the average output length and the truncation rate for each row in Table 2, and rerun the main comparison with a 32K limit and more samples (as in SeerAttention-R)? If SABER still matches the 4K/8K baselines at a lower budget, most of my concern about Table 2 would be resolved. 
  2. Generalization beyond math. Can the math-trained router be used on LiveCodeBench without retraining? 
  3. Is the predicted budget sufficient? Could you report attention recall at the predicted K vs. fixed K, and the held-out accuracy and calibration of the predictors? This is the main evidence I would need to believe the probe predicts budget sufficiency rather than question difficulty. 
  4. Simple budget rules. How does SABER compare with non-learned rules at the same average budget (e.g., per-task budget, budget proportional to context length), and with a fixed Quest baseline at around 2.6K on LongBench? 
  5. Decode-time sparse methods. How does SABER compare with TidalDecode and LessIsMore at a similar budget, and can it be combined with them? 
  6. Batched decoding. When requests in a batch use different K, how is this executed? If the batch is padded to the largest K, the savings may largely disappear, especially with mixed tasks. 
  7. Latency breakdown. In Fig. 4(b), why does the token selector time drop from about 22us to 7us, given that Quest scores all pages regardless of K?
6 comments

W1 & Q1. “This goes against SeerAttention-R's own finding that less accurate sparse attention makes reasoning longer, which should hurt under a hard limit. Without output lengths and truncation rates, I can't tell whether the gains come from better budgets or simply from how often each method finishes within 12K.” & “Truncation under the 12K limit. Could you report the average output length and the truncation rate for each row in Table 2, and rerun the main comparison with a 32K limit and more samples (as in SeerAttention-R)? If SABER still matches the 4K/8K baselines at a lower budget, most of my concern about Table 2 would be resolved.”

(1) Output lengths and truncation rates under the original 12K limit. The following table reports average output tokens / truncation rate for the fixed-budget baselines and SABER:

MethodAIME24AIME25MATH500
SeerAttention-R 1K9847.6 / 26.46%10019.7 / 32.29%4622.3 / 2.40%
SeerAttention-R 2K9523.7 / 24.58%9851.6 / 30.21%4466.3 / 2.60%
SeerAttention-R 4K9394.6 / 22.71%9905.6 / 33.33%4414.6 / 1.80%
SeerAttention-R 8K9457.4 / 26.46%9851.6 / 31.67%4398.4 / 2.60%
SABER9484.4 / 22.92%9893.3 / 32.92%4404.8 / 2.00%

These statistics are at similar levels across methods. SABER does not consistently generate shorter outputs or finish within the limit more often than the 4K/8K baselines.

(2) SABER retains its accuracy advantage under a 32K generation limit. The following table reports AIME24/25 (4 seeds) and MATH500 (1 seeds) accuracy with the generation limit increased to 32K:

MethodAIME24AIME25MATH500
SeerAttention-R 4K68.33%62.50%86.00%
SeerAttention-R 8K68.33%67.50%87.00%
SABER73.33%69.17%86.80%

(3) SABER continues to use substantially smaller attention budgets under the 32K limit. The following table compares SABER’s average attention budgets under the original 12K and extended 32K generation limits:

DatasetAvg. budget, 12K limitAvg. budget, 32K limit
AIME2424452518.23
AIME2524972675.42
MATH50010421087.5

The average budget increases modestly while remaining substantially below both fixed-budget baselines. With longer generations, SABER tends to allocate larger budgets in later reasoning stages, consistent with its trajectory-aware budget adjustment.

In conclusion, these results show that SABER’s gain comes from better budget allocation, not simply from how often each method finishes within 12K. This gain persists under the extended 32K limit: SABER achieves competitive or even higher accuracy than the 4K/8K baselines on AIME24/25 and MATH500 while using a substantially smaller average attention budget. We hope these additional results address your concern about Table 2.

W1 & Q2. “In addition, reasoning is trained and tested only on math, so it is unclear whether the router transfers to other tasks such as LiveCodeBench. Since the online router is the main new part of the paper, this weighs heavily in my score.” & “Can the math-trained router be used on LiveCodeBench without retraining? ”

Yes, the math-trained router can be used on LiveCodeBench without retraining. We directly apply the same math-trained router without retraining to the first 100 problems in LiveCodeBench v6, comprising 27 easy, 28 medium, and 45 hard problems. The results are:

MethodAccuracyAvg. attention budget
SeerAttention-R 4K46.00%4096
SeerAttention-R 8K47.00%8192
SABER47.00%2452.7

These results show that SABER trained-on-math has competitive performance compared with the 4K/8K baselines on LiveCodeBench while using a substantially smaller average attention budget, supporting zero-shot generalization from mathematical reasoning to code generation. This observation is consistent with literature that linear probes over LLM hidden states can generalize to unseen datasets [1].

Our LongBench experiments also provide additional evidence of cross-task generalization. As described in Section 4.1, the long-context budget estimator is trained only on QA data. Nevertheless, Table 1 shows that it generalizes to other LongBench task categories, such as coding and summarization, supporting the generalization of our method. [1] Dhananjay Ashok and Jonathan May. Language Models Can Predict Their Own Behavior. NeurIPS, 2025

W2. “The design of the modules is largely empirical… Many choices are not motivated or analyzed: using the last prompt token from layers 24/28, one predictor per budget, the 512-token check interval, and the utility p − λC(b) with a hand-picked λ.”

These choices are motivated by prior work and the requirements of budget-conditioned prediction, with an interval ablation already reported in Table 5. Following the reviewer’s valuable suggestions, we will make their rationale and supporting evidence more explicit in the revised manuscript.

(1) Our use of the last prompt token and L24/L28 is motivated by prior probing literature. Zou et al. [3] use last-token representations by default and demonstrate effective readouts from middle-to-late layers. Marks and Tegmark [2] identify linearly decodable truth representations in intermediate layers, while Yan [4] shows later layers’ representations aligning with output prediction in LLaMA-3-8B on arithmetic tasks. These findings motivate concatenating L24 (intermediate) and L28 (later) to combine information from two stages of representation processing.

(2) We use one predictor per budget to match the budget-specific formulation of the long-reasoning prediction task. Each predictor addresses a binary classification task: whether the current trajectory will lead to a correct answer under its corresponding attention budget.

(3) Table 5 analyzed the performance of different token check intervals. The 512-token setting achieves the highest AIME24/25 accuracy among these configurations while using a smaller average attention budget than the 256-token setting, supporting the selected accuracy–budget trade-off.

(4) Our utility p-λC(b) follows a standard quality–cost trade-off formulation in adaptive routing [5]. Here, λ controls the relative weight of computational cost. We use λ=0.2 consistently across the reasoning benchmarks.

W2. “The controller is also trained on scores produced by the predictors rather than on the real outcome of switching budgets.”

(1) This is SABER’s design choice, not a weakness. Separating semantic estimation from routing decisions enables efficient policy learning. And our experiments (empirical performance) demonstrate that predictor scores provide effective supervision for budget control.

(2) Training on predictor scores is not merely “not a weakness”; it is a strength that reduces supervision-collection costs. Our current setting requires one prefix replay per candidate budget, followed by supervised learning. Direct supervision from switching outcomes would require multiple budget-switching continuations from the same prefix, each generated to completion, substantially increasing trace-collection costs. Online policy optimization, such as PPO, would introduce further rollout and optimization costs.

[2] Marks and Tegmark. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. 2023. [3] Zou et al. Representation Engineering: A Top-Down Approach to AI Transparency. 2023. [4] Yao Yan. Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs. Findings of EMNLP, 2025. [5] Somerstep et al. CARROT: A Cost Aware Rate Optimal Router. 2025

W2 & Q3. “All the evidence is end-task accuracy vs. average budget. There is no attention-level check, e.g., attention recall at the predicted K compared with fixed K, and no accuracy or calibration results for the predictors, even though the method thresholds their probabilities directly.” & “Could you report attention recall at the predicted K vs. fixed K, and the held-out accuracy and calibration of the predictors?”

(1) Attention recall. We report average attention-mass recall on MultifieldQA-en, where SABER selects a budget for each input and uses an average budget of 2879 tokens:

Budget configurationAvg. attention-mass recall
SABER, avg. K=287989.38%
Fixed K=25677.86%
Fixed K=102487.25%
Fixed K=409696.71%
Fixed K=819299.46%

SABER uses a smaller average budget than the 4K/8K baselines and correspondingly retains less attention mass. This is expected: at the same attention state, a smaller K under the same top-k sparse-attention backend always retains less attention mass. Nevertheless, SABER maintains competitive task performance, showing that lower attention recall does not necessarily imply an insufficient budget.

(2) Predictor accuracy and calibration. Our predictors estimate completion correctness under a specified attention budget, directly assessing whether that budget is sufficient for the task. Their held-out validation accuracy and calibration results are:

PredictorAccuracyECEBrierNLL
Long-context79.00%0.09180.15160.4755
Long-reasoning89.74%0.07490.08990.3677

Together, these results support the effectiveness of SABER’s budget-sufficiency predictions. The accuracy and calibration results provide direct evidence of predictor quality, while the end-task accuracy and attention recall demonstrate that the SABER preserves competitive performance despite retaining less attention mass.

W2 & Q4. “As a result, it is hard to tell whether the probe learns "this budget is sufficient" or just "this question is hard". In the latter case, simple rules such as a per-task budget or a budget proportional to context length might work equally well, but they are not compared.” & “Simple budget rules. How does SABER compare with non-learned rules at the same average budget (e.g., per-task budget, budget proportional to context length), and with a fixed Quest baseline at around 2.6K on LongBench?”

SABER outperforms both uniform and task-wise fixed-budget allocation at the same average attention budget. The following table compares their LongBench performance, with scores averaged equally across the 13 tasks:

MethodAvg. attention budgetLongBench avg. score
Quest, uniform fixed K265448.96
Quest, task-wise fixed K265448.76
SABER265449.81

For the uniform fixed baseline, each task uses a constant K equal to SABER’s overall average budget for LongBench. For the task-wise baseline, each task uses a constant K equal to SABER’s average budget for that task.

These results show that simple rules such as a per-task budget do not work equally well as SABER. SABER’s learned budget-sufficiency predictions enable more effective input-dependent allocation, improving task performance at the same average budget.

Reviewer_FHCjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SABER which employs lightweight probes to read the intermediate hidden states of LLMs. It dynamically allocates a sparse attention budget during long-context prefilling and long-sequence inference, thereby reducing computational overhead while maintaining accuracy.

Critical strengths
  1. Uses hidden-state probes instead of attention scores to determine budget allocation is interesting and useful.
  2. It supports both pre-allocation for long texts and dynamic routing during long-context inference.
  3. Probes feature extremely low latency and are plug-and-play.
Critical weaknesses
  • The method lacks both elegant theoretical modeling and a unified design.
  • The approach relies on coarse-grained switching between only five predefined, discrete budget tiers. Moreover, long-context inference requires a costly offline prefix replay process for every new model and budget level. It maybe severely limits flexible deployment.
  • The paper fails to specify which hidden layers are extracted for long-context inference (in the context of Qwen3) or how prefix sequences are cached. It also mixed the distribution shift that occurs between the mixed-budget trajectories used during online routing and the single-budget trajectories used during training.
  • The authors use SeerAttention to measure accuracy for long-context inference, yet switch to Quest operators when benchmarking speed.
Questions for authors
  1. Which hidden layer is used for extraction during long-sequence inference? How do you address the distribution shift between mixed-budget inference and single-budget training? 2. Why is SeerAttention used for accuracy evaluation, while Quest is used for measuring inference speed? 3. Do you plan to open-source the code and model weights?
3 comments

W1. “The method lacks both elegant theoretical modeling and a unified design.”

(1) SABER provides a unified approach to deciding how much computation a sparse-attention backend needs. The problem we study is how to allocate sufficient attention budget across inputs and reasoning stages while avoiding unnecessary computation. Our unified principle is to use the model’s hidden states to predict budget sufficiency. Figures 2 and 3 apply this same principle to different inference requirements: long-context inference allocates a budget after prefilling, whereas long reasoning adjusts it as decoding progresses. These are two applications of a unified budget-allocation design.

(2) SABER’s novelty is to make budget allocation a separate, learnable control problem, rather than embedding it within the sparse-attention algorithm. Representative adaptive methods, such as top-p pruning [1,2], determine budgets through attention-mass estimation and pruning within the attention computation. SABER instead introduces a lightweight estimator and controller that decides how much computation is needed, while the existing backend determines which tokens to retain.

This is a methodological contribution with practical deployment benefits. SABER adds adaptive budget control without requiring new attention kernels or fine-tuning the base model, allowing existing sparse-attention implementations to benefit from the new allocation mechanism. Beyond our main experiments, the LessIsMore results (reviewer 2GZ1 Q5) demonstrate that SABER further improves an existing backend’s accuracy–budget trade-off by reusing predictors and controller trained with another backend without retraining.

W2. “The approach relies on coarse-grained switching between only five predefined, discrete budget tiers. Moreover, long-context inference requires a costly offline prefix replay process for every new model and budget level. It maybe severely limits flexible deployment.”

(1) The reason why we use discrete budget tiers is to preserve GPU kernel efficiency. Prior sparse-attention implementations commonly use discrete budget configurations, including powers of two, to support efficient GPU execution. SABER follows this practice and selects among these configurations according to the input or reasoning state.

(2) We respectfully disagree that the offline preparation cost severely limits flexible deployment. This cost is incurred once per adapted model and amortized across subsequent inference runs. As reported in Table 11, only 20.47 GPU-hours of preparation enable a 67.6% reduction in average attention budget on LongBench relative to the 8K setting, while preserving competitive accuracy. For long reasoning, 204.12 GPU-hours enable a 51.3% reduction relative to the 4K setting, again with competitive accuracy. The trained components are reused as inference savings accumulate.

Rather than severely limiting flexible deployment, SABER’s plug-in design makes it easier to deploy adaptive sparse attention. Other sparse-attention approaches introduce deployment costs through specialized kernels, which can require additional porting effort on accelerators with restricted custom-kernel support, or through substantial training that updates the base model’s parameters. SABER instead trains separate lightweight components and reuses existing backend kernels, enabling flexible deployment with minimal changes to the existing inference stack.

[1] Chaofan Lin et al. Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning. NeurIPS, 2025.

[2] Wentao Ni et al. Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs. arXiv:2602.05191, 2026.

W3 & Q1. “The paper fails to specify which hidden layers are extracted for long-context inference (in the context of Qwen3) or how prefix sequences are cached. It also mixed the distribution shift that occurs between the mixed-budget trajectories used during online routing and the single-budget trajectories used during training.”

(1) For long-reasoning inference with Qwen3, SABER uses the final-layer hidden states. As stated in the appendix, “We adopt the hidden circuit encoder architecture in (Ghasemabadi and Niu, 2025) as the backbone of our predictor.” This architecture uses final-layer hidden states. We will explicitly specify this layer choice in the revised manuscript to avoid ambiguity.

(2) Prefix-level features are cached through offline replay, as described in Appendix A.1. We split generated completions into prefixes at 512-token intervals. Each prefix is replayed under each candidate budget, and we cache the corresponding predictor features, logit margins, and correctness probabilities for subsequent controller training.

(3) We indeed use single-budget trajectories for training, but the empirical performance proves our effectiveness on mix-budget trajectories. Collecting training rollouts across different budget-switching sequences would introduce substantial additional costs. We therefore use single-budget trajectories and replay their prefixes under each candidate budget to construct supervision. Despite this simpler training setup, our result shows that SABER preserves competitive accuracy with a smaller average attention budget during mixed-budget inference, while Table 9 further demonstrates the effectiveness of the learned controller.

W4 & Q2. “The authors use SeerAttention to measure accuracy for long-context inference, yet switch to Quest operators when benchmarking speed. Why is SeerAttention used for accuracy evaluation, while Quest is used for measuring inference speed?”

We want to clarify that SABER uses the same backend for accuracy and efficiency evaluation within each inference setting. For long-context inference, both the LongBench results in Table 1 and the efficiency evaluation in Figure 4 use are for Quest. For long-reasoning inference, both the accuracy results in Table 2 and the runtime results in Table 3 are for SeerAttention-R. Thus, the different backends correspond to different inference settings, rather than a switch between accuracy and speed evaluation. We will make it more explicit in the revised manuscript.

Q3. Do you plan to open-source the code and model weights?

Yes.

Dear Reviewer FHCj,

We would like to follow up on our responses to your comments. We would appreciate your feedback on whether these responses address your concerns and would be happy to clarify any remaining questions. Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Remarks for the Reviewers and Area Chair

We thank all reviewers for their constructive feedback. We appreciate that Reviewers 2GZ1 and EEX1 indicated that our responses and additional evidence addressed most of their concerns, with 2GZ1 raising their score and EEX1 maintaining a positive assessment. We have also provided detailed responses to Reviewer FHCj and hope they will consider these clarifications and additional results in their final assessment. We summarize the key responses below.

  • Generation-limit sensitivity (2GZ1): Under a 32K limit, SABER retains higher accuracy on AIME24/25 and competitive accuracy on MATH500 with substantially smaller budgets. Together with the 12K output-length and truncation statistics, this shows that its advantage is not simply an artifact of the shorter limit.

  • Generalization beyond math (2GZ1): SABER generalizes to code generation by directly reusing its math-trained predictors and controller without retraining.

  • Backend compatibility (2GZ1): We reuse the SeerAttention-R-trained predictors and controller with another sparse-attention backend, LessIsMore, without retraining.

  • Predictor validation (2GZ1): We provided validation accuracy, calibration metrics, and attention-recall measurements to supplement end-task evaluations and assess predictor quality.

  • Simple budget rules (2GZ1): SABER outperforms uniform and task-wise fixed-budget allocation at the same average attention budget, showing the effectiveness of its input-dependent budget decisions.

  • Methodological contribution (EEX1): We clarified SABER’s distinction from related approaches: it predicts budget sufficiency and controls allocation while retaining the existing sparse-attention backend. The cross-task and cross-backend results above further support its practical value and generality.

  • Unified design and implementation (FHCj): Both inference settings use hidden states to predict budget sufficiency, enabling allocation after prefilling and online adjustment during reasoning. We also specified hidden-state extraction and prefix replay, and explained how supervision from single-budget trajectories supports effective mixed-budget inference.

  • Preparation costs and flexible deployment (FHCj): The reported preparation costs cover all candidate budgets, rather than a single tier, and are amortized across subsequent inference requests. The LiveCodeBench and LessIsMore results above additionally demonstrate reuse across tasks and backends without repeating the training process.

  • Runtime evaluation (FHCj): Table 3 compares fixed-budget and adaptive-budget execution using the same kernel implementation; Appendix A.2 explains the measurement setup and rationale.

We thank the reviewers and Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.