acceptodds
Under review as a conference paper at ICLR 2027

Adamas: Hadamard Sparse Attention for Efficient Long-context Inference

Abstract

Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods reduce these costs by attending to only a subset of key-value (KV) tokens, but fixed or coarse-grained selection strategies can miss important tokens or retain substantial redundancy under constrained token budgets. We introduce **Adamas**, an accurate and efficient token-level sparse attention mechanism for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and uses lightweight distance-based estimation for efficient top- candidate selection. The selected positions are then used to gather the corresponding original KV states for sparse attention. Comprehensive evaluations show that Adamas maintains near-lossless performance within the effective context length of Llama-3.1-8B-Instruct. In efficiency evaluations, Adamas achieves up to self-attention speedup and end-to-end decoding speedup at a 32K context length. Adamas also maintains performance comparable to full attention on challenging long-form reasoning benchmarks, including AIME 2024 and AIME 2025. Code is publicly available at [Anonymous Adamas Repo](https://anonymous.4open.science/r/Adamas-36EA).

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model, Sparse Attention, Long-context Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WrjeOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Adamas uses a Hadamard transform and two-bit query/key codes to select KV tokens. It then runs attention on the selected original KV states. The method needs no training and uses custom kernels.

Critical strengths
  • Quality is better than the tested sparse-attention baselines at small token budgets.
  • The paper measures kernel and decoding time, and tests the transform and quantization choices.
Critical weaknesses
  • Practical gains are limited. The largest speedups come from MHA, while end-to-end speedup on GQA is only about 1.1x. The paper does not show whether selection overhead is worthwhile for MLA or hybrid models with fewer full-attention layers.
  • Quest expands GQA to MHA in this implementation, which adds cost and causes OOMs. The runtime gap is therefore partly an implementation issue, not just a better selection method.
Questions for authors
  1. Can you separate selection and attention costs, with native GQA support for Quest?
  2. Does the method provide meaningful end-to-end gains on MLA or hybrid attention models, including selection overhead?
  3. What batch sizes were used? Do the GQA gains hold with batching?
4 comments
AuthorsOpenReview ↗

Replies to Reviewer Wrje (1)

Thank you for the thoughtful comments. Below we provide our detailed responses to each concern. For closely related questions and concerns, we provide a combined clarification.

W1 & Q2: End-to-End Efficiency and Accuracy on Hybrid Attention Models

Thank you for raising this concern. To evaluate Adamas on hybrid attention architectures, we conducted additional experiments using Qwen3.5-9B, a hybrid attention model that combines Gated DeltaNet (GDN) and full attention layers in a 3:1 ratio. We replaced all full attention layers (i.e., only one-quarter of the model's attention layers) with Adamas and evaluated both accuracy and end-to-end latency.

Accuracy. We evaluated the model on six LongBench datasets. The results are reported below:

ConfigurationQasperNarrativeQAHotpotQAMultiFieldQA-enTriviaQAGovReportAverage
Full48.9530.8560.8454.9990.4630.8552.82
Adamas-25647.6627.4159.5755.3491.2130.3251.92
Adamas-51247.3928.0160.1054.2190.8830.3851.83
Adamas-102448.5529.6760.6254.6690.8030.6452.49
Adamas-204847.7230.5761.7154.7490.9630.6852.73
Adamas-409649.5330.5261.9254.5090.9630.9753.07

End-to-end latency. We further measured the end-to-end speedup, including selection overhead, at different input lengths. We report results for context lengths of 32K and above:

Input Length256512102420484096
32K1.026×1.025×1.023×1.016×1.008×
64K1.065×1.063×1.061×1.054×1.046×
128K1.138×1.137×1.134×1.128×1.118×

Overall, Adamas maintains accuracy close to full attention, with the average LongBench score ranging from 51.83 to 53.07, compared with 52.82 for full attention. More importantly, despite replacing only one-quarter of the model's attention layers, Adamas achieves consistent end-to-end speedups at context lengths of 32K and above, reaching up to 1.138× at 128K. These results demonstrate that Adamas can provide practical efficiency gains in hybrid attention architectures, even when only a fraction of the attention layers are eligible for acceleration. We will include this analysis in the revised manuscript.

W2 & Q1: Native-GQA Quest Comparison at Matched Quality

Thank you for raising this concern. We agree that comparing Adamas against Quest's original MHA fallback may introduce implementation-related overhead and thus does not provide a fair assessment of their relative efficiency under GQA. To address this issue, we implemented a native-GQA version of Quest, which performs selection and attention directly over shared KV heads without expanding them into separate query heads. We then re-evaluated the decoding latency of both methods.

The updated decoding latency results are reported below:

Context LengthToken BudgetAdamasQuest (Native GQA)Full
32K25612.66112.86313.251
51212.77713.08013.251
102413.00713.21013.251
204813.14913.26213.251
409613.41213.85913.251
64K25613.82614.65715.681
51213.93314.70315.681
102414.06815.12115.681
204814.32115.26015.681
409614.55315.75515.681
128K25617.92718.42220.556
51218.01118.81120.556
102418.11618.74020.556
204818.38919.06620.556
409618.96319.56220.556

Adamas consistently achieves lower decoding latency than native-GQA Quest across all evaluated context lengths and token budgets.

To further compare the two methods at closely matched task quality, we selected two pairs of operating points based on their LongBench normalized scores: Adamas-256 (92.92%) vs. Quest-1024 (91.15%), and Adamas-512 (97.95%) vs. Quest-4096 (97.20%). Their corresponding decoding latencies are shown below.

Context LengthBudget (Adamas / Quest)LongBench (Adamas / Quest)Adamas LatencyQuest LatencySpeedup
32K256/102492.92%/91.15%12.66113.0071.027×
512/409697.95%/97.20%12.77713.8591.085×
64K256/102492.92%/91.15%13.82614.0681.018×
512/409697.95%/97.20%13.93315.7551.131×
128K256/102492.92%/91.15%17.92718.1161.011×
512/409697.95%/97.20%18.01119.5621.086×

These results show that Adamas achieves comparable LongBench performance with substantially smaller token budgets. Even at closely matched quality, Adamas consistently achieves lower decoding latency than native-GQA Quest, with speedups ranging from 1.011× to 1.131× across the evaluated context lengths.

Overall, these additional experiments demonstrate that Adamas's efficiency advantage persists when compared against a native-GQA implementation of Quest, and is therefore not solely attributable to the MHA expansion overhead in the original Quest implementation. We will include this comparison in the revised manuscript.

AuthorsOpenReview ↗

Replies to Reviewer Wrje (2)

Q3: Efficiency Gains across Batch Sizes

Thank you for this question. All latency experiments in the original paper were conducted with a batch size of 1. To evaluate whether the speedups persist under batched inference, we conducted additional experiments with batch sizes of 1, 2, 4, and 8, covering the batch sizes that fit within the 80 GB memory capacity of an NVIDIA H800. The results are reported below, where the speedup is calculated as the latency of Full Attention divided by that of Adamas.

ModelContext LengthB=1B=2B=4B=8
Llama-3.18K0.972×0.999×1.071×1.128×
16K1.014×1.079×1.231×1.414×
32K1.095×1.219×1.526×1.889×
LongChat4K1.031×1.110×1.212×1.333×
8K1.145×1.293×1.554×1.849×
12K1.241×1.476×1.881×2.286×

The results show that Adamas maintains its efficiency benefits under batched inference for both GQA (Llama-3.1) and MHA (LongChat) models. Moreover, the speedup generally increases with batch size. For example, Adamas achieves 1.89× speedup on Llama-3.1 at a batch size of 8 and 32K context length, and 2.29× on LongChat at a batch size of 8 and 12K context length.

These results demonstrate that the efficiency gains of Adamas extend beyond batch-size-1 inference and remain effective across a range of batch sizes. We will include these additional results in the revised manuscript.

Thanks for the additional experiments. They help clarify the trade-offs, though I still find the gains at near-full quality modest. I’m keeping my weak reject recommendation.

Thank you for the follow-up and for acknowledging that the additional experiments help clarify the trade-offs. We agree that the speedups at near-full-attention quality are relatively modest. However, we believe the broader results in the revised manuscript demonstrate the practical value of Adamas beyond this operating point. In particular, the additional batching experiments show that its speedups increase with batch size, reaching 1.89× for Llama-3.1 (GQA) at 32K and 2.29× for LongChat (MHA) at 12K, while the hybrid-model experiments further demonstrate its applicability to models with hybrid attention architectures. Together with the quality–latency comparisons, these results provide a more comprehensive picture of Adamas across different inference settings and operating points. We hope these additional results help clarify the scope and practical significance of our contribution.

Reviewer_NgbDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper proposes Adamas, a training-free sparse attention method for autoregressive decoding. Queries and keys are transformed with a Hadamard matrix, discretized into 2-bit codes, and compared using an integer distance estimator. The selected positions are used to fetch the original keys and values for attention. The paper also introduces shared candidate selection for GQA and custom CUDA kernels. Evaluations cover long-context understanding, retrieval, language modeling, and mathematical reasoning. The accuracy results are promising, while the practical speedups are substantially larger for MHA than for GQA.

Strengths

S1. The method addresses a practical inference bottleneck with a deployable design. The compressed codes are used only for candidate selection, while attention uses the original KV states. This avoids retraining the model or quantizing the final attention computation.

S2. The accuracy evaluation is reasonably broad. Table 1 reports 97.65% and 98.04% of full-attention performance on Llama-3.1 and Qwen3, respectively, averaged over the evaluated LongBench budgets. The passkey and PG19 results provide complementary evidence. Appendix B also evaluates both selectors on the same full-attention trajectories, which helps separate selection quality from differences in generated text.

S3. The implementation and ablations support the proposed combination. The paper reports kernel and end-to-end latency on two GPUs and studies the Hadamard transform, quantization width, and distance metric. The strong degradation without the transform under small budgets makes its role clear.

Weaknesses

W1. The comparison does not include a closely related token-level selector. The motivation emphasizes the limitations of fixed patterns and coarse page selection, but inexpensive query-dependent token selection already exists. For example, SparQ Attention, ICML 2024, estimates scores using selected query dimensions and then fetches the original top-k KV pairs; it also describes grouped selection for GQA. The current comparisons with StreamingLLM, SnapKV, and Quest do not establish the benefit of Hadamard-based codes relative to this alternative. A direct comparison of candidate recall and the accuracy-latency trade-off on a representative MHA and GQA setting would better establish the contribution. This is a question about the selector, rather than a request for a longer list of baselines.

W2. The practical efficiency advantage at comparable task quality is not yet clear, particularly for GQA. On A6000 with Llama-3.1 at 32K and budget 256, Table 12 reports 28.88 ms for Adamas, 19.33 ms for SnapKV, and 32.35 ms for full attention. On H800 at 48K, the corresponding Adamas/full-attention numbers are 14.70/15.68 ms. Appendix G acknowledges these setting-dependent results. However, the latency tables alone do not show which method offers the best trade-off at a given task quality. The paper also acknowledges that Quest expands GQA heads, so its latency and OOM failures partly reflect implementation compatibility. Please compare latency at similar task quality and distinguish these implementation effects from the algorithmic benefit. This would make the practical scope of the speedup claims easier to assess.

Critical strengths

S1. The method addresses a practical inference bottleneck with a deployable design. The compressed codes are used only for candidate selection, while attention uses the original KV states. This avoids retraining the model or quantizing the final attention computation.

Critical weaknesses

W1. The comparison does not include a closely related token-level selector. The motivation emphasizes the limitations of fixed patterns and coarse page selection, but inexpensive query-dependent token selection already exists. For example, SparQ Attention, ICML 2024, estimates scores using selected query dimensions and then fetches the original top-k KV pairs; it also describes grouped selection for GQA. The current comparisons with StreamingLLM, SnapKV, and Quest do not establish the benefit of Hadamard-based codes relative to this alternative. A direct comparison of candidate recall and the accuracy-latency trade-off on a representative MHA and GQA setting would better establish the contribution. This is a question about the selector, rather than a request for a longer list of baselines.

W2. The practical efficiency advantage at comparable task quality is not yet clear, particularly for GQA. On A6000 with Llama-3.1 at 32K and budget 256, Table 12 reports 28.88 ms for Adamas, 19.33 ms for SnapKV, and 32.35 ms for full attention. On H800 at 48K, the corresponding Adamas/full-attention numbers are 14.70/15.68 ms. Appendix G acknowledges these setting-dependent results. However, the latency tables alone do not show which method offers the best trade-off at a given task quality. The paper also acknowledges that Quest expands GQA heads, so its latency and OOM failures partly reflect implementation compatibility. Please compare latency at similar task quality and distinguish these implementation effects from the algorithmic benefit. This would make the practical scope of the speedup claims easier to assess.

Questions for authors

Q1. What accuracy-latency trade-off does Adamas achieve against a token-level method such as SparQ under the same model and execution setup?

Q2. In Eq. 11, candidates favored by any query head can fill the shared GQA budget. How does this minimum-distance aggregation affect per-head mass recall? Appendix B uses an MHA model; applying its existing diagnostic to GQA would directly test this design choice.

Q3. Were the calibration examples excluded from the reported LongBench evaluation? Were the same numerical thresholds transferred across all models, or was calibration repeated for each model? Appendix A usefully studies threshold multipliers, but does not make these protocol details clear.

7 comments
AuthorsOpenReview ↗

Replies to Reviewer NgbD (1)

Thank you for the thoughtful comments. Below we provide our detailed responses to each concern. For closely related questions and concerns, we provide a combined clarification.

W1 & Q1: Token-Level Selector Comparison with SparQ Attention

Thank you for this valuable suggestion. We conducted a direct comparison between Adamas and SparQ Attention using SparQ's official implementation. To ensure a fair comparison, we aligned the execution setup and operators across both methods, with the selection algorithm being the only difference. All experimental configurations follow those in our paper. For SparQ, we use the recommended configuration with r=16r=16.

Latency comparison. The decoding latency results are reported below:

Context LengthToken BudgetAdamasSparQ(r=16r=16)FullSparQ/Adamas
32K25612.66124.44813.2511.931×
51212.77724.56413.2511.923×
102413.00724.87613.2511.913×
204813.14924.33213.2511.850×
409613.41225.30013.2511.886×
64K25613.82623.98815.6811.735×
51213.93324.33715.6811.747×
102414.06824.21315.6811.721×
204814.32124.34815.6811.700×
409614.55327.76715.6811.908×
128K25617.92724.48920.5561.366×
51218.01125.24120.5561.401×
102418.11626.16520.5561.444×
204818.38927.61320.5561.502×
409618.96331.37820.5561.655×

Accuracy comparison. We further evaluated both methods on LongBench using Llama-3.1-8B. The results are shown below:

Token BudgetSparQAdamasFullSparQ − Adamas
25651.2448.1151.77+3.13
51251.5950.7151.77+0.88
102451.8551.0051.77+0.85
204851.9551.3351.77+0.62
409651.7851.6351.77+0.15

Overall, SparQ achieves slightly higher LongBench scores than Adamas, although the accuracy gap becomes smaller as the token budget increases. However, this comes at a substantial latency cost: SparQ is 1.37×–1.93× slower than Adamas across the evaluated configurations. In particular, at a budget of 4096, the two methods achieve nearly identical accuracy (51.78 vs. 51.63), while Adamas remains substantially faster.

These results demonstrate that Adamas offers a more favorable accuracy-latency trade-off: it achieves comparable accuracy with considerably lower latency, making it more effective for practical efficient inference. We will include this baseline in the revised manuscript.

W2: Quality-Matched Efficiency Comparison and Native-GQA Baseline

Thank you for this valuable suggestion. We agree that latency comparisons at fixed token budgets alone do not fully characterize the practical efficiency of Adamas, and that comparisons with Quest's MHA fallback may conflate algorithmic differences with GQA compatibility overhead.

To address the latter issue, we implemented a native-GQA version of Quest, which performs selection and attention directly over shared KV heads without expanding them to MHA, and re-evaluated its decoding latency against Adamas. The results are reported below:

Context LengthToken BudgetAdamasQuest (Native GQA)Full
32K25612.66112.86313.251
51212.77713.080
102413.00713.210
204813.14913.262
409613.41213.859
64K25613.82614.65715.681
51213.93314.703
102414.06815.121
204814.32115.260
409614.55315.755
128K25617.92718.42220.556
51218.01118.811
102418.11618.740
204818.38919.066
409618.96319.562

Adamas achieves lower decoding latency than native-GQA Quest across all evaluated context lengths and token budgets.

We further compare the two methods at closely matched LongBench quality. Specifically, we select two pairs of operating points: Adamas-256 (92.92%) vs. Quest-1024 (91.15%), and Adamas-512 (97.95%) vs. Quest-4096 (97.20%). Their corresponding decoding latencies and speedups are shown below:

Context LengthBudget (Adamas/Quest)LongBench (Adamas/Quest)Adamas LatencyQuest LatencySpeedup
32K256/102492.92%/91.15%12.66113.0071.027×
512/409697.95%/97.20%12.77713.8591.085×
64K256/102492.92%/91.15%13.82614.0681.018×
512/409697.95%/97.20%13.93315.7551.131×
128K256/102492.92%/91.15%17.92718.1161.011×
512/409697.95%/97.20%18.01119.5621.086×

At these matched-quality operating points, Adamas achieves lower decoding latency than native-GQA Quest across all evaluated context lengths, with speedups ranging from 1.011× to 1.131×. These comparisons also show that Adamas reaches comparable LongBench performance with smaller token budgets.

Together, these results provide a comparison that accounts for GQA compatibility and task quality: the latency advantage of Adamas persists against native-GQA Quest and is not solely attributable to the MHA expansion of Quest implementation. We will include this comparison in the revised manuscript.

AuthorsOpenReview ↗

Replies to Reviewer NgbD (2)

Q2: Per-Head Attention-Mass Recall under GQA Group-wise Sharing

Thank you for the suggestion. To directly evaluate the effect of the minimum-distance aggregation in Eq. 11, we extend the attention-mass recall analysis in Appendix B to two GQA models, Llama-3.1-8B and Qwen3-8B. We compare Adamas's group-wise shared selection (shared selection) against independently selecting Top-B KV candidates for each query head (independent selection). This comparison allows us to examine both the average per-head recall and the low-recall tails.

Attention-mass recall.

ModelBudgetShared mean (%)Independent mean (%)Shared P5 (%)Independent P5 (%)
Llama-3.125680.1983.5953.8260.24
51285.4088.1664.3270.09
102490.1892.1975.4679.07
204894.2895.5485.0587.39
409697.3597.9892.7294.00
Qwen325674.7179.3241.8350.87
51281.7885.5152.9661.73
102488.0490.7464.4372.47
204893.2494.9177.6882.89
409696.9997.8289.2991.86

Shared selection yields lower attention-mass recall than independent selection on both models, as expected from sharing a candidate set across query heads. The gap in mean recall decreases with the budget: from 3.40 to 0.63 percentage points for Llama-3.1-8B and from 4.61 to 0.83 percentage points for Qwen3-8B. The low-recall tails show a similar trend. For example, at a budget of 1024 on Llama-3.1-8B, shared selection has a P5 recall 3.61 percentage points lower than independent selection.

LongBench performance. We further evaluate both selection strategies on LongBench to assess whether the recall differences translate into downstream task performance differences.

BudgetLlama sharedLlama independentΔ\DeltaQwen3 sharedQwen3 independentΔ\Delta
25648.1149.93+1.8345.7448.63+2.88
51250.7151.46+0.7448.8049.62+0.82
102451.0051.64+0.6449.7350.70+0.97
204851.3351.57+0.2349.7950.98+1.19
409651.6351.31−0.3149.5951.16+1.57
Average50.5651.18+0.6348.7350.22+1.49

Δ\Delta denotes independent selection minus shared selection.

Averaged across budgets, shared selection scores 0.63 points lower than independent selection on Llama-3.1-8B and 1.49 points lower on Qwen3-8B. The difference is more pronounced at a budget of 256, while the two strategies achieve relatively close performance at budgets of 512–4096.

End-to-end decoding latency. We also compare the end-to-end decoding latency of the two strategies. Speedup is calculated as the latency of independent selection divided by that of shared selection.

Context LengthBudgetSharedIndependentSpeedup
32K25612.63512.8811.019×
51212.79513.1631.029×
102412.90513.5311.049×
204813.18514.0931.069×
409613.56814.8371.094×
64K25613.77414.4201.047×
51213.86114.6031.054×
102414.16615.2211.074×
204814.43216.1271.117×
409614.85817.4581.175×
128K25618.13619.0021.048×
51218.20419.3621.064×
102418.35820.0141.090×
204818.78121.0781.122×
409619.38922.7981.176×

Shared selection consistently achieves lower end-to-end decoding latency across all evaluated context lengths and budgets, with speedups ranging from 1.019× to 1.176×. Its efficiency advantage generally increases with the token budget and context length.

Overall, group-wise minimum-distance aggregation introduces a trade-off: it reduces per-head attention-mass recall compared with independent selection, particularly at smaller budgets, but also reduces end-to-end decoding latency. The resulting LongBench performance remains relatively close between the two strategies, with average score differences of 0.63 and 1.49 points on Llama-3.1-8B and Qwen3-8B, respectively. These results quantify the recall and downstream-performance impact of the shared selection design in Eq. 11, alongside its efficiency benefit. We will include this analysis in the revised manuscript.

AuthorsOpenReview ↗

Replies to Reviewer NgbD (3)

Q3: Calibration Data Independence and Model-Specific Thresholds

Thank you for raising this concern. In our original experiments, the Q/K thresholds were calibrated using LongBench data, which may raise concerns about potential dependence on the evaluation data. To investigate this, we conducted additional calibration experiments using C4-English, which is disjoint from the evaluation datasets used in our paper. We also clarify that the thresholds are calibrated separately for each model, rather than shared across models.

The newly calibrated thresholds are compared with the original ones below:

ModelQ: LongBench → C4ChangeK: LongBench → C4Change
Llama-3.1-8B1.3517 → 1.3960+3.28%2.2577 → 2.2305−1.20%
Qwen3-8B1.5656 → 1.5989+2.13%3.1817 → 3.1872+0.17%
LongChat-7B1.4534 → 1.4904+2.55%1.7977 → 1.7863−0.64%

As shown above, changing the calibration dataset leads to only minor changes in the calibrated Q/K thresholds across all three models. To further evaluate the impact of these changes, we tested Llama-3.1-8B on LongBench using the newly calibrated C4 thresholds and compared the results with those obtained using the original thresholds (both reported as normalized scores):

Token BudgetOriginal ThresholdsC4-Calibrated ThresholdsChange (percentage points)
25692.92%93.19%+0.27
51297.95%96.66%−1.29
102498.51%98.78%+0.27
204899.15%99.42%+0.27
409699.72%100.10%+0.39
Average across budgets97.65%97.63%−0.02

The average LongBench score across all five token budgets changes by only 0.02 percentage points when using the C4-calibrated thresholds. This indicates that the calibration results are stable across different datasets and that the choice of calibration data has a negligible impact on downstream performance.

These results provide evidence that Adamas calibration does not rely on LongBench-specific examples and transfers well to held-out evaluation data. We will include these results and clarify the calibration protocol in the revised appendix.

Reviewer_abQXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Adamas, a token-level sparse-attention method that requires no training. Queries and keys are passed through a Hadamard transform, the transformed coordinates are discretized into four calibrated buckets, and packed 2-bit key representations are stored next to the original KV cache. At decoding time, a low-bit distance estimator chooses candidate positions, and attention is then computed with the original query and the selected original KV states. Under GQA, each KV-head group shares one candidate selection. Over six LongBench tasks and three models, the reported average normalized score reaches 98.78% of full attention across the evaluated budgets. Diagnostics on shared trajectories show more retained attention mass than Quest, and the custom kernels yield large attention speedups for MHA, with smaller end-to-end gains under GQA. Retrieval, perplexity, and long-form mathematical reasoning are also evaluated.

Critical strengths
  1. The method practically divides inexpensive candidate discovery from accurate final attention. Only selection relies on the compressed representation; the attention computation still has access to the original KV values. Ablations over the Hadamard transform, bit width, and distance metric justify the design choices instead of crediting the result to sparsity alone. Notably, low-budget accuracy suffers considerably without the transform, whereas moving from two to three bits yields comparatively little.

  2. The experiments link the quality of the selector to downstream behavior. In Appendix B, both methods are evaluated on identical full-attention trajectories, which removes divergent generated sequences as a confound. With 512 selected tokens, Adamas keeps 92.03% of attention mass compared with 88.54% for Quest; with 1024 tokens, the figures are 94.70% and 92.27%, respectively. The LongBench, retrieval, perplexity, and AIME evaluations then check whether these candidates stay useful during real generation. Taken together, this is more convincing than either retrieval recall or task accuracy on its own.

Critical weaknesses
  1. A more carefully matched GQA comparison is needed to support the practical efficiency advantage. According to the paper, the Quest implementation used for evaluation expands shared KV heads into an MHA-style representation, which contributes to its latency and memory failures. Although the native GQA implementation in Adamas is a valuable engineering contribution, this comparison conflates selection quality with overhead from the baseline's implementation. The distinction is important: Section 4.3 reports end-to-end acceleration of up to 1.8× for MHA but only 1.1× for GQA, and Figure 3 shows GQA attention acceleration of just 1.3× on H800 at 48K. I would like the main conclusion to separate the large MHA gain from the more modest GQA result, and to include a comparison with a native-GQA selector at matched quality or matched retained attention mass.

  2. Support for broadly near-lossless selection is stronger on empirical grounds than on mechanistic or statistical ones. Preserving dot products prior to quantization does not show that the L1 distance between bucket codes preserves the dot-product ranking, especially when key norms and per-head distributions vary. Although the recall analysis helps, it covers only one MHA model; the same analysis matters most for the group-wise GQA rule, in which a single candidate set is shared by all query heads. Because calibration draws on subsets of LongBench tasks, the manuscript should clarify whether calibration is model-specific and whether those examples are excluded from the reported evaluation. Lastly, running greedy decoding repeatedly on the same 30 AIME questions yields no independent evidence of non-inferiority. These limitations narrow the scope of the claim; they do not negate the reported accuracy gains.

Questions for authors
  1. Against a baseline that performs selection and attention directly over shared KV heads, without MHA expansion, how much of the GQA latency and memory advantage persists? Please report operating points at matched quality or matched recall.
  2. Could you report per-head attention-mass recall for Llama-3.1 and Qwen3, including the low-recall tails, and isolate the effect of sharing a single selected set within each GQA group?
  3. Are the calibration examples disjoint from the LongBench examples used for evaluation, and are the quoted thresholds shared across models or estimated separately for each? A result showing that calibration transfers to held-out data would strengthen the generality claim.
4 comments
AuthorsOpenReview ↗

Replies to Reviewer abQX (1)

Thank you for the thoughtful comments. Below we provide our detailed responses to each concern. For closely related questions and concerns, we provide a combined clarification.

W1 & Q1: Native-GQA Quest Comparison at Matched Quality

Thank you for this valuable suggestion. We agree that although the original Quest implementation does not support GQA, comparing against its MHA fallback is insufficient to fully characterize its performance. To address this concern, we implemented a native-GQA version of Quest, which performs selection and attention directly over shared KV heads without MHA expansion, and re-evaluated the latency of both methods.

The updated decoding latency results are reported below:

Context LengthToken BudgetAdamasQuest (Native GQA)Full
32K25612.66112.86313.251
51212.77713.08013.251
102413.00713.21013.251
204813.14913.26213.251
409613.41213.85913.251
64K25613.82614.65715.681
51213.93314.70315.681
102414.06815.12115.681
204814.32115.26015.681
409614.55315.75515.681
128K25617.92718.42220.556
51218.01118.81120.556
102418.11618.74020.556
204818.38919.06620.556
409618.96319.56220.556

Adamas consistently achieves lower decoding latency than native-GQA Quest across all evaluated context lengths and token budgets.

Furthermore, to provide a comparison at closely matched quality (reported as normalized scores), we selected two pairs of operating points based on their LongBench results: Adamas-256 (92.92%) vs. Quest-1024 (91.15%), and Adamas-512 (97.95%) vs. Quest-4096 (97.20%). Their corresponding decoding latencies are shown below.

Context LengthBudget (Adamas / Quest)LongBench (Adamas / Quest)Adamas LatencyQuest LatencySpeedup
32K256 / 102492.92% / 91.15%12.66113.0071.027×
512 / 409697.95% / 97.20%12.77713.8591.085×
64K256 / 102492.92% / 91.15%13.82614.0681.018×
512 / 409697.95% / 97.20%13.93315.7551.131×
128K256 / 102492.92% / 91.15%17.92718.1161.011×
512 / 409697.95% / 97.20%18.01119.5621.086×

These results show that Adamas achieves comparable LongBench performance with substantially smaller token budgets. More importantly, even at closely matched quality, Adamas consistently outperforms native-GQA Quest in decoding latency, with speedups ranging from 1.011× to 1.131× across the evaluated context lengths.

Overall, these additional experiments demonstrate that the efficiency advantage of Adamas persists against a native-GQA implementation of Quest, rather than arising solely from the MHA expansion overhead in the original Quest implementation. We will include this comparison in the revised manuscript.

Part of W2: Statistical Reliability of the AIME Evaluation

Thank you for this comment. We agree that repeated greedy decoding on the same 30 questions does not provide independent statistical evidence. Our primary motivation for including AIME is different: these challenging problems require Qwen3's reasoning mode, resulting in long generation trajectories. Adamas achieves comparable or better accuracy than full attention under this setting, providing empirical evidence that its approximation errors do not accumulate severely over long generation trajectories. We will clarify this motivation in the revised manuscript and avoid interpreting these results as a statistical non-inferiority claim.

AuthorsOpenReview ↗

Replies to Reviewer abQX (2)

Q2: Per-Head Attention-Mass Recall and the Impact of GQA Group-wise Sharing

Thank you for the suggestion. To further analyze the effect of Adamas's sparse selection mechanism (referred to as shared selection) on attention quality, we extend the attention-mass recall evaluation in Appendix B to Llama-3.1-8B and Qwen3-8B. We also compare shared selection with independent KV selection for each query head under GQA (referred to as independent selection), including the low-recall tails.

Attention-mass recall.

ModelBudgetShared mean (%)Independent mean (%)Shared P5 (%)Independent P5 (%)
Llama-3.125680.1983.5953.8260.24
51285.4088.1664.3270.09
102490.1892.1975.4679.07
204894.2895.5485.0587.39
409697.3597.9892.7294.00
Qwen325674.7179.3241.8350.87
51281.7885.5152.9661.73
102488.0490.7464.4372.47
204893.2494.9177.6882.89
409696.9997.8289.2991.86

The results show that shared selection achieves lower attention-mass recall than independent selection on both models. The difference in mean recall decreases as the token budget increases, from 3.40 to 0.63 percentage points for Llama-3.1-8B and from 4.61 to 0.83 percentage points for Qwen3-8B. A similar trend is observed in the low-recall tails. For example, at a budget of 1024 on Llama-3.1-8B, the P5 recall of shared selection is 3.61 percentage points lower than that of independent selection.

LongBench performance. We further evaluate both selection methods on LongBench to examine their impact on downstream task performance.

BudgetLlama sharedLlama independentΔ\DeltaQwen3 sharedQwen3 independentΔ\Delta
25648.1149.93+1.8345.7448.63+2.88
51250.7151.46+0.7448.8049.62+0.82
102451.0051.64+0.6449.7350.70+0.97
204851.3351.57+0.2349.7950.98+1.19
409651.6351.31−0.3149.5951.16+1.57
Average50.5651.18+0.6348.7350.22+1.49

Δ\Delta denotes independent selection minus shared selection.

On average across budgets, shared selection achieves scores 0.63 points lower than independent selection on Llama-3.1-8B and 1.49 points lower on Qwen3-8B. The difference is more pronounced at a budget of 256, while the two methods achieve relatively close performance at budgets of 512–4096.

End-to-end decoding latency. We additionally compare the end-to-end decoding latency of the two selection methods. The results are reported below, where speedup is calculated as the latency of independent selection divided by that of shared selection.

Context LengthBudgetSharedIndependentSpeedup
32K25612.63512.8811.019×
51212.79513.1631.029×
102412.90513.5311.049×
204813.18514.0931.069×
409613.56814.8371.094×
64K25613.77414.4201.047×
51213.86114.6031.054×
102414.16615.2211.074×
204814.43216.1271.117×
409614.85817.4581.175×
128K25618.13619.0021.048×
51218.20419.3621.064×
102418.35820.0141.090×
204818.78121.0781.122×
409619.38922.7981.176×

Shared selection consistently achieves lower end-to-end decoding latency than independent selection across all evaluated context lengths and token budgets, with speedups ranging from 1.019× to 1.176×. The efficiency advantage generally increases with the token budget and context length.

Overall, these results characterize the trade-off introduced by group-wise candidate sharing. Compared with independent selection, shared selection retains slightly less attention mass and achieves somewhat lower LongBench scores, while reducing end-to-end decoding latency by up to 1.176×. This demonstrates the efficiency benefit of shared selection alongside its impact on attention recall and downstream task performance. We will include this analysis in the revised manuscript.

AuthorsOpenReview ↗

Replies to Reviewer abQX (3)

W2 & Q3: Calibration Data Independence and Model-Specific Thresholds

Thank you for raising this concern. We understand the concern regarding the use of LongBench data for calibration in the original experiments. To investigate whether Adamas calibration depends on the choice of calibration dataset, we repeated the calibration using the C4-English dataset, which is entirely disjoint from the evaluation datasets used in our paper. We also clarify that the thresholds are calibrated separately for each model, rather than shared across models.

The newly calibrated thresholds are compared with the original ones below:

ModelQ: LongBench → C4ChangeK: LongBench → C4Change
Llama-3.1-8B1.3517 → 1.3960+3.28%2.2577 → 2.2305−1.20%
Qwen3-8B1.5656 → 1.5989+2.13%3.1817 → 3.1872+0.17%
LongChat-7B1.4534 → 1.4904+2.55%1.7977 → 1.7863−0.64%

As shown above, changing the calibration dataset leads to only minor changes in the calibrated Q/K thresholds across all three models. To further evaluate the impact of these changes, we tested Llama-3.1-8B on LongBench using the newly calibrated C4 thresholds and compared the results with those obtained using the original thresholds (both reported as normalized scores):

Token BudgetOriginal ThresholdsC4-Calibrated ThresholdsChange (percentage points)
25692.92%93.19%+0.27
51297.95%96.66%−1.29
102498.51%98.78%+0.27
204899.15%99.42%+0.27
409699.72%100.10%+0.39
Average across budgets97.65%97.63%−0.02

The average LongBench score across all five token budgets changes by only 0.02 percentage points when using the C4-calibrated thresholds. This indicates that the calibration results are stable across different datasets and that the choice of calibration data has a negligible impact on downstream performance.

These results provide evidence that Adamas calibration does not rely on LongBench-specific examples and transfers well to held-out evaluation data. We will include these results and clarify the calibration protocol in the revised appendix.

Thank you again for your feedback. We have revised the manuscript and conducted additional experiments to address the concerns raised by the reviewers. As the discussion deadline is approaching, we would appreciate it if you could let us know whether any concerns remain unresolved or if you have any further questions regarding our revisions. We would be happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.