acceptodds
Under review as a conference paper at ICLR 2027

Scalable Attention: The Inference Compute-Cache Scaling Law

Abstract

We present Scalable Attention (SA), which scales attention compute without enlarging the KV cache, adding marginal parameters. Based on SA, we propose the compute-cache scaling law, which models loss as separate power laws in compute and cache. Given unit prefill and decode costs, set by prefix caching, workload, speculative decoding, and hardware, the law returns the optimal compute and cache for any target loss. At the 3.3B scale, SA with twice the attention compute outperforms GQA in validation loss while using only of its KV cache.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Full Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iMJSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Scalable Attention to increase attention compute without enlarging the KV cache and fits a compute-cache scaling law to guide inference resource allocation.

Critical strengths
  1. The paper conducts a substantial pretraining experiment, training eleven approximately 3.3B configurations on 98.3B tokens and reporting validation and downstream results.
Critical weaknesses
  1. Limited novelty. The added linear projections amount to a factorized expansion of query heads. Their value beyond ordinary attention parameterization is not established. This is essentially a LoRA-like addition of trainable parameters, without a clear new insight. Without the normalization before the added projection, its linear map can be absorbed into the original query projection.
  2. Insufficient baselines. The experiments compare against only one GQA configuration, while the MQA baseline fails to train. The reported quality gains are very marginal, and the limited comparisons do not establish a convincing advantage over existing attention designs.
  3. Unsupported law claim. Fitting five parameters to eleven configurations at one model scale does not establish a general inference law. Neither the assumed compute-cache separability nor the predicted deployment optimum is independently validated.
Questions for authors

How does performance change if the normalization before the added query projection is removed?

17 comments

We thank the reviewer for the helpful suggestions and feedback. We would first like to clarify a few points that may have been misunderstood and ignored. We address the concerns below:

W1: The added linear projections amount to a factorized expansion of query heads. Their value beyond ordinary attention parameterization is not established. This is essentially a LoRA-like addition of trainable parameters, without a clear new insight. Without the normalization before the added projection, its linear map can be absorbed into the original query projection.

A: As discussed in Sections 3.2 and 3.3, this extension scales attention compute at only a small parameter cost. In contrast, simply repeating the existing components would add compute without adding representational capacity. The additional query projections cannot be absorbed into the original projections, because the total number of sub-heads exceeds the original number of heads. Absorbing them would require duplicating the original projection parameters across sub-heads.

W2: The experiments compare against only one GQA configuration, while the MQA baseline fails to train. The reported quality gains are very marginal, and the limited comparisons do not establish a convincing advantage over existing attention designs.

A: We respectfully note that in LLM pre-training, a loss reduction of 0.005 is already considered significant. SA-4 (4d_h) reduces the loss by 0.007 relative to GQA, which exceeds this threshold. Regarding baselines, we focus on GQA because it is the most widely adopted attention design in current frontier LLMs. MHA and MLA are rarely used in recent large-scale models. We will include MQA results in the final version, although MQA generally underperforms GQA in model quality. More importantly, our contribution goes beyond a lower loss at a fixed budget. Model quality can always be improved by enlarging the KV cache or increasing attention compute, but both raise inference cost. We therefore derive an inference compute-cache scaling law that identifies the lowest-cost configuration for a target model quality. Existing attention designs offer no comparable guidance, and this is a key advantage of our method.

W3: Fitting five parameters to eleven configurations at one model scale does not establish a general inference law. Neither the assumed compute-cache separability nor the predicted deployment optimum is independently validated.

A: Our scaling law is intentionally scoped. We fix the model size and training data, and model only how loss varies with attention compute and KV cache, the two quantities that determine inference cost. In this setting, model size and data size are constants and do not need to appear in the law. Our five-parameter form also matches the complexity of widely used scaling laws such as Chinchilla. Extending the law to vary model and data size jointly would require additional parameters and substantially more training runs, which we leave to future work. Regarding separability, we also fitted non-separable forms that couple compute and cache, and they gave a worse fit than the separable form. Regarding the deployment optimum, the optimal configuration depends on the prefill-decode unit cost of real user requests. This unit cost varies across requests, and estimating it requires large-scale production data that we do not have.

Q: How does performance change if the normalization before the added query projection is removed?

A: In a preliminary ablation, removing RMS normalization increased the loss by approximately 0.005 within the first 5,000 training steps. As noted above, a difference of this size is significant in LLM pre-training, so we stopped the ablation at that point and retained RMS normalization in all subsequent experiments.

Dear Reviewer iMJS,

As the discussion period closes in two days, we want to gently follow up and ask whether you could review our rebuttal. We would be grateful for any further feedback.

Reviewer_iMJSOpenReview ↗

Response

Thank you for the response. My main concerns remain unresolved.

  1. Limited novelty. The added linear projections amount to a factorized expansion of query heads, closely related to ordinary attention parameterization. Without intermediate normalization, the linear maps can be absorbed into effective query projections. Adding normalization does not by itself establish a substantive architectural novelty. The conceptual advance of this combination remains unclear.

  2. Lack of significance. The paper does not establish a clear practical advantage that would justify adopting SA over existing attention designs. GQA can also vary query-head count, head dimension, and the sharing ratio (query heads per KV head). At fixed KV-head count and head dimension, increasing query heads raises this ratio without increasing the KV cache. At fixed total query-projection width, more query heads can instead be obtained by reducing head dimension. The relevant comparison is therefore with GQA configurations at comparable parameter, compute, and cache budgets, with remaining differences explicitly reported, to establish the value of SA's derived heads.

  3. Insufficient evidence. The following experiments are needed to support the conclusions.

    • Cache-matched GQA comparisons. Please evaluate GQA with 1, 2, and 4 KV heads to match SA's 2d_h, 4d_h, and 8d_h cache budgets. Alternatively, evaluate SA at GQA's 16d_h budget using compatible head configurations. Adjust query-head counts to compare at similar attention compute, and report parameter counts and remaining compute differences.

    • Additional attention baselines. Please include successful comparisons with MQA, MLA, and GLA under comparable cache and compute budgets, using the same training data, token budget, and evaluation protocol. A single GQA configuration and an unsuccessful MQA run do not establish an advantage over these alternatives. Promising these comparisons for a later version does not resolve the current gap.

    • Robustness and model scale. Quality gains are very marginal and inconsistent. In Table 4, SA-4 at 4d_h improves validation loss by about 0.007, yet average accuracy falls from 59.11% to 58.70%. Please repeat matched comparisons across independent seeds, report means and standard deviations, and test whether the conclusions hold beyond the single 3.3B model scale.

    • Scaling-law evidence. The experimental coverage is too limited to support a general scaling law. The paper fits five parameters to only eleven SA configurations at a single 3.3B model scale. The new held-out fits remain within this same study and do not show that the relationship holds at other model scales or for other attention architectures. Broader experiments are needed to justify the law's claimed generality.

I maintain my original recommendation.

Reviewer_JqtBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Scalable Attention (SA), an attention architecture that increases computation without proportionally increasing its KV-cache size or parameter count. Building on GQA, SA projects each query head into sas_a sub-heads that share a common KV head (hkvh_{kv} in total), which serves as both key and value. The outputs are combined through learned projections. The authors thoroughly studied this architecture, by training 11 3.3B scale SA models with different sas_a and hkvh_{kv} configurations, and proposed a inference compute-cache scaling law that models validation loss as a sum of independent power-law terms in attention compute and KV-cache size. SA with 2× attention computation achieves a lower validation loss than GQA (2.1785 vs. 2.1809) while requiring only 25% of its KV cache. Additional decoding kernel benchmarks on NVIDIA B200 GPUs demonstrate that moderately increasing attention computation incurs limited decoding latency overhead.

Critical strengths
  • SA introduces a straightforward mechanism for increasing the number of attention distributions through lightweight query projections while maintaining a compact KV cache.
  • The authors train 11 configurations at the 3.3B scale and characterize how attention computation and KV-cache size affect validation loss, and then connects this trade-off to hardware characteristics and supports its analysis with decoding kernel benchmarks on B200 GPUs.
Critical weaknesses
  • The experimental evaluation primarily compares SA against a single GQA configuration, without comparisons to other KV-cache-efficient architectures such as MLA or GLA. Although the authors justify excluding these baselines based on their incremental prefill inefficiencies, this does not establish whether SA achieves a competitive quality-efficiency trade-off. Furthermore, SA differs from GQA in several architectural aspects, including tied key-value representations and additional query sub-heads. Without cache-matched baselines or appropriate ablations, it is difficult to isolate the benefits of the proposed design. This is particularly concerning given the representational restrictions introduced by SA and the inconsistent downstream accuracy improvements reported in Table 4.

  • The proposed scaling law is fitted to only eleven SA configurations at a single model scale, without evaluating its predictive accuracy on unseen configurations or its applicability to other attention architectures. Moreover, the law is derived entirely from FineWeb-Edu validation loss at a context length of 4,096, which may not adequately reflect broader model capabilities, particularly long-context performance. Although the authors use this law to derive optimal compute-cache allocations, they do not validate these predictions through end-to-end inference measurements under realistic workloads. Consequently, the results appear to characterize SA's specific design space rather than establish a generalizable inference compute-cache scaling law.

Questions for authors
  • Can the authors provide model-quality comparisons against MLA, GLA, or other competitive cache-efficient attention architectures under matched parameter and KV-cache budgets? How does SA compare against GQA configurations with reduced KV-head counts or tied key-value representations?

  • How well does the proposed scaling law generalize, e.g. to other SA configurations, or to other models structures?

  • Does the observed compute-cache relationship hold for downstream capabilities, particularly long-context tasks such as retrieval and long-context understanding? The downstream accuracy improvements reported in Table 4 are less consistent than the validation-loss improvements.

2 comments

We thank the reviewer for the helpful suggestions and feedback. We address the concerns below:

W1 The experimental evaluation primarily compares SA against a single GQA configuration, without comparisons to other KV-cache-efficient architectures such as MLA or GLA. Although the authors justify excluding these baselines based on their incremental prefill inefficiencies, this does not establish whether SA achieves a competitive quality-efficiency trade-off. Furthermore, SA differs from GQA in several architectural aspects, including tied key-value representations and additional query sub-heads. Without cache-matched baselines or appropriate ablations, it is difficult to isolate the benefits of the proposed design. This is particularly concerning given the representational restrictions introduced by SA and the inconsistent downstream accuracy improvements reported in Table 4.

A SA builds on GQA, which is currently the most widely used attention mechanism in frontier LLMs and has been shown to offer a competitive quality-efficiency trade-off. SA inherits this trade-off. Moreover, because of KV tying, SA requires only half the KV cache of GQA for the same number of KV heads. For example, SA-1 (8d_h) uses half the KV cache of GQA (16d_h), and its loss is only 0.0045 higher. We cannot provide SA-1 (16d_h) because a configuration with 16 KV heads is not feasible in our setting. We will add GQA with KV tying in the final version to enable a direct comparison at the same KV cache size.

W2 The proposed scaling law is fitted to only eleven SA configurations at a single model scale, without evaluating its predictive accuracy on unseen configurations or its applicability to other attention architectures. Moreover, the law is derived entirely from FineWeb-Edu validation loss at a context length of 4,096, which may not adequately reflect broader model capabilities, particularly long-context performance. Although the authors use this law to derive optimal compute-cache allocations, they do not validate these predictions through end-to-end inference measurements under realistic workloads. Consequently, the results appear to characterize SA's specific design space rather than establish a generalizable inference compute-cache scaling law.

A Our scaling law is intentionally scoped. We fix the model size and training data, and model only how loss varies with attention compute and KV cache, the two quantities that determine inference cost. In this setting, model size and data size are constants and do not need to appear in the law. Our five-parameter form also matches the complexity of widely used scaling laws such as Chinchilla. Extending the law to vary model and data size jointly would require additional parameters and substantially more training runs, which we leave to future work. Evaluating performance at long context lengths would require long-context training, which exceeds our current computational resources. We lack the industrial-grade infrastructure needed to measure end-to-end latency on real-user requests.

Q1 Can the authors provide model-quality comparisons against MLA, GLA, or other competitive cache-efficient attention architectures under matched parameter and KV-cache budgets? How does SA compare against GQA configurations with reduced KV-head counts or tied key-value representations?

A MLA has been deprecated by DeepSeek, there is no need to compare with MLA. SA inherits this trade-off. Moreover, because of KV tying, SA requires only half the KV cache of GQA for the same number of KV heads. For example, SA-1 (8d_h) uses half the KV cache of GQA (16d_h), and its loss is only 0.0045 higher. We cannot provide SA-1 (16d_h) because a configuration with 16 KV heads is not feasible in our setting. We will add GQA with KV tying in the final version to enable a direct comparison at the same KV cache size.

Q2 How well does the proposed scaling law generalize, e.g. to other SA configurations, or to other models structures?

A Existing attention architectures cannot scale attention compute at a low parameter cost. As a result, it is difficult to establish a comparable inference compute-cache scaling law for them. The design space relevant to deployment is narrow. In practice, a KV cache of 4d_h or 8d_h and 1–4× attention compute cover the useful configurations, and our eleven configurations span this range. The law is therefore intended to guide choices within this practical range rather than to extrapolate beyond it.

Q3 Does the observed compute-cache relationship hold for downstream capabilities, particularly long-context tasks such as retrieval and long-context understanding? The downstream accuracy improvements reported in Table 4 are less consistent than the validation-loss improvements.

A Evaluating performance at long context lengths would require long-context training, which exceeds our current computational resources.

Dear Reviewer JqtB,

As the discussion period closes in two days, we want to gently follow up and ask whether you could review our rebuttal. We would be grateful for any further feedback.

Reviewer_jr6kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Scalable Attention (SA), which expands each query head into independently projected sub-heads while sharing the base KV cache, and proposes an additive power-law model L(C,M) = E + A/C^alpha + B/M^beta for trading attention compute against KV-cache size. On eleven 3.3B models trained from scratch, the fitted law has RMSE 0.00066, and SA-2 (hkv=4) reports validation loss 2.1785 versus GQA's 2.1809 while caching 25% as many d_h-values; a B200 kernel reports at most a 5% latency increase at s_a=2. The idea is technically interesting and potentially useful, but the broad scaling and serving-optimality claims need stronger cross-scale, repeated-run, and end-to-end evidence.

Critical strengths
  1. The query-side expansion is a clear and technically motivated mechanism. The paper explains why repetition collapses to a reparameterized single head, gives the full SA forward pass, and shows that independent query projections create distinct attention distributions without materializing additional KV heads.
  2. The compute-cache formulation connects model design to deployment conditions in a useful way. Separating C and M, deriving the iso-loss tradeoff and the cost-weighted optimum, and relating the unit costs to prefix caching, workload shape, speculative decoding, and hardware provides a concrete co-design perspective beyond a single memory-saving metric.
  3. Within its stated 3.3B setting, the study covers eleven SA configurations, matches total parameter counts, reports validation and out-of-domain perplexities plus seven zero-shot tasks, and checks the roofline prediction with a B200 decoding kernel. The reported SA-2/GQA cache-quality comparison and the near-flat s_a=2 latency result are specific, interpretable evidence for the proposed tradeoff.
Critical weaknesses
  1. The scaling-law fit is estimated from eleven configurations at one model size, corpus, tokenizer, training budget, and context length, with no seed-to-seed variability, confidence intervals, or held-out configurations. The RMSE and the reported 0.002–0.005 loss differences therefore do not establish that the additive separability, fitted exponents, or break-even/optimality conclusions persist beyond these points.
  2. The law treats C=s_a and M=h_kv as proxies for serving cost, but the cost terms are not measured for the trained SA/GQA models in an end-to-end setting. The kernel study uses a different h_q=16, h_kv=1, n_q=2 configuration on B200, while the trained models use h_q=24; it does not report prefill latency, total throughput, KV traffic, or the contribution of projections and the rest of the layer across realistic prompt/output workloads.
  3. The main quality comparison is only against one GQA configuration, while the omitted MQA run diverged and MLA/SAS/TPA/MLRA are discussed without matched measurements. SA also changes several architectural details at once (tied K/V, normalization, RoPE/value handling, and retuned FFN width), so the reported advantage does not isolate how much comes from query-side expansion versus these accompanying choices.

Justification of Recommendation

I recommend weak acceptance because the query-expansion mechanism and the demonstrated cache-quality tradeoff constitute a useful contribution within the reported setting. The decisive weaknesses are the single-scale fit without held-out or repeated-run validation, the mismatch between kernel measurements and trained models, and the architectural changes bundled with query expansion. I would rate the paper higher with held-out and cross-scale validation, ablations isolating query expansion, and end-to-end serving measurements for the trained configurations.

Questions for authors
  1. Can the authors report multiple-seed validation losses and downstream results with uncertainty estimates, and test the fitted law on held-out SA configurations, a second model scale, or a different context/training budget? This would distinguish an interpolating fit from a transferable scaling relation.
  2. Can the authors measure end-to-end prefill and decode latency/throughput and peak memory for the actual h_q=24 SA and GQA models, across prompt/output lengths and batch sizes, and compare the measured deployment optimum with the τ-prefill/τ-decode prediction? Please include projection, FFN, and KV-traffic costs rather than attention score FLOPs alone.
  3. Can the authors add controlled baselines that match K/V tying, normalization, RoPE, value projection, parameter count, and total FLOPs, together with stable implementations of the most relevant cache-efficient alternatives? Which portion of the SA-2 gain remains after these factors are held fixed?
6 comments

We thank the reviewer for the helpful suggestions and feedback. We address the concerns below:

W1 The scaling-law fit is estimated from eleven configurations at one model size, corpus, tokenizer, training budget, and context length, with no seed-to-seed variability, confidence intervals, or held-out configurations. The RMSE and the reported 0.002–0.005 loss differences therefore do not establish that the additive separability, fitted exponents, or break-even/optimality conclusions persist beyond these points.

A Regarding run-to-run variability, we previously pretrained SA-8 with both 8d_h and 4d_h KV caches on B200 and H200 GPUs, and the loss difference between runs was within 0.001, which is negligible. Regarding the number of configurations, we agree that more data points would make the fit more robust. However, the design space relevant to deployment is narrow. In practice, a KV cache of 4d_h or 8d_h and 1–4× attention compute cover the useful configurations, and our eleven configurations span this range. The law is therefore intended to guide choices within this practical range rather than to extrapolate beyond it.

W2 The law treats C=s_a and M=h_kv as proxies for serving cost, but the cost terms are not measured for the trained SA/GQA models in an end-to-end setting. The kernel study uses a different h_q=16, h_kv=1, n_q=2 configuration on B200, while the trained models use h_q=24; it does not report prefill latency, total throughput, KV traffic, or the contribution of projections and the rest of the layer across realistic prompt/output workloads.

A Using h_q = 24 for training and h_q = 16 for latency measurement does not affect our conclusions. Our latency analysis targets the arithmetic intensity of decoding, which is determined by the ratio h_q/h_kv rather than by the absolute value of h_q. Measuring unit serving cost, however, requires production infrastructure and a large volume of real user requests, which are not available to us. Similarly, fully characterizing prefill latency, total throughput, KV traffic, and the cost of projections and other layer components under realistic prompt and output workloads would require larger models and substantially more compute than we currently have.

W3 The main quality comparison is only against one GQA configuration, while the omitted MQA run diverged and MLA/SAS/TPA/MLRA are discussed without matched measurements. SA also changes several architectural details at once (tied K/V, normalization, RoPE/value handling, and retuned FFN width), so the reported advantage does not isolate how much comes from query-side expansion versus these accompanying choices.

A We would like to clarify several architectural details. First, GQA also uses RMS normalization, so this component does not favor our method. Second, the RoPE and value handling follows from the KV-tied design. Because keys and values are tied, we apply inverse RoPE to the values to remove the positional rotation introduced on the values. A KV-separate variant is also possible, but KV tying halves the KV cache for the same number of KV heads. To ensure a fair comparison, we therefore evaluate models across multiple KV cache sizes. As shown in Table 4, at each KV cache size, adjusting the attention compute allows our method to outperform GQA. Third, we adjust the FFN width only to keep the total parameter count equal across models, and no additional components are introduced. Finally, due to the time constraints of the rebuttal period, we will include GQA with KV tying in the final version.

Q1 Can the authors report multiple-seed validation losses and downstream results with uncertainty estimates, and test the fitted law on held-out SA configurations, a second model scale, or a different context/training budget? This would distinguish an interpolating fit from a transferable scaling relation.

A Regarding run-to-run variability, we previously pretrained SA-8 with both 8d_h and 4d_h KV caches on B200 and H100 GPUs, and the loss difference between runs was within 0.001, which is negligible. So there is no need to compute multi-seed experiments. Due to limited computing resources, we are unable to run experiments at larger model scales. We will include experiments at smaller scales in the final version to examine whether the observed trends hold across model sizes. As discussed above, our inference compute-cache scaling law is defined at a fixed model size and data size. Extending it across scales would require a new law that jointly models compute, cache, model size, and data size, which we leave to future work.

Q2 Can the authors measure end-to-end prefill and decode latency/throughput and peak memory for the actual h_q=24 SA and GQA models, across prompt/output lengths and batch sizes, and compare the measured deployment optimum with the τ-prefill/τ-decode prediction? Please include projection, FFN, and KV-traffic costs rather than attention score FLOPs alone.

A We sincerely apologize, but we currently lack the kernel infrastructure needed to measure these metrics. Implementing the decoding kernel on B200 GPUs alone required substantial time and effort, and extending this capability to the additional measurements is not feasible at present.

Q3 Can the authors add controlled baselines that match K/V tying, normalization, RoPE, value projection, parameter count, and total FLOPs, together with stable implementations of the most relevant cache-efficient alternatives? Which portion of the SA-2 gain remains after these factors are held fixed?

A In a preliminary ablation, removing RMS normalization increased the loss by approximately 0.005 within the first 5,000 training steps. As noted above, a difference of this size is significant in LLM pre-training, so we stopped the ablation at that point and retained RMS normalization in all subsequent experiments. For other components, such as KV tying, we are unable to obtain results within the rebuttal period.

Dear Reviewer jr6k,

As the discussion period closes in two days, we want to gently follow up and ask whether you could review our rebuttal. We would be grateful for any further feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

After-Rebuttal

Over multiple rounds of discussion with Reviewer iMJS, we found that the reviewer repeatedly raised the same issues even after we had addressed them, and some of the requests were highly unreasonable. For example, on the question of random seeds, the reviewer continued to press the point after we had provided results from independent runs. Our LLM-based assessment also indicates that the reviewer’s comments were likely AI-generated and unusually harsh. We regret that such unjustified low scores are harmful to the community, and we respectfully ask the PC, AC, and other reviewers to look into this case.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.