- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Scalable Attention to increase attention compute without enlarging the KV cache and fits a compute-cache scaling law to guide inference resource allocation.
Critical strengths
- The paper conducts a substantial pretraining experiment, training eleven approximately 3.3B configurations on 98.3B tokens and reporting validation and downstream results.
Critical weaknesses
- Limited novelty. The added linear projections amount to a factorized expansion of query heads. Their value beyond ordinary attention parameterization is not established. This is essentially a LoRA-like addition of trainable parameters, without a clear new insight. Without the normalization before the added projection, its linear map can be absorbed into the original query projection.
- Insufficient baselines. The experiments compare against only one GQA configuration, while the MQA baseline fails to train. The reported quality gains are very marginal, and the limited comparisons do not establish a convincing advantage over existing attention designs.
- Unsupported law claim. Fitting five parameters to eleven configurations at one model scale does not establish a general inference law. Neither the assumed compute-cache separability nor the predicted deployment optimum is independently validated.
Questions for authors
How does performance change if the normalization before the added query projection is removed?
17 comments
Dear Reviewer iMJS,
As the discussion period closes in two days, we want to gently follow up and ask whether you could review our rebuttal. We would be grateful for any further feedback.
Response
Thank you for the response. My main concerns remain unresolved.
-
Limited novelty. The added linear projections amount to a factorized expansion of query heads, closely related to ordinary attention parameterization. Without intermediate normalization, the linear maps can be absorbed into effective query projections. Adding normalization does not by itself establish a substantive architectural novelty. The conceptual advance of this combination remains unclear.
-
Lack of significance. The paper does not establish a clear practical advantage that would justify adopting SA over existing attention designs. GQA can also vary query-head count, head dimension, and the sharing ratio (query heads per KV head). At fixed KV-head count and head dimension, increasing query heads raises this ratio without increasing the KV cache. At fixed total query-projection width, more query heads can instead be obtained by reducing head dimension. The relevant comparison is therefore with GQA configurations at comparable parameter, compute, and cache budgets, with remaining differences explicitly reported, to establish the value of SA's derived heads.
-
Insufficient evidence. The following experiments are needed to support the conclusions.
-
Cache-matched GQA comparisons. Please evaluate GQA with 1, 2, and 4 KV heads to match SA's 2d_h, 4d_h, and 8d_h cache budgets. Alternatively, evaluate SA at GQA's 16d_h budget using compatible head configurations. Adjust query-head counts to compare at similar attention compute, and report parameter counts and remaining compute differences.
-
Additional attention baselines. Please include successful comparisons with MQA, MLA, and GLA under comparable cache and compute budgets, using the same training data, token budget, and evaluation protocol. A single GQA configuration and an unsuccessful MQA run do not establish an advantage over these alternatives. Promising these comparisons for a later version does not resolve the current gap.
-
Robustness and model scale. Quality gains are very marginal and inconsistent. In Table 4, SA-4 at 4d_h improves validation loss by about 0.007, yet average accuracy falls from 59.11% to 58.70%. Please repeat matched comparisons across independent seeds, report means and standard deviations, and test whether the conclusions hold beyond the single 3.3B model scale.
-
Scaling-law evidence. The experimental coverage is too limited to support a general scaling law. The paper fits five parameters to only eleven SA configurations at a single 3.3B model scale. The new held-out fits remain within this same study and do not show that the relationship holds at other model scales or for other attention architectures. Broader experiments are needed to justify the law's claimed generality.
-
I maintain my original recommendation.
We thank the reviewer for the helpful suggestions and feedback. We would first like to clarify a few points that may have been misunderstood and ignored. We address the concerns below:
W1: The added linear projections amount to a factorized expansion of query heads. Their value beyond ordinary attention parameterization is not established. This is essentially a LoRA-like addition of trainable parameters, without a clear new insight. Without the normalization before the added projection, its linear map can be absorbed into the original query projection.
A: As discussed in Sections 3.2 and 3.3, this extension scales attention compute at only a small parameter cost. In contrast, simply repeating the existing components would add compute without adding representational capacity. The additional query projections cannot be absorbed into the original projections, because the total number of sub-heads exceeds the original number of heads. Absorbing them would require duplicating the original projection parameters across sub-heads.
W2: The experiments compare against only one GQA configuration, while the MQA baseline fails to train. The reported quality gains are very marginal, and the limited comparisons do not establish a convincing advantage over existing attention designs.
A: We respectfully note that in LLM pre-training, a loss reduction of 0.005 is already considered significant. SA-4 (4d_h) reduces the loss by 0.007 relative to GQA, which exceeds this threshold. Regarding baselines, we focus on GQA because it is the most widely adopted attention design in current frontier LLMs. MHA and MLA are rarely used in recent large-scale models. We will include MQA results in the final version, although MQA generally underperforms GQA in model quality. More importantly, our contribution goes beyond a lower loss at a fixed budget. Model quality can always be improved by enlarging the KV cache or increasing attention compute, but both raise inference cost. We therefore derive an inference compute-cache scaling law that identifies the lowest-cost configuration for a target model quality. Existing attention designs offer no comparable guidance, and this is a key advantage of our method.
W3: Fitting five parameters to eleven configurations at one model scale does not establish a general inference law. Neither the assumed compute-cache separability nor the predicted deployment optimum is independently validated.
A: Our scaling law is intentionally scoped. We fix the model size and training data, and model only how loss varies with attention compute and KV cache, the two quantities that determine inference cost. In this setting, model size and data size are constants and do not need to appear in the law. Our five-parameter form also matches the complexity of widely used scaling laws such as Chinchilla. Extending the law to vary model and data size jointly would require additional parameters and substantially more training runs, which we leave to future work. Regarding separability, we also fitted non-separable forms that couple compute and cache, and they gave a worse fit than the separable form. Regarding the deployment optimum, the optimal configuration depends on the prefill-decode unit cost of real user requests. This unit cost varies across requests, and estimating it requires large-scale production data that we do not have.
Q: How does performance change if the normalization before the added query projection is removed?
A: In a preliminary ablation, removing RMS normalization increased the loss by approximately 0.005 within the first 5,000 training steps. As noted above, a difference of this size is significant in LLM pre-training, so we stopped the ablation at that point and retained RMS normalization in all subsequent experiments.