- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Basis Decomposition Attention (BDA), a method that exploits low-rank structure and basis selection to accelerate attention projections. BDA can be used both to reformulate projections in existing models and as a parameterization for training models from scratch. The authors also develop a dedicated kernel implementation and evaluate the method on several models and tasks. across several models and tasks demonstrate promising projection-level acceleration.
Critical strengths
-
A simple and potentially useful reformulation. The method exploits the low-rank structure of attention projections to reduce computational and parameter costs. The idea is intuitive and may offer practical benefits when applied to suitable model components.
-
Hardware-aware implementation. The authors develop a dedicated kernel to realize the proposed decomposition, going beyond theoretical complexity analysis and demonstrating actual acceleration at the projection level.
-
Reasonably broad experimental coverage. The experiments cover multiple model configurations and task types, including both inference-oriented evaluations and training from scratch, providing useful evidence for the method's applicability.
Critical weaknesses
-
Novelty and generality require further clarification. The relationship between BDA and prior basis-based low-rank methods, particularly PIFA, is not sufficiently clear. Although the paper reports throughput comparisons, a more comprehensive comparison would help establish the specific advantages of BDA over existing approaches. In addition, the applicability of BDA depends on the structure of the attention formulation. For example, with standard RoPE, the positional rotation intervenes in the QK product, preventing a straightforward application of the proposed decomposition to the combined projection matrix. The paper discusses handling decoupled RoPE, but this raises questions about how broadly the method can be applied to modern attention architectures. A clearer discussion of these limitations and potential extensions would strengthen the paper.
-
The practical end-to-end benefit remains unclear. The kernel results demonstrate that BDA can accelerate individual projections, but the overall impact depends on how much runtime these projections account for in the full model. In realistic LLM inference, the bottleneck may also lie in attention computation, MoE communication, or memory movement. The FLOP-based estimates are useful, but do not fully establish the realized end-to-end speedup. A quantitative breakdown of projection costs, together with direct end-to-end measurements, would help clarify the practical value of the method.
Questions for authors
-
Could the authors provide a more comprehensive comparison with PIFA beyond throughput, including model quality and other relevant trade-offs under comparable settings? It would be helpful to understand which benefits arise specifically from BDA's basis selection strategy and kernel design.
-
What fraction of total model computation and runtime is attributable to the projections accelerated by BDA, particularly QK projection generation? Based on this fraction, what end-to-end speedup can theoretically be expected? Could the authors also provide direct end-to-end measurements to validate the practical gains?
-
Can BDA provide additional benefits for Multi-head Latent Attention (MLA), given that MLA already exploits low-dimensional latent representations? Which components might benefit, and what additional savings could be expected beyond those already provided by MLA?
-
Beyond the attention projections studied in this paper, could BDA be applied to other model components or efficient architectures that exploit low-rank structure? A discussion of promising applications and their limitations would help clarify the broader impact of the method.
5 comments
Reply to Reviewer Qsfp — Part 2/4
Weakness 2 / Question 2:
The practical end-to-end benefit remains unclear. The kernel results demonstrate that BDA can accelerate individual projections, but the overall impact depends on how much runtime these projections account for in the full model. In realistic LLM inference, the bottleneck may also lie in attention computation, MoE communication, or memory movement. The FLOP-based estimates are useful, but do not fully establish the realized end-to-end speedup. A quantitative breakdown of projection costs, together with direct end-to-end measurements, would help clarify the practical value of the method.
What fraction of total model computation and runtime is attributable to the projections accelerated by BDA, particularly QK projection generation? Based on this fraction, what end-to-end speedup can theoretically be expected? Could the authors also provide direct end-to-end measurements to validate the practical gains?
Reply:
BDA reduces full-model prefill and decode latency in the tested configurations. We integrated shared continuous-basis BDA for DeepSeek-V2-Lite into vLLM 0.11.0 V1 on A100-SXM4-80GB, using BF16 and retaining the native Triton MLA backend, paged latent KV cache, MoE computation, scheduler, and sampling. Measurements use the offline vLLM engine API, with eager execution for both baseline and BDA.
In prefill, the K/V up-projections targeted by BDA account for 2.486% and 2.376% of major model matrix-multiplication FLOPs at 512 and 2048 input tokens, respectively. The K-side projection used to generate attention keys contributes half of each share. BDA removes 25% of these projection FLOPs, giving model-level FLOP reductions of 0.622% and 0.594%, respectively. During absorbed decode, BDA removes computation from the query-to-latent and value-side transforms; the corresponding model-level reductions are shown in the decode table below. The FLOP counts include active MoE computation and the final-token vocabulary projection. FLOP reductions provide a compute-based reference, rather than a bound on latency improvement.
FLOP calculation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_theory.py
For the runtime contribution, separate profiling of the baseline implementation at batch size 1 with 512 input tokens places the K/V up-projections at 1.36% of prefill CUDA kernel time; this measures GPU activity rather than end-to-end wall time.
Profiling code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_profile.py
Prefill. We used kernel block sizes optimized for prefill and measured model-level time to first token (TTFT), including scheduling and sampling. Each configuration uses 24 paired measurements with alternating execution order. The table reports median paired TTFT reductions for the tested configurations with prefill benefits.
| Batch / input tokens | Model FLOP reduction | Measured TTFT reduction | 95% confidence interval |
|---|---|---|---|
| 1 / 512 | 0.622% | 0.554% | 0.157%–0.726% |
| 1 / 2048 | 0.594% | 0.373% | 0.346%–0.448% |
| 4 / 512 | 0.622% | 0.466% | 0.179%–0.547% |
Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/prefill_benchmark.py
Decode. For 128 generated tokens, time per output token (TPOT) is measured over the 127 decode steps following prefill. Across four configurations, we measured 64 paired baseline/BDA comparisons with alternating execution order. The FLOP reduction is calculated over these same decode steps using their actual context lengths.
| Batch / input tokens | Model FLOP reduction | Measured TPOT reduction | 95% confidence interval |
|---|---|---|---|
| 1 / 512 | 0.520% | 0.436% | 0.385%–0.514% |
| 1 / 2048 | 0.411% | 0.409% | 0.331%–0.470% |
| 4 / 512 | 0.520% | 0.450% | 0.338%–0.547% |
| 4 / 2048 | 0.411% | 0.499% | 0.405%–0.553% |
All reported TPOT reductions have 95% confidence intervals above zero, calculated by bootstrap resampling of prompt groups. We compute the TPOT reduction for each baseline/BDA pair and report the median of those reductions. All pairs are retained, including different generated sequences that may affect MoE routing.
Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_benchmark.py
These measurements establish model-level gains with the remaining attention, MoE, and memory costs included. We will add the projection-cost breakdown and the full-model prefill and decode measurements to Appendix E, distinguishing the projection-level arithmetic reduction from the realized model-level benefit.
Reply to Reviewer Qsfp — Part 3/4
Question 3:
Can BDA provide additional benefits for Multi-head Latent Attention (MLA), given that MLA already exploits low-dimensional latent representations? Which components might benefit, and what additional savings could be expected beyond those already provided by MLA?
Reply:
Yes. BDA acts on the projection weight products that remain after MLA introduces a latent KV representation. Using the paper's row-vector convention and omitting the head index, let be the query-projection input and the cached KV latent vector. MLA's non-RoPE score contribution and the corresponding head's output contribution can be written as
where are the attention weights. Weight absorption evaluates on the query side and applies after aggregating the latent vectors.
Our implementation applies column-basis BD to and . For valid bases, grouping the selected latent coordinates first for notation gives
The selected contiguous coordinates can differ between QK and VO. The basis factors and replace the corresponding non-RoPE query and output projection weights. Absorbed decode then evaluates
\ell_s^{\mathrm{non\text{-}RoPE}}=\widetilde q c_s^\top,$$ $$y=(z_1+z_2C_v^\top)B_v^\top,$$ where $z_1$ contains the selected VO coordinates and $z_2$ the remaining coordinates. In the query-to-latent transform, the identity block copies $uB_q$ into the selected coordinates; only the remaining coordinates require multiplication by $C_q$. On the value side, the identity block contributes $z_1$ directly, so only $z_2C_v^\top$ requires multiplication. The same factors also support prefill: the K/V up-projections use $c_1+c_2C_q^\top$ and $c_1+c_2C_v^\top$, respectively, with each split following its own selected coordinates. These identities preserve attention scores and outputs in exact arithmetic. They leave the latent KV cache and the decoupled RoPE branch unchanged, which is why BDA is compatible with MLA weight absorption. Appendix E also describes an alternative row-basis construction for the absorbed QK product. For V2-Lite, the latent dimension is 512 and the non-RoPE head dimension and value head dimension are both 128. BDA reduces each affected dense multiplication from $512\times128$ to $384\times128$, or the transposed dimensions, saving **25% of its matrix-multiplication FLOPs**. The identity contribution is handled by direct copying or addition. The table below summarizes the corresponding model-level FLOP reductions and measured latency reductions for full-model V2-Lite inference on A100 (BF16, vLLM). The baseline already uses MLA weight absorption and latent KV caching. Each range covers the same configurations reported in our reply to Question 2 (Part 2/4): three prefill configurations and four decode configurations. | Phase | Model FLOP reduction | Measured model latency reduction | | --- | ---: | ---: | | Prefill (time to first token, TTFT) | 0.594%–0.622% | 0.373%–0.554% | | Decode (time per output token, TPOT) | 0.411%–0.520% | 0.409%–0.499% | These results show additional model-level benefits on top of MLA. The model FLOP reductions provide a compute-based reference, rather than a bound on latency improvement. FLOP calculation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_theory.py Prefill code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/prefill_benchmark.py Decode code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_benchmark.py For numerical quality, full WikiText-2 PPL in the same V2-Lite serving implementation changes from **6.310404 to 6.318902** (+0.135%). Quality evaluation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_quality.py We also evaluated all 60 layers of DeepSeek-V2 in a separate full-model quality experiment. With QK/VO products formed in BF16 and continuous-basis BD applied without coefficient regularization, WikiText-2 PPL changes from **3.820219 to 3.824063** (+0.1006%). This extends the numerical-quality evaluation to the larger MLA model; the latency measurements are for V2-Lite. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/v2_ppl/full_ppl.py We will extend Appendix E to identify the affected prefill and absorbed-decode transforms, quantify their additional arithmetic savings, and include the corresponding model-level measurements.
Reply to Reviewer Qsfp — Part 1/4
Thank you for recognizing the potential of the reformulation and asking how its benefits extend to practical inference and other architectures. We have added a controlled PIFA-style comparison covering quality and execution; measured full-model MLA inference; and evaluated continuous BD in low-rank language and diffusion models.
Weakness 1 / Question 1:
Reply:
BD uses the basis-and-coefficient representation of ID/PIFA. Applied to compatible QK/VO products, BDA exposes removable identity blocks and aligns their coordinates across heads using shared contiguous bases.
On A100, we compared complete K/V coefficient projections using shared continuous bases against our PIFA-style attention adaptation with per-head QR-selected bases. Both use factors from V2-Lite layers 0, 13, and 26, with 16 heads, latent dimension 512, head dimension 128, and synthetic latent inputs. Timings include input selection, matrix multiplication, addition, and output layout conversion. QR fuses the first three operations; both implementations use the same rounding convention. At 2048 input tokens, continuous BD achieves projection speedups of 6.31×–6.62× in FP16 and 6.29×–6.63× in BF16 over fused QR. The ranges cover K and V projections from layers 0, 13, and 26.
Each case uses 24 paired measurements with alternating order and 100 CUDA Graph replays per measurement. Fused QR also beats its PyTorch batched-matmul implementation in every case. These speedups reflect the combined layout/kernel benefit. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/projection_qr/benchmark.py
Full-model V2-Lite WikiText-2 PPL is shown below. QR quality uses per-head input selection and PyTorch batched matrix multiplication, separately from the fused QR timing implementation:
QR has smaller PPL changes; continuous bases offer the execution gains above. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/pifa_experiment.py
With standard RoPE, we preserve Q/K rotation and apply BD to VO. On Qwen2.5-7B, retaining native Q/K, RoPE, V bias and KV sharing, BF16 WikiText-2 PPL is 6.851355→6.859181 (+0.1142%). FLOPs decrease by 3.571% for V projection, or 0.2232% for all QKVO projections. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/qwen_ptb/qwen_vo.py
We will clarify the ID/PIFA relationship, the roles of the attention formulation, layout and kernel, and the standard-RoPE restriction on QK and applicability to VO.