acceptodds
Under review as a conference paper at ICLR 2027

Accelerating Attention Projection with Basis Decomposition

Abstract

Attention is a core operation in large language models (LLMs). We present BD Attention (**BDA**), a *lossless algorithmic reformulation* of attention that accelerates its linear projections. BDA is enabled by a simple matrix identity from Basis Decomposition (**BD**), which restructures multi-head projections into a compact form while preserving attention outputs in exact arithmetic. Complementary to I/O-aware system optimizations such as FlashAttention, BDA provides an architecture-agnostic reduction in projection FLOPs. BDA requires only **4s** of offline preparation on DeepSeek-V2-Lite (16B), **with no retraining required**. On modern GPUs, it achieves **34% faster** key/value projections in FP16 and **25% smaller** K/V projection weights. These gains come with a perplexity (PPL) increase of just **0.02%** (FP16) or **0.0004%** (FP32) on DeepSeek-V2-Lite—a negligible effect on model performance. These results position BDA as a theoretically exact method for lossless attention projection acceleration that is complementary to existing engineering-level optimizations. Our code is available at https://anonymous.4open.science/r/Basis-decomp-57B8.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention, Efficient AI, Large Language Model, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QsfpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Basis Decomposition Attention (BDA), a method that exploits low-rank structure and basis selection to accelerate attention projections. BDA can be used both to reformulate projections in existing models and as a parameterization for training models from scratch. The authors also develop a dedicated kernel implementation and evaluate the method on several models and tasks. across several models and tasks demonstrate promising projection-level acceleration.

Critical strengths
  1. A simple and potentially useful reformulation. The method exploits the low-rank structure of attention projections to reduce computational and parameter costs. The idea is intuitive and may offer practical benefits when applied to suitable model components.

  2. Hardware-aware implementation. The authors develop a dedicated kernel to realize the proposed decomposition, going beyond theoretical complexity analysis and demonstrating actual acceleration at the projection level.

  3. Reasonably broad experimental coverage. The experiments cover multiple model configurations and task types, including both inference-oriented evaluations and training from scratch, providing useful evidence for the method's applicability.

Critical weaknesses
  1. Novelty and generality require further clarification. The relationship between BDA and prior basis-based low-rank methods, particularly PIFA, is not sufficiently clear. Although the paper reports throughput comparisons, a more comprehensive comparison would help establish the specific advantages of BDA over existing approaches. In addition, the applicability of BDA depends on the structure of the attention formulation. For example, with standard RoPE, the positional rotation intervenes in the QK product, preventing a straightforward application of the proposed decomposition to the combined projection matrix. The paper discusses handling decoupled RoPE, but this raises questions about how broadly the method can be applied to modern attention architectures. A clearer discussion of these limitations and potential extensions would strengthen the paper.

  2. The practical end-to-end benefit remains unclear. The kernel results demonstrate that BDA can accelerate individual projections, but the overall impact depends on how much runtime these projections account for in the full model. In realistic LLM inference, the bottleneck may also lie in attention computation, MoE communication, or memory movement. The FLOP-based estimates are useful, but do not fully establish the realized end-to-end speedup. A quantitative breakdown of projection costs, together with direct end-to-end measurements, would help clarify the practical value of the method.

Questions for authors
  1. Could the authors provide a more comprehensive comparison with PIFA beyond throughput, including model quality and other relevant trade-offs under comparable settings? It would be helpful to understand which benefits arise specifically from BDA's basis selection strategy and kernel design.

  2. What fraction of total model computation and runtime is attributable to the projections accelerated by BDA, particularly QK projection generation? Based on this fraction, what end-to-end speedup can theoretically be expected? Could the authors also provide direct end-to-end measurements to validate the practical gains?

  3. Can BDA provide additional benefits for Multi-head Latent Attention (MLA), given that MLA already exploits low-dimensional latent representations? Which components might benefit, and what additional savings could be expected beyond those already provided by MLA?

  4. Beyond the attention projections studied in this paper, could BDA be applied to other model components or efficient architectures that exploit low-rank structure? A discussion of promising applications and their limitations would help clarify the broader impact of the method.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer Qsfp — Part 1/4

Thank you for recognizing the potential of the reformulation and asking how its benefits extend to practical inference and other architectures. We have added a controlled PIFA-style comparison covering quality and execution; measured full-model MLA inference; and evaluated continuous BD in low-rank language and diffusion models.

Weakness 1 / Question 1:

Novelty and generality require further clarification. The relationship between BDA and prior basis-based low-rank methods, particularly PIFA, is not sufficiently clear. Although the paper reports throughput comparisons, a more comprehensive comparison would help establish the specific advantages of BDA over existing approaches. In addition, the applicability of BDA depends on the structure of the attention formulation. For example, with standard RoPE, the positional rotation intervenes in the QK product, preventing a straightforward application of the proposed decomposition to the combined projection matrix. The paper discusses handling decoupled RoPE, but this raises questions about how broadly the method can be applied to modern attention architectures. A clearer discussion of these limitations and potential extensions would strengthen the paper.

Could the authors provide a more comprehensive comparison with PIFA beyond throughput, including model quality and other relevant trade-offs under comparable settings? It would be helpful to understand which benefits arise specifically from BDA's basis selection strategy and kernel design.

Reply:

BD uses the basis-and-coefficient representation of ID/PIFA. Applied to compatible QK/VO products, BDA exposes removable identity blocks and aligns their coordinates across heads using shared contiguous bases.

On A100, we compared complete K/V coefficient projections using shared continuous bases against our PIFA-style attention adaptation with per-head QR-selected bases. Both use factors from V2-Lite layers 0, 13, and 26, with 16 heads, latent dimension 512, head dimension 128, and synthetic latent inputs. Timings include input selection, matrix multiplication, addition, and output layout conversion. QR fuses the first three operations; both implementations use the same rounding convention. At 2048 input tokens, continuous BD achieves projection speedups of 6.31×–6.62× in FP16 and 6.29×–6.63× in BF16 over fused QR. The ranges cover K and V projections from layers 0, 13, and 26.

Each case uses 24 paired measurements with alternating order and 100 CUDA Graph replays per measurement. Fused QR also beats its PyTorch batched-matmul implementation in every case. These speedups reflect the combined layout/kernel benefit. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/projection_qr/benchmark.py

Full-model V2-Lite WikiText-2 PPL is shown below. QR quality uses per-head input selection and PyTorch batched matrix multiplication, separately from the fused QR timing implementation:

MethodFP16 PPLBF16 PPL
Native model6.3068256.310469
Per-head QR6.3070036.310180
Shared continuous basis6.3072206.320450

QR has smaller PPL changes; continuous bases offer the execution gains above. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/pifa_experiment.py

With standard RoPE, we preserve Q/K rotation and apply BD to VO. On Qwen2.5-7B, retaining native Q/K, RoPE, V bias and KV sharing, BF16 WikiText-2 PPL is 6.851355→6.859181 (+0.1142%). FLOPs decrease by 3.571% for V projection, or 0.2232% for all QKVO projections. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/qwen_ptb/qwen_vo.py

We will clarify the ID/PIFA relationship, the roles of the attention formulation, layout and kernel, and the standard-RoPE restriction on QK and applicability to VO.

AuthorsOpenReview ↗

Reply to Reviewer Qsfp — Part 2/4

Weakness 2 / Question 2:

The practical end-to-end benefit remains unclear. The kernel results demonstrate that BDA can accelerate individual projections, but the overall impact depends on how much runtime these projections account for in the full model. In realistic LLM inference, the bottleneck may also lie in attention computation, MoE communication, or memory movement. The FLOP-based estimates are useful, but do not fully establish the realized end-to-end speedup. A quantitative breakdown of projection costs, together with direct end-to-end measurements, would help clarify the practical value of the method.

What fraction of total model computation and runtime is attributable to the projections accelerated by BDA, particularly QK projection generation? Based on this fraction, what end-to-end speedup can theoretically be expected? Could the authors also provide direct end-to-end measurements to validate the practical gains?

Reply:

BDA reduces full-model prefill and decode latency in the tested configurations. We integrated shared continuous-basis BDA for DeepSeek-V2-Lite into vLLM 0.11.0 V1 on A100-SXM4-80GB, using BF16 and retaining the native Triton MLA backend, paged latent KV cache, MoE computation, scheduler, and sampling. Measurements use the offline vLLM engine API, with eager execution for both baseline and BDA.

In prefill, the K/V up-projections targeted by BDA account for 2.486% and 2.376% of major model matrix-multiplication FLOPs at 512 and 2048 input tokens, respectively. The K-side projection used to generate attention keys contributes half of each share. BDA removes 25% of these projection FLOPs, giving model-level FLOP reductions of 0.622% and 0.594%, respectively. During absorbed decode, BDA removes computation from the query-to-latent and value-side transforms; the corresponding model-level reductions are shown in the decode table below. The FLOP counts include active MoE computation and the final-token vocabulary projection. FLOP reductions provide a compute-based reference, rather than a bound on latency improvement.

FLOP calculation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_theory.py

For the runtime contribution, separate profiling of the baseline implementation at batch size 1 with 512 input tokens places the K/V up-projections at 1.36% of prefill CUDA kernel time; this measures GPU activity rather than end-to-end wall time.

Profiling code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_profile.py

Prefill. We used kernel block sizes optimized for prefill and measured model-level time to first token (TTFT), including scheduling and sampling. Each configuration uses 24 paired measurements with alternating execution order. The table reports median paired TTFT reductions for the tested configurations with prefill benefits.

Batch / input tokensModel FLOP reductionMeasured TTFT reduction95% confidence interval
1 / 5120.622%0.554%0.157%–0.726%
1 / 20480.594%0.373%0.346%–0.448%
4 / 5120.622%0.466%0.179%–0.547%

Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/prefill_benchmark.py

Decode. For 128 generated tokens, time per output token (TPOT) is measured over the 127 decode steps following prefill. Across four configurations, we measured 64 paired baseline/BDA comparisons with alternating execution order. The FLOP reduction is calculated over these same decode steps using their actual context lengths.

Batch / input tokensModel FLOP reductionMeasured TPOT reduction95% confidence interval
1 / 5120.520%0.436%0.385%–0.514%
1 / 20480.411%0.409%0.331%–0.470%
4 / 5120.520%0.450%0.338%–0.547%
4 / 20480.411%0.499%0.405%–0.553%

All reported TPOT reductions have 95% confidence intervals above zero, calculated by bootstrap resampling of prompt groups. We compute the TPOT reduction for each baseline/BDA pair and report the median of those reductions. All pairs are retained, including different generated sequences that may affect MoE routing.

Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_benchmark.py

These measurements establish model-level gains with the remaining attention, MoE, and memory costs included. We will add the projection-cost breakdown and the full-model prefill and decode measurements to Appendix E, distinguishing the projection-level arithmetic reduction from the realized model-level benefit.

AuthorsOpenReview ↗

Reply to Reviewer Qsfp — Part 3/4

Question 3:

Can BDA provide additional benefits for Multi-head Latent Attention (MLA), given that MLA already exploits low-dimensional latent representations? Which components might benefit, and what additional savings could be expected beyond those already provided by MLA?

Reply:

Yes. BDA acts on the projection weight products that remain after MLA introduces a latent KV representation. Using the paper's row-vector convention and omitting the head index, let uu be the query-projection input and csc_s the cached KV latent vector. MLA's non-RoPE score contribution and the corresponding head's output contribution can be written as

ℓsnon-RoPE=uAcs⊤,A=WqWk⊤,\ell_s^{\mathrm{non\text{-}RoPE}}=uAc_s^\top,\qquad A=W_qW_k^\top,

y=zD,D=WvWo,z=∑sascs,y=zD,\qquad D=W_vW_o,\qquad z=\sum_s a_s c_s,

where asa_s are the attention weights. Weight absorption evaluates uAuA on the query side and applies DD after aggregating the latent vectors.

Our implementation applies column-basis BD to AA and D⊤D^\top. For valid bases, grouping the selected latent coordinates first for notation gives

A=Bq[I,Cq],D⊤=Bv[I,Cv].A=B_q[I,C_q],\qquad D^\top=B_v[I,C_v].

The selected contiguous coordinates can differ between QK and VO. The basis factors BqB_q and Bv⊤B_v^\top replace the corresponding non-RoPE query and output projection weights. Absorbed decode then evaluates

\ell_s^{\mathrm{non\text{-}RoPE}}=\widetilde q c_s^\top,$$ $$y=(z_1+z_2C_v^\top)B_v^\top,$$ where $z_1$ contains the selected VO coordinates and $z_2$ the remaining coordinates. In the query-to-latent transform, the identity block copies $uB_q$ into the selected coordinates; only the remaining coordinates require multiplication by $C_q$. On the value side, the identity block contributes $z_1$ directly, so only $z_2C_v^\top$ requires multiplication. The same factors also support prefill: the K/V up-projections use $c_1+c_2C_q^\top$ and $c_1+c_2C_v^\top$, respectively, with each split following its own selected coordinates. These identities preserve attention scores and outputs in exact arithmetic. They leave the latent KV cache and the decoupled RoPE branch unchanged, which is why BDA is compatible with MLA weight absorption. Appendix E also describes an alternative row-basis construction for the absorbed QK product. For V2-Lite, the latent dimension is 512 and the non-RoPE head dimension and value head dimension are both 128. BDA reduces each affected dense multiplication from $512\times128$ to $384\times128$, or the transposed dimensions, saving **25% of its matrix-multiplication FLOPs**. The identity contribution is handled by direct copying or addition. The table below summarizes the corresponding model-level FLOP reductions and measured latency reductions for full-model V2-Lite inference on A100 (BF16, vLLM). The baseline already uses MLA weight absorption and latent KV caching. Each range covers the same configurations reported in our reply to Question 2 (Part 2/4): three prefill configurations and four decode configurations. | Phase | Model FLOP reduction | Measured model latency reduction | | --- | ---: | ---: | | Prefill (time to first token, TTFT) | 0.594%–0.622% | 0.373%–0.554% | | Decode (time per output token, TPOT) | 0.411%–0.520% | 0.409%–0.499% | These results show additional model-level benefits on top of MLA. The model FLOP reductions provide a compute-based reference, rather than a bound on latency improvement. FLOP calculation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_theory.py Prefill code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/prefill_benchmark.py Decode code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_benchmark.py For numerical quality, full WikiText-2 PPL in the same V2-Lite serving implementation changes from **6.310404 to 6.318902** (+0.135%). Quality evaluation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_quality.py We also evaluated all 60 layers of DeepSeek-V2 in a separate full-model quality experiment. With QK/VO products formed in BF16 and continuous-basis BD applied without coefficient regularization, WikiText-2 PPL changes from **3.820219 to 3.824063** (+0.1006%). This extends the numerical-quality evaluation to the larger MLA model; the latency measurements are for V2-Lite. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/v2_ppl/full_ppl.py We will extend Appendix E to identify the affected prefill and absorbed-decode transforms, quantify their additional arithmetic savings, and include the corresponding model-level measurements.
Reviewer_GeMgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BDA, a reparameterization that reduces attention projection costs while preserving outputs in exact arithmetic. It represents the per-head QK and VO weight products using basis matrices and reconstruction coefficients, allowing identity blocks to be omitted from storage and computation. Its main implementation choice is to align contiguous basis coordinates across heads. The experiments show faster projection operators, small changes in model quality, and additional efficiency gains on already compressed low-rank models.

Critical strengths
  1. I like that the method reduces projection costs without retraining the model or introducing a rank approximation. The derivation appears correct under the stated assumptions, and the idea could be useful in attention architectures where the transformation applies.
  2. The experimental results are encouraging. The authors report projection speedups across several GPU generations, with only small changes in perplexity and downstream accuracy. This provides reasonable evidence that the optimization can work in practice.
Critical weaknesses
  1. I am not convinced by the novelty claim. The underlying representation and its parameter savings are already established in interpolative decomposition and PIFA. The main contribution seems to be applying this representation to attention and arranging the bases so that the heads can be computed efficiently together. The paper should be clearer about this distinction.
  2. My main concern is that it remains unclear how much the method helps in optimized inference. The main attention benchmarks measure individual projection operators. Appendix E explains how BDA could work with MLA weight absorption and latent KV caching, but this setting is not benchmarked. Without those measurements, it is difficult to judge the practical value of the attention contribution.
Questions for authors
  1. What would you say is the main advance over ID and PIFA? It would help to separate the contribution of the attention formulation from the benefits of the basis layout and kernel implementation.
  2. Do the speedups carry over to full-model inference when MLA uses weight absorption and latent KV caching? Could you provide timings for both prefill and decoding?
  3. Appendix G extends the basis search to improve numerical accuracy. What happens if all candidate blocks remain poorly conditioned—is there a fallback? Could you also report the largest per-head reconstruction errors, rather than only the averages in Table 5?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer GeMg — Part 1/3

Thank you for recognizing the correctness of the reformulation and its ability to remove projection work without retraining. We have focused the additional experiments on your three questions: separating the attention/layout contribution from the underlying representation, testing optimized MLA inference, and quantifying worst-case numerical behavior.

Weakness 1 / Question 1:

I am not convinced by the novelty claim. The underlying representation and its parameter savings are already established in interpolative decomposition and PIFA. The main contribution seems to be applying this representation to attention and arranging the bases so that the heads can be computed efficiently together. The paper should be clearer about this distinction.

What would you say is the main advance over ID and PIFA? It would help to separate the contribution of the attention formulation from the benefits of the basis layout and kernel implementation.

Reply:

ID and PIFA establish the basis-and-coefficient representation. BDA develops its use in attention: compatible QK/VO products expose removable identity blocks, shared contiguous basis coordinates align these blocks across heads, and the implementation computes the resulting projections jointly. The attention reformulation identifies where projection computation can be removed while preserving attention outputs in exact arithmetic. The shared basis layout and kernel turn this reduction into efficient joint computation across heads. With different basis coordinates for each head, input selection is head-specific even when the arithmetic reduction is the same.

To evaluate the execution benefit of shared contiguous bases, we compared complete K and V coefficient projections against our PIFA-style attention adaptation with per-head QR-selected bases. On A100, both use factors from V2-Lite layers 0, 13, and 26, with 16 heads, latent dimension 512, head dimension 128, and synthetic latent inputs of 2048 tokens. Timings include input selection, matrix multiplication, addition, and output layout conversion. The QR comparator fuses input selection, matrix multiplication, and addition, and is faster than its PyTorch batched-matmul implementation in every reported case. Both implementations use the same rounding convention. Each case uses 24 paired measurements with alternating execution order and 100 CUDA Graph replays per measurement.

PrecisionContinuous BD speedup over fused QR
FP166.31×–6.62×
BF166.29×–6.63×

The ranges cover K and V projections from the three layers. These measurements quantify the combined execution benefit of the shared contiguous layout and its kernel.

Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/projection_qr/benchmark.py

We will clarify in the introduction how the attention reformulation, shared contiguous bases, and fused kernel contribute to the method, and consistently identify the comparator as our PIFA-style attention adaptation. The identity and the sufficient nonsingularity condition provide the mathematical basis; the contribution is their attention-specific construction and efficient realization.

AuthorsOpenReview ↗

Reply to Reviewer GeMg — Part 2/3

Weakness 2 / Question 2:

My main concern is that it remains unclear how much the method helps in optimized inference. The main attention benchmarks measure individual projection operators. Appendix E explains how BDA could work with MLA weight absorption and latent KV caching, but this setting is not benchmarked. Without those measurements, it is difficult to judge the practical value of the attention contribution.

Do the speedups carry over to full-model inference when MLA uses weight absorption and latent KV caching? Could you provide timings for both prefill and decoding?

Reply:

BDA reduces full-model prefill and decode latency in the tested configurations with MLA weight absorption and latent KV caching. We integrated shared continuous-basis BDA for DeepSeek-V2-Lite into vLLM 0.11.0 V1 on A100-SXM4-80GB, using BF16 and retaining the native Triton MLA backend, paged latent KV cache, MoE computation, scheduler, and sampling. Prefill follows the native vLLM MLA path; during absorbed decoding, the coefficient transform acts on queries while the cache remains latent. This uses the column-basis realization of absorbed MLA; Appendix E describes the alternative row-basis realization. Measurements use the offline vLLM engine API, with eager execution for both baseline and BDA.

We compare measured latency reductions with the corresponding model-level FLOP reductions. The FLOP counts include the model’s major matrix multiplications, active MoE computation, and the final-token vocabulary projection. Both reductions use the baseline as the denominator. FLOP reductions provide a compute-based reference, rather than a bound on latency improvement.

FLOP calculation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_theory.py

Prefill. In the prefill benchmark, we used kernel block sizes optimized for prefill and measured model-level time to first token (TTFT), including scheduling and sampling. Each configuration uses 24 paired measurements with alternating execution order. The table reports median paired TTFT reductions for the tested configurations with prefill benefits.

Batch / input tokensPrefill FLOP reductionMeasured TTFT reduction95% confidence interval
1 / 5120.622%0.554%0.157%–0.726%
1 / 20480.594%0.373%0.346%–0.448%
4 / 5120.622%0.466%0.179%–0.547%

Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/prefill_benchmark.py

Decode. For 128 generated tokens, time per output token (TPOT) is measured over the 127 decode steps following prefill. Across four configurations, we measured 64 paired baseline/BDA comparisons with alternating execution order. The FLOP reduction is calculated over these same decode steps using their actual context lengths.

Batch / input tokensDecode FLOP reductionMeasured TPOT reduction95% confidence interval
1 / 5120.520%0.436%0.385%–0.514%
1 / 20480.411%0.409%0.331%–0.470%
4 / 5120.520%0.450%0.338%–0.547%
4 / 20480.411%0.499%0.405%–0.553%

All reported TPOT reductions have 95% confidence intervals above zero, calculated by bootstrap resampling of prompt groups. For batch 4 with 2048 input tokens, median TPOT decreases from 54.513 to 54.255 ms. We compute the TPOT reduction for each baseline/BDA pair and report the median of those reductions. All pairs are retained, including different generated sequences that may affect MoE routing.

Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_benchmark.py

We further tested chunked prefill with mixed-length, 12-request bursts. At 16 requests/s, total completion time decreases by 0.561% (95% confidence interval: 0.328%–0.771%); a baseline-versus-baseline control gives −0.053% (95% confidence interval: −0.201%–0.072%). This extends the result to the tested mixed-request workload.

Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mixed_serving/mixed_serving_dynamic_fast.py

To evaluate the numerical effect of the serving implementation, we measured full WikiText-2 perplexity in the same engine: 6.310404→6.318902 (+0.135%). We will add the model-level prefill and decode measurements, their execution settings, and the corresponding FLOP reductions to Appendix E.

Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_quality.py

AuthorsOpenReview ↗

Reply to Reviewer GeMg — Part 3/3

Question 3:

Appendix G extends the basis search to improve numerical accuracy. What happens if all candidate blocks remain poorly conditioned—is there a fallback? Could you also report the largest per-head reconstruction errors, rather than only the averages in Table 5?

Reply:

A poorly conditioned basis can require large reconstruction coefficients. Rounding the basis and these coefficients to FP16 or BF16 can then produce substantial reconstruction error, even when the unrounded factors reconstruct the weights accurately. If this issue persists across all candidate bases after offset search, we can use regularized least squares to suppress large coefficients while retaining the selected continuous basis and the same inference structure.

For the column-basis form, write W=[B,R]=B[I,C\*]W=[B,R]=B[I,C_\*], grouping the selected basis columns first for notation. We keep the identity block unchanged and regularize only the coefficients of the remaining columns:

Cλ=arg⁡min⁡C(∣BC−R∣F2+λ∣C∣F2),λ>0.C_\lambda=\arg\min_C\left(\\|BC-R\\|_F^2+\lambda\\|C\\|_F^2\right),\qquad\lambda>0.

This suppresses large coefficients associated with small singular values of BB. Specifically, the inverse factor 1/σi1/\sigma_i is replaced by σi/(σi2+λ)\sigma_i/(\sigma_i^2+\lambda), giving ∣Cλ∣F≤∣R∣F/(2λ)\\|C_\lambda\\|_F\leq\\|R\\|_F/(2\sqrt{\lambda}).

To relate this directly to reconstruction accuracy, let B^=B+ΔB\widehat B=B+\Delta B and C^λ=Cλ+ΔCλ\widehat C_\lambda=C_\lambda+\Delta C_\lambda denote the rounded factors. Their product satisfies

=(BC_\lambda-R)+\Delta B\\,C_\lambda+B\\,\Delta C_\lambda+\Delta B\\,\Delta C_\lambda.$$ Under round-to-nearest with unit roundoff $u$, assuming no overflow or underflow, the factor-rounding contribution therefore gives $$\\|\widehat B\widehat C_\lambda-R\\|_F \leq \underbrace{\\|BC_\lambda-R\\|_F}_{\text{regularization bias}} +(2u+u^2)\\|B\\|_F\\|C_\lambda\\|_F.$$ Here the product of the rounded factors is evaluated exactly to isolate factor-rounding error. For an input $X$, the error in the non-basis output columns, $\\|X(\widehat B\widehat C_\lambda-R)\\|_F$, is bounded by $\\|X\\|_2$ times this bound. Thus, reducing the coefficient norm limits the amplification of factor-rounding errors in the forward computation. Regularization introduces approximation bias: for a full-column-rank basis with $R=BC_\*$, $\\|BC_\lambda-R\\|_F\leq\lambda\\|R\\|_F/(\sigma_{\min}(B)^2+\lambda)$. The choice of $\lambda$ therefore balances this bias against rounding-error amplification; it does not change the condition number of the fixed basis. We will add this optional regularized solve and the rounding-error analysis to Appendix G. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/ridge/ridge_quality.py Thank you for suggesting that we examine the largest per-head reconstruction errors. This motivated us to select the continuous basis by minimizing the maximum reconstruction error across heads, which also improves model-level perplexity. Compared with the original offset search, this criterion reduces the WikiText-2 PPL increase above the baseline by **29.1% in FP16 and 6.4% in BF16**. We evaluated all 864 head-products per precision in V2-Lite without coefficient regularization. The table reports relative Frobenius reconstruction error after rounding factors to the target precision, rather than the original table's mean NMSE: | Continuous-basis policy | FP16 mean / maximum | BF16 mean / maximum | | --- | ---: | ---: | | First block | 1.058% / 8.339% | 4.179% / 8.776% | | Offset search | 0.555% / 3.602% | 3.406% / 8.796% | | Select by maximum error | 0.556% / 2.083% | 3.637% / 7.566% | Both search policies evaluate the 25 offsets $0,16,\ldots,384$. In each layer, they retain one shared offset across heads for QK and one for VO. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/basis_search.py The corresponding full WikiText-2 evaluation shows the task-level effect of these basis-selection policies: | Method | FP16 PPL | BF16 PPL | | --- | ---: | ---: | | Baseline | 6.306825 | 6.310469 | | Original offset search | 6.307220 | 6.320450 | | Selection by maximum error | 6.307105 | 6.319813 | The relative reduction in PPL increase is calculated as $$\frac{\mathrm{PPL}_{\mathrm{offset}}-\mathrm{PPL}_{\mathrm{max\text{-}error}}}{\mathrm{PPL}_{\mathrm{offset}}-\mathrm{PPL}_{\mathrm{baseline}}}.$$ Basis selection uses weight reconstruction errors, not test PPL. We will add these maximum per-head errors and the corresponding task-level results to Appendix G. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/lite_quality.py
Reviewer_iEw3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Basis Decomposition (BD), which represents a rank-r matrix as r of its own rows or columns plus a coefficient matrix (Sec. 3.1, Eq. 2). It applies BD to the per-head QK and VO weight products in attention to get BD Attention (BDA). A shared contiguous basis (first or last d_h, chosen by residual) lets all heads be computed in one GEMM (Eq. 8, Alg. 2). On DeepSeek-V2-Lite, BDA increases WikiText2 perplexity by 0.02% in FP16, and a fused Triton kernel gives about 1.34× k_proj speedup across consumer Ampere/Ada/Blackwell cards and an A6000 (Fig. 3). Further experiments train BDA from scratch on IWSLT’14 (Table 4) and apply BD on top of low-rank-pruned LLaMA2 (Table 3).

Critical strengths

It’s exact, and the precision study backs that up. Table 5 and Table 6 show reconstruction error and perplexity change per format. The Residual-min and offset-search variants (Section 3.2, App. G) are a sensible, cheap answer to conditioning issues. Zero-shot results in Table 7 sit inside one standard error. I don’t doubt the method does what it says at the operator level.

The shared contiguous basis is the actually useful engineering idea here. The PIFA-style per-head QR comparison in Tables 8 and 9 makes the point well. Per-head basis selection runs slower than plain MHA, while aligned bases get close to the theoretical bound.

Critical weaknesses

The headline numbers don’t reflect real model-level gains, and the paper doesn’t say so up front. The abstract and Fig. 3b report k_proj speedup. Table 10 in Appendix H estimates attention speedups of 1.87% to 3.77%, and that’s attention only, not the full model with MoE FFNs. The measured setting also assumes explicitly expanded keys and values. Appendix E admits that under MLA weight absorption, which is how MLA decode is usually served, the savings move to the query and output side. No numbers are given for that case. The 25% ratio itself comes from d = 512 being the KV latent dimension (Sec. 4.2, Table 8 caption). For standard MHA or GQA with d = 4096 and d_h = 128, the same algebra gives about 3% on the projections. That’s never stated, and I think readers will walk away overestimating applicability.

Novelty is thin. Appendix B says BD is a special case of interpolative decomposition and CUR. PIFA already uses a basis-and-coefficient representation for low-rank layers. Theorem 3.1 is a standard measure-zero argument, and leaning on Mandt et al. to justify absolute continuity of trained weights is a stretch (though in practice the residual check makes the theorem mostly irrelevant). What’s left is the head-aligned contiguous basis and the kernel. That’s a fine workshop contribution, but it doesn’t match “a new algorithmic foundation” (Sec. 1).

Questions for authors

(Clarification) Can you report end-to-end prefill and decode latency for DeepSeek-V2-Lite inside an actual serving stack, with the absorbed MLA path BD is meant to run in? If the real-world gain is near the Table 10 estimates, I’d want the abstract rewritten. If it’s meaningfully higher, I’d raise my score.

(Clarification) All GPU results are on RTX cards and an A6000. How does the fused kernel compare against cuBLAS on A100/H100 at these shapes, where these GEMMs are smaller and could be memory-bound? I’m not convinced the 1.33× FLOP reduction carries over there.

(Clarification) Table 4 shows BDA beating MHA by about 1 BLEU at LR scale 1. How many seeds were run? Without variance I can’t tell if “comparable” is supported. Appendix D’s claim that LLM pipelines can migrate “at essentially no cost” goes well beyond a 6-layer, 4-head translation model.

(Suggestion for future work) Please quantify savings for GQA/MQA models and standard MHA models with RoPE. Appendix E says QK isn’t exact under vanilla RoPE, so for LLaMA-style models only VO applies, and the ratio is small.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer iEw3 — Part 1/4

Thank you for recognizing the exact reformulation and the value of the shared contiguous basis. We have completed new A100 benchmarks, full-model prefill and decode measurements in vLLM with latent KV caching, and five-seed training experiments. These results support the paper's projection-acceleration and comparable-quality conclusions while quantifying their model-level impact.

Weakness 1:

The headline numbers don’t reflect real model-level gains, and the paper doesn’t say so up front. The abstract and Fig. 3b report k_proj speedup. Table 10 in Appendix H estimates attention speedups of 1.87% to 3.77%, and that’s attention only, not the full model with MoE FFNs. The measured setting also assumes explicitly expanded keys and values. Appendix E admits that under MLA weight absorption, which is how MLA decode is usually served, the savings move to the query and output side. No numbers are given for that case. The 25% ratio itself comes from d = 512 being the KV latent dimension (Sec. 4.2, Table 8 caption). For standard MHA or GQA with d = 4096 and d_h = 128, the same algebra gives about 3% on the projections. That’s never stated, and I think readers will walk away overestimating applicability.

Reply:

The abstract's 34% figure refers explicitly to K/V projections. We will add the end-to-end results to the abstract and clearly distinguish them from the projection-level speedups.

At an input length of 4096 tokens, BD reduces the total major model matmul FLOPs of V2-Lite by 0.561% (FLOP calculation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/prefill/compute_theory.py). With the same scaled dot-product attention (SDPA) and MLP optimizations applied to the baseline and BD implementations, full-model prefill latency decreases by 0.448% (prefill benchmark code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/prefill/run_packed_model.py). The measured latency reduction is approximately 80% of the theoretical FLOP reduction.

These measurements relate the projection-level saving to its model-level effect on V2-Lite. To examine the benefit in a model with more eligible low-rank computation, we first compressed the projection matrices of LLaMA-7B into low-rank factors, retaining 90% of their original parameters, and then measured the additional benefit of BD on A100. In this model, continuous BD on all 224 projections reduces prefill latency, including KV-cache creation, by 10.89%–13.24% across four long-input configurations (LLaMA timing code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/lowrank/cached_inference.py). PPL changes from 8.895749 to 8.904175 (+0.0947%) (PPL evaluation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/lowrank/llama_full.py). These results measure BD's additional benefit over the same low-rank baseline, with latency measured for prefill. The experiment extends the low-rank application already studied in Table 3.

The 25% projection-level saving follows from dh/d=128/512d_h/d = 128/512 in V2-Lite; it is not a model-wide or architecture-independent percentage. With d=4096d = 4096 and dh=128d_h = 128, the corresponding saving is 3.125% for an eligible projection.

We will add the absorbed-MLA prefill and decode measurements to Appendix E and state the architecture-dependent savings and standard-RoPE/GQA applicability conditions in Section 4.2 and Appendix E. The detailed measurements and architecture-specific calculations are provided in our response to Question 1 (Part 3/4) and the suggestion for future work (Part 4/4), respectively.

AuthorsOpenReview ↗

Reply to Reviewer iEw3 — Part 2/4

Weakness 2:

Novelty is thin. Appendix B says BD is a special case of interpolative decomposition and CUR. PIFA already uses a basis-and-coefficient representation for low-rank layers. Theorem 3.1 is a standard measure-zero argument, and leaning on Mandt et al. to justify absolute continuity of trained weights is a stretch (though in practice the residual check makes the theorem mostly irrelevant). What’s left is the head-aligned contiguous basis and the kernel. That’s a fine workshop contribution, but it doesn’t match “a new algorithmic foundation” (Sec. 1).

Reply:

As discussed in Appendix B, BD shares the basis-and-coefficient representation underlying ID/CUR and PIFA. Our contribution is the attention reformulation built on this representation: identifying compatible QK/VO products, exposing identity blocks whose computation can be removed, and aligning their basis coordinates across heads for efficient joint execution.

Theorem 3.1 provides a justification for the shared contiguous basis used in BDA. Under the stated distributional assumptions, the same prescribed contiguous indices form a valid basis across heads almost surely. This allows us to choose the basis layout for efficient execution. Although the nonsingularity argument is classical, this consequence is directly relevant to our design. Residual checks complement the result: a valid basis can still be poorly conditioned, so we also evaluate reconstruction errors in finite precision.

Regarding Mandt et al., we use their local SGD model to motivate the distributional assumption. We will clarify that this does not establish the assumption for every trained checkpoint. The exact BD reformulation itself requires a valid selected basis, regardless of how the weights were trained.

To evaluate the execution benefit of shared contiguous bases, we compared complete K and V coefficient projections against our PIFA-style attention adaptation with per-head QR-selected bases. On A100, both use factors from V2-Lite layers 0, 13, and 26, with 16 heads, latent dimension 512, head dimension 128, and synthetic latent inputs of 2048 tokens. Timings include input selection, matrix multiplication, addition, and output layout conversion. The QR comparator fuses input selection, matrix multiplication, and addition, and is faster than its PyTorch batched-matmul implementation in every reported case. Both implementations use the same rounding convention. Each case uses 24 paired measurements with alternating execution order and 100 CUDA Graph replays per measurement. The following ranges cover the K and V projections from the three layers:

PrecisionProjection speedup
FP166.31×–6.62×
BF166.29×–6.63×

Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/projection_qr/benchmark.py

We will revise “a new algorithmic foundation” to “an exact attention reformulation with shared contiguous bases and fused execution” in the introduction, so that the stated contribution directly reflects the formulation and implementation developed in this work.

AuthorsOpenReview ↗

Reply to Reviewer iEw3 — Part 3/4

Question 1:

(Clarification) Can you report end-to-end prefill and decode latency for DeepSeek-V2-Lite inside an actual serving stack, with the absorbed MLA path BD is meant to run in? If the real-world gain is near the Table 10 estimates, I’d want the abstract rewritten. If it’s meaningfully higher, I’d raise my score.

Reply:

We implemented shared continuous-basis BDA in vLLM 0.11.0 V1 on A100-SXM4-80GB, BF16, retaining the native Triton MLA backend, paged latent KV cache, MoE computation, scheduler, and sampling. Prefill follows the native vLLM MLA path; during absorbed decoding, the coefficient transform acts on queries while the cache stays latent. This uses the column-basis realization of absorbed MLA; Appendix E describes the alternative row-basis realization. We measured requests through the offline vLLM engine API. Both baseline and BDA use eager execution.

Table 10 estimates attention-level savings. For the end-to-end measurements requested here, we compute the corresponding model-level FLOP reductions under absorbed MLA and compare them with measured latency reductions. The FLOP counts include the model's major matrix multiplications, including active MoE computation and the final-token vocabulary projection. Both reductions use the baseline as the denominator. FLOP reductions provide a compute-based reference, rather than a bound on latency improvement. FLOP calculation code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_theory.py

Prefill latency. In the prefill benchmark, we used kernel block sizes optimized for prefill and measured model-level time to first token (TTFT), including scheduling and sampling. Each configuration uses 24 balanced pairs:

Batch / input tokensBaseline → BDA TTFT (ms)Prefill FLOP reductionMeasured TTFT reduction95% confidence interval
1 / 51252.898 → 52.6370.622%0.554%0.157%–0.726%
1 / 204891.775 → 91.2990.594%0.373%0.346%–0.448%
4 / 51289.164 → 88.8530.622%0.466%0.179%–0.547%

The TTFT results identify beneficial configurations; we do not claim an improvement at every batch/context size. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/prefill_benchmark.py

Decode latency. For 128 generated tokens, time per output token (TPOT) is measured over the 127 decode steps following prefill. Across four configurations, we measured 64 paired baseline/BDA comparisons with balanced execution order. The FLOP reduction is calculated over these same decode steps using their actual context lengths:

Batch / input tokensDecode FLOP reductionMeasured TPOT reduction95% confidence interval
1 / 5120.520%0.436%0.385%–0.514%
1 / 20480.411%0.409%0.331%–0.470%
4 / 5120.520%0.450%0.338%–0.547%
4 / 20480.411%0.499%0.405%–0.553%

All reported TPOT reductions have 95% confidence intervals above zero, calculated by bootstrap resampling of prompt groups. For batch 4 with 2048 input tokens, median TPOT is 54.513→54.255 ms. We compute the TPOT reduction for each baseline/BDA pair and report the median of those reductions. All pairs are retained, including different generated sequences that may affect MoE routing. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_benchmark.py

Mixed requests. We further tested chunked prefill with mixed-length, 12-request bursts. At 16 requests/s, total completion time decreases by 0.561% (95% confidence interval: 0.328%–0.771%); a baseline-versus-baseline control gives −0.053%, with a confidence interval spanning zero. Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mixed_serving/mixed_serving_dynamic_fast.py

Numerical quality. Full WikiText-2 PPL in the serving engine changes from 6.310404 to 6.318902 (+0.135%). Code: https://anonymous.4open.science/r/Basis-decomp-57B8/experiments/mla/serving_quality.py

These measurements show that BDA provides additional model-level benefits with absorbed MLA and latent KV caching. We will revise the abstract to state the projection-level scope of the headline speedups and add the measured model-level prefill and decode results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.