acceptodds
Withdrawn from ICLR 2027

Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Efficient autoregressive generation is increasingly limited by the memory footprint of the key-value (KV) cache, which grows linearly with sequence length, batch size, and model depth. Existing approaches such as Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and Tensor Product Attention (TPA) reduce this cost through different trade-offs, but often sacrifice expressivity, require additional positional-encoding machinery, or impose restrictive low-rank structure. We propose Kronecker Product Attention (KPA), which represents key and value activations as sums of Kronecker products of compact, token-dependent matrix factors. KPA strictly generalizes TPA: rank-one vector-outer-product factorization arises as a degenerate Kronecker split, while non-degenerate matrix factors can represent richer block-structured patterns. We further show that a structured rotary embedding operator matched to the Kronecker representation preserves the relative-position property of RoPE while rotating only the compact factors, removing the need for a decoupled RoPE branch as in MLA. Empirically, KPA matches or exceeds TPA, MHA, MQA, GQA, and MLA across four model scales and nine downstream benchmarks under identical training conditions, while achieving up to 10× KV cache compression. The Kronecker head split is a tuneable hyperparameter that allows practitioners to navigate the compression–accuracy trade-off at design time without architectural changes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Kronecker Product, KV Cache Compression, Efficient Autoregressive Generation, Memory-Efficient Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9BYGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Kronecker Product Attention (KPA), which represents token-dependent QKV matrices as sums of Kronecker products of compact matrix factors. By coupling head and feature indices within each factor, KPA allows different attention heads to have more diverse feature directions than standard Tensor Product Attention (TPA) at a given number of factor terms. The paper derives exact factorized attention computations and a compatible rotary operator, and reports improved average zero-shot accuracy over several attention baselines across four model scales, together with 6–10× fewer KV-cache entries than MHA.

Critical strengths
  1. KPA is a meaningful extension of TPA. By increasing the dimensions of the Kronecker factors, it provides a way to increase representational capacity while retaining the benefits of structured factorization. I am interested in the possibility of exploiting the Kronecker structure to reduce computation and compute attention scores efficiently, although some of these benefits may already be attributable to TPA itself.

  2. The paper examines how different factor allocations affect performance under comparable KV cache budgets. Although the choice of factor dimensions still appears to require empirical tuning, the reported results suggest that KPA can remain competitive across different allocations, indicating certain robustness and practicality of the method.

Critical weaknesses
  1. The practical inference efficiency of KPA is not sufficiently established. The paper reports substantial reductions in the number of stored KV cache entries, but this does not directly establish comparable reductions in actual GPU memory usage or improvements in end-to-end inference speed. The factorized representation introduces its own computation and memory-access patterns, and the practical outcome will depend on how these operations are implemented. In particular, the paper does not provide end-to-end prefill and decode latency or throughput measurements, nor an analysis of kernel fusion, intermediate tensor materialization, and opportunities for parallel execution or overlap. The paper itself acknowledges the need for efficient factored decoding kernels.

  2. The empirical evidence does not yet convincingly establish the breadth of KPA's performance advantages. The experiments provide preliminary evidence that KPA can outperform TPA. However, all experiments are conducted using Llama-based models and evaluated on a relatively narrow set of downstream benchmarks. Specially, alhough the reported accuracy of MHA, MQA, TPA are imported from the TPA paper, the reported accuracy of MQA is nearly identicaly or even better than that of MHA. This raises questions about the reported results and makes me less convinced by the claim that KPA substantially outperforms MHA and other common baselines. More comprehensive evaluations, including validation perplexity and tests on more advanced models and longer sequences, would strengthen the empirical evidence.

Questions for authors
  1. The paper only report KV savings of KPA compared to MHA. Can the authors also report the comparisons to MQA, GQA, and MLA? Additionally, can they provide a preliminary analysis of the potential benefits of kernel fusion and reduced memory traffic?

  2. Can the authors evaluate KPA on more advanced model architectures and report validation perplexity? Evaluations on longer sequences would also help assess whether the method preserves language modeling performance.

  3. Since reducing KV cache storage is one of the main motivations for KPA, is it necessary to apply KPA to Q? Ablations comparing these design choices would help clarify this point.

  4. Can the authors further analyze how the Kronecker-separable RoPE design affects performance?

Reviewer_1SjmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Kronecker Product Attention (KPA), which represents queries, keys, and values using sums of Kronecker products of token-dependent matrix factors. Unlike standard TPA, each factor combines part of the head index with part of the feature index. The paper derives attention computation directly from these factors and a compatible rotary operator. Experiments compare zero-shot accuracy across four model scales and report analytical KV-cache storage.

Critical strengths
  1. KPA provides a useful alternative to the factor grouping used in TPA. Its matrix factors allow different heads to receive different feature directions within a single term. The paper clearly explains this distinction and its relationship to standard TPA.
  2. The paper derives exact attention computation from the cached factors and specifies a rotary operator that preserves the relative-position identity. The experiments across four model scales also provide evidence that KPA can maintain competitive accuracy with compact representations.
Critical weaknesses
  1. The practical benefits for long-context inference are not experimentally established. The reported 6–10× compression measures the number of cached scalar entries, rather than actual GPU memory usage. The paper does not report measured memory, decoding latency, or throughput, and the accuracy evaluation does not include long-context tasks. It is therefore unclear how much practical benefit remains after accounting for factor computation and temporary memory. Measurements of memory, speed, and accuracy across context lengths are needed to assess the proposed method.

  2. The comparison with the closest alternatives is incomplete. TPA already supports factor caching and attention computation from the factors, while higher-order TPA also discusses factor-level rotary encoding and includes a small-model evaluation. KPA’s main distinction is how head and feature indices are grouped, but the experiments compare only against standard TPA. Moreover, the methods are not compared under a matched KV-cache budget, and Appendix B does not report the cache size of the TPA baseline used in the accuracy tables. The current results therefore do not establish whether the proposed grouping offers a better accuracy–storage trade-off.

Questions for authors
  1. Can you report actual GPU memory usage, decoding latency or throughput, and long-context accuracy across several context lengths using factor-only KV caching?
  2. Can you report the actual cache sizes and parameter counts of the compared configurations, and compare KPA with standard and higher-order TPA under matched KV-cache and training budgets?
Reviewer_ytYZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Kronecker Product Attention (KPA), an extension of Tensor Product Attention that enables more expressivity than TPA while maintaining a strong compression rate compared to standard MHA. The authors carefully describe how they leverage the Kronecker product, and proceed to implement a relative positional embedding on top of their mechanism. They then validate their new inductive bias for several shape configurations, across 4 sizes from ~350M to ~1.5B parameters, reporting improvements in both performance compared to TPA and KV cache usage compared to MHA.

Critical strengths
  • Elegant and logical contribution: whether the paper's framing is really clear or the extension is genuinely straightforward, extending TPA through higher-dimensional Kronecker products feels like a natural and elegant extension. It should allow more flexibility than TPA while retaining the KV cache compression benefits. It is nicely motivated, and the mathematical layout is sensible.
Critical weaknesses
  • Suspicious results and Missing experiments: My main concern about this paper is the general validity and breadth of the experimental results. First, the authors only report zero-shot evaluation scores, and do not report training dynamics or validation perplexity. These zero-shot downstream results are suspiciously good; for instance, the 350M KPA h=(8×2) reaches 66.16 on ARC-E, outperforming all 1.5B baselines except TPA while using x5 less parameters. Given that this model better matches the performance of the baselines on other benchmarks, this raises concerns about the performance gains on this specific benchmark. This kind of gains should be more thoroughly explained or justified. Second, the paper would largely benefit from context-related evaluations, whether it is by monitoring rolling perplexity along documents, measuring retrieval abilities, or evaluating long-context performance (e.g. using RULER or LongBench). Finally, an decoding speed benchmark across sizes and shapes would be beneficial to assess the direct applicability of the methods.
  • Insufficient discussion of experimental details: By my understanding, KPA should reduce the parameter count of a given architecture, all things equal. It is unclear from reading the paper how this discrepancy is handled, i.e. whether other aspects of the models are adjusted to match the same parameter counts as baselines. There is also no discussion of the KV cache compression ratio of baselines such as MLA or TPA, and it is thus unclear if KPA offers beneficial points on the compression-performance frontier, compared to MLA with various compression rates for instance.
Questions for authors
  • How did you account for parameter count reduction in KPA?
  • How do you justify the discrepancy of the result on the ARC-E benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.