acceptodds
Under review as a conference paper at ICLR 2027

Sequential Functional Structured Tucker Compression for Large Language Model Attentions

Abstract

Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Compression, Post-Training Compression, Tensor Decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jy1kOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FTC, a post-training compression framework for large language model attention layers that aims to reduce model storage while preserving accuracy. FTC jointly compresses query, key, and value projections using a shared Tucker representation, incorporates activation information to guide compression, and processes layers sequentially to account for changes introduced by earlier compression. It also optimizes the allocation of compression budgets between tensor ranks and sparse-core coefficients and handles the attention output projection separately.

Critical strengths
  • FTC introduces an activation-induced metric to guide Tucker decomposition, improving compression quality by accounting for the functional importance of model weights.
  • FTC uses activations from the current compressed model to guide subsequent layer compression, accounting for representation shifts caused by earlier compression.
Critical weaknesses
  • Although the paper reports compression overhead, it does not evaluate end-to-end inference latency or throughput. It remains unclear whether FTC's more complex compressed representation provides practical inference acceleration.

  • FTC compresses only attention projections, which account for a relatively small fraction of parameters in modern GQA models. Consequently, even aggressive attention compression yields modest whole-model storage savings, limiting its practical impact.

Questions for authors

See the weakness

Reviewer_pveDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FTC, a post-training framework for compressing LLM attention projections without gradient-based fine-tuning. It jointly approximates Q/K/V using shared Tucker factors and head-specific sparse cores, with an activation-aware metric updated sequentially from the compressed model. Tucker ranks and core sparsity are selected jointly under a storage budget that includes support metadata. The output projection is calibrated and compressed separately to account for changes in attention representations. Experiments on seven decoder-only models from 6B to 32B parameters report improved perplexity and downstream performance over the evaluated baselines, particularly under aggressive attention compression. Component ablations and calibration sensitivity analyses further examine the sources and robustness of these gains.

Critical strengths

1.Substantial empirical gains under aggressive compression. At an attention-projection storage ratio of 0.2, FTC achieves substantially lower perplexity than the evaluated alternatives across all five GQA models. Improvements also transfer to downstream multiple-choice tasks, supporting the practical value of the overall framework.

2.Broad evaluation with useful controls. Experiments span seven models from 6B to 32B parameters and include a matched sequential per-matrix baseline, component ablations, and calibration sensitivity analyses. These results provide meaningful evidence that the improvements extend across architectures and help identify which components contribute to performance.

Critical weaknesses

1.The methodological contribution beyond closely related work is not sufficiently established. Activation-aware whitening has been explored in SVD-LLM, adaptation to compression-induced input shifts is explicitly addressed by AA-SVD, and shared attention Tucker representations and sparse cores have been investigated in TensorLLM and LeSTD. While FTC differs in its native GQA representation, budget-coupled rank–sparsity allocation, and separate output-projection calibration, the paper does not sufficiently explain why these choices constitute a substantive advance beyond integrating established techniques.

2.The experiments demonstrate the effectiveness of the overall framework, but do not fully resolve this concern. Seq-SVD uses independent matrix factorizations, whereas enh-LeSTD retains a data-free objective; neither directly isolates the proposed contribution over an activation-aware, sequentially calibrated sparse Tucker baseline. A clearer technical distinction from these closely related methods, supported by a focused comparison, could materially change my assessment.

Questions for authors

1.Contribution clarification: What substantive technical advance does FTC provide beyond combining activation-aware sequential compression with an existing sparse Tucker representation? Please clarify the differences from SVD-LLM, AA-SVD, TensorLLM, and LeSTD, and explain which comparisons isolate the benefits of FTC’s distinctive design choices. This clarification could materially affect my assessment.

2.Compatibility with other attention mechanisms: The paper currently focuses on MHA and GQA. Could the authors briefly discuss FTC’s compatibility with other attention mechanisms, such as linear attention and sparse attention, including any adaptations that would be necessary? This is a suggestion to clarify the framework’s scope, rather than a request for additional experiments.

Reviewer_DELAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FTC compresses LLM attention post-training, without fine-tuning, by fitting the stacked Q/K/V heads with a shared-factor Tucker model and sparse per-head core under an activation metric built sequentially from the current compressed model, with W_O calibrated and compressed separately. Its contributions are this sequential joint head-wise representation, a budget-coupled allocation of Tucker ranks versus sparse-core capacity with gradual prune–refit, and a matched per-matrix control isolating the benefit of joint tensorization. Across seven 6B–32B models and six baselines, it attains the lowest WikiText-2 perplexity on all five GQA models at every ratio, with the largest gains at ρ = 0.2 (1.21–1.83× dense, 39–88% below the best baseline) and clear downstream improvements, though it uses more calibration data than baselines, compresses only attention weights (11–32% of parameters), and reports no speedups.

Critical strengths
  1. Large, controlled gains under aggressive compression. At ρ = 0.2, FTC stays within 1.21–1.83× dense perplexity on all five GQA models while baselines reach 2–37×, and the Seq-SVD control (same metric and sequential calibration, independent factorization) loses most of this advantage, credibly attributing the gain to joint head-wise tensorization.
  2. Principled, fully accounted budget allocation. Storage counts factors, core values, and the support mask, an exact subspace-plus-sparsification error decomposition yields a cheap rank/sparsity search, W_O gets its own metric, and one fixed recipe works across all seven models and four ratios with every key component ablated.
Critical weaknesses
  1. The practical benefit is not demonstrated: core sparsity leaves the reconstructed weights dense, no kernel, latency, or memory measurement is reported, and ρ applies only to attention projections (11–32% of parameters), so whole-model savings top out near 26%. Since the headline gains occur at ρ = 0.2, exactly where the deployable 2:4 variant degrades most (up to 366% on Qwen2.5-7B), I cannot judge whether the perplexity advantage survives deployment.
  2. The comparison is not matched: FTC uses 256 calibration windows versus 128 for baselines, and Table 16 shows Qwen2.5-7B at ρ = 0.2 rises from 12.55 to 43.9 with 128 windows, undermining one of the most striking results. With calibration and evaluation both on WikiText-2, the GQA LeSTD baseline being the authors’ own port, and the abstract’s “every tested ratio” claim failing on MHA models at ρ = 0.8, I am uncertain how much of the margin is intrinsic rather than protocol-driven.
Questions for authors
  1. Can you report all data-dependent baselines (SVD-LLM, MPIFA, Basis Sharing, Seq-SVD) with 256 calibration windows, or FTC with 128 everywhere, at least for the four 7–8B GQA models at ρ = 0.2? Given Table 16, I need to know whether the Qwen2.5-7B margin survives a matched calibration budget.
  2. Does the sparse latent core yield any actual inference benefit? Please report latency and/or memory for at least one model at ρ = 0.2 using either the sparse rank-one expansion of Eq. (21) or the 2:4 core, compared with the dense model and with a plain low-rank baseline at the same ρ. If no kernel exists, please state that the sparse core currently provides storage savings only.
  3. How does FTC compare with sequential, activation-aware methods when the W_O calibration is also given to them? Seq-SVD isolates joint tensorization, but it is unclear whether baselines like SVD-LLM or MPIFA would close part of the gap if equipped with your sequential student metric and closed-form W_O target.
  4. For the MHA models, can you report native LeSTD alongside enh-LeSTD at all ratios, so the reader can verify the GQA port does not weaken the baseline?
  5. The abstract claims the best perplexity “at every tested keep ratio,” but Table 6 shows enh-LeSTD ahead on GPT-J-6B and Llama-2-13B at ρ = 0.8. Please confirm whether this is a reporting error or the claim should be qualified to GQA models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.