acceptodds
Under review as a conference paper at ICLR 2027

Basis Optimization for Spectral Compression of Deep Learning Matrices

Abstract

Large-scale inference pipelines are typically memory-bound, so smaller stored forms of their matrices reduce latency. Spectral compression trades a controlled error for a shorter description by discarding the least important directions of a matrix. This paper analyzes whether the basis in which the truncation occurs is an optimization target that yields savings beyond plain spectral compression. We (i) formalize basis-optimized spectral compression as a task scored by two costs, the parameters a representation stores and the values a product reads, (ii) introduce Basis-Native Disentanglement (BND), which adapts tensor disentanglement from quantum many-body physics by anchoring the basis search at the native coordinates and by building the rotation from a bounded number of planes to control its parameter cost, and (iii) run it on 1337 matrices from six families of deep learning matrices and two controls. At a relative error of one percent, BND never stores more parameters than raw storage or plain SVD and stores fewer than raw storage on 86% of the matrices. Gradients and state spaces are natively low rank and compress more than tenfold without any rotation, weights gain barely more than noise, and key value caches hold statistically significant structure beyond their spectrum that only a learned basis reveals. Saved parameters convert to load time at the rate of the channel. Our findings mark basis-optimized spectral compression as a real but narrow axis for compressing deep learning matrices.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model compression, low-rank approximation, spectral compression, efficient inference, tensor disentanglement, matrix compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2geaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether learned changes of tensor basis improve spectral compression after charging for the transformation itself. BND combines a native-basis anchor, a bounded-plane Cayley rotation, and rank search. Experiments on 1,337 matrices, including synthetic controls, find modest additional structure in caches and weights, with much larger gains on planted examples. Storage and matrix-product costs are evaluated separately.

Critical strengths

The cost accounting includes transformation overhead and distinguishes storage costs from matrix-product costs, avoiding the misleading assumption that rank reduction necessarily translates into execution savings. The spectrum-matched controls and component ablations are informative, and the paper reports negative results that help distinguish the benefits of the learned basis from those of compression itself.

Critical weaknesses

W1. The incremental contribution on real matrices is limited. The native anchor and bounded planes are reasonable adaptations of the tensor-disentanglement framework of Wei et al. (2025), but their additional benefit on real matrices is modest. Table 2 reports storage compression factors of 1.18 versus 1.16 for SVD on caches and 1.14 versus 1.13 on weights. The large gains on gradients and state-space matrices require no learned rotation, while compression factors for frontier weights remain near 1.00. The strongest improvements occur on planted controls constructed using this type of rotation. Moreover, the no-worse-than-SVD guarantee follows from retaining SVD as a candidate and does not, by itself, establish the value of basis optimization. W2. A practically useful quality–cost operating point has not been established. Table 1 reports matrix-product speedups of approximately 1.00× for caches and 1.01× for weights, and Section 6 acknowledges that the search is unsuitable for transient caches. Appendix G provides downstream evaluations, but these primarily concern plain SVD. The BND experiment on distilgpt2 (Table 19) reduces storage by only about 0.2–0.4% relative to SVD, while perplexity reaches 81.5 and 174.3 at the two tested budgets, compared with 47.0 for the original model. These results do not demonstrate worthwhile compression while preserving model quality. The isolated transfer measurements also leave factor preparation and the complete inference workflow unevaluated.

Questions for authors
  1. Which operating point demonstrates a useful advantage of BND over SVD at comparable downstream quality? Please distinguish stored size, device memory, and inference latency, and account for factor preparation and the cost of learning the basis, including how that cost is amortized.
  2. Equation (9) defines squared relative error, but Section 3.3 compares it directly with the budget used for unsquared relative error in Equation (1). Does the implementation compare the squared tail with the square of the budget? Please clarify the actual certification criterion.
1 comment

Q1/W2 Fair criticism. We understand that all reviewers consider at least one useful operating point necessary to warrant a higher score. We expand on the promising results on LLM caches by focusing on large, idle (non-transient) key caches of long-running agentic LLM systems with 13k-25k tokens. We tune ϵ\epsilon to the lossless regime of SVD (ϵ=0.10−0.15\epsilon=0.10-0.15, ΔF1\Delta F1 -0.002 [-0.010, +0.005] not statistically significant). Within this regime, BND stores 2.6-3.7% fewer key parameters than SVD using the procedure described in our paper. Device memory and per-token latency are unchanged, since keys are decompressed once on resume; factor preparation takes 0.7 ms (BND) vs 0.3 ms (SVD) for an average 17k-token history, and resume time is on par with SVD from host memory and faster from SSD or network storage by up to its 2.7–3.6% byte saving. Learning the basis takes about 1–4 CPU-minutes per 2048 × 128 block, run offline while the agent is idle and parallel across blocks; it amortizes over the time a cache stays stored, which we state as the method's main limitation.

W1 Fair criticism. We would merely argue that the low magnitude of the findings is an artifact of our study approach, since it favors generality (one single method for all matrix types) over establishing a single practically useful operating point, which we happily accept as a weakness. Since this is, to our knowledge, the first study searching for deeper matrix structure and locating it on a compression scale between random noise and planted structure, we also believe the admittedly large portion of negative results could be interesting to the community.

Q2 The implementation certifies the unsquared relative Frobenius error, so the term in Section 3.3 is wrong. Thank you for catching this!

Reviewer_SiVqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BND, which combines index reshaping and a learned orthogonal transformation to improve on ordinary SVD compression. Truncation is performed in the transformed basis, allowing the reconstructed matrix to have high rank in its original layout. The transformation preserves Frobenius error, and the representation supports matrix–vector products without reconstructing the full matrix. Experiments on real and synthetic matrices show large gains on planted structures, modest improvements on real weights and caches, and limited evidence of downstream benefits.

Critical strengths

The main strength is the ability to represent matrices beyond the usual low-rank constraint with a compact set of parameters. The transformation preserves Frobenius error, while the derived forward computation avoids reconstructing a dense weight matrix. Together, these properties make the method an interesting extension of spectral compression. The spectrum-matched controls are also well motivated. By fixing the singular values and randomizing the basis, they help isolate the contribution of the original basis structure. The results on caches and, to a lesser extent, weights provide evidence that the learned transformation captures structure beyond what ordinary SVD can exploit.

Critical weaknesses

My main concern is the limited improvement over SVD on real data. Average storage compression increases from 1.13× to 1.14× on weights and from 1.16× to 1.18× on caches. These gains come with additional optimization cost and a more complex forward pass, without a demonstrated speed advantage over SVD. The large compression ratios on gradients and state-space matrices are already achieved by SVD and therefore do not establish a benefit from the learned transformation.

Model-level validation of the learned transformation is limited. The direct comparison on distilgpt2 shows better quality than SVD and an additional 0.2–0.4% reduction in parameters, but both methods substantially degrade the original model. The results do not yet show whether BND provides a useful advantage when model quality is largely preserved.

Questions for authors

For real matrices where a learned rotation is selected, how do GPU memory usage and matrix–vector latency compare with SVD at the same reconstruction-error budget? In which cases do the compression gains outweigh the correction overhead?

2 comments

W1/W2 Fair criticism. We expand on the promising results on LLM caches by focusing on large, idle (non-transient) key caches of long-running agentic LLM systems with 13k–25k tokens. We tune ϵ\epsilon to the lossless regime of SVD (ϵ\epsilon = 0.10–0.15; ΔF1\Delta F1 −0.002 [−0.010, +0.005], not significant). Within this regime, BND stores 2.6–3.7% fewer key parameters than SVD using the procedure described in our paper, with F1 matching SVD's (ΔF1\Delta F1 −0.003 [−0.010, +0.004], 1,813 paired questions). Factor preparation takes 0.7 ms (BND) vs 0.3 ms (SVD) for an average 17k-token history. When idle caches are resumed from host memory with transfer and decompression overlapped, BND's resume time is on par with SVD's (within ±1\pm 1%). When they are resumed from SSD or network storage (below ~7–10 GB/s), BND resumes faster, by up to its 2.7–3.6% byte saving, since the extra decompression is smaller than the transfer it saves.

Q1 For matrices where a rotation is selected, (i) GPU memory is reduced, since fewer stored parameters are loaded. (ii) Matrix-vector latency is bottlenecked by the identity lane, which streams k(m + n) values from device memory. A rotation is kept only when it lowers the rank, so the identity lane reads fewer values, while the rotation lane adds 2h(lc + rb) values small enough to stay in on-chip cache (Eq. 8). The gains therefore outweigh the overhead for products that are memory-bound and large enough that the saved reads exceed the fixed launch overhead. In practice, the gains reported in our paper are largely absorbed by small matrix executions being dominated by that overhead.

Thank you for the additional experiments. The idle-cache results provide a useful example where BND offers storage savings while largely preserving task quality. However, the practical gains remain modest, and the GPU computation overhead is still not sufficiently quantified, particularly the matrix–vector latency relative to SVD under the same reconstruction-error budget. I continue to find the representation interesting and maintain my weak accept recommendation.

Reviewer_kancOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem of reducing the size of different kind of matrices that are commonly used in deep learning models through low rank factorization. Instead of using a plain svd, they leverage the tensor disentanglement approach from quantum many-body physics by introducing a basis rotation to their decomposition that may lead to a smaller size for a given truncation error. This basis rotation consists in reshaping the matrix into a four-index tensor, choosing a matricization to this tensor along a different regrouping of the indices and applying a rotation to the regrouped matrix. Their contributions consist in formalizing this framework and using Basis-Native Disentanglement (BND) to allow a practical way to build the basis rotations. They perform experiments on 1337 matrices sampled from real-world models (weights, kv cache, etc.) to evaluate their approach.

Critical strengths

The idea is interesting. It astutely leverages a decomposition from physics to try to reach better performance than plain svd by exploiting possible hidden structure in DL matrices. These matrices can be large and costly for training and inference, so compressing them is an important research topic.

Critical weaknesses

I found the paper very hard to read, with many abstract metaphors ("a real matrix that gains more than its twin can only owe the excess to its basis", "product net is reported as served, one wherever the load time probe runs the dense product.", etc.) and a heavy use of defined-once jargon ("native cut", "crossed cut", “rotation lane”, “spectrum twin”, “excess”). Appendix A states that "It is meant to be read before the results." which seems suggests that the main paper is not self-contained.

The large compression performance seem to be obtained with plain svd. The improvements of the new approach seem, in general, modest compared to svd (for example improving from 1.13 to 1.14 on weights), which makes it difficult to evaluate the relevance of adding this level of complexity.

The main text of the paper contains no accuracy or perplexity comparison which is the main interest of such compression approaches. In the appendix, only one such comparison (table 19) is made on a model where perplexity severely degrades.

Questions for authors

Could you provide more results on the improvement in terms of perplexity or accuracy of your approach compared with plain svd? In particular on models where that are still functional (negligible loss of accuracy or perplexity)

Could you provide more insights on why cache matrices seem to hold more structure than other matrix families? Does it align with the real structure of such matrices (eg. heads × head dimension, or tokens)?

3 comments

W1 We apologize that you found the paper difficult to read and will take that criticism to heart when rewriting.

W1.1 Several terms follow established usage, e.g., the term "cut" is used in tensor network literature (Pastawski et al 2015, Levine et al 2018), or "lane" follows the hardware sense of parallel computing (Hennessy & Patterson). Regardless, we see the point as valid and will drop all other terms that are borderline jargon or metaphorical with more robust terms.

W1.2 That Appendix A "should" be read first is a bad claim and will be revised. The section pre-emptively scopes non-claims of the paper, which is definitely not essential to understand the paper, but was merely intended as a small aid for easier interpretation of the work as a whole.

W2 Fair criticism. As stated in the abstract, the paper analyzes whether the truncation basis is an optimization target worth pursuing, across deep learning matrices on a broad scale; to our knowledge no prior work covers this. We argue that the magnitude of gains over SVD and which matrix families hold them is the object of study. The findings identify where impactful future work can build (key caches and weights hold statistically significant structure beyond their spectrum), which is why we see value in the work for the community beyond the method itself.

Q1/W3. Fair criticism. We extend our existing experimental pipeline, unchanged, to idle long-horizon agents' pre-RoPE key caches, i.e., agents that go unused for some time before the cache is loaded again. At budgets where SVD leaves answer F1 unchanged (ΔF1\Delta F1 −0.002 [−0.010, +0.005], 1,813 questions across Qwen2.5-1.5B, Llama-3.1-8B and Qwen3-8B, not significant), BND compresses keys 1.9–3.1× over raw storage and stores 2.6–3.7% fewer parameters than SVD. Its F1 matches SVD's (ΔF1\Delta F1 −0.003 [−0.010, +0.004], pooled over the three models).

Q2 Largely speculative for now, but we find that gains are not uniform. They concentrate in middle layers (6-9% per layer for layers 14-19 against under 1% for layers 0-2 on Qwen2.5-1.5B) and in the low-rank regime. The rotation couples 64-token segments with group channels within each head rather than following the head layout.

I thank the authors for their answer and the new experiment.

My concerns about the readibility remain. Regarding the added experiment, it shows that BND matches SVD in F1 with a parameter reduction of only 3%. This confirms my concern that the improvement over svd is marginal relative to the added complexity and the cost. I will therefore keep my score.

Thank you for your response and for engaging with our rebuttal. We understand your position and do not intend to revisit your score for this submission with this comment.

Instead, since we plan to further improve this work, we would appreciate your guidance on what you would consider a meaningful improvement over SVD in this setting, i.e., what level of parameter reduction you would have expected to make the trade-off against the added complexity and cost worthwhile.

Comments on the paper

1 comment

We thank all reviewers for their careful and constructive feedback. With the discussion period closing, we briefly summarize our perspective.

We see our contribution primarily as a study of hidden structure in deep learning matrices, for which no comparable study exists so far. It tells readers (i) which matrix families are worth targeting (key caches and weights hold significant structure beyond their spectrum; others do not), and (ii) which additional factors such an evaluation must account for, from the parameters a learned basis itself consumes to load and reconstruction times, and how one factor need not necessitate another.

Our contributions spread reviewer scores from strong reject to weak accept. Regardless of overall score, the primary criticism lay in the small magnitude of the gains (though we believe it is under-appreciated that our method competes with SVD, whose optimality within its representation class has been known since 1936). In response, we demonstrated (iii) a small but clear improvement of 2.6–3.7% fewer parameters than SVD on the key caches of idle agents, at unchanged answer quality across three models.

We feel encouraged by the reviews that a method tailored towards a single matrix family showing larger gains is desirable. In our view, that step is best taken once the overall premise and its targets have been verified and validated first, which is what this paper aims to provide. We hope this perspective is considered in the overall decision, regardless of what that decision may be.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.