acceptodds
Under review as a conference paper at ICLR 2027

Learning Functional Subspaces for Neural Network Compression

Abstract

Modern transformers couple impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and therefore efficient on standard hardware. However, several existing methods choose the subspace removed from each weight matrix by _local_ closed-form criteria—activation energy, layer-wise reconstruction error, or a quadratic loss approximation—and thus ignore how errors propagate through the rest of the network. At aggressive compression ratios, these errors compound through depth and performance collapses. The number of directions removed from each layer is set before training according to that layer's measured effect on the output. We introduce _Learnable Subspace Projections_ (LSP), which instead learns the removed subspaces end-to-end. Each linear layer, or _tied group_ of layers that read the same activations, receives an orthogonal projector. All projectors are optimized jointly against a _global_ objective, the KL divergence to the dense model's output distribution, while the pretrained weights stay frozen. Projectors are initialized from a whitened SVD truncation, and ranks are allocated by measured cost: the output KL divergence each projector induces, per parameter saved. After training, the projectors merge into standard low-rank factors, and each tied group shares one input factor, so attention can cache a single narrow latent instead of full keys and values. Across decoder-only LLMs (OPT-125M/1.3B, Qwen3-4B, Llama-2-7B) and a Vision Transformer (ViT-B/16), LSP outperforms activation-based, curvature-based, and rank-learning baselines by a margin that grows with the compression ratio. At 70% compression, LSP brings Llama-2-7B to WikiText-2 perplexity versus for the strongest baseline, and to mean zero-shot accuracy versus . The factorized model decodes up to faster than the dense model at small batch sizes, and caching the shared latent reduces the combined memory of weights and KV cache by at a 128k-token context, versus at most for untied factorizations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model compression, low-rank, subspace projections

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_haXwOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learnable Subspace Projections (LSP), a low-rank neural network compression method that learns functional subspaces through end-to-end optimization. The authors argue that conventional SVD and activation-aware low-rank compression methods mainly rely on local statistics or reconstruction objectives, without fully accounting for how compression errors propagate through the network. To address this limitation, LSP introduces learnable orthogonal projectors for individual linear layers or groups of layers sharing the same inputs. The projectors are initialized using whitened SVD, while rank budgets are allocated based on the full-model output KL divergence induced by compressing each unit individually. All projectors are then jointly optimized to preserve the dense model's output distribution, with the pretrained weights kept frozen. After training, the projectors are merged into standard low-rank factors, and shared input factors across Q/K/V projections can further reduce computation and potentially KV-cache storage. Experiments on four decoder-only LLMs and ViT-B/16 demonstrate the effectiveness of LSP. At 70% parameter compression on Llama-2-7B, LSP achieves a WikiText-2 perplexity of 10.9 and an average accuracy of 42.2% across six zero-shot tasks. The paper also reports up to 1.56× faster short-context decoding and an estimated 13.5× reduction in combined weight and KV-cache memory using latent caching.

Critical strengths

Strength 1: The paper presents a clear approach to function-aware low-rank compression by learning which orthogonal subspaces to remove, rather than relying solely on fixed singular-vector rankings or local reconstruction criteria. Unlike conventional activation-aware SVD methods, LSP jointly optimizes layer-wise projections using the KL divergence between the compressed and dense models' output distributions, allowing the retained subspaces to reflect global functional sensitivity. The method has a well-defined optimization objective and enables end-to-end learning of the projection matrices while keeping the pretrained weights frozen.

Strength 2: The paper demonstrates competitive results at high compression ratios. In particular, at 70% compression on Llama-2-7B, LSP reduces WikiText-2 perplexity from 13.3 with the SVD-LLM recovery baseline to 10.9, while improving average accuracy across six zero-shot tasks from 36.0% to 42.2%. The NoLSP ablation further shows that learning the projections substantially improves model quality under the same initialization, rank allocation, and shared structure. The paper also evaluates multiple compression ratios, different training objectives, and three random seeds, along with calibration distribution shifts and cross-task transfer on ViT-B/16.

Strength 3: LSP can be merged into standard low-rank factors for deployment, without retaining the QR operations or additional projections used during training. Sharing input factors across Q/K/V and gate/up projections also reduces redundant computation and enables low-dimensional latent caching under certain attention configurations. The paper reports measured decode throughput and analyzes how shared factors and rank allocation affect inference efficiency, demonstrating practical benefits beyond compression accuracy.

Critical weaknesses

Weakness 1: MatryoshkaKV [3] has already explored training orthogonal projection matrices to compress KV representations while optimizing low-dimensional subspaces based on model performance. LLRC [4] studies differentiable selection of singular directions, while QSVD [5] introduces joint Q/K/V low-rank factorization to reduce parameter and KV-cache costs. Therefore, optimizing projection subspaces through training objectives and exploiting shared Q/K/V representations for efficient inference both have clear precedents. The main distinction of LSP appears to be extending trainable projections to linear layers throughout the network and jointly optimizing shared subspaces against a global KL objective. Although this extension has potential value, the paper does not sufficiently discuss its relationship to MatryoshkaKV and LLRC, nor does it provide direct comparisons under matched compression budgets to establish the independent advantages of joint subspace learning over existing trainable subspace methods.

Weakness 2: The NoLSP ablation demonstrates the importance of joint training, and the comparison with SVD-LLM [1] using LoRA recovery is valuable. However, LSP requires multiple full-model forward and backward passes, QR orthonormalization, and projection optimization, while the baselines differ in their trainable parameter counts, training iterations, and optimization flexibility. In particular, LSP directly modifies the retained subspaces by training high-dimensional projection matrices, whereas SVD-LLM uses a relatively small LoRA recovery configuration. Consequently, the performance gains cannot be attributed entirely to orthogonal subspace learning. The paper lacks comparisons with direct low-rank factor optimization or more extensive recovery training under matched calibration data, optimization steps, trainable parameter budgets, and GPU-hours.

Weakness 3: Eq. (3) measures the KL divergence induced by compressing each unit individually while keeping the rest of the network dense, and Eq. (17) uses these isolated costs for greedy rank allocation based on marginal KL per parameter saved. However, these isolated measurements do not capture error propagation and interactions among multiple low-rank projections applied simultaneously.

Weakness 4: The paper highlights a 13.5× reduction in combined weight and KV-cache memory at 70% compression, along with an estimated single-GPU context capacity of approximately 322k tokens. However, Appendix E explicitly states that these results are analytical estimates based on low-rank factor dimensions, while the measured decode throughput uses a standard full-width KV cache rather than actual latent-cache inference. Therefore, the experiments do not directly validate the peak GPU memory usage, runtime latency, additional K/V reconstruction overhead, or long-context generation quality of the proposed latent caching scheme.

Weakness 5: The largest LLM evaluated is Llama-2-7B, leaving the scalability of projection training, KL probing, and rank allocation unverified on larger 13B/32B/70B models. The downstream evaluation mainly covers six zero-shot commonsense and basic mathematical reasoning tasks, without more challenging benchmarks such as GSM8K, MATH-500, AIME, MMLU, or code generation. Moreover, the LSP compression pipeline involves full-model KL probing and multiple rounds of projection training, requiring several hours and multiple GPUs for some configurations. The code is also planned for release only after acceptance, limiting independent reproducibility at this stage. Given the offline compression efficiency of Swift-SVD [8] and other lower-cost methods, a clearer analysis of the trade-off between compression cost, training memory, and model quality is needed.

[1] Wang, X., Zheng, Y., Wan, Z., and Zhang, M. SVD-LLM: Truncation-Aware Singular Value Decomposition for Large Language Model Compression. In International Conference on Learning Representations (ICLR), 2025.

[2] Wang, Q., Ke, J., Tomizuka, M., Keutzer, K., and Xu, C. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives. In International Conference on Learning Representations (ICLR), 2025.

[3] Lin, B., Zeng, Z., Xiao, Z., Kou, S., Hou, T., Gao, X., Zhang, H., and Deng, Z. MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection. In International Conference on Learning Representations (ICLR), 2025.

[4] Sundrani, S., Tudisco, F., and Minervini, P. Low-Rank Compression of Language Models via Differentiable Rank Selection. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC), pp. 10031–10045, 2026.

[5] Wang, Y., Wang, H., and Zhang, S. Q. QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.

[6] Chang, C.-C., Lin, W.-C., Lin, C.-Y., Chen, C.-Y., Hu, Y.-F., Wang, P.-S., Huang, N.-C., Ceze, L., Abdelfattah, M. S., and Wu, K.-C. Palu: KV-Cache Compression with Low-Rank Projection. In International Conference on Learning Representations (ICLR), 2025.

[7] Abbasi, A., Thrash, C., Qin, H., Sharma, S., Seifi, S., and Kolouri, S. Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression. In Proceedings of the 43rd International Conference on Machine Learning (ICML), pp. 84–100, 2026.

[8] Qi, R., Liu, Y., Wu, X., Wang, X., Li, M., Chen, C., Chen, J., Chen, Y., and Weng, Q. Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

Questions for authors

Q1: MatryoshkaKV [3] has already explored training orthogonal projections to optimize low-dimensional KV representations, LLRC [4] uses differentiable learning to select low-rank directions, and QSVD [5] investigates joint Q/K/V low-rank compression. Could the authors further clarify the independent technical contributions of LSP compared with these methods, particularly what new optimization properties or practical advantages arise from jointly learning orthogonal projectors across the entire network? Could the authors also provide direct comparisons under comparable compression settings to better establish the independent contribution of the proposed subspace learning approach?

Q2: LSP requires multiple full-model forward and backward passes, QR orthonormalization, and projection optimization, whereas the LoRA recovery stage of SVD-LLM [1] uses a relatively limited number of trainable parameters. Could the authors compare LSP with more extensive SVD-LLM recovery or direct low-rank factor optimization under matched calibration data, optimization steps, trainable parameter budgets, and GPU-hours? In particular, could the authors better distinguish whether the performance gains come from the orthogonal subspace parameterization itself or from additional end-to-end optimization?

Q3: Eq. (3) estimates the KL cost of each compression unit independently, while Eq. (17) uses these isolated costs for greedy rank allocation without explicitly accounting for interactions between compressed layers. Appendix B further reports that, on Llama-2-7B, the joint compression KL can be 2.1–4.3× the sum of the isolated KL costs. Could the authors further analyze how this discrepancy affects the final rank allocation and compare the current strategy with joint-KL evaluation or interaction-aware allocation? Additionally, could the authors clarify the efficiency or accuracy advantages of the proposed greedy allocation over more direct global optimization approaches?

Q4: The reported 13.5× reduction in combined weight and KV-cache memory and the estimated 322k-token context capacity are primarily based on analytical estimates, while the measured decode throughput still uses a standard full-width KV cache. Could the authors provide end-to-end experiments with actual latent-cache inference, including GPU peak memory, TTFT, prefill and decode throughput, and K/V reconstruction overhead? In particular, could the authors evaluate the actual memory and computational benefits across different context lengths and batch sizes, together with long-context generation quality, to verify whether the theoretical storage advantages translate into practical deployment benefits?

Q5: The largest LLM evaluated is Llama-2-7B, while KL probing and joint projector training introduce additional computational and memory costs. Could the authors extend the evaluation to larger 13B/32B models and include more challenging reasoning and code generation benchmarks, such as GSM8K, MATH-500, AIME, MMLU, HumanEval, or MBPP, to assess capability preservation under aggressive compression? Additionally, could the authors provide detailed GPU-hours, peak GPU memory, and total compression time for KL probing, projector training, and model merging across different model sizes, and compare the resulting accuracy–compression-cost trade-off more directly with lower-cost methods such as Swift-SVD [8]?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer haXw [1/3]

We thank the reviewer for the detailed and constructive review. W1/Q1, W2/Q2 and the harder benchmarks of Q5 are addressed with new experiments in the Global Response (GR), summarized below; we answer W3–W5 in full. Table, section and appendix numbers refer to the revised PDF.

[W1/Q1] Not sufficiently discussing differences with MatryoshkaKV, LLRC, QSVD and Palu.

We thank the reviewer for the references. We note that QSVD (related work) and Palu (Sec. 3.3 of the submission, Sec. 3.4 in the revision) were already cited in the paper, but agree that a clearer presentation and comparison with LSP helps the manuscript. Experimentally, we now compare with MatryoshkaKV under matched compression (GR, T3), while no code is available for LLRC.

  • MatryoshkaKV (MKV) proposes KV cache projections, after RoPE, separated by head, with post-training allocation. LSP instead projects the weight matrices, before RoPE and across heads, sharing projections among layers with the same input or output, and allocates before training. Empirically, LSP restricted to K/V consistently outperforms MKV, by 4.0–19.9 Avg6 points (see the detailed comparison in GR, T3).
  • LLRC and Zero-Sum SVD keep a subset of the singular components of a fixed activation-aware or whitened SVD and zero the rest, choosing it with a learned mask (LLRC) or first-order loss estimates (Zero-Sum SVD). LSP is not restricted to these components: it learns the retained subspace itself, so it can keep a fraction between 0 and 1 of each singular direction (summing to the rank), whereas selection keeps each direction entirely or not at all (see Figure 4).
  • QSVD and Palu share two design choices with LSP, which we now credit more clearly: QSVD's joint factorization of Q/K/V with one cached latent, which LSP applies to every group of layers that share the same input or output, and Palu's latent-cache deployment with keys rebuilt under RoPE (W4). Both choose the subspace in closed form and allocate ranks by Fisher information. In our experiments, LSP's measured-KL allocation beats a diagonal-Fisher one (Table 10), and learning the subspace takes the closed-form start from 222.8 to 10.9 perplexity at −70% on Llama-2-7B, and adds up to 20.1 Avg6 points over NoLSP-KV (GR, T3).

[W2/Q2] SVD-LLM uses limited resources compared to LSP (GR, T1-T2-T4-T5).

We refer to GW3 in the Global Response for the full results; in short:

  • Matched training is not enough. SVD-LLM trained like LSP (same data, budget, KL objective, and a LoRA rank matched to LSP's parametrization) still trails LSP in all six Qwen3-4B and Llama-2-7B settings (T4), in mean score at every ratio on both Qwen3.5 models (T1–T2), and at almost every epoch (T5).
  • The projector adds on top. From the same start and with the same training, fine-tuning the low-rank factors directly (NoLSP+RFT) reaches about LSP's perplexity on Qwen3.5-4B, but trails LSP's mean by 0.5–1.3 points on Qwen3.5-4B and by 1.7–3.1 on Qwen3.5-9B (T1–T2).
  • Fairness. We note that the projector is restricted in its parameterization, as it can only remove spectral energy from the pretrained weights' directions, never add, while free factors and LoRA recovery are unconstrained. Its extra training cost (QR, projections) is paid once; after merging, LSP is plain low-rank factors with the same parameter count and FLOPs as the baselines (W5/Q5).

[W3/Q3] Isolated KL costs in the allocation. As mentioned in the limitations (App. A), the KL probe is indeed an additive surrogate (the joint KL is 2.1–4.3× the sum of the isolated costs on Llama-2-7B (App. B)). However, the greedy allocator only uses the relative order of marginal costs per saved parameter, so interactions that inflate all costs alike leave the allocation unchanged. In practice, we compare the KL allocation with fixed-rank, uniform, and diagonal-Fisher allocations, and find KL to perform best overall. A joint search would need a forward pass of the compressed network per candidate move and allocation state; the isolated probe runs once per model and serves every ratio and objective. We leave interaction-aware allocation to future work.

AuthorsOpenReview ↗

Author Response to Reviewer haXw [2/3]

[W4/Q4] The latent-cache scheme is not validated end to end. The reviewer is right. In the paper, only decode throughput was measured (prefill and decode FLOPs were counted), and the 13.5× reduction of weight-plus-cache memory and the 322k-token capacity were analytical estimates from the factor widths, as App. E and the Figure 3 caption state. The cost of rebuilding keys under RoPE was only discussed, and long-context quality was not evaluated.

What we ran now. We implemented the latent-cache deployment, as in Palu (a static latent buffer with chunked prefill, values folded into the output projection, and a fused Triton kernel that rebuilds keys from the latent, applies RoPE and computes scores in one pass). We measured it on the WikiText-2-calibrated Llama-2-7B at −70%, on one GH200 in bf16: peak memory, the longest context that fits, prefill time (the time to first token), decode throughput at several context lengths and batch sizes, and the key-rebuild overhead against a full-width cache of the same weights.

Llama-2-7B, −70%DenseLSP, full-width cacheLSP, latent cacheSVD-LLM (W), latent cache*
Cached floats/token/layer819281925561248
Longest context at batch 8 (measured)20.5k22.5k333.8k149.5k
Peak memory at 8×128k tokensOOMOOM40.1GiB83.6GiB
Prefill 8×16k: time/peak7.9s/80.4GiB5.2s/72.2GiB5.2s/12.4GiB4.9s/17.8GiB
Prefill 8×128k: time/peakOOMOOM198.4s/42.9GiB198.4s/86.1GiB
Decode, batch 1, tok/s at 512/4k/16k/128k134/115/100/35203/162/132/38212/167/92/16186/147/81/14
Decode, batch 8, tok/s at 4k/64k/256k627/OOM/OOM783/OOM/OOM424/33/8380/29/OOM

SVD-LLM (W): SVD-LLM's whitened truncation (its LoRA recovery keeps the same shapes). *Ranks padded to multiples of 16 (1228 floats unpadded); see the correction below.

The measurements confirm the paper's estimates, conservatively:

  • Memory. The latent cache holds 333.8k tokens per sequence at batch 8 (322.1k estimated) and completes a real 8 × 256k-token prefill at a 77.7 GiB peak, where the full-width cache stops at 22.5k tokens.
  • Quality. It is exact (WikiText-2 perplexity 10.9498 vs. 10.9502 with the full-width cache), so long-context quality is that of the compressed model (RULER at 128k on Qwen3.5-9B, GR T2: 89.9 at −20%, dense 95.6).
  • Speed. Rebuilding keys under RoPE costs decode speed as context and batch grow: at batch 1, the latent cache decodes as fast as the full-width one up to 4k tokens, and at 0.69× / 0.42× its speed at 16k / 128k; at batch 8, the cost already shows at 4k (0.54×). Palu pays the same cost, which DeepSeek-V2 avoids by design with a separate, decoupled RoPE key.
  • Use. Both deployments compute the same model, so a server can pick one per request: the full-width cache when the context fits and speed matters, the latent cache when memory is the limit, where it serves contexts the full-width cache cannot hold (at batch 8, beyond 22.5k and up to 256k tokens).

Correction to the paper. Re-timed with rank-aligned baselines (ranks zero-padded to multiples of 16, which is exact), LSP remains the fastest factorization in all nine cells, but by +11–14% at 512 tokens rather than +15–25%, and untied factorizations are no longer slower than dense at −30%. We correct Sec. 3.4 and App. E accordingly.

AuthorsOpenReview ↗

Author Response to Reviewer haXw [3/3]

[W5/Q5] LSP is not tested at larger scale or on harder benchmarks, and it is not clear when it is preferable to cheaper alternatives.

Larger models. Beyond Llama-2-7B, we now compress the slightly larger Qwen3.5-9B (GR, T2); given the other experiments requested, a 13B–32B run did not fit the rebuttal period. We note that, compared with training-free methods, training-based methods are more limited by model size, which we added as a limitation to the revised paper.

Harder benchmarks. We now add harder benchmarks that evaluate them on reasoning, knowledge, instruction following and code (GSM8K, MATH-500, AIME, MMLU, MMLU-Pro, GPQA-Diamond, BBH, IFEval, HumanEval+, MBPP+), long context (LongBench v2; RULER at 32k and 128k) and tool use (BFCL v4, single- and multi-turn). We refer to (GW1, T1–T2) for all the new benchmark results.

Cost of the KL probe. On Llama-2-7B, the probe takes around 10% of LSP's compression time at −50% and −70%, and one probe serves every ratio and objective; it costs 0.2 / 1.7 / 1.1 wall-clock hours on OPT-125M / Qwen3-4B / Llama-2-7B (App. F). When compression time matters, it can also be skipped: uniform allocation stays within 1.3 Avg6 points of the measured-KL allocation in five of six settings (Table 11).

Pareto optimality. App. F reports compression cost against perplexity for every method: LSP is more expensive than the alternatives, but it is Pareto-optimal at higher budgets, as at every model and ratio one of its objectives gives the lowest perplexity within 0.5–10 hours. Training-free methods are much cheaper but degrade at high compression (Swift-SVD reaches 203.6 perplexity at −70% on Llama-2-7B, against LSP's 10.9). Moreover, compression is paid once per model and ratio, while quality and inference cost are paid at every query, and LSP also lowers the latter (faster decoding and a shared latent cache, W4).

Code. We confirm that the full code will be released upon acceptance.


We hope these answers address the reviewer's concerns, and ask if there are any other concerns or questions.

Reviewer_3zrROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Learnable Subspace Projections (LSP), which learns orthogonal projection subspaces through global KL distillation while keeping pretrained weights frozen. The method combines whitened SVD initialization, KL-based rank allocation, and shared low-rank factors. Experiments on LLMs and ViT demonstrate strong compression performance, improved decoding efficiency, and potential KV-cache memory savings.

Critical strengths
  1. Well-motivated formulation with effective global optimization. The paper provides a clear motivation for moving beyond local reconstruction objectives toward preserving the overall network behavior. Learning the removed subspaces through end-to-end distillation is a natural and effective approach. The substantial improvement over NoLSP provides convincing evidence for the importance of jointly optimizing compression directions.

  2. Strong empirical results and practical efficiency benefits. The experiments cover multiple architectures and compression ratios, with particularly impressive improvements under aggressive compression. The ability to merge learned projections into standard low-rank factors is practically appealing. The shared-factor design also provides additional opportunities for reducing inference computation and KV-cache memory.

Critical weaknesses
  1. The relationship to closely related projection-learning methods is insufficiently clarified. MatryoshkaKV [1] also addresses the limitations of local PCA-based compression by jointly learning orthogonal projections through global knowledge distillation. It further employs output KL divergence to allocate compression dimensions across layers and heads. Although the two works target different compression settings, they share important methodological principles. LSP extends these ideas to network-wide weight compression and introduces additional designs, including whitened initialization and shared low-rank factors. However, the paper does not discuss MatryoshkaKV, making it difficult to clearly distinguish the methodological contributions from those of this closely related work.

  2. The specific benefits of the orthogonal projector parameterization are not fully isolated. The comparison with NoLSP clearly demonstrates the importance of learning compression subspaces. However, it remains unclear whether the proposed orthogonal projector parameterization offers advantages over directly optimizing general low-rank factors with the same global distillation objective. Existing recovery baselines, such as SVD-LLM with LoRA, differ in their parameterization, trainable parameter counts, and optimization procedures. Consequently, the reported improvements cannot be attributed solely to the proposed parameterization, leaving some uncertainty about its specific contribution to the observed performance gains.

Overall, I am open to revisiting my assessment if these concerns are satisfactorily addressed during the rebuttal.

[1] Lin, B., Zeng, Z., Xiao, Z., Kou, S., Hou, T., Gao, X., Zhang, H., and Deng, Z. (2025). MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection. In The Thirteenth International Conference on Learning Representations (ICLR 2025).

Questions for authors
  1. What are the key methodological differences between LSP and MatryoshkaKV, particularly regarding globally optimized orthogonal projections and KL-based rank allocation?

  2. To what extent do the performance improvements arise from the orthogonal projector parameterization itself, rather than from end-to-end KL optimization of the compressed model?

  3. Do the accuracy and efficiency measurements in Figure 3 correspond to the same compressed checkpoints? Table 2 reports results with Alpaca calibration, whereas Appendix E describes efficiency evaluations on WikiText-2-calibrated checkpoints.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 3zrR

We thank the reviewer for the careful review and feedback. Both weaknesses are addressed with new experiments in the Global Response (GR); we summarize the main points here. Table, section and appendix numbers refer to the revised PDF.

[W1/Q1] The relationship with MatryoshkaKV is not sufficiently clarified (GR, T3).

We refer to (GR, T3) for detailed qualitative and quantitative comparison. Summarizing, MatryoshkaKV (MKV) projects the cached K/V representations, after RoPE, separately per head; LSP projects each linear layer's weights directly (before RoPE, without splitting attention heads), with projectors shared among layers that share the same input or output, to decompose them into low-rank factors. Both allocate by output KL, but MKV distributes cache dimensions after its nested training, while LSP distributes a parameter budget across all layer types before training. Empirically, we find that at matched cache size, LSP restricted to K/V beats MKV by 4.0–19.9 Avg6 points.

[W2/Q2] It is not clear which factors contribute to LSP performance (GR, T1-T2-T4-T5).

In (GR, T1-T2-T4-T5) we compare LSP in detail against a version of SVD-LLM matched in budget, capacity and recipe, and against full recovery fine-tuning of the NoLSP factors (NoLSP+RFT). To what extent is it the projector? Part of the gap to the SVD-LLM performance is the recipe, as the LSP-matched version SVD-LLM (matched) gains on Qwen3.5 show (T1–T2). The projector adds on top: from LSP's start, with the same KL objective, data and budget, training full low-rank factors instead (NoLSP+RFT) trails LSP by 0.5–1.3 mean points on Qwen3.5-4B and 1.7–3.1 on Qwen3.5-9B, even though the projector is the more constrained parameterization.

[Q3] Do the accuracy and efficiency measurements in Figure 3 correspond to the same compressed checkpoints?

Yes for LSP, and we thank the reviewer for pointing out that the paper does not say so. Figure 3 uses the Alpaca-calibrated checkpoints of Table 2 on every axis (throughput from their factor shapes, context capacity from their ranks), while Table 5, App. E and the abstract (13.5×) use the WikiText-2-calibrated ones. Allocation depends on the calibration data, so the shapes differ slightly. At −70%, speed agrees within 0.3%, while the Alpaca latent is wider (604 vs. 556 floats per token per layer), so capacity is 8% lower (296k vs. 322k tokens) and the memory reduction at 128k is 12.5× instead of 13.5×, against at most 6.5× for untied factorizations on either corpus. The baselines barely change: SVD-LLM's ranks do not depend on the calibration data, Swift-SVD's capacity differs by under 0.01%, and Dobi-SVD's capacity uses its Alpaca ranks and its speed a WikiText-2 run within 1.3% in parameters. The revised caption names the checkpoints, and App. E reports both.

We hope the matched MKV comparison and the parameterization controls resolve both concerns, and we ask the reviewer if they have any remaining concern or question.

Thank you for your detailed response.

My concerns have been resolved. I would like to raise my score to 3.

Reviewer_Az5gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper finds that existing low-rank compression keeps high-energy directions instead of the directions the network output actually depends on. It proposes Learnable Subspace Projections (LSP), which learn the removed subspaces end-to-end against the dense model's output. At 70% compression, LSP reaches 10.9 WikiText-2 perplexity and 42.2% zero-shot accuracy on Llama-2-7B.

Critical strengths
  1. Clearly written.
Critical weaknesses
  1. Low-rank compression has basically been thoroughly studied in the era of small models. The core idea of this article is almost identical to "Compressing Transformers: Features Are Low-Rank, but Weights Are Not!" (AAAI 2023). It brings no incremental ideas/contributions to the academic world.
  2. The experimental scale and ablation studies are insufficient. It's already 2026. Please do not conduct experiments on OPT/LLaMA2/Qwen3 models, nor use CIFAR-100. It has no meaning at all. At the very least, it should be Qwen3.5. However, the modern LLM is usually a hybrid architecture, and the redundant weights within it have been significantly reduced. The practicality of the Low-rank method on it is questionable.
  3. Besides, the most important scenario for LLMs now is long context/agentic. No relevant ablation experiments have been conducted here. The datasets (ARC-e, PIQA, OpenB., WinoG, HellaS, MathQA, etc.) are currently outdated. Please do not conduct experiments on those small datasets. It is very difficult for me to believe that this article has any practical value.
Questions for authors

Please see weakness. Before being accepted as a formal conference paper, this article needs to undergo significant revisions in its methodology design, and experimental results.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer Az5g

We thank the reviewer for their time and feedback. We address the reviewer’s concerns below; the new experiments are detailed in the Global Response (GR). Table, section and appendix numbers refer to the revised PDF.

[W1] The idea is almost identical to Yu & Wu (AAAI 2023) [1].

We thank the reviewer for the reference, but we push back on the argument that the two methods are almost identical. Specifically, [1] shares one main component with LSP: ranks allocated greedily under a parameter budget, from the output KL of compressing each layer in isolation (closely related to our KL measure after applying each projection in LSP), for which we thank the reviewer for the note, and which we now credit.

As per other components, the activations-based initialization is quite common in low-rank compression methods ([1], SVD-LLM, Swift-SVD), which LSP initializes its projections from, and extends to tied projections (Prop. 1). Lastly, [1] performs recovery fine-tuning on the full low-rank factors to L2-match the dense model’s features, while LSP learns subspace projections to the frozen weights, trained on the output KL ([1] reports L2 feature matching to outperform output-KL distillation, while in early experiments we found the opposite).

LSP differs in what it learns, in how layers are factorized, and in what it reveals about each layer's functional subspace. Our results show that

  • (i) learning the subspace alone suffices: at -70% on Llama-2-7B it lowers perplexity from 222.8 (untrained start) to 10.9;
  • (ii) the learned subspaces drop high-energy directions and keep low-energy ones (Figure 4), showing that activations’ spectral energy does not rank directions by function;
  • (iii) LSP beats SVD-LLM with LoRA (GR, T1-T2-T4) and the closest analogue of [1]'s fine-tuning stage: training the factors themselves from LSP's start, with the same KL objective, data and budget (NoLSP+RFT), which trails LSP's mean in all six Qwen3.5 settings (GR, T1-T2).

In addition, LSP extends the learnable projectors to weights that share the same activations (Q/K/V, gate/up; Proposition 1), demonstrating that least-functional directions can be shared as well, and that removing them is beneficial (at equal budget, this raises Avg6 at every ratio (Table 9), and gives one narrow latent cache and faster decoding (Table 5)).

[W2-3] The models (OPT, Llama-2, Qwen3), CIFAR-100 and the zero-shot benchmarks (ARC-e, PIQA, OpenBookQA, WinoGrande, HellaSwag, MathQA) are outdated, the ablations are insufficient, and low-rank compression may not pay off on modern hybrid, long-context and agentic LLMs.

Choice of models and benchmarks.

  • As mentioned in GW1, our choice of models and benchmarks followed the protocols of the low-rank compression baselines, and is up to date with most recent related papers (compressing Qwen3-4B on the same benchmarks of Swift-SVD, ICML26).
  • CIFAR-100 appears only in the ViT study, as the ViT's task and calibration pool, to test how calibration data affect transfer to unseen datasets (Pets, Aircraft, Places365).
  • The paper ablates initialization, tying, allocation and seeds (App. B–C), the objective (LSP vs. LSP$^T$) and calibration shift (Sec. 3.3); the rebuttal adds factor fine-tuning from NoLSP, SVD-LLM with LSP's recipe and capacity, and MatryoshkaKV (GR).

New up-to-date results. We agree that newer models and harder tasks make the evaluation more informative, and thank the reviewer for the suggestions. We now compress two hybrid models, the pre-trained Qwen3.5-4B and the post-trained Qwen3.5-9B (DeltaNet layers included), and evaluate them on reasoning, knowledge, instruction following and code (GSM8K, MATH-500, AIME, MMLU, MMLU-Pro, GPQA-Diamond, BBH, IFEval, HumanEval+, MBPP+), long context (LongBench v2; RULER at 32k and 128k) and tool use (BFCL v4, single- and multi-turn; GW1, T1–T2).

LSP has the highest mean at every ratio on both models. Hybrid models remain compressible, at a task-dependent cost: at −20%, LSP keeps 86% of the 9B's dense mean (71.7 vs. 83.0), and 93% and 98% of its long-context and tool-use scores; hard reasoning (AIME, GPQA) degrades most, and LSP's lead is largest there (AIME26: 50.8 vs. 24.2 for SVD-LLM). The shared latent also makes long contexts cheaper: measured on one GPU, the −70% Llama-2-7B fits 333.8k cached tokens per sequence at batch 8, against 149.5k with SVD-LLM's latent cache and 20.5k for the dense model, and it decodes up to 1.58× faster than dense at batch 1 (App. E, Table 5).

We hope these clarifications and experiments address the reviewer's concerns, and we ask the reviewer if they have any remaining concern or question.


[1] H. Yu, J. Wu. Compressing Transformers: Features Are Low-Rank, but Weights Are Not!, AAAI23.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Global Author Response [1/4]

We thank all reviewers for their time and detailed feedback.

We appreciate the acknowledgement of the well-motivated and clearly defined formulation [3zrR, haXw], the strong gains at aggressive compression [3zrR, haXw], the practicality of the resulting shared-latent cache [3zrR, haXw], and the clear writing [Az5g, 3zrR].

We uploaded a revised PDF with changes in blue; all table, section and appendix numbers below refer to it. This rebuttal adds:

  • (i) two recent hybrid models, the pre-trained Qwen3.5-4B and the post-trained Qwen3.5-9B (Sec. 3.2, Table 3, App. H) [Az5g, haXw];
  • (ii) harder benchmarks for reasoning, knowledge, code, long context (RULER up to 128k, LongBench v2) and tool use (BFCL v4) [Az5g, haXw];
  • (iii) a matched comparison with MatryoshkaKV: LSP-KV leads by 4.0–19.9 Avg6 points (App. J) [3zrR, haXw];
  • (iv) SVD-LLM trained with LSP's recipe and capacity (App. I), and full recovery fine-tuning from LSP's start (App. H) [3zrR, haXw];
  • (v) a measured latent-cache deployment: on one GPU, Llama-2-7B at −70% holds 333.8k tokens per sequence at batch 8, against 20.5k for the dense model (App. E) [haXw].

Below we answer the concerns shared by several reviewers (GW1–GW3); individual points are answered in each reviewer's thread.

[GW1-Az5g,haXw] The paper should test harder benchmarks in more up-to-date configurations

We thank the reviewers for the model [Az5g] and benchmark [haXw] recommendations. Our original setup followed the protocol of the baselines we compare against (SVD-LLM, SliceGPT, Dobi-SVD, Swift-SVD): the most recent, Swift-SVD (ICML26), reports the same six zero-shot benchmarks on Qwen3-4B, its newest model. We agree that newer models and harder tasks make the evaluation more informative.

We now compress the pre-trained Qwen3.5-4B and the post-trained Qwen3.5-9B (which supports thinking and tool calling), and evaluate both on harder benchmarks (T1–T2 below; Sec. 3.2 and App. H).

Models. The two Qwen3.5 models are both hybrid architectures, interleaving Gated DeltaNet linear attention with gated attention. Every linear layer is compressed with the paper's recipe, the DeltaNet projections included.

Calibration. As the main paper's calibration on C4 would not carry reasoning through compression [1,2], we build a calibration mix for each of the two new setups. For the 4B pre-trained model, we use 6.6M calibration tokens: half is pretraining-style text with math and code (25% FineWeb-Edu, 15% FineMath-4+, 10% GitHub Python), half is the training data of GSM8K, MATH and MBPP and MMLU's auxiliary-train set. For the 9B post-trained model, following the idea of RAC [2], we calibrate on 6.8M tokens of the dense model's own responses to Tülu 3 [3] and tool-calling prompts.

Baselines. We compare against the paper's strongest baseline, SVD-LLM with LoRA recovery, in two configurations: (i) the standard version with fixed epochs and rank 8; and (ii) a new improved version matched in budget (number of epochs), capacity (parametrization), and recipe (objective function, early stopping) to LSP, ensuring a like-for-like comparison. In addition, we add a baseline that performs recovery fine-tuning with same LSP distillation signal from NoLSP (NoLSP+RFT).

AuthorsOpenReview ↗

Global Author Response [2/4]

[GW1, continued] Results on Qwen3.5.

  • T1 — Qwen3.5-4B-Base. Share of compressible parameters removed; WikiText-2 perplexity ↓; accuracy or pass@1, % ↑. Mean = mean of the seven new benchmarks.
RatioMethodWiki2pplGSM8KMATH-500MMLUMMLU-ProBBHHumanEval+MBPP+Mean
—Dense8.0680.649.873.048.657.643.371.760.7
−30%SVD-LLM15.6462.723.054.127.340.814.038.937.2
SVD-LLM(matched)14.2867.927.254.129.443.715.240.739.8
NoLSP+RFT11.4565.924.058.130.545.815.943.940.6
LSP11.3866.526.257.830.546.715.944.241.1
−50%SVD-LLM26.4442.57.840.816.835.45.516.423.6
SVD-LLM(matched)22.4852.412.441.717.335.58.525.127.6
NoLSP+RFT17.1850.713.243.717.136.19.124.327.8
LSP16.4951.014.644.618.736.211.626.729.1
−70%SVD-LLM51.8021.44.831.212.730.81.81.914.9
SVD-LLM(matched)37.8129.95.833.213.130.13.79.017.8
NoLSP+RFT25.4935.25.433.814.631.54.911.919.6
LSP25.1335.36.835.014.631.34.314.020.2
  • T2 — Qwen3.5-9B, thinking mode, %. Reasoning, knowledge and code (AIME avg@4, GPQA-Diamond avg@2), long context and tool use ↑. Mean = mean of the fifteen benchmark scores.
RatioMethodGSM8KMATH-500AIME25AIME26GPQA-DMMLUIFEvalHumanEval+MBPP+LongBenchv2RULER-32kRULER-128kBFCL-non-liveBFCL-liveBFCL-multi-turnMean
—Dense95.496.484.285.881.177.989.687.873.554.596.595.686.681.359.483.0
−20%SVD-LLM93.475.629.224.235.464.475.266.550.831.279.568.885.077.032.559.2
SVD-LLM(matched)92.675.837.536.738.666.080.672.058.539.092.481.784.278.550.065.6
NoLSP+RFT92.582.043.345.850.874.380.673.861.146.993.389.687.179.249.570.0
LSP92.783.645.050.852.374.781.979.363.046.194.389.986.580.555.671.7
−40%SVD-LLM81.045.00.02.516.448.556.027.430.227.263.547.380.671.97.840.4
SVD-LLM(matched)83.244.80.02.513.953.166.943.339.922.576.159.682.773.631.646.2
NoLSP+RFT81.540.20.02.515.260.364.143.933.125.878.862.885.274.328.946.4
LSP83.647.60.03.318.760.769.547.041.327.279.063.484.576.836.049.2
−60%SVD-LLM35.47.80.00.09.632.730.73.02.114.740.125.964.535.50.420.2
SVD-LLM(matched)48.59.80.00.08.834.641.87.910.618.351.338.773.759.311.027.6
NoLSP+RFT41.59.40.00.09.641.835.911.012.422.954.032.576.259.91.227.2
LSP46.213.00.00.015.941.148.816.519.016.553.936.376.863.47.630.3

Conclusions. New hybrid models remain compressible, at a cost that depends on the task: at −20%, LSP keeps 86% of the dense Qwen3.5-9B mean (71.7 vs. 83.0), and hard reasoning (AIME, GPQA) degrades most. LSP has the highest mean for both models at every ratio; at −20% on the 9B, its largest lead over SVD-LLM is on AIME26 (50.8 vs. 24.2).

AuthorsOpenReview ↗

Global Author Response [3/4]

[GW2-3zrR,haXw] How does LSP differ from a MatryoshkaKV applied to the network?

We thank the reviewers for this reference, which the revised paper now discusses (Sec. 4, App. J). MatryoshkaKV (MKV) and LSP both train orthogonal projections from a spectral initialization and allocate ranks by output KL. They differ in two ways:

  • (i) MKV trains projections of the cached K/V states (after RoPE) to shrink the KV cache; LSP trains projections of the weights (before RoPE) to factorize them into low-rank factors, which also results in compressing the KV cache when K and V share one projector and its latent is cached.
  • (ii) MKV learns a separate basis for each head, and for K and V; LSP learns one projector shared by all head, and in general among layers that share the same activations
  • (iii) MKV allocates ranks after training by output KL; LSP allocates ranks before training, by measuring the output KL of projecting each weight matrix in isolation.

Empirical comparison. We compare MKV with LSP applied only to the K/V matrices (LSP-KV), at the same cache size and on the same calibration data. Against MKV, LSP-KV still leads by 3.9–19.9 points.

  • T3 — KV-only compression at matched cache size: Avg6 ↑ / WikiText-2 perplexity ↓. NoLSP-KV is LSP-KV's untrained start. Perplexity is on the WikiText-2 test set. Methods are calibrated on WikiText-2 (OPT-1.3B), C4 (Qwen3-4B), and Alpaca (Llama-2-7B), so this test is in-domain only for OPT-1.3B.
Model (dense)KV-cacheMKVNoLSP-KVLSP-KVΔAvg6(LSP-KV−MKV)
OPT-1.3B (46.1/14.6)37.5%41.3/14.245.7/15.745.3/14.8+4.0
25%38.8/17.244.1/16.544.5/14.9+5.6
12.5%33.7/55.339.1/22.843.1/15.8+9.4
Qwen3-4B (61.1/7.9)37.5%51.8/12.140.3/35.557.6/8.9+5.8
25%46.4/18.234.5/82.954.6/9.9+8.1
12.5%34.6/230.732.8/346.049.1/12.9+14.5
Llama-2-7B (56.9/5.5)37.5%50.8/11.156.3/5.956.6/5.6+5.8
25%47.8/20.452.3/7.756.3/6.0+8.5
12.5%34.4/463.243.7/18.654.3/7.4+19.9

LSP-KV has the higher zero-shot average performance (Avg6) at every model and budget, by 4.0 to 19.9 points, and the gap widens as the cache shrinks: at 12.5% on Llama-2-7B, MKV's perplexity reaches 463 while LSP-KV's stays at 7.4. Even the untrained NoLSP-KV surpasses MKV in Avg6 on OPT-1.3B and Llama-2-7B.

AuthorsOpenReview ↗

Global Author Response [4/4]

[GW3-3zrR,haXw] It is not clear if LSP advantage comes from the method, from larger capacity, or from better signal

In the paper, we reported SVD-LLM as proposed by the original paper: whitened truncation followed by two rank-8 LoRA fine-tuning stages (U, then V) for a fixed number of epochs. LSP uses a different recipe: a KL loss, a larger budget of 30 epochs with early stopping, and a larger parametrization space (despite being restricted to the model's weights subspace). We now compare LSP with baselines matched in recipe, budget and capacity (some already reported above in T1–T2).

(i) Does SVD-LLM fill the gap with LSP at matched setup? No.

We introduce a matched control, with the same 30-epoch budget and validation-based early-stop selection, KL loss, and increased rank to match LSP's parametrization capacity, reported here as "SVD-LLM(matched)":

  • T1 and T2 reported above, show results for Qwen3.5-4B and Qwen3.5-9B. SVD-LLM(matched) improves over SVD-LLM but does not reach LSP performance.
  • T4 — Zero-shot accuracy (Avg6 ↑) of SVD-LLM trained like LSP. The SVD-LLM column keeps its original cross-entropy recipe; its counterpart is LSP$^T$.
SettingRatioSVD-LLMSVD-LLM(matched)LSPLSP$^T$
Llama-2-7B, Alpaca−30%51.652.352.952.9
−50%44.646.847.847.3
−70%36.040.342.240.4
Qwen3-4B, C4−20%51.153.657.256.4
−40%44.748.150.750.6
−60%35.640.843.843.3

Trained like LSP, SVD-LLM (matched) gains 0.7–5.2 points over its published recipe, but LSP still leads in all six cells: by +0.6 / +1.0 / +1.9 on Llama-2-7B and by +3.6 / +2.6 / +3.0 on Qwen3-4B.

(ii) Does LSP lead only at the end? No.

  • T5 — Avg6 after k epochs, Qwen3-4B (C4). SVD-LLM: at each epoch, the best of four variants (CE or KL; LoRA rank 8 or matched).
RatioMethodep1ep2ep3ep6ep10ep15final
−20%SVD-LLM(best-of-variants)51.852.853.553.653.753.453.8
LSP54.355.555.756.156.757.157.2
−40%SVD-LLM(best-of-variants)45.246.647.547.747.547.648.1
LSP47.348.749.149.950.650.350.7
−60%SVD-LLM(best-of-variants)36.237.638.839.940.640.740.8
LSP37.940.040.241.342.343.143.8

LSP leads at almost every epoch, and keeps improving after SVD-LLM saturates (3–12 epochs).

(iii) Does the gain come from the projector parameterization, or from optimizing the subspace end to end?

  • T1 and T2 reported above isolate the projector parameterization with NoLSP+RFT: it starts from the same compressed model (LSP's epoch 0: same allocation, ties and whitened initialization), trains with the same KL objective, data, epoch budget and validation-based epoch selection, and changes only what is trained: both low-rank factors of every layer, directly. On perplexity, projectors and factors reach about the same quality from the same start (WikiText-2 within 0.6–4.2% on Qwen3.5-4B). On benchmarks, the projector matches or exceeds direct factor training: NoLSP+RFT trails LSP's Mean by 0.5 / 1.3 / 0.6 points on Qwen3.5-4B, and by 1.7 / 2.8 / 3.1 points on Qwen3.5-9B, scored with long thinking-mode generation. LSP gets there by only projecting frozen pre-trained weights.

[1] Bandari et al., Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning, EMNLP24

[2] Lucas et al., Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction, ICLR26

[3] Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, COLM25

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.