- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Learnable Subspace Projections (LSP), a low-rank neural network compression method that learns functional subspaces through end-to-end optimization. The authors argue that conventional SVD and activation-aware low-rank compression methods mainly rely on local statistics or reconstruction objectives, without fully accounting for how compression errors propagate through the network. To address this limitation, LSP introduces learnable orthogonal projectors for individual linear layers or groups of layers sharing the same inputs. The projectors are initialized using whitened SVD, while rank budgets are allocated based on the full-model output KL divergence induced by compressing each unit individually. All projectors are then jointly optimized to preserve the dense model's output distribution, with the pretrained weights kept frozen. After training, the projectors are merged into standard low-rank factors, and shared input factors across Q/K/V projections can further reduce computation and potentially KV-cache storage. Experiments on four decoder-only LLMs and ViT-B/16 demonstrate the effectiveness of LSP. At 70% parameter compression on Llama-2-7B, LSP achieves a WikiText-2 perplexity of 10.9 and an average accuracy of 42.2% across six zero-shot tasks. The paper also reports up to 1.56× faster short-context decoding and an estimated 13.5× reduction in combined weight and KV-cache memory using latent caching.
Critical strengths
Strength 1: The paper presents a clear approach to function-aware low-rank compression by learning which orthogonal subspaces to remove, rather than relying solely on fixed singular-vector rankings or local reconstruction criteria. Unlike conventional activation-aware SVD methods, LSP jointly optimizes layer-wise projections using the KL divergence between the compressed and dense models' output distributions, allowing the retained subspaces to reflect global functional sensitivity. The method has a well-defined optimization objective and enables end-to-end learning of the projection matrices while keeping the pretrained weights frozen.
Strength 2: The paper demonstrates competitive results at high compression ratios. In particular, at 70% compression on Llama-2-7B, LSP reduces WikiText-2 perplexity from 13.3 with the SVD-LLM recovery baseline to 10.9, while improving average accuracy across six zero-shot tasks from 36.0% to 42.2%. The NoLSP ablation further shows that learning the projections substantially improves model quality under the same initialization, rank allocation, and shared structure. The paper also evaluates multiple compression ratios, different training objectives, and three random seeds, along with calibration distribution shifts and cross-task transfer on ViT-B/16.
Strength 3: LSP can be merged into standard low-rank factors for deployment, without retaining the QR operations or additional projections used during training. Sharing input factors across Q/K/V and gate/up projections also reduces redundant computation and enables low-dimensional latent caching under certain attention configurations. The paper reports measured decode throughput and analyzes how shared factors and rank allocation affect inference efficiency, demonstrating practical benefits beyond compression accuracy.
Critical weaknesses
Weakness 1: MatryoshkaKV [3] has already explored training orthogonal projection matrices to compress KV representations while optimizing low-dimensional subspaces based on model performance. LLRC [4] studies differentiable selection of singular directions, while QSVD [5] introduces joint Q/K/V low-rank factorization to reduce parameter and KV-cache costs. Therefore, optimizing projection subspaces through training objectives and exploiting shared Q/K/V representations for efficient inference both have clear precedents. The main distinction of LSP appears to be extending trainable projections to linear layers throughout the network and jointly optimizing shared subspaces against a global KL objective. Although this extension has potential value, the paper does not sufficiently discuss its relationship to MatryoshkaKV and LLRC, nor does it provide direct comparisons under matched compression budgets to establish the independent advantages of joint subspace learning over existing trainable subspace methods.
Weakness 2: The NoLSP ablation demonstrates the importance of joint training, and the comparison with SVD-LLM [1] using LoRA recovery is valuable. However, LSP requires multiple full-model forward and backward passes, QR orthonormalization, and projection optimization, while the baselines differ in their trainable parameter counts, training iterations, and optimization flexibility. In particular, LSP directly modifies the retained subspaces by training high-dimensional projection matrices, whereas SVD-LLM uses a relatively small LoRA recovery configuration. Consequently, the performance gains cannot be attributed entirely to orthogonal subspace learning. The paper lacks comparisons with direct low-rank factor optimization or more extensive recovery training under matched calibration data, optimization steps, trainable parameter budgets, and GPU-hours.
Weakness 3: Eq. (3) measures the KL divergence induced by compressing each unit individually while keeping the rest of the network dense, and Eq. (17) uses these isolated costs for greedy rank allocation based on marginal KL per parameter saved. However, these isolated measurements do not capture error propagation and interactions among multiple low-rank projections applied simultaneously.
Weakness 4: The paper highlights a 13.5× reduction in combined weight and KV-cache memory at 70% compression, along with an estimated single-GPU context capacity of approximately 322k tokens. However, Appendix E explicitly states that these results are analytical estimates based on low-rank factor dimensions, while the measured decode throughput uses a standard full-width KV cache rather than actual latent-cache inference. Therefore, the experiments do not directly validate the peak GPU memory usage, runtime latency, additional K/V reconstruction overhead, or long-context generation quality of the proposed latent caching scheme.
Weakness 5: The largest LLM evaluated is Llama-2-7B, leaving the scalability of projection training, KL probing, and rank allocation unverified on larger 13B/32B/70B models. The downstream evaluation mainly covers six zero-shot commonsense and basic mathematical reasoning tasks, without more challenging benchmarks such as GSM8K, MATH-500, AIME, MMLU, or code generation. Moreover, the LSP compression pipeline involves full-model KL probing and multiple rounds of projection training, requiring several hours and multiple GPUs for some configurations. The code is also planned for release only after acceptance, limiting independent reproducibility at this stage. Given the offline compression efficiency of Swift-SVD [8] and other lower-cost methods, a clearer analysis of the trade-off between compression cost, training memory, and model quality is needed.
[1] Wang, X., Zheng, Y., Wan, Z., and Zhang, M. SVD-LLM: Truncation-Aware Singular Value Decomposition for Large Language Model Compression. In International Conference on Learning Representations (ICLR), 2025.
[2] Wang, Q., Ke, J., Tomizuka, M., Keutzer, K., and Xu, C. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives. In International Conference on Learning Representations (ICLR), 2025.
[3] Lin, B., Zeng, Z., Xiao, Z., Kou, S., Hou, T., Gao, X., Zhang, H., and Deng, Z. MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection. In International Conference on Learning Representations (ICLR), 2025.
[4] Sundrani, S., Tudisco, F., and Minervini, P. Low-Rank Compression of Language Models via Differentiable Rank Selection. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC), pp. 10031–10045, 2026.
[5] Wang, Y., Wang, H., and Zhang, S. Q. QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
[6] Chang, C.-C., Lin, W.-C., Lin, C.-Y., Chen, C.-Y., Hu, Y.-F., Wang, P.-S., Huang, N.-C., Ceze, L., Abdelfattah, M. S., and Wu, K.-C. Palu: KV-Cache Compression with Low-Rank Projection. In International Conference on Learning Representations (ICLR), 2025.
[7] Abbasi, A., Thrash, C., Qin, H., Sharma, S., Seifi, S., and Kolouri, S. Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression. In Proceedings of the 43rd International Conference on Machine Learning (ICML), pp. 84–100, 2026.
[8] Qi, R., Liu, Y., Wu, X., Wang, X., Li, M., Chen, C., Chen, J., Chen, Y., and Weng, Q. Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.
Questions for authors
Q1: MatryoshkaKV [3] has already explored training orthogonal projections to optimize low-dimensional KV representations, LLRC [4] uses differentiable learning to select low-rank directions, and QSVD [5] investigates joint Q/K/V low-rank compression. Could the authors further clarify the independent technical contributions of LSP compared with these methods, particularly what new optimization properties or practical advantages arise from jointly learning orthogonal projectors across the entire network? Could the authors also provide direct comparisons under comparable compression settings to better establish the independent contribution of the proposed subspace learning approach?
Q2: LSP requires multiple full-model forward and backward passes, QR orthonormalization, and projection optimization, whereas the LoRA recovery stage of SVD-LLM [1] uses a relatively limited number of trainable parameters. Could the authors compare LSP with more extensive SVD-LLM recovery or direct low-rank factor optimization under matched calibration data, optimization steps, trainable parameter budgets, and GPU-hours? In particular, could the authors better distinguish whether the performance gains come from the orthogonal subspace parameterization itself or from additional end-to-end optimization?
Q3: Eq. (3) estimates the KL cost of each compression unit independently, while Eq. (17) uses these isolated costs for greedy rank allocation without explicitly accounting for interactions between compressed layers. Appendix B further reports that, on Llama-2-7B, the joint compression KL can be 2.1–4.3× the sum of the isolated KL costs. Could the authors further analyze how this discrepancy affects the final rank allocation and compare the current strategy with joint-KL evaluation or interaction-aware allocation? Additionally, could the authors clarify the efficiency or accuracy advantages of the proposed greedy allocation over more direct global optimization approaches?
Q4: The reported 13.5× reduction in combined weight and KV-cache memory and the estimated 322k-token context capacity are primarily based on analytical estimates, while the measured decode throughput still uses a standard full-width KV cache. Could the authors provide end-to-end experiments with actual latent-cache inference, including GPU peak memory, TTFT, prefill and decode throughput, and K/V reconstruction overhead? In particular, could the authors evaluate the actual memory and computational benefits across different context lengths and batch sizes, together with long-context generation quality, to verify whether the theoretical storage advantages translate into practical deployment benefits?
Q5: The largest LLM evaluated is Llama-2-7B, while KL probing and joint projector training introduce additional computational and memory costs. Could the authors extend the evaluation to larger 13B/32B models and include more challenging reasoning and code generation benchmarks, such as GSM8K, MATH-500, AIME, MMLU, HumanEval, or MBPP, to assess capability preservation under aggressive compression? Additionally, could the authors provide detailed GPU-hours, peak GPU memory, and total compression time for KL probing, projector training, and model merging across different model sizes, and compare the resulting accuracy–compression-cost trade-off more directly with lower-cost methods such as Swift-SVD [8]?
3 comments
Author Response to Reviewer haXw [2/3]
[W4/Q4] The latent-cache scheme is not validated end to end. The reviewer is right. In the paper, only decode throughput was measured (prefill and decode FLOPs were counted), and the 13.5× reduction of weight-plus-cache memory and the 322k-token capacity were analytical estimates from the factor widths, as App. E and the Figure 3 caption state. The cost of rebuilding keys under RoPE was only discussed, and long-context quality was not evaluated.
What we ran now. We implemented the latent-cache deployment, as in Palu (a static latent buffer with chunked prefill, values folded into the output projection, and a fused Triton kernel that rebuilds keys from the latent, applies RoPE and computes scores in one pass). We measured it on the WikiText-2-calibrated Llama-2-7B at −70%, on one GH200 in bf16: peak memory, the longest context that fits, prefill time (the time to first token), decode throughput at several context lengths and batch sizes, and the key-rebuild overhead against a full-width cache of the same weights.
| Llama-2-7B, −70% | Dense | LSP, full-width cache | LSP, latent cache | SVD-LLM (W), latent cache* |
|---|---|---|---|---|
| Cached floats/token/layer | 8192 | 8192 | 556 | 1248 |
| Longest context at batch 8 (measured) | 20.5k | 22.5k | 333.8k | 149.5k |
| Peak memory at 8×128k tokens | OOM | OOM | 40.1GiB | 83.6GiB |
| Prefill 8×16k: time/peak | 7.9s/80.4GiB | 5.2s/72.2GiB | 5.2s/12.4GiB | 4.9s/17.8GiB |
| Prefill 8×128k: time/peak | OOM | OOM | 198.4s/42.9GiB | 198.4s/86.1GiB |
| Decode, batch 1, tok/s at 512/4k/16k/128k | 134/115/100/35 | 203/162/132/38 | 212/167/92/16 | 186/147/81/14 |
| Decode, batch 8, tok/s at 4k/64k/256k | 627/OOM/OOM | 783/OOM/OOM | 424/33/8 | 380/29/OOM |
SVD-LLM (W): SVD-LLM's whitened truncation (its LoRA recovery keeps the same shapes). *Ranks padded to multiples of 16 (1228 floats unpadded); see the correction below.
The measurements confirm the paper's estimates, conservatively:
- Memory. The latent cache holds 333.8k tokens per sequence at batch 8 (322.1k estimated) and completes a real 8 × 256k-token prefill at a 77.7 GiB peak, where the full-width cache stops at 22.5k tokens.
- Quality. It is exact (WikiText-2 perplexity 10.9498 vs. 10.9502 with the full-width cache), so long-context quality is that of the compressed model (RULER at 128k on Qwen3.5-9B, GR T2: 89.9 at −20%, dense 95.6).
- Speed. Rebuilding keys under RoPE costs decode speed as context and batch grow: at batch 1, the latent cache decodes as fast as the full-width one up to 4k tokens, and at 0.69× / 0.42× its speed at 16k / 128k; at batch 8, the cost already shows at 4k (0.54×). Palu pays the same cost, which DeepSeek-V2 avoids by design with a separate, decoupled RoPE key.
- Use. Both deployments compute the same model, so a server can pick one per request: the full-width cache when the context fits and speed matters, the latent cache when memory is the limit, where it serves contexts the full-width cache cannot hold (at batch 8, beyond 22.5k and up to 256k tokens).
Correction to the paper. Re-timed with rank-aligned baselines (ranks zero-padded to multiples of 16, which is exact), LSP remains the fastest factorization in all nine cells, but by +11–14% at 512 tokens rather than +15–25%, and untied factorizations are no longer slower than dense at −30%. We correct Sec. 3.4 and App. E accordingly.
Author Response to Reviewer haXw [3/3]
[W5/Q5] LSP is not tested at larger scale or on harder benchmarks, and it is not clear when it is preferable to cheaper alternatives.
Larger models. Beyond Llama-2-7B, we now compress the slightly larger Qwen3.5-9B (GR, T2); given the other experiments requested, a 13B–32B run did not fit the rebuttal period. We note that, compared with training-free methods, training-based methods are more limited by model size, which we added as a limitation to the revised paper.
Harder benchmarks. We now add harder benchmarks that evaluate them on reasoning, knowledge, instruction following and code (GSM8K, MATH-500, AIME, MMLU, MMLU-Pro, GPQA-Diamond, BBH, IFEval, HumanEval+, MBPP+), long context (LongBench v2; RULER at 32k and 128k) and tool use (BFCL v4, single- and multi-turn). We refer to (GW1, T1–T2) for all the new benchmark results.
Cost of the KL probe. On Llama-2-7B, the probe takes around 10% of LSP's compression time at −50% and −70%, and one probe serves every ratio and objective; it costs 0.2 / 1.7 / 1.1 wall-clock hours on OPT-125M / Qwen3-4B / Llama-2-7B (App. F). When compression time matters, it can also be skipped: uniform allocation stays within 1.3 Avg6 points of the measured-KL allocation in five of six settings (Table 11).
Pareto optimality. App. F reports compression cost against perplexity for every method: LSP is more expensive than the alternatives, but it is Pareto-optimal at higher budgets, as at every model and ratio one of its objectives gives the lowest perplexity within 0.5–10 hours. Training-free methods are much cheaper but degrade at high compression (Swift-SVD reaches 203.6 perplexity at −70% on Llama-2-7B, against LSP's 10.9). Moreover, compression is paid once per model and ratio, while quality and inference cost are paid at every query, and LSP also lowers the latter (faster decoding and a shared latent cache, W4).
Code. We confirm that the full code will be released upon acceptance.
We hope these answers address the reviewer's concerns, and ask if there are any other concerns or questions.
Author Response to Reviewer haXw [1/3]
We thank the reviewer for the detailed and constructive review. W1/Q1, W2/Q2 and the harder benchmarks of Q5 are addressed with new experiments in the Global Response (GR), summarized below; we answer W3–W5 in full. Table, section and appendix numbers refer to the revised PDF.
[W1/Q1] Not sufficiently discussing differences with MatryoshkaKV, LLRC, QSVD and Palu.
We thank the reviewer for the references. We note that QSVD (related work) and Palu (Sec. 3.3 of the submission, Sec. 3.4 in the revision) were already cited in the paper, but agree that a clearer presentation and comparison with LSP helps the manuscript. Experimentally, we now compare with MatryoshkaKV under matched compression (GR, T3), while no code is available for LLRC.
[W2/Q2] SVD-LLM uses limited resources compared to LSP (GR, T1-T2-T4-T5).
We refer to GW3 in the Global Response for the full results; in short:
[W3/Q3] Isolated KL costs in the allocation. As mentioned in the limitations (App. A), the KL probe is indeed an additive surrogate (the joint KL is 2.1–4.3× the sum of the isolated costs on Llama-2-7B (App. B)). However, the greedy allocator only uses the relative order of marginal costs per saved parameter, so interactions that inflate all costs alike leave the allocation unchanged. In practice, we compare the KL allocation with fixed-rank, uniform, and diagonal-Fisher allocations, and find KL to perform best overall. A joint search would need a forward pass of the compressed network per candidate move and allocation state; the isolated probe runs once per model and serves every ratio and objective. We leave interaction-aware allocation to future work.