acceptodds
Under review as a conference paper at ICLR 2027

ShareSVD: Sharing Key Subspaces for Low-Rank Generative Transformers

Abstract

Singular value decomposition provides a practical low-rank factorization of attention projections, reducing parameter count and computation while supporting compact latent KV caches. Existing methods typically factorize Query, Key, and Value projections independently, ignoring both their unequal sensitivity to rank reduction and cross-head subspace redundancy. Our systematic study across vision-language models (VLMs) reveals a consistent asymmetry: Value is the most fragile projection, whereas Key is the most robust. Motivated by this observation, we introduce ShareSVD, which adaptively groups geometrically compatible Key heads and shares a low-dimensional input basis within each group while retaining head-specific output factors. This sharing reduces the parameters required for Key factorization, enabling more low-rank directions to be retained in the more sensitive Query and Value projections. At a retained QKV ratio of 0.3, ShareSVD improves average accuracy over the competitive baseline by 8.57% across three VLMs and two benchmarks, while reducing low-rank KV-cache memory by 14.7%. Consistent perplexity improvements on dense and MoE Qwen3 models further demonstrate that ShareSVD generalizes beyond multimodal architectures. Code is available at https://anonymous.4open.science/r/ShareSVD-F0E0.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Singular Value Decomposition, Vision language model, Large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CdhuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ShareSVD groups Key heads using activation-aware subspace distances, shares an input basis obtained through joint SVD while retaining head-specific output factors, reallocates Q/K/V capacity using task-aware direction scores and a heuristic allocation rule, and performs Fisher-weighted local refinement. Evaluation covers three VLMs on ScienceQA-IMG and SEED-Bench-IMG, with the strongest reported gains at low retention ratios. Additional results examine two Qwen3 models, quantization compatibility, and materialized low-rank KV-cache allocation. The appendix provides a conditional least-squares result, group statistics, and a refinement ablation.

Critical strengths
  1. A useful shared representation with targeted support for its design choices. Activation-aware grouping, a shared Key input factor, and head-specific output factors provide a practical structural extension of per-head low-rank factorization. Table 2 reports higher average accuracy for geometry-based grouping on all three models while matching group-size profiles. Proposition 1 supports the initialization's conditional optimality for a fixed subspace. Together, these provide meaningful support beyond an unexamined change in factorization.

  2. Reported low-budget benefits extend across models and datasets. At ρ=0.3, Table 1 improves on WSVD in all six VLM–dataset combinations by approximately 8.57 accuracy percentage points on average. Improvements on SEED-Bench-IMG after ScienceQA calibration provide some cross-dataset evidence; Figure 6 reports lower perplexity for two Qwen3 models at most evaluated retention ratios. Table 4 also directly supports approximately 14.7% lower materialized low-rank cache allocation on average at the measured configuration. These constitute the main empirical case for acceptance, conditional on verification of the realized budgets.

Critical weaknesses
  1. The relationship between the equal-budget reference and the claimed sharing savings is not coherently specified. Equation 10 defines RrefR^{ref} using the cost d+dhd+d_h of an independent per-head direction. The following text then selects the top RrefR^{ref} group candidates. Under the literal reading ∑GrG=Rref\sum_G r_G=R^{ref}, the shared cost is ∑GrG(d+gGdh)=(d+dh)Rref+dh∑G(gG−1)rG\sum_G r_G(d+g_Gd_h)=(d+d_h)R^{ref}+d_h\sum_G(g_G-1)r_G, exceeding that reference budget. The saving d∑G(gG−1)rGd\sum_G(g_G-1)r_G in Eq.11 instead compares against a different reference in which every member head retains rGr_G directions. The sharing identity itself is correct, but the conversion between these references is missing. This affects the central reallocation mechanism and interpretation of equal-budget performance. Page 6 and Algorithm 1 state that realized costs are finally reconciled, so this establishes a problem in the written accounting rather than proving that the implementation or experiments exceed their budgets.

  2. The contributions of sharing and rank reallocation are not sufficiently separated. The Share-Q/K/V and random-grouping comparisons support design choices but also involve capacity allocation. The paper does not clearly compare adjusting Q/K/V ranks alone, sharing with fixed per-projection budgets, and the full method under the same realized parameter budget, direction scoring, and refinement setup. The appendix does provide a refinement ablation: Table 7 increases SmolVLM accuracy from 16.16 to 47.23 at ρ=0.3, showing that refinement matters substantially. The evidence supports the reported combined pipeline but does not yet sufficiently attribute its gains to the proposed sharing-and-reallocation mechanism. A bounded component comparison would materially affect the assessment of its knowledge contribution.

  3. The memory–latency trade-off of actual inference is not sufficiently established. Section 4.3, Figure 4, and Table 4 do support approximately 14.7% lower materialized BF16 low-rank KV-buffer allocation at ρ=0.3,B=1,L=2048\rho=0.3,B=1,L=2048. However, the paper reports neither peak memory nor latency/throughput for actual prefill and decoding, and does not specify whether latent caches are retained throughout, whether per-head K/V tensors are temporarily reconstructed, or which attention implementation is used. Equation 9 shares input factors while retaining head-specific output factors, making runtime compute and temporary storage implementation-dependent. For a method motivated by compression and cache savings, this limits assessment of its deployment value. The appropriate request is to quantify benefits and costs, not to assume it must be faster. If the claims are restricted to the measured buffers, the concern limits the scope and practical evidence rather than invalidating the existing measurements.

Questions for authors
  1. How does the reference direction count in Eq.10 map to group ranks and final direction counts in Eqs.11–12? Please use a two-head group to illustrate how Algorithm 1 enforces the final budget, and report realized Q/K/V factor parameter counts and deviations from target budgets for representative configurations.

  2. Under matched realized QKV budgets, sharing structures, calibration, and refinement, how does the proposed rule compare with uniform budget allocation and importance-based allocation? Please report accuracy and Q/K/V rank assignments.

  3. Could you jointly report accuracy, actual GPU memory usage, and inference throughput across representative compression settings to characterize the accuracy–efficiency trade-off?

If the authors address these concerns satisfactorily, I would be willing to raise my rating.

1 comment

I thank the authors for their time in responding to my concerns. I have decided not to increase my score because my main concerns remain unresolved. First, the introduction states that the primary motivation for this method is that it provides improved parameter compression rates by leveraging the robustness of the key heads. As shown in Figure 1, this is in fact not the case in two of the three models tested. If the authors aim to instead emphasize that this approach is a more performant KV cache compression method over prior work then the authors should instead state that in the introduction and include relevant KV cache compression baselines. Currently, the KV cache comparisons are presented as secondary.

Second, as mentioned in my earlier comment, the steps taken in the various stages in Section 3 lack formal justification. For instance, the authors compute the principal angles using Cholesky factors but not provide a formal justification for this approach. Moreover the authors refer to their decomposition as “activation aware” but also do not provide formal justification for this approach over naive non-activation aware approaches.

Reviewer_StLpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a method for adaptively compressing Q/K/V weights in transformer models. Their approach is comprised of a number of stages including the grouping of Key head weights, compression of the groups, reallocation of the compression budget to Q and V heads.

Critical strengths
  • The work shows good performance compared with baselines
Critical weaknesses
  • Saying that K is remarkably robust is too strong. If I understand correctly, Figure 1 shows that in 2/3 architectures Q is slightly more robust than K?
  • The author's method consists of a number of heuristics that lack justification or further analysis. For instance, it is unclear why exactly the SVD of S[W_1, …, W_g] is performed as opposed to simply performing the SVD of [W_1, …, W_g]. The authors refer to this as an activation aware approach but do not provide sufficient formal justification for this approach. Similarly, in the scoring stage the authors do not compare their approach against a simpler variant such as merely using phi scores to adaptively select the rank allocation for each matrix.
  • The authors present their task utility score (eq. 7) as a novel contribution, however an earlier work has proposed a similar score which has not been cited [1].

[1] https://arxiv.org/pdf/2606.00573v1

Questions for authors

There are a number of sentences I do not understand. What is meant by these:

  • “The full Cholesky factor pre-serves cross-channel activation correlations, so discrepancies in the transformed weights reflect re- construction error on the calibration outputs.”
  • ”Complete-linkage agglomeration partitions the h Key heads into disjoint groups”
Reviewer_BpbaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper compresses attention projections in VLMs and LLMs with low-rank factorization. It first shows that reducing rank in V hurts most and in K hurts least (Fig. 1). ShareSVD then clusters Key heads by principal-angle distance between their whitened input subspaces, and has each group share one input factor A_K while keeping head-specific B_K. The parameters saved go to Q and V through a ratio-dependent rule (section 3.6), directions are picked by a gradient-weighted score, and factors are refined with 100 steps of Fisher-weighted reconstruction. The evaluation covers SmolVLM, LLaVA-1.5-7B and LLaVA-1.6-13B on ScienceQA-IMG and SEED-Bench-IMG, plus WikiText-2 perplexity on Qwen3-8B and Qwen3-30B-A3B. Ablations cover which projection to share, max group size, random grouping, W8A8, and fine-tuning.

Critical strengths

The single-projection sensitivity study (Fig. 1) lines up with the projection-sharing ablation. At ρ = 0.3, sharing K keeps accuracy at 58.39%, against 45.64% for Share-V and 52.42% for Share-Q (Fig. 3, Section 4.3), and it also cuts cache memory relative to WSVD (Fig. 4, Table 4). That’s a practical design rule other SVD methods could adopt even without the grouping machinery.

The ablations are honest about where the gains come from. Table 7 shows fine-tuning is responsible for most of the ρ = 0.3 improvement on SmolVLM, and Table 6 shows realized group sizes stay small (2.87 at g_max = 8). I appreciated that they didn’t hide either.

Critical weaknesses

Positioning and efficiency. Sharing A across heads in a group is the same structure as factorizing the concatenated group weight at rank r. PaLU already proposes a group-head variant of its low-rank decomposition, yet PaLU isn’t a baseline and the paper doesn’t discuss how ShareSVD differs from it. As far as I can tell, the actual new pieces are geometry-based grouping and Q/V reallocation, and only grouping is ablated against a meaningful alternative (Table 2, and that’s random grouping, not fixed contiguous groups).

On efficiency, the only measurement is allocated BF16 cache memory at B = 1, L = 2048 and ρ = 0.3 (section 4.3, Table 4). All the evaluated models use rotary embeddings, so caching X·A_K means keys have to be rebuilt and rotated at every decode step. The paper never mentions this or reports latency or throughput. For a paper whose motivation is deployment cost, that’s a real gap.

The headline number is fragile. The 8.57-point average gain at ρ = 0.3 (abstract, 4.2) is absolute accuracy, and about two thirds of it comes from the two SmolVLM rows (+15.71 and +18.78). In that setting WSVD sits at 31.52 ± 15.74 over three seeds (Table 5), so I don’t trust the gap much. Without fine-tuning, ShareSVD on SmolVLM at ρ = 0.3 drops to 16.16 ± 18.07 (Table 7), below WSVD.

The baselines also aren’t on equal footing. section 4.1 says each baseline uses its official defaults, so ASVD and SVD-LLM get no local refinement, while ShareSVD gets 100 steps on the same calibration samples. ASVD near 0% on SmolVLM up through ρ = 0.6 looks like a configuration problem more than a fair comparison.

Questions for authors

(Clarification) Did WSVD get the same 100 fine-tuning iterations on the same 256 calibration samples as ShareSVD? If not, I’d want the ρ = 0.3 to 0.5 comparison with matched refinement, since Table 7 suggests that’s where much of the gain lives.

(Clarification) How does ShareSVD compare to PaLU’s grouped-head decomposition with fixed group sizes at the same QKV budget? This is the comparison that would separate the value of geometric grouping and reallocation from the value of sharing at all.

(Clarification) How are rotary embeddings handled when keys are reconstructed from the shared latent at decode time, and what are the decode latency and throughput against WSVD and FP16? A memory number at B = 1 isn’t enough to back the deployment claims.

(Clarification) In Table 2, SmolVLM and LLaVA-13B report identical values at ρ = 0.5 for both random grouping (73.67) and Grassmann (73.49), and the 73.49 also appears for both models in Table 1. Table 5 gives them different standard deviations, so it may be coincidence, but please confirm there’s no transcription error.

(Clarification) Does the Γ_K = ρ·S_K split in section 3.6 actually bind after the final ϕ-based selection? And was it compared against letting ϕ allocate the freed budget with no fixed split?

(Future work) Given that V is fragile but contributes most of the latent cache, have you tried sharing one basis between K and V within a head group, MLA-style, but built from pretrained weights? It would be a natural follow-up and would test whether the asymmetry holds in that setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.