acceptodds
Under review as a conference paper at ICLR 2027

DynSVD: Dynamic SVD via Token-Adaptive Basis Selection

Abstract

Existing Singular Value Decomposition (SVD)-based compression methods construct static low-rank bases that are fixed across all inputs, even though the contribution of each basis depends jointly on its singular value and the input token. As a result, singular directions with small singular values, which can be highly activated for specific inputs, are permanently discarded by static truncation. To address this issue, we propose DynSVD, a training-free, token-adaptive inference framework that selects bases dynamically on top of static SVD. We prove that dynamic selection is optimal for each token and that its error adds to the static truncation error without cross terms, which lets DynSVD complement static SVD methods. Experiments show that, at the same computation budget, DynSVD lowers perplexity by 42% on average relative to static SVD. Within a fixed GPU memory budget, one stored DynSVD model offers a flexible, continuous range of operating points through a single threshold, and with custom Triton kernels it yields up to higher decoding throughput at the same perplexity and up to 27% lower perplexity at the same throughput than static SVD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Rank Decomposition, SVD Compression, Adaptive Inference, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D6mXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SVD-based compression methods reduce the memory footprint and inference latency of deep learning models by replacing large weight matrices with a multiplication of two low rank matrices. On the other hand, adaptive inference methods address memory and latency by activating only specific portions of the model based on the input. This work introduces a new adaptive inference method which builds on SVD-based compression. Once a model has been compressed using an existing SVD-based compression algorithm, their method selects only a subset of the low rank factors to activate at each layer. As a result, a larger number of parameters may be retained in each layer while maintaining similar memory and inference latency to a model compressed with a larger compression rate. The authors show that their method is able to outperform existing SVD-based compression methods while using the same amount of memory at inference.

Critical strengths
  • An adaptive inference method based on a low rank decomposition of the model's weight matrices is genuinely interesting. If it can be shown to have advantages over other adaptive inference methods I believe it is a nice contribution.
Critical weaknesses
  • The paper’s experiments exclusively compare with SVD compression methods. However, I believe it is necessary to provide a comparison with other adaptive inference methods. Ideally, the experiments would clearly show what advantage an SVD-based adaptive inference method presents when compared with existing methods.
  • The method presented builds directly upon existing SVD compression methods. In doing so, the authors avoid identifying novel decomposition strategies that account for their adaptive inference method.
Questions for authors

Can the authors clarify what advantages the SVD-based adaptive inference method has over existing adaptive inference methods such as TEAL?

4 comments

Thank you for your careful review and constructive feedback. Below, we answer each point and note where we have revised the manuscript.

[W1], [Q1] Comparison with adaptive inference methods

Activation sparsity methods such as TEAL (to input xx) and DynSVD (to intermediate z^\hat{z}) both reduce the computation for each token. TEAL on the original model keeps all weights in memory, so it is a good choice when the original model fits. The advantage of DynSVD appears when the model has to be compressed to fit in memory: (1) TEAL on the original model does not fit, while DynSVD runs on the compressed model, and (2) on the same SVD-compressed model, DynSVD reaches lower perplexity and faster decoding than TEAL at similar active parameters.

The table below compares DynSVD with TEAL on LLaMA-7B, on the original model and on the same SVD-LLM model under 10 and 8 GB budgets. We calibrate TEAL's per-module thresholds on each model and use its official sparse kernel.

ModelMemory (GiB)Act. / Total (B)tok/sWiki2↓
Original13.476.48 / 6.4891.8 (1.00×)5.70
Original + TEAL 25%13.504.86 / 6.48106.7 (1.16×)5.91
Original + TEAL 50%13.503.24 / 6.48137.2 (1.49×)6.75
SVD-LLM9.674.43 / 4.43112.5 (1.00×)10.46
DynSVD-T9.673.95 / 4.43116.4 (1.03×)10.61
DynSVD-T9.673.47 / 4.43126.3 (1.12×)11.90
SVD-LLM + TEAL 25%9.683.92 / 4.43110.9 (0.99×)11.76
SVD-LLM + TEAL 50%9.683.40 / 4.43121.8 (1.08×)15.19
SVD-LLM7.553.30 / 3.30136.1 (1.00×)24.56
DynSVD-T7.552.94 / 3.30139.8 (1.03×)25.09
DynSVD-T7.552.58 / 3.30149.3 (1.10×)30.03
SVD-LLM + TEAL 25%7.612.92 / 3.30133.7 (0.98×)31.85
SVD-LLM + TEAL 50%7.612.53 / 3.30145.2 (1.07×)45.09

(1) Adaptive inference under limited memory

DynSVD is useful when memory is limited and the available compute changes over time, as on personal devices. TEAL on the original model preserves quality well but needs 13.5 GiB at every sparsity level, so it does not fit under 10 or 8 GB. There, the model has to be compressed first, and adaptive inference then runs on the compressed model, where DynSVD-T and TEAL use the same stored SVD-LLM model.

(2) Lower perplexity and faster decoding than TEAL on the same compressed model

At similar active parameters, DynSVD-T is faster and has lower perplexity than SVD-LLM + TEAL in all four cases (table above). For example, under 10 GB it reaches 11.90 perplexity at 126.3 tok/s, against 15.19 at 121.8 tok/s for TEAL at 50%. On an SVD-compressed model, TEAL can skip only the first factor of each layer, so 25% sparsity removes only 11–12% of the parameters read per token and gives no speed-up (110.9 vs. 112.5 tok/s under 10 GB).

This advantage comes from a selection designed for the SVD-compressed model. The magnitude ∣z^i∣|\hat{z}_i| theoretically tells how much each basis contributes to the output of a token (Theorem 1), which does not hold for the input xx, since the columns of a weight matrix are neither orthogonal nor of equal norm. We also applied three selection rules to the same SVD-LLM model at the same active parameters on LLaMA-2-7B (table below). TEAL (to xx) sparsifies the input as in the original method, and TEAL (to z^\hat{z}) applies the same per-module threshold rule to z^\hat{z}. At 40%, the gap between TEAL (to xx) (20.60) and TEAL (to z^\hat{z}) (13.67) shows the effect of where the selection is made, and the gap between TEAL (to z^\hat{z}) and DynSVD-T (10.24) shows the effect of our threshold design.

Active comp. ratioMethodWiki2↓tok/s
40%DynSVD-T10.24115.3
40%SVD-LLM + TEAL (to xx)20.60110.1
40%SVD-LLM + TEAL (to z^\hat{z})13.67111.1
60%DynSVD-T37.85147.3
60%SVD-LLM + TEAL (to xx)163.0143.4
60%SVD-LLM + TEAL (to z^\hat{z})56.77143.2

We have added this comparison with Tables 5 and 6 to Section 4.5, and introduced TEAL as a baseline in Section 4.1 (Experimental Setup).

[W2] Decomposition designed for our method

While exploring a decomposition tailored to adaptive inference is worthwhile, compatibility with existing static SVD methods is in itself one of the key strengths of DynSVD. Theorem 2 shows that the truncation error of the static SVD and the selection error of DynSVD add up without a cross term, so our selection can be applied on top of static SVD methods whose left factor has orthonormal columns. This compatibility allows token-adaptive inference to benefit from advances in static SVD methods. For example, Table 1 shows that DynSVD-T (from SVD-LLM) and DynSVD†-T (from Dobi-SVD) reduce WikiText-2 perplexity from 59.96 to 28.27 and from 46.18 to 26.82 at 60% active compression. These results illustrate how DynSVD's dynamic selection can be directly combined with static SVD methods, independently of their offline transformations or calibrations.

We also see room for decompositions tailored to adaptive selection (e.g., shared component selection across a batch), so we have added this direction as future work in Appendix A.

Dear Reviewer D6mX,

Thank you again for your valuable comments. To summarize our responses above:

  • [W1], [Q1]: Under fixed memory budgets, DynSVD-T is faster and has lower perplexity than TEAL on the same SVD-compressed model (added to Section 4.5).
  • [W2]: We added a decomposition designed for dynamic selection as future work (Appendix A), and also discussed compatibility with existing static SVD methods as one of DynSVD's key strengths.

We hope our responses have addressed your concerns. If any points remain unclear, we would be glad to address them before the discussion period ends.

I thank the authors for their responses. I appreciate the additional comparison with adaptive inference methods as well as the additional clarification that your method is more appropriate in settings that require model compression. Given that your method appears to be superior to applying generic adaptive inference methods to models compressed with SVD compression, I believe that this work provides a nice contribution and am raising my score accordingly.

Reviewer_rD4iOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DynSVD, a training-free approach that combines static SVD compression with token-dependent selection of retained singular components. Selection uses the magnitude of intermediate activations, either through exact top-rdr_d selection or a calibrated threshold. The paper establishes optimality for reconstructing an individual layer's output under an orthonormal basis and shows that static truncation and dynamic selection errors have no cross term. Experiments evaluate perplexity, downstream accuracy, and decoding throughput, including custom Triton kernels. The principal contribution is an improved quality–computation trade-off for SVD-compressed models, particularly during batch-one decoding.

Critical strengths
  1. The method is simple and well motivated. Selecting components according to their actual contribution to a token addresses a limitation of fixed spectral truncation. Theorems 1 and 2 provide a clear justification within their stated layerwise reconstruction setting. I also appreciate that the paper explicitly accounts for the cost of computing the intermediate activations: the active-parameter count is nrs+mrdn r_s + m r_d, rather than counting only the selected output components.

  2. The evaluation goes beyond reconstruction error and theoretical operation counts. Table 1 shows improvements in perplexity and downstream accuracy, while Figure 4 and Table 4 examine measured throughput under memory constraints. The comparison between exact, per-layer threshold, and shared-threshold selection helps explain the method's behaviour. The additional architecture results and kernel measurements strengthen the empirical contribution.

Critical weaknesses
  1. The runtime-flexibility claim is framed too strongly. Section 4.4 states that a static SVD model must be replaced by a smaller model to run faster. However, for a fixed whitened decomposition without rank-dependent parameter updates, smaller-rank approximations can be obtained by evaluating prefixes of the stored factors. Given stored factors L^\widehat{L} and R^\widehat{R} of rank rsr_s, this amounts to evaluating L^:,1:k(R^1:k,:x)\widehat{L}_{:,1:k}(\widehat{R}_{1:k,:}x) for k≤rsk \leq r_s. A static baseline can therefore change its operating point without recomputing the decomposition or loading another model. If the smaller SVD-LLM models in Figure 4 were constructed this way, their quality results already represent this variable-rank baseline, although the throughput of evaluating prefixes within the larger stored model remains to be confirmed. This does not invalidate the reported Pareto advantage. It does mean that runtime flexibility alone is not distinctive: the contribution is the quality–computation trade-off enabled by token-dependent selection and, with the shared threshold, budget reallocation across layers. The claims should reflect this distinction.

  2. The strongest practical evidence remains relatively narrow. At matched active parameters, DynSVD stores more weights: for example, Table 1 reports 5.44B stored parameters versus 3.88B for static SVD at the 40% setting. The paper acknowledges this and provides memory-constrained experiments, which is helpful. However, the headline matched-quality throughput improvement is up to 1.10×1.10\times and is demonstrated on LLaMA-7B with an A100. The newer-model results primarily establish perplexity improvements, sometimes at substantial absolute degradation. This limits how confidently the practical gains can be generalised beyond the measured setting.

Questions for authors
  1. Can you clarify whether the smaller SVD-LLM models in Figure 4 correspond to prefixes of the larger models' factors, using the same whitening statistics and no rank-dependent updates? If so, could you report representative throughput measurements when evaluating those prefixes from one resident model, within the same memory limit and using the same static-SVD kernels? This would establish whether Figure 4 also represents a static baseline with runtime rank adjustment.

  2. Can you provide repeated timing measurements, including variability, for the principal matched-quality operating points in Figure 4?

  3. Does the quality–throughput advantage hold at a modest compression level on one of the newer architectures already evaluated in Appendix B?

3 comments

Thank you for your careful review and positive evaluation of our work. Below, we answer each point and note where we have revised the manuscript.

[W1], [Q1] Runtime flexibility and the prefix baseline

We agree that our phrasing such as "must be replaced" in Section 4.4 was too strong. Our contribution is not runtime flexibility itself, but that DynSVD reaches better operating points than static SVD by changing only a single threshold at run time.

In Figure 4, each SVD-LLM model is stored separately at its own compression ratio, not run as a prefix of a larger model. However, its weights are identical to the prefix of the larger model, because SVD-LLM whitens once at full rank, sorts the bases by singular value, and keeps the top ones for each compression ratio.

We also measured the prefix directly, keeping the larger model in memory and running only its prefix with the same static-SVD kernels (table below, 10 runs). The prefix has the same speed and perplexity as the separate model but needs the memory of the whole stored model. So, the static curve in Figure 4 holds for both cases. At the same memory (9.67 GiB), DynSVD-T is 5.2% faster than the prefix at a lower perplexity (129.92 vs. 123.55 tok/s, 14.08 vs. 14.50).

ModelMemory (GiB)Act. / Total (B)tok/sWiki2↓
SVD-LLM9.674.43 / 4.43112.37 ± 0.01 (1.00×)10.46
DynSVD-T9.673.95 / 4.43116.30 ± 0.04 (1.04×)10.61
DynSVD-T9.673.47 / 4.43125.70 ± 0.07 (1.12×)11.90
DynSVD-T9.673.23 / 4.43129.92 ± 0.09 (1.16×)14.08
SVD-LLM9.314.21 / 4.21116.38 ± 0.03 (1.04×)11.85
SVD-LLM8.623.88 / 3.88123.64 ± 0.12 (1.10×)14.50
SVD-LLM8.163.56 / 3.56130.12 ± 0.08 (1.16×)18.76
SVD-LLM, prefix9.674.21 / 4.43116.42 ± 0.06 (1.04×)11.85
SVD-LLM, prefix9.673.88 / 4.43123.55 ± 0.07 (1.10×)14.50
SVD-LLM, prefix9.673.56 / 4.43130.21 ± 0.05 (1.16×)18.76

To make our claim more precise, we have revised the sentences below and added the prefix results to Table 14 in Appendix G.

  • Section 4.4. "A static SVD model has a single speed for a given memory, and it must be replaced by a smaller model to run faster. In contrast, DynSVD-T can adjust its threshold on the same stored weights and thus change its active parameters."
    → A static SVD model can run faster only by using a prefix of\color{blue}{\text{A static SVD model can run faster only by using a prefix of}} its stored bases, which is the same as replacing it with a\color{blue}{\text{its stored bases, which is the same as replacing it with a}} smaller model (Appendix G, Table 14). DynSVD-T instead\color{blue}{\text{smaller model (Appendix G, Table 14). DynSVD-T instead}} adjusts its threshold on the same stored weights and selects\color{blue}{\text{adjusts its threshold on the same stored weights and selects}} the bases for each token.\color{blue}{\text{the bases for each token.}}
  • Abstract. "one stored DynSVD model offers a flexible, continuous range of operating points through a single threshold"
    → one stored DynSVD model offers a continuous range of better\color{blue}{\text{one stored DynSVD model offers a continuous range of better}} operating points through a single threshold\color{blue}{\text{operating points through a single threshold}}
  • Conclusion. "A static SVD model has one operating point per stored model, whereas one DynSVD model moves between speed and quality at run time by changing a single threshold."
    → Unlike a static SVD model, which can only use a prefix of\color{blue}{\text{Unlike a static SVD model, which can only use a prefix of}} its stored bases, one DynSVD model selects bases for each\color{blue}{\text{its stored bases, one DynSVD model selects bases for each}} token through a single threshold and gives better operating\color{blue}{\text{token through a single threshold and gives better operating}} points at run time.\color{blue}{\text{points at run time.}}

[Q2] Repeated timing measurements

We re-measured the operating points of Figure 4 and Table 4 in 10 separate processes each on one A100 40GB (batch 1, prompt 128, 128 decoded tokens), as shown in the table below. The standard deviation is below 0.1% of the mean, much smaller than the differences between operating points.

Modelrd/rsr_d/r_sMemory (GiB)Act. / Total (B)tok/sWiki2↓
SVD-LLM–11.735.54 / 5.5494.73 ± 0.03 (1.00×)7.56
DynSVD-T0.911.735.24 / 5.5496.38 ± 0.02 (1.02×)7.56
0.811.734.94 / 5.5499.95 ± 0.03 (1.06×)7.61
0.711.734.64 / 5.54103.88 ± 0.04 (1.10×)7.73
0.611.734.34 / 5.54108.34 ± 0.05 (1.14×)8.04
0.511.734.04 / 5.54112.81 ± 0.08 (1.19×)8.81
SVD-LLM–9.674.43 / 4.43112.37 ± 0.01 (1.19×)10.46
SVD-LLM–9.674.43 / 4.43112.37 ± 0.01 (1.00×)10.46
DynSVD-T0.99.674.19 / 4.43112.28 ± 0.05 (1.00×)10.48
0.89.673.95 / 4.43116.30 ± 0.04 (1.04×)10.61
0.79.673.71 / 4.43120.64 ± 0.04 (1.07×)10.98
0.69.673.47 / 4.43125.70 ± 0.07 (1.12×)11.90
0.59.673.23 / 4.43129.92 ± 0.09 (1.16×)14.08
SVD-LLM–8.163.56 / 3.56130.12 ± 0.08 (1.16×)18.76

The table shows the 12 and 10 GB budgets, and we have added the full table as Table 14 in Appendix G.


[W2], [Q3] Storage and speed on a newer architecture at modest compression

We repeated the Figure 4 and Table 4 experiment on Qwen3-8B from Appendix B, and the advantage also holds at modest compression (14.7 and 12.4 GiB). DynSVD-T decodes 1.04–1.12× faster than the stored SVD-LLM model, while a smaller static SVD-LLM model is no faster and has higher perplexity (16.04 vs. 14.72, 26.82 vs. 21.17). Most of the perplexity increase over the original model comes from the stored SVD-LLM model (9.71 to 13.91), and DynSVD-T adds little (13.99–14.72).

ModelMemory (GiB)Act. / Total (B)tok/sWiki2↓
Original16.456.95 / 6.9582.65 ± 0.049.71
SVD-LLM14.705.94 / 5.9482.93 ± 0.01 (1.00×)13.91
DynSVD-T14.705.29 / 5.9486.56 ± 0.01 (1.04×)13.99
DynSVD-T14.704.97 / 5.9489.91 ± 0.01 (1.08×)14.24
DynSVD-T14.704.64 / 5.9493.09 ± 0.01 (1.12×)14.72
SVD-LLM13.085.21 / 5.2191.12 ± 0.04 (1.10×)16.04
SVD-LLM12.424.76 / 4.7696.46 ± 0.05 (1.00×)18.65
DynSVD-T12.424.24 / 4.76100.99 ± 0.03 (1.05×)18.87
DynSVD-T12.423.98 / 4.76104.13 ± 0.02 (1.08×)19.51
DynSVD-T12.423.71 / 4.76107.29 ± 0.01 (1.11×)21.17
SVD-LLM11.324.17 / 4.17105.58 ± 0.02 (1.09×)26.82

We have added the Pareto plot and the table above as Figure 6 and Table 11 in Appendix B, and Section 4.4 refers to them.

I appreciate and thank the authors' efforts dealing with all the proposed and suggested items in their rebuttal and I am happy to keep me positive score.

Reviewer_bnjxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DynSVD a training-free framework that dynamically selects token-adaptive bases on a static SVD compressed model weights. They found out each token may activate a different set of singular directions and the standard static SVD-based compression method may ignore per token specific basis and based on this proposed a threshold based selection algorithm for activation components selection.

Critical strengths
  1. The Algorithm itself is simple and theoretically can apply to any existing static SVD-based compression method on the fly within half an hour

  2. The experiments are relatively comprehensive. The authors evaluate DynSVD across multiple model families and sizes, including models with MHA, GQA, and MoE architectures.

Critical weaknesses
  1. The core idea is essentially a mean based masking function. While the method is reasonable, I am not convinced that it provides sufficient algorithmic novelty beyond existing activation sparsity methods.

  2. The main result does not give sufficient quality memory trade-off. In Table 1, the authors compare DynSVD with other SVD compression methods under the same number of active parameters. DynSVD indeed gets a better performance. However, it stores significantly more parameters than the static baselines. For example, at 40% compression, DynSVD-T stores 5.44B parameters compared to 3.88B for SVD-LLM. So the method is using more memory for better accuracy.

  3. Since the proposed method essentially performs dynamic activation sparsity on SVD factors, it is unclear why this approach should be preferred over more general pruning or activation sparsity methods. The paper mainly compares against static SVD compression baselines, while the comparison with existing dynamic sparsity methods is limited.

Questions for authors
  1. Could the authors consider evaluating DynSVD on LowRankArena (https://arxiv.org/abs/2608.26389)? It provides a standardized evaluation framework for SVD-based LLM compression methods. I think it would be helpful to compare DynSVD under the same evaluation protocol, particularly in terms of downstream accuracy, memory usage, and actual serving efficiency.

  2. The descriptions on Table4 includes some results that does not appear on the actual table "under 10 GB DynSVD-T decodes 130.8 tokens per second at 14.08 perplexity"

  3. Since the algorithm is more like applying activation-based masking to the intermediate representations of an SVD-compressed model. I am curious about the fundamental difference between DynSVD and existing pruning or activation sparsity methods. What is the advantage of applying dynamic selection on top of SVD-compressed weights rather than directly performing pruning or activation sparsity on the original model?

6 comments

Thank you for your careful review and constructive feedback. Below, we answer each point and note where we have revised the manuscript.

[Q1] Evaluation on LowRankArena

We evaluated DynSVD-T under the LowRankArena (LRA) protocol on LLaMA-7B. Baseline quality is taken from LRA and memory and speed are measured on our A100 40GB with LRA's code and released checkpoints. DynSVD-T is measured in the same way and matched in active parameters, as in Table 1 of our paper.

Active comp. ratioMethodWiki2↓C4↓MCQ↑Math↑Mem↓tok/s↑ (Single, no pad)tok/s↑ (Single, padded)tok/s↑ (Batched)
–Original5.677.200.6490.27012.8884.984.9460.2
20%ASVD8.6011.040.6090.22610.46–––
SVD-LLM v17.8816.420.5010.25810.4837.086.7459.4
DoBi-SVD9.2319.010.5260.25110.7443.285.4458.6
Basis Sharing7.7415.510.5740.22210.3539.378.3454.8
MoDeGPT6.9210.870.5520.27910.57–––
DynSVD-T6.8611.640.5980.27012.51–89.0454.6
40%ASVD3839.834268.560.3570.2358.05–––
SVD-LLM v113.7455.030.3990.2598.1947.3104.5467.7
DoBi-SVD15.2448.370.4260.2198.3154.2102.3466.1
Basis Sharing12.4241.130.4650.2387.8555.293.7463.6
MoDeGPT12.4323.400.5090.2578.08–––
DynSVD-T9.0620.720.5270.26010.48–106.1459.9

Single (no pad): LRA's released checkpoints as they are. Single (padded): ranks zero-padded to a multiple of 8. ASVD and MoDeGPT don't load in LRA's serving path.

  • Quality: At matched active parameters, DynSVD-T is better than its backbone SVD-LLM on every benchmark at both ratios (e.g., WikiText-2 perplexity is 13% and 34% lower). Among all methods, DynSVD-T is the best on all quality metrics at 40%, while ASVD and MoDeGPT are better on some metrics at 20%.
  • Speed: DynSVD-T is the fastest for single-batch decoding, using our custom Triton kernels. The ranks of LRA's released checkpoints are not multiples of 8, which is inefficient for GPU kernels, so they decode a single request at only 37–55 tok/s. For a fair comparison, we zero-pad the baseline ranks to a multiple of 8, as for DynSVD-T, which gives 78–105 tok/s without changing the outputs. In batched serving, the per-token selection does not directly reduce computation (Section 4.6), so we make no speed claim there and leave it to future work.
  • Memory: At the same budget (10.48 GiB for SVD-LLM at 20% and DynSVD-T at 40%), DynSVD-T reads fewer parameters per token and decodes 22% faster. Its perplexity is higher, but its accuracy on the downstream benchmarks (MCQ and Math) is comparable or slightly higher.

Although DynSVD-T needs more memory than the baselines at matched active parameters, our target is not memory reduction alone, but a more practical setting where memory, quality, and speed are considered together. This is why Figure 4 and Table 4 evaluate quality and speed under a fixed memory budget, as discussed in our next response (W2).

LowRankArena is valuable for a fair comparison with other SVD methods, so we have added the table above and its discussion to Appendix I.

[W2] Storage at matched active parameters

Table 1 matches methods in active parameters (same computation), so DynSVD stores more parameters, because it selects bases from a larger stored pool. Looking only at quality and memory, DynSVD can be seen as using more memory for better quality. However, as noted in our motivation, personal devices have limited memory, and the compute available to the model changes over time. What matters there is how fast and accurate the model is within the given memory, which Section 4.4 (Figure 4 and Table 4) evaluates.

With memory fixed at 12, 10, and 8 GB, DynSVD-T achieves lower perplexity at the same speed. For example, under 10 GB (table below), SVD-LLM (3.56B / 3.56B) scores 18.76 perplexity at 130.2 tok/s, while DynSVD-T (3.23B / 4.43B) reaches 14.08 at a similar speed (130.8 tok/s). SVD-LLM (4.43B / 4.43B) achieves 10.46 but at only 112.5 tok/s.

ModelMemory (GiB)Act. / Total (B)tok/sWiki2↓
SVD-LLM9.674.43 / 4.43112.5 (1.00×)10.46
DynSVD-T9.673.95 / 4.43116.4 (1.03×)10.61
DynSVD-T9.673.71 / 4.43120.8 (1.07×)10.98
DynSVD-T9.673.47 / 4.43126.3 (1.12×)11.90
DynSVD-T9.673.23 / 4.43130.8 (1.16×)14.08
SVD-LLM, prefix9.673.56 / 4.43130.2 (1.16×)18.76
SVD-LLM8.163.56 / 3.56130.2 (1.16×)18.76

Static SVD models can also gain speed within the same stored parameters by using only a prefix of the factors. So, SVD-LLM (3.56B / 4.43B) is faster than SVD-LLM (4.43B / 4.43B), but its perplexity is the same as SVD-LLM (3.56B / 3.56B).

Notably, this implies that while static SVD can trade quality for speed within a fixed memory budget, it moves along a lower-quality curve (shown in Figure 4). This setup is closer to local LLM deployment, where memory, speed, and quality matter together, and there DynSVD-T gives a better trade-off between speed and quality than static SVD within the same memory.


[Q2] Numbers in the Table 4 description

Thank you for pointing this out. We have added the two missing rows to Table 4 in Section 4.4, as shown below.

ModelMemory (GiB)Act. / Total (B)tok/sWiki2↓
DynSVD-T9.673.23 / 4.43130.8 (1.16×)14.08
SVD-LLM8.163.56 / 3.56130.2 (1.16×)18.76

[W1], [W3], [Q3] Novelty/Difference/Advantage compared to activation sparsity or pruning (1/3)

In short, our answer on activation sparsity and pruning is as follows, and the details are given in the next two comments.

  • For activation sparsity, methods such as TEAL preserve quality well on the original model, but they keep all weights in memory, so the model still has to be compressed when memory is limited. DynSVD is designed for SVD-compressed models: it selects on the SVD bases, where the magnitude of each activation tells how much its basis contributes to the output (Theorem 1), with a threshold designed for them. On the same stored model, it achieves lower perplexity and faster decoding than TEAL at similar active parameters.
  • For pruning, methods such as LLM-Pruner and SliceGPT compress the stored weights as SVD does, while DynSVD is built on top of a static SVD model, so static SVD is its direct baseline. A pruned model, with or without activation sparsity, is a different stored model, so the comparison shows the difference between pruning and SVD compression rather than the effect of DynSVD's selection. We also show the results of this comparison with LLM-Pruner and SliceGPT, with and without TEAL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.