- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SVD-based compression methods reduce the memory footprint and inference latency of deep learning models by replacing large weight matrices with a multiplication of two low rank matrices. On the other hand, adaptive inference methods address memory and latency by activating only specific portions of the model based on the input. This work introduces a new adaptive inference method which builds on SVD-based compression. Once a model has been compressed using an existing SVD-based compression algorithm, their method selects only a subset of the low rank factors to activate at each layer. As a result, a larger number of parameters may be retained in each layer while maintaining similar memory and inference latency to a model compressed with a larger compression rate. The authors show that their method is able to outperform existing SVD-based compression methods while using the same amount of memory at inference.
Critical strengths
- An adaptive inference method based on a low rank decomposition of the model's weight matrices is genuinely interesting. If it can be shown to have advantages over other adaptive inference methods I believe it is a nice contribution.
Critical weaknesses
- The paper’s experiments exclusively compare with SVD compression methods. However, I believe it is necessary to provide a comparison with other adaptive inference methods. Ideally, the experiments would clearly show what advantage an SVD-based adaptive inference method presents when compared with existing methods.
- The method presented builds directly upon existing SVD compression methods. In doing so, the authors avoid identifying novel decomposition strategies that account for their adaptive inference method.
Questions for authors
Can the authors clarify what advantages the SVD-based adaptive inference method has over existing adaptive inference methods such as TEAL?
4 comments
[W2] Decomposition designed for our method
While exploring a decomposition tailored to adaptive inference is worthwhile, compatibility with existing static SVD methods is in itself one of the key strengths of DynSVD. Theorem 2 shows that the truncation error of the static SVD and the selection error of DynSVD add up without a cross term, so our selection can be applied on top of static SVD methods whose left factor has orthonormal columns. This compatibility allows token-adaptive inference to benefit from advances in static SVD methods. For example, Table 1 shows that DynSVD-T (from SVD-LLM) and DynSVD†-T (from Dobi-SVD) reduce WikiText-2 perplexity from 59.96 to 28.27 and from 46.18 to 26.82 at 60% active compression. These results illustrate how DynSVD's dynamic selection can be directly combined with static SVD methods, independently of their offline transformations or calibrations.
We also see room for decompositions tailored to adaptive selection (e.g., shared component selection across a batch), so we have added this direction as future work in Appendix A.
Dear Reviewer D6mX,
Thank you again for your valuable comments. To summarize our responses above:
- [W1], [Q1]: Under fixed memory budgets, DynSVD-T is faster and has lower perplexity than TEAL on the same SVD-compressed model (added to Section 4.5).
- [W2]: We added a decomposition designed for dynamic selection as future work (Appendix A), and also discussed compatibility with existing static SVD methods as one of DynSVD's key strengths.
We hope our responses have addressed your concerns. If any points remain unclear, we would be glad to address them before the discussion period ends.
I thank the authors for their responses. I appreciate the additional comparison with adaptive inference methods as well as the additional clarification that your method is more appropriate in settings that require model compression. Given that your method appears to be superior to applying generic adaptive inference methods to models compressed with SVD compression, I believe that this work provides a nice contribution and am raising my score accordingly.
Thank you for your careful review and constructive feedback. Below, we answer each point and note where we have revised the manuscript.
[W1], [Q1] Comparison with adaptive inference methods
Activation sparsity methods such as TEAL (to input x) and DynSVD (to intermediate z^) both reduce the computation for each token. TEAL on the original model keeps all weights in memory, so it is a good choice when the original model fits. The advantage of DynSVD appears when the model has to be compressed to fit in memory: (1) TEAL on the original model does not fit, while DynSVD runs on the compressed model, and (2) on the same SVD-compressed model, DynSVD reaches lower perplexity and faster decoding than TEAL at similar active parameters.
The table below compares DynSVD with TEAL on LLaMA-7B, on the original model and on the same SVD-LLM model under 10 and 8 GB budgets. We calibrate TEAL's per-module thresholds on each model and use its official sparse kernel.
(1) Adaptive inference under limited memory
DynSVD is useful when memory is limited and the available compute changes over time, as on personal devices. TEAL on the original model preserves quality well but needs 13.5 GiB at every sparsity level, so it does not fit under 10 or 8 GB. There, the model has to be compressed first, and adaptive inference then runs on the compressed model, where DynSVD-T and TEAL use the same stored SVD-LLM model.
(2) Lower perplexity and faster decoding than TEAL on the same compressed model
At similar active parameters, DynSVD-T is faster and has lower perplexity than SVD-LLM + TEAL in all four cases (table above). For example, under 10 GB it reaches 11.90 perplexity at 126.3 tok/s, against 15.19 at 121.8 tok/s for TEAL at 50%. On an SVD-compressed model, TEAL can skip only the first factor of each layer, so 25% sparsity removes only 11–12% of the parameters read per token and gives no speed-up (110.9 vs. 112.5 tok/s under 10 GB).
This advantage comes from a selection designed for the SVD-compressed model. The magnitude ∣z^i∣ theoretically tells how much each basis contributes to the output of a token (Theorem 1), which does not hold for the input x, since the columns of a weight matrix are neither orthogonal nor of equal norm. We also applied three selection rules to the same SVD-LLM model at the same active parameters on LLaMA-2-7B (table below). TEAL (to x) sparsifies the input as in the original method, and TEAL (to z^) applies the same per-module threshold rule to z^. At 40%, the gap between TEAL (to x) (20.60) and TEAL (to z^) (13.67) shows the effect of where the selection is made, and the gap between TEAL (to z^) and DynSVD-T (10.24) shows the effect of our threshold design.
We have added this comparison with Tables 5 and 6 to Section 4.5, and introduced TEAL as a baseline in Section 4.1 (Experimental Setup).