- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method to replace the weight matrices in neural network architectures with low-rank versions of them. Instead of optimizing for the best approximation to the original matrices, it learns the best lower-rank approximation in activation space, using a training data. A VAE is proposed to learn a latent distribution of sketching matrices that transform an original matrix into its lower-rank version. An additional term is included in the -VAE objective where the decoder is also optimized to generate sketching matrices that perform well on a particular task and model. Thus, both a (pretrained) task model and the VAE are trained end-to-end.
Results on transformer-based language models outperform baselines at different compression levels, and sometimes even outperforms uncompressed baselines. The VAE adjusts well to depth upscaling and domain adaptation.
Critical strengths
- The work is generally presented clearly.
- The solution is principled and elegant.
- The task performance gain is significant and well supported.
- The authors offer a performance analysis of the hyper-parameters assigned to the loss terms.
Critical weaknesses
-
A few things are not clear:
- How do the compression ratios reported map to the ranks chosen? Are the ranks reduced uniformly across all weights or is it more dynamic?
- Details on how the datasets of sketching matrices are constructed in the experiments are missing. I assume that the pretrained models and their original datasets are used. Table 4 seems to confirm so but I think it should be addressed explicitly in the main document.
- Wouldn’t each data sample map 1:1 to a sketching matrix ? Why do we make a difference between the number of data samples and the number of sketching matrices ?
- It is not clear if we must sample a sketching matrix for each inference step or not. Do the theoretical guarantees only apply in expectation? In that case, there must be a runtime overhead of sampling a sketching matrix and then performing truncated SVD on for each inference step. This overhead does not seem to apply to the baselines considered since (when applicable) it only needs to be done once for all inference steps. Does this overhead diminish the computational advantage of running the model with lower-rank matrix decompositions? In that case, is the efficiency advantage just model compression? If so, it would seem relevant to compare against other compression approaches (e.g. quantization, pruning, etc).
-
The authors suggest that “sketching [is] a powerful tool with input data-independent theoretical guarantees” however since we are optimizing in the activation space, it does seem like any theoretical guarantees would have to be data dependent. In any case, some rank-distortion theoretical analysis would have strengthened the claims and increase understanding.
-
It would have been nice to extend the evaluation to other models and tasks but I wouldn’t deem it necessary to get this work published.
Questions for authors
I have raised my questions as weaknesses. If the authors can help me improve my understanding I can reevaluate my scores.
5 comments
W5. Input data-independent theoretical guarantees: The statement that sketching offers input data-independent theoretical guarantees refers to established results for classical data-oblivious sketching methods like [Sarlos, 2006] and [Woodruff, 2014]. As correctly pointed out by the reviewer, we do not establish such a result in our paper. We agree that the current wording could suggest that the proposed sketching method automatically inherits those guarantees though and thus will revise the text to make this distinction explicit.
We thank the reviewer for requesting a rank-distortion analysis. We provide two complementary results: an exact distortion-versus-rank curve for the activation-space approximation used to construct our training targets, and a deterministic stability bound for the deployed compression step. We have added these results as section A.2 in the appendix of the revised draft.
Setup. Fix a layer , a target rank , and an activation matrix . Write , let denote the best approximation of rank at most obtained by truncated SVD, and let denote the spectral norm. Choose an orthogonal projector of rank at most satisfying . Sketches are square, with dimension equal to the number of rows of , as in our compression step . For Proposition 1, Equation (3) is applied to this same activation matrix . When the truncation is non-unique, denotes the truncation induced by the SVD used in Equation (3); Proposition 2 holds for any truncated SVD choice of . The statements concern exact arithmetic.
Proposition 1: Optimal activation-space rank-distortion curve. The target weights constructed by Equation (3) satisfy and therefore attain Indeed, every feasible has rank at most , so Eckart-Young supplied the lower bound.
With the compact SVD , Equation (3) reads using and . This construction has rank at most and satisfies , since the row space of is contained in that of . The optimal layerwise distortion is thus characterised by the activation spectrum.
Proposition 2: Distortion of the deployed compression. For every square sketch , including every sample from the trained VAE, the deployed weights satisfy To prove this, write . The triangle inequality yields Because has rank at most , optimality of truncated SVD implies Consequently, the sharper intermediate bound holds. Submultiplicativity yields the stated proposition.
When , its additive term can equivalently be written as
The factor arises from bounding weight-space truncation error in the activation-space metric.
Interpretation and scope. Proposition 2 holds for every realised sketch, regardless of how its distribution was learned. It bounds excess distortion above the optimal rank- benchmark by the sketch’s deviation from an optimal activation-space projector. At , already has rank at most , so the final truncation leaves it unchanged, and the optimal distortion is attained exactly.
This is a deterministic stability result, not a claim that VAE prior samples are close to during compression with a specified probability. It is also a layerwise statement for prescribed activations, rather than an end-to-end perplexity guarantee. It applies to fixed held-out activations when the corresponding oracle and projector are recomputed for analysis, while keeping the deployed weights fixed. Population-level conclusions however require additional assumptions and analysis.
Our bound depends explicitly on the activations through , and the norm factors. Such dependence does not itself preclude distribution-free validity: classical low-rank approximation bounds also involve the input’s singular-value tail. Classical oblivious constructions sample independently of the input matrix, with sketch size determined by parameters such as dimensions, target rank, accuracy, and confidence. Their approximation guarantees are proved for specified sketch distributions. Our learned distribution is data-aware, and we do not claim that is satisfies these hypotheses. Proposition 2 instead supplies a deterministic, per-sketch bound valid for arbitrary fixed weights and activations, without assuming a data-generating distribution.
We thank the reviewer for their feedback, insights and suggestions. In the following, we will address each of the critical weaknesses. We additionally uploaded an updated version of the paper and highlighted the changes both in the main part of the paper and the appendix in blue.
W1. Definition of compression ratio and compression schedule: We define the compression ratio cr as the rank ratio that a weight matrix is compressed. It can be used to compute the rank of the compressed matrix via rc=ruc∗(1−cr) for the compressed rank rc and uncompressed rank ruc. Assume e.g., the original rank of a matrix was 10, for a compression ratio of 0.5 the rank of the compressed matrix thus is 5, whereas the rank of the compressed matrix is 6 for a compression ratio of 0.4. We thus work directly on rank compression.
In this paper, we fix the same rank for each of the layers and thus compress uniformly. Advancing the proposed method with a layer-wise dynamic compression schedule can be a promising direction of future work.
W2. Construction of training dataset of sketching matrices: As pointed out by the reviewer, details regarding the construction of the training dataset of the sketching matrices can be found in the appendix of the paper (Table 4, p. 18). We have added a short paragraph to Section 4.1 in the main part of the paper to clarify this further.
W3. Number of training sketching matrices: We distinguish between the number of data samples M and the number of training sketching matrices N as we compute a training sketch for every data sample and layer, i.e., N=M⋅L for L being the number of layers.
W4. Clarification of inference: The model compression step has to be applied only once — not at each inference step. Hence, there is no runtime overhead due to sampling of S and the following truncation during inference. This is comparable to the other baselines tested.
[Sarlos, 2006] Tamas Sarlos. Improved approximation algorithms for large matrices via random projections. In 2006 47th annual IEEE symposium on foundations of computer science (FOCS’06), pp. 143–152. IEEE, 2006.
[Woodruff, 2014] David P Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends in Theoretical Computer Science, 10(1-2):1–157, 2014.