acceptodds
Under review as a conference paper at ICLR 2027

Learning to Sketch for Low-rank Model Compression

Abstract

Transformer blocks are the cornerstones of modern deep neural networks. While energy consumption used to be dominated by training of the models, widespread adoption and intensive usage shifted it towards model inference - thus highlighting the importance of post-training model compression for transformers. Sketching describes the field of probabilistic compression that use random projections, sampled from a probability distribution. While being widely used for data compression, sketching for model compression remains widely unexplored. One key component of the sketching methods is the choice of probability distribution over sketching matrices. We propose data-aware sketching by learning the sketching distribution through training a variational autoencoder (VAE). To preserve the utility of the low-ranked model on the downstream task, we train the VAE with a carefully tailored loss function. We show that our method outperforms data-oblivious sketching and low-rank compression baselines. *For compression ratios of up to %, our compressed model outperforms even the uncompressed baseline*. Additionally, we show that for both depth upscaling and domain adaptation, a finetuning step of the VAE decoder is sufficient, highlighting the applicability of the data-aware sketching.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
sketching, random projections, model compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VhNfOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method to replace the weight matrices in neural network architectures with low-rank versions of them. Instead of optimizing for the best approximation to the original matrices, it learns the best lower-rank approximation in activation space, using a training data. A VAE is proposed to learn a latent distribution of sketching matrices that transform an original matrix into its lower-rank version. An additional term is included in the β\beta -VAE objective where the decoder is also optimized to generate sketching matrices that perform well on a particular task and model. Thus, both a (pretrained) task model and the VAE are trained end-to-end.

Results on transformer-based language models outperform baselines at different compression levels, and sometimes even outperforms uncompressed baselines. The VAE adjusts well to depth upscaling and domain adaptation.

Critical strengths
  • The work is generally presented clearly.
  • The solution is principled and elegant.
  • The task performance gain is significant and well supported.
  • The authors offer a performance analysis of the hyper-parameters assigned to the loss terms.
Critical weaknesses
  • A few things are not clear:

    • How do the compression ratios reported map to the ranks chosen? Are the ranks reduced uniformly across all weights or is it more dynamic?
    • Details on how the datasets of sketching matrices are constructed in the experiments are missing. I assume that the pretrained models and their original datasets are used. Table 4 seems to confirm so but I think it should be addressed explicitly in the main document.
    • Wouldn’t each data sample XjX_j map 1:1 to a sketching matrix SiS_i ? Why do we make a difference between the number of data samples MM and the number of sketching matrices NN ?
    • It is not clear if we must sample a sketching matrix for each inference step or not. Do the theoretical guarantees only apply in expectation? In that case, there must be a runtime overhead of sampling a sketching matrix and then performing truncated SVD on W^l\hat{W}_l for each inference step. This overhead does not seem to apply to the baselines considered since (when applicable) it only needs to be done once for all inference steps. Does this overhead diminish the computational advantage of running the model with lower-rank matrix decompositions? In that case, is the efficiency advantage just model compression? If so, it would seem relevant to compare against other compression approaches (e.g. quantization, pruning, etc).
  • The authors suggest that “sketching [is] a powerful tool with input data-independent theoretical guarantees” however since we are optimizing in the activation space, it does seem like any theoretical guarantees would have to be data dependent. In any case, some rank-distortion theoretical analysis would have strengthened the claims and increase understanding.

  • It would have been nice to extend the evaluation to other models and tasks but I wouldn’t deem it necessary to get this work published.

Questions for authors

I have raised my questions as weaknesses. If the authors can help me improve my understanding I can reevaluate my scores.

5 comments

We thank the reviewer for their feedback, insights and suggestions. In the following, we will address each of the critical weaknesses. We additionally uploaded an updated version of the paper and highlighted the changes both in the main part of the paper and the appendix in blue.

W1. Definition of compression ratio and compression schedule: We define the compression ratio cr\text{cr} as the rank ratio that a weight matrix is compressed. It can be used to compute the rank of the compressed matrix via rc=ruc∗(1−cr)r_c = r_{\text{uc}} * (1-\text{cr}) for the compressed rank rcr_c and uncompressed rank rucr_{\text{uc}}. Assume e.g., the original rank of a matrix was 1010, for a compression ratio of 0.50.5 the rank of the compressed matrix thus is 55, whereas the rank of the compressed matrix is 66 for a compression ratio of 0.40.4. We thus work directly on rank compression.

In this paper, we fix the same rank for each of the layers and thus compress uniformly. Advancing the proposed method with a layer-wise dynamic compression schedule can be a promising direction of future work.

W2. Construction of training dataset of sketching matrices: As pointed out by the reviewer, details regarding the construction of the training dataset of the sketching matrices can be found in the appendix of the paper (Table 4, p. 18). We have added a short paragraph to Section 4.1 in the main part of the paper to clarify this further.

W3. Number of training sketching matrices: We distinguish between the number of data samples MM and the number of training sketching matrices NN as we compute a training sketch for every data sample and layer, i.e., N=M⋅LN = M \cdot L for LL being the number of layers.

W4. Clarification of inference: The model compression step has to be applied only once — not at each inference step. Hence, there is no runtime overhead due to sampling of SS and the following truncation during inference. This is comparable to the other baselines tested.

[Sarlos, 2006] Tamas Sarlos. Improved approximation algorithms for large matrices via random projections. In 2006 47th annual IEEE symposium on foundations of computer science (FOCS’06), pp. 143–152. IEEE, 2006.

[Woodruff, 2014] David P Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends in Theoretical Computer Science, 10(1-2):1–157, 2014.

W5. Input data-independent theoretical guarantees: The statement that sketching offers input data-independent theoretical guarantees refers to established results for classical data-oblivious sketching methods like [Sarlos, 2006] and [Woodruff, 2014]. As correctly pointed out by the reviewer, we do not establish such a result in our paper. We agree that the current wording could suggest that the proposed sketching method automatically inherits those guarantees though and thus will revise the text to make this distinction explicit.

We thank the reviewer for requesting a rank-distortion analysis. We provide two complementary results: an exact distortion-versus-rank curve for the activation-space approximation used to construct our training targets, and a deterministic stability bound for the deployed compression step. We have added these results as section A.2 in the appendix of the revised draft.

Setup. Fix a layer ll, a target rank rr, and an activation matrix XX. Write A=WlXA= W_l X, let [⋅]r[\cdot]_r denote the best approximation of rank at most rr obtained by truncated SVD, and let ∥⋅∥2\| \cdot \|_2 denote the spectral norm. Choose an orthogonal projector PrP_r of rank at most rr satisfying PrA=[A]rP_r A = [A]_r. Sketches SS are square, with dimension equal to the number of rows of WlW_l, as in our compression step Wl∗=[SWl]rW^*_l = [SW_l]_r. For Proposition 1, Equation (3) is applied to this same activation matrix XX. When the truncation is non-unique, [A]r[A]_r denotes the truncation induced by the SVD used in Equation (3); Proposition 2 holds for any truncated SVD choice of [SWl]r[SW_l]_r. The statements concern exact arithmetic.

Proposition 1: Optimal activation-space rank-distortion curve. The target weights W^l\hat{W}_l constructed by Equation (3) satisfy W^lX=[A]r\hat{W}_l X = [A]_r and therefore attain Dl∗(r):=min⁡rank(W~)≤r∥WlX−W~X∥F=(∑i>rσi(A))12.D^{\ast}_l (r) := \min_{\text{rank}(\tilde{W}) \leq r} \Vert W_l X - \tilde{W}X \Vert_F = (\sum_{i > r} \sigma_i (A))^{\frac{1}{2}}. Indeed, every feasible W~X\tilde{W}X has rank at most rr, so Eckart-Young supplied the lower bound.

With the compact SVD X=UΣVTX = U \Sigma V^T, Equation (3) reads W^l=[WlUΣ]rΣ−1UT=[A]rX†,\hat{W}_l = [W_l U \Sigma]_r \Sigma^{-1}U^T = [A]_r X^{\dagger}, using [WlUΣ]rVT=[A]r[W_l U \Sigma]_r V^T = [A]_r and X†=VΣ−1UTX^{\dagger}=V \Sigma^{-1}U^T. This construction has rank at most rr and satisfies W^lX=[A]r\hat{W}_l X = [A]_r, since the row space of [A]r[A]_r is contained in that of XX. The optimal layerwise distortion is thus characterised by the activation spectrum.

Proposition 2: Distortion of the deployed compression. For every square sketch SS, including every sample from the trained VAE, the deployed weights Wl∗=[SWl]rW^{\ast}_l = [SW_l]_r satisfy ∥WlX−Wl∗X∥F≤Dl∗(r)+(∥WlX∥2+∥Wl∥2∥X∥2)∥S−Pr∥F.\boxed{\Vert W_lX-W^{\ast}_lX\Vert_F \leq D^{\ast}_l(r) + (\Vert W_lX\Vert_2 + \Vert W_l\Vert_2\Vert X\Vert_2)\Vert S-P_r\Vert_F}. To prove this, write C=[SWl]rC=[SW_l]_r. The triangle inequality yields ∥A−CX∥F≤Dl∗(r)+∥(Pr−S)A∥F+∥(SWl−C)X∥F.\Vert A-CX\Vert_F \leq D^{\ast}_l(r)+\Vert(P_r - S)A\Vert_F + \Vert(SW_l - C)X\Vert_F. Because PrWlP_rW_l has rank at most rr, optimality of truncated SVD implies ∥SWl−C∥F≤∥(S−Pr)Wl∥F.\|SW_l -C\|_F \leq \|(S-P_r)W_l\|_F. Consequently, the sharper intermediate bound ∥A−CX∥F≤Dl∗(r)+∥(S−Pr)A∥F+∥X∥2∥(S−Pr)Wl∥F\|A-CX\|_F \leq D^*_l(r) + \|(S-P_r)A\|_F + \|X\|_2\|(S-P_r)W_l\|_F holds. Submultiplicativity yields the stated proposition.

When ∥WlX∥2>0\|W_lX\|_2 > 0, its additive term can equivalently be written as (1+ρl)∥WlX∥2∥S−Pr∥F,ρl=∥Wl∥2∥X∥2∥WlX∥2≥1.(1+\rho_l)\|W_lX\|_2\|S-P_r\|_F, \quad \rho_l = \frac{\|W_l\|_2 \|X\|_2}{\|W_lX\|_2} \geq 1.

The factor ρl\rho_l arises from bounding weight-space truncation error in the activation-space metric.

Interpretation and scope. Proposition 2 holds for every realised sketch, regardless of how its distribution was learned. It bounds excess distortion above the optimal rank-rr benchmark by the sketch’s deviation from an optimal activation-space projector. At S=PrS= P_r, SWlSW_l already has rank at most rr, so the final truncation leaves it unchanged, and the optimal distortion is attained exactly.

This is a deterministic stability result, not a claim that VAE prior samples are close to PrP_r during compression with a specified probability. It is also a layerwise statement for prescribed activations, rather than an end-to-end perplexity guarantee. It applies to fixed held-out activations when the corresponding oracle and projector are recomputed for analysis, while keeping the deployed weights fixed. Population-level conclusions however require additional assumptions and analysis.

Our bound depends explicitly on the activations through Dl∗(r),PrD^*_l(r), P_r, and the norm factors. Such dependence does not itself preclude distribution-free validity: classical low-rank approximation bounds also involve the input’s singular-value tail. Classical oblivious constructions sample independently of the input matrix, with sketch size determined by parameters such as dimensions, target rank, accuracy, and confidence. Their approximation guarantees are proved for specified sketch distributions. Our learned distribution is data-aware, and we do not claim that is satisfies these hypotheses. Proposition 2 instead supplies a deterministic, per-sketch bound valid for arbitrary fixed weights and activations, without assuming a data-generating distribution.

Reviewer_KdSEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

To compress a model, the authors train a VAE to generate sketch matrices that enable to reduce the rank of the weight matrices during training instead of performing a costly SVD. The optimization is performed in the activation space, not in the weight space.

Critical strengths

None

Critical weaknesses
  • The comparison is performed with very old baselines: RSVD 2011, CUR 2009. Recent ones like ASVD (2023), SVD-LLM (2025) are absent.
  • The VAE is trained on Wikitext-2 (lignes 358-359), not the baselines. This weakens a lot the results and can explain why their compressed model beats the uncompressed baseline (which sounds very counter-intuitive).
  • The VAE+truncation does not improve compression time.
  • Inference latency is not reported, so the benefit of the method at inference remains unverified.
  • The compression ratio is not cleary defined: is it weight compression, rank compression ? Even if the authors seem to work on rank compression, reducing the rank doesn't mean reducing the model size, or FLOPs, or memory, which is the main goal of compression.
Questions for authors

Clarifications:

  • What is the point of the VAE since SVD is computed at inference ?
  • Which matrices are concerned by the compression ?

Requests:

  • Compare to baselines without adding fine-tuning only to your model for fairer comparison.
  • Compare to more recent baselines like ASVD, SVD-LLM to strengthen your approach.
  • Define clearly the compression ratio. How much memory does the compression save ?
4 comments

We thank the reviewer for their insightful feedback and suggestions and will address their comments below. We additionally also uploaded and updated version of the paper and highlighted the changes both in the main part of the paper and the appendix in blue.

W1 & R1,2. Additional baselines: As the proposed method is a probabilistic method, we restricted the reported baselines to related stochastic low-rank approximation methods for model compression, i.e., randomised SVD and CUR decomposition. As truncated SVD is the theoretically ideal deterministic baseline in weight space, we additionally included this.

As requested, we now additionally compare our method to ASVD and SVD-LLM and report results for GPT-2 Medium and Gemstone-256x23 in Table R1 and Table R2, respectively. We use the same dataset partition and number of samples as used in the generation of training data for the VAE for both ASVD and SVD-LLM to ensure a fair comparison between the approaches.

The proposed method outperforms both ASVD and SVD-LLM for the GPT-2 Medium model. For the Gemstone-256x23 model, ASVD and SVD-LLM outperform the remaining methods with our approach achieving the third best performance. As shown in Tables R3 and R4, ASVD and SVD-LLM however also take significantly more time to compress the model than the proposed sketching method does. We hence argue that even though for some models our method cannot outperform those additional deterministic compression methods, for the same rank, compressing with our method is faster than compressing with ASVD and SVD-LLM while also achieving better performance than other probabilistic methods.

Table R1: GPT-2 Medium (uncompressed PPL = 18.00)

Method0.50.40.30.20.1
SVD1794.19179.0522.0418.4618.26
RSVD1137.70125.6020.3218.4718.25
CUR3.08e81.48e91.18e81.89e101.05e9
ASVD101.3429.7521.4218.7218.18
SVD-LLM94.7864.5239.7820.4019.00
VAE53.01 ± 2.6920.84 ± 0.0317.90 ± 0.0316.65 ± 0.0015.73 ± 0.01

Table R2: Gemstone-256x23 (uncompressed PPL = 32.53)

Method0.50.40.30.20.1
SVD171.4289.2757.3141.8736.09
RSVD195.1198.1257.2942.1836.07
CUR9.50e66.36e61.93e62.06e73.41e6
ASVD56.3544.0538.6035.0633.38
SVD-LLM44.5839.0836.0234.1433.16
VAE77.36 ± 0.2155.14 ± 0.0444.92 ± 0.1739.41 ± 0.0534.68 ± 0.04

W2. Wikitext-2 training: As the full original pretraining corpus was not publicly released for GPT-2 models and the original GPT-2 paper [Radford et al., 2019] demonstrates strong zero-shot performance on Wikitext-2, we use the openly available Wikitext-2 dataset for training of the VAE. For the additional ASVD and SVD-LLM baselines added, we make use of the same data corpus that was used for VAE training and show that our method outperforms those baselines for GPT-2 Medium. We thus argue that using Wikitext-2 does not weaken the results and that training on the GPT-2 pretraining corpus would have not been possible.

[Radford et al., 2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.

W3. Missing compression time improvement: While the proposed method has a similar compression time compared to truncated SVD, we argue that a missing improvement in terms of compression time should not be seen as a disadvantage. In contrast to methods with similar compression times, our method achieves significantly lower perplexity values (see Tables R3 and R4 for GPT-2 Medium and Gemstone-256x23, respectively).

Table R3: GPT-2 Medium (compression ratio 0.4)

MethodCompression time (ms)Calibration (ms)Truncation (ms)Total time (ms)Perplexity
Uncompressed--------18.00
SVD51,044----51,044179.05
RSVD25,191----25,191125.60
CUR74,251----74,2511.48e+09
ASVD51,04816,584--67,63229.75
SVD-LLM55,634459,211--514,84464.52
VAE550--51,18551,73520.84 ± 0.03

Table R4: Gemstone-256x23 (compression ratio 0.4)

MethodCompression time (ms)Calibration (ms)Truncation (ms)Total time (ms)Perplexity
Uncompressed--------32.53
SVD7,780----7,78089.27
RSVD4,507----4,50798.12
CUR12,010----12,0106.36e+06
ASVD7,78917,765--25,55444.05
SVD-LLM8,24039,850--48,09039.08
VAE53.9--7,8367,89055.14 ± 0.04

W4 & Q1. Inference: We would like to clarify the proposed approach with respect to inference costs and time. As any of the other low-rank approximation model compression methods in this paper, the proposed method is applied only once on the trained uncompressed LLM. Once compressed, the weights are stored as low-rank factorisations of the original weight matrix — as common for low-rank approximation model compression methods. At inference, only a forward pass through the compressed model is thus needed - neither a forward pass through the VAE nor an SVD. Inference latency, memory costs and parameter count thus equal those of any of the other baseline methods.

W4. Inference latency: As in our experiments we assume the rank to be fixed and we only compare low-rank approximation methods for model compression, the number of parameters of the low-rank model does not differ between methods. The compressed language model thus has the same inference latency for the proposed approach as well as the baselines as in this case inference latency is depending on parameter count only. For completeness, we have thus added Table 3 in the appendix in the updated version of the draft, showing the parameter count for the uncompressed versus compressed layers of the GPT-2 Small and GPT-2 Medium models for different compression ratios. If by inference latency you refer to the overhead due to the low-rank compression itself, the results are depicted in Tables R3, R4 in the rebuttal and in Figure 2 in the revised draft.

W5 & R3. Definition of compression ratio: As correctly pointed out by the reviewer, the method compressed the rank of the weight matrices. To simplify comparability between models and model sizes, we report the compression ratio rather than an explicit rank. The compression ratio cr\text{cr} is defined as the rank ratio that a weight matrix is compressed and can be used to compute the rank of the compressed matrix via rc=ruc∗(1−cr)r_c = r_{\text{uc}} * (1-\text{cr}) for the compressed rank rcr_c and uncompressed rank rucr_{\text{uc}}. Assume e.g., the original rank of a matrix was 1010, for a compression ratio of 0.50.5 the rank of the compressed matrix thus is 55, whereas the rank of the compressed matrix is 66 for a compression ratio of 0.40.4.

W5 & R3. Model size, FLOPs and memory: While reducing the rank does not necessarily reduce the model size, FLOPs or memory, this is a general issue of low-rank approximation methods for model compression and not specific to the proposed approach. Table R5 gives an overview of the parameter counts for uncompressed versus compressed layers for GPT-2 Small, GPT-2 Medium and Gemstone-256x23 models for different compression ratios and ranks.

Table R5: Parameter count for uncompressed vs. compressed layers

ModelCompression ratioRankCompressed layers (%)Compressed layers (M)Uncompressed layers (M)
GPT-2 Small0.1691100.035.435.4
0.261499.935.435.4
0.353890.131.935.4
0.446180.028.335.4
0.538470.024.835.4
GPT-2 Medium0.1922100.0125.8125.8
0.2819100.0125.8125.8
0.371790.0113.3125.8
0.461480.0100.6125.8
0.551270.088.1125.8
Gemstone-256x230.1230100.09.09.0
0.2205100.09.09.0
0.317991.68.39.0
0.415483.57.59.0
0.512875.06.89.0

We have additionally added the same table to the appendix of the updated version of the paper.

Dear Reviewer KdSE,

Thanks again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

Reviewer_QmiuOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes learning a distribution over sketching matrices for low-rank LLM compression. Training targets S are obtained from an activation-aware rank-r approximation via least squares. A VAE is trained on these S, and at inference, S is sampled per layer. On GPT-2 Small/Medium and Gemstone-256, the method reports lower WikiText-2 perplexity than SVD/RSVD/CUR, and in some cases lower than the uncompressed model.

Critical strengths

To the best of my knowledge, learning a distribution over projection operators is novel. Decoder-only finetuning for transfer is a practical idea.

Critical weaknesses

Train/eval domain overlap The γ term is trained on WikiText-2 train, and evaluation is on WikiText-2 test. The gains may therefore come from domain adaptation rather than better compression. Comparing Table 1 with Table 2, the advantage disappears once the LM is finetuned on WikiText-2.

Missing key baselines The method should be compared with Ŵ, ASVD, and SVD-LLM.

Practical benefit. The compression overhead (latency, memory, number of parameters) is not reported. Missing ablation. It is unclear how a layer-agnostic S can suit every layer, and whether stochastic sampling is beneficial.

Suspicious CUR results Perplexities above 10⁷ suggest a possible implementation problem.

Limited experimental scope All models are small, and no other tasks are evaluated, even though many larger open-weight LLMs are available.

Questions for authors
  1. What are the parameter count and memory consumption of the compressed models?
  2. Which split was used to select β, γ, and the best epoch?
4 comments

We thank the reviewer for their insightful feedback and will address their questions and critical weaknesses below. Additionally, we uploaded an updated version of the paper and highlighted the changes both in the main part of the paper and the appendix in blue.

W2. Additional baselines: As the proposed method is a probabilistic method, we restricted the reported baselines to related stochastic low-rank approximation methods for model compression, i.e., randomised SVD and CUR decomposition. We additionally include truncated SVD (W^\hat{W}) as it is the theoretically ideal deterministic baseline in weight space.

As requested, we now provide additional results and compare our VAE method to ASVD and SVD-LLM and report results for GPT-2 Medium and Gemstone-256x23 in Table R1 and Table R2, respectively. We use the same dataset partition and number of samples as used in the generation of training data for the VAE for both ASVD and SVD-LLM to ensure a fair comparison between the approaches.

The proposed method outperforms both ASVD and SVD-LLM for the GPT-2 Medium model. For the Gemstone-256x23 model, ASVD and SVD-LLM outperform the remaining methods with our approach achieving the third best performance. As shown in Tables R3 and R4, ASVD and SVD-LLM however also take significantly more time to compress the model than the proposed sketching method does. We hence argue that even though for some models our method cannot outperform those additional deterministic compression methods, for the same rank, compressing with our method is faster than compressing with ASVD and SVD-LLM while also achieving better performance than other probabilistic methods.

Table R1: GPT-2 Medium (uncompressed PPL = 18.00)

Method0.50.40.30.20.1
SVD1794.19179.0522.0418.4618.26
RSVD1137.70125.6020.3218.4718.25
CUR3.08e81.48e91.18e81.89e101.05e9
ASVD101.3429.7521.4218.7218.18
SVD-LLM94.7864.5239.7820.4019.00
VAE53.01 ± 2.6920.84 ± 0.0317.90 ± 0.0316.65 ± 0.0015.73 ± 0.01

Table R2: Gemstone-256x23 (uncompressed PPL = 32.53)

Method0.50.40.30.20.1
SVD171.4289.2757.3141.8736.09
RSVD195.1198.1257.2942.1836.07
CUR9.50e66.36e61.93e62.06e73.41e6
ASVD56.3544.0538.6035.0633.38
SVD-LLM44.5839.0836.0234.1433.16
VAE77.36 ± 0.2155.14 ± 0.0444.92 ± 0.1739.41 ± 0.0534.68 ± 0.04

Table R3: Compression time GPT-2 Medium, compression ratio 0.4

MethodCompression (s)Per-layer median (ms)Calibration (s)Truncation (s)Total (s)
SVD51.041062.5––51.04
RSVD25.19532.2––25.19
CUR74.251546.0––74.25
ASVD51.051063.116.58–67.63
SVD-LLM55.631193.7459.21–514.84
VAE0.5511.5–51.1851.73

Table R4: Compression time Gemstone-256x23, compression ratio 0.4

MethodCompression (s)Per-layer median (ms)Calibration (s)Truncation (s)Total (s)
SVD7.78114.5––7.78
RSVD4.5162.9––4.51
CUR12.01176.0––12.01
ASVD7.79114.417.76–25.55
SVD-LLM8.24118.739.85–48.09
VAE0.0540.71–7.847.89

W1. Wikitext-2 and domain adaptation:

We use the Wikitext-2 training dataset during training of the VAE as the full original pretraining corpus for the GPT-2 models was not publicly released. As stated in the original GPT-2 paper [Radford et al., 2019], GPT-2 models demonstrated strong zero-shot performance on Wikitext-2, hence why we chose this text corpus.

To prove or disprove the domain adaptation claim, we empirically test ASVD and SVD-LLM on the same dataset partition that was used for creating the VAE training data. As shown in Table R1, ASVD and SVD-LLM are not able to outperform the uncompressed baseline even though they are using the Wikitext-2 train partition. As the original model language model is used as part of the γ\gamma loss term for training the VAE, a potential domain adaptation effect should be more pronounced for ASVD and SVD-LLM. This therefore contradicts any domain adaptation claims.

[Radford et al., 2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.

Q1 & W3. Parameter counts and memory consumption: As we fix the target rank per layer, all low-rank approximation methods including the proposed method one yield to an identical number of parameters and memory requirement. For completeness, we have added Table 3 to the appendix of the updated PDF — showing the parameter count for uncompressed versus compressed layers for GPT-2 Small, GPT-2 Medium and Gemstone-256x23 models for different compression ratios. The compression time, i.e., the latency overhead added due to compression, is shown in Tables R3, R4 in the rebuttal and in Figure 2 in the revised draft.

W3. Practical benefit: As highlighted through the results on the SVD, ASVD and SVD-LLM baselines, stochastic sampling is beneficial for a performance-computation tradeoff. The proposed approach consistently outperforms truncated SVD as well as ASVD and SVD-LLM for selected models while requiring significantly less computation time (see tables above).

We argue that given the experimental results, training a VAE per layer to learn the distribution of sketching matrices per layer is not required nor desirable and refrain from this ablation due to impracticality. Training and using a separate VAE per layer would drastically increase the computational costs, especially for deep networks. We furthermore argue that this “critical weakness” in fact can be seen as a strength as the trained VAE generalises over layers.

W4. CUR results: We carefully double checked the CUR implementation and can confirm that the high perplexity values do not stem from a possible implementation problem. We rather suspect the random sampling of rows and columns of the weight matrix as performed by the method to be incompatible with the structure inherent in weight matrices of neural networks - leading to high perplexity values.

Q2. β,γ\beta, \gamma split: As training of the VAE especially for small LLMs is computationally cheap, we used the full training partition as described in Section A.3 in the appendix of the paper to determine the β\beta and γ\gamma parameters. We used estimations based on GPT-2 Small to find a suitable parameter range for larger models.

W5. Experiments on larger models: As our experiments on larger models are still ongoing, we plan to add them as soon as we obtain results.

Dear Reviewer Qmiu,

Thanks again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

Sorry for the late reply. Thanks for the additional experiments and clarifications. They resolve some of my earlier concerns, but a couple of points are still open.

W1. I’m not convinced the ASVD and SVD-LLM results rule out domain adaptation. Those methods only use activations for calibration, while your VAE is trained directly on WikiText-2, so the comparison doesn’t quite settle it. Cross-dataset results (WikiText-2 → C4 and C4 → WikiText-2) would.

Q2. How were β, γ, and the best epoch chosen? In particular, was WikiText-2 test perplexity used for tuning or checkpoint selection? A sensitivity analysis over β and γ would also show whether the method is robust to these choices.

I still think the main claims need more validation, but the new results help, so I’m raising my score by one point to Weak Reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.