acceptodds
Under review as a conference paper at ICLR 2027

Slide&Prune: A Unified Pruning Framework for Joint Low Rank Compression and Channel Sparsification via Sliding Window

Abstract

Low-rank compression reduces the deployment cost of large language models (LLMs), but preserving model quality under aggressive compression remains challenging. Conventional top-k singular-value truncation is optimal for the corresponding layer-wise reconstruction objective, yet this guarantee does not extend to the behavior of multiple composed Transformer blocks. We introduce Slide&Prune (S&P), a post-training framework that extends sliding-window reconstruction to low-rank compression, enabling out-of-order singular-component selection according to its effects across neighboring blocks. This formulation generalizes rank truncation into a pruning problem, allowing learnable masks to jointly select singular components, attention heads, and feed-forward network channels under a shared parameter budget. S&P uses a two-stage pipeline of global allocation and sliding-window refinement while keeping pretrained weights and SVD factors fixed. Experiments across OPT, Vicuna, and LLaMA models from 6.7B to 30B parameters establish state-of-the-art compression–quality trade-offs among the evaluated structured pruning and low-rank compression methods, with particularly strong gains under aggressive compression. On LLaMA-1-7B at a compression ratio of 0.4, S&P achieves WikiText-2 perplexity of 13.75, compared with 45.17 for the strongest evaluated baseline.

Primary area:
generative models
Keywords:
Efficient AI, Low Rank Compression, Structured Pruning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ooz4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new approach for low-rank compression of LLMs. They provide strong experimental results backing their approach.

Critical strengths

1/ The experimental results of the approach seems to be quite strong.

Critical weaknesses

1/ The paper proposes a more computationally expensive method than some of the baselines; it would be interesting to compare both at matched compute and runtime budgets.

2/ The paper’s approach provides limited novelty. Low-rank selection based on pruning singular values in a differentiable way or not is a common approach in this field, leaving very limited novelty ideas in the paper.

Questions for authors

1/ Can the authors provide more experiments matching compute budget or comparing with other methods using finetuning post compression?

2/ Can the authors argue more about the novelty of the approach? I would be happy to reconsider my review given a strong rebuttal.

9 comments

We thank Reviewer ooz4 for the careful and constructive review. We respond to each of the reviewer's concerns in detail below.


1. Comparison of methods with matched compute

Original Comment: "[W1] The paper proposes a more computationally expensive method than some of the baselines; it would be interesting to compare both at matched compute and runtime budgets. [Q1] Can the authors provide more experiments matching compute budget or comparing with other methods using finetuning post compression?"

We thank the reviewer for raising concerns about compute and runtime cost. Compression is a one-time, pre-deployment cost, whereas the resulting accuracy gains persist for the lifetime of the deployed model. The additional compression time of S&P is therefore well justified. S&P takes 99.4 minutes to compress LLaMA-1-7B on a single H200, compared with 11.7 minutes for SVD-LLM and 8.7 minutes for ZS-SVD (Table 6). For fair comparison, we have re-evaluated relevant strong SVD baselines (AA-SVD, Dobi-SVD, ZS-SVD) alongside applying remapping under identical experimental and evaluation settings; only Dobi-SVD without remapping is quoted from its paper.

CRMethodRemappingWiki2PTBC4OBQAARC-eARC-cWinoGHellaSPIQAMathQAAvg
1Dense5.688.357.080.340.750.420.70.570.790.270.55
0.8ZS-SVDX6.7611.4710.110.30.720.380.670.490.740.250.51
0.8AA-SVD†X7.6816.3414.870.290.640.310.640.430.680.240.46
0.8Dobi-SVD‡X8.5414.8310.010.260.590.310.660.440.70.230.46
0.8S&PX6.2710.549.180.290.720.370.680.510.760.250.51
0.8Dobi-SVDO6.089.028.070.330.730.40.690.550.770.270.53
0.8ZS-SVDO5.98.837.560.340.750.420.70.560.780.260.55
0.8S&PO5.798.677.310.320.730.410.690.560.780.270.54
0.6ZS-SVDX11.5843.4534.580.210.520.260.620.350.630.220.4
0.6AA-SVD†X12.0147.2334.760.20.440.230.570.330.590.230.37
0.6Dobi-SVD‡X13.5446.3823.540.220.410.270.580.340.610.230.38
0.6S&PX8.2825.9319.020.240.60.290.610.40.680.230.43
0.6Dobi-SVDO7.6912.3612.150.270.650.340.680.480.730.240.48
0.6ZS-SVDO712.3610.880.310.720.380.670.480.740.240.51
0.6S&PO6.3510.689.590.290.710.370.660.510.750.250.51
0.4ZS-SVDX41.76285.46209.110.160.310.20.550.280.560.210.32
0.4AA-SVD†X29.44212.28118.480.150.320.190.50.270.540.220.31
0.4Dobi-SVD‡X46.18238.91190.620.150.310.20.520.280.540.220.32
0.4S&PX13.7585.948.730.20.440.240.550.320.60.220.37
0.4Dobi-SVDO14.9849.1143.90.210.440.240.610.340.60.220.38
0.4ZS-SVDO13.1351.4145.780.230.50.270.60.340.620.220.4
0.4S&PO8.222.7517.890.260.60.280.610.410.670.230.44

† AA-SVD evaluated without its block-level local refinement step.
‡ Quoted from the Dobi-SVD paper, as no checkpoint is available; Dobi-SVD with remapping was re-run by us.

Our results show that the S&P compressed model is substantially more accurate, with the largest gains under aggressive compression; at CR 0.4, S&P reaches a WikiText-2 perplexity of 13.75, compared with 29.44 for the strongest evaluated no fine-tuning / weight update baseline and zero-shot accuracy average of 0.37 compared to 0.32.

The same one-time-cost argument applies to the baselines: if extra compression time is acceptable, a practitioner could instead spend it fine-tuning a cheaper method after compression. To test this directly, we compare S&P against baselines that use a post-compression update: AA-SVD with block-level local refinement, ZS-SVD with its projected-gradient weight updates, and SVD-LLM with LoRA fine-tuning. We also apply AA-SVD's block-level refinement to S&P, using the same 256 calibration sequences, to check whether S&P's advantage persists once both sides are fine-tuned. All results are on LLaMA-1-7B at CR 0.8, 0.6 and 0.4:

CRMethodFine-tuning / Weight UpdateUpdate DataUpdate SamplesWiki2PTBC4OBQAARC-eARC-cWinoGHellaSPIQAMathQAAvg
1Dense5.688.357.080.340.750.420.700.570.790.270.55
0.8AA-SVDOWiki-2 (Calib. Set)2566.9012.3211.430.300.710.360.660.480.720.250.50
0.8ZS-SVD 5x‖OWiki-2 (Calib. Set)206.4311.1710.420.320.720.390.670.500.750.250.51
0.8SVD-LLM + LoRA*OAlpaca500007.73-12.230.330.67-0.690.550.790.26-
0.8S&PXX-6.2710.549.180.290.720.370.680.510.760.250.51
0.8S&P + Refinement§OWiki-2 (Calib. Set)2566.109.738.840.300.720.380.670.520.770.260.52
0.6AA-SVDOWiki-2 (Calib. Set)2568.3525.0518.510.260.610.310.620.410.660.230.44
0.6ZS-SVD 5x‖OWiki-2 (Calib. Set)209.4536.5226.200.230.560.290.620.380.660.220.42
0.6SVD-LLM + LoRA*OAlpaca500009.27-15.630.290.59-0.680.520.690.20-
0.6S&PXX-8.2825.9319.020.240.600.290.610.400.680.230.43
0.6S&P + Refinement§OWiki-2 (Calib. Set)2567.4019.5514.100.260.650.310.640.440.700.240.46
0.4AA-SVDOWiki-2 (Calib. Set)25613.6174.1745.790.180.440.230.560.330.600.230.37
0.4ZS-SVD 10x‖OWiki-2 (Calib. Set)4018.49144.8976.860.170.380.230.550.300.580.220.35
0.4SVD-LLM + LoRA*OAlpaca5000031.79-43.710.110.23-0.210.140.170.08-
0.4S&PXX-13.7585.9048.730.200.440.240.550.320.600.220.37
0.4S&P + Refinement§OWiki-2 (Calib. Set)25610.4647.9727.360.220.490.240.580.360.620.230.39

* SVD-LLM fine-tuned with LoRA on 50K Alpaca samples; "-" denotes results not reported.
§ S&P followed by AA-SVD's block-level local refinement on the same 256 calibration sequences.
‖ ZS-SVD (ZeroSum SVD) with 5× / 10× denotes the number of correction iterations.

Reviewer_Z9gGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Slide&Prune (S&P), a post-training framework that jointly selects singular components, attention heads, and FFN channels under a shared parameter budget while keeping pretrained weights and SVD factors fixed. It first learns a global allocation through differentiable masks, then refines these masks using reconstruction objectives over overlapping Transformer-block windows, allowing non-prefix singular-component selection. The learned selections are materialized as compact dense or factorized operators according to their parameter costs. Experiments on OPT, Vicuna, and LLaMA models report improved compression–quality trade-offs, with the largest gains demonstrated under aggressive compression on LLaMA-1-7B. Ablations examine sliding-window optimization and non-prefix selection, alongside measurements of compression cost and inference efficiency.

Critical strengths

1.A well-motivated problem formulation. The paper addresses the gap between layer-wise reconstruction quality and the behavior of composed Transformer blocks. Jointly selecting singular components, attention heads, and FFN channels under a shared budget is a meaningful direction, since these choices affect both reconstruction quality and one another’s parameter costs. This formulation provides a useful basis for studying compression decisions beyond isolated matrices.

Critical weaknesses

1.The experimental coverage is insufficient to establish the generality of the main claims. Although several model families are evaluated, aggressive-compression results and key ablations are concentrated on LLaMA-1-7B; the other models are mainly evaluated at CR=0.8. The evaluation also relies on older architectures. Furthermore, comparisons with structured pruning methods are limited to LLM-Pruner and FLAP on LLaMA-1-13B at CR=0.8, leaving their relative performance under aggressive compression unclear. Directly relevant hybrid compression baselines are also missing. These limitations weaken the evidence that the proposed joint framework consistently improves over alternative compression strategies.

2.Important experimental details and attribution studies are missing. The paper does not specify key settings such as window size, sliding stride, optimization steps, learning rates, and the Stage 2 temperature schedule, limiting reproducibility and assessment of its substantially higher compression cost. The ablations also do not adequately isolate the benefit of jointly pruning heads and FFN channels beyond low-rank compression, or establish whether the sliding-window comparison uses matched optimization budgets. Consequently, the sources of the reported gains remain insufficiently resolved.

Questions for authors

1.Experimental coverage: Could you extend the evaluation to a representative recent model, particularly one using GQA, at an aggressive compression ratio such as CR=0.4 or 0.6? This would help establish whether the gains demonstrated on LLaMA-1-7B generalize to newer architectures.

2.Baseline coverage: Could you include representative recent structured pruning and hybrid compression methods under matched parameter and calibration budgets? In particular, comparisons at aggressive compression ratios would help determine whether S&P’s advantage extends beyond the primarily SVD-based comparisons.

3.Reproducibility and attribution: Please specify the window size, stride, optimization steps, learning rates, and temperature schedule. Could you also clarify whether the sliding-window ablation uses matched optimization budgets and provide a low-rank-only control to isolate the benefit of jointly pruning heads and FFN channels?

11 comments

We thank Reviewer Z9gG for the careful and constructive review. We are glad the reviewer found that jointly selecting singular components, attention heads, and FFN channels under a shared budget is a meaningful direction, and that the reviewer recognized our unified formulation as a useful basis for studying compression decisions beyond individual matrices. We respond to each of the reviewer's concerns in detail below.


1. Evaluation protocol and reproducibility

Original comment: "[W2] Important experimental details and attribution studies are missing. The paper does not specify key settings such as window size, sliding stride, optimization steps, learning rates, and the Stage 2 temperature schedule, limiting reproducibility and assessment of its substantially higher compression cost. Reproducibility and attribution: Please specify the window size, stride, optimization steps, learning rates, and temperature schedule."

We sincerely apologize for omitting the key hyperparameter details necessary to reproduce our results. To ensure full reproducibility, the table below provides the exact calibration, initialization, and hyperparameter configurations used to compress all model variants.

StageHyperparameterValue
CalibrationCalibration datasetWikiText-2 (train) or C4 (train)
Calibration samples256
Sequence length2048
Batch size1
Seed42
InitWhiteningCholesky of calibration Gram matrix
Initial rank kktarget ratio × break-even rank = ratio·mn/(m+n)
Head / MLP-channel gate init ZZ1
MLP mask placementintermediate
Head gate placemento_proj input
Stage 1Epochs5
OptimizerAdamW, weight decay 0
Rank kk lr5
Gate lr (head/FFN gates)0.01
Rank kk beta0.1
μ\mu (gate steepness)5 → 15 (linear)
γ\gamma init1
γ\gamma lr10
Lossrelative MSE on final hidden states (Eq. 4)
Stage 2Window size8 blocks for 7B, 8B models; 10 for 13B; 12 for 30B
StrideWindow size / 2
Epochs10
OptimizerAdamW, weight decay 0
Gate lr (singular-value/head/FFN gates)0.01
μ\mu, singular-value gates5 → 50 (linear)
μ\mu, head/MLP gates15 → 50 (linear)
γ\gammaone per window, init 1.0
Window compression ratio targetsrescaled by Stage-1 overshoot
SlicingFinal maskZ>0.5Z>0.5

Additionally, upon careful review, we discovered that acc_norm was inadvertently reported instead of standard accuracy (acc) for the compressed model results. To ensure a strictly fair comparison, we have re-evaluated strong baselines (ZS-SVD, AA-SVD, and Dobi-SVD with remapping) alongside our S&P method using an identical calibration setup, evaluation harness, and benchmark metric. We have also re-evaluated structured pruning baselines and hybrid compression baselines under identical setup.

The following table describes the exact evaluation setting:

EvaluationDataset / SettingDetails
PPLWikiText-2test split, 2048-token chunks
PPLPTBtest split joined with '\n\n', 2048-token chunks
PPLC4256 random 2048-token windows from validation shard
AccuracyHarnesslm-evaluation-harness v0.4.2, zero-shot, batch 8
AccuracyMetricplain acc for all tasks
AccuracyTasksOBQA, ARC-e, ARC-c, WinoGrande, HellaSwag, PIQA, MathQA

2. Evaluation on newer architectures and more aggressive compression ratios

Original comment: "[Q1] Experimental coverage. The experimental coverage is insufficient to establish the generality of the main claims. The evaluation also relies on older architectures. Could you extend the evaluation to a representative recent model, particularly one using GQA, at an aggressive compression ratio such as CR=0.4 or 0.6? This would help establish whether the gains demonstrated on LLaMA-1-7B generalize to newer architectures."

We agree that evaluating additional model architectures and compression ratios is important. We therefore evaluate our method on newer architectures that use GQA: LLaMA-3.1-8B and Qwen3-8B. We evaluate these architectures across compression ratios of 0.8, 0.6, and 0.4. We evaluated all baselines and S&P based on matching eval settings and hyperparameters. In addition, we include AA-SVD, a recent and competitive SVD-based method, as a new baseline for a more comprehensive comparison. The main results reported in the original AA-SVD paper include its block-level local refinement step. Since our method does not use such refinement, we reran all AA-SVD experiments without refinement for a fair comparison. The following table shows results for comparing S&P against all baselines for all compression ratios on LLaMA-3.1 8B:

LLaMA-3.1-8B (bold: best, italic: second best)

CRMethodOBQAARC-eARC-cWinoGHellaSPIQAMathQAAvg
1.0Dense0.330.810.510.730.600.800.400.60
0.8SVD0.160.240.210.510.250.520.180.30
0.8SVD-LLM0.240.600.310.670.430.710.260.46
0.8ZS-SVD0.290.710.370.690.490.770.260.51
0.8AA-SVD†0.150.350.190.530.330.610.200.34
0.8S&P0.300.730.390.690.540.780.290.53
0.6SVD0.150.240.220.500.260.520.180.29
0.6SVD-LLM0.150.320.190.570.290.580.210.33
0.6ZS-SVD0.190.430.210.580.310.620.220.37
0.6AA-SVD†0.150.330.180.530.300.600.210.33
0.6S&P0.240.590.280.590.430.720.240.44
0.4SVD0.120.250.210.510.260.530.200.30
0.4SVD-LLM0.120.270.210.520.260.530.200.30
0.4ZS-SVD0.120.280.210.500.260.540.200.30
0.4AA-SVD†0.150.300.180.500.270.560.200.31
0.4S&P0.190.420.210.530.320.630.220.36

† AA-SVD evaluated without its block-level local refinement step.

Reviewer_91WVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a post-training LLM compression method that jointly learns masks over singular components, attention heads, and FFN channels under a shared parameter budget. It uses a two-stage pipeline: global allocation followed by sliding-window, multi-block refinement.

Critical strengths

Strengths: As far as I could find, the paper offers moderate novelty, since it jointly optimizes low-rank compression with attention-head and FFN-channel pruning under a single budget. This would be particularly interesting if the authors showed that the hybrid outperforms each approach alone in a head-to-head setting. The paper shows large gains over the compared SVD baselines and is competitive with recent published results, although the SOTA claim is harder to defend since some baselines are missed. The experiments cover several models, datasets, and tasks.

Critical weaknesses

Overstated claims and incomplete related work. The paper overstates its claims, including state-of-the-art results, and its review of prior work is limited. It omits a substantial body of directly relevant work that should at least be discussed, and the novelty should be justified against it, particularly AA-SVD, SAES-SVD, GRASPrune, LLRC, Hinge, Collaborative Compression, DF, LoRAP, and MoDeGPT. This list is not exhaustive, and the related-work section should be substantially updated to cover recent work. In particular, AA-SVD achieves results competitive with S&P, which weakens the SOTA claim.

Missing ablations. No ablation isolates pruning-only from low-rank-only compression at the same budget. The paper breaks down the removed parameters by component (Figure 2), but this does not establish that hybrid compression is better than either approach alone. More broadly, the method combines several components whose effects interact. Each should be isolated to show its individual contribution.

Loss analysis and hyperparameters. The loss function should be analyzed, and the progress of each of its terms should be reported during training. The paper also omits several key hyperparameters.

Accuracy above the uncompressed baseline. In Table 1 (at CR 0.8) and for all models in Table 2, the compressed models achieve higher accuracy than the uncompressed baseline. Small fluctuations can occur, but gains this consistent should be carefully double-checked and justified.

Fragile loss balance. The absolute-value budget penalty can oscillate around the target. In addition, the dual update uses the signed gap while the penalty uses the absolute gap, which is inconsistent.

The baseline results appear to be copied from other papers, and the paper does not state their provenance. The authors should evaluate at least one baseline, and the uncompressed model, with their own pipeline (same evaluation harness version and metrics) and show that the reported numbers are reproduced.

There are several typos in the paper: Examples: lines 115-116 and 306-307.

References

Eo et al., “A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression,” 2023.

Hu et al., “SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression,” 2026.

Li et al., “Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression,” 2020.

Li et al., “Towards Compact CNNs via Collaborative Compression,” 2021.

Li et al., “LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models,” 2024.

Lin et al., “MoDeGPT: Modular Decomposition for Large Language Model Compression,” 2025.

Sinha et al., “AA-SVD: Anchored and Adaptive SVD for Large Language Model Compression,” 2026.

Sundrani et al., “Low-Rank Compression of Language Models via Differentiable Rank Selection,” 2025.

Solgi et al., “Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM,” 2025.

Wang et al., “GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models,” 2026.

Questions for authors

How would light fine-tuning after compression affect these results? Does hybrid compression have any positive or negative effect on post-compression fine-tuning?

Given that this method requires gradients, one may wonder how, in principle and for the low-rank part only, it differs from double-sided whitening methods such as OBD-LLM and GFWSVD.

Is it also possible to replace the activation-aware SVD with double-sided whitening, and how would the results change?

Chekalina et al., “Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models,” 2025.

Li et al., “Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation,” 2026.

15 comments

We thank Reviewer 91WV for the careful and constructive review. We appreciate that the reviewer found joint optimization of low-rank compression with attention-head and FFN-channel pruning under a single budget to be novel, noted the large gains over the compared SVD baselines and competitiveness with recent published results, and recognized the breadth of our experiments across models, datasets and tasks. In response to the reviewer's concerns, we address each point in detail below.


1. Evaluation protocol and reproducibility

Original Comment: "[W4] Accuracy above the uncompressed baseline. In Table 1 (at CR 0.8) and for all models in Table 2, the compressed models achieve higher accuracy than the uncompressed baseline. Small fluctuations can occur, but gains this consistent should be carefully double-checked and justified."

We sincerely apologize for the inconsistency in our tables. Upon carefully reviewing, we discovered that acc_norm was inadvertently reported instead of standard accuracy (acc) for the compressed model results. To ensure a strictly fair comparison, we have re-evaluated strong baselines (ZS-SVD, AA-SVD, and Dobi-SVD with remapping) alongside our S&P method using an identical calibration setup, evaluation harness, and benchmark metric. We have also re-evaluated structured pruning baselines and hybrid compression baselines under identical setup.

The following table describes the exact evaluation setting:

EvaluationDataset / SettingDetails
PPLWikiText-2test split, 2048-token chunks
PPLPTBtest split joined with '\n\n', 2048-token chunks
PPLC4256 random 2048-token windows from validation shard
AccuracyHarnesslm-evaluation-harness v0.4.2, zero-shot, batch 8
AccuracyMetricplain acc for all tasks
AccuracyTasksOBQA, ARC-e, ARC-c, WinoGrande, HellaSwag, PIQA, MathQA

Original Comment: "[W3] The paper also omits several key hyperparameters"

We sincerely apologize for omitting the key hyperparameter details necessary to reproduce our results. To ensure full reproducibility, the table below provides the exact calibration, initialization, and hyperparameter configurations used to compress all model variants.

StageHyperparameterValue
CalibrationCalibration datasetWikiText-2 (train) or C4 (train)
Calibration samples256
Sequence length2048
Batch size1
Seed42
InitWhiteningCholesky of calibration Gram matrix
Initial rank kktarget ratio × break-even rank = ratio·mn/(m+n)
Head / MLP-channel gate init ZZ1
MLP mask placementintermediate
Head gate placemento_proj input
Stage 1Epochs5
OptimizerAdamW, weight decay 0
Rank kk lr5
Gate lr (head/FFN gates)0.01
Rank kk beta0.1
μ\mu (gate steepness)5 → 15 (linear)
γ\gamma init1
γ\gamma lr10
Lossrelative MSE on final hidden states (Eq. 4)
Stage 2Window size8 blocks for 7B, 8B models; 10 for 13B; 12 for 30B
StrideWindow size / 2
Epochs10
OptimizerAdamW, weight decay 0
Gate lr (singular-value/head/FFN gates)0.01
μ\mu, singular-value gates5 → 50 (linear)
μ\mu, head/MLP gates15 → 50 (linear)
γ\gammaone per window, init 1.0
Window compression ratio targetsrescaled by Stage-1 overshoot
SlicingFinal maskZ>0.5Z>0.5

2. Effectiveness of hybrid compression compared with individual components

Original Comment: "This would be particularly interesting if the authors showed that the hybrid outperforms each approach alone in a head-to-head setting … [W2] Missing ablations. No ablation isolates pruning-only from low-rank-only compression at the same budget. The paper breaks down the removed parameters by component (Figure 2), but this does not establish that hybrid compression is better than either approach alone. More broadly, the method combines several components whose effects interact. Each should be isolated to show its individual contribution."

We agree that directly comparing hybrid compression against its individual components is important. To address this concern, we conduct controlled experiments on LLaMA-1-7B, comparing the following three variants: (1) S&P SVD-only, (2) S&P structured pruning-only, and (3) S&P hybrid. All three approaches are evaluated for LLaMA-1 7B under identical compression ratios of 0.8, 0.6, and 0.4, with consistent evaluation settings.

At CR 0.8 the three are close, with joint best on all three perplexities and tied with rank-only on accuracy. At CR 0.6 the result is mixed: joint is best on WikiText-2 (8.28 vs. 9.29 and 10.83), prune-only is best on PTB and C4, and rank-only is marginally ahead on average accuracy (0.44 vs. 0.43). At CR 0.4 neither mechanism alone remains usable: prune-only collapses (WikiText-2 perplexity 2459) and rank-only reaches 20.92, while joint reaches 13.75, is best on all three corpora and has the highest average accuracy (0.37 vs. 0.34). The value of joint selection therefore grows with compression.

CRMethodCompressionWiki2PTBC4OBQAARC-eARC-cWinoGHellaSPIQAMathQAAvgDrop
1Dense5.688.357.080.340.750.420.700.570.790.270.550.00
0.8S&Prank-only6.5011.029.590.300.710.380.680.500.760.250.517.10
0.8S&Pprune-only6.7511.649.210.300.710.370.640.510.760.240.508.10
0.8S&Pjoint6.2710.549.180.290.720.370.680.510.760.250.517.10
0.6S&Prank-only9.2932.1522.380.260.600.280.620.390.670.220.4420.60
0.6S&Pprune-only10.8323.6117.010.240.540.250.560.380.670.210.4125.90
0.6S&Pjoint8.2825.9319.020.240.600.290.610.400.680.230.4320.90
0.4S&Prank-only20.92161.2383.610.170.380.230.550.300.580.210.3437.10
0.4S&Pprune-only2459.00557.52342.520.160.360.240.550.290.560.220.3437.90
0.4S&Pjoint13.7585.9048.730.200.440.240.550.320.600.220.3732.70

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your careful reading and constructive suggestions. We have addressed each review in its individual response, and the additional comparisons and analyses you asked for have made the paper substantially more complete. We have uploaded a revised PDF for both the main paper and the supplementary material, with all changes highlighted in blue. The main revisions are:

  • Evaluation: all baselines and the uncompressed model are re-evaluated under a single pipeline, with the full setup and hyperparameters reported in the appendix.
  • Baselines: new comparisons with a recent SVD method (AA-SVD), a structured pruning method (GRASPrune), and hybrid methods (LoSparse, LoRAP), including results on GQA models (LLaMA-3.1-8B and Qwen3-8B).
  • Ablations: hybrid versus SVD-only and prune-only compression at equal budgets.
  • Additional analyses: post-compression fine-tuning, double-sided whitening, and training dynamics of the loss terms.
  • Related work: expanded to cover the methods the reviewers identified, with the novelty of S&P clarified against them.

We thank you for your time and effort.

Sincerely,

The Authors

AuthorsOpenReview ↗

General Summary and Key Clarifications for Area Chair and Reviewers

Dear Area Chair and Reviewers,

We thank you for your constructive feedback. Below we (1) summarize our contributions, (2) clarify two points raised by Reviewer ooz4, and (3) list the new evidence added during the rebuttal: a corrected evaluation, results on GQA models, additional pruning and hybrid baselines, and expanded ablations.

Key Contribution

S&P is, to our knowledge, the first post-training LLM compression method that jointly learns structured removal (attention heads, FFN channels) and rank reduction of the retained operators under a single budget, without fine-tuning. Joint selection matters most under aggressive compression: at CR 0.4 on LLaMA-1-7B, prune-only collapses (WikiText-2 PPL 2459), rank-only reaches 20.92, and S&P reaches 13.75.

Clarifications

  1. Novelty (ooz4): Our contribution is not differentiable rank selection itself, but the joint rank–structure allocation, overlapping multi-block windows as the selection objective (per-layer optimization degrades PPL from 8.28 to 147.67), and the finding that out-of-order selection concentrates in attention and grows with compression.
  2. Compute cost (ooz4): At matched compute, S&P is on par with AA-SVD including its refinement. When both receive the same light refinement, S&P is best on all metrics at every compression ratio.

Additional Evidence

  • Corrected evaluation (91WV, Z9gG): We fixed an acc_norm/acc reporting error and re-evaluated all methods with plain acc in one pipeline. After re-evaluation, S&P remains the strongest method overall in both perplexity and accuracy across compression ratios 0.8, 0.6, and 0.4. In particular, S&P performs strongly under aggressive compression, achieving a WikiText-2 PPL of 13.75 vs. 29.44 and an average accuracy of 0.37 vs. 0.32 against the strongest baseline at CR 0.4.
  • GQA models (Z9gG): S&P is best among SVD methods at CR 0.6 and 0.4 on LLaMA-3.1-8B and Qwen3-8B.
  • Pruning and hybrid baselines (Z9gG, 91WV): S&P is best or tied-best in 4/6 settings and within 0.01 in the rest, against FLAP, LLM-Pruner, GRASPrune, LoRAP and LoSparse.
  • Ablations (all reviewers): We added rank-only/prune-only/joint ablations, post-compression fine-tuning, double-sided whitening, loss dynamics (Appendix Fig. 3), the full hyperparameter table, and an expanded related-work discussion.

All results support our original conclusions and are reflected in the revised version. We thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.