acceptodds
Under review as a conference paper at ICLR 2027

Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference

Abstract

Scaling laws continue to reward increases in model scale, while the resulting computational and memory demands drive greater reliance on cloud infrastructure for inference. Yet unreliable network connectivity, communication delays, and privacy requirements can make such reliance impractical, motivating local execution on edge devices. Local deployment requires models to fit within limited memory and execute efficiently with the available computational resources. Recent work has shown that learning sparse-plus-low-rank representations enables elastic compression of a single checkpoint to multiple memory budgets. However, fewer parameters do not necessarily translate into faster inference. Indexing overhead and irregular memory access in unstructured sparse components can offset computational savings. To address this critical deployment barrier, we introduce S-SALAAD, a unified framework that jointly learns the model weights and their low-rank and structured sparse components during training. The framework supports a family of hardware-compatible structured sparsity patterns. It couples this structure learning with budget-aware compression and dedicated CPU and GPU kernels that exploit the learned structure for acceleration, enabling efficient inference from a single checkpoint across memory budgets. Experiments on Llama2 and Qwen3 models, covering both pretraining and distillation, show that S-SALAAD advances the empirical Pareto frontier in perplexity, memory, and inference latency across multiple hardware platforms relative to the evaluated baselines. These results show that structured learning can translate elastic compressibility into reduced memory requirements and lower inference latency, enabling practical deployment of large models on edge devices.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Structured Sparsity, Model Compression, Sparse-and-Low-Rank Decomposition, Efficient Inference, Elastic Deployment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Fw91OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes S-SALAAD, a training framework that learns structured sparse-plus-low-rank representations for language-model weights. It extends SALAAD by introducing hardware-friendly sparsity patterns, aiming to turn parameter savings into faster inference. An iterative SVD reduces the cost of learning the low-rank components, while a budget-aware refinement procedure compresses one checkpoint to different model sizes without calibration data. Dedicated CPU and GPU kernels execute the compressed representations. Experiments on Llama pretraining and Qwen3 distillation show improved tradeoffs between model quality, memory usage, and prefill throughput compared with the evaluated baselines.

Critical strengths

1. The paper addresses a practical problem: reducing model size does not necessarily make inference faster. Learning hardware-friendly sparsity during training is well motivated, and the custom kernels help translate the learned structure into measured inference speedups.

2. The implementation goes beyond reporting theoretical parameter savings. The custom kernels demonstrate measured acceleration, and the inference-memory measurements include sparse indices and working buffers. The comparison with exact SVD also supports the use of the cheaper iterative update, which substantially reduces training overhead while preserving compression quality.

Critical weaknesses

1. The inference evaluation mainly measures throughput with relatively large batches. Since the paper targets edge deployment, it is unclear whether the reported gains also hold for batch-size-one inference and token-by-token generation. Quantized baselines are also missing, so the reported gains do not establish an advantage over a common alternative under similar memory constraints.

2. The choice and sensitivity of the target rank ratio and sparsity level are insufficiently investigated. Although the selected values are reported, it is unclear how much tuning is required for a new model or whether small changes substantially affect compression quality. This limits confidence that the reported tradeoffs can be reproduced beyond the evaluated configurations.

Questions for authors

1. Does S-SALAAD retain its speed advantage at batch size one and during autoregressive generation? Could you report inference speed and peak memory in these settings?

2. How does S-SALAAD compare with 8-bit or 4-bit quantization under similar memory budgets? Can the two approaches be combined to further reduce memory usage?

3. How were the target rank ratio and sparsity level selected? Does changing these values substantially affect the accuracy of the compressed models?

4. The evaluation includes a recent model architecture, but remains limited to relatively small models. Does S-SALAAD retain its compression quality and inference advantage on models with 7B or more parameters?

3 comments
AuthorsOpenReview ↗

New Experiments on Autoregressive Decoding, Quantization, and Target Sensitivity

Dear Reviewer Fw91,

Thank you for your careful review and constructive feedback. We respond to each weakness and question below.

1. Batch-size-one autoregressive generation (Weakness 1 and Question 1)

We agree that batch-size-one autoregressive generation is important for edge deployment. We have added experiments on Llama-1B and Qwen3-1.7B across Quadro RTX 6000, RTX 2080 Ti, and AMD EPYC 7662 CPUs, evaluating decode latency and peak memory across parameter budgets.

We highlight two representative results below; the complete results are available in Appendix H, Table 15. Latency and peak memory are shown as dense to S-SALAAD; memory reduction is relative to dense inference.

Model (budget)DeviceLatency (ms/token)Peak memory (GB)Memory reduction
Llama-1B (500M)Quadro RTX 60005.4 → 3.22.76 → 1.0661.6%
Llama-1B (500M)4× AMD EPYC 7662 CPUs280.1 → 143.25.49 → 2.0363.0%

Compared with batched prefill, speedups are generally smaller, consistent with reduced weight reuse and lower Tensor Core utilization during single-token decoding. Relative memory savings are greater because activation storage is smaller. Some GPU configurations at larger budgets are slightly slower than dense while still reducing peak memory.

2. Comparison and combination with quantization (Weakness 1 and Question 2)

S-SALAAD induces compressible structures during pretraining and can complement post-hoc methods. Section 5.1 and Appendix G.1 demonstrate synergies with SliceGPT and SVD-LLM on its L-only checkpoints. Our new results also demonstrate compatibility with quantization.

We added W8A8 and W4A4 experiments on Llama-1B and Qwen3-1.7B (Appendix F, Tables 12–14). W8A8 causes only small perplexity increases, as for dense baselines. The table highlights Qwen3-1.7B at a 1B S-SALAAD budget on Quadro RTX 6000 during batched inference.

ConfigurationPPLPeak mem. (GB)Throughput (ktok/s)
Dense + W8A812.724.1733.50
S-SALAAD, unquantized14.064.2333.23
S-SALAAD + W8A814.083.5139.77

At similar peak memory, dense W8A8 achieves lower perplexity than unquantized S-SALAAD, with similar throughput. Combining W8A8 with S-SALAAD further reduces its peak memory by 17.0% and increases throughput by 19.7%, with only a 0.02 perplexity increase. The full results show additional throughput and memory benefits across all three platforms, supporting the complementarity of learned compressibility and quantization.

3. Target selection and sensitivity (Weakness 2 and Question 3)

Our I-Controller dynamically updates the regularization strengths α\alpha and β\beta to regulate rank and sparsity (Appendix C.1, Eq. 16). These updates depend jointly on the target errors and controller step sizes Δα\Delta \alpha and Δβ\Delta \beta. To simplify tuning, we fix the empirical targets (Γ^\hat{\Gamma},Υ^\hat{\Upsilon}) at (0.15, 0.95) in the main block-sparse experiments and adjust the step sizes. These targets encourage a low rank ratio and high sparsity during training.

Appendix D.3 already explains that structured sparsity uses a larger Δβ\Delta\beta and smaller Δα\Delta\alpha than the unstructured baseline. Table 6 lists the values, which can generally be reused across model scales within each sparsity pattern.

We have added a sensitivity study in Appendix E.5 on Llama-350M with 16×16 and 32×32 blocks, comparing the default target pair (0.15, 0.95) with (0.10, 0.97), (0.20, 0.93), and (0.00, 1.00). The results support robustness to target choices. Figure 23 shows broadly consistent convergence trends across the layers and block sizes, and Table 11 shows small differences in final structural statistics between the default and nearby targets. Figure 24 shows limited changes in perplexity at larger budgets, with pronounced differences mainly under extreme compression.

Therefore, within the tested range, S-SALAAD is relatively insensitive to target choices at larger parameter budgets, and the shared empirical defaults can be reused without fine-grained target tuning. Under extreme compression, target selection has a larger effect on quality, and additional tuning may be beneficial.

4. Extension to models with 7B or more parameters (Question 4)

We agree that validation on larger models would strengthen the evaluation. Our current computational budget limits our experiments to models up to Qwen3-1.7B. We are actively seeking additional computational resources to extend these experiments to 7B and larger models and validate our findings on compression quality and inference efficiency at these scales.

The corresponding textual additions are marked in red in the revised manuscript. Thank you again for helping us improve the evaluation. If these additional results and clarifications address your concerns, we would sincerely appreciate your consideration of a higher rating.

Best,
The Authors

Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!

I thank the authors for their detailed response and the additional experiments. The rebuttal resolves most of my concerns, and I will maintain my positive assessment.

Reviewer_pn45OpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes to jointly learn the model weights and their low-rank and structured sparse component during pre-training. They used ADMM as the optimizer for solving the optimization problem. Results are promising.

Critical strengths

Pruning models so that they can be executed locally is a timely topic. The optimization formulation is principled.

Critical weaknesses

It's unclear what the novelty is. In the pre-LLM era of deep learning, learning structured sparsity using ADMM is exhaustively explored, but the paper didn't discuss how the proposed technique relates to that body of literature or if LLM introduces anything special.

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Clarifying Novelty and Connections to Pre-LLM Literature

Dear Reviewer pn45,

Thank you for raising this concern. We agree that sparsity and low-rank methods have a long history in machine learning. We have therefore added a paragraph to the Related Work section acknowledging these foundations and explicitly stating that our work builds on them.

In addition, following your suggestion, we reviewed relevant work from the pre-LLM era of deep learning on ADMM-based structured learning. We have incorporated the following three studies into the Related Work section to clarify their connections to S-SALAAD. Among these works, StructADMM [1] uses ADMM to induce filter-wise, channel-wise, and shape-wise sparsity, and demonstrates practical hardware acceleration. Whereas StructADMM learns sparse weights under specified pruning constraints, S-SALAAD jointly learns low-rank and structured-sparse components and supports subsequent compression of one checkpoint to different deployment budgets. Ma et al. [2] provide a closer precedent for the representation itself, combining low-rank and column-wise sparse components through ADMM with regularization. Their method reconstructs pretrained CNN layer responses using sample data and subsequently fine-tunes the network. S-SALAAD instead learns the representation jointly with model weights under the language-model pretraining objective, followed by compression without calibration data or any fine-tuning. Guo et al. [3] further demonstrate learning low-rank-plus-sparse representations from scratch, directly using low-rank factors and an unstructured sparse component. S-SALAAD incorporates hardware-compatible structured sparsity into this learning setting and provides budget allocation and refinement for elastic deployment. These additional works do not change the core contribution claims of this paper. The principal similarities and differences are summarized below.

AspectStructADMM [1]Ma et al. [2]Guo et al. [3]S-SALAAD
RepresentationSparse weightsLow-rank + column-wise sparseLow-rank + unstructured sparseLow-rank + structured sparse
Learning settingPretraining under specified pruning constraintsLayer-wise response reconstruction of a pretrained CNNPretraining from scratchPretraining from scratch
Sparsity patternsFilter, channel, shapeColumn-wiseUnstructuredBlock-wise, row-/column-wise, N:M
Compression workflowProjection and masked retrainingData-based reconstruction and fine-tuningEnergy-based pruning, with fine-tuning evaluatedWeight-only allocation and refinement; multiple budgets from one checkpoint without calibration or fine-tuning

Compared with the pre-LLM era, modern LLM training and deployment present the following new challenges. First, high training costs make repeated fine-tuning for different budgets expensive. Using jointly learned low-rank and structured-sparse components, EGAR allocates and refines capacity across projections to adapt one checkpoint to multiple budgets without calibration data or fine-tuning. Second, repeated exact SVD updates of large projection matrices are costly. Warm-started I-SVD reduces per-step training time from 890 to 632 ms on Llama-1B with eight H100 GPUs, compared with 609 ms for vanilla training. Third, fewer parameters do not guarantee faster inference: kernel-specific patterns and granularities call for a unified framework to induce compatible structures during pretraining. S-SALAAD supports block-wise, N:M, and row-/column-wise sparsity within one ADMM formulation through the choice of regularizer and corresponding update. We implement dedicated CPU/GPU kernels for block-sparse deployment and evaluate inference gains across budgets.

The corresponding changes are marked in red in the revised manuscript.

Thank you again for helping us improve the positioning of our work. If these clarifications address your concern, we would sincerely appreciate your consideration of a higher rating.

Best,
The Authors

References

  1. Zhang et al. StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs. Preprint, 2018.
  2. Ma et al. A Unified Approximation Framework for Compressing and Accelerating Deep Neural Networks. ICTAI, 2019.
  3. Guo et al. Compressing by Learning in a Low-Rank and Sparse Decomposition Form. IEEE Access, 2019.

Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!

Thanks for your response and for adding the new results. Unfortunately, the three papers you compared against are not representative of the most seminal work before. I would have liked a more comprehensive comparison with works like https://arxiv.org/pdf/1804.03294.

Reviewer_qMJQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the challenges of efficiently deploying large models on edge devices. Although learning sparse-plus-low-rank representations enables elastic compression of a single checkpoint to multiple memory budgets, indexing overhead and irregular memory access in unstructured sparse components can offset computational savings. To address this barrier, the authors proposed S-SALAAD, a plug-and-play framework that learns structured sparse-and low-rank representations during pretraining for elastic compression and efficient edge inference. Experiments on Llama2 and Qwen3 models show that S-SALAAD advances the empirical Pareto frontier in perplexity, memory, and inference latency across multiple hardware platforms.

Critical strengths
  1. The research question is well-defined, has practical significance, and emphasizes actual runtime efficiency. S-SALAAD is a unified framework that jointly learns the model weights and their low-rank and structured sparse components during training
  2. Each checkpoint supporting multiple compression budgets is of practical value. Compared to methods that fix the model capacity during training, S-SALAAD is well-suited for the deployment environments with varying storage and computational resources.
  3. The work is comprehensive and provides targeted validation of the proposed modules, such as I-SVD, EGAR and so forth.
Critical weaknesses
  1. Table 1 compares only Vanilla and S-SALAAD and lacks comparisons with other state-of-the-art relevant baselines.
  2. The paper presented the trend about larger models can accommodate coarser, more efficient blocks while retaining elastic compressibility derived from increasing from 16×16 on Llama-350M to 64×64 on Llama-1B. There is a lack of results from mainstream large models at other scales, as well as verification of the “breakpoint” for extremely small models.
  3. In Appendix D.3, the target effective-rank ratio is fixed at 0.15 and target sparsity is fixed at 0.95, lacking a sensitivity analysis in other settings.
Questions for authors

S-SALAAD is a practical and well-motivated framework for the efficient deployment of LLMs. It pointed out that fewer parameters do not necessarily translate into faster inference. To address this critical deployment barrier, S-SALAAD supports a family of hardware-compatible structured sparsity patterns, coupling this structure learning with budget-aware compression and dedicated CPU and GPU kernels. The framework has been thoroughly evaluated across various models and hardware platforms. Given the core contributions, the current recommendation decision is “Weak Accept”. However, addressing the weaknesses mentioned above in the rebuttal is crucial to the final decision.

4 comments
AuthorsOpenReview ↗

New Experiments: Downstream Comparisons, Block Granularity, and Target Sensitivity

Dear Reviewer qMJQ,

Thank you for your careful review and positive assessment of the practical value of S-SALAAD. We respond to your three concerns below.

1. Downstream comparisons with other baselines (Weakness 1)

We have expanded Table 1 to include Wanda, SparseGPT, SliceGPT, and SVD-LLM on Llama-1B and Qwen3-1.7B across six zero-shot tasks. The post-training baselines compress the corresponding vanilla checkpoints, and we report each method's achieved parameter count.

Compared with Vanilla, S-SALAAD retains useful downstream capability as the parameter budget tightens, supporting elastic deployment from a single checkpoint without a collapse in task accuracy.

The added baselines further show a favorable accuracy-parameter trade-off. On Llama-1B, S-SALAAD at 700M parameters outperforms Wanda and SparseGPT (both 748M) on six and five tasks, respectively. On Qwen3-1.7B, its 800M variant outperforms every reported post-training baseline configuration (973-1592M) on all six tasks. Thus, S-SALAAD achieves higher accuracy on most tasks with fewer parameters in these comparisons. We have added the corresponding discussion in Section 5.1.

We highlight representative Qwen3-1.7B results below (zero-shot accuracy in %, higher is better).

MethodParams. (M)PIQASciQWinoG.HellaSwagARC-EOBQA
Vanilla167873.2992.2056.6745.7575.7629.80
S-SALAAD80069.7589.0056.3540.2067.5526.20
Wanda (2:4)97365.1880.1052.9634.4155.9821.00
SparseGPT (2:4)97365.6784.8053.7534.1461.0323.60
SliceGPT (80%)159265.6186.1053.0435.7262.4221.20
SVD-LLM (80%)139668.8285.3056.0437.9764.5225.20

The complete downstream comparison for both models is provided in Table 1 of the revised manuscript.

2. Model scale and block granularity (Weakness 2)

We have added a Llama-60M experiment in Figure 6(a) to examine block granularity at an extremely small model scale. At approximately 45M compressed parameters, 4×4 blocks achieve a perplexity of 52.03, close to 52.30 for L-only, whereas 8×8 blocks yield 66.72. This comparison shows that finer blocks are needed at this scale to preserve the L-only compression trade-off.

Together with the existing results, the granularity that approximately preserves this trade-off increases from 4×4 on Llama-60M to 16×16 on Llama-350M and 64×64 on Llama-1B. These results extend the empirical evidence that larger models can accommodate coarser blocks. The 60M experiment characterizes the smaller-scale regime.

We agree that experiments on larger models are needed to verify whether they can accommodate even coarser blocks while retaining elastic compressibility. These experiments are currently constrained by our computational budget, and we are actively seeking additional resources to extend the block-granularity study to larger model scales.

3. Target selection and sensitivity (Weakness 3)

Our I-Controller dynamically updates the regularization strengths α\alpha and β\beta to regulate rank and sparsity (Appendix C.1, Eq. 16). These updates depend jointly on the target errors and controller step sizes Δα\Delta \alpha and Δβ\Delta \beta. To simplify tuning, we fix the empirical targets (Γ^\hat{\Gamma},Υ^\hat{\Upsilon}) at (0.15, 0.95) in the main block-sparse experiments and adjust the step sizes. These targets encourage a low rank ratio and high sparsity during training.

Appendix D.3 already explains that structured sparsity uses a larger Δβ\Delta\beta and smaller Δα\Delta\alpha than the unstructured baseline. Table 6 lists the values, which can generally be reused across model scales within each sparsity pattern.

We have added a sensitivity study in Appendix E.5 on Llama-350M with 16×16 and 32×32 blocks, comparing the default target pair (0.15, 0.95) with (0.10, 0.97), (0.20, 0.93), and (0.00, 1.00). The results support robustness to target choices. Figure 23 shows broadly consistent convergence trends across the layers and block sizes, and Table 11 shows small differences in final structural statistics between the default and nearby targets. Figure 24 shows limited changes in perplexity at larger budgets, with pronounced differences mainly under extreme compression.

Therefore, within the tested range, S-SALAAD is relatively insensitive to target choices at larger parameter budgets, and the shared empirical defaults can be reused without fine-grained target tuning. Under extreme compression, target selection has a larger effect on quality, and additional tuning may be beneficial.

The corresponding textual additions are marked in red in the revised manuscript. Thank you again for helping us improve the evaluation. If these additional results and clarifications address your concerns, we would sincerely appreciate your consideration of a higher rating.

Best,
The Authors

Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!

Thank you again for reviewing our work. We would appreciate your feedback on whether our response and the additional experiments on downstream performance, block granularity, and target sensitivity address your concerns. We would be happy to clarify any remaining points.

I thank the authors for their response. Although the Weakness 2 has been partially resolved, I still maintain my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.