- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ASQ, an ADMM-based method for jointly sparsifying and quantizing LLM weights. The dense weights are trained with Adam on the language modeling loss, while an auxiliary variable is projected onto a joint sparse and quantized set. For a fixed quantization grid, this projection has a simple closed-form solution, so the sparse support and the quantized values are decided together. The authors further add quantization scale search and a Fisher-based weighting, and they provide a convergence analysis. Experiments on LLaMA and Qwen models show clear gains over existing joint compression methods at high compression rates; the gap is especially large at 90% sparsity with 4-bit weights. The authors also report up to a 3.2x decode speedup over BF16 for a 2:4 sparse INT4 model.
Critical strengths
-
The formulation is clean, and handling all the discrete decisions in the projection step is a neat idea. The paper is also clear and easy to follow.
-
The results at high compression are strong and consistent across models. The ablation in Table 1 is convincing as well. Replacing the joint projection with a sequential prune-then-quantize step almost doubles WikiText-2 perplexity at 40x, which shows that the joint projection itself makes a real difference.
Critical weaknesses
-
My main concern is the training budget. ASQ fine-tunes the model on C4 data, while the baselines only use 128 calibration sequences. The ablation shows that the joint projection helps within ASQ, but Table 1 also suggests that much of the gain over the baselines comes from the training itself. Even with all three components removed, ASQ still reaches a perplexity of 83.6 on Qwen3-8B at 40x, versus 388 for the best baseline. A comparison with one-shot methods followed by fine-tuning on the same data, or with ADMM-based sparsification followed by GPTQ, would make the claim much more convincing. It would also help to report the training cost.
-
Some claims are stronger than what the experiments support. The abstract says the gap widens in low-bit regimes, but all experiments use 4-bit quantization. The conclusion claims superior trade-offs in hardware-oriented settings. Under 2:4 + INT4, however, ASQ is only marginally better than the best baselines (Table 10), and at 8x it is even behind the best baseline on all four models. The compression rate also ignores the overhead of masks and scales. For instance, the 2:4 + INT4 model is 8x by this formula but only 4.78x in measured memory. I would suggest reporting bits per weight and focusing the claims on the higher compression rates, where the gains are clear. Even there, the models remain far from dense quality, so saying that they retain high fidelity seems too strong.
-
The theoretical part is weaker than presented. The convergence theorem requires lambda > 5(beta + eps), while the experiments use lambda between 1e-4 and 1e-2. It also does not cover scale search or Fisher weighting, so it is better seen as an analysis of an idealized algorithm. In addition, ADMM-based joint pruning and quantization has been explored before for DNNs, and recent ADMM-based LLM sparsification work is only briefly cited. The paper should discuss how ASQ differs from these.
Minor: the choice of hyperparameters is not explained, and the largest model is 14B. ASQ is also trained on C4 and evaluated on C4 perplexity, which may favor it. The reported speedups come from the 2:4 + INT4 format rather than from ASQ itself.
Questions for authors
- What is the training cost of ASQ in terms of tokens, GPU hours and memory?
- How does ASQ compare with one-shot methods followed by fine-tuning on the same data?
- Does the advantage still hold at lower bit-widths?
- How were the hyperparameters tuned?
4 comments
[W3] Convergence Analysis and Distinction from Prior ADMM Methods
Regarding the concerns about the theoretical analysis, we would first like to clarify the role of our theoretical convergence guarantee. In general, convergence analyses of complex nonconvex optimization algorithms do not aim to provide unconditional guarantees covering every component of the practical implementation. Rather, their significance lies in rigorously characterizing the properties of the core optimization mechanism under appropriate assumptions.
From this perspective, ASQ's theoretical result goes beyond merely demonstrating a convergence trend. Based on exact joint projection onto a nonconvex and discrete sparse–quantized constraint set, we prove that the residual between the continuous and compressed variables converges to zero. Furthermore, we establish finite-time identification, ensuring that the sparse support and quantized values stabilize after a finite number of changes. Finally, we establish convergence to a -stationary point and derive an explicit upper bound on the number of changes in the discrete representation.
Meanwhile, the theoretical condition cannot be assessed solely from the absolute value of , as it depends on the smoothness of the objective function and the approximation accuracy of the x-update. Therefore, it is difficult to determine whether this condition is satisfied based only on the values of used in the experiments. However, since we have not verified this sufficient condition under our current experimental settings, we clearly distinguish the scope of the theoretical guarantee as applying to the core formulation.
We also explicitly discuss the scope of the theoretical guarantee in the Discussion section, where we clarify that the convergence guarantee applies to the core exact-projection formulation and provide separate empirical evidence for the optimization dynamics of practical ASQ (Figure 3). Therefore, the theoretical guarantee and empirical validation complement each other in supporting the optimization properties of ASQ.
Additionally, we thank the reviewer for suggesting that we discuss the closely related work on DNNs. Although [3] similarly uses ADMM as a tool for joint compression, it differs from our work in two important aspects: (i) its primary objective is to automatically determine how sparsity and bitwidth should be allocated across layers, and (ii) it handles sparsity and quantization constraints through separate auxiliary variables and projection steps. In contrast, our work is motivated by the need to account for the interaction between sparsification and quantization constraints, and accordingly formulates the problem and proposes a joint projection. We acknowledge that this work should have been discussed in our paper and will clarify these differences in the paper.
[Q4 & Minor] Hyperparameter Selection, Scalability, and Evaluation Protocol
Finally, we would like to address several minor issues raised by the reviewer.
- Choice of hyperparameters: We selected the learning rate () and ADMM penalty parameter () based on validation. We will clarify the search space and selection procedure in Appendix C.
- Scalability: We conducted additional experiments on Qwen2.5-32B, and present the main results below. The following table reports WikiText-2 perplexity (↓) on Qwen2.5-32B under INT4 quantization.
| Method | 70% + INT4 | 80% + INT4 | 90% + INT4 |
|---|---|---|---|
| SparseGPTQ | 12.41 | 30.30 | 654.69 |
| SLiM | 12.06 | 26.10 | 831.33 |
| ASQ | 10.49 | 14.43 | 32.97 |
These results show that our method also achieves strong performance on a larger model. We will include the full results in the paper.
- C4: Using C4 for optimization is a common practice in LLM compression. Importantly, C4 perplexity is evaluated on a held-out set, separate from the optimization data. We also evaluate ASQ on WikiText-2 and various zero-shot benchmarks, where the performance improvements remain consistent.
Finally, we sincerely thank the reviewer again for their careful assessment of our work and constructive suggestions. We hope that our responses and additional results help address the concerns raised.
References
[1] Lee et al., "The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM," ICLR, 2026.
[2] Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers," ICLR, 2023.
[3] Yang et al., "Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-Based Approach," CVPR, 2020.
Thank you for the additional experiments. The budget-matched comparison with ELSA followed by GPTQ addresses my main concern, especially at 90% sparsity. The reported training cost and the new INT3 and 32B results also strengthen the evaluation. I am raising my rating from 2 to 3.
We sincerely thank the reviewer for carefully considering our responses and additional experiments, and for raising their rating. We are pleased that the budget-matched comparison, training cost analysis, and additional INT3 and 32B results have helped address your concerns. Your thoughtful feedback and constructive suggestions have been invaluable in strengthening our experimental evaluation and improving the overall quality of our work. We will incorporate these additional results into the revised manuscript. Thank you again for your time and constructive review.
We sincerely thank the reviewer for their careful assessment and constructive suggestions. We particularly appreciate their recognition of the joint projection, informative ablations, and strong results under aggressive compression. Below, we address the reviewer's comments and suggestions with additional experiments and clarifications.
[W1 & Q1 & Q2] Training Budget Fairness and Computational Cost
First, we would like to address the reviewer's main concern regarding the training budget. The ablation study that the reviewer positively evaluated demonstrates the independent contribution of joint projection within the same optimization framework. We would also like to emphasize that ASQ is not a method that applies joint projection in isolation, but is designed to integrate task-aware iterative optimization and joint sparse–quantized projection into a unified ADMM framework.
However, we agree that the comparison with existing baselines did not fully isolate the effect of additional optimization on performance. To investigate this concern, we conducted additional experiments with the reviewer-suggested sequential baseline, ADMM-based LLM sparsification (ELSA [1]) followed by GPTQ [2], using a training budget matched to ASQ.
The following table reports WikiText-2 perplexity (↓) and average zero-shot accuracy (↑) on Qwen3-8B-Base. Values are presented as PPL (Zero-shot Avg.).
Even under the same training budget, ASQ achieves lower PPL and higher zero-shot accuracy across all settings, with the PPL gap becoming larger at higher compression ratios. These results indicate that ASQ's advantages cannot be explained solely by the additional training budget. We will incorporate these results into the Experiments section.
We also report ASQ's training configuration and computational cost on Qwen3-8B-Base below.
One point we would like to emphasize is that compression incurs a cost before deployment. In other words, when a compressed model is used repeatedly, this cost can be amortized over an extended period. ASQ achieves better performance than existing methods at high compression ratios and also outperforms the budget-matched baseline. Therefore, we believe that its efficiency should be evaluated in conjunction with the performance achieved.
[W2 & Q3] Clarification of Claims
Next, we acknowledge that some of our claims were somewhat stronger than what the presented experimental results support. We appreciate the reviewer's careful examination of our results and their suggestions for clarifying the scope of our claims.
First, we conducted additional experiments at 3-bit precision, which is lower than the bitwidth considered in our original experiments.
The following table presents WikiText-2 perplexity (↓) on Qwen3-8B-Base under 3-bit quantization.
As the results show, ASQ also achieves strong performance at a lower bitwidth. We present the main results here and will include the results for all baselines in the paper.
Additionally, we agree that the performance difference in the hardware-compatible INT4+2:4 setting is relatively small, and that the theoretical compression ratio does not fully reflect actual storage costs. Accordingly, we will define effective bits/weight accounting for metadata overhead and incorporate the corresponding storage costs of ASQ. We will also refine our claims that go beyond the experimental evidence.