Taking Low-Rank LLM Compression a Step Further: A Global Perspective with Fused Inference

Low-rank decomposition has emerged as a promising paradigm for compressing Large Language Models (LLMs). However, current methods universally rely on greedy, layer-wise reconstruction objective. This approach induces cascading error propagation, where localized truncation errors amplify across deep architectures. To mitigate this limitation, we formulate the low-rank decomposition problem from a global perspective using the Alternating Direction Method of Multipliers (ADMM). By decoupling the non-convex rank constraints from the holistic language modeling objective, our method preserves model quality. Systemically, the standard sequential execution of factorized matrices often fails to yield practical wall-clock speedups due to kernel launch overheads and activation memory traffic. We bridge the gap by developing a fused GPU operator. Extensive experiments across diverse architectures demonstrate that our algorithm-agnostic method significantly benefits existing methods. For example, at a 30% compression ratio on Qwen-3-8B, our method achieves a perplexity of 10.05—nearly indistinguishable from the uncompressed model's 9.85—while further improving zero-shot reasoning accuracy. Furthermore, by integrating our custom fused operator into the widely adopted inference ecosystems, we translate theoretical FLOPs reduction into practical inference speedup.