UDA-SVD: Upstream- and Downstream-Aware Low-Rank Approximation and Adaptive Rank Allocation for LLMCompression

The rapid growth of large language models (LLMs) has intensified the demand for efficient compression, and low-rank factorization provides a simple and broadly compatible solution. However, in sequential low-rank compression, the effectiveness of compressing each module is shaped by two factors: 182 upstream compression perturbs the activations received by the current module; and 183 approximation errors introduced at the current module propagate downstream and can have unequal effects on the final prediction. Existing methods typically model these effects in isolation. Conventional layer-wise approaches account for neither explicitly, whereas cumulative error-compensation methods primarily address upstream deviations and loss-aware methods focus on downstream sensitivity. Moreover, uniform rank allocation ignores differences in rank sensitivity across modules, leading to inefficient capacity allocation. We propose UDA-SVD, an upstream- and downstream-aware framework for low-rank LLM compression. UDA-SVD captures both activation deviations accumulated through upstream compression and the sensitivity of the language-modeling objective to compression errors at the current module. By integrating these two signals into a unified objective, UDA-SVD provides a common optimization basis for module-wise low-rank approximation and adaptive rank allocation under a fixed parameter budget. Across multiple compression ratios and model architectures, UDA-SVD consistently achieves the highest downstream accuracy among the compared SVD-based methods while substantially reducing language-modeling perplexity, including a nearly 40% reduction on WikiText-2. These results demonstrate the effectiveness of combining upstream drift, downstream sensitivity, and adaptive rank allocation, highlighting UDA-SVD as a promising framework for effective low-rank LLM compression.