MORS: Metric-Orthogonal Sparse Factorization for Loss-Aware LLM Compression

Sparse dictionary learning has shown advantages over SVD-based compression by providing a more flexible weight representation, yet loss-aware formulations have remained largely confined to low-rank methods. We introduce **MORS**, a loss-aware sparse dictionary learning framework that incorporates second-order information from a Kronecker-factored empirical Fisher approximation into both the factorization objective and compression allocation. MORS optimizes the factorization directly in the original weight domain while constraining the dictionary to be orthogonal under the activation-side curvature metric. The general formulation supports full output-side curvature, while a diagonal surrogate decouples sparse support selection and admits efficient closed-form updates. Across diverse model families and compression ratios, MORS consistently outperforms strong SVD-based baselines, establishing loss-aware sparse dictionary compression as a competitive alternative to low-rank factorization.