acceptodds
Under review as a conference paper at ICLR 2027

MORS: Metric-Orthogonal Sparse Factorization for Loss-Aware LLM Compression

Abstract

Sparse dictionary learning has shown advantages over SVD-based compression by providing a more flexible weight representation, yet loss-aware formulations have remained largely confined to low-rank methods. We introduce **MORS**, a loss-aware sparse dictionary learning framework that incorporates second-order information from a Kronecker-factored empirical Fisher approximation into both the factorization objective and compression allocation. MORS optimizes the factorization directly in the original weight domain while constraining the dictionary to be orthogonal under the activation-side curvature metric. The general formulation supports full output-side curvature, while a diagonal surrogate decouples sparse support selection and admits efficient closed-form updates. Across diverse model families and compression ratios, MORS consistently outperforms strong SVD-based baselines, establishing loss-aware sparse dictionary compression as a competitive alternative to low-rank factorization.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
loss-aware model compression, sparse dictionary learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9aq4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MORS, which is a sparse dictionary learning framework with second-order information. It fills the gap where sparse dictionary methods have lacked loss-aware objectives. Experiments across diverse model families show that MORS consistently outperforms SVD-based baselines at moderate compression ratios, while remaining compatible with quantization.

Critical strengths

MORS is designed to solve the problem that SVD methods can only be combined from the same shared subspace, so it is based on sparse dictionary representation. Specifically, it introduce loss aware into dictionary learning, using second-order approximation to obtain loss information, where second-order information is constructed using Kronecker Factored Approximate Curve (KFAC) approximation to construct empirical Fisher instead of Hessian.

  • This paper has a good writing with reproducibility.
  • It Introduces loss aware compression for sparse dictionaries, filling the gap.
  • This method is more flexible in terms of representation ability.
  • The experiment is conducted on multiple dimensions.
Critical weaknesses
  • MORS introduce loss aware into dictionary learning, using second-order approximation to obtain loss information, where second-order information is constructed using Kronecker Factored Approximate Curve (KFAC) approximation to construct empirical Fisher instead of Hessian. Is there a way to measure the error caused by these approximations.
  • The implementation of the method uses various tools such as KFAC approximate empirical Fisher, Cholesky whitening, orthogonal constraints, Procrustes update, etc. These seem like an engineering pile up of many methods, with a bit rigid and compositional contribution, losing the feeling of a loss perception mechanism tailored for the sparse dictionary structure itself. Therefore, the question is whether there is a more natural loss aware parameterization method for the structure of sparse dictionaries that is different from quadratic loss and low rank loss? What are the essential differences between the detailed steps of CoSpaDi or COMPOT in practical operation.
  • This paper claims that it introduces a computationally efficient loss-aware sparse dictionary compression strategy. Since this method require calculating KFAC, Cholesky decomposition, 20 rounds of alternating optimization. Is the calculation and engineering cost of the compression stage itself high.
  • Will an increase in compression rate lead to a decrease in advantage. When will a projection wise local loss approximation under severe disturbances occur.
  • The acceleration ratio in Table 4 ranges from 1.12 × to 1.20 ×. Is this result a standalone improvement of the algorithm or an advantage brought by the collaborative design of the system hardware. If we switch to a different hardware, will the acceleration advantage still exist?
  • The results come from LEMS, and the comparison of results does not come from the reproduction of results. So, is the experimental setup exactly the same, and is there any inconsistency caused by misalignment.
  • Suggest providing a compression ratio curve to demonstrate robustness.
Questions for authors
  • Is there a way to measure the error caused by these approximations.
  • The question is whether there is a more natural loss aware parameterization method for the structure of sparse dictionaries that is different from quadratic loss and low rank loss?
  • What are the essential differences between the detailed steps of CoSpaDi or COMPOT in practical operation.
  • Is the calculation and engineering cost of the compression stage itself high.
  • Will an increase in compression rate lead to a decrease in advantage.
  • When will a projection wise local loss approximation under severe disturbances occur.
  • Is this result a standalone improvement of the algorithm or an advantage brought by the collaborative design of the system hardware. If we switch to a different hardware, will the acceleration advantage still exist?
  • Is the experimental setup exactly the same, and is there any inconsistency caused by misalignment.
  • Suggest providing a compression ratio curve to demonstrate robustness.
1 comment

Dear Reviewer 9aq4,

We wanted to kindly follow up and hope that our additional experiments and clarifications, particularly regarding curvature approximations, compression costs, higher compression ratios, and the baseline comparisons and calibration protocols, have addressed your concerns.

We would be grateful to hear if any questions remain. Thank you again for your time and constructive feedback!

Reviewer_LP5GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MORS, a method for compressing post-training model using sparse dictionary learning. Unlike previous works which focus on reproducing the pretrained weights or layer activations, this work focuses on preserving model loss, thus called loss-aware. To achieve this, it studys the approximated curvature of the loss wrt weights, and then approximate the weights using an orthogonal tall dictionary and sparse coefficients. The sparse dictionary learning process follows an alternating optimization manner with convergence guarantee on the sequence of cost function value. To facilitate the optimization process, only the diagonal terms of the output side curvature matrix are preserved. Furthermore, to move forward from uniform compression ratios to a more flexible heterogeneous scheme, this paper introduces the dynamic compression allocation and a cheap scoring to evaluate each candidate without exactly computing them. Experiments demonstrate improvements over existing compression methods across several language models, with additional evaluations on speech and vision-language models, quantization, and inference speed.

Critical strengths
  1. Compression improvements across multiple models. The proposed pipeline improves over baselines across several language models and compression budgets.

  2. An efficient optimization procedure. Metric orthogonality enables cheap hard thresholding and Procrustes updates, with a descent guarantee for the diagonal surrogate actually optimized. This computational cheapness matters because an accurate but expensive factorization procedure would be less useful at applications at this scale.

Critical weaknesses

My main concern is about the scoring. The score thresholds coefficients after a full output curvature transformation in (20). Mapping them back gives (S=\widetilde ZL_g^{-1}), which generally does not preserve column sparsity when (L_g^{-1}) is non-diagonal (due to the usage of a full output curvature matrix). The final deployable factorization is, however, obtained using a diagonal right curvature matrix. Although the paper explicitly treats the score as a surrogate and table 5 demonstrates downstream benefits from the allocator, i am concerned that it does not directly assess whether this score reliably ranks the losses achievable by the final compressor.

Questions for authors
  1. how well does the fast score rank the final factorization’s full curvature error and the actual task-loss increase?
  2. please clarify the MORS/MORS^\dagger labels in table 17 and the corresponding ASR description in sec. 4.2, which cites the uniform variant’s results as MORS results at 40% compression. If the table labels are correct, why does dynamic allocation underperform uniform MORS at 30% and 40% compression?
1 comment
AuthorsOpenReview ↗

Response to Reviewer LP5G

We thank the reviewer for the thoughtful follow-up and constructive feedback. We appreciate the opportunity to clarify the strengths and limitations of our allocation strategy. We will incorporate the additional experimental results, including the full-versus-diagonal curvature comparison and its implications, into the revised manuscript, along with the corrected ASR labeling.

Thank you again for your valuable suggestions, which have helped us improve the clarity and completeness of our work.

Reviewer_HF3vOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MORS, a post-training compression method that represents each projection matrix as a dense dictionary and column-sparse coefficients. Activation and output-gradient statistics define a loss-aware reconstruction objective, with the dictionary constrained to be orthogonal under the activation metric. The local solver uses a diagonal output metric and alternates hard-thresholding and Procrustes updates. A separate allocator uses the full output metric to score one-step approximations and distribute the storage budget. Experiments cover several language models and compression ratios, with additional evaluations of quantization, inference latency, speech recognition, and vision-language tasks.

Critical strengths
  1. The main results are consistent across models and compression ratios. In Table 1, MORS improves average accuracy over COMPOT in all 12 combinations of model and compression ratio, with gains of 2.39 to 6.86 percentage points. Each method uses its own dynamic allocator in this comparison. WikiText perplexity improves in all 12 settings and LAMBADA perplexity in 11. The authors report reproducing COMPOT under the same evaluation protocol, and the comparisons include both Llama3-8B and Qwen3-8B. These results are the main reason for my positive assessment.

  2. The paper supports the method with valid local updates and useful ablations. The hard-thresholding and metric-orthogonal Procrustes steps follow from the diagonal output metric, and Appendix C.4 states the convergence result with appropriate qualifications. Table 1 compares local compressors under uniform allocation. Tables 5 and 16 then hold the MORS compressor fixed and compare allocation strategies on Qwen3-0.6B at three compression ratios. MORS achieves the best average accuracy and LAMBADA perplexity in these comparisons, while LEMS performs better on WikiText perplexity.

Critical weaknesses
  1. The methodological advance is incremental. Under the diagonal output metric, the local problem can be rewritten as orthogonal dictionary learning on a reweighted matrix, so the update structure closely follows COMPOT. I see the main added value in applying loss information to this sparse representation and in the allocation score. The full-method results support this combination, but do not show as clearly which change drives the gains on the 8B models. Under uniform allocation, average accuracy is mixed on these models. The comparisons against competing allocators in Tables 5 and 16 hold the local MORS compressor fixed, but are limited to Qwen3-0.6B. They also do not isolate the effect of sparse-candidate scoring from the loss metric and other allocation choices.

  2. A smaller concern is the omission of CLEAR (Wei and Lee, 2026) from the related-work discussion. CLEAR also combines activation-whitened sparse dictionaries with global budget allocation, although its allocation uses a spectral-energy proxy. A short comparison would help clarify what MORS adds through its sparse-candidate loss score. Reference: Jingjiang Wei and Ook Lee. CLEAR: Water-Filling Rank Allocation with Sparse Dictionary Representations for Training-Free LLM Compression. Electronics, 15(15):3395, 2026. DOI: 10.3390/electronics15153395.

Questions for authors
  1. With the local MORS compressor fixed, how do the MORS, COMPOT, and LEMS allocators compare on one of the 8B models? A comparison at one compression ratio already used in the paper, with the same calibration data and total compression budget, would help establish whether the controlled results in Tables 5 and 16 extend to a larger model.

  2. Can the existing results, or a small controlled comparison, clarify the benefit of scoring sparse candidates rather than using a spectral proxy? Keeping the loss metric and the other allocation choices fixed would help show what the proposed score adds.

1 comment

Dear Reviewer HF3v,

Following your suggestions, we have added controlled experiments on the 8B models, investigated sparse-candidate versus spectral allocation, and clarified the differences from CLEAR.

We hope these additional results help resolve your concerns regarding the methodological contribution of MORS. We would be glad to hear your thoughts.

Thank you again for your constructive feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.