- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MORS, which is a sparse dictionary learning framework with second-order information. It fills the gap where sparse dictionary methods have lacked loss-aware objectives. Experiments across diverse model families show that MORS consistently outperforms SVD-based baselines at moderate compression ratios, while remaining compatible with quantization.
Critical strengths
MORS is designed to solve the problem that SVD methods can only be combined from the same shared subspace, so it is based on sparse dictionary representation. Specifically, it introduce loss aware into dictionary learning, using second-order approximation to obtain loss information, where second-order information is constructed using Kronecker Factored Approximate Curve (KFAC) approximation to construct empirical Fisher instead of Hessian.
- This paper has a good writing with reproducibility.
- It Introduces loss aware compression for sparse dictionaries, filling the gap.
- This method is more flexible in terms of representation ability.
- The experiment is conducted on multiple dimensions.
Critical weaknesses
- MORS introduce loss aware into dictionary learning, using second-order approximation to obtain loss information, where second-order information is constructed using Kronecker Factored Approximate Curve (KFAC) approximation to construct empirical Fisher instead of Hessian. Is there a way to measure the error caused by these approximations.
- The implementation of the method uses various tools such as KFAC approximate empirical Fisher, Cholesky whitening, orthogonal constraints, Procrustes update, etc. These seem like an engineering pile up of many methods, with a bit rigid and compositional contribution, losing the feeling of a loss perception mechanism tailored for the sparse dictionary structure itself. Therefore, the question is whether there is a more natural loss aware parameterization method for the structure of sparse dictionaries that is different from quadratic loss and low rank loss? What are the essential differences between the detailed steps of CoSpaDi or COMPOT in practical operation.
- This paper claims that it introduces a computationally efficient loss-aware sparse dictionary compression strategy. Since this method require calculating KFAC, Cholesky decomposition, 20 rounds of alternating optimization. Is the calculation and engineering cost of the compression stage itself high.
- Will an increase in compression rate lead to a decrease in advantage. When will a projection wise local loss approximation under severe disturbances occur.
- The acceleration ratio in Table 4 ranges from 1.12 × to 1.20 ×. Is this result a standalone improvement of the algorithm or an advantage brought by the collaborative design of the system hardware. If we switch to a different hardware, will the acceleration advantage still exist?
- The results come from LEMS, and the comparison of results does not come from the reproduction of results. So, is the experimental setup exactly the same, and is there any inconsistency caused by misalignment.
- Suggest providing a compression ratio curve to demonstrate robustness.
Questions for authors
- Is there a way to measure the error caused by these approximations.
- The question is whether there is a more natural loss aware parameterization method for the structure of sparse dictionaries that is different from quadratic loss and low rank loss?
- What are the essential differences between the detailed steps of CoSpaDi or COMPOT in practical operation.
- Is the calculation and engineering cost of the compression stage itself high.
- Will an increase in compression rate lead to a decrease in advantage.
- When will a projection wise local loss approximation under severe disturbances occur.
- Is this result a standalone improvement of the algorithm or an advantage brought by the collaborative design of the system hardware. If we switch to a different hardware, will the acceleration advantage still exist?
- Is the experimental setup exactly the same, and is there any inconsistency caused by misalignment.
- Suggest providing a compression ratio curve to demonstrate robustness.
Dear Reviewer 9aq4,
We wanted to kindly follow up and hope that our additional experiments and clarifications, particularly regarding curvature approximations, compression costs, higher compression ratios, and the baseline comparisons and calibration protocols, have addressed your concerns.
We would be grateful to hear if any questions remain. Thank you again for your time and constructive feedback!