- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SPRBM, a pruning method based on proposed task and complexity gradient agreement. It is evaluated on the trainable LoRA parameter reduction of DistilGPT2.
Critical strengths
- The intuition of using task and complexity gradient agreement as a parameter-wise signal may be interesting.
- The paper is honest about the limitations.
Critical weaknesses
- The motivation, i.e., gradient alignment is used to design a pruning criterion, can not be substantiated. First, gradient alignment alone is not strongly correlated with measuring the redundancy of the pruned parameters. If a parameter is important to the task, and its complexity gradient direction happens to be consistent with the task gradient, how should this be handled? Moreover, no theoretical basis is given to explain why the proposed criterion is more suitable than importance or magnitude criteria for determining pruning. Therefore, the claim stated, "Can parameter-wise pruning pressure be derived from task–complexity gradient agreement", is not substantiated.
- Theoretically, in the First-Order Interpretation section, only a first-order Taylor expansion is used for the analysis. How are the perturbations caused by higher-order terms measured? In addition, the analysis is parameter-wise, ignoring the interactions between parameters. Moreover, it lacks convergence, optimality, or approximation guarantee for the overall procedure. At last, the designs of hyper-parameters and mask are heuristic, with no theory. Overall, it is only a local analysis that can provide some intuitive guidance, but the analysis is not rigorous enough.
- Please explain the motivation for the threshold design. Please explain why the sigmoid function is used to transform the parameter-specific coefficient. In addition, numerous hyper-parameters such as the global scale, alignment sensitivity, and coefficient are introduced, which increases the difficulty of tuning. Moreover, there is no experiment on how these hyper-parameters affect the results.
- For the experiments, are the baselines selected the state-of-the-art methods among magnitude, regularization-based, random, and SNIP-style criteria? Why not compare with Movement Pruning or PLATON? If the method proposed is a new criterion derived from task and complexity gradient agreement, then it needs to be compared with pruning methods of different styles, and with relatively SOTA specific methods in every criteria.
- The model scale is extremely small, so the generalizability of the conclusions is limited. Whether it can be transferred to models that truly require pruning is completely unknown. In addition, the experiment is conducted on LoRA, which is itself a parameter-efficient fine-tuning method; therefore, the marginal benefit brought by sparsification is extremely small. Furthermore, the actual benefit is only a reduction in adapter storage, with no inference acceleration. In terms of results, the performance improvement is also not obvious.
- The ablation claimed in the paper cannot prove that this information comes from the gradient alignment mechanism itself. The information provided by the ablation experiments is very limited and not comprehensive.
Questions for authors
- If an important parameter happens to have aligned gradients, it still gets pruned. How to deal with it?
- Your analysis is local and first-order. Why does that justify a discrete pruning decision?
- What motivates the sigmoid mapping, hyper-parameter designs and mask designs?
- You have many hyperparameters. How sensitive are results to them?
- Why no comparison with strong baselines?
- Your models are tiny. Will this work on real-scale models?
- LoRA is already efficient. What is the actual gain?
- Please provide more ablations.