- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes LIZO, a sparse zeroth order method for LLM finetuning. The authors use normalized parameter values (based on the mean and std of all parameter values of the model ) as an importance metric. They then use the lower 25% of parameters based on the importance metric to perturb during zeroth-order finetuning. The paper proves that such low-scoring parameters exhibit smaller expected change of score when using Gaussian perturbation. Experimental results show LIZO improves over several ZO baselines in LLM finetuning (particularly in classification tasks) in the paper's evaluation settings.
Critical strengths
1- The importance metric used is simple and easy to implement.
2- The reported classification results are promising.
3- The mask swapping ablation is useful.
Critical weaknesses
1- The analysis of magnitude-base perturbation (Both figure 1 and the results provided in figure 5) is not convincing as the values apparently are for a single training seed and only one task is used (assuming figure 5 is also on SST-2. The authors don't mention what task figure 5 is about). The differences seem insignificant and more tasks and more training seeds are needed. Particularly for these small benchmarks that are very fast to train on.
2- The novelty appears limited. LIZO applies an existing energy-based score to ZO finetuning. The point of S-MEZO was to show that using sparse ZO finetuning helps and that a magnitude-based rule can be helpful. There are of course a number of other existing metrics that can be used for sparse ZO finetuning. This paper then uses the metric proposed in SimAM (which is cited). There are several other metrics from the pruning and sparsity literature that might outperform magnitude-based metrics, and applying an existing metric to sparse ZO finetuning does not by itself show a substantial contribution.
3- Experimental results are limited. Only two models and only one (if the same across all tasks) training seed and older benchmarks which might not be useful today. Also LIZO does not "consistently improves over existing ZO methods across a range of settings" as the abstract suggests (LIZO is outperformed by the baselines in open ended QA tasks ). Larger models with current important benchmarks (math, coding, ...) and more training seeds are needed.
4- The paper does not quantify how different its masks are from S-MEZO. Figure 6 is not enough. Since the proposed metric essentially becomes magnitude based if the mean of parameters is 0, what is driving the improvements over S-MEZO if the improvements persist with more training seeds.
Questions for authors
1- The appendix mentions results from existing papers are used. How have authors made sure that the evaluation data is identical in different methods? Are the evaluation settings identical?
2- "LIZO leads all seven Llama3-8B classification tasks" is incorrect. FZOO outperform LIZO on MultiRC. Also the bolded entry for that model-task is incorrect.
3- Is the mean and std computed over the entire model weights? Is bottom 25% over the entire model weights?
4- If like the paper claims for efficiency purposes all weights use the same mean and variance, equation 6 should produce at most two curves when plotted against absolute weight size. One for positive weights and one for negative weights. Why does figure 6 show a scatter? Were different means and variances used for different weights?