- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to select a fine-tuning configuration for large language models that balances accuracy, latency, and memory overhead, within a limited number of evaluation trials. AEPS-OT learns to select evolutionary operators and leverages configurations from past tasks to assist in the search process. Experimental results show that this method achieves the best average performance, although ablation studies indicate that most of the gains come from initializing the search with historical configurations.
Critical strengths
The experimental section in the paper provides a detailed validation of the results. Instead of directly using the best results obtained during the search process, the authors retrain the selected configurations and evaluate them on an independent test set, which makes the experimental results more credible.
Critical weaknesses
-
The improvement brought by the new mechanism is still not clear. As shown in Table 2, using only historical configurations for initialization has already achieved most of the gains, and the additional benefits from subsequent transfer are relatively small. It's unclear whether it's necessary to introduce a scoring mechanism that weights by relevance, stability, and preference. Moreover, all experiments use the same number of transfer slots, which resembles simply selecting configurations under a fixed budget. This still doesn't demonstrate that the method can effectively determine how much transfer budget should be allocated. The authors need to conduct a fair comparison with simpler strategies such as randomly selecting historical sources or choosing configurations based solely on performance on the target task.
-
There are many comparison methods, but the comparisons directly relevant to this paper remain insufficient. The experiments mainly compare general optimization algorithms, while PEFT configuration search methods such as AutoPEFT and BIPEFT mentioned in the paper were not included in the comparison. It's still difficult to assess the actual improvements of this work over existing approaches. The soundness of the resource-constrained experiments is also limited. All configurations checked in Appendix D meet the latency and memory requirements, and the final evaluation mainly focuses on accuracy, without demonstrating how trade-offs should be made under resource constraints. Simply adjusting the scale of hyper-volume calculation after the experiments end does not indicate that the method can adapt its search strategy according to deployment requirements.
Questions for authors
-
Why are there no methods directly compared to AutoPEFT, BIPEFT, etc.? Could an experiment be added after reasonably considering the search space and training cost differences?
-
If using the same initial configuration and transfer, does the complete scoring mechanism perform better than randomly selecting sources or selecting only based on performance on the target task? Are the improvements consistent across different random seeds?
-
If delay or memory limits are tightened, making some high-accuracy configurations unavailable, does the proposed method still have advantages? Compared to search with fixed preferences, which different configurations would it select?