- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new version of LoRA called TriBLoRA that splits its matrices into blocks and adds a trainable matrix between each pair of blocks. The authors also propose use different ranks for different blocks while keeping the same number of parameters. The paper includes theoretical results about the proposed method. The authors test it on GLUE with RoBERTa-base and on several tasks with LLaMA-2-7B. The results show small improvements over LoRA. The authors also study how different parts of the method affect the results.
Critical strengths
The main strength of the paper is detailed experiments. The authors compare TriBLoRA with several existing PEFT methods and provide many ablations: different interaction matrices, initialization methods, block numbers, and rank allocations. However, I think it is not enough to compensate for small improvements in performance and the fact that the method is quite similar to existing approaches.
Critical weaknesses
The proposed method is quite similar to existing block-based LoRA methods like BoRA. The main difference is use of dense matrices instead of diagonal. Theory results follow from standard matrix factorization and low-rank analysis, rank allocation considers a simplified problem, so theory is limited.
The improvements over LoRA are quite small on both GLUE and LLaMA-2-7B. Learned matrices replaced by identity does not affect the performance. Rank allocation shows no improvement over uniform. It seems it is additional complexity which is not justified and does not support accept at ICLR.
Questions for authors
I do not have specific questions. My main concerns are about the contribution and small improvements in the experiments, not unclear explanations.