- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- The paper proposes GeDRA (Group Decoupled Reservoir Adaptation), a PEFT method that indepentently modulates rotation and deformation in weight transformations. The motivation behind GeDRA is to perform high rank updates while lowering trainable parameters.
- GeDRA structures the parameter updates as , where is the geometric adaptation generator.
- is constructed as $U_\ell = \operatorname{Diag}(\xi_\ell) P_\ell, \qquad Q = I_n + \frac{1}{\sqrt{2M}} \sum_{\ell=1}^{M} U_\ell$ where is sampled uniformly from a set of permutation matrices, and .
- The reservoir is decomposed into skwe-symmetric component which is the rotation-generator template and symmetric component which is the deformation-generation template.
- is obtained by adding two matrices and , where and are trainable vectors.
- The method is evaluated on commonsense reasoning, natural language understanding, visual recognition and sibject driven generation.
Critical strengths
- The rotation and deformation analysis of full fine-tuning, LoRA and OFT, and the result that GeDRA update shows pattern similar to full fine-tuning is interesting.
Critical weaknesses
- The performance of LoRA in all the benchmarks looks much lower than expected. For example, on commonsense reasoning, [1] reports 86.3% accuracy for LoRA with Llama3 8B. This paper reports 80.19%. While a few point difference is expected, this difference is too large, which suggests that the hypserparameters for LoRA were not tuned well or the evaluation settings like sampling are different.
- This makes the gains over LoRA questionable.
- The does not cite WeGeFT [1], which is highly related and has the same structural form: WeGeFT uses an update when applied to the output dimensions of the weights (or when applied to the input dimensions), where is a low rank matrix with and learned directly instead of . The work should include a comparison with WeGeFT in shared and unshared modes.
References
[1] Savadikar et al., "WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models", ICML 2025
Questions for authors
- How are the hyperparameters tuned for LoRA?