- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the placement problem of LoRA modules when trainable parameters are limited. A gradient-based metric, PAGE, is proposed to estimate the initial trainable gradient energy of candidate LoRA modules. PAGE is used to identify the downprojected modules of FFNs and designates them as "dominant adaptive modules." Based on this, the DomLoRA algorithm is proposed, which places a single LoRa adapter within this module. Experimental results show that DomLoRA achieves better average performance with a significant reduction in trainable parameters.
Critical strengths
-
The paper's motivation is reasonable, exploring where LoRA should be placed to achieve better results when the parameter budget is relatively limited.
-
The paper's method is simple and easy to understand, and easy to follow.
-
This paper achieves good performance using fewer trainable parameters.
-
The theoretical derivation is good, explaining why DomLoRA is effective, but the overall logic is slightly rigid, and readability needs improvement.
Critical weaknesses
-
The method's innovation is relatively limited; using Fisher + gradient to explore module sensitivity is not novel.
-
The experimental results do not support the paper's main claims; the authors seem to have failed to distinguish between better fit and less capability degradation. In Table 1, I found that on most tasks, the performance of the Domlora-trained model was worse than the base model. What is the point of training it then? The advantage over LoRA mostly comes from the TruthfulQA dataset; even after removing this dataset, Domlora still underperforms the base model. This is puzzling.
-
The baselines used in the experiments are relatively old; more recent baselines need to be included for a fair comparison.
4.The identity in Theorem 1 is not wrong, but it only characterizes the initial sample-by-sample gradient energy and does not prove that this energy can predict batch optimization progress or final fit quality. The paper needs more comprehensive ranking-to-actual training benefit correlation experiments to extrapolate from sensitivity to placement effectiveness.
Questions for authors
please refer the weaknesses
2 comments
Dear Reviewer,
Thank you again for your thoughtful review and constructive suggestions, which have helped us strengthen our paper.
As the discussion period is approaching its end, we would be very grateful for your feedback on whether our responses and additional experiments have addressed your concerns. If any questions remain, we would sincerely appreciate the opportunity to clarify them before the deadline.
We understand the demands on your time and truly appreciate your consideration.
Best regards, Authors of Submission 3099
We hope this message finds you well. We are writing to kindly follow up on our previous correspondence regarding the rebuttal for our manuscript. We understand that you have a busy schedule, but we would greatly appreciate any feedback or responses you may have regarding our paper.
We are eager to address any further concerns to help improve the paper. Please let us know if there is anything further we can provide.
Thank you once again for your time and consideration.
Best regards,
Authors of Submission 3099