- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GenLoRA, which generates LoRA factors from shared learnable latent vectors using grouped RBF generators to reduce the parameter cost of adding rank components. The method combines within-group normalization, Gaussian RBF expansions, and a base-activation branch, while retaining compatibility with weight merging. Experiments cover mathematical reasoning, commonsense reasoning, and code generation, with additional matched-rank comparisons, related baselines, and visual classification results in the appendix. The reported results indicate competitive adaptation performance with substantially fewer trainable parameters and training times close to LoRA. The analysis establishes structural rank bounds and conditional local gradient boundedness.
Critical strengths
- Matched-rank comparisons support parameter efficiency. Appendix G.5 reports approximately 8.7–15.3 times fewer trainable parameters than LoRA at (r=32), with the highest average accuracy in four of six reasoning settings. These results directly support the practical value of compact factor generation.
- The evaluation and component analysis are substantial. Experiments cover multiple backbones and task families, with ablations examining normalization, grouping, generator bases, and latent-vector learnability. The method also retains weight-merging compatibility, which is useful for deployment.
Critical weaknesses
- The correspondence between the initialization description and released implementation would benefit from clarification. Appendix K.3 describes random nonzero initialization of (Z_B), whereas line 141 of mypeft/Genlora.py in the supplied snapshot initializes this latent vector using torch.zeros(out_features). With zero B-side generator coefficients, both settings preserve a zero initial update, but it is unclear which setting was used for the main results. The authors may consider clarifying the experimental setting and reconciling the manuscript with the code; a version note would help if the implementation changed.
- The HumanEval+ evaluation configuration could be specified more explicitly. Section 4.1, Appendix K.1, and Table 22 report using BigCode Evaluation Harness with 50 samples per problem and Pass@1/5/10, but leave the framework revision and decoding settings unspecified. I could not locate the corresponding HumanEval+ evaluation entry point or configuration in the supplied code snapshot. The authors may consider providing the framework revision, evaluation command, and task configuration, including temperature, top-p, and the generation-length limit. Clarifying whether native prompts, stopping criteria, and post-processing rules were retained, together with any overrides or modifications, would help; explicit pointers to existing scripts would also suffice.
Questions for authors
-
Did the main experiments use the random initialization of (Z_B) described in Appendix K.3 or the zero initialization in the supplied code snapshot? Could the authors identify the corresponding code version?
-
Could the authors provide the framework version or commit, evaluation command, and task and decoding configurations used for HumanEval+? Please also clarify whether native prompts, stopping criteria, and post-processing rules were retained, and identify any configuration overrides or implementation changes.
2 comments
Dear Reviewer 93MN,
We would appreciate your feedback on whether our rebuttal has adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarification and continue the discussion.
Thank you for your time and consideration.
Response to Reviewer 93MN
Hi, Reviewer 93MN:
Thank you for recognizing GenLoRA's parameter efficiency at matched rank, broad evaluation and component analysis, and weight-merging compatibility.
Your remaining concerns focus on (i) consistency between the initialization description and implementation and (ii) reproducibility of the HumanEval+ evaluation. We address them below.
W1 and Q1: Initialization of $Z_B$ and the corresponding implementation.R1: Thank you for identifying this inconsistency. The main experiments used zero-initialized ZB, consistent with torch.zeros(out_features) in the supplied mypeft/Genlora.py snapshot (line 141). Appendix K.3 incorrectly described random initialization of ZB; this is now corrected.
The actual initialization was:
Here, dgroup is the latent-group dimension. Under Eqs. (7)-(9), zeroing both B-side coefficient sets gives
B0=0,ΔW0=B0A0=0,
preserving the pretrained model's initial output.
Revision location: Corrected Appendix K.3 (p. 30); Appendix N.1, Table 30 (p. 37), records these parameters and the supplied implementation snapshot/routine. The original frozen-latent ablation used Kaiming initialization before freezing and is distinguished from the main experiments.
W2 and Q2: HumanEval+ framework version, evaluation command, task configuration, and decoding protocol.R2: Thank you for pointing out the missing reproducibility details.
We use the BigCode Evaluation Harness vendored in GraLoRA, with the GraLoRA repository revision fixed at commit
853e7c9b0d5c7eaa01767566ab53c14ef5479c5a. The HumanEval+ task definition, scoring code, andcode_evaluate.pyare unchanged from this revision.The evaluation entry point is
code_evaluate.py, and the task identifier ishumanevalplus, defined inbigcode_eval/tasks/humanevalplus.py. The repository also provides an evaluation example script as a reference for the evaluation interface.The checked records corresponding to the main-table experiments for LoRA, MELoRA, HiRA, DoRA, AuroRA, and GenLoRA share the following settings:
The training seed was 42; the generation seed was 0. Appendix N.2 documents these separately. The saved outputs contain all 164 problems with 50 candidates per problem.
We retained the task's native prompts, stopping criteria, and post-processing. No instruction/chat template or additional prefix was added. These task routines were not modified for the reported evaluation.
We also verified the Pass@5 scoring entry point. Let ci be the number of candidates passing the tests among the 50 generated for problem i. The reported estimator is
Pass@5=1641∑i=1164(1−(550)(550−ci)).
Scores are presented as percentages. The estimator uses all 50 candidates per problem; generating 10 candidates per call is a generation grouping setting.
Revision location: Appendix N.2, Table 31 and Eq. (48) (p. 37), supplements Appendix K.1/Table 22 with these decoding settings, length limit, seeds, task-processing details, and the Pass@5 estimator.
Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns and clarify any doubts. Please let us know of any further questions.