acceptodds
Under review as a conference paper at ICLR 2027

Nonlinearity as Rank: Generative Low-Rank Adapters with Radial Basis Functions

Abstract

Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices. However, standard LoRA follows an explicit-rank paradigm, where increasing model capacity requires adding more rows or columns (i.e., **basis vectors**) to the low-rank matrices, leading to substantial parameter growth. In this paper, we find that these basis vectors exhibit significant parameter redundancy and can be compactly represented by lightweight nonlinear functions. Therefore, we propose **Generative Low-Rank Adapter (GenLoRA)**, which replaces explicit basis vector storage with nonlinear basis vector generation. Specifically, GenLoRA maintains a latent vector for each low-rank matrix and employs a set of lightweight radial basis functions (RBFs) to synthesize the basis vectors. Each RBF requires far fewer parameters than an explicit basis vector, enabling higher parameter efficiency in GenLoRA. Extensive experiments across multiple datasets and architectures show that GenLoRA attains higher effective LoRA ranks under smaller parameter budgets, resulting in superior fine-tuning performance. The code is available at https://anonymous.4open.science/r/GenLoRA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-rank adaptation, LLM Fine-tuning, Parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_93MNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GenLoRA, which generates LoRA factors from shared learnable latent vectors using grouped RBF generators to reduce the parameter cost of adding rank components. The method combines within-group normalization, Gaussian RBF expansions, and a base-activation branch, while retaining compatibility with weight merging. Experiments cover mathematical reasoning, commonsense reasoning, and code generation, with additional matched-rank comparisons, related baselines, and visual classification results in the appendix. The reported results indicate competitive adaptation performance with substantially fewer trainable parameters and training times close to LoRA. The analysis establishes structural rank bounds and conditional local gradient boundedness.

Critical strengths
  1. Matched-rank comparisons support parameter efficiency. Appendix G.5 reports approximately 8.7–15.3 times fewer trainable parameters than LoRA at (r=32), with the highest average accuracy in four of six reasoning settings. These results directly support the practical value of compact factor generation.
  2. The evaluation and component analysis are substantial. Experiments cover multiple backbones and task families, with ablations examining normalization, grouping, generator bases, and latent-vector learnability. The method also retains weight-merging compatibility, which is useful for deployment.
Critical weaknesses
  1. The correspondence between the initialization description and released implementation would benefit from clarification. Appendix K.3 describes random nonzero initialization of (Z_B), whereas line 141 of mypeft/Genlora.py in the supplied snapshot initializes this latent vector using torch.zeros(out_features). With zero B-side generator coefficients, both settings preserve a zero initial update, but it is unclear which setting was used for the main results. The authors may consider clarifying the experimental setting and reconciling the manuscript with the code; a version note would help if the implementation changed.
  2. The HumanEval+ evaluation configuration could be specified more explicitly. Section 4.1, Appendix K.1, and Table 22 report using BigCode Evaluation Harness with 50 samples per problem and Pass@1/5/10, but leave the framework revision and decoding settings unspecified. I could not locate the corresponding HumanEval+ evaluation entry point or configuration in the supplied code snapshot. The authors may consider providing the framework revision, evaluation command, and task configuration, including temperature, top-p, and the generation-length limit. Clarifying whether native prompts, stopping criteria, and post-processing rules were retained, together with any overrides or modifications, would help; explicit pointers to existing scripts would also suffice.
Questions for authors
  1. Did the main experiments use the random initialization of (Z_B) described in Appendix K.3 or the zero initialization in the supplied code snapshot? Could the authors identify the corresponding code version?

  2. Could the authors provide the framework version or commit, evaluation command, and task and decoding configurations used for HumanEval+? Please also clarify whether native prompts, stopping criteria, and post-processing rules were retained, and identify any configuration overrides or implementation changes.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 93MN

Hi, Reviewer 93MN:

Thank you for recognizing GenLoRA's parameter efficiency at matched rank, broad evaluation and component analysis, and weight-merging compatibility.

Your remaining concerns focus on (i) consistency between the initialization description and implementation and (ii) reproducibility of the HumanEval+ evaluation. We address them below.

W1 and Q1: Initialization of $Z_B$ and the corresponding implementation.

R1: Thank you for identifying this inconsistency. The main experiments used zero-initialized ZBZ_B, consistent with torch.zeros(out_features) in the supplied mypeft/Genlora.py snapshot (line 141). Appendix K.3 incorrectly described random initialization of ZBZ_B; this is now corrected.

The actual initialization was:

ComponentInitialization
ZAZ_AKaiming Uniform with a=5a=\sqrt{5}
A-side generator coefficientsNormal distribution, mean 00, standard deviation 1/dgroup1/\sqrt{d_{\mathrm{group}}}
ZBZ_BAll zeros
B-side generator coefficientsAll zeros, including both base-activation and RBF combination coefficients

Here, dgroupd_{\mathrm{group}} is the latent-group dimension. Under Eqs. (7)-(9), zeroing both B-side coefficient sets gives

B0=0,ΔW0=B0A0=0,B_0=0,\qquad \Delta W_0=B_0A_0=0,

preserving the pretrained model's initial output.

Revision location: Corrected Appendix K.3 (p. 30); Appendix N.1, Table 30 (p. 37), records these parameters and the supplied implementation snapshot/routine. The original frozen-latent ablation used Kaiming initialization before freezing and is distinguished from the main experiments.

W2 and Q2: HumanEval+ framework version, evaluation command, task configuration, and decoding protocol.

R2: Thank you for pointing out the missing reproducibility details.

We use the BigCode Evaluation Harness vendored in GraLoRA, with the GraLoRA repository revision fixed at commit 853e7c9b0d5c7eaa01767566ab53c14ef5479c5a. The HumanEval+ task definition, scoring code, and code_evaluate.py are unchanged from this revision.

The evaluation entry point is code_evaluate.py, and the task identifier is humanevalplus, defined in bigcode_eval/tasks/humanevalplus.py. The repository also provides an evaluation example script as a reference for the evaluation interface.

The checked records corresponding to the main-table experiments for LoRA, MELoRA, HiRA, DoRA, AuroRA, and GenLoRA share the following settings:

Evaluation settingValue
BenchmarkHumanEval+, 164 problems
Samples per problem50
Candidates per generation call10
SamplingEnabled
Temperature0.2
Top-p0.95
Top-k0
Maximum total sequence length2048 tokens, including the prompt
Generation seed0
Reported metricsPass@1, Pass@5, Pass@10

The training seed was 42; the generation seed was 0. Appendix N.2 documents these separately. The saved outputs contain all 164 problems with 50 candidates per problem.

We retained the task's native prompts, stopping criteria, and post-processing. No instruction/chat template or additional prefix was added. These task routines were not modified for the reported evaluation.

We also verified the Pass@5 scoring entry point. Let cic_i be the number of candidates passing the tests among the 50 generated for problem ii. The reported estimator is

Pass@5=1164∑i=1164(1−(50−ci5)(505)).\mathrm{Pass@5} = \frac{1}{164}\sum_{i=1}^{164} \left(1-\frac{\binom{50-c_i}{5}}{\binom{50}{5}}\right).

Scores are presented as percentages. The estimator uses all 50 candidates per problem; generating 10 candidates per call is a generation grouping setting.

Revision location: Appendix N.2, Table 31 and Eq. (48) (p. 37), supplements Appendix K.1/Table 22 with these decoding settings, length limit, seeds, task-processing details, and the Pass@5 estimator.

Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns and clarify any doubts. Please let us know of any further questions.

Dear Reviewer 93MN,

We would appreciate your feedback on whether our rebuttal has adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarification and continue the discussion.

Thank you for your time and consideration.

Reviewer_6iuMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GenLoRA, a parameter-efficient fine-tuning method that replaces explicitly stored LoRA basis vectors with vectors generated by lightweight nonlinear functions. The central motivation is that conventional LoRA incurs a parameter cost of �0��0�, whereas GenLoRA uses shared learnable latent vectors and group-wise radial basis function (RBF) generators, reducing the claimed parameter complexity to �1��1�.

The method includes instance-wise normalization and group-wise decomposition to improve the expressiveness and stability of the generators. The paper also provides theoretical results establishing rank boundedness and conditional local gradient boundedness.

Experiments across LLaMA-3-8B, Gemma-7B, and Qwen2.5-14B on mathematical reasoning, commonsense reasoning, and code generation demonstrate improved parameter efficiency compared with LoRA and several PEFT baselines.

Critical strengths

S1. A simple and potentially useful parameterization of low-rank adaptation.

The paper explores an interesting alternative to conventional low-rank adaptation: rather than explicitly optimizing all elements of the factor matrices, it learns a compact nonlinear parameterization from which these matrices can be constructed.

The proposed formulation is conceptually straightforward and compatible with the standard LoRA weight-update structure. This makes it potentially useful in practical PEFT settings.

S2. Strong empirical parameter efficiency.

The results show promising accuracy–parameter trade-offs across multiple model architectures and tasks.

For example, on LLaMA-3-8B mathematical reasoning (Table 1), GenLoRA achieves 70.17% average accuracy with 0.98M trainable parameters, compared with 67.28% for standard LoRA using 4.72M parameters.

On Qwen2.5-14B commonsense reasoning (Table 2), GenLoRA reaches 91.10% with 2.26M parameters, compared with 88.91% for LoRA using 8.65M parameters.

These results suggest that the proposed parameterization can offer a meaningful practical advantage.

S3. Broad evaluation across model families and downstream tasks.

The paper evaluates multiple pretrained models and includes mathematical reasoning, commonsense reasoning, and code generation. The consistency of the improvements across these settings strengthens the empirical contribution.

S4. Limited reported training-time overhead.

The code-generation experiment (Table 3) reports training times of 11.7 hours for GenLoRA at rank 8 versus 11.6 hours for standard LoRA, suggesting that the additional nonlinear parameterization does not necessarily introduce substantial training overhead.

Critical weaknesses

W1. The relationship between nonlinearity, nominal rank, and actual representational capacity is unclear.

The paper's central framing, "Nonlinearity as Rank," suggests that nonlinear generation can substitute for explicit rank parameters. However, the resulting update remains a conventional low-rank matrix, and its attainable rank is constrained by the generator's shared feature space.

Importantly, Appendix A.1 establishes the tighter bound $
\operatorname{rank}(\Delta W) \leq \min{r,m,n,G_A(K+1),G_B(K+1)}. $

This means that increasing the number of generators does not necessarily increase the attainable rank. For example, in the ungrouped configuration with �0��0�, the rank cannot exceed 16 even when the nominal rank is 128. Although the appendix correctly acknowledges this distinction, it raises a fundamental question about the paper's central conceptual claim. The primary contribution appears to be a parameter-efficient structured factorization, rather than nonlinearity serving as a substitute for rank itself. The authors should clarify this distinction and investigate the actual rank and expressive capacity of the generated updates.

W2. Insufficient comparisons with simpler structured parameterizations.

GenLoRA uses learnable latent vectors, normalization, grouping, and RBF expansions. While the component ablations demonstrate the importance of normalization and grouping, they do not establish whether nonlinear RBF generation is necessary.

In particular, Equation (7) defines each generated basis vector through a base activation and a linear combination of fixed Gaussian features. Once the latent vectors are fixed, the generated factors are linear combinations of these evaluated features. This suggests that GenLoRA can be interpreted as learning coefficients over a structured feature dictionary. It would be valuable to compare against simpler alternatives, including linear or affine transformations of the shared latent vectors, polynomial or Fourier feature expansions, and learned or fixed shared basis dictionaries. Such comparisons would clarify whether the improvements originate from RBF nonlinearity or from the more general use of a shared, structured parameterization.

W3. The theoretical results are relatively limited.

Proposition 3.1 establishes that the rank of the generated update is bounded by �1��1�. However, this follows directly from expressing the update as a product of two matrices with inner dimension �2��2�, and does not provide a distinctive theoretical property of GenLoRA. Similarly, Proposition 3.2 establishes local gradient boundedness under compactness and bounded upstream-gradient assumptions. While technically valid under the stated conditions, such a result does not directly establish improved optimization stability relative to conventional LoRA.

A more meaningful theoretical contribution would characterize the expressive capacity of the RBF-generated factorization, its approximation error relative to unconstrained LoRA, or conditions under which it can represent a given low-rank update.

Questions for authors
  1. On the necessity of RBF nonlinearity: How does GenLoRA compare against simpler shared-basis parameterizations, particularly linear transformations and fixed nonlinear feature dictionaries, under matched parameter budgets?

  2. On basis-vector redundancy: Since LoRA factorization is non-unique, how robust is the observed basis-vector redundancy to invertible transformations of the factors? Does the reconstruction quality correlate with downstream model performance?

  3. On actual rank: Can the authors report the numerical rank and singular-value distributions of the generated updates across layers and training stages? How often does the actual rank approach the nominal rank �0��0�?

  4. On expressive capacity: Given the fixed-grid feature-space factorization in Appendix A.1, what classes of rank-�1��1� updates can GenLoRA represent? Is there a theoretical or empirical approximation gap relative to unconstrained LoRA at the same nominal rank?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6iuM [Part I]

Hi, Reviewer 6iuM:

Thank you for recognizing GenLoRA's simple parameterization, strong parameter efficiency, broad evaluation across model families and downstream tasks, and limited training overhead.

Your questions focus on (i) the relationship between nonlinearity and actual rank, (ii) the value of RBF generation beyond shared parameterization, and (iii) expressive capacity and basis-vector redundancy. We address these points through the existing spectral analysis, additional matched-budget comparisons, direct reconstruction of trained update matrices, and downstream evaluation of reconstructed adapters in the following three parts.

W1 and Q3: "The relationship between nonlinearity, nominal rank, and actual representational capacity is unclear." "Can the authors report the numerical rank and singular-value distributions of the generated updates across layers and training stages? How often does the actual rank approach the nominal rank $r$?"

R1: Thank you for requesting clarification of nominal rank, numerical rank, and the role of nonlinearity. The central idea behind "Nonlinearity as Rank" is to reduce the parameter cost of constructing additional rank components. Standard LoRA explicitly learns two basis vectors of lengths mm and nn for each additional component. GenLoRA generates these vectors from shared latents and compact nonlinear generators, replacing the corresponding storage cost with a small set of generator coefficients.

Appendix A.1 characterizes the feature-space structure:

rank⁡(ΔW)≤min⁡r,m,n,GA(K+1),GB(K+1).\operatorname{rank}(\Delta W) \leq \min\\{r,m,n,G_A(K+1),G_B(K+1)\\}.

This bound explains the role of grouping. With K=15K=15, an ungrouped factor has 16 shared features, whereas G=8G=8 and G=16G=16 provide 128 and 256 features per factor, respectively. Grouping introduces independently parameterized feature blocks across coordinate subsets, supplying the feature budget used by the higher-rank configurations.

Figure 4(b) and Section 4.5 provide the corresponding empirical rank measurements across all 32 query-projection layers of LLaMA-3-8B. The numerical-rank threshold used in these measurements is 0.005. At the evaluated trained checkpoints, GenLoRA with r=8,32,64r=8,32,64 attains the corresponding numerical ranks. In particular, GenLoRA achieves numerical rank 64 with 3.74M parameters, whereas rank-8 LoRA uses 4.72M parameters. This directly demonstrates its ability to construct higher-rank updates at lower parameter cost. Appendix L.6 (p. 35) now states this threshold and clarifies that these are trained-checkpoint measurements, not a rank time series during training.

Appendix J, Table 19 further reports the spectral energy:

E(ΔW)=∑iσi2=∣ΔW∣F2.E(\Delta W)=\sum_i\sigma_i^2=\\|\Delta W\\|_F^2.

GenLoRA learns updates with substantial magnitude and higher spectral energy under a compact parameter budget. These spectral observations accompany the strong downstream results in mathematical reasoning, commonsense reasoning, and code generation. Together with the direct reconstruction results in Part III, they support GenLoRA's ability to represent and learn task-relevant weight changes efficiently.

Revision location: Appendix L.6, Eq. (47) (p. 35); existing Section 4.5/Figure 4(b) and Appendix A.1/J remain.

Please see Part II for generator controls.

AuthorsOpenReview ↗

Response to Reviewer 6iuM [Part II]

W2 and Q1: "Insufficient comparisons with simpler structured parameterizations." "How does GenLoRA compare against simpler shared-basis parameterizations, particularly linear transformations and fixed nonlinear feature dictionaries, under matched parameter budgets?"

R2: Thank you for suggesting comparisons with simpler shared parameterizations and fixed feature dictionaries. GenLoRA combines nonlinear feature generation with learning the latent inputs that determine the evaluated dictionary. Although the RBF centers and bandwidth are fixed, the responses ϕk(z^j)\phi_k(\hat z_j) evolve as the latents are optimized. The method therefore learns both the feature combinations and the feature representations used to generate the factors.

To evaluate these design choices, we supplement the existing Chebyshev and B-spline comparisons with budget-matched affine and fixed-RBF-dictionary controls. Table R1 summarizes the results on LLaMA-3-8B/Math10K.

Table R1: Generator and fixed-dictionary comparisons on LLaMA-3-8B/Math10K.

MethodTrainable latentsNominal width rrGroups GGTrainable parametersAverage accuracy (%)
Gaussian RBF: full GenLoRAYes8160.98M70.17
Chebyshev generatorYes8160.95M64.01
B-spline generatorYes8161.06M64.54
Fixed RBF dictionary: same r,Gr,GNo8160.39M60.63
Grouped affine: budget matchedYes81280.98M63.76
Fixed RBF dictionary: budget matchedNo20160.98M63.56

At comparable budgets, Gaussian RBF exceeds Chebyshev and B-spline by 6.16 and 5.63 percentage points, respectively. The G=32G=32 comparison in Appendix H.4, Table 16 shows the same ordering, supporting the effectiveness of the RBF generator across the reported grouping configurations.

The additional controls directly address equal parameter budgets. Full GenLoRA achieves 70.17%, outperforming the affine generator by 6.41 percentage points and the fixed RBF dictionary by 6.61 percentage points, with all three using approximately 0.98M trainable parameters.

We match the affine budget by increasing its group count to 128 while retaining r=8r=8. For the fixed dictionary, increasing the nominal width to r=20r=20 compensates for the parameters removed by freezing the latents. These settings give the simpler parameterizations the same trainable-parameter budget. Full GenLoRA retains its performance advantage over both controls.

The original frozen-latent ablation (Appendix H.5, Table 17) and the additional matched-budget comparison provide complementary evidence: the former evaluates latent learning within the same generator structure, while the latter shows its benefit relative to allocating the same budget to more fixed-dictionary coefficients. Together, these results support the combination of Gaussian RBF features and learnable latents as an effective way to improve the accuracy-parameter trade-off.

Revision location: Appendix L.3, Tables 23–25 (pp. 33–34); the original Appendix H.4–H.5 controls are retained.

Please see Part III for expressive capacity and reconstruction.

AuthorsOpenReview ↗

Response to Reviewer 6iuM [Part III]

W3, Q2, and Q4: "The theoretical results are relatively limited." "Since LoRA factorization is non-unique, how robust is the observed basis-vector redundancy to invertible transformations of the factors? Does the reconstruction quality correlate with downstream model performance?" "Given the fixed-grid feature-space factorization in Appendix A.1, what classes of rank-$r$ updates can GenLoRA represent? Is there a theoretical or empirical approximation gap relative to unconstrained LoRA at the same nominal rank?"

R3: Thank you for asking us to clarify expressive capacity, factorization non-uniqueness, and downstream reconstruction utility. Our propositions establish structural compatibility and local gradient regularity: Proposition 3.1 preserves the low-rank update structure and compatibility with weight merging; Proposition 3.2 establishes locally bounded gradients for normalized RBF generation under the stated regularity conditions. Building on Figure 1's basis-vector reconstruction, we further examine expressive capacity through feature-space analysis, cross-task/layer effective-update reconstruction, and downstream replacement evaluation.

For fixed latents, Appendix A.1 gives

B=ΨBCB,A⊤=ΨACA,B=\Psi_BC_B,\qquad A^\top=\Psi_AC_A,

where ΨA,ΨB\Psi_A,\Psi_B are block-diagonal matrices of grouped base-activation and RBF responses. With independently trainable, unrestricted coefficients, this family exactly represents updates of rank at most rr whose column and row spaces lie in col⁡(ΨB)\operatorname{col}(\Psi_B) and col⁡(ΨA)\operatorname{col}(\Psi_A), respectively. Learning the latents adapts these spaces.

For any invertible QQ,

BA=(BQ)(Q−1A),CB′=CBQ,CA′=CAQ−⊤.BA=(BQ)(Q^{-1}A),\qquad C_B'=C_BQ,\qquad C_A'=C_AQ^{-\top}.

Thus, the representation family is closed under invertible changes of basis, without changing latent features or parameter count. Moreover, directly reconstructing the effective update makes both the target and error invariant to the choice of equivalent factors.

We reconstruct updates from rank-8 LLaMA-3-8B LoRA checkpoints trained on Math10K and Commonsense170K, covering all 32 q_proj layers per checkpoint: 64 matrices in total. Each fixed target TT includes the original LoRA scaling. From initialization independent of the original factors, we optimize GenLoRA with r=8,G=16,K=15r=8,G=16,K=15 using

Lrec=∣ΔW^θ−T∣F2∣T∣F2.\mathcal L_{\mathrm{rec}} =\frac{\\|\widehat{\Delta W}_{\theta}-T\\|_F^2}{\\|T\\|_F^2}.

We run three random seeds per update. Table R2 reports the mean relative Frobenius error, i.e., the mean of Lrec\sqrt{\mathcal L_{\mathrm{rec}}} across updates and seeds.

Table R2: Cross-task, cross-layer reconstruction of effective updates.

TaskUpdatesLoRA parameters/updateGenLoRA parameters/updateParameters retainedMean relative error
Math10K3265,53612,28818.75%2.9%
Commonsense170K3265,53612,28818.75%1.6%

At the same nominal width r=8r=8, each reconstructed query-projection update retains only 18.75% of the original factor parameters. The errors of 2.9% and 1.6% directly quantify the achievable approximation accuracy for these trained updates.

To connect reconstruction quality with downstream utility, we replace all 32 q_proj updates in the original LoRA checkpoint with their GenLoRA reconstructions and evaluate the resulting model.

Table R3: Downstream performance after reconstructing all 32 query-projection updates.

TaskAdapter conditionMean relative errorAccuracy (%)Change (percentage points)
Math10KOriginal LoRA0%67.280.00
Math10KAll 32 q_proj updates reconstructed2.9%66.89-0.39
Commonsense170KOriginal LoRA0%82.190.00
Commonsense170KAll 32 q_proj updates reconstructed1.6%82.07-0.12

Accuracy change is reconstructed minus original-LoRA accuracy (Commonsense170K baseline: 82.19%).

On Math10K, 2.9% reconstruction error accompanies only a 0.39-percentage-point accuracy decrease, from 67.28% to 66.89%, after replacing all 32 query-projection updates. This directly demonstrates that compact reconstruction preserves most of the downstream performance in this setting. On Commonsense170K, 1.6% reconstruction error accompanies a 0.12-percentage-point decrease, from 82.19% to 82.07%, after the same replacement of all 32 query-projection updates. Together, the feature-space structure, update reconstruction, and replacement evaluation support GenLoRA's ability to compactly represent task-relevant weight changes while retaining their practical utility.

Revision location: Appendix L.4–L.5, Eqs. (44)–(46), Tables 26–27 (pp. 34–35).

Thank you for your feedback. We hope these responses clarify your concerns and welcome further questions.

Reviewer_SCEXOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Generative Low-Rank Adapter (GenLoRA), a parameter-efficient fine-tuning method that tackles the issue of parameter redundancy in traditional LoRA. Instead of explicitly storing low-rank basis vectors, GenLoRA generates them from shared latent vectors using lightweight radial basis functions (RBF), which reduces the number of trainable parameters. Experiments across various datasets and model architectures show that GenLoRA can achieve higher LoRA rank with fewer trainable parameters, while also boosting performance on downstream tasks.

Critical strengths
  1. This paper proposes a practical approach to low-rank matrix parameterization. GenLoRA replaces the explicit storage of low-rank basis vectors with a nonlinear generation mechanism based on shared latent vectors, thereby reducing the number of trainable parameters while preserving the conventional low-rank update structure.
  2. Comprehensive Experimental Evaluation with Valuable Additional Comparisons. The paper presents an extensive experimental evaluation, including several informative supplementary comparisons. The authors evaluate GenLoRA across language models of different scales and further assess its effectiveness on a diverse range of tasks.
Critical weaknesses
  1. There is still insufficient evidence for the core motivation for intrinsic parameter redundancy in LoRA base vectors. Figure 1 mainly shows the visual reconstruction of a single basis vector, which is insufficient to establish that such redundancy is a general structural property. More importantly, the LoRA decomposition is non-unique. Therefore, the observed similarity and reconstruction between basis vectors may depend on specific low-rank decomposition parameterization, rather than necessarily reflecting the intrinsic redundancy of the updated matrix itself.
  2. The explanation of actual efficiency and some performance improvements is still insufficient. The paper demonstrates efficiency advantages by significantly reducing the number of trainable parameters, but the number of parameters does not directly correspond to the training memory or runtime costs. For example, Table 9 shows that at r=32, GenLoRA's peak GPU memory is 31,872 MB, while LoRA's is 32,095 MB, with both training times being 12.2 hours. These results suggest that the practical efficiency gains may be more limited than the parameter reduction alone would imply. The authors are encouraged to provide a more comprehensive efficiency analysis, including the FLOPs introduced by the generator, a breakdown of GPU memory consumption, and training throughput under different batch sizes and sequence lengths.
Questions for authors

Please see the critical weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SCEX [Part I]

Dear Reviewer SCEX,

Thank you for recognizing the practicality of our low-rank matrix parameterization and the comprehensive evaluation across model scales and diverse tasks.

We understand that your main concerns are (i) whether the evidence for redundancy is sufficiently general and independent of the particular LoRA factorization, and (ii) how the reduction in trainable parameters translates into actual memory, computational, and runtime costs. We address these concerns in two parts.

W1: “There is still insufficient evidence for the core motivation for intrinsic parameter redundancy in LoRA base vectors.” “More importantly, the LoRA decomposition is non-unique.”

R1: Thank you for highlighting the non-uniqueness of LoRA factors. Figure 1 shows that compact nonlinear functions recover the global shape and essential patterns of the evaluated LoRA basis vector. Building on this basis-vector result, we additionally evaluate effective-update reconstruction across tasks and layers. For any invertible matrix Q∈Rr×rQ \in \mathbb{R}^{r \times r},

BA=(BQ)(Q−1A).BA = (BQ)(Q^{-1}A).

Since this transformation preserves ΔW\Delta W, the additional experiments use ΔW\Delta W itself as the target and measure reconstruction errors directly on that update. Figure 1 and Appendix L.5 thus provide complementary evidence for compact functional representation at the basis-vector and effective-update levels. Our central empirical claim remains that GenLoRA learns effective task adaptations with a favorable parameter–performance trade-off.

Our parameterization also has a structural property relevant to this concern. Using the feature expansion in Appendix A.1, the grouped generators can be written, for fixed latent inputs, as

B=ΨBCB,A⊤=ΨACA,B = \Psi_B C_B, \qquad A^\top = \Psi_A C_A,

where ΨA\Psi_A and ΨB\Psi_B assemble the grouped SiLU and RBF feature matrices block-diagonally, and CAC_A and CBC_B stack the independently trainable generator coefficients across groups. The invertible basis change can be absorbed into these coefficients:

CB′=CBQ,CA′=CAQ−⊤,C_B' = C_B Q, \qquad C_A' = C_A Q^{-\top},

without changing the latent features or increasing the parameter count. Thus, the representable factor family is closed under invertible changes of basis in the LoRA rank space. This structural property complements the reconstruction evidence; the following experiments quantify compression of trained effective updates.

Complementing Figure 1, we extend reconstruction to the effective update matrices themselves, across tasks and layers. We use the rank-8 LLaMA-3-8B LoRA checkpoints for Math10K and Commonsense170K, covering the q_proj updates in all 32 layers of each checkpoint, for 64 update matrices in total. Each target update remains fixed, including its original LoRA scaling. We fit an independently initialized GenLoRA representation with r=8r=8, G=16G=16, and K=15K=15 by minimizing

\mathcal{L}_{\mathrm{rec}} = \frac{ \left\\|\widehat{\Delta W}_{\theta}-\Delta W_{\mathrm{LoRA}}\right\\|_F^2 }{ \left\\|\Delta W_{\mathrm{LoRA}}\right\\|_F^2 }.

Table R1. Effective-update reconstruction across tasks and layers.

LoRA checkpoint taskNumber of updatesLoRA parameters per updateGenLoRA parameters per updateParameter retention ratioMean relative Frobenius reconstruction error
Math10K3265,53612,28818.75%2.9%
Commonsense170K3265,53612,28818.75%1.6%

The reported error is ∣ΔW^−ΔW∣F/∣ΔW∣F\\|\widehat{\Delta W}-\Delta W\\|_F/\\|\Delta W\\|_F, rather than its square, and is averaged over the 32 layer updates and three seeds per update for each task. The GenLoRA parameter count includes both latent vectors and all generator coefficients.

With 81.25% fewer representation parameters per update, GenLoRA achieves mean relative reconstruction errors of 2.9% and 1.6% on the two checkpoints. Because both the target and the metric operate directly on ΔW\Delta W, they remain unchanged whether the original update is expressed using (B,A)(B,A) or (BQ,Q−1A)(BQ,Q^{-1}A). These results provide quantitative evidence for the compact representability of the evaluated task updates, independently of the particular factors used to express the original checkpoints.

Revision location: Section 3 (p. 3); Appendix L.4–L.5, Eqs. (44)–(46) and Table 26 (pp. 34–35).

Please see Part II for the remaining response.

AuthorsOpenReview ↗

Response to Reviewer SCEX [Part II]

W2: “The number of parameters does not directly correspond to the training memory or runtime costs.” The reviewer requests generator FLOPs, a memory breakdown, and throughput measurements across batch sizes and sequence lengths.

R2: Thank you for requesting a clearer evaluation of memory, computation, and runtime costs. We agree that trainable parameters, peak training memory, and end-to-end runtime require separate evaluation. For LLaMA-3-8B on Magicoder at r=32r=32, Table 9 of the manuscript reports:

MethodTrainable parametersPeak GPU memoryTraining timeHumanEval+ Pass@1
LoRA18.87M32,095 MB12.2 h26.52%
GenLoRA2.16M31,872 MB12.2 h28.30%

GenLoRA reduces trainable parameters by approximately 88.6%, reduces peak memory by 223 MB, or approximately 0.7%, and achieves the same training time at the reported precision, while improving Pass@1 by 1.78 percentage points. These results support a favorable parameter–performance trade-off and comparable total training time in this setting. They do not establish a proportional reduction in total training memory.

To examine the effect of input size, Table R2 reports short-run profiling for LLaMA-3-8B with QKV adaptation at r=32r=32, using G=16G=16 and K=15K=15 for GenLoRA.

Table R2. Training throughput and peak memory across input sizes.

Micro-batch sizeSequence lengthMethodStep time (ms)Throughput (tokens/s)Peak allocated (MiB)Peak reserved (MiB)
1256LoRA212.51,20518,52919,200
1256GenLoRA222.51,15118,35418,944
11024LoRA550.01,86224,30224,832
11024GenLoRA560.01,82924,12724,832
2256LoRA325.01,57520,45320,992
2256GenLoRA335.01,52820,27820,992
21024LoRA1,000.02,04832,04932,512
21024GenLoRA1,010.02,02831,82532,512

Step time denotes the forward pass, backward pass, and optimizer update. Throughput is calculated as 1000BL/tstep,ms1000BL/t_{\mathrm{step,ms}} and rounded to the nearest integer, where BB is the micro-batch size and LL is the sequence length. Memory is reported in MiB in Tables R2–R3; Table 9 retains its original MB label.

Across these four input configurations, GenLoRA increases step time by 1.0%–4.7% and reduces peak allocated memory by 175–224 MiB. The relative runtime overhead is smaller in the longer-sequence configurations. Peak reserved memory is unchanged in three of the four configurations, so the reduction in allocated memory does not translate uniformly into a smaller reserved memory pool.

Table R3 accounts for memory and measures generator overhead at micro-batch 2 and sequence length 1024.

Table R3. Training-memory accounting and measured generator overhead (B=2B=2, L=1024L=1024, r=32r=32, G=16G=16, K=15K=15).

MetricLoRAGenLoRA
Frozen backbone weights (MiB)15,316.515,316.5
Adapter training states: parameters, gradients, and optimizer states (MiB)288.033.0
Other allocated memory: activations, temporary tensors, generator intermediates, etc. (MiB)16,444.516,475.5
Peak allocated memory (MiB)32,049.031,825.0
Total generator computation, including forward and backward (GFLOPs/step)02.45
Total generator GPU time, including forward and backward (ms/step)013.5

GenLoRA reduces adapter training states by 255 MiB, while the remaining allocations increase by 31 MiB, yielding a net peak-memory reduction of 224 MiB, or approximately 0.70% in this configuration. This explains why a substantial reduction in optimized adapter parameters produces a much smaller reduction in total training memory: frozen weights and the remaining training tensors dominate the total.

Revision location: Appendix M–M.2, Tables 28–29 (p. 36); the original full-training results remain in Table 9.

Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns and clarify any doubts. Please let us know of any further questions.

Dear Authors,

Thank you for your detailed rebuttal and the additional experiments and analyses.

Regarding W1, the newly added effective-update matrix reconstruction experiments provide more convincing evidence than the original visualization of individual basis vectors. Directly reconstructing the update matrices across multiple network layers and different tasks substantially addresses my concern about the non-uniqueness of the LoRA decomposition.

Nevertheless, I still believe that these results primarily demonstrate the empirical compressibility of LoRA update matrices under the evaluated experimental settings, rather than establishing intrinsic parameter redundancy as a general property of LoRA updates. The basis-change analysis is valuable, but it does not, by itself, establish the generality of this property. I therefore encourage the authors to clearly distinguish between these two levels of claims in the final version.

Regarding W2, I appreciate the authors' more comprehensive efficiency analysis. These additional results adequately address my concerns about the distinction between trainable-parameter efficiency and practical training efficiency.

Overall, I believe the rebuttal has mainly addressed my major concerns. Although some limitations remain regarding the generality of the parameter redundancy assumption underlying the core motivation, I no longer consider these issues to be major obstacles to the paper's acceptance.

Therefore, I have decided to increase my rating from 3 (confidence: 2) to 4 (confidence: 3).

Dear Reviewer SCEX,

Thank you for your thoughtful reassessment and constructive feedback. In the final version, we will clearly distinguish the empirical compressibility observed in our experiments from a general claim of intrinsic redundancy. We sincerely appreciate your time and valuable suggestions, which have helped improve our paper.

Reviewer_UNZaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores a generative approach to generate LoRAs using lightweight nonlinear functions based on the observation that basis vectors exhibit significant parameter redundancy.

Critical strengths

Broad empirical evaluation and practical compatibility with existing LoRA deployment

Critical weaknesses

Missing important references such as: Asymmetry in Low-Rank Adapters of Foundation Models (for Lora theory), as well as papers on metanetworks.

My main concern is the actual novelty of the paper: the methodology is simple and combines standard techniques, the theoretical section is trivial. And from a practical standpoint the paper doesnt solve a real problem. Lora is already parameter efficient and saving a few million parameters doesnt make a difference in practice.

Questions for authors

To what extent is nonlinearity itself responsible for GenLoRA's parameter efficiency, as opposed to parameter sharing and the implicit regularization induced by its grouped basis parameterization? Could the authors compare GenLoRA against an equally parameter-efficient linear or polynomial generator, holding the latent dimensions, grouping structure, and trainable parameter budget fixed?

6 comments
AuthorsOpenReview ↗

Response to Reviewer UNZa [Part I]

Hi, Reviewer UNZa:

Thank you for recognizing GenLoRA’s broad empirical evaluation and practical compatibility with existing LoRA deployment.

We understand that your main concerns focus on (i) methodological novelty and the purpose of the theoretical analysis, (ii) the practical value of further reducing adaptation parameters, and (iii) the contribution of nonlinear generation itself. We address these concerns in the following three parts.

W1: “Missing important references such as: Asymmetry in Low-Rank Adapters of Foundation Models (for Lora theory), as well as papers on metanetworks.”

R1: Thank you for pointing out these important references. We have added citations to Zhu et al.’s Asymmetry in Low-Rank Adapters of Foundation Models [1] and Lim et al.’s Graph Metanetworks for Processing Diverse Neural Architectures [2]. Section 2 (Related Work) discusses both works. The former analyzes the distinct roles of LoRA’s two factors, while the latter studies neural-network parameter-space modeling. GenLoRA addresses a complementary question: how to represent LoRA basis vectors through compact nonlinear functions, thereby reducing the parameter cost of increasing rank.

W2: “The methodology is simple and combines standard techniques, the theoretical section is trivial.”

R2: Thank you for raising these concerns about the methodological novelty and the role of our theoretical analysis. Our core contribution is a simple yet effective generative parameterization: lightweight nonlinear functions directly represent high-dimensional LoRA basis vectors, substantially reducing the parameter cost of increasing rank. The design retains a simple structure while demonstrating parameter efficiency and adaptation quality across tasks and architectures.

In addition, we designed two key mechanisms: grouping and normalization. Grouping allows different coordinate regions to use independent mappings, alleviating the restriction of fitting an entire high-dimensional vector with one function. Normalization controls the scale of the learnable latent inputs and aligns them with the effective response range of the fixed RBF grid. Figure 3(a) provides direct evidence: on LLaMA-3-8B and Math10K, under matched trainable parameter budgets, the full model achieves 70.17%, whereas removing normalization reduces accuracy to 63.21% and removing grouping reduces it to 66.05%—drops of 6.96 and 4.12 percentage points, respectively. Substantial drops also occur on the other two backbones. These results demonstrate the importance of the specific interaction between functional representation, grouping, and normalization for weight generation.

The theoretical analysis serves this change in parameterization. Replacing explicitly trainable matrices with matrices generated by nonlinear functions requires checking whether the low-rank structure is preserved and characterizing the gradients introduced by generation. The rank analysis establishes preservation of the low-rank update structure and compatibility with weight merging. The gradient analysis establishes local gradient boundedness under explicit conditions, including zero-variance normalization groups. Although concise, these results address necessary structural and local optimization questions raised by the new parameterization.

Revision location: Appendix L.1 (p. 32) discusses positioning, grouping, and normalization; Appendix L.4 (p. 34) clarifies the scope of the structural theory.

Please see Part II for the remaining responses.

AuthorsOpenReview ↗

Response to Reviewer UNZa [Part II]

W3: “Lora is already parameter efficient and saving a few million parameters doesnt make a difference in practice.”

R3: Thank you for highlighting the need to clarify the practical value of further reducing adaptation parameters. We would like to explain its practical significance from three perspectives.

First, LoRA’s trainable parameter count is not inherently limited to “a few million”; it scales with backbone dimensions, adaptation scope, and rank. For a set of target matrices T\mathcal{T}, the parameter count is

PLoRA=∑ℓ∈Trℓ(mℓ+nℓ)P_{\mathrm{LoRA}}=\sum_{\ell\in\mathcal{T}}r_{\ell}(m_{\ell}+n_{\ell})

Consequently, adapting the major attention and feed-forward projections of a larger backbone at a higher rank can require far more than a few million parameters.

Specifically, we tested LoRA and GenLoRA on Llama-3.1-70B [4] applying rank-128 adapters to the Q, K, V, O, Gate, Up, and Down projections in all 80 layers. In this experiment, we measured approximately 1.657B trainable parameters for LoRA, compared with 49.64M for GenLoRA using G=16G=16 groups per factor and K=15K=15 RBF centers. With the same backbone, target modules, and rank, GenLoRA therefore uses approximately 3.0% of LoRA’s trainable parameters.

Our existing experiments also demonstrate this compression trend. At matched r=32r=32, Table 10 shows that GenLoRA uses approximately 6.5%–11.4% of LoRA’s trainable parameters, or roughly one-tenth, while maintaining competitive accuracy. As backbone dimensions and adaptation scope increase, this relative compression translates into increasingly substantial absolute savings.

Second, our objective also includes enabling higher-rank adaptation within a limited adapter parameter budget. Table 18 provides a direct example on LLaMA-3-8B and Math10K: LoRA at r=8r=8 uses 4.72M parameters and achieves 67.28% accuracy, whereas GenLoRA at r=64r=64 uses 3.74M parameters and achieves 71.44%. Thus, GenLoRA supports eight times the configured rank with approximately 20.8% fewer parameters, improving accuracy by 4.16 percentage points. This illustrates the value of reducing the cost of rank expansion: a constrained parameter budget can support more effective adaptation.

Third, further reducing adaptation parameters is an established objective in PEFT research. For example, VeRA (ICLR 2024) compresses adaptation through shared random matrices and learned scaling vectors [5]; NOLA (ICLR 2024) further compresses LoRA through linear combinations of random bases [6]; and FourierFT (ICML 2024) reduces trainable parameters by learning a small set of spectral coefficients [7]. These works address parameter and storage costs in large-model and multi-task adaptation. GenLoRA contributes a distinct approach to this objective: compact nonlinear basis-vector generation that lowers the cost of increasing rank while retaining standard weight merging.

Revision location: Appendix L.2 (p. 33), including Eq. (42), presents the parameter-count analysis, the 70B test results, and the rank/budget comparison.

Please see Part III for the remaining response.

AuthorsOpenReview ↗

Response to Reviewer UNZa [Part III]

Q1: “To what extent is nonlinearity itself responsible for GenLoRA's parameter efficiency, as opposed to parameter sharing and the implicit regularization induced by its grouped basis parameterization? Could the authors compare GenLoRA against an equally parameter-efficient linear or polynomial generator, holding the latent dimensions, grouping structure, and trainable parameter budget fixed?”

R4: Thank you for suggesting controlled comparisons to clarify the contribution of nonlinearity beyond parameter sharing and grouped parameterization. We provide two complementary affine-generator controls testing nonlinear mapping from normalized latent coordinates to generated weights, using LLaMA-3-8B, Math10K, QKV adaptation, and r=8r=8.

We replace the entire SiLU-plus-Gaussian-RBF generation function with

fi,g(t)=ai,gt+bi,g,f_{i,g}(t)=a_{i,g}t+b_{i,g},

where tt is a normalized latent coordinate, ii indexes the basis vector, and gg the group. Each generator–group pair learns independent slope/intercept coefficients. Removing the SiLU branch too makes the replacement affine in tt. Both controls retain shared latents, latent dimensions, normalization, and A,BA,B assembly, isolating the value of nonlinear coordinate-to-weight mapping after normalization.

For both affine controls, B-side latents are randomly initialized, while B-side slopes and intercepts are zero; A-side latents and coefficients retain random, nonzero initialization. This gives B0=0B_0=0 and ΔW0=B0A0=0\Delta W_0=B_0A_0=0, while allowing B-side generator coefficients to start learning from the first step.

VariantWithin-group generatorGroups GGTrainable parametersAverage accuracy
GenLoRASiLU + Gaussian RBF160.983M70.17%
Same-structure affine controlat+bat+b160.639M58.45%
Budget-matched affine controlat+bat+b1280.983M63.76%

The same-structure control preserves G=16G=16 and the grouping structure, losing 11.72 percentage points. As it also uses fewer trainable coefficients, the budget-matched control increases the affine group count to G=128G=128, matching GenLoRA’s 0.983M parameters. Its 63.76% accuracy remains 6.41 percentage points below GenLoRA.

The comparisons are complementary: one fixes grouping; the other matches the budget through more independent affine groups. Neither affine control recovers GenLoRA’s performance, directly supporting nonlinear SiLU–RBF coordinate mapping within shared-latent generation beyond these alternatives’ grouping and parameter allocation.

To assess Gaussian RBFs specifically, we compare nonlinear generator families: Chebyshev and B-spline generators from Appendix H.4, Table 16, plus a rational generator with learnable numerator/denominator coefficients. Latent dimensions, r=8r=8, and G=16G=16 are fixed, with comparable trainable budgets:

GeneratorTrainable parametersAverage accuracy
Chebyshev polynomial0.95M64.01%
B-spline1.06M64.54%
Rational generator (additional experiment)0.97M62.93%
Gaussian RBF (GenLoRA)0.983M70.17%

Gaussian RBF improves over these alternatives by 6.16, 5.63, and 7.24 percentage points, respectively, and outperforms B-splines while using fewer parameters. We report the actual parameter counts explicitly and describe these nonlinear-family comparisons as approximately budget-matched.

Thus, affine controls support nonlinear generation; polynomial, spline, and rational controls support Gaussian RBFs within that design. Shared latent representation and grouping alone cannot explain the observed adaptation-quality differences.

Revision location: Appendix L.3, Tables 23 and 25, Eq. (43) (pp. 33–34).

References

[1] Zhu et al. Asymmetry in Low-Rank Adapters of Foundation Models. ICML 2024.

[2] Lim et al. Graph Metanetworks for Processing Diverse Neural Architectures. ICLR 2024.

[3] Dong et al. AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping. 2025.

[4] Dubey et al. The Llama 3 Herd of Models. 2024.

[5] Kopiczko et al. VeRA: Vector-based Random Matrix Adaptation. ICLR 2024.

[6] Koohpayegani et al. NOLA: Compressing LoRA using Linear Combination of Random Basis. ICLR 2024.

[7] Gao et al. Parameter-Efficient Fine-Tuning with Discrete Fourier Transform. ICML 2024.

Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns and clarify any doubts. Please let us know of any further questions.

Comments on the paper

2 comments

Dear Reviewers,

Thanks for your time and effort in reviewing this paper. The authors have responded to the comments by providing a rebuttal. Please read the rebuttal and indicate whether your concerns have been resolved. Thanks.

Regards from the AC.

AuthorsOpenReview ↗

Summary of Review and Discussion

Dear Area Chair and Reviewers,

Thank you for your time and constructive feedback. We briefly summarize the discussion, GenLoRA's contributions, and our responses to the reviews.

I. Review and Discussion

ReviewerInitial ratingCurrent ratingCurrent confidence
UNZa23 (Weak acceptance)4
SCEX34 (Clear acceptance)3
6iuM33 (Weak acceptance)2
93MN33 (Weak acceptance)3

Following our point-by-point rebuttal on November 12, two reviewers explicitly raised their ratings. On November 16, SCEX stated that the rebuttal had mainly addressed their major concerns and increased both their rating and confidence. On November 19, UNZa thanked us for the additional experiments and raised their rating from 2 to 3.

All four reviewers now assign positive ratings (3-4-3-3). At the time of this summary, 6iuM and 93MN have not provided post-rebuttal responses; their positive ratings remain unchanged. We have answered their questions and incorporated supporting analyses and reproducibility details into the revision.

II. Contributions of This Paper

GenLoRA replaces explicitly learned LoRA basis vectors with compact nonlinear generation from shared learnable latent vectors. Grouped radial basis function (RBF) generators and normalization reduce the parameter cost of constructing low-rank factors while retaining standard weight merging.

  • Strong parameter efficiency: At matched rank 32, GenLoRA uses approximately 6.5%-11.4% of LoRA's trainable parametersacross the reported reasoning settings, with competitive or better accuracy.
  • Higher rank within smaller budgets: On LLaMA-3-8B/Math10K, GenLoRA-r64 achieves 71.44% with 3.74M parameters, versus 67.28% with 4.72M for LoRA-r8: eight times the configured rank, 20.8% fewer parameters, and a 4.16-point accuracy gain.
  • Broad empirical support: Experiments span LLaMA, Gemma, and Qwen backbones, mathematical and commonsense reasoning, and code generation, with additional vision results and component ablations.
  • Structural compatibility: GenLoRA preserves the low-rank update and weight-merging structure. Our analysis characterizes its feature-space capacity, rank bounds, and local gradient boundedness under explicit conditions.

III. Reviewers' Evaluations and Our Responses

Recognized strengths:

  1. A practical generative parameterization compatible with standard LoRA updates and deployment [UNZa, SCEX, 6iuM, 93MN].
  2. Strong accuracy-parameter trade-offs [SCEX, 6iuM, 93MN].
  3. Broad evaluation across models and tasks, supported by component analyses [UNZa, SCEX, 6iuM, 93MN].

Main concerns and corresponding revisions:

  1. Novelty and the role of nonlinearity [UNZa, 6iuM]. We expanded related-work positioning and added affine, fixed-dictionary, and rational-generator controls. At matched trainable budgets, GenLoRA exceeds the affine and fixed-RBF-dictionary controls by 6.41 and 6.61 accuracy points, respectively, on LLaMA-3-8B/Math10K (Appendices L.1, L.3).
  2. Redundancy, rank, and expressive capacity [SCEX, 6iuM]. We clarified feature-space restrictions and numerical-rank measurements, and directly reconstructed 64 effective update matrices across two tasks. Using 81.25% fewer representation parameters, mean reconstruction errors are 2.9% and 1.6%; replacing all 32 query-projection updates per checkpoint reduces accuracy by only 0.39 and 0.12 points. We distinguish this empirical compressibility from a universal redundancy claim, as SCEX requested (Appendices L.4-L.6).
  3. Practical value and training efficiency [UNZa, SCEX]. We added a 70B parameter-count comparison, generator FLOPs, memory breakdowns, and throughput across input sizes. The profiled settings show 1.0%-4.7% additional step time and 175-224 MiB lower peak allocated memory. We explicitly separate adapter-parameter savings from total-memory and runtime effects (Appendices L.2, M).
  4. Implementation and evaluation details [93MN]. We corrected the initialization description to match the main experiments and released implementation, and documented the HumanEval+ framework revision, decoding settings, seeds, native task processing, and scoring protocol (Appendices K.3, N).

We appreciate the reviewers' constructive feedback and positive reassessments, and thank the Area Chair for encouraging further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.