acceptodds
Under review as a conference paper at ICLR 2027

GDPLoRA: Partitioned Low-Rank Adaptation with Gap-Driven Local Rank Allocation

Abstract

Low-Rank Adaptation (LoRA) applies a uniform rank to each weight projection, implicitly assuming that all parameters of a projection require equal adaptation capacity. We show that this assumption does not hold at a finer granularity: local submatrices within a single projection exhibit markedly different spectral concentration, so a monolithic low-rank factorization fits them unevenly. Crucially, this structural signal is already present in the frozen pretrained weights, before any fine-tuning. Building on this observation, we propose **Gap-Driven Partitioned LoRA (GDPLoRA)**, a fine-grained adaptation framework that tailors rank allocation to local weight sub-structures. GDPLoRA partitions each frozen weight matrix into functionally cohesive domains via direction-based clustering, then redistributes a fixed per-projection rank budget according to local spectral concentration. Because domains and rank assignments are determined offline from the frozen weights, GDPLoRA requires no input-dependent routing and preserves exact mergeability into the pretrained weights, leaving inference overhead unchanged. Experiments demonstrate that GDPLoRA outperforms competitive PEFT baselines across multiple backbones and tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PxPfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GDPLoRA partitions pretrained projections into channel groups, allocates a fixed total rank using their frozen-weight spectra, and initializes local adapters from singular vectors. The grouping and ranks remain fixed during training, allowing the resulting update to be merged without routing.

Critical strengths
  1. The local adaptation design is motivated and remains easy to deploy. Uneven reconstruction errors across channel groups motivate local adapters, and the construction preserves channel correspondence across coupled feed-forward projections. Fixed groups and ranks allow exact merging without routing.

  2. Comparisons and ablations support the main design choices. Tables 1–2 include GraLoRA and routed alternatives across RoBERTa-large and two 8B backbones at comparable reported parameter counts. Table 3 separately tests grouping, initialization, and allocation. These results support the combined method, although gains are not uniform, as Qwen's SVAMP result illustrates.

Critical weaknesses
  1. The spectral allocation rule remains only partly explained. Sections 3–4 acknowledge that the frozen-spectrum rationale is a hypothesis, and the uniform/reversed-allocation controls provide some support. It remains unclear whether the descriptor predicts which groups benefit from extra rank, particularly when spectral truncation depends on group shape. The evidence supports the tested heuristic more clearly than a general allocation principle.

  2. The small allocation advantage needs a more controlled comparison. With unequal row groups, parameter cost is ∑iri(din+dout,i)\sum_i r_i(d_{\mathrm{in}} + d_{\mathrm{out},i}), so fixed total rank does not ensure an equal parameter budget. Exact allocation-ablation counts are missing. Table 3's LRP/LRP advantage over HRP/HRP is only 0.34 points on a 100-point scale (0.8751 versus 0.8717), with three-seed means but no dispersion. Counts and paired seed-level results are needed to assess whether this difference reflects the allocation policy.

Questions for authors
  1. What are the exact parameter counts and paired seed-level differences for the allocation ablations? Does their ordering survive matching parameter costs?
  2. Does the preference for LRP/LRP also hold under the main RoBERTa configuration?
5 comments
AuthorsOpenReview ↗

Responses to Reviewer PxPf​​ (1/3)

We thank the reviewer for the detailed comments on the spectral allocation rule, the comparability of parameter budgets, and the allocation ablations. In response, we have:

  • clarified what the truncated spectral descriptor measures and provided marginal gradient-capture evidence while stating its limitations (W1).
  • reported exact trainable-parameter counts and paired seed-level differences for the R=64R=64, K=8K=8 allocation ablation (W2 & Q1).
  • repeated the allocation comparison under the main RoBERTa-large configuration (R=56R=56, K=4K=4), including seed-level results and actual parameter costs (W2 & Q2).

Below we respond to each concern in turn.

W1. Predictiveness of the spectral descriptor and sensitivity to group shape.

We thank the reviewer for this insightful comment. To address the concerns about the spectral descriptor's predictive value and sensitivity to truncation, we provide further mechanistic clarification and additional gradient-based evidence below.

(1) Mechanistic rationale and predictive evidence.

The effective-rank descriptor characterizes the spectral concentration of frozen pretrained weights, which does not directly determine the optimal rank for downstream adaptation. Our low-eRank prioritization (LRP) is motivated by a capacity-complementarity hypothesis: domains with more concentrated pretrained spectra may benefit from additional trainable adaptation capacity.

To investigate this hypothesis, we conducted a marginal gradient-capture analysis on RoBERTa-large across four GLUE tasks (CoLA, SST-2, MRPC, and RTE). Specifically, we measured the additional task-gradient energy captured when expanding the SVD-initialized rank from 8 to 12.

For FFN up-projections, the mean within-layer Spearman correlations were

\rho(e,Z_4)=-0.3642,$$ where $M_4$ measures marginal gradient capture and $Z_4$ normalizes it against a random-direction baseline. These negative correlations indicate that lower-eRank domains tend to capture more task-gradient energy through additional adaptation directions, providing localized evidence consistent with the capacity-complementarity hypothesis. However, marginal gradient capture does not directly establish larger downstream performance gains. **(2) Module-dependent behavior and empirical scope.** The negative correlations observed in FFN up-projections do not extend to attention projections, where Q/K/V exhibit positive correlations. This may reflect the interactions among Q, K, and V in computing attention weights and aggregating information, making the effect of additional adaptation capacity module-dependent. Therefore, we evaluate LRP and HRP jointly across QKVO and FFN rather than directly generalizing the FFN findings. We also acknowledge that the effective-rank descriptor is affected by spectral truncation and local group dimensions. Accordingly, we use it as a relative structural indicator within each projection, without assuming that it provides a shape-invariant estimate of the optimal adaptation rank. As detailed in our responses to Questions 1 and 2, **LRP/LRP achieves the highest average performance under both $R=64$, $K=8$ and the main $R=56$, $K=4$ configurations**, providing downstream empirical support for the joint allocation strategy. Together, these results provide mechanistic motivation and empirical support for LRP within the evaluated settings, while highlighting the need for further investigation into its generalizability across different architectures and tasks.
AuthorsOpenReview ↗

Responses to Reviewer PxPf​​ (2/3)

W2 & Q1. Exact parameter counts and paired seed-level differences.

We agree that a fixed total rank does not guarantee a fixed parameter budget when the row groups have unequal sizes. To address this concern, we report the exact trainable-parameter counts and paired seed-level performance differences for all four allocation strategies across three random seeds (41, 42, and 43), enabling a more transparent assessment of their performance and parameter efficiency.

We supplement the original R=64R=64, K=8K=8 allocation ablation with exact trainable-parameter counts and paired seed-level performance differences. The results show that LRP/LRP achieves the highest average performance and consistently outperforms the other three strategies across all three seeds, with mean paired improvements of 0.717, 0.629, and 0.338 percentage points over HRP/LRP, LRP/HRP, and HRP/HRP, respectively. The two-sided paired t-tests indicate statistically significant improvements over HRP/LRP (p=0.0213p=0.0213) and LRP/HRP (p=0.0125p=0.0125) at the 0.05 significance level, while the smaller improvement over HRP/HRP (p=0.0800p=0.0800).

QKVO / FFNSeed 41Seed 42Seed 43Mean ±\pm SDPaired Differences (41/42/43)Paired Gain ±\pm SDPaired t(2)t(2)pp-value
LRP / LRP87.46487.52887.53887.510 ± 0.040--------
HRP / LRP86.79686.96686.61886.793 ± 0.174+0.668 / +0.562 / +0.920+0.717 ± 0.1846.7500.0213
LRP / HRP86.97686.83886.82886.881 ± 0.083+0.488 / +0.690 / +0.710+0.629 ± 0.1238.8760.0125
HRP / HRP86.94287.35887.21587.172 ± 0.211+0.522 / +0.170 / +0.323+0.338 ± 0.1763.3200.0800
QKVO / FFNSeed 41Seed 42Seed 43Mean ParamsΔ ParamsRelative
LRP / LRP12,750,94712,754,64412,740,12512,748,572----
HRP / LRP12,796,28912,800,02812,788,91912,795,079+46,507+0.36%
LRP / HRP12,899,97112,896,23212,907,34112,901,181+152,609+1.20%
HRP / HRP12,945,31312,941,61612,956,13512,947,688+199,116+1.56%
AuthorsOpenReview ↗

Responses to Reviewer PxPf​​ (3/3)

W2 & Q2. Allocation under the main RoBERTa-large configuration.

To examine whether the preference for LRP/LRP also holds under the main RoBERTa-large configuration, we conducted additional allocation ablations with R=56R=56 and K=4K=4, evaluating all four QKVO/FFN allocation strategies across three random seeds (41, 42, and 43).The results confirm that the preference for LRP/LRP also holds under the main RoBERTa-large configuration. LRP/LRP achieves the highest average performance (87.455±0.28387.455\pm0.283) and outperforms all three alternative strategies in each random seed. Compared with HRP/LRP, LRP/HRP, and HRP/HRP, it improves the five-task average by 0.673, 2.061, and 0.292 percentage points, respectively.

Importantly, LRP/LRP also requires the fewest trainable parameters, with HRP/HRP using approximately 2.54% more parameters while achieving lower average performance. Together with the original R=64R=64, K=8K=8 ablation, these results demonstrate a consistent empirical preference for LRP/LRP across both evaluated configurations, supporting its effectiveness and parameter efficiency.

QKVO / FFNSeed 41Seed 42Seed 43Mean ±\pm SD
LRP / LRP87.47487.72887.16287.455 ± 0.283
HRP / LRP86.31487.22886.80486.782 ± 0.457
LRP / HRP86.93884.94284.30085.393 ± 1.376
HRP / HRP87.11087.64286.73687.163 ± 0.455
QKVO / FFNSeed 41Seed 42Seed 43Mean ParamsΔ ParamsRelative
LRP / LRP13,274,94813,263,39613,269,29713,269,214----
HRP / LRP13,306,02413,293,89613,302,66713,300,862+31,649+0.24%
LRP / HRP13,569,88413,582,01213,573,24113,575,046+305,832+2.30%
HRP / HRP13,600,96013,612,51213,606,61113,606,694+337,481+2.54%
Reviewer_2kEsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GDPLoRA, a fine-grained low-rank adaptation method motivated by the observation that different local regions within the same pretrained weight matrix exhibit substantially different spectral characteristics. Instead of applying a single uniform-rank LoRA update to the whole projection, GDPLoRA first partitions the frozen weight matrix into local domains using direction-based clustering, then characterizes the local spectral concentration of each domain and redistributes a fixed rank budget accordingly. Each domain is assigned an independent LoRA branch with its own rank, and the factors are initialized using local singular directions. Since the partitioning and rank allocation are determined offline from the frozen weights, the method introduces no dynamic routing and can be merged into the pretrained weights for zero-overhead inference. Experiments on RoBERTa-large, LLaMA-3-8B, and Qwen3-8B across language understanding, mathematical reasoning, code generation, and instruction-following tasks show consistent improvements over several existing PEFT baselines under comparable parameter budgets.

Critical strengths

The paper has a clear and well-motivated problem formulation. The authors provide convincing empirical evidence that pretrained weight matrices exhibit substantial local spectral heterogeneity, suggesting that a uniform-rank LoRA parameterization may not be optimal for all regions within the same projection. This observation provides a meaningful motivation for introducing fine-grained local adaptation and non-uniform rank allocation.

Critical weaknesses

The main weakness is that the novelty and justification of the proposed local rank allocation are not yet fully established. Closely related fine-grained LoRA methods such as G-LoRA, which also models within-matrix heterogeneity and allocates additional capacity to important local regions, are not discussed or directly compared. Moreover, GDPLoRA assigns larger ranks to regions with smaller effective-rank descriptors, but this choice is somewhat counterintuitive and is mainly supported by empirical ablation rather than a clear theoretical or mechanistic justification.

Questions for authors
  1. The comparison with closely related fine-grained LoRA methods appears incomplete. In particular, G-LoRA also explicitly models within-matrix parameter heterogeneity and allocates additional adaptation capacity to important local regions, but it is neither discussed nor included as a baseline. Given the close overlap in motivation, a direct comparison would be important for clarifying the novelty and empirical advantage of GDPLoRA.
  2. The rationale behind the proposed rank-allocation strategy is not fully convincing. GDPLoRA assigns larger ranks to local domains with smaller effective-rank descriptors, i.e., regions whose pretrained spectral energy is more concentrated in a few dominant directions. However, this choice is somewhat counterintuitive, since regions with more dispersed spectra may appear to require higher-rank adaptation capacity. Although the ablation results favor the proposed low-eRank prioritization strategy, the paper does not provide a principled explanation for why this allocation rule should generally be preferable. The current justification therefore appears largely empirical, and a stronger theoretical or mechanistic analysis would help support the proposed design.
5 comments
AuthorsOpenReview ↗

Responses to Reviewer 2kEs (1/2)

We appreciate the reviewer’s thoughtful comments on the novelty of GDPLoRA and the rationale behind its rank-allocation strategy. To address these concerns, we have:

  • expanded the Related Work section (Lines 124–128) to discuss G-LoRA, clarified its methodological differences from GDPLoRA, and added direct RoBERTa-large GLUE comparisons (W1 & Q1).
  • conducted additional experiments on LLaMA-3-8B using GSM8K and SVAMP to evaluate the effectiveness of GDPLoRA beyond encoder-based models (W1 & Q1).
  • provided a mechanistic explanation and additional gradient-capture analysis to support the capacity-complementarity hypothesis, complemented by joint allocation ablations (W2 & Q2).

Below, we address each concern in detail.

W1 & Q1. Novelty and direct comparison with G-LoRA.

We thank the reviewer for highlighting G-LoRA as a highly relevant fine-grained adaptation method. We agree that a direct comparison with G-LoRA is important for clarifying the novelty and empirical advantages of GDPLoRA. Accordingly, we have expanded the Related Work section (Lines 124–128) to discuss G-LoRA and clarify its methodological differences from GDPLoRA. We have also included G-LoRA as an additional baseline and conducted multi-seed evaluations and computational-cost comparisons. Below, we present the detailed methodological distinctions and experimental results.

(1) Methodological distinctions.

Both G-LoRA and GDPLoRA exploit within-matrix heterogeneity, but differ in their adaptation strategies. G-LoRA identifies important regions using gradient-based importance estimates and combines global and local adapters. In contrast, GDPLoRA constructs non-overlapping local domains from frozen pretrained weights and allocates adaptation ranks using spectral descriptors, without an additional global adapter. Thus, GDPLoRA's key distinction lies in its pretrained-spectrum-guided local-domain construction and rank allocation, rather than local adaptation itself.

(2) Direct RoBERTa-large GLUE comparison.

To provide a direct empirical comparison, we additionally evaluate G-LoRA under the same RoBERTa-large GLUE setting. The results are presented below.GDPLoRA achieves higher mean performance on all eight GLUE tasks, improving the overall average from 0.8803 to 0.8941 (+1.38 percentage points) while using approximately 1.3% fewer trainable parameters.

MethodParamsCoLASST-2MRPCMNLI-mmQNLIRTEQQPSTS-BAvg.
G-LoRA13.44M0.67630.94300.93210.89510.94060.84960.88380.92230.8803 ± 0.0026
GDPLoRA13.27M0.70400.96600.93460.90480.94880.87480.89330.92650.8941 ± 0.0017

(3) Mathematical Reasoning Evaluation on LLaMA-3-8B.

To evaluate GDPLoRA beyond encoder-based models, we further compare it with G-LoRA on LLaMA-3-8B using two mathematical reasoning benchmarks, GSM8K and SVAMP.GDPLoRA outperforms G-LoRA by 10.20 and 9.11 percentage points on GSM8K and SVAMP, respectively, while using 1.159M fewer trainable parameters. These results demonstrate a favorable performance–parameter trade-off and provide additional evidence of GDPLoRA's effectiveness on decoder-only language models.

MethodParams. (M)GSM8K ↑SVAMP ↑
G-LoRA298.84465.30267.556
GDPLoRA (Ours)297.68575.50076.670
AuthorsOpenReview ↗

Responses to Reviewer 2kEs (2/2)

W2 & Q2. Mechanistic justification for low-eRank prioritization.

We agree that a more dispersed pretrained spectrum generally requires a higher rank for accurate weight reconstruction. However, GDPLoRA keeps pretrained weights frozen and learns low-rank updates for task-specific adaptation. Therefore, pretrained effective rank does not directly determine the required adaptation rank. Our capacity-complementarity hypothesis suggests that domains with concentrated pretrained spectra may benefit from additional trainable directions.

GDPLoRA establishes a mathematical connection between pretrained spectral structure and adaptation capacity through SVD-based initialization. Specifically, the LoRA factors are initialized as Ai(0)=Vi,ri⊤A_i^{(0)}=V_{i,r_i}^{\top} and Bi(0)=0B_i^{(0)}=0, yielding the initial gradient ∇BiL∣0=ρGiVi,ri\nabla_{B_i}\mathcal{L}|_0=\rho G_iV_{i,r_i}, where GiG_i denotes the task-loss gradient. Increasing the rank introduces additional pretrained singular directions into the initial adaptation subspace, although this does not guarantee that lower-eRank domains always benefit more.

To further investigate this relationship, we conduct a marginal gradient-capture analysis on RoBERTa-large across four GLUE tasks (CoLA, SST-2, MRPC, and RTE) , measuring the gradient energy captured by four additional SVD directions when increasing the rank from 8 to 12. For FFN up-projections, the mean within-layer Spearman correlations are ρS(e,M4)=−0.3698\rho_{\mathrm S}(e,M_4)=-0.3698 and ρS(e,Z4)=−0.3642\rho_{\mathrm S}(e,Z_4)=-0.3642, indicating that lower-eRank domains tend to capture more marginal gradient energy from additional adaptation directions. This provides localized mechanistic evidence consistent with our capacity-complementarity hypothesis.

The same tendency does not uniformly extend to attention projections, where Q/K/V exhibit positive correlations. This may be because Q and K jointly determine which tokens receive attention, while V provides the information being aggregated, making the benefit of additional adaptation ranks dependent on their interactions rather than individual spectral properties alone. Nevertheless, joint QKVO/FFN ablations across five GLUE tasks and three seeds show that LRP/LRP achieves the highest average score (0.8751), outperforming HRP/HRP (0.8717), uniform allocation (0.8706), and both mixed configurations (0.8679 and 0.8688).

Together, these results establish a mathematical connection through SVD initialization, localized mechanistic evidence from FFN adaptation, and empirical support from joint allocation ablations. We therefore position low-eRank prioritization as an empirically effective pretrained-structure-guided allocation heuristic under the evaluated settings, rather than a universally optimal rank-allocation principle.

The additional experiments, particularly the direct comparisons with G-LoRA, have largely alleviated my concerns. I have no further questions regarding this issue.

Reviewer_fASeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GDPLoRA, a method that divides pretrained weight matrices into smaller groups and allocates different LoRA ranks based on their spectral properties. The authors use pretrained weights to determine the groups and ranks before fine-tuning, without requiring additional training data for rank allocation. They evaluate their method on RoBERTa-large, LLaMA-3-8B, and Qwen3-8B across several tasks. The results show some improvements over several LoRA baselines with comparable trainable parameters.

Critical strengths

I think the idea of allocating different ranks within the same weight matrix is interesting. The method is also evaluated on multiple models and tasks, and the authors provide ablation studies to support their approach.

Critical weaknesses
  1. The improvements over some baselines are relatively small, especially on GLUE. I would like to see more discussion of whether these improvements are significant.
  2. The method requires additional preprocessing, including clustering and SVD. But the authors claim no additional inference overhead, but I would like to see a clearer comparison of the total computational cost, including preprocessing.
Questions for authors
  1. Could the authors provide more information about the significance of the improvements?
  2. Could the authors compare the preprocessing and total training costs with standard LoRA and other baselines?
3 comments
AuthorsOpenReview ↗

Responses to Reviewer fASe​​ (1/2)

We appreciate the reviewer for the careful evaluation and constructive comments on the statistical reliability and computational efficiency of GDPLoRA. To address these concerns, we provide the following additional analyses and clarifications:

  • Seed-level statistical analysis (W1 & Q1): We conduct two-sided Welch's tt-tests across three random seeds to evaluate the significance of the GLUE improvements, with nominally significant gains over 9 of the 10 baselines (p<0.05p < 0.05).
  • Cross-task consistency (W1 & Q1): We provide pairwise task-level comparisons across eight GLUE tasks, showing that GDPLoRA outperforms each individual baseline on at least seven tasks.
  • Computational efficiency (W2 & Q2): We compare initialization overhead, fine-tuning runtime, GPU memory consumption, and inference latency against LoRA, G-LoRA, and MoELoRA across RoBERTa-large, Llama-3-8B, and Qwen3-8B.

We address each concern in detail below.

W1 & Q1. Statistical significance and robustness of GLUE improvements.

We thank the reviewer for raising concerns regarding the statistical significance and reliability of the performance improvements. To address these concerns, we provide additional seed-level statistical analysis and cross-task consistency comparisons to further assess the robustness of GDPLoRA's improvements.We first evaluate the statistical significance of GDPLoRA's performance improvements over ten baselines on eight GLUE tasks using three random seeds (41, 42, and 43).

GDPLoRA achieves the highest overall GLUE average score of 0.8941 ± 0.0017, outperforming all ten baselines in terms of mean performance. Moreover, GDPLoRA achieves a higher overall average score than every baseline under each of the three individual seeds, indicating that the observed improvements are not driven by a single favorable run.To further assess the statistical reliability of these improvements, we conduct exploratory two-sided Welch's tt-tests based on the seed-level average scores. The results show that GDPLoRA achieves nominally significant improvements over nine of the ten baselines (p<0.05p < 0.05).The complete statistical comparisons are summarized below:

BaselineImprovement (pp)Welch's pp-value
Full FT+0.740.0184
LoRA+0.450.0227
PiSSA+0.500.0313
DoRA+0.320.0605
AdaLoRA+0.870.0074
RandLoRA+0.670.0070
GraLoRA+0.480.0205
G-LoRA+1.380.0028
MoELoRA-Top1+0.520.0160
MoELoRA-Top4+0.410.0413

Beyond statistical testing of the overall GLUE average, we further examine whether the improvements are consistently distributed across different downstream tasks. The results shows that GDPLoRA achieves higher mean performance on at least seven of the eight GLUE tasks compared with each individual baseline.These results demonstrate that GDPLoRA achieves consistent performance improvements across diverse GLUE tasks.

BaselineWinsTiesLossesNon-winning TasksOne-Sided Exact Sign Test (pp, Ties Excluded)
Full FT701QQP0.0352
LoRA800None0.0039
PiSSA701MRPC0.0352
DoRA701MNLI-mm0.0352
AdaLoRA701MNLI-mm0.0352
RandLoRA701MRPC0.0352
GraLoRA800None0.0039
G-LoRA800None0.0039
MoELoRA-Top1701MRPC0.0352
MoELoRA-Top4710QNLI (Tie)0.0078
AuthorsOpenReview ↗

Responses to Reviewer fASe​​ (2/2)

W2 & Q2. Preprocessing, training cost, and inference overhead.

We thank the reviewer for raising this important question regarding the computational overhead introduced by GDPLoRA.To address these concerns, we conduct additional computational profiling experiments comparing GDPLoRA with four of the strongest competing PEFT baselines—standard LoRA, GraLoRA, G-LoRA, and MoELoRA on three model architectures: RoBERTa-large, Llama-3-8B, and Qwen3-8B.

Regarding inference, GDPLoRA introduces no additional inference-time operations, as its domain-specific low-rank updates can be merged into the pretrained weight matrices after fine-tuning, resulting in the same theoretical inference complexity as standard LoRA. Consistent with this property, GDPLoRA achieves nearly identical inference latency to LoRA, GraLoRA, and G-LoRA, with identical peak inference memory. Compared with MoELoRA, which retains dynamic expert routing, GDPLoRA achieves approximately 6.3× and 5.3× faster inference on Llama-3-8B and Qwen3-8B, respectively, while also requiring less GPU memory.

ModelMethodInference Latency (s)Memory (GiB)
Llama-3-8BLoRA2.82815.443
GraLoRA2.77015.443
G-LoRA2.84115.443
MoELoRA17.87116.988
GDPLoRA2.84115.443
Qwen3-8BLoRA3.84815.752
GraLoRA3.76315.752
G-LoRA3.90315.752
MoELoRA20.78517.289
GDPLoRA3.90015.752

We acknowledge that GDPLoRA introduces additional computational overhead due to its domain-specific preprocessing and training operations. We will explicitly discuss this limitation and clarify the trade-off between performance and computational efficiency in the revised manuscript. The detailed training costs are reported below.

Regarding initialization, GDPLoRA incurs additional overhead from weight-domain clustering, truncated SVD, and rank allocation. However, this preprocessing is performed only once before fine-tuning and does not need to be repeated during subsequent training or inference. Its initialization is slower than LoRA, GraLoRA, and MoELoRA, but consistently faster than G-LoRA, with 84.6% and 86.8% lower initialization time on Llama-3-8B and Qwen3-8B, respectively.Regarding fine-tuning, GDPLoRA reduces training runtime compared with MoELoRA by 29.7%, 42.2%, and 54.4% on RoBERTa-large, Llama-3-8B, and Qwen3-8B, respectively. It is slower than LoRA and GraLoRA due to additional domain-specific matrix operations and indexing overhead. Compared with G-LoRA, GDPLoRA has higher training runtime on RoBERTa-large and Llama-3-8B, but achieves a 16.0% reduction on Qwen3-8B. Notably, GDPLoRA achieves the lowest peak training memory among all five methods across the three architectures. Regarding total training time, GDPLoRA consistently requires less time than MoELoRA across all three models.

ModelMethodInit. (s)Init. Mem. (GiB)Step (s/step)Training (s)Total (h)Train Mem. (GiB)
RoBERTa-largeLoRA0.4721.3740.21622.7060.1614.271
GraLoRA0.3501.3740.31732.5960.2204.660
G-LoRA11.8191.4530.38538.7780.2845.829
MoELoRA0.2241.3730.70171.3790.5539.043
GDPLoRA9.0191.3940.49750.1680.3804.179
Llama-3-8BLoRA5.33216.1310.73944.27040.54020.393
GraLoRA4.00516.1310.94756.77750.12720.393
G-LoRA403.92417.3771.36785.87262.38320.951
MoELoRA2.85416.0883.105187.014139.31921.428
GDPLoRA62.07016.8841.785108.11680.41620.237
Qwen3-8BLoRA5.56116.4920.88653.48645.26721.150
GraLoRA4.32216.4920.98559.11663.40721.150
G-LoRA457.22117.5781.57398.76379.87321.323
MoELoRA3.14516.4363.021181.945151.14121.733
GDPLoRA60.58417.1751.37282.98367.08821.137

Dear Reviewer fASe

Thank you once again for your valuable feedback on our work. We would like to kindly remind you that we have provided a response to your remaining concerns, as noted above. With the revision deadline approaching, we would greatly appreciate it if you could take a moment to review our response and share any additional feedback or concerns. Your thoughtful comments have been instrumental in improving our work, and we want to ensure that we address any further suggestions you may have before the deadline. We look forward to continuing the discussion with you.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.