- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GDPLoRA partitions pretrained projections into channel groups, allocates a fixed total rank using their frozen-weight spectra, and initializes local adapters from singular vectors. The grouping and ranks remain fixed during training, allowing the resulting update to be merged without routing.
Critical strengths
-
The local adaptation design is motivated and remains easy to deploy. Uneven reconstruction errors across channel groups motivate local adapters, and the construction preserves channel correspondence across coupled feed-forward projections. Fixed groups and ranks allow exact merging without routing.
-
Comparisons and ablations support the main design choices. Tables 1–2 include GraLoRA and routed alternatives across RoBERTa-large and two 8B backbones at comparable reported parameter counts. Table 3 separately tests grouping, initialization, and allocation. These results support the combined method, although gains are not uniform, as Qwen's SVAMP result illustrates.
Critical weaknesses
-
The spectral allocation rule remains only partly explained. Sections 3–4 acknowledge that the frozen-spectrum rationale is a hypothesis, and the uniform/reversed-allocation controls provide some support. It remains unclear whether the descriptor predicts which groups benefit from extra rank, particularly when spectral truncation depends on group shape. The evidence supports the tested heuristic more clearly than a general allocation principle.
-
The small allocation advantage needs a more controlled comparison. With unequal row groups, parameter cost is , so fixed total rank does not ensure an equal parameter budget. Exact allocation-ablation counts are missing. Table 3's LRP/LRP advantage over HRP/HRP is only 0.34 points on a 100-point scale (0.8751 versus 0.8717), with three-seed means but no dispersion. Counts and paired seed-level results are needed to assess whether this difference reflects the allocation policy.
Questions for authors
- What are the exact parameter counts and paired seed-level differences for the allocation ablations? Does their ordering survive matching parameter costs?
- Does the preference for LRP/LRP also hold under the main RoBERTa configuration?
5 comments
Responses to Reviewer PxPf (2/3)
W2 & Q1. Exact parameter counts and paired seed-level differences.
We agree that a fixed total rank does not guarantee a fixed parameter budget when the row groups have unequal sizes. To address this concern, we report the exact trainable-parameter counts and paired seed-level performance differences for all four allocation strategies across three random seeds (41, 42, and 43), enabling a more transparent assessment of their performance and parameter efficiency.
We supplement the original , allocation ablation with exact trainable-parameter counts and paired seed-level performance differences. The results show that LRP/LRP achieves the highest average performance and consistently outperforms the other three strategies across all three seeds, with mean paired improvements of 0.717, 0.629, and 0.338 percentage points over HRP/LRP, LRP/HRP, and HRP/HRP, respectively. The two-sided paired t-tests indicate statistically significant improvements over HRP/LRP () and LRP/HRP () at the 0.05 significance level, while the smaller improvement over HRP/HRP ().
| QKVO / FFN | Seed 41 | Seed 42 | Seed 43 | Mean SD | Paired Differences (41/42/43) | Paired Gain SD | Paired | -value |
|---|---|---|---|---|---|---|---|---|
| LRP / LRP | 87.464 | 87.528 | 87.538 | 87.510 ± 0.040 | -- | -- | -- | -- |
| HRP / LRP | 86.796 | 86.966 | 86.618 | 86.793 ± 0.174 | +0.668 / +0.562 / +0.920 | +0.717 ± 0.184 | 6.750 | 0.0213 |
| LRP / HRP | 86.976 | 86.838 | 86.828 | 86.881 ± 0.083 | +0.488 / +0.690 / +0.710 | +0.629 ± 0.123 | 8.876 | 0.0125 |
| HRP / HRP | 86.942 | 87.358 | 87.215 | 87.172 ± 0.211 | +0.522 / +0.170 / +0.323 | +0.338 ± 0.176 | 3.320 | 0.0800 |
| QKVO / FFN | Seed 41 | Seed 42 | Seed 43 | Mean Params | Δ Params | Relative |
|---|---|---|---|---|---|---|
| LRP / LRP | 12,750,947 | 12,754,644 | 12,740,125 | 12,748,572 | -- | -- |
| HRP / LRP | 12,796,289 | 12,800,028 | 12,788,919 | 12,795,079 | +46,507 | +0.36% |
| LRP / HRP | 12,899,971 | 12,896,232 | 12,907,341 | 12,901,181 | +152,609 | +1.20% |
| HRP / HRP | 12,945,313 | 12,941,616 | 12,956,135 | 12,947,688 | +199,116 | +1.56% |
Responses to Reviewer PxPf (3/3)
W2 & Q2. Allocation under the main RoBERTa-large configuration.
To examine whether the preference for LRP/LRP also holds under the main RoBERTa-large configuration, we conducted additional allocation ablations with and , evaluating all four QKVO/FFN allocation strategies across three random seeds (41, 42, and 43).The results confirm that the preference for LRP/LRP also holds under the main RoBERTa-large configuration. LRP/LRP achieves the highest average performance () and outperforms all three alternative strategies in each random seed. Compared with HRP/LRP, LRP/HRP, and HRP/HRP, it improves the five-task average by 0.673, 2.061, and 0.292 percentage points, respectively.
Importantly, LRP/LRP also requires the fewest trainable parameters, with HRP/HRP using approximately 2.54% more parameters while achieving lower average performance. Together with the original , ablation, these results demonstrate a consistent empirical preference for LRP/LRP across both evaluated configurations, supporting its effectiveness and parameter efficiency.
| QKVO / FFN | Seed 41 | Seed 42 | Seed 43 | Mean SD |
|---|---|---|---|---|
| LRP / LRP | 87.474 | 87.728 | 87.162 | 87.455 ± 0.283 |
| HRP / LRP | 86.314 | 87.228 | 86.804 | 86.782 ± 0.457 |
| LRP / HRP | 86.938 | 84.942 | 84.300 | 85.393 ± 1.376 |
| HRP / HRP | 87.110 | 87.642 | 86.736 | 87.163 ± 0.455 |
| QKVO / FFN | Seed 41 | Seed 42 | Seed 43 | Mean Params | Δ Params | Relative |
|---|---|---|---|---|---|---|
| LRP / LRP | 13,274,948 | 13,263,396 | 13,269,297 | 13,269,214 | -- | -- |
| HRP / LRP | 13,306,024 | 13,293,896 | 13,302,667 | 13,300,862 | +31,649 | +0.24% |
| LRP / HRP | 13,569,884 | 13,582,012 | 13,573,241 | 13,575,046 | +305,832 | +2.30% |
| HRP / HRP | 13,600,960 | 13,612,512 | 13,606,611 | 13,606,694 | +337,481 | +2.54% |
Responses to Reviewer PxPf (1/3)
We thank the reviewer for the detailed comments on the spectral allocation rule, the comparability of parameter budgets, and the allocation ablations. In response, we have:
Below we respond to each concern in turn.
We thank the reviewer for this insightful comment. To address the concerns about the spectral descriptor's predictive value and sensitivity to truncation, we provide further mechanistic clarification and additional gradient-based evidence below.
(1) Mechanistic rationale and predictive evidence.
The effective-rank descriptor characterizes the spectral concentration of frozen pretrained weights, which does not directly determine the optimal rank for downstream adaptation. Our low-eRank prioritization (LRP) is motivated by a capacity-complementarity hypothesis: domains with more concentrated pretrained spectra may benefit from additional trainable adaptation capacity.
To investigate this hypothesis, we conducted a marginal gradient-capture analysis on RoBERTa-large across four GLUE tasks (CoLA, SST-2, MRPC, and RTE). Specifically, we measured the additional task-gradient energy captured when expanding the SVD-initialized rank from 8 to 12.
For FFN up-projections, the mean within-layer Spearman correlations were
\rho(e,Z_4)=-0.3642,$$ where $M_4$ measures marginal gradient capture and $Z_4$ normalizes it against a random-direction baseline. These negative correlations indicate that lower-eRank domains tend to capture more task-gradient energy through additional adaptation directions, providing localized evidence consistent with the capacity-complementarity hypothesis. However, marginal gradient capture does not directly establish larger downstream performance gains. **(2) Module-dependent behavior and empirical scope.** The negative correlations observed in FFN up-projections do not extend to attention projections, where Q/K/V exhibit positive correlations. This may reflect the interactions among Q, K, and V in computing attention weights and aggregating information, making the effect of additional adaptation capacity module-dependent. Therefore, we evaluate LRP and HRP jointly across QKVO and FFN rather than directly generalizing the FFN findings. We also acknowledge that the effective-rank descriptor is affected by spectral truncation and local group dimensions. Accordingly, we use it as a relative structural indicator within each projection, without assuming that it provides a shape-invariant estimate of the optimal adaptation rank. As detailed in our responses to Questions 1 and 2, **LRP/LRP achieves the highest average performance under both $R=64$, $K=8$ and the main $R=56$, $K=4$ configurations**, providing downstream empirical support for the joint allocation strategy. Together, these results provide mechanistic motivation and empirical support for LRP within the evaluated settings, while highlighting the need for further investigation into its generalizability across different architectures and tasks.