- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SHARE, a high-rank PEFT method that reuses a small number of trainable parameter banks across multiple layers and modules. Weight matrices with compatible update shapes share the same bank, which can also enable reuse of update construction and, in some cases, matrix multiplication.
The method is evaluated on commonsense reasoning, GSM8K, and HumanEval using Llama-2-7B, Llama-3-8B, and Qwen3-4B under matched trainable-parameter budgets. The paper also studies training efficiency, update rank, aggressive sharing, and layer-segment scaling.
Critical strengths
-
The empirical results are consistent across three backbone models and multiple task families under matched trainable-parameter budgets and multiple random seeds.
-
SHARE achieves strong predictive performance while maintaining competitive training throughput relative to other high-rank PEFT baselines. The implementation also exploits shared updates to reduce redundant computation.
Critical weaknesses
-
The conceptual novelty is moderate. Cross-layer parameter sharing has already been studied in PEFT, while compact high-rank adaptation has also been studied separately. SHARE mainly combines these two directions in a specific shared high-rank parameterization.
-
The experiments do not fully disentangle the effects of high-rank adaptation and cross-layer parameter sharing. Although the comparisons match the number of trainable parameters, they do not include a direct high-rank control with an otherwise comparable sharing mechanism, making it difficult to isolate the contribution of SHARE’s specific parameterization.
-
The scalability of the shared-bank design at larger parameter budgets is only lightly validated. Full-depth sharing imposes an upper bound on the size of each parameter bank, and the proposed layer-segment workaround is demonstrated only in a limited GSM8K experiment on Llama-2-7B.
Questions for authors
Could the authors compare against a straightforward shared-parameter version of a strong high-rank baseline such as HiRA or MoSA? This would help clarify whether the gain comes from SHARE’s specific parameterization or from applying cross-layer sharing to high-rank adaptation more generally.
3 comments
Thank you for the detailed response and additional experiments. The scalability results are helpful.
However, I still have some concerns about the methodological novelty and the lack of a direct comparison with shared high-rank baselines. I will maintain my current rating.
Dear Reviewer pErG,
Thank you for your follow-up and thoughtful feedback. We appreciate your acknowledgment of the additional scalability results. To further address your question, we conducted an additional Shared-HiRA experiment on Llama-2-7B commonsense reasoning. Shared-HiRA shares low-rank factors across layers within each projection type while maintaining the same trainable parameter budget. Unlike SHARE, it still produces distinct high-rank updates through element-wise modulation by each layer's pretrained weights.
| Method | BoolQ | PIQA | SIQA | ARC-C | ARC-E | OBQA | HellaSwag | WinoG | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Shared-HiRA | 72.52 | 84.83 | 79.93 | 72.75 | 87.67 | 84.28 | 93.56 | 85.49 | 82.63 |
| SHARE | 74.06 | 86.41 | 80.72 | 76.08 | 89.33 | 85.85 | 94.44 | 85.69 | 84.07 |
Under the matched parameter budget, SHARE achieves 84.07% average accuracy, compared with 82.63% for Shared-HiRA.
We hope this direct comparison helps clarify the distinction between sharing factors that generate high-rank updates and directly sharing the resulting updates. Thank you again for your time and consideration.
Best regards,
The Authors
We thank the reviewer for the constructive feedback.
W1. Conceptual novelty
We appreciate the concern. SHARE differs from existing approaches not merely by combining high-rank adaptation and parameter sharing, but in what is shared and how the adaptation budget is utilized.
What is shared. While existing sharing-based PEFT methods primarily reduce parameter or storage costs by sharing low-rank components, SHARE directly reuses entire high-rank updates across layers and modules. This enables computational reuse without independently constructing an update for every module.
How the adaptation budget is utilized. Under a fixed parameter budget, SHARE uses more parameters to construct each shared update while reducing the need for separate parameters across modules. Importantly, this design does not require modules to perform similar functions. SHARE groups weight matrices based on their shapes, allowing the same update to induce distinct effects through different hidden representations and pretrained computations.
Our experiments support the feasibility of this design. Table 8 demonstrates robustness to fixed-coefficient choices, while the Single-Bank experiment achieves comparable accuracy (83.75% vs. 84.07%) even when parameters are shared across heterogeneous modules through different fixed assignments.
Thus, SHARE introduces a distinct parameterization that combines high-rank adaptation, direct reuse of complete updates, and effective sharing across modules with different functions.
W2 & Q1. he Role of Parameter Sharing in High-Rank Adaptation
We appreciate this important question. SHARE goes beyond combining high-rank adaptation with parameter sharing by directly constraining how weight updates are shared across modules. We clarify this distinction and discuss the supporting experimental evidence.
Distinction from Shared-HiRA. A straightforward Shared-HiRA variant would share low-rank factors across layers, while each module would still construct a different high-rank update through element-wise modulation by its pretrained weights. In contrast, SHARE directly shares the resulting weight updates, up to fixed scaling or transposition. Specifically, modules sharing the same bank satisfy Rs−1(ΔWs)/αs=Db, where vec(Db)=Hbθb. This imposes a direct constraint on weight updates, rather than only on the factors used to generate them. As formalized in Appendix B.2, this can be interpreted as the hard-constraint limit of an update-consistency penalty. Under a fixed parameter budget, SHARE also allocates more parameters to each shared update instead of learning separate updates for individual modules.
Evidence beyond high-rank adaptation. HiRA and MoSA already produce high-rank updates, yet SHARE achieves stronger performance under matched parameter budgets. Figure 2 confirms that SHARE preserves high-rank updates despite extensive sharing, while Figure 3 demonstrates its effectiveness under small parameter budgets. More importantly, Figure 4 examines the role of applying shared updates across layers while keeping SHARE's parameterization and parameter budget unchanged. As the forward drop probability increases from 0 to 0.75, accuracy decreases from 84.07% to 81.80%. This trend is consistent with the importance of using shared updates across multiple layers during training, rather than relying on high-rank capacity alone.
W3. Scalability to larger parameter budgets
We appreciate this concern and extend the layer-segment experiments to both Llama-2-7B and Llama-3-8B on GSM8K and HumanEval:
The results show different trends across the two models:
These results extend our scalability evaluation beyond a single model and task.