acceptodds
Under review as a conference paper at ICLR 2027

Share Once, Adapt Everywhere: High-Rank Adaptation Across Layers and Modules

Abstract

Recent high-rank parameter-efficient fine-tuning (PEFT) methods improve the expressiveness of weight updates while retaining a separate set of trainable parameters for each target weight matrix. This leaves open whether high-rank adaptation can accommodate more extensive parameter sharing across layers and modules. We introduce SHARE, a high-rank adaptation method that organizes target weight matrices according to their update shapes and reuses a small set of trainable parameter banks across layers and modules. Weight matrices assigned to the same bank reuse the corresponding high-rank update. We provide a first-order analysis showing that fixed parameter sharing defines a reparameterization of the weight updates and derive the resulting update to the model weights. The shared structure also allows each high-rank update to be constructed once and reused by all target weight matrices assigned to the same bank during training. Extensive experiments on commonsense reasoning, mathematical reasoning, and code generation across multiple backbone models show that SHARE consistently outperforms strong PEFT baselines, while achieving competitive training throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Weight Update Reuse, High-rank adaption

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_pErGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SHARE, a high-rank PEFT method that reuses a small number of trainable parameter banks across multiple layers and modules. Weight matrices with compatible update shapes share the same bank, which can also enable reuse of update construction and, in some cases, matrix multiplication.

The method is evaluated on commonsense reasoning, GSM8K, and HumanEval using Llama-2-7B, Llama-3-8B, and Qwen3-4B under matched trainable-parameter budgets. The paper also studies training efficiency, update rank, aggressive sharing, and layer-segment scaling.

Critical strengths
  1. The empirical results are consistent across three backbone models and multiple task families under matched trainable-parameter budgets and multiple random seeds.

  2. SHARE achieves strong predictive performance while maintaining competitive training throughput relative to other high-rank PEFT baselines. The implementation also exploits shared updates to reduce redundant computation.

Critical weaknesses
  1. The conceptual novelty is moderate. Cross-layer parameter sharing has already been studied in PEFT, while compact high-rank adaptation has also been studied separately. SHARE mainly combines these two directions in a specific shared high-rank parameterization.

  2. The experiments do not fully disentangle the effects of high-rank adaptation and cross-layer parameter sharing. Although the comparisons match the number of trainable parameters, they do not include a direct high-rank control with an otherwise comparable sharing mechanism, making it difficult to isolate the contribution of SHARE’s specific parameterization.

  3. The scalability of the shared-bank design at larger parameter budgets is only lightly validated. Full-depth sharing imposes an upper bound on the size of each parameter bank, and the proposed layer-segment workaround is demonstrated only in a limited GSM8K experiment on Llama-2-7B.

Questions for authors

Could the authors compare against a straightforward shared-parameter version of a strong high-rank baseline such as HiRA or MoSA? This would help clarify whether the gain comes from SHARE’s specific parameterization or from applying cross-layer sharing to high-rank adaptation more generally.

3 comments

We thank the reviewer for the constructive feedback.


W1. Conceptual novelty

We appreciate the concern. SHARE differs from existing approaches not merely by combining high-rank adaptation and parameter sharing, but in what is shared and how the adaptation budget is utilized.

  • What is shared. While existing sharing-based PEFT methods primarily reduce parameter or storage costs by sharing low-rank components, SHARE directly reuses entire high-rank updates across layers and modules. This enables computational reuse without independently constructing an update for every module.

  • How the adaptation budget is utilized. Under a fixed parameter budget, SHARE uses more parameters to construct each shared update while reducing the need for separate parameters across modules. Importantly, this design does not require modules to perform similar functions. SHARE groups weight matrices based on their shapes, allowing the same update to induce distinct effects through different hidden representations and pretrained computations.

Our experiments support the feasibility of this design. Table 8 demonstrates robustness to fixed-coefficient choices, while the Single-Bank experiment achieves comparable accuracy (83.75% vs. 84.07%) even when parameters are shared across heterogeneous modules through different fixed assignments.

Thus, SHARE introduces a distinct parameterization that combines high-rank adaptation, direct reuse of complete updates, and effective sharing across modules with different functions.


W2 & Q1. he Role of Parameter Sharing in High-Rank Adaptation

We appreciate this important question. SHARE goes beyond combining high-rank adaptation with parameter sharing by directly constraining how weight updates are shared across modules. We clarify this distinction and discuss the supporting experimental evidence.

  • Distinction from Shared-HiRA. A straightforward Shared-HiRA variant would share low-rank factors across layers, while each module would still construct a different high-rank update through element-wise modulation by its pretrained weights. In contrast, SHARE directly shares the resulting weight updates, up to fixed scaling or transposition. Specifically, modules sharing the same bank satisfy Rs−1(ΔWs)/αs=DbR_s^{-1}(\Delta W_s)/\alpha_s=D_b, where vec⁡(Db)=Hbθb\operatorname{vec}(D_b)=H_b\theta_b. This imposes a direct constraint on weight updates, rather than only on the factors used to generate them. As formalized in Appendix B.2, this can be interpreted as the hard-constraint limit of an update-consistency penalty. Under a fixed parameter budget, SHARE also allocates more parameters to each shared update instead of learning separate updates for individual modules.

  • Evidence beyond high-rank adaptation. HiRA and MoSA already produce high-rank updates, yet SHARE achieves stronger performance under matched parameter budgets. Figure 2 confirms that SHARE preserves high-rank updates despite extensive sharing, while Figure 3 demonstrates its effectiveness under small parameter budgets. More importantly, Figure 4 examines the role of applying shared updates across layers while keeping SHARE's parameterization and parameter budget unchanged. As the forward drop probability increases from 0 to 0.75, accuracy decreases from 84.07% to 81.80%. This trend is consistent with the importance of using shared updates across multiple layers during training, rather than relying on high-rank capacity alone.


W3. Scalability to larger parameter budgets

We appreciate this concern and extend the layer-segment experiments to both Llama-2-7B and Llama-3-8B on GSM8K and HumanEval:

Reference RankLlama-2 GSM8KLlama-2 HumanEvalLlama-3 GSM8KLlama-3 HumanEval
3259.6727.2479.1959.76
6461.2928.0579.3960.37
12861.6731.1079.4759.14

The results show different trends across the two models:

  • Llama-2-7B: Increasing the budget improves performance on both tasks, with gains of 2.00 and 3.86 percentage points at reference rank 128.
  • Llama-3-8B: Performance exhibits diminishing returns as the budget increases, potentially reflecting reduced benefits from additional adaptation capacity or overfitting.

These results extend our scalability evaluation beyond a single model and task.

Thank you for the detailed response and additional experiments. The scalability results are helpful.

However, I still have some concerns about the methodological novelty and the lack of a direct comparison with shared high-rank baselines. I will maintain my current rating.

Dear Reviewer pErG,

Thank you for your follow-up and thoughtful feedback. We appreciate your acknowledgment of the additional scalability results. To further address your question, we conducted an additional Shared-HiRA experiment on Llama-2-7B commonsense reasoning. Shared-HiRA shares low-rank factors across layers within each projection type while maintaining the same trainable parameter budget. Unlike SHARE, it still produces distinct high-rank updates through element-wise modulation by each layer's pretrained weights.

MethodBoolQPIQASIQAARC-CARC-EOBQAHellaSwagWinoGAvg.
Shared-HiRA72.5284.8379.9372.7587.6784.2893.5685.4982.63
SHARE74.0686.4180.7276.0889.3385.8594.4485.6984.07

Under the matched parameter budget, SHARE achieves 84.07% average accuracy, compared with 82.63% for Shared-HiRA.

We hope this direct comparison helps clarify the distinction between sharing factors that generate high-rank updates and directly sharing the resulting updates. Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_TAvTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a parameter-efficient fine-tuning method which combines high-rank adaptation and parameter sharing. The proposed method shares weight updates across layers using by creating "parameter banks". This results in improved downstream task performance as well as computational throughput during training.

Critical strengths
  • High-rank adaptation is an interesting direction to fine-tuning and seems to yield more performant/memory-efficient adaptation. Combining this with parameter sharing is a natural extension.

  • Empirical results are quite strong, with almost dominant performance in every tested benchmark.

  • Parameterization and caching scheme seems relatively straightforward, with a simple first-order analysis.

Critical weaknesses
  • The baselines are limited to a somewhat limited set with fixed rank updates. I understand that the experiments are focusing on matched parameter counts, but at the end of the day if another method gives better performance for a moderate increase in parameters/memory, then that method would be preferred. E.g. the paper discusses VeRA and ShareLoRA as motivating parameter-sharing baselines yet these are not compared against. This also matters because the proposed method requires an extra ~1GB of memory for caching which is not afforded to other methods (and also renders the parameter count matching moot).

  • The paper would benefit from an ablation showing that the efficiency gains in Table 3 come from parameter sharing. Currently, the method under-performs LoRA, out-performs MoSA, and matches HiRA. But each method has different parameterizations/implementations, so it's not clear what is influencing throughput. Ablations would help disentangle where the efficiency/performance gains are coming from. This may also show where the additional downstream performance gains come from. Why does parameter sharing improve performance?

  • There is limited analysis here investigating where the downstream task improvements come from. If sharing is primarily motivated as a way to reduce redundant computations, why does this improve task performance so much?

Somewhat minor: this is a fast moving field, but there are quite a few other high-rank adaptation methods in the past year that should probably be considered, e.g. [1-3]

[1] Gurung, A. and Campbell, J.. Hyperadapt: Simple high-rank adaptation. TMLR 2026.

[2] Wang, Y., Meng, F., Zhang, X., Jiang, F., Tang, P. and Zhang, M.. Hd-pissa: High-rank distributed orthogonal adaptation. EMNLP 2025.

[3] Chen, Z., Dangovski, R., Loh, C., Dugan, O., Luo, D. and Soljačić, M.. Quanta: Efficient high-rank fine-tuning of llms with quantum-informed tensor adaptation. Neurips 2024.

Questions for authors
  1. Where does the improvement in downstream task performance come from? How does parameter sharing improve this?
1 comment

We thank the reviewer for the constructive feedback and insightful questions.


W1. Baseline comparisons and memory efficiency

We appreciate the reviewer's suggestion. Existing sharing-based PEFT methods, such as VeRA, ShareLoRA, and VB-LoRA, primarily leverage parameter sharing to reduce trainable parameters or storage costs while retaining low-rank adaptation. In contrast, SHARE investigates how parameter sharing can improve adaptation under a fixed parameter budget.

We additionally evaluate VB-LoRA, which uses globally shared vector banks, under our experimental setting:

MethodLlama-2 GSM8KLlama-2 HumanEvalLlama-3 GSM8KLlama-3 HumanEval
VB-LoRA49.4017.6871.9550.61
SHARE59.6727.2479.1959.76

SHARE outperforms VB-LoRA across all four settings.

Regarding memory efficiency, we acknowledge SHARE's additional caching overhead. However, SHARE achieves strong performance even with smaller trainable-parameter budgets. On Llama-3-8B commonsense reasoning, SHARE with a rank-16 reference budget achieves 87.49%, surpassing HiRA with a rank-32 budget (86.73%).

This highlights SHARE's parameter efficiency, with smaller budgets also reducing trainable-parameter and optimizer-state memory, although caching introduces a separate memory cost.


W2. Where do the efficiency gains come from?

We appreciate this suggestion and conducted an additional ablation on Llama-2-7B, progressively enabling the computational reuse mechanisms while keeping the SHARE parameterization unchanged.

ConfigurationStep Time (ms)Speedup
Naive (no reuse)3769.481.00×
+ Shared-update caching1316.292.86×
+ QKV matrix-multiplication reuse1005.603.75×

The results isolate the contributions of the two reuse mechanisms:

  • Shared-update caching avoids redundant update construction, reducing step time from 3769.48 ms to 1316.29 ms (2.86× speedup).
  • QKV matrix-multiplication reuse further reduces step time to 1005.60 ms, providing an additional 1.31× speedup.

Together, these optimizations achieve a 3.75× speedup over the naive implementation.


W3 & Q1. Why does parameter sharing improve downstream performance?

We believe SHARE's effectiveness can be understood through two complementary properties: structural regularization induced by parameter sharing and finer-grained adaptation under a fixed parameter budget.

  • Structural regularization. Parameter sharing introduces a structural inductive bias rather than merely reducing parameter count. In SHARE, the fixed parameterization Δw=Pθ\Delta w=P\theta restricts updates to a shared subspace, while ∇θL=P⊤g\nabla_\theta\mathcal{L}=P^\top g combines gradient contributions from different layers and modules. This coupling may discourage unnecessary module-specific changes and favor adaptations that accommodate multiple computational contexts. Importantly, sharing the same update does not imply identical functional behavior, since modules operate within distinct pretrained computations.

  • Adaptation capacity. For NN same-shaped matrices and KK trainable parameters, independent adapters allocate approximately K/NK/N parameters per matrix, whereas SHARE uses KK parameters to construct a shared high-rank update reused across all NN matrices. Thus, SHARE allows more parameters to construct each shared update without increasing the total parameter budget, while limiting independent adaptation across modules.

Our experiments are consistent with this interpretation:

  • Figure 3 shows that SHARE with a rank-1 reference budget nearly matches rank-32 LoRA performance (80.69% vs. 80.73% on Llama-2-7B), suggesting effective use of limited adaptation capacity.
  • Figure 2 confirms that extensive sharing preserves high-rank updates. The Single-Bank experiment further achieves comparable accuracy (83.75% vs. 84.07%) despite sharing parameters across heterogeneous modules through different fixed assignments.

Together, these findings suggest that structural regularization and finer-grained adaptation may jointly contribute to SHARE's strong downstream performance.

Reviewer_bfWqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • As opposed to learning a layer specific ΔW\Delta W, SHARE proposes a PEFT framework where the pretrained weights of the same dimensions and the pretrained weights that have the same dimensions after being transposed (m×nm \times n or n×mn \times m) share the learnable parameters through a parameter bank.
  • Each parameter bank bb provides a matrix Db∈Rmb×nbD_b \in \mathbb{R}^{m_b \times n_b} shared across all the weight matrices assigned to that bank. The bank assigned to the index ss defines an operator Rs(D):Rmb×nb→Rdsout×dsin\mathcal{R}_s(D): \mathbb{R}^{m_b \times n_b} \rightarrow \mathbb{R}^{d_s^{out} \times d_s^{in}}
  • Each bank maintains a trainable vector θb=[θb,1,...,θb,Kb]∈RKb\theta_b = [\theta_{b,1},...,\theta_{b,K_b}] \in \mathbb{R}^{K_b}. The matrix DbD_b is constructed by assigning an element of the vector θb\theta_b to a predetermined index (i,j)(i,j) of DbD_b.
  • The residual matrix ΔWs\Delta W_s, which is shared among the weights of the same shape ss is obtained as ΔWs=αsRs(Db(s))\Delta W_s = \alpha_s\mathcal{R}_s(D_{b(s)}), where αs\alpha_s is a fixed coefficient.
  • (D)\mathcal(D) is implemented as vec(ΔWs)=αsRsHbθbvec(\Delta W_s) = \alpha_sR_sH_b\theta_b, where vec()vec() vectorizes the matrix by stacking its columns, and, HbH_b is a fixed assignment matrix that assigns elements from θb\theta_b, and RsR_s is a permutation matrix.
  • The method is evaluated on commonsense reasoning, arithmetic reasoning and code generation using Llama 2 (7B), Llama 3 (8B) and Qwen 3 (4B).
Critical strengths
  • SHARE produced high rank updates under the same parameter budget, which is interesting.
  • The first order analysis and the result that the parameter update is proportional to the projected full fine-tuning direction is interesting.
Critical weaknesses
  1. The baselines look untuned. The difference between LoRA and SHARE for all the benchmarks is too large. In my experience, this is not possible once LoRA is tuned well. The paper also does not mention the hyperparameter tuning strategy for the baselines, which is suspicious.
  2. The training throughput for SHARE is much lower than LoRA (Table 3). Coupled with (1), the amount of complexity that the method adds (parameter sharing across layers is not condusive to distributed training like FSDP), and the fact that LoRA is a proven method with support across multiple frameworks makes the contribution limited.
  3. The paper misses related methods that share parameters across layers WeGeFT [1] and VB-LoRA [2]. The paper should include a comparison with both the methods.
  4. The paper does not reveal any details of how the assignment is done, and how the the permutation metrix $$R_s$ is defined. This makes it very hard to follow and understand the paper.

References

[1] Savadikar et al., "WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models"

[2] Li et al., "VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks"

Questions for authors
  1. How are the baseline hyperparameters tuned?
2 comments

We thank the reviewer for the constructive comments.


W1 & Q1. Baseline tuning

We apologize for the unclear baseline settings. LoRA used LR=1e-4 (following HiRA) and batch size 72, while HiRA and MoSA followed their reported settings (LR=1e-4, batch size 32; LR=1e-5, batch size 16, respectively).

To further examine LoRA's sensitivity to learning rate, we conducted an additional sweep on Llama-3-8B:

LoRA Learning RateGSM8KHumanEval
1e-4 (Original)71.1844.51
2e-473.3647.56
1e-362.6543.90

While tuning improves LoRA, its best results among the tested learning rates remain below those of other high-rank PEFT methods, including SHARE, on both GSM8K and HumanEval. Importantly, our original LoRA baseline is consistent with previously reported results. Comparable LoRA performance has been reported in prior studies, such as GateRA [1] and TLoRA [2], under similar experimental settings, further supporting the reasonableness of our baseline. Overall, these findings indicate that SHARE's performance advantage cannot be attributed solely to the original LoRA learning-rate choice.


W2. Training efficiency

We acknowledge that LoRA achieves higher raw training throughput. However, SHARE primarily aims to make high-rank adaptation more computationally efficient, rather than outperform LoRA in throughput alone. By directly reusing complete high-rank updates, SHARE avoids redundant update construction and, when inputs are shared, matrix multiplications. This yields 2.02–2.78× higher throughput than MoSA across three backbones while achieving stronger downstream performance. SHARE also supports multi-GPU training with DDP, achieving a step time of 596.1 ms on two GPUs (batch size 4 per GPU), compared with 1005.6 ms on a single GPU (batch size 8). While cross-layer sharing requires additional care under FSDP, it does not inherently prevent distributed training.


W3. Comparisons with related methods

We thank the reviewer for highlighting these relevant methods. We additionally evaluated VB-LoRA under our experimental setting:

MethodLlama-2 GSM8KLlama-2 HumanEvalLlama-3 GSM8KLlama-3 HumanEval
VB-LoRA49.4017.6871.9550.61
SHARE59.6727.2479.1959.76

SHARE achieves higher performance across all four settings.

Regarding the methodological distinction:

  • Existing sharing-based methods. Recent sharing-based PEFT methods, such as VeRA, ShareLoRA, and VB-LoRA, primarily leverage parameter sharing to reduce parameter or storage costs while retaining low-rank adaptation. WeGeFT further improves parameter, computation, and memory efficiency through shared low-rank weight generators.

  • SHARE. Rather than merely reducing parameter count, SHARE uses parameter sharing to organize adaptation capacity under a fixed budget. It directly reuses entire high-rank updates across layers and modules, allowing more parameters to construct each shared update while enabling computational reuse. This distinguishes SHARE from approaches that share low-rank factors or weight generators but construct separate updates for individual modules.


W4. Parameter assignment and the definition of RsR_s

We apologize for the insufficient implementation details. We clarify the parameter assignment and transformation operator below.

  • Parameter assignment. For each bank bb with an update matrix containing Mb=mbnbM_b=m_bn_b entries and KbK_b trainable parameters, we randomly assign matrix positions to trainable parameters, with approximately equal numbers of positions per parameter. This defines a fixed mapping κb(i,j)\kappa_b(i,j), where Db[i,j]=θb,κb(i,j)D_b[i,j]=\theta_{b,\kappa_b(i,j)}. The assignment is generated once before training and remains unchanged. All modules assigned to the same bank reuse this mapping and the resulting update matrix.

  • Transformation operator. The operator RsR_s applies either identity or transposition to match the target weight shape, i.e., Rs(Db)=DbR_s(D_b)=D_b or Rs(Db)=Db⊤R_s(D_b)=D_b^\top. It is a deterministic shape-matching operation, rather than a learned or randomly generated transformation.

Unlike methods that share only the factors used to construct module-specific updates, SHARE directly reuses complete weight updates based on shape compatibility, even when the modules perform different functions. The same update can induce distinct functional effects through different hidden representations and pretrained computations. The Single-Bank experiment further demonstrates effective parameter sharing across heterogeneous modules using different fixed assignments (83.75% vs. 84.07% accuracy).

[1] Jie Ou et al., "GateRA: Token-Aware Modulation for Parameter-Efficient Fine-Tuning." In AAAI, 2026.

[2] Weicheng Lin et al., "TLoRA: Task-aware Low Rank Adaptation of Large Language Models." In ACL, 2026.

Thank you for your rebuttal. While some of my concerns are addressed, a few still remain.

  1. Hyperparameter tuning: I appreciate the authors running a learning rate sensitivity check. However, I recommend the authors run a transparant hyperparameter tuning phase for all baseline methods and benchmarks to compare under fair setting, and describe it in the revised version. Untill the baselines are tuned well (for example, details about hyperpareter tuning for each method, the train validation split, etc) and run with the exact same conditions, the gains cannot be attributed to the proposed method

  2. Comparisons: I recommend adding comparisons with WeGeFT and VB-LoRA both, with the tuned baselines.

I also recommend the authors provide a motivation for why high rank tuning is required, since for most practical applications, low rank tuning has proven to be sufficient.

Given the gaps in the tuning setup and lack of clear motivation for why high rank tuning is necessary, I will retain my score.

Reviewer_31PSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

By organizing target weight matrices based on their update shapes, the proposed method shares a compact set of trainable parameter banks across modules and layers and involves sharing of high-rank updates.

Critical strengths

Through a first-order analysis, the authors demonstrate via their proposed SHARE method that parameter sharing can be modeled as a fixed mapping from shared parameters to model weights, allowing them to derive the corresponding weight updates. Because this analysis applies to any fixed sharing pattern, it remains independent of the specific relationships between sharing layers or modules. This flexibility enables the grouping of weight matrices by their update shapes, allowing a single high-rank update to be constructed once and reused by multiple matrices. Furthermore, when these matrices share both the same update and the same input, the underlying matrix multiplication can also be reused. SHARE directly shares the high-rank update across multiple weight matrices.

Critical weaknesses
  1. Comparisons and connections with some more high rank based adaptation methods such as https://openreview.net/forum?id=eKngc0MXXP and connections to structured sparsity for modeling high-rank updates would give more insights.
  2. The results of 94.39, 94.29, 94.44 in Table 8, say under hell-swag or the average column, look so close by. How do you posit that this is substantial improvement?
Questions for authors

1.) How are gradients from multiple weight matrices using the same bank accumulated correctly? Are these treated as sparse full weight matrices?

2.) How is the stable rank so low at around 1.5? Is it an artefact of the dataset and model used?

1 comment

We thank the reviewer for the constructive comments and insightful questions.


W1. Connections to high-rank adaptation and structured sparsity

We appreciate the suggested related work and address both aspects below.

  • Comparison with high-rank adaptation. We additionally evaluate ABBA (ICLR 2026), a recent high-rank PEFT method, under our experimental setting:
MethodLlama-2 GSM8KLlama-2 HumanEvalLlama-3 GSM8KLlama-3 HumanEval
ABBA58.8528.0577.0857.32
SHARE59.6727.2479.1959.76

SHARE outperforms ABBA in three of the four settings, further demonstrating its competitiveness against recent high-rank adaptation methods.

  • Distinction from structured sparsity. The key difference lies in how the adaptation space is constrained. Sparse fine-tuning typically restricts which weight entries can be updated, whereas SHARE uses a fixed sparse assignment matrix HbH_b to tie multiple entries to shared trainable parameters. Consequently, SHARE produces dense, potentially high-rank updates rather than restricting the support of weight modifications.

More importantly, SHARE extends this parameter tying across layers and modules by directly reusing complete weight updates. This constrains how different modules adapt, rather than independently selecting trainable weights within each module. Under a fixed parameter budget, SHARE can construct finer updates and reuse them efficiently across modules.


W2. Small differences in Table 8

We agree that the differences are small, which is precisely the intended observation. Table 8 evaluates SHARE's robustness to different fixed-coefficient constructions, rather than seeking improvements from any particular choice. The comparable performance across all-one, random-sign, and continuous coefficients (83.82%–84.07%) indicates that effective adaptation does not require carefully designed module-specific scaling.

This finding is consistent with SHARE's shape-based sharing design. Although modules may serve different functions, a shared weight update can induce distinct effects through different hidden representations and pretrained computations. The results suggest that SHARE can effectively reuse updates across these contexts without relying on carefully tuned coefficients to differentiate modules.


Q1. Gradient accumulation across shared banks

Gradients are automatically accumulated through the shared computational graph. Each bank constructs a dense update tensor from its trainable parameter vector using a fixed index mapping. When multiple modules reuse this tensor, their gradient contributions are summed into the corresponding shared parameters during backpropagation. No separate trainable sparse full-weight matrices are maintained, and caching preserves the computational graph and correct gradient accumulation.


Q2. Low stable rank of LoRA

Stable rank measures the concentration of singular-value energy rather than the number of nonzero singular values. A rank-32 LoRA update can therefore exhibit a low stable rank when its spectral energy is concentrated in the leading singular directions.

  • Original analysis. Figure 2 shows rapidly decaying singular-value spectra on Llama-2-7B, with LoRA's average stable ranks of 1.51 (attention) and 1.44 (FFN), compared with 58.0 and 102.1 for SHARE, respectively.

  • Additional analysis. We further examined rank-32 LoRA updates on Llama-3-8B fine-tuned with MetaMath100K. Across five sampled layers, the average stable ranks are 2.19 (attention) and 2.35 (FFN). These results show that strong spectral energy concentration also occurs with a different backbone and mathematical reasoning task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.